Distributed Trading

SWE 622Spring 2009

Group Members

Dave Johnson Akhil Pathania Sohrab Rezvan Nick Ton

Overview

System Design Experiments and Measurements

System Design

Peer to Peer Technology,Asynchronous Messaging

Message Types

Registration Ping: heartbeat to check for

connectivity Ping Acknowledge Time: used to calculate offset Buy Sell

Trader/Exchange Heartbeat

Reason To check if Trader or Exchange goes down Also use to check if Trader or Exchange come

back up Benefits

Timely notification of when a Trader or Exchange goes down

Consequence Ping messages are queued there; is currently

no way to flush the queue of backlog messages

More network traffic

/** * Sends Ping message to all exchange. This reduces thread overhead. */private void sendPing() {

StringBuffer ping = new StringBuffer();ping.append(MessageType.PING);ping.append(",");ping.append(System.currentTimeMillis());ping.append(",");ping.append(this.traderIP);ping.append(",");ping.append(this.traderPort);ping.append(",");ping.append(this.traderName);

Enumeration<String> en = connectionMap.keys();while (en.hasMoreElements()) {ExchangeInfo exInfo = connectionMap.get(en.nextElement());send(ping.toString(), exInfo.exchangeAddress, Integer.parseInt(exInfo.exchangePort));}}

private void sendPingEcho(ExchangePingMessage epm){StringBuffer ping = new StringBuffer();ping.append(MessageType.PINGACK);ping.append(",");ping.append(System.currentTimeMillis());ping.append(",");ping.append(this.traderIP);ping.append(",");ping.append(this.traderPort);ping.append(",");ping.append(this.traderName);

send(ping.toString(), epm.getExchangeIP(), epm.getExchangePort());}

Time Between Trade and Other Activity (Sell 21)

In JFrameTraderGUI.java

public void actionPerformed(ActionEvent e) {final String request = e.getActionCommand();... else if(request == IOperationsConsole.__SELL21){ String exchange = _exchange.getSelectedItem().toString(); String product = _product.getText(); String price = _price.getText(); String qt = _qt.getText();... LOGGER.info("<====EXPERIMENT:1 START>"); LOGGER.info("<====EXPERIMENT:1 capture trades 2 - 21 elapsed time>"); long elapsedTime = 0; long time = 0; long curTime = 0; for(int i=1; i<=21;i++){ _trader.sell(exchange, product, price, qt); curTime = System.currentTimeMillis(); elapsedTime = curTime - time; time = curTime; LOGGER.info("<====Experiment:1 Trade:" + i + ":ms since last trade. " + elapsedTime ); } LOGGER.info("<====EXPERIMENT:1 END>"); }}

Exchange: Parse Incoming Messages

public void receive(String message) { String mType = MessageType.getMessageType(message); boolean stale = false;

if (mType.equals(MessageType.REGISTRATION)) { RegistrationMessage rm = new RegistrationMessage(message); addRemoteEndpoint(rm.getTraderIP(), Integer.parseInt(rm.getTraderListenPort())); sendRegistrationAck(rm); } else if(mType.equals(MessageType.TRADE)){ TradeMessage tm = new TradeMessage(message);

if (tm.getOperation().equals(MessageType.BUY)){ if (productList.contains(tm.getProduct()) == true){ gui.logOperation(tm.getTraderName(), tm.getEchoNumber(), tm.getOperation(), tm.getProduct(), tm.getPrice(), tm.getQt(), stale); sendTradeEcho(tm); }else{ tm.rollback(); //decrements the echo number counter } }else if (tm.getOperation().equals(MessageType.SELL)){ if (!productList.contains(tm.getProduct())){ productList.add(tm.getProduct()); } gui.logOperation(tm.getTraderName(), tm.getEchoNumber(), tm.getOperation(), tm.getProduct(), tm.getPrice(), tm.getQt(), stale); sendTradeEcho(tm); } }else if (mType.equals(MessageType.PING)){ TraderPingMessage tpm = new TraderPingMessage(message); sendPingEcho(tpm); }else if(mType.equals(MessageType.PINGACK)){

}else if(mType.equals(MessageType.TIME)){ TimeAckMessage ttm = new TimeAckMessage(message); Calendar cal = Calendar.getInstance(); String startTime = Long.toString(cal.getTimeInMillis() + cal.getTimeZone().getOffset(cal.getTimeInMillis())); send(new String(ttm.getOperation() + "," + startTime + "," + (cal.getTimeInMillis() + cal.getTimeZone().getOffset(cal.getTimeInMillis())) + ",” + exchangeName + "," + ttm.getId()), ttm.getTraderIp(), Integer.parseInt(ttm.getTraderPort())); }}

Exchange: Sending Error

public void errorSending(String message, String remoteHost, int remotePort, IOException exception) {

Calendar cal = Calendar.getInstance(); System.out.println("Error sending " + message); System.out.println("Removing remote host " + remoteHost); removeRemoteEndpoint(remoteHost, remotePort); iTraderCount-=1; bUpdateTraderInfo = true; if(iTraderCount==0) gui.updateTraderInfo(iTraderCount, 0); LOGGER.info(remoteHost + " was removed at :" + (cal.getTimeInMillis() + cal.getTimeZone().getOffset(cal.getTimeInMillis())));}

Exchange-Trader Offset Calculation The calculation is based on the method described in

“Distributed Systems: Principles & Paradigms” by Andrew S. Tanenbaum.

Figure 1, shows a similar Exchange-Trader arrangement, where:

T1 is the time when the request leaves the Trader. T2 is the time when Exchange receives the request. T3 is the time when Exchange sends the response

back to the Trader. T4 is the time when Trader receives Exchange

response.

Figure 1 Figure 2

Prototype Requirements Traders receive information about all correctly processed orders, and only about those.

All traders should display the orders processed by each exchange in the exact same (processing) order, regardless of the possible reordering of messages by the network.

Traders should receive information on orders no later than 10 seconds after an exchange processes them, and the sooner, the better. Information received after 10 seconds needs to be marked as stale.

Traders should be able to determine automatically how long ago an order was processed. Note: traders may not assume that the local timer is synchronized with the one on the exchange, unless your prototype takes definitive measures to make that happen.

In case a trader crashes, all other components should be able to continue operation. Exchanges should recognize the failure and stop echoing processed orders to the trader. Upon restart, and re-register, the trader should be readmitted to the system.

In case an exchange crashes, all other components should be able to continue operation. Traders should recognize the failure and send orders only to the surviving exchanges. Recovery mechanisms don’t need to be provided for exchange crashes.

Experiments and Measurements

Experiment 1 Latency between

starting to send an order and proceeding with other activities, averaged over at least 20 orders

Average latency: 2.68 ms (3 sets of 21 trades)

Maximum latency: 11 ms

Created a button on the GUI to initiate a sequence of 21 sells

Since our messages are sent from the trader asynchronously, we expected the latency to be very low

Experiment 2 Latency between

issuing an order and the last trader receiving it’s echo, averaged over at least 20 orders

Average latency: 473 ms Tested with 1 exchange

and 3 traders 1 trader sent all 21

trades Captured time message

was sent and each message was received, factoring in offsets

Experiment 3 Latency between a

trader failure and the last exchange stopping echoing orders, averaged over at least 5 failures

Average latency: 18 secs

Tested with 3 exchanges, 1 trader

Time of trader closing is written to the log file

When exchange detects failure (exception raised from message failure), time is recorded

Experiment 4a: Skew Average and max

observed during operation Clock skew

Average: 13.5 sec Max: 1 min 14 sec Each trader calculates

the offset with every exchange to which it is connected by “time” messages sent at startup

Configuration file includes parameter for number of hours to wait before calculating offset again

Experiment 4a: Offset

Akhil Dave Nick SohrabAkhil -03:17:875 -04:02:680 -00:01:195Dave 03:16:438 -00:46:609 03:17:532Nick 04:05:531 00:43:216 04:05:617Sohrab 00:01:226 -03:19:256 -04:03:875

Akhil Dave Nick SohrabAkhil -03:27:051 -03:55:781 -00:05:375Dave 03:23:218 -00:28:809 03:21:535Nick 03:53:516 00:28:740 03:50:422Sohrab 00:04:071 -03:21:014 -03:50:312

Offset: Thursday, April 16, 2009

Minutes : Seconds : Milliseconds

Offset: Monday April 27, 2009

Minutes : Seconds : Milliseconds

Experiment 4b: Latency Average and max

observed during operation network latency

Average: 189 ms (124 trades, 1 direction)

Max: 470 ms (124 trades, 1 direction)

On both exchange and trader, checkbox and spinner control allow for the simulation of network latency (on/off, 1 - 15 sec)

“…assume the propagation delays from A to B is roughly the same as B to A…”

Experiment 5 Storage footprint on

each exchange required to echo orders to traders, as a function of the number of traders (use empirical and/or analytical methods)

Storage footprint of Exchange: 1.61 MB

Storage footprint on Exchange per Trader: 59 KB

x = memory utilized by an idle exchange

y = memory used by exchange to communicate with each trader connected

n = number of traders connected to exchange

z = total memory used by exchange

z = x + ny y = (z – x) / n

Experiment 5: Memory Usage

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Number of Traders

MB

Development Measurements 1: Lines of Code

Product LOC Difference from Previous

Original P2P Software 6387 N/A

Original P2P Software (2 versions)* 12774 N/A

Prototype (Project 2) 14238 1464

Prototype with Experiments (Project 3) 14562 324

* P2P-calculate was copied to two versions (trader and exchange)

Development Measurements 2: Level of Effort

Design 42Development 120Experiments 57Presentation 27Total 246

Measurement 3 Network traffic: 1

Exchange, 1 Trader, approximately 5 min 37 sec

1 Registration 1 RegistrationAck 10 Time 10 TimeAck 34 Ping (trader) 34 PingAck (exch) 23 Ping (exch) 23 PingAck (trader) = 136 Messages