What I Learned Writing a Trillion $

Trading System

Nick Zeeb & Antonio Molina @NickZeeb @ajmolinacom

23rd October 2015

$$

Challenges

{ FX Market }

Source: http://www.fxkeys.com, April 29th, 2015 by Chris Pottorff

{ $1.9 Trillion / year }{ $7.3 Billion / day }{ $5.0 Million / minute }

Challenges

{ spikes of 250,000 transactions / second }

7

Challenges

{ Innovation }{ Risk of Failure }

{ simpler is better }

Plain Old Java(JavaSE)

{ integrate integrate integrate }

12

{ feedback }

{ embargo }

Nick and Antonio at GeeCon Prague!

{ invest }- Virtualized dedicated system- Acceptance Tests : 664 cores, 4.9TB RAM, 5.4

TB SSD Storage, 3.3TB SAS Storage, 10.1TB NL-SAS Storage.

- Commit build: 80 cores, 128GB RAM, 1.2TB Storage.

- Staging: 240 cores, 2.3TB RAM, 4.8TB SSD Storage, 24TB NL-SAS.

- Native 10gig networking throughout CI.

Classification: Public

{ everyone is a tester }

Go Crazy on Tests

Acceptance Tests Integration tests

Unit tests

Software unit

Unit tests

Software unit

Unit tests

Software unit

{ test like crazy }

Client

{ acceptance tests }

{ acceptance tests }

@Testpublic void shouldNotLoginWithInvalidCredentials(){ tradingUI.login("user: demo", "password: NotCorrect1.",

"failureMessage: BAD_CREDENTIALS");

logsAPI.checkCustomerAuditLogContains("user: demo", "expression: Failed log on: BAD_CREDENTIALS");

}

{ clean logs }

20

{ reliability }

@Testpublic void shouldFailOverToSecondaryMatchingEngine() {

//when coordinator.killService("primary matching engine") tradingGateway.placeOrder("id: myOrder",

"side: buy", "price: 10",

"quantity: 10"); //then tradingGateway.waitForOrder("id: myOrder",

"expectStatus: OK");}

{ reliability }

{ performance }

{ what does NOT work }

O(x^2)

{ be a scientist }

{ performance tests }

Microbenchmarks

(class, algo level)

Macrobenchmarks

(whole system)

shorter time to feedback

longer time to feedback

less realistic more realistic

Mesobenchmarks

(business operation

level)

{ microbenchmarks }

{ microbenchmarks }public BigDecimal myDivision(long quantity){ BigDecimal bd = BigDecimal.valueOf(quantity)

.setScale(SCALE, UNNECESSARY);

return bd.divide(DIVISOR, UNNECESSARY);}

public BigDecimal myDivision(long quantity){ BigDecimal bd = BigDecimal.valueOf(quantity); return bd.divide(DIVISOR);}

{ microbenchmarks }

{ macrobenchmarks }- Representative of production.- We need load generators:

- Statistical models.- Spiky scenarios.- Maintained load (red line).

- Real usage from journals.

{ releases }

Revision X

Revision X+1

{ releases }

Software changesdata migration

{ compatibility }download & import data -> migrate -> run test

@Testpublic void shouldHaveASecurityRoleForEveryRoleIdInUse() {

Collection<Integer> validIds = SecurityRole.getIds();

roleDao.getAllRolIdInUse().stream().forEach(roleId -> {

assertThat("Unexpected role ID on the DB: " + roleId, validIds.contains(roleId), is(true));

} );}

{ compatibility }

{ staging release }

Deploy the current production revision to staging

Load production data (sanitized)

Migrate staging to the new revision that we are going to

deploy

START and CHECK system is working as expected

Host1

Host3 Host2

Host4

Real Exchange(with $$$)

Testing in Live Exchange

{ testing in live }physical view of the system logical view of the system

Testing in Live(set of representative

Acceptance Test)

{ other tests... }{ static analysis }{ invariant test }

{ configuration test }{ disaster recovery test }

{ culture }

Have Standup Meetings

{ standup }

Swarm on Stories

{ swarm }

Collectively Own the Code

{ everyone owns the code }

Pair Program

{ pair }

Don’t be a Jerk

{ be nice }

{ rotate }

Take Time to Sharpen Your Tools

{ sharpen your tools }

Share Your Knowledge

{ share your knowledge }

Share Your Code { share your code }

{ share your process }

@NickZeeb@ajmolinacom

Be Pragmatic

{ be pragmatic }

{ last responsible moment }

Avoid Irreversible Decisions

{ avoid irreversible decisions }

It’s OK to Change Your Mind

{ it’s OK to change your mind }

The Business Must Trust You{ trust }

The Most Important Thing You’ll Do

{ find the right people }

55

{ thanks! ;) }

@ajmolinacom@NickZeeb

Images thanks to

Classification: Public

• Icons and predesigned images: Pixabay http://www.pixabay.com• Rafael Matsunaga https://www.flickr.com/photos/rednuht/ • Philippe Lewicki https://www.flickr.com/photos/lewicki/ • Martin Fisch https://www.flickr.com/photos/marfis75/ • Talus https://www.flickr.com/photos/nocallerid_man/ • Kevin Jarrett https://www.flickr.com/photos/kjarrett/ • Jeffrey Beall https://www.flickr.com/photos/denverjeffrey/ • Improve It https://www.flickr.com/photos/improveit/ • Where is wally image https://upload.wikimedia.org/wikipedia/commons/2/27/Where%E2%80%

99s_Wally_World_Record_(5846121513).jpg• Vajrapani666

https://commons.wikimedia.org/w/index.php?title=User:Vajrapani666&action=edit&redlink=1 • MECU https://commons.wikimedia.org/wiki/User:MECU • Ehsan Lebatihem https://www.flickr.com/photos/lebatihem/ • Massimo Valiani https://www.flickr.com/photos/leader_maximo/ • Skip Steuart https://www.flickr.com/photos/omnidirectional/ • Karl-Ludwig Poggemann https://www.flickr.com/photos/hinkelstone/ • Open Clipart https://openclipart.org • Miguel Librero https://www.flickr.com/photos/dark_imp666/ • Police Cars of the Ohio Valley & the Great Lakes https://www.flickr.com/groups/1597741@N24/pool/38396373@N08/• Gaming4JC at English Wikipedia http://commons.wikimedia.org/wiki/File:3DCrystal_ball.png• some guy called Darren https://www.flickr.com/photos/59239214@N08/ • JD Hancock https://www.flickr.com/photos/jdhancock/