The data deluge driven by Next Generation Sequencing is transforming life sciences and its computational needs

Simon RasmussenAssistant Professor

Center for Biological Sequence AnalysisDepartment of Systems Biology

Technical University of Denmarksimon@cbs.dtu.dk

Helicobacter acinonychis str Sheeba

Helicobacter pylori P12

Helicobacter pylori B8

Helicobacter pylori 26695

Helicobacter pylori G

27H

elicobacter pylori B38H

elicobacter pylori HPAG

1H

elicobacter pylori Shi470H

elicobacter pylori J99H

elicobacter cinaedi CC

UG

18818H

elicobacter hepaticus ATCC

51449H

elicobacter mustelae 12198

Helicobacter bilis ATC

C 43879

Helicobacter pullorum

MIT 98−5489

Helicobacter canadensis M

IT 98−5491H

elicobacter wingham

ensis ATCC

BAA−430W

olinella succinogenes DSM

1740

Cam

pylobacter concisus 13826C

ampylobacter curvus 52592

Cam

pylobacter rectus RM

3267C

ampylobacter show

ae RM

3277

Cam

pylobacter fetus subsp fetus 82−40C

ampylobacter hom

inis ATCC

BAA−381C

ampylobacter gracilis R

M3268

Sulfurospirillum deleyianum

DSM

6946N

itratiruptor sp SB155−2

Sulfurimonas denitrificans D

SM 1251

Arcobacter nitrofigilis DSM

7299Arcobacter butzleri R

M4018

Sulfurovum sp N

BC37−1

Nautilia profundicola Am

H

GU

649V1.CD

18.3G

U649V1.C

D35.0

Fusobacterium sp D

11

Fusobacterium sp 3 1 33

Fusobacterium sp 7 1

Fusobacterium nucleatum

subsp nucleatum ATC

C 25586

Fusobacterium nucleatum

subsp nucleatum ATC

C 23726

Fusobacterium sp 3 1 27

Fusobacterium sp 4 1 13

Fusobacterium sp 3 1 36A2

Fusobacterium sp 2 1 31

Fusobacterium sp 1 1 41FAA

Fusobacterium periodonticum

ATCC

33693Fusobacterium

sp D12

Fusobacterium gonidiaform

ans ATCC

25563

Fusobacterium sp 3 1 5R

Fusobacterium varium

ATCC

27725

Fusobacterium ulcerans ATC

C 49185

Fusobacterium m

ortiferum ATC

C 9817

Sebaldella termitidis ATC

C 33386

Leptotrichia goodfellowii F0264

Leptotrichia hofstadii F0254

Leptotrichia buccalis C−1013−b

Streptobacillus moniliform

is DSM

12112

Nostoc punctiform

e PCC

73102

Nostoc sp PC

C 7120

Anabaena variabilis ATCC

29413N

ostoc azollae 0708Trichodesm

ium erythraeum

IMS101

Cyanothece sp PC

C 7425

Thermosynechococcus elongatus BP−1

Acaryochloris marina M

BIC11017

Synechococcus elongatus PCC

7942

Synechococcus elongatus PCC

6301Synechocystis sp PC

C 6803

Cyanothece sp PC

C 8802

Cyanothece sp PC

C 8801

Cyanothece sp ATC

C 51142

Cyanothece sp PC

C 7424

Microcystis aeruginosa N

IES−843

Synechococcus sp PCC

7002

cyanobacterium U

CYN−A

Synechococcus sp WH

8102

Synechococcus sp CC

9605Synechococcus sp C

C9902

Synechococcus sp WH

7803

Synechococcus sp CC

9311Prochlorococcus m

arinus str MIT 9303

Prochlorococcus marinus str M

IT 9313Prochlorococcus m

arinus str MIT 9211

Prochlorococcus marinus subsp m

arinus str CC

MP1375

Prochlorococcus marinus str N

ATL2A

Prochlorococcus marinus str N

ATL1A

Synechococcus sp RC

C307

Prochlorococcus marinus str M

IT 9312

Prochlorococcus marinus str M

IT 9215

Prochlorococcus marinus str AS9601

Prochlorococcus marinus str M

IT 9301Prochlorococcus m

arinus str MIT 9515

Prochlorococcus marinus subsp pastoris str C

CM

P1986

Synechococcus sp JA−3−3Ab

Synechococcus sp JA−2−3Ba(2−13)

Gloeobacter violaceus PC

C 7421

GU

729MH

0021

GU

967MH

0067

GU

768V1.CD

19.0

GU

715MH

0183G

U439M

H0043

GU

484V1.UC

40.0G

U815M

H0137

GU

815O2.U

C44.0

GU

815O2.U

C44.2

GU

196MH

0038G

U306V1.C

D28.0

Rothia mucilaginosa DY−18

Rothia mucilaginosa ATCC 25296

Rothia dentocariosa ATCC 17931

Kocuria rhizophila DC2201Arthrobacter sp FB24

Arthrobacter chlorophenolicus A6

Arthrobacter aurescens TC1

Renibacterium salmoninarum ATCC 33209

Micrococcus luteus NCTC 2665

Micrococcus luteus SK58

Brevibacterium mcbrellneri ATCC 49030

Kytococcus sedentarius DSM 20547

Clavibacter michiganensis subsp sepedonicus

Clavibacter michiganensis subsp michiganensis NCPPB 382

Leifsonia xyli subsp xyli str CTCB07

Kineococcus radiotolerans SRS30216

Mobiluncus mulieris 28−1

Mobiluncus mulieris ATCC 35243

Mobiluncus curtisii ATCC 43063Actinomyces odontolyticus ATCC 17982

Actinomyces odontolyticus F0309

Actinomyces coleocanis DSM 15436

Actinomyces urogenitalis DSM 15434

Actinomyces sp oral taxon 848 str F0332

Arcanobacterium haemolyticum DSM 20595

Cellulomonas flavigena DSM 20109

Sanguibacter keddieii DSM 10542

Xylanimonas cellulosilytica DSM 15894

Jonesia denitrificans DSM 20603

Beutenbergia cavernae DSM 12333

Brachybacterium faecium DSM 4810

Frankia sp EAN1pec

Frankia alni ACN14a

Frankia sp CcI3

Geodermatophilus obscurus DSM 43160

Kribbella flavida DSM 17836

Nocardioides sp JS614

Aeromicrobium marinum DSM 15272

Propionibacterium freudenreichii subsp shermanii CIRM−BIA1

Propionibacterium acnes J139

Propionibacterium acnes J165

Propionibacterium acnes KPA171202

Propionibacterium acnes SK187

Propionibacterium acnes SK137

Bifidobacterium bifidum NCIMB 41171

GU234V1.CD36.0

Bifidobacterium longum subsp infantis ATCC 15697

Bifidobacterium longum subsp longum ATCC 55813

Bifidobacterium longum subsp infantis CCUG 52486

Bifidobacterium longum subsp longum F8

Bifidobacterium longum DJO10A

Bifidobacterium longum NCC2705

Bifidobacterium longum subsp longum JDM301

Bifidobacterium breve DSM 20213

GU69V1.CD36.0

Bifidobacterium adolescentis ATCC 15703

Bifidobacterium adolescentis L2−32

Bifidobacterium pseudocatenulatum DSM 20438

Bifidobacterium catenulatum DSM 16992

Bifidobacterium dentium Bd1

Bifidobacterium dentium ATCC 27678

Bifidobacterium angulatum DSM 20098

Bifidobacterium animalis subsp lactis AD011

Bifidobacterium animalis subsp lactis DSM 10140

Bifidobacterium animalis subsp lactis Bl−04

Bifidobacterium gallicum DSM 20093

Gardnerella vaginalis ATCC 14019

Gardnerella vaginalis 409−05

Parascardovia denticolens F0305

Scardovia inopinata F0304

Tropheryma whipplei str TwistTropheryma whipplei TW0827

Tsukamurella paurometabola DSM 20162Rhodococcus jostii RHA1Rhodococcus opacus B4Rhodococcus erythropolis PR4Rhodococcus erythropolis SK121

Rhodococcus equi ATCC 33707

Nocardia farcinica IFM 10152Gordonia bronchialis DSM 43247

Mycobacterium abscessus ATCC 19977Mycobacterium sp JLSMycobacterium sp KMS

Mycobacterium sp MCSMycobacterium smegmatis str MC2 155

Mycobacterium gilvum PYR−GCKMycobacterium vanbaalenii PYR−1

Mycobacterium tuberculosis F11Mycobacterium tuberculosis KZN 1435

Mycobacterium tuberculosis H37Rv

Mycobacterium tuberculosis CDC1551

Mycobacterium tuberculosis H37Ra

Mycobacterium bovis BCG str Tokyo 172

Mycobacterium bovis BCG str Pasteur 1173P2

Mycobacterium bovis AF212297

Mycobacterium marinum MMycobacterium ulcerans Agy99

Mycobacterium parascrofulaceum ATCC BAA−614

Mycobacterium avium subsp paratuberculosis K−10

Mycobacterium avium 104

Mycobacterium leprae TN

Nakamurella multipartita DSM 44233Actinosynnema mirum DSM 43827Saccharopolyspora erythraea NRRL 2338

Saccharomonospora viridis DSM 43017

Corynebacterium tuberculostearicum SK141Corynebacterium pseudogenitalium ATCC 33035

Corynebacterium accolens ATCC 49725

Bacteroides ovatus SD CC 2a

Bacteroides xylanisolvens SD C

C 1b

Bacteroides sp D1

Bacteroides sp 2 1 22

Bacteroides xylanisolvens XB1A

Bacteroides ovatus SD CMC 3f

Bacteroides ovatus ATCC 8483

Bacteroides sp 2 2 4Bacteroides sp D2

Bacteroides caccae ATCC 43185

Bacteroides finegoldii DSM 17565

Bacteroides thetaiotaomicron VPI−5482

Bacteroides sp 1 1 6Bacteroides fragilis NCTC 9343

Bacteroides fragilis YCH46

Bacteroides sp 2 1 16Bacteroides sp 3 2 5

Bacteroides fragilis 3 1 12

Bacteroides cellulosilyticus DSM 14838

Bacteroides intestinalis DSM 17393

Bacteroides sp D20

Bacteroides uniformis ATCC 8492

Bacteroides eggerthii DSM 20697

Bacteroides stercoris ATCC 43183

GU633M

H0143Bacteroides vulgatus PC510

Bacteroides sp 4 3 47FAA

Bacteroides vulgatus ATCC 8482

Bacteroides dorei DSM 17855

Bacteroides sp 3 1 33FAA

Bacteroides dorei 5 1 36D4

Bacteroides sp 9 1 42FAABacteroides coprocola DSM

17136

Bacteroides coprophilus DSM 18228

Bacteroides plebeius DSM 17135

GU702M

H0047

GU702M

H0135G

U462V1.CD38.0

GU116M

H0047

GU116M

H0006G

U755V1.CD19.0

GU617M

H0046

GU5226O

2.UC43.0

GU891M

H0057Prevotella tannerae ATCC 51259

GU474MH0006

Prevotella bergensis DSM 17361

GU924MH0069

Prevotella bivia JCVIHMP010

Prevotella melaninogenica ATCC 25845

Prevotella melaninogenica D18

Prevotella veroralis F0319

GU164V1.UC56.0

Prevotella copri DSM 18205

Prevotella buccae D17

Prevotella oris F0302

GU1320MH0057

GU1320O2.UC57.0

GU301V1.CD13.0Prevotella buccalis ATCC 35310

Prevotella timonensis CRIS 5C−B1

Prevotella sp oral taxon 472 str F0295

Prevotella sp oral taxon 317 str F0108

Prevotella sp oral taxon 299 str F0039

GU255MH0011

GU255V1.UC55.4

GU1185MH0107

GU1058V1.CD19.0

GU592MH0168

GU520MH0045

GU520MH0012

Prevotella ruminicola 23

GU20MH0012

GU20MH0061

GU51O2.UC37.0

GU118V1.CD15.3

Parabacteroides merdae ATCC 43184

Parabacteroides johnsonii DSM 18315

Bacteroides sp 2 1 7

Bacteroides sp 2 1 33B

Parabacteroides sp D13

Parabacteroides distasonis ATCC 8503

GU2MH0020

GU2MH0074

GU279MH0020

GU279O2.UC18.2

Porphyromonas uenonis 60−3

Porphyromonas endodontalis ATCC 35406

Porphyromonas gingivalis ATCC 33277

Porphyromonas gingivalis W

83

GU1031V1.CD20.4

GU927V1.CD29.0

GU927O2.UC40.2

GU927O2.UC40.0

GU873O2.UC60.0

GU485O2.UC60.0

Candidatus Azobacteroides pseudotrichonymphae genomovar CFP2

GU67O2.UC48.2

GU67MH0012

Alistipes putredinis DSM 17216

GU29MH0002

GU29MH0074

Alistipes shahii WAL 8301

GU268MH0054

GU157V1.UC11.5

GU14MH0012

GU14O2.UC48.2

GU788MH0016

GU788V1.UC49.1

GU561O2.UC51.2

GU561V1.UC49.1GU709MH0158

GU770MH0006

GU770MH0022

GU545MH0009

GU435MH0012

GU514MH0009

GU514MH0031

GU1060MH0044

GU831MH0143

GU831MH0071

Pedobacter heparinus DSM 2366

Sphingobacterium spiritivorum ATCC 33300

Sphingobacterium spiritivorum ATCC 33861

Cytophaga hutchinsonii ATCC 33406

Dyadobacter fermentans DSM 18053

Spirosoma linguale DSM 74

Flavobacterium psychrophilum JIP0286

Flavobacterium johnsoniae UW101

Croceibacter atlanticus HTCC2559

Gramella forsetii KT0803

Zunongwangia profunda SM−A87

Robiginitalea biformata HTCC2501

Capnocytophaga ochracea DSM 7271

Capnocytophaga sputigena ATCC 33612

Capnocytophaga gingivalis ATCC 33624

Flavobacteriaceae bacterium 3519−10

Chryseobacterium gleum ATCC 35910

Chitinophaga pinensis DSM 2588

Candidatus Amoebophilus asiaticus 5a2

Blattabacterium sp (Periplaneta americana) str BPLAN

Blattabacterium sp (Blattella germanica) str Bge

Candidatus Carsonella ruddii PV

Ruminococcus gnavus ATCC 29149

Candidatus Sulcia muelleri GWSS

Candidatus Sulcia muelleri DMIN

Candidatus Sulcia muelleri SMDSEM

Salinibacter ruber

Salinibacter ruber DSM 13855

Rhodothermus marinus DSM 4252

Chlorobium luteolum DSM 273

Chlorobium phaeovibrioides DSM 265

Pelodictyon phaeoclathratiforme BU−1

Chlorobium limicola DSM 245

Chlorobium phaeobacteroides DSM 266

Chlorobium chlorochromatii CaD3

Chlorobaculum parvum NCIB 8327

Chlorobium tepidum TLS

Prosthecochloris aestuarii DSM 271

Chlorobium phaeobacteroides BS1

Chloroherpeton thalassium ATCC 35110

Gemmatimonas aurantiaca T−27

Fibrobacter succinogenes subsp succinogenes S85

Chlamydia trachomatis AHAR−13

Chlamydia trachomatis BTZ1A828OT

Chlamydia trachomatis DUW−3CX

Chlamydia trachomatis BJali20OT

Chlamydia trachomatis L2bUCH−1proctitis

Chlamydia trachomatis 434Bu

Chlamydia muridarum Nigg

Chlamydophila pneumoniae J138

Chlamydophila pneumoniae TW−183

Chlamydophila pneumoniae CWL029

Chlamydophila pneumoniae AR39

Chlamydophila felis FeC−56

Chlamydophila caviae GPIC

Chlamydophila abortus S263

Candidatus Protochlamydia amoebophila UWE25

Waddlia chondrophila WSU 86−1044

GU154MH0012

GU154MH0002

GU154V1.CD31.0

GU344V1.CD7.4

Akkermansia muciniphila ATCC BAA−835

Methylacidiphilum infernorum V4

Opitutus terrae PB90−1

Coraliomargarita akajimensis DSM 45221

Rhodopirellula baltica SH 1

Pirellula staleyi DSM 6068

Planctomyces limnophilus DSM 3776

Borrelia burgdorferi B31

Borrelia burgdorferi ZS7

Borrelia afzelii PKo

Borrelia garinii PBi

Borrelia turicatae 91E135

Borrelia hermsii DAH

Borrelia recurrentis A1

Borrelia duttonii Ly

Treponema vincentii ATCC 35580

Treponema denticola ATCC 35405

Treponema pallidum subsp pallidum SS14

Treponema pallidum subsp pallidum str Nichols

Leptospira biflexa serovar Patoc strain Patoc 1 (Ames)

Leptospira biflexa serovar Patoc strain Patoc 1 (Paris)

Leptospira borgpetersenii serovar Hardjo−bovis L550

Leptospira borgpetersenii serovar Hardjo−bovis JB197

Leptospira interrogans serovar Copenhageni str Fiocruz L1−130

Leptospira interrogans serovar Lai str 56601

Brachyspira hyodysenteriae WA1

Brachyspira murdochii DSM 12563

Elusimicrobium minutum Pei191

uncultured Termite group 1 bacterium phylotype Rs−D17

Thermosipho melanesiensis BI429

Thermosipho africanus TCF52B

Fervidobacterium nodosum Rt17−B1

Thermotoga petrophila RKU−1

Thermotoga naphthophila RKU−10

Thermotoga sp RQ2

Thermotoga maritima MSB8

Thermotoga neapolitana DSM 4359

Thermotoga lettingae TMO

Kosmotoga olearia TBF 1951

Petrotoga mobilis SJ95

Dictyoglomus turgidum DSM 6724

Dictyoglomus thermophilum H−6−12

Coprothermobacter proteolyticus DSM 5265

Candidatus Cloacamonas acidaminovorans

Dehalococcoides ethenogenes 195

Dehalococcoides sp VS

Dehalococcoides sp GT

Dehalococcoides sp CBDB1

Dehalococcoides sp BAV1

Dehalogenimonas lykanthroporepellens BL−DC−9

Sphaerobacter thermophilus DSM 20745

Thermomicrobium roseum DSM 5159

Thermobaculum terrenum ATCC BAA−798

Chloroflexus sp Y−400−fl

Chloroflexus aurantiacus J−10−fl

Chloroflexus aggregans DSM 9485

Roseiflexus castenholzii DSM 13941

Roseiflexus sp RS−1

Herpetosiphon aurantiacus DSM 785

Synergistetes bacterium SGP1

Aminobacterium colombiense DSM 12261

Anaerobaculum hydrogeniformans ATCC BAA−1850

Thermanaerovibrio acidaminovorans DSM 6589

Pyramidobacter piscolens W5455

Jonquetella anthropi E3 33 E1

Meiothermus ruber DSM 1279

Meiothermus silvanus DSM 9946

Thermus thermophilus HB8

Thermus thermophilus HB27

Deinococcus deserti VCD115

Deinococcus geothermalis DSM 11300

Deinococcus radiodurans R1

Truepera radiovictrix DSM 17093

Life science data deluge• Massive unstructured

data from several areas DNA, patient journals, proteomics, imaging, ...

• Impacts Industry, Environment, Health

• Societal grand challenges

• Cheap sequencing technologies results in explosion of DNA data

What does DNA do?How to make a car? Car blueprint

What does DNA do?How to make a human? DNA

DNA contains the information on how to create an organism!

DNA: strings...GGATCAGCTGACTCGCCTGGCTCTGAGCCCCGCCGCCGCGCTCGGGCTCCGTCAGTTTCCTCGGCAGCGGTAGGCGAGAGCACGCGGAGGAGCGTGCGCGGGGGCCCCGGGAGACGGCGGCGGTGGCGGCGCGGGCAGAGCAAGGACGCGGCGGATCCCACTCGCACAGCAGCGCACTCGGTGCCCCGCGCAGGGTCGCGATGCTGCCCGGTTTGGCACTGCTCCTGCTGGCCGCCTGGACGGCTCGGGCGCTGGAGGTACCCACTGATGGTAATGCTGGCCTGCTGGCTGAACCCCAGATTGCCATGTTCTGTGGCAGACTGAACATGCACATGAATGTCCAGAATGGGAAGTGGGATTCAGATCCATCAGGGACCAAAACCTGCATTGATACCAAGGAAGGCATCCTGCAGTATTGCCAAGAAGTCTACCCTGAACTGCAGATCACCAATGTGGTAGAAGCCAACCAACCAGTGACCATCCAGAACTGGTGCAAGCGGGGCCGCAAGCAGTGCAAGACCCATCCCCACTTTGTGATTCCCTACCGCTGCTTAGTTGGTGAGTTTGTAAGTGATGCCCTTCTCGTTCCTGACAAGTGCAAATTCTTACACCAGGAGAGGATGGATGTTTGCGAAACTCATCTTCACTGGCACACCGTCGCCAAAGAGACATGCAGTGAGAAGAGTACCAACTTGCATGACTACGGCATGTTGCTGCCCTGCGGAATTGACAAGTTCCGAGGGGTAGAGTTTGTGTGTTGCCCACTGGCTGAAGAAAGTGACAATGTGGATTCTGCTGATGCGGAGGAGGATGACTCGGATGTCTGGTGGGGCGGAGCAGACACAGACTATGCAGATGGGAGTGAAGACAAAGTAGTAGAAGTAGCAGAGGAGGAAGAAGTGGCTGAGGTGGAAGAAGAAGAAGCCGATGATGACGAGGACGATGAGGATGGTGATGAGGTAGAGGAAGAGGCTGAGGAACCCTACGAAGAAGCCACAGAGAGAACCACCAGCATTGCCACCACCACCACCACCACCACAGAGTCTGTGGAAGAGGTGGTTCGAGAGGTGTGCTCTGAACAAGCCGAGACGGGGCCGTGCCGAGCAATGATCTCCCGCTG...

Human: 3 bill

Bacteria: 4 mill

Virus: 10k

A, C, G and T

Some sequencing examples

• Sequence 1000s of human genomes: who are we, who are you, drug effects, diseases, cancers, ...

• Sequence environmental samples, thousands of different bacteria: novel enzymes, the human microbiome, bacteria producing electricity

The sequencing data avalanche

• Computer speed and storage capacity is doubling every 18 months and this rate is steady

• DNA sequence data is doubling every 6-8 months over the last 3 years!

Distributed data productionWorld wide >900 centers

Data transfer and storage becomes

an issue

GenomeDK (KU, DTU, AU)

We ship harddrives...

>60 Pb pr year

http://omicsmaps.com

What does this mean?

• First human genome draft in 2001, final 2004

• Estimated costs $3 billion, time 13 years

• Today: 1 week, $8000

• Towards $1000 genome

Storage and analysis

Highest cost is not the experimentbut storage and analysis

A standard human (30-40x) whole-genome sequencing exp. would create

150 Gb (compressed) data

High strain on IO - read/writing GB->TBs

Analysis: Two basic approaches

• Alignment: We compare to a known genome

• de novo assembly: The genome is unknown we must create it ourselves

• Algorithm development

• is very dynamic - code optimization no longer vital

• What we used 2 years ago we don’t use today!

Alignments: Human data

• Using a known reference genome to assemble our data

• Where does the each of the 100-character strings match in the genome?

• Originally hash based algorithms - problem: high memory demand and slow

1.2 bill 100-character DNA-strings

3.2 bill genome

BWT alignment

• Burrows Wheelers Transformation (known from bzip2)

• Reversible transformation rearranging a character string into runs of similar characters

• Important because genomes tend to have many similar strings!

• Combine with suffix arrays to quickly find all possible matches

• High speed, high precision, low memory usage

Human project example• 51 human genomes from around the world

• Compute cluster resources used >20 CPU-years, 43 Tb storage

• >30 of algorithms/software used for only this project

• Application pipelines (sequential code and data flow) - need versatile compute facilities!

• Time to solution is key

• Competing with Stanford University, we could outperform them on compute time - we publish

• Time matters! - Accessibility matters!

de novo assembly algorithms• If no genome is known for a species we need

to make it

• Graph theory - de bruijn graphs

• Example: Polarbear

• Raw data: ~4 bill 100-character DNA strings

• Eg. total 400 bill characters!

• Originally: All vs. all comparisons - no chance

De Bruijn graph assemblers

• Directed graph of overlapping items (here DNA sequences)

• Graph is created by 1 pass of the data and assembly by walking Eulerian path

• Lots of RAM required (up to 2Tb or more)

• Data is unstructured - placement of each data string can be anywhere in the graph!

• High strain on communication between nodes in SMP systems!

Example genome de Bruijn graphs

only a handful of near-identical repeats longer than 200 bp (Fig. 3), whereas complex genomes, such as the human, usually have their repeat length determined by whether there has been an active LINE or SINE transposable element (usually around 4 kb in length for the former and between 500 bp and 1 kb for the lat-ter). As the ability to produce longer read pairs (also referred to as ‘mate pairs’ to distinguish them from the shorter read pairs) has only recently been optimized for next-generation technologies, assemblies of complex genomes have been rare.

The other main barrier for large, complex genome assem-blies is the memory overhead for these methods. Although the de Bruijn data structure is compressed, all the methods use some sort of adjunct data structures in addition to the core de Bruijn graph to map the reads to the graph. These adjunct structures are critical for leveraging additional information required for accurate assemblies, such as read pair information.

sequence length from a reference assembly. The read lengths need only be over the k-mer length to generate a reasonable assembly (in theory, k must be over 15 bp, though in practice 19 is the lowest sensible k-mer, and larger k-mers are always better, although at the expense of having to generate more coverage to support these large k-mer sizes).

The first assembler to exploit this technology was Roche’s 454 assembler, Newbler, which adapted the scheme specifically to handle the main source of error in 454 sequencing—namely, ambiguity in the length of homopolymer runs. In late 2007 and early 2008, sev-eral second-generation de Bruijn graph assemblers were released for very short reads, compatible with the Solexa technology, includ-ing SHARCGS27, VCAKE28, VELVET29, EULER-SR30, EDENA31, ABySS32 and ALLPATHS33. Some of these methods, such as VELVET, EULER-SR and ABySS, explicitly use de Bruijn graphs, whereas other methods implicitly explore a de Bruijn graph—for example, constrained by read-pair behavior, as in ALLPATHS. The methods differ in how they treat errors and to what extent they use read-pair information. Read pairs are defined as two short DNA sequence reads generated from different ends of a longer DNA molecule—for example, 35-bp reads generated from both ends of a 500 bp frag-ment. One does not know the identity of the sequence between the read pairs, but one usually has an estimate of the length of the inter-vening sequence. As it is only marginally more expensive to generate short reads in read-pair format than as single reads, extremely high coverage of read pairs is routinely available. The more advanced de Bruijn graph assemblers29,30,32,33 can use read pairs to provide long assemblies. A particular challenge has been the two-base-encoding ‘color space’ of ABI SOLiD technology. In this two-base encoding, a single error produces a systematic translation error on all subse-quent decoding of the bases for the rest of the read. In the context of an alignment, such an encoding scheme can be integrated into the alignment routine, and there is an argument that the double base encoding provides better discrimination between errors and observed differences. In de novo assembly, however, there is no ref-erence. The solution has been to perform the assembly directly in color space and then ‘key’ the resulting color space assembly to one of the four feasible base-pair assemblies using either a small amount of traditional sequence or the presence of a known base at the start of each SOLiD read.

Whichever sequencing technology and assembly method are used, the ability to provide long assemblies critically requires that at least a proportion of the read pairs are longer than the longest common near-identical repeat in the genome. This var-ies considerably between genomes. Bacterial genomes often have

Linear stretches

Tips

! ! ! ! ! ! ! !

! ! !

! ! ! !

! ! ! ! ! !

!

! ! ! !

!

!!!!!!!!!

a

b

1. Sequencing (for example, Solexa or 454)

2. Hashing

3. Simplification of linear stretches

4. Error (tip and bubble) removal Bubble

!

Figure 3 | Constructing and visualizing a de Bruijn graph of a DNA sequence. (a) An example de Bruijn graph assembly for a short genomic sequence without polymorphism. Sequence at top represents the genome, which is then sampled using shotgun sequencing in base space with 7-bp reads (step 1). Some of the reads have errors (red). In step 2, the k-mers in the reads (4-mers in this example) are collected into nodes and the coverage at each node is recorded. There are continuous linear stretches within the graph, and the sequencing errors create distinctive, low-coverage features through out the graph. In step 3, the graph is simplified to combine nodes that are associated with the continuous linear stretches into single, larger nodes of various k-mer sizes. In step 4, error correction removes the tips and bubbles that result from sequencing errors and creates a final graph structure that accurately and completely describes in the original genome sequence. (b) A full de Bruijn graph from a bacterial genome that shows the general lack of repetitive structure within the entire genome.

S10 | VOL.6 NO.11s | NOVEMBER 2009 | NATURE METHODS SUPPLEMENT

REVIEW

Figure 2.9: Graphical representation of the de Bruijn of the Streptococcussuis genome

In this representation, node sequences are represented as curves, whichconnect at their tips.

2.7.2 Local reference based visualisation

It quickly became clear that troubleshooting some of the algorithms de-scribed in the following chapters would require some adequate visualisa-tion techniques. Given that most tests were done on species with a knownreference, it was possible to use this sequence as a guide to the graph’scomplexity.

The first solution consisted in following the path of the referencesequence through the graph, recording the properties of the nodes beingtraversed. Figure 2.10 represents the length and multiplicity of successivenodes on the reference path.

In this diagram, long contigs are interrupted by two types of breaks.Sometimes, two long contigs are separated by a very short, isolated fea-

39

Simple genome

A bit more complex genome

>1Tb RAM580 days of compute

>5 Tb storage

Conclusions• The data deluge is fundamentally changing life science and the

required computational resources

• Analysis requires High Performance Computing facilities, CPU, Memory, Storage, IO and fast data links

• Time to solution - need accessible compute resource

• Dynamic algorithm development - very fast algorithm turnaround

• A need for shared compute (cloud) and storage facilities - computable storage

Acknowledgements

• Center for Biological Sequence analysis (DTU)

• Søren Brunak

• John Damm Sørensen

• Bent Petersen

Preprocessing and SNP calling Natasja S. Ehlers, PhD student Center for Biological Sequence Analysis Functional Human Variation Group