BENE Jo Vanvoorden

Upgrade je Blackboard 7.2

servers naar 9.0

Implicaties van de upgrade

voor systeembeheerders.

Overzicht

• Upgrade pad• Voorbereiding• Servers• Timeframe systeem beheerders• Doing the job• Performance + bevindingen• Nazorg

Upgrade pad

• 7.2.383.0 => 7.2.383.45• 7.2.383.45 => 8.0.260.7• 8.0.260.7 => Xythos Unique ID tool• 8.0.260.7 => 9.0.351.4• 9.0.351.4 => 9.0.505.0• 9.0.505.0 => 9.0.505.0 + Patchset + MR

Voorbereiding

• Gestart met testfase upgrade 8• Pad verder uitgeschreven voor BB9• Verschillende versies getest• Clean install• Full install• 2 FTE +2maanden fulltime restore + testen• Uitbouw hulpmiddelen

– Patch deploy script– Smsmewakker script– Comparesystems script– Installblackboardfromscratch script

Servers

• Applicatie Servers ?

Cygnus (albireo, sadir,gienah,azelfafage,ruchba)

Lyra ( tolbb9app2t)

tolbb9dev1t

tolbb9dev2t

tolbb9dev3t

tolbb9dev4t

tolbb9dev5t

tolbb9dev6t

Muliphein (tolbb9app3t)Tolbb9app1t

Tolbb9app4tAlshain,altair

Canis-major (sirius,mirzam)

Servers

• Applicatie Servers ?• Database Servers

– Tolora10db6, elune, polaris– Tolora10db1t, tolora10db2t, tolora10db3t,tolora10db4t,

tolora10db5t, altair

• Storage– Backend Netapp -> full system restores– DB -> full restores 5 test db servers (500G/server)

• Hardware– Full restores => fysieke hardware– Development restores => vmware esx

Timeframe systeem beheerders

• Testen :– Run1

– Run 2 : 14 h 30– Run 3 : 15 h– Run …13h

upgrade 7.2 -> 7.2.383.45

18min?

upgrade 7.2 -> 8

40min

UNIQUE ID TOOL

10.34 -> 16h gestopt opnieuw gestart in screen

16h -> 21h.... gestopt met errors

2010-01-20T20:53:44Z] Trace XythosSession ContextImpl:201 Transaction Aborted com.xythos.storageServer.api.StorageServerException: ContextImpl:199 Transaction Aborted

(com.xythos.storageServer.api.EntryReadException: FileSystemEntryException on entry bbcms//users/j-1013000000027004/calculations.doc

upgrade 8 -> 9.0.351.4

7h

upgrade 9.0.351.4 -> 9.0.505.0

1.5h

Doing the job

• Vrijdagavond 12 februari - 20.00– Cold backup db + app + snapshots backend, gdlo down +

redirect in place, start restore reference system

• 21.07 DB in noarchivelog + andere voorbereidingen• 21.21 PONR

– 7.2.383.0 => 7.2.383.45

• 21.56– Reconfigure bb-tasks, disable PA ,next installer ( => 8.0.184.3 )

• 22.29– Reconfigure bb-tasks, Unique ID tool

Doing the job

• Zaterdagmorgen• 04.40

– Xythosid finished -> patchen ongedaan maken

• 5.08– java upgrade, classpath aanpassen, blackboard aanpassen +

volgende installer (9.0.351.4)

• 6.33– Custom patches, upgrade java, new init script

• 7.04– next installer (9.0.505.0)

• 7.32– Patchset

Doing the job

• Zaterdagmorgen– Migration reminders

• Loadbalanced specific aanpassingen• Custom authentication aanpassingen (shib, import/export)• Queued tasks• Permgensize• Wrapper.conf• Nieuwe certificaten• Nieuwe firewall (startup booster)• Nieuw loadbalancer check script• Restore client files• Allowlinking• Sql tuning

• 9.20 => Blackoard release naar toledo team

Nazorg

• Zaterdag– Bijstaan toledo team – performance

• Zondag– 9.00 – 19.30 Verdere afwerking

• Collab server – https• Undotbs, temptbs shrinken• End-user performance tuning (oracle EM)• Deploy on all servers in cluster

– 19.30 – Blackboard available for end users

• Lopend– Upgrade applicatie servers RHEL4 => RHEL5– Upgrade database server 10.2.0.3 => 10.2.04– End user experience (slow)

Performance + bevindingen

• New = cool– +/- 2 weken overload op de servers

• Gemiddeld 300 CPS/ cluster (cluster 4-6 servers)• Na upgrade 250 CPS/ server

Iedereen wil het nieuwe zien

• Garbage collection problemen– Stop the world ...

• Behind.blackboard.com– Informatie, maar niet altijd even duidelijk

• Veel tuning nodig aan de databank

Vragen

• Vragen ?