#SummitNow

The Art of the UpgradeKyle AdamsSolutions Engineer at Alfresco@kylefadams

#SummitNow

#SummitNow

Objectives• Arm you with the strategy and

tactics to perform a successful upgrade of the core Alfresco repository.

#SummitNow

#SummitNow

Strategy and Tactics"Strategy without tactics is the slowest route to victory.” “Tactics without strategy is the noise before defeat.” – Sun Tzu, The Art of War

#SummitNow

#SummitNow

Strategy and Tactics“...tactics are the actual means used to gain an objective, while strategy is the overall campaign plan…”[1]

[1] http://en.wikipedia.org/wiki/Tactic_(method)

#SummitNow

#SummitNow

Upgrade StrategyStrategy: Accurately predict how my Alfresco instancewill behave throughout the upgrade process.

Tactic:Prepare

Tactic:Execute

Tactic:Monitor

Tactic:Validate

#SummitNow

#SummitNow

Upgrade StrategyStrategy: Accurately predict how my Alfresco instancewill behave throughout the upgrade process.

80% of Work

Tactic:Prepare

Tactic:Execute

Tactic:Monitor

Tactic:Validate

#SummitNow

#SummitNow

Prepare"To ... not prepare is the greatest of crimes; to be prepared beforehand for any contingency is the greatest of virtues.”[1]

[1] Sun Tzu, The Art of War

#SummitNow

#SummitNow

Prepare – Current State• Count of all database tables• Size and number of files in content

store• Size of lucene or Solr indexes• Custom Content Models• Custom Process definitions• Installed modules• Customizations

#SummitNow

#SummitNow

Prepare – Upgrade Path• http://docs.alfresco.com/4.2/topic/com.alfresco.enterprise.doc/con

cepts/upgrade-path.html

#SummitNow

#SummitNow

Prepare – Backup/Restore• Backup Prod Database

• Restore into new instance• Backup Prod lucene/solr indices

• Restore into new instance• Backup Prod ContentStore

• Restore into new instance

#SummitNow

#SummitNow

Prepare – Backup/Restore• Backup Prod ContentStore• What if your ContentStore is too

large to restoring into a lower environment for testing?

#SummitNow

#SummitNow

Prepare – Backup/Restore• No excuse!• We can mock up the ContentStore

against the database using small sample files.

• https://github.com/kylefernandadams/alfresco-mock-content-generator • CAUTION: Only use this for

testing purposes.

#SummitNow

#SummitNow

Prepare – Alfresco Instances• Install an out-of-the-box Alfresco instance

for each upgrade segment• Current Version: 3.3.4• Target Version: 4.2.0• Path: 3.3.4 -> 3.3.5 -> 4.2.0

• Don’t forget to request license files for each minor version from Alfresco support!

• Point each instance to common content store, database, and indices.

• Do not perform an in-place upgrade!

#SummitNow

#SummitNow

Prepare – Alfresco Instances• Do not perform an in-place upgrade!

#SummitNow

#SummitNow

Prepare – Patches• Know exactly which patches will run

during each segment of the upgrade.

• alfresco/WEB-INF/classes/alfresco/patch/patch-services-context.xml

• Patch Java docs:http://dev.alfresco.com/resource/docs/java/repository/org/alfresco/repo/admin/patch/impl/package-summary.html

#SummitNow

#SummitNow

Prepare – Patches• Still unsure? • Perform a dry run of the upgrade

with OOTB installation.• Set logging to debug for the

following packages• org.alfresco.repo.admin.patch• org.alfresco.repo.domain.patch

• Review the logging output

#SummitNow

#SummitNow

Prepare – Customizations• Only include bare minimum

extensions and overrides (Example: content model)• Restore a Prod backup into the

extensionRoot

#SummitNow

#SummitNow

Prepare – Tools for Monitoring• Logs

• tail, text editor, Splunk• JVM

• YourKit, JProfiler• DB Queries

• Vendor-specific tools (MySQL Workbench, SQL Developer, PgAdmin, SQL Server Mgmt Studio)

• Quickly toggle log4j logging for java.sql.*

#SummitNow

#SummitNow

Prepare – Tools for Monitoring• Hardware & OS (CPU, Memory, Disk,

etc)• top, nmon, ProcessExplorer

#SummitNow

#SummitNow

Upgrade StrategyStrategy: Accurately predict how my Alfresco instancewill behave throughout the upgrade process.

Tactic:Prepare

Tactic:Execute

Tactic:Monitor

Tactic:Validate

#SummitNow

#SummitNow

Execute• Validate your newly restored

instance is working as expected. • Example: 3.3.4

• Shutdown the instance• Startup the instance for the first

segment of the upgrade. • Example: 3.3.4 -> 3.3.5

#SummitNow

#SummitNow

Upgrade StrategyStrategy: Accurately predict how my Alfresco instancewill behave throughout the upgrade process.

Tactic:Prepare

Tactic:Execute

Tactic:Monitor

Tactic:Validate

#SummitNow

#SummitNow

Monitor - Logs• Based on logging output from your

dry run, you should be able to determine the overall progress of your upgrade.

• What if the logging output appears to be stalled?

#SummitNow

#SummitNow

Monitor - Logs• Stalled?

#SummitNow

#SummitNow

Monitor - JVM• Leverage a Java profiling tool such

as YourKit• YourKit will tell you exactly which

class and methods are being called.

• What if it appears that the upgrade has stalled on a given method call?

#SummitNow

#SummitNow

Monitor – JVM• Still Stalled???

#SummitNow

#SummitNow

Monitor – SQL Queries• Use your DB vendor-specific tools to

monitor the queries being executed• Don’t have MySQL Workbench, SQL

Developer, PgAdmin, or SQL Server Mgmt Studio?• Quickly toggle log4j logger for

java.sql.*• You must add java.sql to custom-

log4j.properties

#SummitNow

#SummitNow

Upgrade StrategyStrategy: Accurately predict how my Alfresco instancewill behave throughout the upgrade process.

Tactic:Prepare

Tactic:Execute

Tactic:Monitor

Tactic:Validate

#SummitNow

#SummitNow

Validate – Smoke Tests• Test basic Alfresco functionality.• Test one or more of your major use

cases. • For the latest 4.x versions, leverage

WebDrone.

#SummitNow

#SummitNow

Upgrade StrategyStrategy: Accurately predict how my Alfresco instancewill behave throughout the upgrade process.

Tactic:Prepare

Tactic:Execute

Tactic:Monitor

Tactic:Validate

Repeat for each segment

#SummitNow

#SummitNow

Gotchas – Cyclical ACL’s• FixAclInheritancePatch could fail with

StackOverflowException during 3.4 upgrade.

• Added cyclical ACL detection in 3.4.13

• Jira: MNT-2580

#SummitNow

#SummitNow

Gotchas – Orphaned Nodes• Orphaned Nodes and Associations to

deleted nodes• Reindex could fail with

“InvalidNodeRefException: Node does not exist”

#SummitNow

#SummitNow

Gotchas – Orphaned Nodes• Find orphaned nodes:

SELECT n.*FROM alf_node n, alf_qname q, alf_namespace nswhere q.id = n.type_qname_idand ns.id = q.ns_idand not (ns.uri = 'http://www.alfresco.org/model/system/1.0' AND q.local_name = 'deleted')AND NOT EXISTS (SELECT 1 FROM alf_store where root_node_id = n.id)AND NOT EXISTS (SELECT 1 FROM alf_child_assoc where child_node_id = n.id)

#SummitNow

#SummitNow

Gotchas – Orphaned Nodes• Find associations to deleted nodes:

SELECT a.*FROM alf_child_assoc a, alf_node n, alf_qname q, alf_namespace nswhere (n.id = a.child_node_id or n.id = a.parent_node_id)and q.id = n.type_qname_idand ns.id = q.ns_idand ns.uri = 'http://www.alfresco.org/model/system/1.0' AND q.local_name = 'deleted'

#SummitNow

#SummitNow

Gotchas – Orphaned Nodes• CAUTION: Do not attempt to fix

these by hacking the database!• Detection and fix via Java has been

added as of 4.1.4• Jira: MNT-8531

#SummitNow

#SummitNow

Gotchas – Nodes without Txn’s• May cause schema updates to fail

with *ContraintViolationExceptions.

#SummitNow

#SummitNow

Gotchas – Nodes without Txn’s• Find nodes without transactions:SELECT *

FROM alf_node node WHERE node.transaction_id NOT IN (

SELECT id FROM alf_transaction);

#SummitNow

#SummitNow

Gotchas – Nodes without Txn’s• CAUTION: Do not alter the database

without the consent or the assistance of Alfresco Support or Alfresco Consulting.• Fix:UPDATE alf_node set transaction_id = (SELECT MIN(id) from alf_transaction) WHERE transaction_id NOT IN (SELECT id FROM alf_transaction);

#SummitNow

#SummitNow

Gotchas – Long-running Patches• MigrateVersionStorePatch (2.x 3.x)• Run asynchronously as a scheduled

job# Never: * * * * * ? 2099# Once every thirty minutes: 0 0/30 * * * ?version.store.migrateVersionStore.cronExpression=* * * * * ? 2099# Limit number of version histories to migrate per job cycle, where -1 = unlimited. Note: if limit > 0 then need to schedule job to run regularly in order to complete the migration.version.store.migrateVersionStore.limitPerJobCycle=-1version.store.migrateVersionStore.runAsScheduledJob=false

#SummitNow

#SummitNow

Gotchas – Long-running Patches• ContentUrlConverterPatch (...

3.2.x)• Run asynchronously as a scheduled

job# Never: * * * * * ? 2099# Once every thirty minutes: 0 0/30 * * * ?system.content.contentUrlConverter.cronExpression=* * * * * ? 2099system.content.contentUrlConverter.threadCount=2system.content.contentUrlConverter.batchSize=500system.content.contentUrlConverter.runAsScheduledJob=false

#SummitNow

#SummitNow

Gotchas – Long-running Patches• NodeDeleted.sql (… 4.1.x)• Renames alf_node to t_alf_node• Recreate the alf_node table without

the node_deleted column• Insert all rows from t_alf_node into

alf_node• Drop the temporary t_alf_node table

#SummitNow

#SummitNow

Gotchas – Database Statistics• Don’t forget to recompute your

database statistics!• Without histograms if you’re

using Oracle

#SummitNow

#SummitNow

Summary• Be meticulous!• Have attention to detail!• Leave nothing up to mystery!

• There should be no surprises.• “Know thy self, know thy enemy. A

thousand battles, a thousand victories”[1]

• [1] Sun Tzu, The Art of War

#SummitNow

#SummitNow

Summary• “The supreme art of war is to

subdue the enemy without fighting.”[1]

• [1] Sun Tzu – The Art of War

#SummitNow

#SummitNow

Contact Information• Kyle Adams• Solutions Engineer at Alfresco• Email: kyle.adams@alfresco.com• Twitter: @kylefadams

#SummitNow