Building an OpenMRS Distribution

Developer lessons from KenyaEMR

Distribution?

• Thinking in terms of “distributions” appears a recent development in OpenMRS

• Nationwide implementations of OpenMRS are becoming more common (e.g. Rwanda)

• Managing 100 sites is a very different challenge to managing 10 sites

Challenges of scale

• Need for consistency across site installations• Need for simple or automated upgrades and

maintenance• Need for scalable user support

Site inconsistency

• What we want to avoid:

• Endless different environments, most of which won’t have been tested

• Becomes impossible to manually track

Site OpenMRS Concepts Module 1 Module 2 …

Clinic 1 1.9.3 20140714 2.5.3 0.3

Clinic 2 1.9.3 20131215 2.4.1 1.0-BETA

Clinic 3 1.9.7 20131215 2.4 0.3.1

…

Site inconsistency

• More chance of site-specific bugs– Development team might not have tested a site’s

particular environment• Complex site-specific upgrade processes– Single components upgraded individually

• Confusion for users trained in different environments– National implementations often rely on

centralised training events

Site consistency

• A distribution should define a consistent environment

• We can break that down into components:– A version of OpenMRS core– A set of modules with specified versions– A set of metadata objects• Might be versioned SQL dumps or MDS packages• Might include a separately versioned concept

dictionary

Distribution example

• An implementation’s own custom modules will only be a part of the larger distribution:

MyModule

Reporting

Idgen

HtmlFormEntry

CIEL

MyDistro

OpenMRS

Distribution versioning

• Specific versioned releases of all the components make up a single versioned release of the distribution, e.g.

MyModule 1.0

Reporting 0.8

Idgen 2.6

HtmlFormEntry 2.5

CIEL 20120931

MyDistro 1.0

OpenMRS 1.9.3

MyModule 1.1

Reporting 0.8.1

Idgen 2.6

HtmlFormEntry 2.5.1

CIEL 20140107

MyDistro 2.0

OpenMRS 1.9.7

Distribution versioning

• At every stage of the development lifecycle, we should be working with a version of the complete distribution, e.g.

mydistro-2.0-SNAPSHOT

mydistro-2.0-RC1

mydistro-2.0-RC2

mydistro-2.0

Development with an “in-progress” version

Testing with release candidate versions

Installations and upgrades with released versions

Distribution modules

• Rather than manage a distribution as a separate project, it can be easier to tie it to a “distribution module”

• Its version is the distribution version• Defines the required versions of all other

components• Depends on all of the other modules

Distribution modules

• By requiring all of the other modules the, distribution module ensures that the distribution is always run as a whole

<require_modules> <require_module version="0.8.1"> org.openmrs.module.reporting </require_module> <require_module version="2.6"> org.openmrs.module.idgen </require_module> <require_module version="2.5.1"> org.openmrs.module.htmlformentry </require_module></require_modules>

Continuous integration

• Don’t want to develop different components in insolation and only realise integration problems during testing

• Developers should work with the latest version of the distribution

• CI server should be used to keep a testing server up to date with changes to any component

Buildable distributions

• Need to make it easy for developers (and CI servers) to deploy a particular version of a distribution

• Useful to have a zip archive of the different component modules

mydistro-2.0-SNAPSHOT mymodule-1.1.omod

reporting-0.8.1.omod

idgen-2.6.omod

htmlformentry-2.5.1.omodmyd

istro

-2.0

-dist

ro.z

ip

Buildable distributions

• Maven provides a convenient mechanism to produce an archive of a project with its dependencies – called an assembly

• Thus our distribution module can have two build outputs:– A regular omod– A distribution zip archive

For an example of how to implement this, see: https://github.com/I-TECH/openmrs-module-kenyaemr/blob/master/distro

Metadata consistency

• The idea of site consistency should apply also to metadata

• For example:– The distribution does patient registration– This saves encounters of type “Registration”– The distribution can’t function if that encounter

type doesn’t exist– Can we guarantee that the encounter type means

the same thing in different installations?

User managed metadata

• Non-distribution modules often expect the user to manage metadata, e.g.– Every time some code tries to access the

“Registration” encounter type, check for null– If it doesn’t exist, show the user an error message

to tell them to create it– Tell user to set the

mydistro.registrationEncounterType global property to reference the new object

Distribution managed metadata

• If we want to be sure that metadata is the same across all sites, we manage it via code rather than users

• Distributions should install required metadata automatically

• Distribution code should assume that the metadata exists– If it doesn’t, it is a developer problem rather than a

user problem

Fail fast assumptions

• If we assume that metadata always exists, we should fail-fast if that assumption turns out to be incorrect, e.g.

• Helps developers find problems right away• Easier than tracking down source of a NPE

EncounterType ret = Context.getEncounterService().getEncounterTypeByUuid(uuid);if (ret == null) { throw new IllegalArgumentException("No such encounter type with uuid " + uuid);}

The Metadata Deploy module provides fail-fast fetch methods for most metadata classes: https://github.com/I-TECH/openmrs-module-metadatadeploy

Metadata identity

• Database ids are not reliable for identifying the same metadata in different installations

• Anytime distribution code references metadata it should use one of:– UUID: all OpenMRS classes have these and they

can be kept consistent across installations– Reference terms: these are globally consistent and

unique identifiers for concepts (and soon also drugs)

Metadata installation

• One approach is to bundle metadata packages with the distribution and install these on startup

• Weaknesses of this approach:– Metadata is not easily readable or editable – Packages typically have to be managed on an

external server, exported and embedded into the code

– Package installation is slow so usually not appropriate to use the same packages in unit tests

Metadata deploy

• Module was developed to address these issues– Allows metadata to be defined in code– Metadata is easy to read and edit– Fast installation suitable for unit tests

For more information about the Metadata Deploy module go to https://wiki.openmrs.org/display/docs/Metadata+Deploy+Module

Metadata deploy

• Metadata that’s too lengthy to be described in code can be loaded from CSV files etc– Still more readable than zip archives

• Support for synchronization of large sets– Used to synchronize OpenMRS locations with all

9500 facilities in the Kenya Master Facility List:• Clean database synchronization: 1min• Subsequent synchronization: 3-4secs• Previous MDS package load: 20-25mins

Concepts?

• So far SQL dumps have been best for these:– We don’t manage/edit them directly (we use CIEL)– Database dump is only quick way to install

50,000+ concepts• Groovy script used used to generate XML

dataset file of just those concepts used by KenyaEMR– Used for unit tests that need access to same

concepts as production environment

Example metadata bundle

@Component@Requires({ BaseMetadata.class })public class MyMetadata extends AbstractMetadataBundle { public static final class _EncounterType { public static final String ENCOUNTER_TYPE1 = "d3e3d723-7458-4b4e-8998-408e8a551a84"; } public static final class _Form { public static final String FORM1 = "4b296dd0-f6be-4007-9eb8-d0fd4e94fb3a"; public static final String FORM2 = "89994550-9939-40f3-afa6-173bce445c79"; } @Override public void install() { install(encounterType("Encounter Type #1", "Something...", _EncounterType.ENCOUNTER_TYPE1)); install(form("Form #1", null, _EncounterType.ENCOUNTER_TYPE1, "1", _Form.FORM1)); install(form("Form #2", null, _EncounterType.ENCOUNTER_TYPE1, "1", _Form.FORM2)); // A form that should be retired if it exists uninstall(possible(Form.class, "73d34479-2f9e-4de3-a5e6-1f79a17459bb"), "Because..."); }}

This “bundle” installs an encounter type

and two forms

Also retires a form that’s no longer needed

Enforcing consistency

• Current OpenMRS UI wasn’t made for this idea of a distribution

• Ideally we want to prevent even the super user account from doing things like:– Stopping modules– Deleting or modifying metadata

• KenyaEMR overrides the regular UI and provides a custom UI without this functionality

Installations and upgrades

• OpenMRS itself is usually only one part of a functioning EMR installation

• Other parts might be:– The database client and server– The JVM– Tomcat or another Java web app server– Database backup scripts run by cron jobs– Help and training materials

Installations and upgrades

• For small or single site implementations, developers often do these

• Not feasible for large implementations:– Developers can’t physically visit every site– Sites often have connectivity issues– Upgrades need to be performed by less technical

users

Installation via virtual machine

• Working installations require correct configuration of all those parts

• Easier to configure once at the office than 100s of times at each site

• Can build virtual machine images and clone for all sites

Installation via virtual machine

• Ongoing experiments with different ways of retaining patient data during upgrade– Moving data from old to new VM via SQL dump– Separate VMs for database (not replaced) and

web app (replaced)• Other experiments using bittorrent to

download VM images to sites– Works even when connectivity is very poor

Maintenance

• Things inevitably go wrong, e.g.– Sites could end up with invalid data due to a

software bug– Might need to convert data as software changes

• Modules can provide liquibase files to make one time changes– Not always easy or possible to do something with

just SQL

Automated fixes

• Requiring user intervention should be a last resort

• KenyaEMR provides its own automated way of making fixes called chores:– Java classes which perform a one-time job– Run at the end of KenyaEMR startup so have

access to all metadata and services

See KenyaEMR source for examples of chore classes: https://github.com/I-TECH/openmrs-module-kenyaemr/tree/master/api/src/main/java/org/openmrs/module/kenyaemr/chore

User support

• Need clear processes for issue tracking:– More technical site staff can create support tickets

directly in ticketing system– Less technical site staff can use email or phone

and then support staff will create ticket• Help materials need to be

easily accessible to site users

User support

• KenyaEMR integrates with external help site

• Users can lookup help documents and videos

• Can provide context sensitive help within different apps

Help site code available at https://github.com/I-TECH/helpsite

Bahati nzuri!