The experiences of migrating a large scale, high performance healthcare network

April 2, 2008 SlipstreamUSA 1

The experiences of migrating a large scale, high performance healthcare network

Larry WilliamsCorporate Manager, Partners HealthCare


In the next half hour…

Partners Healthcare System overview Caché platform architecture & metrics The need to migrate Phased migration approach Benchmark testing and results Discoveries and production enhancements


Partners Healthcare System

Founded in 1994 Brigham & Women’s Hospital Massachusetts General Hospital

Now includes: Community physician network (1200 + 3500 MD’s) PCHi 3 community hospitals 2 rehab hospitals 3 specialty institutions

Enterprise-wide Information Systems 1100 employees Annual budget FY05 approximately $160 million


BWH

MGH

Logan Airport

10 km

6 km

Anchor Hospitals & Airport

http://www.brighamandwomens.org/

http://www.mgh.harvard.edu/

http://www.massgeneral.org/

http://www.brighamandwomens.org/default.asp



MGH

BWH

Newton-Wellesley

Acute Care Hospitals

Community PhysicianPractices

http://www.brighamandwomens.org/

http://www.mgh.harvard.edu/

http://www.massgeneral.org/



http://www.nwh.org/index.asp


Partners Domain Devices

Internet

12,000 Printers

1,450 Servers

32,000 Desktops

Firewall

Closely Managed

~30,000 other devices

Assumed Managed


Windows Production Architecture

3.5 TB


Enterprise Integration

# of

Interfaces

Est. Annual

Transactions

Daily

AverageChange from prior

year

2004 167 610,833,080 1,673,515

2005192 887,649,802 2,431,917 45%

2006170

1,240,712,044 3,399,211 40%

2007 196 1,330,962,017 4,659,035 37%

Over 30% are to and from Caché database


Integration Components


Gigabytes in Use


Annual Database Growth Rate

21%

37%

19%

24%

30%

37%

46%

39%

2000 2001 2002 2003 2004 2005 2006 2007


Database Utilization

Average Database References per day in Billions


The Need to Migrate - Availability

Uptime In Hours In Minutes 99 % 7.3

99.5 % 3.6

99.75% 1.8

99.9 % 44

99.99 % 4.4

99.999% 26 seconds

Monthly Downtime

Current State

Business need


Additional Business Requirements

Increase availability and reliability Decrease database risk from 5 single points of failure More robust hardware and OS Many less servers and OS instances to manage Clustering and automated failover Reduce monthly maintenance needs, updates once or twice per year

--------------------------------------------------------

Improve Performance 64 bit OS, more memory for cache

Caché 5.0.20 to Caché 2008.1, significantly improved ECP performance Increase Scalability

91 Terabytes available on EMC SAN DMX3

On-demand addition of processor cores


Caché Migration Decision Making Process

Only considered first tier vendors and support (IBM, HP) HP assumed much more risk with Professional Services Existing HP business yields more leverage & visibility with

regional office More headroom in HP configuration Price was not a distinguishing factor


Phased migration approach

Proof of Concept (benchmark testing) Completed 10/15/07

Phase 1 – Database tier 4 of 5 servers migrated, anticipated completion 4/14/08

Phase 2 – Application tier Big Bang migration 12/14/08

Phase 3 – Disaster Recovery January 2009


UNIX Benchmark Environment


Database Benchmark Load Testing Results

Goals Simulate current Production user counts &

transaction loads Verify support for load increases up to 300%

Benchmark Environment Isolated LAN, new DMX3 SAN 20 new Windows blade servers (10 app servers,

10 script ‘players’) Scripts for 8 apps (represent heaviest use,

Web/Telnet/VB apps) 2 batch jobs (screensaver simulation, NullGen

LMR functions)

Conclusions Able to simulate production load, 1.5x and 3x

load 2 HP rx8640 can handle growth projections

Metric Production peak (8/21, 11:20 am)

Benchmark “paced” script load

Benchmark full script load

Database Global Refs / sec.

35,000 30,000 135,000

LMR transactions (5 min. period)

11,806 40,000 40,000

LMR avg Caché app time (in sec.)

0.32 0.15 0.66


Design and Configuration Considerations

Database configuration simulation testing 1 to 5 Caché database instances were assessed 1 vs. 5 ECP channels per Caché instance were assessed Number of active cores were accessed (4 active, 2 reserved)

Results and unexpected discoveries Identify 5 Caché database instance as optimal design configuration

Journal synch bottleneck the biggest issue

o High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance)

Maintain same data distribution across 5 DB instances Determine 1 ECP channel per instance optimal

Additional channels did not improve throughput, still have only 1 Journal Deamon


Benchmark Discoveries led to Production Improvements

References to Undefined globals using $Data and $Get These commands require network round trip

Use of $increment Each call to $I requires network round trip

Excessive use of Cache locks Forces more than 1 round trip

Use of large strings Strings that require more than 3900–4000 bytes to represent the

string value are big strings and never cached on the ECP client.

Lesson Learned - Each trip to the database server results in overhead caused by a Journal Synch. Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions .


75% reduction in long running transaction


Phased Migration Approach


Monthly Average Caché Web Transaction Time


Application Models

Old NewBrowser

client

Web server

Cache

Cache

VB client

.Net server

CacheCache

.Net client

Browser client

Web server

Cache Web Services

Browser client.Net client

Scalability/Connection pooling, robustness/error handling,

Vism

Managed Obj.Vism.ocx

Managed Obj.

Cache Web Services

WebLink


The experiences of migrating a large scale, high performance healthcare network

Larry WilliamsCorporate Manager, Partners HealthCare

Date post:	21-May-2015
Category:	Health & Medicine
Upload:	georgejames
View:	816 times
Download:	1 times

The experiences of migrating a large scale, high performance healthcare network

Health & Medicine