Steve Poole IBM

JVM Support for Multitenant Applications

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES.

ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S

CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.

IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS

Important Disclaimers

!Works at IBM’s Hursley Laboratory in the UK

Involved in IBM Java VM development since before Java was 1

Currently leading IBM’s OpenJDK technical engagement

Steve Poole

JVM Support for Multi-Tenant Applications

What this talk is about

!!!!This experimental technology is being developed to help address important pressures on Java !Google ‘IBM Java 8 beta’ for more information !

■ By the end of this session, you should be able to: !

– Understand what multitenancy is and what it’s good for !

– Describe the challenges of multitenant Java deployments !

– Understand ideas for new JDK features to convert existing applications into multitenant deployments

!

Agenda

1. What Multi-tenancy is all about aka how to simplify to save time and money

2. The challenges of building a multi-tenant application in Java 3. How to keep your application running in a multi-tenant world -

dealing with bad behaviour 4. Risk vs. Reward: How dense can we go? 5. Wrap-up: Summary, and next steps

Part 1What is Multitenancy?

!(simplify to save time and money)

Who owns one of these?

Who owns one of these?

Who owns one of these?

Who owns one of these?

At a basic level Multitenancy is a drive to

reduce unnecessary complexity and duplication

this is sophisticated but expensive

this is simpler and cheaper

Who owns one of these?

At a basic level Multitenancy is a drive to

reduce unnecessary complexity and duplication

this is sophisticated but expensive

this is simpler and cheaper

Multitenancy helps with

finding the right balance

Simplifying the software stack by removing all extraneous pieces makes better use of hardware and the people who run it. !

Simple == Cheaper == Predictable == Robust

Its not just a hardware story

“Every piece of knowledge must have a single, unambiguous, authoritative representation within a system” Pragmatic Programmer (Hunt & Thomas)

(or: copy-and-paste encourages problems)

Dont Repeat YourselfSimple

We want to avoid the clone army

•Are your choices only

•fully dedicated machine or shared hardware?

•Multitenancy isn’t just a checkbox.

•There are various levels of ‘tenancy’ defined today.

Are you already using Multitenancy

SaaS Tenancy Spectrum

SaaS Tenancy Spectrum

Your basic laptop, desktop or server machine.

yours, all yours..

SaaS Tenancy Spectrum

A simple shared machine LPAR’d or VM’d into multiple complete

stacks.

SaaS Tenancy Spectrum

Reduced overhead by running multiple app servers on one O/S Keep everything else

separate though. “just in case”

SaaS Tenancy Spectrum

Multiple applications

running on one app server - sharing db server and

middleware!

SaaS Tenancy Spectrum

sharing one app!

SaaS Tenancy Spectrum

Sharing everything including database tables.

SaaS Tenancy Spectrum

A Multitenant JVM covers theses levels of sharing

SaaS Tenancy Spectrum

Merging these bubbles saves money, reduces complexity and can be a real business differentiator.

But it requires engineering effort

SaaS Tenancy Spectrum

Remember, from the end user point of view, they see it like this

■ Customer viewpoint !

– Cost: provider offers the service more cheaply – Time to Value: up and running fast, typically

upgraded often & quickly – Quality of Service: focus on SLA needed not your

ability to run infrastructure – Bypass IT Backlog: streamlined deployment

(handled by provider) !

■ Provider viewpoint – Cost: Minimal moving parts / duplication – Agility: Upgrades, backups, on-boarding

Efficencies

– Fewer Parts == Better Density == $$$ for the service provider

– Fewer Parts == Better Density == £££ for the service provider

Part 2Getting there..

!

Increasing density of the software stack is impacting us all. !Cost of hosting is a business differentiator already !Its only going to increase in importance !If you’re in the cloud you already know !!Moving from ‘free’ to ‘not free’ hosting is an intensive exercise. Your software choices are starting to be decided by CPU and memory usage. With your own hardware you could let it run slow. Can’t do that in the cloud! !So how do move forward with Java and Multitenancy?

Climbing MT tenant

Challenge #1 Isolation

■The fear factor is loosing isolation ■unexpected side effects from other parts

■This is a reasonable concern ■Same number of eggs (apps), fewer baskets ■You want really good baskets arranged carefully

■But it’s not a new problem ■We can reduce the change of failure (more later) ■We can reduce the impact with good choreography

■You need to think about choreography today ■Today we use clusters to provide fail-over and arrange clusters to avoid same machine/rack/site failures

■Apply the same ideas to Multitenancy

http://circa71.wordpress.com/2010/07/

http://bit.ly/e7G1jb

Challenge #2 Cost of Entry

J Easy == No app changes

J Hypervisor sharing only

J Port Collisions

J File System Collisions

J Security Challenges

merge

merge

J Data Isolation between apps

J Control over resource hogs

J JVM can help!!

simplest (hence most popular) Still have neighbours but they are far away

Saves footprint (GB) But now we have collisions Ops guys can help with O/S JVMs can use -Xshareclasses

More footprint savings But now we have real isolation concerns (closer neighbours) We need control over resource hogs

Java Heap consumes 100’s of MB of memory –Heap objects cannot be shared between JVMs –GC has a helper thread-per-core by default

Just-in-Time Compiler consumes 10’s of MB of memory –Generated code is private and big –Generated code is expensive to produce

• Steals time from application • Multiple compilation threads by default

No choreography between JVM instances –Compilation or GC activity can happen at identical (and bad) times

Challenge #2.5 Building isolation

We need to fix the following !L Data Isolation between applications !

L Control over resource hogs

Without forcing people to change their applications!

Challenge #2.5 Building isolation

Data Isolation Challenges

■ Applications embed deployment information like url patterns in code

■ Wait! What happens if we try to deploy two copies of this servlet to a single server?

■ Static variables are bad (for sharing) ■ Most libraries are full of static variables

Wait! What happens ifeach tenant needs a different default locale?

•Usual solutions: – Wrap the whole thing in a ClassLoader (i.e. rewrite your code) – Get ride of the static variable (i.e. rewrite your code) – Use BCI to rewrite the code automatically (erodes robustness)

!!We need a silver bullet!

Wait! What happens ifeach tenant needs a different default locale?

■ Concept: Add a single argument (–Xmt for multi-tenant) to your Java command-line to opt into sharing a runtime with others.

■ Result: Your application behaves exactly as it if had a dedicated JVM, but in reality it runs side-by-side with other applications.

■ Benefits: Smaller, faster, and eventually smarter –Less duplication: (1 GC, 1 JIT), Heap object sharing –Fast Startup: JVM is already running and warm when starting apps

Multitenant JDK

■ Required: No code Changes!

Part 3Demo (of a demo)

Launch your application

■ Opt-in to multitenancy by adding –Xmt

Register with javad daemon

■ JVM will locate/start daemon automatically

locate

Create a new tenant

■ New tenant created inside the javad daemon

locateTenant1

Create a 2nd tenant

■ New tenant created inside the javad daemon

locateTenant1

Create a 2nd tenant

■ New tenant created inside the javad daemon

locateTenant1

Tenant2

One copy of common codelives in the javad process.

Most runtime structures

are shared.

■ What if … the JVM knew about tenants and provided each one with a different view of static variables?

■ Meet the @TenantScope annotation.

■ @TenantScope Semantics: Static variable values are stored per-tenant ■ Each tenant has their own LocaleSettings.defaultLocale ■ Now many tenants can share a single LocaleSettings class

Tenant1

Tenant2

… LocaleSettings.setDefaultLocale( LocaleSettings.UK );

…

… LocaleSettings.setDefaultLocale( LocaleSettings.US );

…

Providing data isolation

Did I say ‘no code changes?’

Application Changes

merge

■ @TenantScope markup gets added automatically as classes are loaded ■ Tenants see dedicated middleware – but behind the curtains classes (and

JIT’ed code) are actually shared

If it’s invisible - why have @TenantScope?

■ Allows middleware to opt out ■ Opportunities for even more density. !

■ Basic operations on Tenants available to the middleware –Data Isolation –Resource Management (more in this in a minute) !

■ Ability for the middleware to differentiate between Tenants –Which one is causing the problem? !

■ Querying the state of Tenants –How much free memory do you have?

Part 4Hey, aren’t we done yet?

Dealing with bad behaviour

images from http://www.rra.memberlodge.org/Neighbourhood-Watch-Reporting

http://bit.ly/ficwkl

■ The closer your neighbours the better your controls must be

■ Multitenant JDK provides controls on –CPU time –Heap size –Thread count –File IO: read b/w, write b/w –Socket IO: read b/w, write b/w

!

Shared environments need resource ctl

■ Simple command-line switches for new resources –-Xlimit:cpu=10-30 // 10% minimum CPU, 30% max –-Xlimit:cpu=30 // 30% max CPU –-Xlimit:netIO=20M // Max bandwidth of 20 Mbps

■ Existing options get mapped for free –-Xms8m –Xmx64m // Initial 8M heap, 64M max

■ Plus some JMX beans to see how much of each resource you are using – i.e. understand how your code uses resources by wrapping in a tenant

Resource Control Ergonomics

■ Throttling at Java layer for portability ■ Or, leveraging OS WLM directly for efficiency (Linux & AIX)

– Note: many WLMs tend to like processes, not groups of threads

Network: XXXKB/S

Hardware resources

CPU: XXX GHZ DISK: XXXKB/S Memory: XXX GB

OS Level Resources Management

Thread

SocketHandler

OS Workload Manager (WLM)

OS resources

JVM Resource Management

Resource Throttle Layer

JSR 284 API

GC (Heap Mgmt)

Tenant Tenant Tenant Tenant Tenant

OS

JVM

Resource native API

Memory CPU Thread File I/O Socket I/O CPU File I/O Socket I/O

Building on JSR 284

JSR-284 Resource Consumption Mgmt API

Round OS as controller JVM as controller

1 1362s 1267s

2 1167s 1239s

3 1452s 1390s

4 1094s 1122s

5 1139s 1123s

6 1244s 1134s

Average 1243s 1212s

Benchmark setting • Duration comparison: Linux AMD64, run a CPU-intensive app

with 10 threads with 100% CPU quota, each thread doing the same Fibonacci calculation, benchmark the duration

• Accuracy comparison: Linux AMD64, run two CPU-intensive apps each doing the same Fibonacci calculation, but with different CPU quota: 60% vs 30%, benchmark the accuracy

Result: JVM control achieves comparable performance, but less accuracy.

Accuracy

cpu throttling in os controller

cpu%

0.00

20.00

40.00

60.00

80.00

time

00:0000:0100:0200:0300:0400:0500:0600:0700:0800:0900:1000:1100:1200:1300:1400:1500:1600:1700:1800:1900:2000:2100:2200:2300:2400:2500:2600:2700:2800:2900:3000:3100:3200:3300:3400:3500:3600:3700:3800:3900:4000:4100:4200:4300:4400:4500:4600:4700:4800:4900:5000:5100:5200:5300:5400:5500:5600:5700:5800:5901:0001:0101:0201:0301:0401:0501:0601:0701:0801:0901:1001:1101:1201:1301:1401:1501:1601:1701:1801:1901:2001:2101:2201:2301:2401:2501:2601:2701:2801:2901:3001:3101:3201:3301:3401:3501:3601:3701:3801:3901:4001:4101:4201:4301:4401:4501:4601:4701:4801:4901:5001:5101:5201:5301:5401:5501:5601:5701:5801:5902:0002:0102:0202:0302:0402:0502:0602:0702:0802:0902:1002:1102:1202:1302:1402:1502:1602:1702:1802:1902:2002:2102:2202:2302:2402:2502:2602:2702:2802:2902:3002:3102:3202:3302:3402:3502:3602:3702:3802:3902:4002:4102:4202:4302:4402:4502:4602:4702:4802:4902:5002:5102:5202:5302:5402:5502:5602:5702:5802:5903:0003:0103:0203:0303:0403:0503:0603:0703:0803:0903:1003:1103:1203:1303:1403:1503:1603:1703:1803:1903:2003:2103:2203:2303:2403:2503:2603:2703:2803:2903:3003:3103:3203:3303:3403:3503:3603:3703:3803:3903:4003:4103:4203:4303:4403:4503:4603:4703:4803:4903:5003:5103:5203:5303:5403:5503:5603:5703:5803:5904:0004:0104:0204:0304:0404:0504:0604:0704:0804:0904:1004:1104:1204:1304:1404:1504:1604:1704:1804:1904:2004:2104:2204:2304:2404:2504:2604:2704:2804:2904:3004:3104:3204:3304:3404:3504:3604:3704:3804:3904:4004:4104:4204:4304:4404:4504:4604:4704:4804:4904:5004:5104:5204:5304:5404:5504:5604:5704:5804:5905:0005:0105:0205:0305:0405:0505:0605:0705:0805:0905:1005:1105:1205:1305:1405:1505:1605:1705:1805:1905:2005:2105:2205:2305:2405:2505:2605:2705:2805:2905:3005:3105:3205:3305:3405:3505:3605:3705:3805:3905:4005:4105:4205:4305:4405:4505:4605:4705:4805:4905:5005:5105:5205:5305:5405:5505:5605:5705:5805:5906:0006:0106:0206:0306:0406:0506:0606:0706:0806:0906:1006:1106:1206:1306:1406:1506:1606:1706:1806:1906:2006:2106:2206:2306:2406:2506:2606:2706:2806:2906:3006:3106:3206:3306:3406:3506:3606:3706:3806:3906:4006:4106:4206:4306:4406:4506:4606:4706:4806:4906:5006:5106:5206:5306:5406:5506:5606:5706:5806:5907:0007:0107:0207:0307:0407:0507:0607:0707:0807:0907:1007:1107:1207:1307:1407:1507:1607:1707:1807:1907:2007:2107:2207:2307:2407:2507:2607:2707:2807:2907:3007:3107:3207:3307:3407:3507:3607:3707:3807:3907:4007:4107:4207:4307:4407:4507:4607:4707:4807:4907:5007:5107:5207:5307:5407:5507:5607:5707:5807:5908:0008:0108:0208:0308:0408:0508:0608:0708:0808:0908:1008:1108:1208:1308:1408:1508:1608:1708:1808:1908:2008:2108:2208:2308:2408:2508:2608:2708:2808:2908:3008:3108:3208:3308:3408:3508:3608:3708:3808:3908:4008:4108:4208:4308:4408:4508:4608:4708:4808:4908:5008:5108:5208:5308:5408:5508:5608:5708:5808:5909:0009:0109:0209:0309:0409:0509:0609:0709:0809:0909:1009:1109:1209:1309:1409:1509:1609:1709:1809:1909:2009:2109:2209:2309:2409:2509:2609:2709:2809:2909:3009:3109:3209:3309:3409:3509:3609:3709:3809:3909:4009:4109:4209:4309:4409:4509:4609:4709:4809:4909:5009:5109:5209:5309:5409:5509:5609:5709:58

cpu throttling in jvm controller

cpu%

0.00

22.50

45.00

67.50

90.00

time

00:0000:0100:0200:0300:0400:0500:0600:0700:0800:0900:1000:1100:1200:1300:1400:1500:1600:1700:1800:1900:2000:2100:2200:2300:2400:2500:2600:2700:2800:2900:3000:3100:3200:3300:3400:3500:3600:3700:3800:3900:4000:4100:4200:4300:4400:4500:4600:4700:4800:4900:5000:5100:5200:5300:5400:5500:5600:5700:5800:5901:0001:0101:0201:0301:0401:0501:0601:0701:0801:0901:1001:1101:1201:1301:1401:1501:1601:1701:1801:1901:2001:2101:2201:2301:2401:2501:2601:2701:2801:2901:3001:3101:3201:3301:3401:3501:3601:3701:3801:3901:4001:4101:4201:4301:4401:4501:4601:4701:4801:4901:5001:5101:5201:5301:5401:5501:5601:5701:5801:5902:0002:0102:0202:0302:0402:0502:0602:0702:0802:0902:1002:1102:1202:1302:1402:1502:1602:1702:1802:1902:2002:2102:2202:2302:2402:2502:2602:2702:2802:2902:3002:3102:3202:3302:3402:3502:3602:3702:3802:3902:4002:4102:4202:4302:4402:4502:4602:4702:4802:4902:5002:5102:5202:5302:5402:5502:5602:5702:5802:5903:0003:0103:0203:0303:0403:0503:0603:0703:0803:0903:1003:1103:1203:1303:1403:1503:1603:1703:1803:1903:2003:2103:2203:2303:2403:2503:2603:2703:2803:2903:3003:3103:3203:3303:3403:3503:3603:3703:3803:3903:4003:4103:4203:4303:4403:4503:4603:4703:4803:4903:5003:5103:5203:5303:5403:5503:5603:5703:5803:5904:0004:0104:0204:0304:0404:0504:0604:0704:0804:0904:1004:1104:1204:1304:1404:1504:1604:1704:1804:1904:2004:2104:2204:2304:2404:2504:2604:2704:2804:2904:3004:3104:3204:3304:3404:3504:3604:3704:3804:3904:4004:4104:4204:4304:4404:4504:4604:4704:4804:4904:5004:5104:5204:5304:5404:5504:5604:5704:5804:5905:0005:0105:0205:0305:0405:0505:0605:0705:0805:0905:1005:1105:1205:1305:1405:1505:1605:1705:1805:1905:2005:2105:2205:2305:2405:2505:2605:2705:2805:2905:3005:3105:3205:3305:3405:3505:3605:3705:3805:3905:4005:4105:4205:4305:4405:4505:4605:4705:4805:4905:5005:5105:5205:5305:5405:5505:5605:5705:5805:5906:0006:0106:0206:0306:0406:0506:0606:0706:0806:0906:1006:1106:1206:1306:1406:1506:1606:1706:1806:1906:2006:2106:2206:2306:2406:2506:2606:2706:2806:2906:3006:3106:3206:3306:3406:3506:3606:3706:3806:3906:4006:4106:4206:4306:4406:4506:4606:4706:4806:4906:5006:5106:5206:5306:5406:5506:5606:5706:5806:5907:0007:0107:0207:0307:0407:0507:0607:0707:0807:0907:1007:1107:1207:1307:1407:1507:1607:1707:1807:1907:2007:2107:2207:2307:2407:2507:2607:2707:2807:2907:3007:3107:3207:3307:3407:3507:3607:3707:3807:3907:4007:4107:4207:4307:4407:4507:4607:4707:4807:4907:5007:5107:5207:5307:5407:5507:5607:5707:5807:5908:0008:0108:0208:0308:0408:0508:0608:0708:0808:0908:1008:1108:1208:1308:1408:1508:1608:1708:1808:1908:2008:2108:2208:2308:2408:2508:2608:2708:2808:2908:3008:3108:3208:3308:3408:3508:3608:3708:3808:3908:4008:4108:4208:4308:4408:4508:4608:4708:4808:4908:5008:5108:5208:5308:5408:5508:5608:5708:5808:5909:0009:0109:0209:0309:0409:0509:0609:0709:0809:0909:1009:1109:1209:1309:1409:1509:1609:1709:1809:1909:2009:2109:2209:2309:2409:2509:2609:2709:2809:2909:3009:3109:3209:3309:3409:3509:3609:3709:3809:3909:4009:4109:4209:4309:4409:4509:4609:4709:4809:4909:5009:5109:5209:5309:5409:5509:5609:5709:58

60% throttling30% throttling

Duration

CPU Throttling

os throttling jvm throttling

Part 5Performance so far

■ Environment: Measure standard benchmarks in a 1 GB + 1 core VirtualBox guest – Advantage: Easy to control, highly reproducible

■ Methodology: Add applications until the system swaps, then it’s ‘full’ – More applications is better – Per tenant cost is amount of RAM / # tenants

Current Performance Data

0

5

10

15

20

instances

Standard JVM Hand tuned Standard JVM MT support

Standard JVM

0

15

30

45

60

instances

Standard JVM Hand tuned Standard JVM MT support

Best hand-tuned JVM config

0

75

150

225

300

instances

Standard JVM Hand tuned Standard JVM MT support

Simple out of the box -Xmt

4MBtenant

Part 6Wrap up

■ Focus to date has been ‘zero application changes’ – We can do even better with tenant-aware middleware

■ API’s used to provide isolation & throttling are available to stack products – JSR-284 (Resource Management) – JSR-121 (Isolates) – @TenantScope fields

■ Java language and frameworks (EclipseLink) are evolving to have first-class multitenant support

■ Stay tuned for progress: watch the IBM Java 8 beta program

This is still ‘experimental’

This is still ‘experimental’

Simplifying the software stack by removing all extraneous pieces makes better use of hardware (and people who run it). !Multitenancy can make us more efficient:

–Trades isolation for footprint and agility –JVM support makes multitenancy safer and easier –Measuring resource usage and load patterns is critical –Multitenant JDK primitives give us room for future growth

Now that you’ve completed this session, you are able to:

– Understand what multitenancy is and what it’s good for • Per-tenant costs measured in single-digit MB are possible

– Describe challenges of multitenant Java deployments • Hard for VM guys, should be easy for you • Choreography of load / deployment is up to you

– Understand new JDK features to convert existing applications into multitenant deployments • Are we on the right track? Could you use this in your business?

Conclusion

Thank you - any questions?