iAssessment’s

High Performance

Gateway Presentation System

• Background

• Implementation

• Pitfalls

• Factors to Consider

• Questions and Answers

Presentation Overview

Background / History

• Web/Portal History– Static – Dynamic – Application based - Portal

• iAssessment Evolution and Products– K-12 ASP Compliance and Reporting– Assessments and Survey– Teacher Resource Cataloging– Professional Development Grant/Policy

Management– Communication Tools

• uPortal – iAssessment Gateway Presentation System (GPS)– User management– Entity management– Individual client branding

• Controlled User Experience– User profile based– Role based applications– Client controlled not user controlled

• Integrated User Experience– Other disparate systems

Background / Solution GPS K12 Smart Portal

Product Map

Implementation / Case Study

• GPS Case Study / CTAP2 California Department of Education– >300,000 teachers & administrators– 50,000 initial students for beta student system (with

one district having ~2 million students)• Use Model

– Administrators– Teachers– User created accounts– Gradual adoption– Peaky use times (grant deadlines, etc.)– Account reuse

Implementation / Case Study

• Current CTAP2 Statistics– ~200,000 accounts– Peaks of 1,000 concurrent logins– Peak “season” average 5,000 logins per day– Tables with more than 20 million rows

• Architecture Requirements– Expandable at top level

• Load balancing• Scaling• Easy machine addition

– Scalable software model• Fixed session at login

Implementation / Case Study

• CTAP2 Hardware– Web/Application Servers

• Sun 220R (2CPU/2GB) NFS Server• Four X1’s (1CPU/1GB)

– Database Server• SunFire v880 (6CPU/12GB)• T3 Disk Array redundant pair

– Cisco content switch– Cisco Pix Firewall

Implementation / Case Study

• Current Hardware Capabilities– X1 can handle ~150 users each– 220R ~400 users– Application front end current max load ~1,000

concurrent users– Database max load 3,000 users (keeping in

mind the different load factor of iAssessment’s applications running in uPortal)

• Still could use optimization as well.

Implementation / Case Study

• Software– Solaris 8 & Linux (web/application servers)– Sybase 12.5 / Solaris 8 (database servers)– Tomcat 4.0– Apache 1.3

• mod_webapp

– JDK 1.4.0– uPortal 2.0

Non-performance-based uPortal changes

• Created generic servlet wrapper channel– Run with and without uPortal

• Layout management modifications– Layout per user type, school type, etc.

• Authentication modifications– First name, last name, id, password

• Database connection pool sharing

Things That Worked

• Planned uPortal extensibility generally worked well

• Relative URL space worked well– Load balancing (

http://ctap2w{1,2,3,4,5}.iassessment.org/)– Integration into existing URL namespace– https might be a problem

Problems & Fixes

• HTML production: XML “well-formed”-ness and XLST processing speed– jTidy didn’t work well– Fearful of XSLT overhead– FIX: avoided XSLT processing

Problems & Fixes

• High memory use (400MB-1GB/process)– Intrinsic needs of applications and ???– Bugs (SmartCache, database connection

pool)• FIX: code changes

– Caused java.lang.Runtime.exec failure (no vfork equivalent in Java)

• FIX: created “exec server”

Problems & Fixes

• Large database hit with large groups– Debug Sybase JDBC driver and ~20,000 UP_USER rows

– uPortal 2.0 queries database for each member in set; long delay during account creation

– FIX: used production JDBC driver; uPortal 2.0 code should be changed

Problems & Fixes

• Proxy servers and “User-Agent” HTTP header removal– FIX: wrapped original HttpServletRequest &

protected BrowserInfo– Better FIX: all in one wrapper

Problems & Fixes

• Cisco content switch problems– Used IP mapping because cookies didn’t work– Load balancing from NAT/proxy didn’t work

well with IP mapping– Single point of failure– No method of determining which machine

user was on– FIX: using content switch for login redirection

only

Problems & Fixes

• Caching of layouts– Performance gain, but required inter-JVM

communication• FIX: created small inter-JVM communication

system

– Aggregated Layouts implication (with multiple JVMs)?

• User-caused problems– Multiple submits of same request

• FIX: used a ‘please wait’ screen where needed

Performance Modifications

• Did layout caching– Per JVM– Reloaded on demand from inter-JVM

message

• Bypassed channel-level XSLT transformations

• Didn’t use debug drivers in production

• Limited duplicate burdensome requests

Scalability Modifications

• Support for multiple web/application server machines

• Support for large JVM size

Lessons Learned

• Memory is limiting factor for GPS

• Well-formed HTML hard to achieve even when we control all the code

• Simple load balancing via publicly exposed web/application servers easiest and most effective

Lessons Learned

• Environment is hostile– Unfriendly proxy servers– “Mega proxies” (AOL)– Impatient users– Impatient firewalls

Things to Consider

• Architecture to support multiple web/application servers– Inter-JVM issues possibly– Per-server information (# of users, etc.)?

• Existing servlet code– To wrap or not?– Bypass XSLT?

Things to Consider

• Environment– Non-technical users– Unfriendly proxies, or firewalls?

• Application types– Large JVM size?– CPU/database intensive?

Questions and Answers

“Why haven’t you checked in the code changes?

“Can we get a copy of the code you did for xyz?”

“How did you avoid XSLT processing?”