Post on 27-Dec-2015
transcript
Infrastructure to Support Infrastructure to Support Extreme Self-ServiceExtreme Self-Service
Session #20653March 15, 2006
Alliance 2006 ConferenceNashville, Tennessee
2
PresenterPresenter
Carol JordanCarol JordanI have been working on PeopleSoft I have been working on PeopleSoft applications since 1997, and have been at applications since 1997, and have been at Stanford since November 2000. Stanford since November 2000. I am the Infrastructure Manager for the I am the Infrastructure Manager for the team supporting Student and HR Systems. team supporting Student and HR Systems.
In addition to the PeopleSoft products, our In addition to the PeopleSoft products, our group also supports Kronos, Resource 25, group also supports Kronos, Resource 25, OnBase Imaging, Resumix and a custom OnBase Imaging, Resumix and a custom Workflow application.Workflow application.
3
OverviewOverview
Before PeopleSoft:- Limited functionality for students- …using a web-enabled mainframe application- …allowing only 35 concurrent student sessions
With PeopleSoft HR/SA and Portal:- Many functions for students- …and for faculty, advisors and staff- …and no portal sessions are turned away
4
Agenda/ContentsAgenda/Contents
In this presentation:In this presentation:• Extreme self-service defined• Our rough start• Where we are now, how we got here• What we learned
5
Overview: Stanford Overview: Stanford UniversityUniversity Stanford University is a private university Stanford University is a private university located 30 miles south of San Francisco and located 30 miles south of San Francisco and just north of Silicon Valley. just north of Silicon Valley.
There are currently 6,700 undergraduate There are currently 6,700 undergraduate and 8,000 graduate students enrolled, with and 8,000 graduate students enrolled, with 1,775 faculty. 1,775 faculty.
Founded in 1891, the university was Founded in 1891, the university was established by Jane and Leland Stanford in established by Jane and Leland Stanford in memory of their son, Leland Stanford, Jr., memory of their son, Leland Stanford, Jr., who died of typhoid fever at the age of 15.who died of typhoid fever at the age of 15.
6
Overview: Application Overview: Application AdministrationAdministration
The Application Administration team is The Application Administration team is part of Administrative Systems, part of Administrative Systems, supporting many of the applications supporting many of the applications used campus-wide. used campus-wide.
There are three AppAdmin teams, There are three AppAdmin teams, supporting PeopleSoft, Oracle supporting PeopleSoft, Oracle Financials, and Reporting.Financials, and Reporting.
The team supporting PeopleSoft is The team supporting PeopleSoft is made up of 5 System Analysts, 3 made up of 5 System Analysts, 3 Oracle DBAs and 3 Windows System Oracle DBAs and 3 Windows System Administrators. Administrators.
Developers and other support staff are Developers and other support staff are part of a separate team within part of a separate team within Administrative Systems.Administrative Systems.
7
Overview: Application Overview: Application AdministrationAdministration
PeopleSoft products currently PeopleSoft products currently installed:installed:
Campus Solutions 8 SP1Campus Solutions 8 SP1 PeopleTools 8.22.05PeopleTools 8.22.05
Enterprise Portal 8.8 SP1Enterprise Portal 8.8 SP1 PeopleTools 8.44.03PeopleTools 8.44.03
Enterprise Learning Management 8.8 SP1Enterprise Learning Management 8.8 SP1 PeopleTools 8.45.12PeopleTools 8.45.12
All on Oracle 9.2.0.6
8
Extreme Self-Service, Extreme Self-Service, defineddefined
Lots of functionality for students, faculty, advisors and staff
and
Lots of users – close to 35,000 logins on our busiest days
9
Extreme Self-Service, Extreme Self-Service, defineddefinedLots of functionality for students
• File or adjust study list and elect grading options• Request an official transcript to be mailed • Print history of courses and grades• Print Enrollment Certification• Declare major and minor• Apply to graduate• Update addresses, emergency contacts• Maintain FERPA elections• View and accept financial aid• View student bill and print statement• View advisors• View degree progress, milestones, program summary• Coming in March…enter or read course evaluations• Coming in April…pay student bill, add money to card plan• Links to other applications: Student Housing, IT Services bill
(Pinnacle), Office of Development
10
Extreme Self-Service, Extreme Self-Service, defineddefinedLots of functionality
• for Faculty− Review class lists− Send e-mail to students in a class− Submit and change grades− Assign a grade proxy − Review previously-submitted grades− Coming in March…read students’ course evaluations
• for Advisors− View lists of current and past advisees− View study lists, grades and unofficial transcripts for current advisees− Place and release holds of current advisees
• for Staff− View on-line pay statement− Make W-4 elections− Enter direct-deposit elections− Register for training, take on-line tests through PeopleSoft ELM− Link to applications: Kronos, on-line W-2
11
Extreme Self-Service, Extreme Self-Service, defineddefined
Axess Usage Summary: Logins - 2005
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
Jan
Feb
Mar
Apr
May Jun
Jul
Aug Sep Oct
Nov
Dec Jan
To
tal
Lo
gin
s
0
150
300
450
600
750
900
1050
Max
Lo
gin
s p
er Q
uar
ter-
ho
ur
Total logins by day Max Logins Per Quarter-hour
Lots of users:
12
Our Rough StartOur Rough Start
• Upgrade to Version 8 went OK
• First couple of months went OK
• …but the system could not support the sustained demand of our busiest days – the first two days of the new academic year
13
Our Rough StartOur Rough Start
Stanford Daily articles with headlines like, “Axess Problems Plague First Week”
What students saw:• log-in attempts that failed• ‘connection refused’ errors between
HR/SA and Portal• pages that didn’t load
14
Our Rough StartOur Rough Start
CIO magazine article: ‘Big Mess on Campus’
“Stanford's IT was still struggling with integrating the enterprise systems when the newly launched PeopleSoft Web portal (called Axess) crashed last fall…Axess couldn't handle the load of all the returning students trying to log in…”
15
Where we are nowWhere we are now
• We are able to handle high load without issues
• Business offices are planning to roll out new functionality using the Axess portal
•…and we’ve managed to stay out of the papers for a while (fingers crossed)
17
How We Got HereHow We Got Here
• Upgraded hardware, added hardware• Server hosting custom single sign-on was the
biggest, earliest bottleneck• Original web servers were old and slow• Got the OK to order new hardware after
capacity issues hit
• Lessons: • Go with your gut…if you’re worrying about it,
fix it; if you can’t fix it, have a contingency plan
• Fight the requests to minimize costs by re-using old hardware
18
How We Got HereHow We Got Here
Web-server changes for HR/SA and Portal• Upgraded JRE• Upgraded Jolt and WebLogic to the latest rolling patch• Reduce network disconnect timeout• Used WebLogic console to fine-tune JVM heapsize
(768M) and thread-count (100) on Windows web servers
• Set Windows PIA service to restart automatically if it goes down
• Installed two logical web-servers per physical server to improve memory utilization under Windows
• Separated administrative and self-service users, to set different time-outs, breadcrumbs, and other configuration parameters for different types of users
19
How We Got HereHow We Got Here
Application-server changes for HR/SA and Portal:
• Upgraded JRE• Upgraded Tuxedo to latest rolling patch• Reduced client-cleanup timeout• For Portal and ELM: implemented shared
application-server cache• For Portal and ELM: re-configured Tuxedo
domains to eliminate spawning – always start the maximum number of PSAPPSRVs− 35 per server for each of two Portal application servers− 8 per server for each of two ELM application servers− 15-25 per server for each of two HR/SA application
servers
20
What we learnedWhat we learned
• We can’t predict load with certainty • High load is loosely tied to the beginning of
the term, when fees are due and students need to know where their classes are
• …so we implemented defensive monitoring to send an e-mail when at 400+ logins per quarter-hour
• Sample e-mail text:The Axess login count as of 00:16, 03/13/2006 is 550 in a 15-minute interval.
21
What we learnedWhat we learned
Because we count logins, we can see patterns in user activity:03/12/2006 22:30 5703/12/2006 22:45 6603/12/2006 23:00 9403/12/2006 23:15 8003/12/2006 23:30 10503/12/2006 23:45 24003/13/2006 00:00 550 03/13/2006 00:15 197 03/13/2006 00:30 181 03/13/2006 00:45 128 03/13/2006 01:00 100 03/13/2006 01:15 90 03/13/2006 01:30 76 03/13/2006 01:45 57
Spring enrollment opened at 12:01am on Monday 3/13
22
What we learnedWhat we learned
Monitor at many levels
• At the web-server layer:• We monitor the load-balanced site• …and individual (logical) web servers• …and the HTTP port used by the load-balancer probe
• At the application-server layer:• We monitor for Tuxedo processes• UNIX SysAdmins monitor memory usage, add swap
space as needed• We monitor for CPU utilization (sample e-mail
follows)
23
CPU monitor – sample e-mailCPU monitor – sample e-mailInfo from sar====================================== Time: 00:40:10 usr%: 77 sys%: 10 wio%: 0idle%: 14
Top output======================================load averages: 4.76, 9.29, 15.18 00:40:10106 processes: 101 sleeping, 1 zombie, 4 on cpu
Memory: 8192M real, 1143M free, 6577M swap in use, 18G swap free
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND26073 a2k_prd 4 0 0 203M 187M cpu0 4:40 10.43% PSAPPSRV 9800 a2k_prd 4 58 0 256M 238M sleep 13:49 8.83% PSAPPSRV10991 a2k_prd 4 50 0 245M 227M sleep 11:18 7.61% PSAPPSRV10819 a2k_prd 4 58 0 267M 251M sleep 13:33 6.82% PSAPPSRV
24
What we learnedWhat we learned
Use your PeopleSoft tools!
• PeopleSoft Performance Monitor for Portal and ELM
• WebLogic console for web servers
25
What we learnedWhat we learned
• Keep Tools up-to-date – watch for BEA updates
• Move AppMessaging to a dedicated host – keep back-end processing out of the transactional infrastructure
• Nobody hits ‘logoff’ – configure your web servers to close idle connections quickly
• Performance testing is mandatory for new projects
• Configuration review, top-to-bottom, with PeopleSoft Consulting was helpful
27
ContactsContacts
Carol JordanCarol JordanInfrastructure Manager, Student and HR SystemsInfrastructure Manager, Student and HR SystemsAdministrative SystemsAdministrative SystemsStanford UniversityStanford UniversityE-mail: E-mail: cajordan@stanford.educajordan@stanford.edu
Jeff ButlerJeff ButlerLead System Analyst, Student & HR SystemsLead System Analyst, Student & HR SystemsStanford UniversityStanford UniversityE-mail: jbutler@stanford.eduE-mail: jbutler@stanford.edu
Ross ShimabukuroRoss ShimabukuroSystem Analyst, Student & HR SystemsSystem Analyst, Student & HR SystemsStanford UniversityStanford UniversityE-mail: ryshima@stanford.eduE-mail: ryshima@stanford.edu
This presentation and all Alliance 2006 This presentation and all Alliance 2006 presentations are available for presentations are available for
download from the Conference Sitedownload from the Conference Site
Presentations from previous meetings are also availablePresentations from previous meetings are also available