Upgrading CA Workload Automation ESP Edition Agents at Verizon
MFX07S #CAWorld
Mainframe
Cedric Persaud, Verizon
Associate Director
2 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Abstract
Verizon Telecommunications made CA Workload
Automation ESP Edition (CA WA ESP) the standard tool
for workload management across the Wireline
organization for both mainframe and distributed
systems using the CA WA ESP agent technology. In this
session, they will share insights on how they
successfully upgraded to the newer version of the CA
WA ESP agent across the enterprise for all of their
distributed applications. They will also review their
processes and procedures used to upgrade ESP, as well
as share lessons learned and best practices.
Cedric Persaud
Verizon
Associate Director
3 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Agenda
COMMUNICATION AND NOTIFICATIONS 5
CONDUCTING ACTUAL UPGRADE TO R11.3 6
LESSONS LEARNED & BEST PRACTICES 8
ISSUES ENCOUNTERED 7
VERIZON ENVIRONMENT 1
REASONS FOR UPGRADING TO R11.3 AND BENEFITS 2
PREPARING AND PLANNING FOR UPGRADE 4
OVERVIEW OF UPGRADE EFFORT 3
SUMMARY & CLOSING REMARKS 9
Q&A 10
4 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Verizon Environment - Overview
Heterogonous Environment (Distributed & Mainframe) – Mainframe applications using CA WA ESP : Billing, Provisioning, etc.
– Distributed applications using CA WA ESP agent: Payroll, Finance, Billing, etc.
– 212 distributed applications using CA WA ESP
1,450 CA WA ESP agents in Distributed Environment (August 2014)
Within past 3 years, Verizon has consolidated data centers and merged CA WA ESP masters. – Currently use 13 production masters & 3 test/dev masters
CA WA ESP agents with CA WA ESP Plug-Ins – SAP
– PeopleSoft
– Remote Execution Agent (REA)
5 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Verizon Environment – Jobs Processed
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
4,752,697
4,284,502 4,522,348 4,434,071
3,942,083 3,756,169
3,904,927 3,911,250 3,861,952
CA WA ESPx Total Jobs Run
6 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Verizon Environment - Architecture Overview
Remote
Execution
Agent Windows
AIX HPUX Solaris
CA WA ESP Workstation 3270
LINUX
SAP PeopleSoft
CA WA ESP Proxy
CA WA ESP Proxy CA WA ESP Proxy
CA WA ESP Master
Cloud servers IBM z/OS
Sysplex
I5 Tandem
CA WA ESP System Agents
CA WA ESP Application Integration
Agents(Plug-ins)
7 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Verizon Environment - CA WA ESPx Infrastructure
Freehold Data Center
Fairland Data Center
Freehold
Mainframe
Fairland
Mainframe
ESPMBHC1
ESPMSAC
ESPMTPA
ESPMFDC2
ESPMOME
ESPMSTE (non-prod)
ESPMDEV (non-prod)
ESPMDEVL (dev)
CA WA ESPx Agents
CA WA ESPx Plug-in
Mainframes & CA WA ESP Masters
8 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Verizon Environment – Organization and My Team
Verizon IT Cloud Services (ITCS)
– Responsible for delivering & managing IT systems, hardware, and
infrastructure across Verizon’s strategic data centers.
Production Control Organization – Supports all scheduling activities for all applications and platforms
– CA WA ESP Administration team responsible for supporting CA WA ESP on mainframe
Open Systems Scheduling (My Team) – 12 FTEs support & manage CA WA ESPx
– Administrators of CA WA ESPx and associated components
– Responsible for deploying new agents and scheduling new batch jobs
– Responsible for providing Tier 2 support of production jobs
– Interface with CA to address CA WA ESPx bugs and/or enhancements
9 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Reasons for Upgrading and Benefits
Reasons for Upgrading from R7 to R11.3 – R7 agent reaching EOL (end of service life): March 2015
– Improved features and scheduling benefits for customers
– R11.3 is compatible with newer operating systems and platforms
Benefits of R11.3 Agent – R11.3 Agent supports Windows 8/Windows 2012 operating systems
– R11.3 adds more detailed logging options to assist in problem resolution
– Performance and optimization improvements
– Many other enhanced features such as:
File watchers with re-trigger option
Event sensors to monitor whether events are loaded
10 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Overview of Upgrade
CA Recommended to Upgrade to R11.3
Started Upgrade in 3Q13 (September/October)
Target Completion Date: 4Q14
Inventory of Agents – Utilized Verizon’s standard inventory tools with discovery capability
– CMDB (HP)
– Discovery Dependency Mapping Inventory tool (HP)
Scope of Project – 1900 R7 agents identified and reviewed across footprint
– 900 R7 agents deemed inactive & removed from environment
– 1,000 R7 agents (or 125 applications) eligible and in scope for upgrade
11 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Preparing and Planning Upgrade
Tested and “Soaked” R11.3 Agent – Installed and performed comprehensive testing of R11.3 agent
No issues found
– Verified compatibility issues with CA WA ESP Master on mainframe (R11.3)
No issues found
– Ensured R11.3 had same functionality as R7 agent
No CA WA ESPx applications or job procedures had to be changed
Packaged 11.3 for Mass Deployment – Verizon has standard packages used for Unix and Windows
– Standard deployment tools used to distribute R11.3 agent
Tivoli (IBM)
System Center Configuration Manager (Microsoft)
12 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Communication and Notification to Users
Users of CA Workload Automation are Verizon Internal Application Teams.
Sought Concurrence and “Buy-In” from Application Teams – Conducted numerous meetings and planning sessions with application
teams.
– Reviewed the benefits and features of R11.3
– Explained R7 EOL as key driver for upgrading to R11.3
– Assured application teams upgrade will be minimally disruptive
– No application code or CA WA ESPx job changes required for upgrade
– Partnered with application teams to perform functional testing of R11.3 agent in test/development environment.
13 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Conducting Upgrade
Prioritized Applications for Upgrade – Initially selected applications running on an CA WA ESP master
targeted for merger/consolidation.
Adhered to Verizon’s Change Management Policy – Submitted change records 3 weeks prior to actual upgrade
Upgrade Steps Performed (average duration per server = 1hr) – Suspended active Jobs/Events (scripted)
– Shutdown agent
– Removed old R7 package (scripted)
– Deployed R11.3 package to target server (scripted)
– Restarted/Recycled agent (scripted)
– Resumed Jobs/Events (scripted)
– Validated change
14 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Issues Encountered with Upgrade
“Agent Notify” issue due to lost connectivity with CA WA ESP master
Out of disk space – upgrade requires more space on server
Discovered additional/unknown servers requiring upgrade
Incomplete removal of R7 package on Windows – Stale files and programs left by CA uninstall program
Server auto-reboot issue on R11.3 package for HP-UX
Default shell issue with R11.3 and R7 (/bin/sh vs. /bin/ksh) – R7 jobs used /bin/ksh. Whereas, R11.3 package defaulted to /bin/sh
– After upgrade, some CA WA ESPx jobs failed because they needed to run using /bin/ksh.
15 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Lessons Learned and Best Practices
Notify application teams to test CA WA ESP agent plug-ins early in development environment.
Partner with CA to perform agentparm.txt file gap analysis & audit.
It’s a good idea to clean-up/remove unused CA WA ESP agents prior to upgrade.
Ensure support has login access to server prior to upgrade
Ensure agent install packages are built per CA’s standards and work with CA to validate.
16 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
Summary & Closing Remarks CA Workload Automation Upgrade is a Worthwhile Effort
Results
Improved benefits and features of R11.3 for application teams
Upgrading provides opportunity to “sanitize” the CA WA ESP
environment.
Verizon customers are satisfied with CA’s support
We are pleased with CA’s responsiveness.
Satisfied with CA’s resolution of technical issues and product
enhancements.
Excellent training offered by CA for Verizon staff and
customers.
17 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
For More Information
To learn more about Mainframe, please
visit:
http://bit.ly/1wbiPkl
Insert appropriate screenshot and text overlay from following “More Info Graphics” slide here;
ensure it links to correct page Mainframe
18 © 2014 CA. ALL RIGHTS RESERVED.
© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
For Informational Purposes Only
© 2014 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
This presentation provided at CA World 2014 is intended for information purposes only and does not form any type of warranty. Content provided in this presentation has not been reviewed for accuracy and is based on information provided by CA Partners and Customers.
Terms of this Presentation