Date post: | 05-Jul-2015 |
Category: |
Technology |
Upload: | rockwell-automation |
View: | 221 times |
Download: | 2 times |
Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.
PUBLIC INFORMATION
Mallinckrodt: Pharmaceutical Plant Saves
$100k with Virtualized PlantPAx® Platform
Don Geers
Senior Maintenance Engineer
Mallinckrodt Pharmaceuticals
Agenda
Company Overview
Project Team
Our Challenge (2 phases)
Justification
What’s Out, What’s Left, What’s New
Virtualization
Factory Acceptance Testing
Qualification & Startup
What Went Well
Lessons Learned
Going Forward
Q&A
2
Company Overview
3
Pharmaceutical manufacturer
Bulk Active Pharmaceutical Ingredients (API)
Specialty Chemicals
Imaging agents
Drug chemicals
Based in St.Louis Mo
140+ year old facility on 40 acres
500+ Employees at the St.Louis plant
Other Manufacturing Sites
Hobart NY, Raleigh NC, Dublin Ireland,
others
FDA Validated Facility
Long term data retention is critical
Project Team
Mallinckrodt Pharmaceuticals Don Geers – Lead Controls Engineer
Lead Process Engineer
Various other Process Engineers, Validation and Quality
personnel.
Rockwell Automation
Jim Wardell – Lead Project Engineer
Minimum of 6 additional engineers for support
Local Rockwell Automation engineering support – Long term
4
Our Challenge
Phase 1
Develop a programming standard for all new PAC based
installations in our facility. Provides for a common
programming structure across our different manufacturing
units.
Phase 2 (the upgrade)
Replace a legacy HMI and Allen-Bradley PLC3 / PLC5 based
Control System.
5
Justification
How did we justify this project.
Cost avoidance.
Process runs 24x7x364. Uptime is extremely important.
Control equipment installed in 1985, so spare parts are getting
hard to find.
Available support for this system was minimal from Mallinckrodt
& vendors.
6
Phase 1 (programming standard)
Why develop a Programming Standard.
There are basically two different control systems in our
facility. • Traditional DCS which has a programming standard.
• The rest are a combination of WonderWare, FactoryLink, RSView®(x),
Siemens, with various types of Allen-Bradley PLC based controls. No
common programming structure.
7
Phase 1 (programming standard)
No common programming
platform for the PAC based
systems.
Develop a programming standard for
future PAC systems that has the same
look & feel as our DCS systems.
For operators, the HMI graphics look
the same. This helps when moving
between manufacturing units with
different control systems.
For programming, the control module
structure is similar between the
DCS & PAC based systems.
8
Phase 1 (programming standard)
What was needed to develop this standard.
Partnered with Rockwell Automation Systems and Solutions
Business (SSB) & local Systems Integrators to help develop the
standard.
Functional Specification jointly developed with Rockwell
Automation (you have to know what you want)
Design Specification developed by Rockwell Automation based on
the Functional Specs.
Test Specification needed to validate the various control modules.
• Once validated, these Standard Modules could now be used on
multiple projects without re-validating.
Lessons learned
• It won’t be perfect the first time. You will miss something.
• We are on version 3 of our standard.
9
Phase 2 (the upgrade)
Scope
Partnered with Rockwell Automation Systems and Solutions
Business (SSB) to execute the project.
• This required very close collaboration of the project team.
Process Engineer, Automation Engineer, Validation, Rockwell
Automation Engineer. All the way through the entire project.
Replace a legacy Allen-Bradley PLC based Control System & HMI.
• Total PLC code re-write (~10K rungs) using S88 phase manager &
FactoryTalk® Batch. Utilized the Standard Control Modules created in
phase 1.
New HMI Graphics re-drawn from original HMI graphics.
• ~200 HMI screens (process graphics and Batch Record screens)
Re-created 60 Digital Batch Records and 6 Process &
Environmental reports utilizing Microsoft Reporting Services.
Utilize existing field I/O to reduce cutover time and risk.
10
Phase 2 (the upgrade)
Scope – continued
As-Built 64 Process Logic Descriptions (PLD’s) per existing PLC
code for accuracy. Then updated to a new Class-Based document
format.
• Managed only one document per operation per Class. Used Excel Mail
Merge function to populate the document with the correct devices. This
concept worked extremely well. A separate spreadsheet contained the unique
elements relative to all module classes in the application.
• 33 Unique Module Designs, + 31 Instances of those Designs. Class-based
documentation can be viewed and printed and be specific to the instance with
respect to Equipment and Devices.
• Class-based Methodology also served (i.e. focused us) to clearly identify all
differences between instances of the same class. (Fail open vs. Fail Closed
valves, extra instrumentation, etc.)
11
Phase 2 (the upgrade)
12
Process Logic Description Development Structure
Documentation Development:
• Each unit on Train 1 has its own Process Logic
Description in MSWord.
• MSExcel is used to provide the Mail Merge data to
each document. The Train 2 documents can be
automatically generated at the click of a button.
• Modifications to one Train are automatically
captured for the other Train.
• Support Systems require their own documents.
Phase 2
13
Mail Merge Demonstration
What’s Out, What’s Left, What’s New
14
Out
1 - PLC3
2 - PLC/5’s
DOS based prog. terminal
4 - FactoryLink HMI operator
terminals (WinNT)
Oracle Historian/Datalogger (WinNT Server)
Batch Record Server (WinNT Server)
3 - Alarm printers
Serial interface to Delta V
control system in adjacent
building
Left
All field I/O
•1771 Remote I/O
•1756 CLX I/O
•Flex I/O
•Profibus PA
•AS-i bus
•Ethernet VFD’s
Used existing I/O to
reduce cutover risk. Plan
to upgrade I/O in the
future.
What’s Out, What’s Left, What’s New
15
All Servers and Terminals are virtualized.
Servers
FactoryTalk® Batch
FactoryTalk® Historian Site
Edition
FactoryTalk® VantagePoint®
FactoryTalk® View Site Edition
(primary & secondary)
FactoryTalk® Directory
MS SQLSvr
Hardware
3 - HMI workstations
1 - Engineering terminal (all thin clients)
3 – ControlLogix® L74 controllers (all redundant)
13 – Ethernet EN2T comm. Modules
3 – Remote I/O Modules
(interface to existing Remote I/O)
Delta V Ethernet Virtual Interface
Module (VIM)
3 – Stratix 8000 Network switches
2 – Blade servers for our existing
VMWare
Batch Record/Graphics printer
Virtualization
Built upon our existing VMware infrastructure.
Two identical systems in two separate buildings.
Automatic backups of virtual images on periodic basis for disaster recovery.
Currently running 16 servers & 30 workstation images on VMWare.
And growing.
Eliminates the need to rebuild a computer due to hardware failure.
Saved ~$100K in computer cost by using virtualization.
16
Factory Acceptance Testing
Joint venture with Rockwell Automation and local Systems Integrators.
Accelerated schedule. Startup moved up by two months.
Performed on-site utilizing existing VMWare system while production unit
was still operating. Allowed us to test the throughput on our actual system.
Three months duration working two shifts.
Still finishing design/code development when FAT started (required
numerous code merge procedures to avoid invalidating FAT efforts).
Leverage FAT testing for Validation. (did not re-test sequencing functions
during shutdown OQ)
17
Qualification & Startup (IQ/OQ)
Installation Qualification (IQ)
Old controls removed and new controls installed in 4 days ready for I/O
checkout.
Round the clock testing.
2 weeks duration working two shifts. Every device manipulated.
• 1000 digital, 500 Analog.
Operational Qualification (OQ)
Introduction of utility chemicals with water as the product
Round the clock coverage
2 weeks duration running multiple batches
18
Qualification & Startup (PQ)
Process Qualification (PQ)
Introduction of actual product.
Five batches were run initially.
• All five passed quality requirements.
Best of all:
No Safety issues
No Environmental issues
No lost batches
19
What went well
20
What went well
Programming Standard (HMI & controllers).
Excellent documentation for FAT testing and validation.
Good in-depth reviews of the Process Logic Descriptions.
VMWare & Thin Client usage.
Utilize our existing virtualization infrastructure to lower control system
maintenance cost & downtime.
FAT performed on-site and on our existing VMWare system. We can
then see how the system will perform before startup.
Get your people involved. Process Engineers, Facilitators and
Operators all know the process the best.
Near-duplicate Virtual Environment at Rockwell Automation Office used
for development, functional testing, mid-project demonstrations
(Webex’s), troubleshooting, and long term support .
Lessons learned
21
Lessons learned
You can’t have enough help.
Get involved earlier in the details of the program code to better
understand its structure by the time the FAT & Startup begins.
Validation protocol reviews could have been better coordinated.
Needed a better understanding of the VMWare infrastructure in our
facility.
Get a better handle on the Ethernet Network Infrastructure in our facility.
Get your Systems Integrator(s) involved with all aspects of the project…
Network Architecture and Configuration, other IS-related functions.
Going Forward
22
Future State
The Programming Standard has been used on 4 upgrade projects to
date, with another grass roots project set to begin in Q3/2014.
Virtualization will only grow in the future as our current control systems
become outdated. This has greatly reduced our computer maintenance
and infrastructure cost.
Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.
We care what you think!
On the mobile app:
1. Locate session using
Schedule or Agenda Builder
2. Click on the thumbs up icon on
the lower right corner of the
session detail
3. Complete survey
4. Click the Submit Form button
23
Please take a couple minutes to complete a quick session survey to tell us how we’re doing.
2
3
4
1
Thank you!!
Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.
PUBLIC INFORMATION
Thank you for participating!
Please remember to tidy up your work area for the next session.
We want your feedback! Please complete the session survey!