1 | Page Root Cause analysis report Root Cause Analysis Report Date: 07/12/2014 DEFINITION Event (Problem statement) Unavailability of Water Injection Pump station P 3221A/B/C on Oct 17, 2014 Date Occurred 17 October 2014 Installation Lekhwair RCA # ORIP ID L00035 SIGNIFICANCE Safety N/A Environmental N/A Production 50000 M3=1.5 million USD Maintenance 1500000 USD for Motor Rewinding, 500000 USD for PUMP overhaul Frequency Seven rewinding for C , three rewinding for B and two rewinding for A RAM Category Actual : A4D Potential: A5D RCA Team RCA Sponsor: Khamis Busaidi, ONO Team leader: Ali Lamki, ONOF RCA Facilitator: Jorge Pirela Galban, ONO6TR RCA Function Support: Guolin Weng, UOM52 Team Members: Amri, Humaid UIE3 (UEE2) Alawi, Salim ONO62 Wardi, Abdulaziz ONO65 Brown, Tim UEP14 RCA Team Charter EXECUTIVE SUMMARY <Executive summary is to include brief event description, root causes and recommendations and conclusion > On October 17, 2014, all three water injection pumps were unavailable due to A motor grounded in Febr, 2014, B motor grounded on Oct 17, 2014, and C motor grounded on October 16, 2014. This incident caused at least 5000 cubic meter that equals to 1.5 million USD production profit loss in oil deferment and more than 2 million maintenance cost. These three motors have a history of rewinding frequently, such as 7 rewinding for P 3221C. Root Causes: 1. The start flow is at far higher than should be (100% or plus than 10%), 2. The discharge pressure is too much higher than need (160 bar vs 100 bar) These two conditions make the motor start current from 1800 A to 2600 A while the normal start current should be 500A to 800A. Motor overheat happens at start up rather than running. Main Recommendations:
Transcript
1 | P a g e
Root Cause analysis report
Root Cause Analysis Report
Date: 07/12/2014 DEFINITION
Event (Problem statement)
Unavailability of Water Injection Pump station P 3221A/B/C on Oct 17, 2014
Date Occurred 17 October 2014
Installation Lekhwair
RCA #
ORIP ID L00035
SIGNIFICANCE
Safety N/A
Environmental N/A
Production 50000 M3=1.5 million USD
Maintenance 1500000 USD for Motor Rewinding, 500000 USD for PUMP overhaul
Frequency Seven rewinding for C , three rewinding for B and two rewinding for A
RAM Category Actual : A4D Potential: A5D
RCA Team RCA Sponsor: Khamis Busaidi, ONO Team leader: Ali Lamki, ONOF RCA Facilitator: Jorge Pirela Galban, ONO6TR RCA Function Support: Guolin Weng, UOM52 Team Members:
<Executive summary is to include brief event description, root causes and recommendations and conclusion> On October 17, 2014, all three water injection pumps were unavailable due to A motor grounded in Febr, 2014, B motor grounded on Oct 17, 2014, and C motor grounded on October 16, 2014. This incident caused at least 5000 cubic meter that equals to 1.5 million USD production profit loss in oil deferment and more than 2 million maintenance cost. These three motors have a history of rewinding frequently, such as 7 rewinding for P 3221C. Root Causes:
1. The start flow is at far higher than should be (100% or plus than 10%), 2. The discharge pressure is too much higher than need (160 bar vs 100 bar)
These two conditions make the motor start current from 1800 A to 2600 A while the normal start current should be 500A to 800A. Motor overheat happens at start up rather than running. Main Recommendations:
2 | P a g e
Root Cause analysis report
1. The min flow control should have delay control at 1 to 2 mins to open at pump start up 2. The pump impeller should be trimmed down diameter 15% to 20% or de-stage to reduce the discharge
pressure, the saving from power will be 1000 KW that equal to USD 613000 per year at 0.07 USD/(KW*h), for twenty four years will save 15 million, more than 5 million saving for 10 pumps per year
Conclusion: The motor grounding failure is caused by in-appropriate design of min flow valve and pump oversize, The quality of re-winding should be excellent and acceptable from any industrial standard and practice, PDO should appreciate what they have done for us as the motor is overloaded too much at start-up.
INTRODUCTION
P 3221 A/B/C are water injection pumps for Lehkair to supply high pressure water at 100 bar to be injected into the well reservoir to lift the oil to the ground well head. Normally two pumps are needed to meet the process requirement. Unavailability of all three pumps will terminate the oil product in Lehkair totally. On October 16 and 17, B and C pump motor were shutdown due to the motor grounding protection was initiated and confirmed later the motors are grounded in their stator winding. A pump was down since February, 2014. This resulted in the unavailability of injection water and caused more than 5000 M3 in oil deferment.
Sequence of Events: P 3221 C
Date & time Event
October 16, 2014 P 3221C shutdown when the motor grounded 7 (motor grounding)
February- October, 2014
237 stop/shutdown for C motor due to pump high axial movement, seal failure, low suction pressure
2011-2014 10 mechanical seal failure after mechanical seal modified from single seal to double seal with clean water as barrier fluid
2008 C motor back in service? P 3221 C motor grounded 7
April, 2007 C motor came back service (EMIB) , RCA completed on May, 2007
Jan, 2007 C motor grounded (EMIB) 6
June, 2005 C motor grounded (EMIB) 5
2005 Pump de-staged from 11 stages to 9 stages
Early 2004 C motor back service (EMIB)
2003 C motor grounded(EMIB) 4, RCA completed on Oct , 2003
2001 C motor back in service (EMIB)
2000 C motor grounded ( EMIB) 3
1998 C motor grounded (WESCO 2
1997 C motor grounded (WESCO) 1
1992 C terminal box overheat ( By motor vendor)
1991 P3221 C commissioned
3 | P a g e
Root Cause analysis report
FINDINGS
<Describe the Analysis done to the Hypotheses here. This section should exhaustively take the reader through the step-by-step process of identifying the root cause with supported evidence>
1. All P 3221A/B/C motor had winding grounding problem and short winding life (from shortest 1 year to 6 years,
average 2.5 years vs 20 years life expected), C motor had more frequent failure. So RCA team focal on C
pump and motor. Then Team will overview all three pumps and motors to see root causes can apply to all of
them.
2. As the motor running current is only 280 A vs the normal design 380A, RCA believe that the normal running
will not overheat and overload the motor that deteriorate the life of winding insulator. RCA team focal on the
motor start to find out any relation between pump and motor
3. As the motor did not have RTD HH protection initiated during start-up and running and HH temperature was
reduced to reasonable value, RCA has more confidence that pump start-up could lead to accelerating motor
winding failure. As C pump had frequently start –up (about 300 times from February to October, 2014).
4. From RCA study, the team also found all motors from new one and re-winded by two different contractors(
WESCO and EMIB) had very short life, RCA team believe that winding quality should be ruled out from the
root cause
5. From the study, RCA team found that Motor start-up for pump is determined by flow times discharge pressure.
The discharge pressure is oversized (160bar vs 100bar needed). Motor start-up current is 5-8 times of the
running current. The total flow of the pump at the start-up will be variable. The higher pump flow, the higher the
motor start-up current. In actual practice, the pump discharge valve should be closed as less as possible to
protect the motor from over-current. Then there are two possibility that high flow could happen at start-up:
5.1 One by-pass valve could be fully opened at start-up ( Evidence B4: P3221C Start up PI Trend)
5.2 The discharge control position could be more than 10% or discharge valve could be eroded to make
the discharge flow higher than that of design
6. In order to test the correctness of our hypothesis, we asked the electrician to test the motor start-up current
and found that the motor start-up current is from 1800A to 2600A.(Evidence B1: P 3221C motor start up
current) While according to calculation, the motor start-up current should be less than 500A-800A. This test
confirms that our hypothesis that higher discharge flow and very higher pump flow are the two major
contributions to the motor winding early failure.
7. The team also found that de-stages from 11 stages to 9 stages helped increase the motor life; this will be
another evidence to support our hypothesis.
8. RCA also found that B motor failure was directly impacted by the not enough cooling due to one piece of the
DE fan baffle plate falling down. But the fan baffle falling down could be caused by high start-up current.
9. RCA team found that B pump shaft has high running out (0.07 mm max vs 0.04 mm API standard) and there is
high potential that C pump could has similar high running out. This high running out or shaft bending could
lead to high pump axial movement and high frequent mechanical seal failure. (11 mechanical failures from
2011 to 2014 after the mechanical seal modified from single seal to double seals with clean water as buffer
liquid). This mechanical seal failure frequency is far more than what should be (1 failure every three years).
RCA team assumed that trimming the impeller OD or de-stage will help increase the seal life as well.
10. During RCA study, the team also found that the pump pressure oversize cost PDO extra electricity usage at
US$ 613000 per year and 15 million US$ in its 24 year service just for one pump. It will save PDO 1 MW for
one pump by trimming or de-stage.
11. RCA suggested that this RCA report to be shared among PDO such as UEOD, engineering department,
project team, oil and gas department as the learning and opportunities of saving for PDO could be potentially
more than 5 – 10 million US$ per year.
4 | P a g e
Root Cause analysis report
CONCLUSIONS
Root Causes:
1. The start flow is at far higher than should be (100% or plus than 10%), min flow valve could be fully opened at start-up
2. The discharge pressure is far much higher than need (160 bar vs 100 bar needed)
5 | P a g e
Root Cause analysis report
RECOMMENDATIONS <The recommendation is to be made considering: a). Corrective Action to fix the present problem; and b). Corrective Action on System to prevent recurrence>
# ACTIONS ACTION PARTY DUE UPDATE
1. Check with WEIR PUMP or a third party about hydraulic
balance including the balance drum size after de-stage from
11 to 9 stages: After checking, if the balance drum size is
correct, then Route A should be ruled out from Root Causes(
Route A). If drum size is wrong balance drum size is to be
replaced to have proper hydraulic balance on pump.
Note: Due date to be revised after study results.
UEP14R 30 June 2015
2. To evaluate the need to perform impeller trimming or de-
stage accordingly (Route B).
UEP14R 30 June 2015
3.1 Replace the discharge control valve if the control valve is
found that internal erosion is high (Route C)
ONO41L 31 Jan 2015
3.2 Review the min flow (recycling valve) logic, add the start up
delay for it if need
ONO43L 31 Jan 2015
4. Evaluate to improve fitting of fan baffle plate including nut
locking and plate quality. (Route D)
ONO62 31 Dec 2015
5 Provide trend indication for winding temperature in DCS and
PI.
ONO43L 31 Jan 2015
6 Review the motor start up procedure to see whether the half
an hour internal between starts is safe enough to protect the
motor (Route E)
ONO62 and UIE3 31 March 2015
7 Review the site procedures on proper start-up procedures ONO23L 31 March 2015
8 Review maintenance routine for HV motors to avoid
accumulation of contaminants in winding.
UIE3 30 June 2015
9 Review test result of winding samples and recommend
according to finding.
ONO6TR / UIE3 31 March 2015
6 | P a g e
Root Cause analysis report
KEY LEARNING
1. PUMP discharge flow concern: For pump system, the motor start power is variable and determined by the equation below:
P =
=
1-1 on normal running condition
Motor normal running current is 280A at A and if the motor was started at this flow, the motor start current will be at least 5 times of 280 A that will be 1400 A. This high current will potentially overheat the winding and even damage it quickly. We have cases that motor was damaged in one year after rewinding. In addition, this start current is after de-staged, the start-up current will be 20% higher than that of before. It would produce 1900 A start-up current. See B1: P 3221 C Motor start up current In order to protect motor at start-up, it is normal practice that discharge valve and recycle (Min Flow) valve should be closed as less as possible so the discharge flow could be reduced to make sure that motor is not overloaded at start-up. For example, P 3221 C start up flow is about 30% of normal flow at Point B, then, the motor start up current should be from 500A-800A as the continuing running current at B is about 99A (50A+30%*(50A+280A)). Assuming that start current at zero flow is 50A, and motor start current is 5-8 times of the running current. The actual power line will be level off at the high flow, but this calculation is enough to give us a guide how to start up the pump. Usually the centrifugal pump can run 2-3 minutes with the discharge valve closed. But in this case, the discharge pressure is very high that could result in hydraulic lock for discharge valve or control valve, for safety reason, the control valve can be opened a little to prevent such lock. For this pump, the control valve opened at 10% and minimum valve fully opened, then the discharge flow could be 110% as the normal flow rate. Refer to B4: P 3221 C Start up PI trend. The motor start current will be more than 1900 A. The actual test start current for P 3221C is from 1800A to 2600A that will
7 | P a g e
Root Cause analysis report
overheat and cook the motor winding insulator and reduce its life dramatically. As this start current only last 2-3 ms, the heat produced does not have enough time to transfer to the RTD and RTD cannot detect this high heat impact and HH temperature will not be initiated. Right now, the P 3221 C could be started at Point C. The actual flow rate refers to the flowing chart.
P 3221C PI data on July 24
2. Pump discharge head oversize concern: P 3221 A/B/C discharge head is 1600 meters now and the process need is only 1000 meters. The pressure throttle valve has to be applied to reduce the discharge head from 1700 meter to 1000 meter. This high discharge pressure will impact on the pump system in the following ways: o Too much load on the impeller and shaft that will cause higher vibration for pump o The start- up current will be higher than should be o In long term, the energy waste on the motor electricity will be huge.
If the discharge head is reduced from 1600 meters to 1100 meters, the new power need is about 2079 KW. The power saving will be about 1000 KW that is about US$ 613000 per year at 0.07USD/KW*hour. For this pump started in 1991, it will save 15 million USD on electricity in 24 years. In PDO there are more pumps on pressure oversize, 10 pumps similar oversize can cost PDO 5 million on extra electricity bill.
8 | P a g e
Root Cause analysis report
Appendices Attachment A: Cause & Effect chart
RCA cause-effect P 3221
Attachment B: Supporting evidences to investigation
B1: P 3221 C motor start up current tested on Oct, 2014
