NAVAIR Public Affairs Release Number 10-0442 with Distribution Statement: A -- Approved for Public Release; distribution is unlimited.

Utilizing Data Mining to Influence Maintenance Actions

Thomas Young Naval Air Systems

Command Lakehurst, NJ

Matthew Fehskens Naval Air Systems

Command Lakehurst, NJ

Paavan Pujara Naval Air Systems

Command Lakehurst, NJ

Michael Burger Naval Air Systems

Command Lakehurst, NJ

Gail Edwards Naval Air Systems

Command Lakehurst, NJ

Abstract - For Aircraft Launch and Recovery Equipment (ALRE), the goal is to get planes in the air and ensure they land safely. Consequently, a high operational availability (Ao) is crucial to ALRE operations. In order to ensure high Ao, it is crucial that the amount of maintenance, both corrective and preventative, is kept to a minimum. Historically, improvements have been reactive in nature to satisfy the Fleet's needs of the moment and are never implemented across the Fleet. One approach to improving maintenance practices is to use historical data in combination with data mining to determine where and how maintenance procedures can be changed or enhanced. For example, if a maintenance manual says to remove three electronics boxes based on a built-in test (BIT) code, but historically, the data shows that removing and replacing two of the boxes never fix the problem, then the maintainer can be directed to first remove and replace the box which the data suggests is the most-likely cause of failure. This type of improvement is where data mining can be used to enhance or modify maintenance procedures. The Integrated Support Environment (ISE) team and the Integrated Diagnostics and Automated Test Systems (IDATS) team of NAVAIR Lakehurst are jointly investigating the use of data mining as an important tool to enhance ALRE systems and to potentially decrease preventive maintenance on-board Navy vessels , thereby reducing the total cost of ownership. The authors' approach is to use maintenance actions, system performance data, and supply information to draw a clear picture of the failures, diagnoses and repair actions for specific components of ALRE systems. The authors are using a commercial off-the-shelf (COTS) data mining suite, called SPSS Clementine, alongside custom software tools to detect the meaningful, yet hidden, patterns within the mountain of data associated with ALRE systems. SPSS Clementine is one of the data mining industry's premier tools, allowing rapid development of models for data mining. Additionally, ALRE subject matter experts (SMEs) were consulted to ensure the validity of the teams' findings. The combination of modern data mining practices and expert knowledge of ALRE systems will be leveraged to improve the maintenance performed at the O-level and to possibly understand why the failure happened in the first place. This paper will describe the forthcoming investigation exemplifying how the data warehouse holding various sources of data about ALRE systems will be utilized to improve the education of maintainers and to enhance maintenance practices, to understand the cause of component failures, as well as provide solutions to diagnose these failures. Utilizing the knowledge and expertise of database systems and data mining which the ISE team provides, combined with SME knowledge, non-trivial

solutions to ALRE maintenance practices shall be uncovered to improve the maintenance environment on-ship

Keywords-data mining, text mining, statistical analysis, operational availability, maintenance

I. INTRODUCTION In a climate where maintenance ailments are cured on an

as-they-come basis, problems are solved when they’ve become a significant burden on the maintainer. In such a reactionary environment, too much emphasis may be placed on the issue du jour. In order to better assess issues before they become “head hurters” it is useful to analyze the multitude of data sources which describe the entire maintenance process. In previous research, the Integrated Diagnostics and Automated Test Systems (IDATS) team, had investigated data mining algorithms on F-18 data [1] as well as methods for properly structuring a database system to store aircraft data [2]. This paper builds on those topics, aiming to investigate the development of models for use in the detection problems before they become critical, by employing a data warehouse of Aircraft Launch and Recovery Equipment (ALRE) data and the techniques of data mining.

II. RESEARCH AND DEVELOPMENT The IDATS team, utilizing the Integrated Support

Environment (ISE), first investigated the variety of data sources available for ALRE systems. Useful sources identified included data from maintenance action forms (MAFs), supply, performance measuring, and availability (AO). Each of these data sources provided a piece of the maintenance puzzle.

Procured from the Open Architectural Retrieval System (OARS) system, MAFs provide information on the action performed by the maintainer at the system. Information provided includes the type of action performed (corrective or preventative), which piece of equipment was maintained, on what date the action was performed, the length of time it took to perform the action, the problem description, the actual solution, and a unique identifier, known as a job control number (JCN).

978-1-4244-7961-0/10/$26.00 ©2010 IEEE

NAVAIR Public Affairs Release Number 10-0442 with Distribution Statement: A -- Approved for Public Release; distribution is unlimited.

The OARS system also provides supply information. Supply information includes all the data relative to parts requested for a particular maintenance action, also uniquely identified by a JCN; using the JCN, MAF and supply data can be integrated. In addition to JCN, the supply data provides the National Item Identification Number (NIIN), the amount of time it took to receive the part, and the quantity issued.

The performance-measuring data investigated for this paper was pulled from the Aircraft Shot and Recovery Log (ASRL) system. This data, as the name implies, is a record of all events for launches, or the taking off of aircraft, and recoveries, or the landing of aircraft, onto naval aircraft carriers. The data provides on which of the various catapults (for launches) or arresting gear (for recoveries) the event occurred, the type of aircraft involved, its weight, wind speeds, and aircraft end speed.

Finally, availability, or AO, information was procured utilizing a system known as Ready For Tasking, Equipment (RFT-E). The RFT-E process gives a snapshot of what caused an equipment failure on-ship when that piece of equipment down. Also included in this data is the specific piece of equipment (i.e., which catapult or arresting gear) and the percentage of events missed as a result of the failure.

Data was collected from each system for calendar year 2009 and, after analyzing these source data structures for entry into model development, it was decided that MAF data would be the best entry point for investigation. The MAF data would be used by itself and in conjunction with ASRL data, to develop models. The MAF data would be related to ASRL data in hopes of identifying a series of events which would lead to the downtime of a system. Additionally, the team would investigate the relationship in MAF data between equipment, identified by Equipment Identification Codes, or EICs, within a system (catapult or arresting gear) which failed in close temporal proximity to each other.

In order to ensure the fidelity and validity of the source information, data mining requires the cleansing of source data, as part of a step known as commonly referred to as data preparation. This step in the data mining process is the most effort-intensive step, usually claiming about 75% of the actual data mining process [3]. This step ensures that the data is in the correct format (i.e. all integer fields contain integers), removes unnecessary data (i.e. a cancelled maintenance action), and that

the data is accurate. Inaccurate, or “dirty,” data occurs for a multitude of reasons, such as human error. Through the knowledge garnered from ALRE subject matter experts (SMEs), as well as the SMEs themselves, computer algorithms have been developed to decrease the effort to cleanse the data, as well as increase the accuracy; what once took months of manpower to do, is now done in minutes through computer analysis with a 94 percent accuracy.

III. METHODOLOGY

A. Data Processing Methodology and Expectations The team decided two separate approaches would be

required to achieve models for the identified objectives. For the use of MAF and ASRL data, trend analysis would be employed to explore the relationship between ASRL-recorded events and a component failing. To assess the relationship between two EICs failing in close temporal proximity, the team decided the MAF data would be analyzed through the use of the apriori algorithm.

In order to analyze MAF and ASRL data, both datasets were merged by date and carrier number. A common date field was generated for the merged dataset drawn from the date of maintenance action and the date of aircraft recovery, respectively. Null values were used in inapplicable fields, i.e. fields from the ASRL dataset were labeled null when adding a MAF record. When sorted by carrier number and common date, the resulting dataset shows clearly relationship between records in ASRL data and records in MAF data, with data for each carrier left disjoint.

Some major measures were calculated from the dataset. The first, operational time, is used to measure the effects of other variables. Under certain conditions, an EIC may experience more corrective maintenance actions. In order to identify these conditions, it is necessary to determine how conditions changed between maintenance actions and the resulting effects on the number of corrective actions. A certain amount of corrective actions are inevitable, but it is preferable that the amount of time between corrective maintenance actions is maximized. Two other measures are used to analyze certain conditions that could cause an increase or decrease in operational time. The first, aircraft composition, seeks to verify the idea that heavier aircraft tend to cause more maintenance issues for certain

Figure 1. Percentage of Daytime Recoveries vs. Operational Time Figure 2. Percentage of Daytime Recoveries vs. Cumulative Distribution

of Operational Time. The intersection of lines is (0.5, 0.5).

NAVAIR Public Affairs Release Number 10-0442 with Distribution Statement: A -- Approved for Public Release; distribution is unlimited.

pieces of equipment. Hence, if more heavy aircraft are utilized, operational time should trend lower. Likewise, due to the relative difficulty of night recoveries, we expect that the resulting increase in variation of secondary landing facts (angle of approach, speed, etc.) would increase the amount of maintenance actions needed for certain components. Therefore, we expect that as the ratio of day landings declines, operational time declines as well.

All three major measures were computed from this sorted dataset, with each carrier calculated independently. We define operational time (o) independently on EICs as the number of days between two corrective actions for an EIC. Likewise, aircraft composition (AC) is defined independently on EIC as the number of each aircraft recovered between two corrective actions for an EIC. AC is normalized across all aircraft types in order to remove a linear trend from our analysis. Additionally, the percentage of day and night landings (PL) was similarly defined on EIC as the ratio of the number of aircraft recovered during the day and the total number of aircraft recovered between two corrective actions for an EIC.

A fourth major measure, preventative maintenance benefit ratio, was defined as the ratio of the number of preventative maintenance actions between corrective actions for an EIC to the operational time for the EIC in the same period. This measure tests the benefit received by increasing the amount of scheduled maintenance actions performed on a piece of equipment. The ratio is important to function as a normalized measure. Likewise, the absolute preventative maintenance benefit is defined as the number of preventative maintenance actions between corrective actions for an EIC. It measures the baseline trend of the benefit obtained by increased maintenance.

B. Apriori Algorithm The second algorithm used to investigate ALRE

maintenance data was the apriori algorithm, commonly referred to as Market-Basket Analysis. The apriori algorithm identifies objects in data which appear together frequently. For instance, in a supermarket, the items peanut butter, jelly, and bread often appear together when apriori analysis is performed; this is because when a customer purchases one, they often purchase the others. Apriori allows the creation of sets of variable size and based on an occurrence threshold, which says that a set of

items must occur a specified number of times to be considered a frequent grouping.

As stated before, this was an investigation on EICs failing in a time-based proximity; that is, EICs on which maintenance occurred within a week of each other for an ALRE system would be grouped together and, if they occur a specified number of times, were considered frequently associated EICs. As clarification, it should be noted that it was ensured that EICs which are part of the same system were grouped together, avoiding useless results, such as grouping catapult and arresting gear EICs together because they occurred within the temporal proximity established.

IV. RESULTS OF DATA MINING

A. ASRL and MAF Data Analysis Using the above methodology, an attempt at identifying

characteristics of an aircraft recovery by the arresting gear was made. The first steps were to look through the data and determine what fields could assist in provide interesting results for identifying characteristics of ASRL data which could be correlated to a corrective action in the MAF data. The findings produced two significant results. First, analysis comparing o to PL identified that the more night landings a ship performs, the less operational time that components tend to see. Fig. 1 shows the findings of these results. The graph displays the occurrence of day landings (on the X-axis) versus the operational time (on the Y-axis). Looking at the graph, one can see that as fewer landings occur during the day (moving left on the graph), operational time goes down, or more corrective maintenance needs to be done on the arresting gear. This trend is more readily seen in the cumulative operational time graph in Fig. 2, where 40% or less daytime flights accounts for little increase in operational time.

One may also notice that the distribution of the plot is indicative of normal curve, which suggests that more daytime landings also results in a decrease in uptime. The historical data is biased towards this sort of normal curve. Aircraft recoveries happen at both day and night, hence it would be expected that, longer a system is up, the more recoveries it will experience, and that the timing of the recoveries will be biased towards the underlying rate of day/night recoveries. That pointed out, the rate of decrease in uptime decreases much more dramatically as the percentage of night recoveries increases. The rate of

Figure 3. F-18 Percentage Composition vs. Operational Time

Figure 4. Preventative Maintenance vs. Operational Time

NAVAIR Public Affairs Release Number 10-0442 with Distribution Statement: A -- Approved for Public Release; distribution is unlimited.

decrease in uptime has a much more gentle decrease as the percentage of day recoveries increase. Fig. 2 also makes this difference clear. If the ratio of daytime flights did not matter, the cumulative distribution of operational time would pass 0.5 at the mean of the ratios of daytime flights, which is slightly less than 0.5. That it does not suggests that more operational time is gained when the percentage of daytime flights is above the mean.

A second finding, also related to corrective actions, was to look at the type of aircraft being recovered within an event. The findings pointed out that the more F-18 landings that occurred on an aircraft, the more corrective actions necessary. Again, in Fig.2 reflects these findings with the occurrence of F-18 landings (on the X-axis) being compared to operational time (on the Y-axis).

Again, the distribution in Fig.3 tends to follow a normal distribution. On the graph, as more F-18s land (moving right on the X-axis), there is less operational time (moving down on the Y-axis). Like the nighttime graph, Fig. 3, as the amount of F-18s which land passes about 40 percent, meaning 40 percent or more of the flights on a carrier are F-18s, the more significantly operational time goes down, or the more corrective maintenance actions go up. As the graph moves left, towards no F-18s being recovered, operational time overall is better, and, even when no F-18s are being launched, the graph shows significant uptimes. The reduction in the uptime as percentage of F-18s decreases may be the result of a lack of data. F-18s make up a large percentage of the flights, therefore performance data over a long period of time when few F-18s are recovered simply do not exist. If a decrease is found, it will not be dramatic, as the previous mentioned results when no F-

18s are flown demonstrate.

A third result is startling and revealing. A study was conducted to determine the effectiveness of preventative maintenance to increase the amount of time between corrective maintenance actions. Fig. 4 demonstrates the absolute preventative maintenance benefit to operational time. As can be seen, a slight positive trend exists, although a much steeper linear trend was expected. More informative is the relative preventative maintenance benefit, shown in Fig. 5. The expected result was a linear relation, albeit with a negative slope indicating diminishing returns from additional maintenance. That the graph takes on the form of the inverse linear function suggests that operational time does not significantly change at all with increasing preventative maintenance.

B. Analysis of EICs EIC analysis via the apriori algorithm has led to some

interesting results. The maintenance records were processed such that a flag was set if a corrective maintenance action occurred for a certain EIC during a given week per carrier. The resulting dataset contained about 900 records, with each record corresponding to one week, and allowed analysis of those corrective maintenance actions that occurred within relatively close time proximity while treating each carrier independently. Results were considered non-trivial only if they exhibit a 2 percent support, that is the records existed in at least 2 percent of the dataset (i.e. 18 records or weeks), and with a confidence of 70 percent. That is to say, a consequent corrective maintenance action occurred in conjunction with the conditional actions at least 70 percent of the time. Table 1 shows the non-trivial results. As further explanation, the first line should be read as “In 74 percent of cases when A/G Wire Support Assy. & Instl. Stbd. (Midspan) required corrective maintenance, A/G Drive System Purchase Cable or Cross Deck Pendant required maintenance in the same week. This combination was found in about 36 weeks out of the 900 examined, or 36 weeks out of the 48 weeks when the A/G Wire Support Assy. & Instl. Stbd. (Midspan) required corrective maintenance.” Even if these non-trivial combinations do not occur very frequently, they can suggest a useful plan of action.

V. HOW TO USE THESE RESULTS The analysis of ASRL recovery data versus MAF corrective

action occurrences, provided outputs which identified that corrective maintenance happens more often in environments

Figure 5. Preventative Maintenance vs. Benefit

TABLE I. COMMON JOINT EIC FAILURES, WEEKLY

Antecedent Consequent Support % Confidence % A/G Wire Support Assy. & Instl. Stbd. (Midspan) A/G Drive System Purchase Cable or Cross Deck Pendant 3.867 74.286 A/G Eng Accumulator Fluid Level Assy. and A/G Engine CROV Valve Display Assy. A/G Drive System Purchase Cable or Cross Deck Pendant 3.094 75 A/G Wire Support Assy. & Instl. Stbd. (Outboard) and A/G Engine CROV Valve Display Assy. A/G Drive System Purchase Cable or Cross Deck Pendant 2.431 77.273 A/G Engine Piping Assy. and A/G Drive System Purchase Cable or Cross Deck Pendant A/G Engine CROV Valve Display Assy. 2.21 70

NAVAIR Public Affairs Release Number 10-0442 with Distribution Statement: A -- Approved for Public Release; distribution is unlimited.

that have either more nighttime landings or more F-18 landings. These findings in turn, can help make decisions, by providing a jumping point to do a deeper dive into the data. One area, for instance, to go would be to investigate if it would be prudent to invest more time doing preventative maintenance to avoid the necessity of corrective actions on carriers with a higher occurrence of nighttime flights or F-18 flights. In light of the benefit provided by additional preventative maintenance, alternative solutions may be considered, such as a policy that expects and reacts more efficiently to corrective maintenance. A study into the best practices in industry may be warranted in order to find the most useful solutions to the problem.

An additional area of investigation would be to examine those component combinations discovered in the apriori analysis. These parts may have some interaction resulting in increased wear on one part as the other part wears. In addition, replacement of both parts in one action may result in savings in man hours due to a reduction in overhead and increased operational time if the components can be replaced in parallel. The authors would stress that these findings ought not to be implemented directly and are instead recommendations for further investigation by engineering and maintenance teams.

The principles proven by these results can be used not only to identify what types of maintenance should be done and with what frequency, but also how to do maintenance. Using the MAF data, corrective actions against the same equipment can be compared, identifying what one maintainer did versus another to alleviate similar problems.

Additionally, the findings of data mining, in general, are not meant as a device to prognosticate. Data mining investigations can rapidly alert the SMEs to areas such as altering technical manuals to provide better routes to problem components or implementing more or less preventative maintenance to, ultimately, cut down the stress to the maintainers, as well as provide models which can be utilized to prognosticate the future needs in maintenance.

VI. CONCLUSIONS The ISE and IDATS team continue to work on data mining

efforts, by investigating the data further and expanding the base of data from which to analyze. The major effort of the team is to first link data from throughout NAVAIR to gain a full-scope picture of what is going on out in the Fleet. This, however, includes not only gathering data from expansive sources, but also ensuring the validity of that data.

Utilizing this full-view approach, the team will look into relationships the range from on-aircraft built-in test (BIT) reporting to supply waiting times. The aim of the team, moving forward, is to assist the various competencies of NAVAIR in making better, data-based decisions.

Data mining, although relatively young, is a discipline which will radically change the way business is done throughout the Department of Defense. Investing time into ensuring the quality of data and the results which data mining can produce will be a leap ahead for maintenance in the military world.

VII. REFERENCES [1] Meseroll, R. J., Kirkos, C. J., and Shannon, R. A. “Data

Mining Navy Flight and Maintenance Data to Affect Repair.” AUTOTESTCON, 2007 IEEE. pp. 476-481.

[2] Kirkos, C., Meseroll, R., Edwards, G., Fehskens, M. “Analyzing Automated Maintenance Architectures to Provide Flexible Smart Maintenance Capabilities.” AUTOTESTCON 2008, IEEE. pp. 382-388.

[3] Pyle, D. Data Preparation for Data Mining, Morgan Kaufmann, 1999. pp 11.