Elysian Aerotech Final Thesis: ICDIC competition 2014

8/10/2019 Elysian Aerotech Final Thesis: ICDIC competition 2014

1/19


2/19

I. AbstractRPMHI is proposed as a means of dynamically recording in flight pilot biometrics with an aim to detect realtime variations in stress levels and perform long term analysis on pilot mental health. The proposed system

has two main system components, measurement and data infrastructure. In this work we lay out our

proposed methods for both the measurement and recording of in-flight pilot biometrics. This work is

aimed at commercial airline companies with the goal of reducing the risk of pilot mental breakdown.

II. IntroductionThe demand for air travel has significantly increased in the past decade. Many people choose flight as a

mode of transportation because it is convenient and time saving, however, the recent disappearance of

Malaysian Airlines flight MH370 suggests that the safety systems present in the aviation industry can be

improved. One of the biggest problems with the rapidly increasing demand for air travel is the upcoming

shortage of airline pilots, this shortage means that pilots are flying more now than ever before which is

leading to huge increases in stress in the cockpit. Stress in the cockpit can be disastrous for the passengers

of the plane as the pilot is required to make complex decisions in seconds while flying the plane. In some

cases, increased stress can lead to mental breakdowns in the cockpit, this was seen in March 2012 when

JetBlue Pilot Clayton F. Osbon left the cockpit and began screaming at passengers, he had to be retained

and refused re-entry to the cockpit by the cabin crew [7].

It has been recorded that human error can be held in account for between 58% and 97% of all aviation

accidents [8]. As human error and pilot wellbeing are inversely correlated, the proposed solution to the

problem of mounting stress and the decline of pilot mental health is to carefully monitor the mental health

of the pilot using an inflight biometrics measurement system.

The remainder of this paper is organised as follows: III. overview of system where a basic high level

abstraction of the intended solution is given.IV. baseline creation where the means of establishing a pilots

in flight baseline is examined. V. system datain which we detail the movement and treatment of data

within the proposed solution.VI. cockpit sensor array, this section constitutes the hardware overview of

the system. Second to last isVII. Summary and Conclusion where the details of proposal are rounded up.

Finally there isVIII. References

.

III. Overview of System

The proposed solution to the problem of monitoring the mental wellbeing of the pilot is an advanced

sensor array built into the plane cockpit that measures the biometrics of the pilot. The raw data from the

sensors is managed and analysed using a python/SQL server; this data is compared to the Baseline data

information which is gathered about the pilot in order to create a unique pilot profile. If the inflight

biometrics are sufficiently dissimilar to the Baseline data the server will provide advice to the pilot and

airline staff relating to the mental and physical wellbeing of the pilot. This solution can act as an early

warning system for pilots whose mounting stress may lead to a mental breakdown while in flight.

P a g e 1 | 18


3/19

Operation Flow Chart

Figure1:Overviewofproposedsolutionsystem

IV. Baseline Creation

This section of the report will describe a strategy for creating a unique set of Baseline parameters for each

individual pilot profile. The baseline consists of a set of values which describe the regular biometrics and

lifestyle of the pilot. As per the sensor design, the biometrics that will be measured are:

Heart Rate

Perspiration

Respiration

Hand Tremor

Motor irregularity (sudden twitching/fidgeting)

These Biometric parameters will also be measured during every in-flight monitoring session using the

sensors installed in the cockpit of the plane. The initial testing of the pilot biometrics establishes the pilots

regular biometric information. Before the initial biometrics tests, the Pilot will take a Profile Questionnaire

which aims to extract information about the pilots lifestyle. The information that will be sought from this

Questionnaire will include:

Pilot Sleep Patterns

Pilot Caffeine intake

In FlightMonitorSystem

DataAggregation

Post FlightData

Transfer

Baseline

Creation

P a g e 2 | 18


4/19

Pilot Alcohol intake

Pilot Stress levels on a day to day basis

Level of Exercise completed on a day to day basis by the pilot

Pilot tobacco intake (Does the pilot smoke?)

Pilot medication intake

Whether the pilot suffers from hyperhidrosis or hypohidrosis

Pilot Weight

Pilot Height

These lifestyle parameters will have an impact on the pilots Biometric information and it can be used to

verify the Biometric data. For instance, a pilot who regularly drinks coffee will have a higher than averageheart rate and a pilot who suffers from hypohidrosis will have a lower than average perspiration level. The

Biometrics measurement process will be repeated on every flight cycle completed by the pilot until there is

sufficient data to finalise the acceptable baseline range.

Figure 2: Baseline Creation Work Flow

The data below shows how the pilot biometrics would be displayed. This system allows the data managers

to determine the unique baseline parameters for the pilot; in this case the data was taken over the course

of 74 flights. The data shown below was generated using a random normal distribution using ranges from

academic studies aiming to measure Heart Rate [9], Perspiration [10], and Hand Tremor [11].

ProfileQuestionnaire

BiometricsMeasurement

DataAquisition

BaselineCreation

P a g e 3 | 18


5/19

Sample data for Biometric Testing

Figure 3.1: Heart Rate vs. Number of Flights

Figure 3.2: Perspiration (Sweat Quantity) vs. Number of Flights

Figure 3.3: Tremor Frequency (Hz) vs. Number of Flights

7475

76

77

78

79

80

81

82

83

84

0 10 20 30 40 50 60 70 80

Heart Rate (bpm) vs Number of Flights

0

0.01

0.02

0.03

0.04

0 10 20 30 40 50 60 70 80

Perspiration (Sweat Quantity) vs Number of

Flights

0

2

4

6

8

0 10 20 30 40 50 60 70 80

Tremor Frequency (Hz) vs Number of Flights

P a g e 4 | 18


6/19

Baseline Profile Example

Figure 4: A typical pre-flight read out of a pilots recorded status

The SQL database will be capable of providing the pilot or airline company with information about their

pilot. This information includes a breakdown of the pilots general mental health and whether the pilot is

deemed to be fit to fly the aircraft at present.

V. System DataThe key components of the proposed data storage and analysis reside on a small control server running a

Linux server OS such as CentOS, CirrOS, Red Hat or Ubuntu server. The reason for utilizing Linux operating

systems is the ease in which these systems may be customised and scaled for application. The server may

run on a simple a SoC (system on a chip) and networked drives in RAID configuration.

On the server SQL [1], OpenCV [2] and Python 2.7.4 [3] libraries will be installed. SQL will serve as the

database component for storing and ordering recorded data. SQL is currently one of the most popular

packages for general database creation, use and management .Python serves as the base primary

programming language to perform analysis, establish secure transparent connections when transferring

data and managing data. Python has the added benefits of being written in C. With the underlying C

libraries recompilation is possible to optimise the Python interpretation and minimise the data and

compute overheads. OpenCV is an open source package for computer vision applications. The computer

vision program written in C using OpenCV libraries is designed to help analyse pilot eye movement, posture

and general movement. The use of the aforementioned packages is not required these are merelysuggested suitable packages.

P a g e 5 | 18


7/19

Figure 5: A high level view of the proposed data operation workflow

Pre-Flight

Import: take-off data relating to both the pilot and co-pilot is imported from the central operating pilot

database, see below. The data is imported as a compressed encrypted SQL database file and contains the

major recorded data for both the pilot and co-pilot and all related metadata. This data will later be used to

create the baseline for the upcoming flight.

Pre-Protocol: Data protection and security is of vital importance and as such pilot data must be properly

encrypted to avoid fraud and malicious activity. Once the database is imported the data must bedecrypted [4]. Once data is decrypted the database must be prepped for writing and modification this is a

can be done through basic SQL shell scripting. As a slight side note, uniform formatting of data necessary as

data may be recorded over several years. Any changes to data format through system updates may be

taken into account at this stage.

Baseline: With the data imported and decrypted we are now in a position to run the algorithm to

determine the pilot and co-pilots flight baseline. Both previous pilot data and the data created from the

pre-flight survey must be considered in the creation of the pre-flight baseline to which everything will be

compared.

Pre-Flight

Post-Flight

Mid-Flight

P a g e 6 | 18


8/19

Mid-Flight

Data Recording: Sensory data is recorded in real time and maintained in raw format files. A sampling

frequency of 2 Hz is used therefore 12 data points are created per second which likely result in

approximately 50 Bytes per second. This data will then be processed by the central analyser. Each data

point will likely be restricted to a single scalar value which will represent an offset from the established

flight baseline. For example, for heart rate a stream of data points: 0, 1, 2, 1, 3, 2, 0, -1 would represent the

beats per minute from the baseline resting heart rate of the pilot or co-pilot.

The openCV program will run constantly to analysis visual data. Due to the nature of visual data the

majority of the data analysed through the openCV program will be temporary. Ultimately only large

variations in visual data will be recorded and received by the analyser. Again, data may be represented as

the integers showing deviation from the baseline by some pre-defined scale.

Analysis: The central analysis Python program will likely run a straightforward algorithm defined for the

RPMHI but machine learning techniques may also be implemented to help create pseudo-personalised

algorithms for each pilot or co-pilot being monitored. Stress data will likely only be analysed every 10

minutes or so to minimise wasteful computation and lessen the effect of non-significant anomalous data.

Considering that flight times can vary from 30 minute up, an analysis frequency of 6 analyses per hour

should be acceptable. It is important to re-state that the proposed system aims at picking up chronic and

long term mental health issues in pilots, issues relating to short term stress induced from intense or

dangerous flying situations fall outside the scope of this proposal.

Analysis is ultimately intended to result in the calculation of the RPMHI of the pilot and co-pilot. Extreme

analysis results may then be reported back to flight staff or ground crew through satellite based

communications. Flight protocol will determine the action taken in response to any real-time data. This

proposal does not suggest any direct intervening with any flight systems by the RPMHI system nor is any

real-time feedback to either the pilot or co-pilot suggested as this may worsen a stressful situation.

Analysis will result in one final data set, which will be the averages and variances of each metric since the

last analysis was carried out. These data points will pay the primary role in determining the pilots RPMHI

result.

Cleaning: At this stage superfluous data is discarded and entries to the SQL database are made. The

cleaning stage is important to minimise unnecessary data and increase the ease of packaging, compression

and encryption during post-flight operations. 10 minute analysis data is maintained along with some final

metadata representing the frequency of large metric variation which is intended to aid in data

interpretation.

Post-Flight

Export: The baseline calculated during pre-flight operation will be packaged alongside all recorded data.

Packaging of the SQL database will be directly followed by compression and encryption operations. Finally,

the transmission of the encrypted data to the central pilot information data is performed. At this point all

P a g e 7 | 18


9/19

sensory technology is inactive and the sensory system may undergo some recalibration during the post-

protocol.

Post-Protocol: In the post-protocol a confirmation of receipt from the central pilot information database is

requested. Once this is received all local data is wiped and the system is once again readied to run pre-

flight operations to import and prep incoming pilot data.

Central Pilot Information Database

The central pilot information database is in principle a single distributed pilot data backing store.

Compressed and encrypted pilot data is received by the database and then stored for later use. In order to

decrease unnecessary complication the received data is decrypted and uncompressed and then combined

with any previous pilot data. Once a request is made to the database for data, all combined pilot data iscompressed and encrypted to be sent by SFTP [6] or similar secure transfer protocols to the intended on

plane system.

The database infrastructure is proposed as a frontal request and incoming worker backed by numerous

worker nodes. The cyber infrastructure used is not of any great importance, but large amounts of storage

will be required. Dedicated is most likely more suitable than cloud or grid based resources in this instance

due to the homogeny of operation and the singular large resource requirement.

VI. Cockpit Sensor Array

The cockpit sensor array consists of 5 sensors to detect various biometrics for the pilot. These sensors are:

Pressure sensor in the back of the seat to detect whether the pilot is in the seat and to measure

breathing

Accelerometer on the joystick to measure hand tremor

A Ventilation Chamber combined with a Thermocouple and Humidity sensor adapter mounted on

the joystick to measure perspiration

An ECG mounted to the headband of the pilots headset to monitor heart rate

A strain gauge built into the seatbelt to measure respiration

In addition to this there will be a camera mounted to the roof of the cockpit which can monitor the pilots

motor functions and a second camera also mounted to the roof of the cockpit that aims to detect pupil

dilation, which can occur if the pilot is under stress.

P a g e 8 | 18


10/19

Pressure sensor in the rear of the seat

Figure6:Proposedcockpitpressuresensorconfiguration[12]

A weight sensor in the rear of the pilot seat can be used to detect whether the pilot is currently sitting in

the chair. If the pilot is not present then the system will not record any data. Data recording will begin as

soon as the plane takes off with a registered pilot in the seat. The sensor itself consists of a series of strain

gauge resistors which generate an electrical signal in response to the weight on the seat from the

occupant. In addition to this, this sensor can also be used in conjunction with the seatbelt mounted

breathing sensor in order to more accurately measure the respiration biometric.

P a g e 9 | 18


11/19

Figure7:Graphshowingtheoutputsignalfromtheseatmountedsensor

The graph above shows a sample output from the weight sensor. At approximately 40 minutes the pilot sits

down and the flight begins. Approximately 200 minutes later, the pilot leaves the seat and the flight ends.

Note that when the pilot is sitting in the seat there is an extra signal from the pilots breathing.

0

1

2

3

4

5

6

7

8

0 50 100 150 200 250 300 350

VoltagefromS

ensor(Volts)

Time (minutes)

Sensor Voltage Vs Time (minutes)

P a g e 10 | 18


12/19

Seatbelt Mounted Breathing sensor

Figure8:Proposedseatbeltmodification

The Seatbelt mounted breathing sensor uses the rise and fall of the pilots chest to calculate a breathing

rate. The breathing rate of the pilot is a key indicator to the pilots wellbeing. If the pilot is breathing

irregularly he may be experiencing health problems and these can be addressed on the landing of the

plane.

The seatbelt breathing sensor consists of a strain gauge that runs along the length of the seatbelt. When

the seatbelt is plugged in, the circuit is complete and the gauge is activated. When the pilot breathes the

rise and fall of his/her chest will cause the strain gauge to output a voltage. This signal is saved as raw data

and processed by the on board control system every 10 minutes with a sample speed of 2Hz.

P a g e 11 | 18


13/19

Figure9:SchematicfortheSeatbeltSensor

Joystick Sensor Systems

Figure10:Modificationtopilotsjoystick

There are two different sensors present on the Joystick of the plane; the Perspiration sensor and the

Accelerometer to detect hand tremor. The Joystick is the main hub for sensory data in the cockpit as hand

tremor and perspiration are two of the most prevalent symptoms of Stress.

P a g e 12 | 18


14/19

The Accelerometer

The Accelerometer will detect hand tremor. In order to rule out turbulence and the movement of the

plane two accelerometers will be used on the joystick and the difference between the two will provide the

control system with a value for pilot hand tremor. The accelerometer system has been used to measure

hand tremor with success in alternative studies [13]. The data from the Accelerometer will be sent to the

Control System which will aggregate the data at a sample speed of 2 Hz (2 samples per second). If the pilot

shows signs of serious hand tremor the Airline and Pilot will be given advice in order to ensure the pilot

stays in good mental health.

Figure11:Sampleaccelerometerreadings

Perspiration Sensor

In the report studied by the group, a Perspiration sensor was constructed using a Relative Humidity sensor

adapter (Specifically the THT-B121 resistive type transmitter), a Flow Meter and a calibrated type K

thermocouple to measure temperature. The perspiration sensor recorded the number of grams of

perspiration from the test subject. This could be implemented to the joystick of the aircraft and could

measure the amount of sweat on the hands of the pilot. The figure below shows the block diagram for the

P a g e 13 | 18


15/19

Perspiration measurement system. This sensor system in particular is crucial in determining the mental

wellbeing of the pilot as sweaty palms are a sign of agitation, anxiety, stress and paranoia.

Figure12:BlockDiagramforPerspirationSensor

P a g e 14 | 18


16/19

ECG Headset

Figure13:BasicECGconfiguration

Built into the headset is an ECG Sensor which is pressed against the pilots temple. The outboard

equipment for the ECG is stored within the control panel of the cockpit to allow for a light and comfortable

headset design. The ECG will record the heart rate of the pilot. The .edf files from the ECG will be sent to

the control system of the plane and if the heart rate is more rapid than the baseline profile then it can be

shown that the pilot is feeling stressed or under pressure and measures can be taken to ensure the pilots

health returns to normal.

Camera

A camera will be present in the cockpit in order to detect pilot movement. One of the key symptoms for

anxiety and stress is fidgeting and rapid twitching movements. The system will use edge detection

algorithms to determine any irregular pilot movement during a flight.

P a g e 15 | 18


17/19

Figure14:Samplecockpitviewusingedgedetection

Fingerprint Scanner

Figure15:Joystickwithaddedfingerprintscanner

A fingerprint scanner will be mounted to the top of the joystick will allow the pilot to login to the system,

thus activating it. The system will not record any data until a pilot with a profile has logged in. In order to

login the pilot simply presses a finger onto the scanner.

VII. Summary and Conclusion

This work outlines a proposal for a data based solution to the rising problem of monitoring commercial

pilot mental health. Described are three main features of a system to aid in the measurement and real-

time analysis of physical factors giving insight into the general mental health and wellness of active

commercial pilots and co-pilots. As mentioned these features are the baseline profile, cockpit sensor array

and the data processing.

P a g e 16 | 18


18/19

Research into this area shows the need for an elegant and efficient solution to monitoring pilot mental

health. At present pilot mental health checks are at the discretion of the airline and the FAA and current

active protocols are insufficient to ensure the long-term mental wellness of active pilots. As pilots are in a

position of great responsibility it is our hope that this work will, to some extent, shed light on the ease and

practicality with which a system may be implemented to help monitor pilot mental health.

The cost of the various system components will likely not exceed $5000. However, the central pilot

information database may require addition of extra cyber infrastructure on top of existing facilities. This

could be improve through the use of opportunistic and/or cloud resources. Added cyber infrastructure will

drive pricing up yet, in the context of the multi-billion airline industry the cost of retrofitting current

aircraft with our proposed system is effectively trivial. Large development cost is a one-time consideration

and will likely focus on the computer vision and data analysis side as the sensory and data technologies are

typical.

In conclusion it is our belief that the work outlined in this paper provides a viable solution to the

monitoring of pilot mental health, and with further development the implementation of such a system is

highly viable. Future work could include the use of similar sensors and techniques in a controlled

environment to provide proof of concept. With further developments this solution will not only ensure the

long term physical and mental health of commercial pilots but also provide greater peace of mind to

customers and companies of the aviation industry.

VIII. References[1] SQL homepage http://www.sql.org/ August 2014

[2] OpenCV hompage http://www.opencv.org/ August 2014

[3] Python homepage https://www.python.org/ August 2014

[4] Olumofin, Femi and Goldberg, Ian. Privacy-preserving Queries over Relational Databases. Proceedings

of the 10th International Conference on Privacy Enhancing Technologies 75-92 2010

[5] Database compression techniques http://technet.microsoft.com/en-

us/library/dd894051(v=SQL.100).aspx August 2014

[6] SFTP overview http://compnetworking.about.com/od/ftpfiletransfer/g/sftp-definition.htm August 2014

[7] Rusche, D. (2012). The JetBlue pilot's breakdown and the high stress of 'safety sensitive

positions'.Available: http://www.theguardian.com/world/2012/mar/30/jetblue-pilot-breakdown-mental-

health. Last accessed 7th Aug 2014.

[8].[J].199910(4):234.

[9] Wood, R. (2012). Resting Heart Rate Table.Available: http://www.topendsports.com/testing/heart-

rate-resting-chart.htm. Last accessed 8th Aug 2014.

P a g e 17 | 18


19/19

[10] Yingjui Tsai, Chiachung Chen. (2010). Development and testing of a perspiration measuring

system. Medical Engineering & Physics. 32 (356-362), 359.

[11] William Dayton, Patricia Collins, Michael Smith, and Ted Selker (2009).Measuring Hand Tremor with a

Mobile Device. Carnegie Mellon, Silicon Valley: Mobility Research Centre.

[12] Rainey, Robert R. (Elkhart, IN, US) . (2005). Vehicle seat weight sensor.Available:

http://www.freepatentsonline.com/6969809.html. Last accessed 8th Aug 2014.

[13] William Dayton, Patricia Collins, Michael Smith, and Ted Selker (2009).Measuring Hand Tremor with a

Mobile Device. Carnegie Mellon, Silicon Valley: Mobility Research Centre.

P a g e 18 | 18

Date post:	02-Jun-2018
Category:	Documents
Upload:	sbrennan970
View:	218 times
Download:	0 times

Elysian Aerotech Final Thesis: ICDIC competition 2014

Documents