SmartV: Intelligent Vigilance Monitoring based onSensor Fusion and Driving Dynamics

N.R.B Wijayagunawardhane., S.D Jinasena., C.B Sandaruwan., W.A.N.S Dharmapriya., Rohan Samarasinghe.

Department of Information TechnologySri Lanka Institute of Information Technology

New Kandy Road, Malabe, Sri Lanka.{nalindalxr, suchithjinasena, cbsandaruwan, nipunisd}@gmail.com, {rohan.s}@sliit.lk

Abstract—Road traffic accidents (RTAs) pose a public healthand development challenge and greatly affect the human capitaldevelopment of every nation. Main causes for a significant per-centage of RTAs are recklessness, fatigue or stress, inexperienceddriving and driving under the influence of alcohol. Reputedautomobile companies and authorities have established standardsand/or proactive/reactive solutions over the years to overcomethis problem but the rates continue to grow each year. Softwaresolutions that focus on preventing RTAs are difficult to be foundin economy class automobiles that are used by the majority ofconsumers. The limited number of successful applications areeither bound to a certain brand/model and/or focus on a specifictask. SmartV is smart phone based vigilance monitoring systemthat focuses on the majority of factors contributing to RTAs.These include DUI(driving under the influence of alcohol), healthissues and vehicle defects. The mobile application communicateswith a heart rate monitor and a Bluetooth OBDII Adapter.These will continuously monitor the heart rate of the driverand the status of the vehicle respectively. During the courseof the drive, the system monitors the behavior of the vehiclefor existence of dangerous driving patterns that are definedin Visual Detection of DWI Motorists[8] a study conducted byU.S.A. N.H.T.S.A. The final product is a versatile, non-intrusive,flexible and most importantly an affordable mobile applicationthat captures the vehicle’s behavior, the health level of the driverand the status of the vehicle. Real environment operation ofSmartV has yielded significant results proving itself to be asuccessful and an affordable Vigilance Monitoring System forthe future.

I. INTRODUCTION

It has become a well-known problem that the numberof road traffic accidents increases each year, resulting inthousands of people losing their lives, property and loved ones.Over the years they have claimed more lives than malariaand are expected by 2030 to be twice as many as thosedue to HIV/AIDS and four times those due to tuberculosis.Comparing the rate of infrastructure growth within the country,the number of vehicles increases at a faster rate signifyingthat the number of future accidents will be not less thantoday. Among many more major factors causing these tragicaccidents are the recklessness of the driver, inexperienceddriving, fatigue or stress, poor health condition and drivingunder influence of alcohol[1].

Reputed automobile manufacturers and authorities haveestablished standards and/or proactive/reactive solutions overyears to overcome this problem but the rates continue to grow

each year. Software solutions that focus on preventing RTAsare difficult to be found in economy class automobiles that areused by the majority of the auto mobile consumers.

So there is a huge potential interest among researchers tocome up with a cost effective solution that operates withoutthe intervention of the driver at a higher rate of accuracy. Theinbuilt car systems that promise safety have constraints uponoperation. The limited number of affordable safety solutionssuch as Anti sleep pilot[2], Toyota alcohol detection system[3]are either vehicle dependent or hard to be integrated to existingenvironments.

In order to come up with an accurate efficient and affordablesolution it was essential, identifying the conditions whereaccidents occur. While driving, the driver is needed to awakenand be attentive. The driver’s information processing errorof inattention is widely regarded to be the most frequentprincipal causal factor in traffic crashes, greatly surpassing lossof alertness. The target system should capture both of thesesituations in order to be accurate and efficient in preventingroad crashes. The loss of attention may be due to health issuesor consumption of alcohol or any other substance that affectsthe functionality of the brain. The loss of alertness may bedue to health issues and/or fatigue. In order to come up witha software solution it was required that the vehicle has somemethod of computing and analyzing the different behaviouraland static variables during the drive.

Today’s smartphone serves as the key computing and com-munication mobile device of choice, and also comes with arich set of embedded sensors, such as an accelerometer, digitalcompass, gyroscope, GPS, microphone, and camera. Collec-tively, these sensors are enabling new applications across awide variety of domains, such as healthcare, social networks,safety, environmental monitoring, and transportation, and giverise to a new area of research called mobile phone sensing.SmartV is a smartphone based driver vigilance monitoringsystem which captures all of the aspects related to monitoringthe eligibility of a driver controlling the vehicle. SmartVcontinuously assesses different components of the system,involving the driver, vehicle and vehicle’s behaviour on road.Productive utilization of the existing resources which accom-panies majority of the mobile devices is a major considerationin its implementation where the mobile device itself acts as

2013 IEEE 8th International Conference on Industrial and Information Systems, ICIIS 2013, Aug. 18-20, 2013, Sri Lanka

507978-1-4799-0910-0/13/$31.00 ©2013 IEEE

the main processing component and the sensor platform.Driver’s health is a crucial factor determining the eligibility

for driving .Continuous monitoring of heart rate is significantenough to identify patterns that classifies the health conditionof the subject. According to U.S.A. H.H.T.S.A. Study VisualDetection of DWI Motorists, aggressive driving patterns suchas swerving and aggressive turns possess identities that dif-ferentiates them from typical non-aggressive driving. Thesedriving patterns have a high probability in suggesting thatthe driver is influenced and contain their own identifiablecharacteristics in terms of acceleration and angular velocity.Additionally vehicle’s active parameters fetched through CANBus are utilized in order to increase accuracy of the overallfunctionality and provide assistance. SmartV packages allof the mentioned features and characteristics to constitutea complete, high performing, versatile and a non-intrusivemobile application that increases the safeness and improve thedriving experience. Increasing popularity of smartphones withtheir embedded sensing and high performing capability andcost effectiveness of SmartV will increase its usability andpopularity amongst its competitors.

II. RELATED WORK

Many research teams and research personnel over thedecades have conducted researches to detect driver drowsinessor driving under influence and they focus on monitoringand preventing driver fatigue. Zhu et al. [4] have discovereda system using two cameras on dashboard to monitor thephysical characteristics of drivers, such as eyelid, gaze andhead movement and facial expression, in order to predictfatigue levels. Lee et al. [5] have used two xed cameras tocapture the drivers sight line and the driving lane path forthe purpose of driving pattern and status recognition. Thesemethods all need multiple cameras to be installed in the vehicleand most probably will be intrusive to the normal drivingactivity, since most cameras are installed right in front ofthe driver. Most of these systems are reluctant to operate inexceptional situations and yield incorrect results whenever thecameras are unable to capture the eye movements intensiveenough to make correct decisions (i.e. when the driver iswearing dark colored spectacles). Besides visual recognitionDesai et al. [6] have installed a force sensor on the acceleratorpedal and have collected data based on the force applied tothe pedal to determine driver fatigue. These researches use theinteractions between human and vehicle to indicate mostlyinfluenced driving. Their systems need major alterations tothe vehicle and are tightly coupled with those add-ons. Othersystems based on head positioning, eye-gaze system, twopupil-based system and in-seat vibration system are hard tobe integrated to the vehicle without major reconstruction ofthe vehicles architecture and design. Leece et al. [7] haveintroduced a new architecture for driving information systemwith specic sensors and GPS receiver. Their work illustratesthat the acceleration reects the features of driving pattern. Eventhough fetching GPS data continuously is not demanding theirarchitecture and research suggests that acceleration reveals

signicant information about driving behavior. According to theUnited States NHTSA’s report [8], lane position maintenanceproblems are 50-75%, speed control problems 45-70%, judg-ment and vigilance problems 40% are the main visual causesfor accidents with the respective probabilities, which suggeststhat lane position maintenance problems are the signicant incausing road accidents among others.

Fig. 1. Problems in maintaining the lane position: (a) weaving, (b) drifting,(c) swerving, (d) turning with a wide radius [8]

If a driver is observed to be weaving, the probability ofinuenced driving is more than 50%. Some patterns such asswerving and sudden acceleration have probabilities closeto 70% of driving under inuence. Furthermore, the proba-bility of inuenced driving increases when a driver exhibitsmore than one of these patterns. These ndings suggest thatdriving pattern provide relatively strong evidence of drivingunder inuence (alcohol, health issues, fatigue, sleepiness andetc.).Transitively these two research articles suggest that thereis a strong correlation between the vehicle driving dynamicsand driving under inuence. But monitoring driving patternalone will not be signicant enough to come to a conclu-sion of possible inuence since a wrong decision may createunforgivable consequences at enterprise level. In addition tomonitoring the driving dynamics it will be advantageous toget real time health information from the driver and vehiclestatistics from the vehicle itself. The normal Heart Rate rangeof a person can be captured by monitoring the heart rate fora certain period of time. Heart Rate has established linksbetween wake/sleep stages and before/after consumption ofalcohol through previous research studies including the studiesof Stepanskiet al. In 1994[9].

Fig. 2. Change in Heart Rate from Baseline during a Performance Task[9]

Analyzing the normal range of the heart rate is thereforecan be an effective method for detecting drivers drowsinessand health issues. Large scale vehicle manufacturers are mov-ing forward with the concept of needing to monitor drivers

2013 IEEE 8th International Conference on Industrial and Information Systems, ICIIS 2013, Aug. 18-20, 2013, Sri Lanka

508

Fig. 3. Changes in Hear Rate due to the effect of alcohol[10]

health and are conducting major researches in implementingsuch systems for their automobiles. One such approach is toimplement a pulse sensing device on the steering wheel whichwill continuously monitor the heart rate of the driver[11]. Butfor the detection of inuenced driving the raw heart rate datawill not be enough.

III. PROPOSED SOLUTION-SMARTV

A. Overview

SmartV is designed with the purpose of minimizing andpreventing RTAs ,mainly caused by disciplinary issues ofdrivers. The system detects in-proper, unsafe driving withouthaving to implement large scale expensive external equipmentto support its decision making. It is capable of providingconsiderable amount of services just by utilizing the inbuiltsensors and technology inside the mobile device itself whichincludes driving pattern recognition. The optional hardwareplug-ins(i.e. Pulse Sensor and/or a OBDII Bluetooth Adapter)will enhance the accuracy and efficiency of decision making.The main source of alerting the driver and interaction betweenthe driver and the application is done through the mobiledevice itself.

B. Methodology

Prior to developing the system the research group spent aconsiderable amount of time to make sure that the applicationis worthwhile and improves the quality of human lives. Once aproper design and the most appropriate features were selectedthe developing team started working on the implementation.The core functionality of the application can be categorizedinto three broader function groups.

1) Dangerous Driving Detection: The functionality is suchthat whenever the vehicle exhibits dangerous driving patternsdefined in the study conducted by U.S.A. N.H.T.S.A (VisualDetection of DWI Motorists)[8] the system recognizes thesedriving patterns individually and alert the driver about potentialdanger. The driving patterns studied here have the probabilityof 0.5 to 0.75 in suggesting that the driver is influencedeither by a drug or alcohol. The problematic patterns usedin SmartV are related to Maintaining Proper Lane Position.These include but not limited to Weaving, Weaving acrosslane lines, Straddling a lane line, Swerving, Turning with awide radius and Drifting.

First the hardware requirements for data capturing wereidentified and aproper mobile device which is equipped withall the required processing power and the sensors was ac-quired. Next a simple data collection algorithm was designedwhere the sensors were configured to provide data at a rateof 40Hz. The initial data collected were analyzed in order tofind significant similarities among similar driving patterns anddifferences among different driving patterns. Meanwhile thethreshold values that define the end points of each drivingpatterns were identified.The metric that was used to detectthe end points of a target driving pattern is the simplemoving average of the square of the angular velocity which ispropositional to the angular kinetic energy.

Fig. 4. Second level of data collection application

The Fig. 4 shows lateral acceleration for 8 different occur-rences of the same driving cue (weaving right). The figureevidences that the driving patterns have close relationship interms of lateral acceleration. The variables used for dangerousdriving detection are (when the mobile device is oriented inportrait mode) the lateral acceleration (Accelerometer X) andthe Angular Velocity (Gyroscope Y). The data from the secondlevel of data gathering process were analyzed using a desktopdata matching program implemented in Java and was used toselect the data files with the most significant correlation. Thedata from these files were taken out to be the template drivingcue information that is used in driving pattern matching.

Fig. 5. Second level of data collection application

The Fig. 5 shows the component distribution of the corefunction group. The vehicle’s driving behavior is monitoredthroughout the course of the drive through the inbuilt sensors.When these sensor data trigger the occurrence of an aggressiveevent through end point detection, these data are fed to thedriving pattern identification sub component. If a positive

2013 IEEE 8th International Conference on Industrial and Information Systems, ICIIS 2013, Aug. 18-20, 2013, Sri Lanka

509

match occurs, the driver is alerted until he/she stops thevehicle.

2) On-Board Diagnosis and Assistance: Vehicle Engineparameters are transferred to the application through a Blue-tooth Adapter which is connected to the vehicle’s in-builtOn-Board Diagnostics interface[12]. This interface is used tofetch continuous data such as speed, fuel economy, fuel leveland Engine RPM(Revolutions per minute) from the vehicle inorder to be used in various occasions for supporting decisionmaking and for providing necessary assistance to the driver.

The application alerts and notifies the driver in responseto a combination of different parameters (i.e. vehicle speed,acceleration, fuel level, and fuel economy). The parametersextracted and their intended usages are described through theFig. 6

Fig. 6. On-Board Diagnosis Feature Extraction

As you can see the OBDII features are highly utilized ina variety of main and sub components including DangerousDriving Detection, Fuel Station Redirection in cases of lowfuel level and Incoming Call Blocking. Availability of someparameters highly depends on the vehicles manufacturer. ButSmartV is designed in a way to retrieve all the requiredparameters that are available in a significant percentage ofvehicles.

3) Heart Rate Monitoring: In order to track the healthlevel of the driver a Pulse Sensor is used which is clippedon to the earlobe of the driver. The Pulse Sensor continuouslyprovides an analog signal to the application which is then usedto calculate the heart rate of the driver. Then the raw signalis recorded and processed in order to retrieve the heart rate ofthe driver. During the initial setup of the system the driver isasked to wear the pulse sensor for an approximate time of fiveminutes in order to configure and store his/her normal heartrate range. Once the normal heart rate range is configured thedriver’s heart rate is continuously monitored and the range iscalculated. Once in every five minutes the current range ischecked against the stored range to detect deviations from thenormal heart rate range.

These changes have a correlation with the driver’s healthstatus as found in previous researches[9]. So the comparisonresults in the activation of Dangerous Driving Detection.

SmartV uses the combination of these core function groupsand the sub components of core groups to come up with adecision that reflects the actual status of the driver’s conditionand eligibility to continue driving. The core functionality (i.e.Dangerous Driving Detection) works even in the absence ofthe external hardware components(i.e. Pulse Sensor and/orBluetooth Adapter) which means that SmartV works witholder vehicles that are not equipped with OBDII.

IV. RESULTS

Once the prototype was successfully implemented, individ-ual components were thoroughly tested in a real environment.The Fig. 7 shows the main interface of the implementedsystem.

Fig. 7. On-Board Diagnosis Feature Extraction

For testing, few vehicles were used including, a 1999Nissan B 14 and a 2006 Maruti Alto, 2011 Kia Sportageand a 2002 Audi A6. With proper tune up to the data andthe algorithms used, the tests showed significant level ofaccuracy and exhibited fast recognition of dangerous drivingpatterns. The recognition alert appeared in less than twoseconds after the event for all tests. Live testing video ofDangerous Driving Detection can be viewed through the linkhttp://youtu.be/Z1czvDWQFps. The Fig. 8 shows the alertsused to notify the driver of potential danger.

The alerts are shown until the driver stops the vehicle(in response to the exhibition of critical driving patterns).The On-Board Diagnostics functionality was tested thoroughlyusing a variety of vehicles including 2002 Audi A6 and2004 Kia Sorento. Some features did not work as intendedas a result of the vehicle’s proprietary OBD standards. Butthe majority of the tested vehicles were using the genericOBDII standards that are used in the software application.For Heart Rate Monitoring the calculated heart rate wascompared with a commercial heart rate monitor (model-OMRON HEM-609). Then it was tested on different An-droid operating system versions such as Ginger Bread, IceCream Sandwich and Jelly Beans. The test results were

2013 IEEE 8th International Conference on Industrial and Information Systems, ICIIS 2013, Aug. 18-20, 2013, Sri Lanka

510

Fig. 8. On-Board Diagnosis Feature Extraction

significant that the calculated values are accurate as ex-pected. Live testing video can be viewed through the linkhttp://www.youtube.com/watch?v=vtG2sV3dxW0&feature=share&list=FLIlvH4pbgERsH9hPq5MtcLQ. The testing videofor the Heart Rate monitoring during the drive and theOBD II Data Retrieval can be viewed through the linkhttps://www.youtube.com/watch?v=lxJfgytvrjc&feature=autoshare. After testing all the main components individually theintegrated system was tested in different devices and differentoperating system versions. The system worked with the mini-mum level of false positivity and passed all the test cases withan accuracy of more than 95%.

V. DISCUSSION

SmartV is developed as a mobile application with theintention to ensure a better and safe driving experience.Themost appropriate features were selected and implemented asso as to fulfill the requirements of the majority of automo-bile consumers. It is designed to do most of its operationsautomatically and to allow the user to get familiar with itsfeatures without any prior knowledge of the technology or theunderlying architecture. Android has become the market dom-inant since it’s introduction and has taken over the majorityof the market share owned by market giants like Apple andWindows[13]. SmartV can be considered as the ideal solutioncompared to other mobile based commercial products such asAnti Sleep Pilot due to the popularity of Android,it’s accuracyand cost effectiveness.

VI. CONCLUSION

The problem in context was that the number of accidentsincreases each year despite of the infrastructure growth andlaw enforcement causing hundreds of people losing their lives,loved ones and property. Software solutions that ensure thesafety of motorists are hard to be found since the majority ofautomobile consumers use economy class vehicles. Whetherit is a luxury sedan or an economy class family vehiclehuman lives are equally valuable to the society. The mainintention of implementing the software solution SmartV was

to ensure safety of people when they are traveling irre-spective to personal interests, ethnic bounds and financialstability. It is influenced by a previous research conducted bythe U.S.A. N.H.T.S.A.Visual Detection of DWI Motorists[8]which strongly provided evidence stating that there are driv-ing patterns that have probabilities ranging from 0.5-0.75in suggesting that the driver is intoxicated. These drivingpatterns’ angular velocity, lateral acceleration can be trackedusing mobile phone’s in-built sensors. A heart rate monitor isclipped on to the earlobe and utilized to continuously monitorthe heart rate of the driver. Additionally a Bluetooth OBDIIadapter[14] is plugged into the vehicle to retrieve data such asspeed, rpm, fuel level and fuel economy. These data add morefeatures and improve accuracy and efficiency of the overallfunctionality of the system. The final product is a versatile,non-intrusive, flexible and most importantly an affordablemobile application that not only captures the vehicle’s behavioron road but monitors the health level of the driver througha pulse sensor and tracks the status of the vehicle throughthe vehicle’s OBDII CAN Bus as well. SmartV constitutesa complete package as its target market segment includesnot only drivers and riders but includes company owners,vehicle lenders and basically everyone who is concerned aboutsafety while driving. The real environment tests on the finalproduct showed significant results. The application is capableof extending itself to deal with more driving patterns in thefuture.

A. Future Work

The system is currently capable of identifying a limitednumber of driving patterns defined in the study conductedby USA NHTSA (Visual Detection of DWI Drivers) due toresource and time limitations. Future contributions will betargeted at extending the pattern recognition to identify moredangerous driving patterns which has a higher probability ofcausing accidents.

REFERENCES

[1] http://www.police.lk/index.php/traffic-police/110[2] http://www.antisleeppilot.com/[3] http://www.nbcnews.com/id/16449687/#.UoTIeHDI26M[4] Z. Zhu and Q. Ji, Real Time and Non-intrusive Driver Fatigue Monitor-

ing,in The 7th International IEEE Conference on Intelligent Transporta-tion Systems, pp. 657-662, Oct. 2004.

[5] J. Lee, J. Li, L. Liu and C. Chen, A Novel Driving Pattern Recognitionand Status Monitoring System, in First pacific rim symposium, PSIVT2006, pp. 504-512, December 2006.

[6] A. V. Desai and M. A. Haque, Vigilance Monitoring for OperatorSafety:A Simulation Study on Highway Driving, in Journal of SafetyResearch,Vol. 37, No. 2, pp. 139-147, 2006.

[7] V. D. Lecce and M. Calabrese, Experimental System to Support Real-Time Driving Pattern Recognition, in Advanced Intelligent ComputingTheories and Applications With Aspects of Artificial Intelligence Annalsof Emergency Medicine, pp. 1192-1199, 2008

[8] U.S. NHTSA, The Visual Detection of DWI Motorists,http://www.nhtsa.gov/staticfiles/nti/pdf/808677.pdf

[9] Stepanski E, Glinn M, Zorick FJ, Roehrs T, Roth T. Heart rate changesin chronic insomnia. Stress Medicine1994, 10:261-266. John Wiley andSons, Limited, Reproduced with permission.

[10] http://www.onlinepcd.com/article/S0033-0620(09)00033-4/abstract

2013 IEEE 8th International Conference on Industrial and Information Systems, ICIIS 2013, Aug. 18-20, 2013, Sri Lanka

511

[11] Ashutosh Gupta and Betsy Thomas, ”Wireless Sensor Embedded Steer-ing Wheel For Real Time Monitoring Of Driver Fatigue Detection”,04/2012; DOI:10.3850/978-981-07-1403-1 In proceeding of: Proc. ofthe Intl. Conf. on Advances in Computer Science and ElectronicsEngineering

[12] http://www.obdii.com/[13] http://techcrunch.com/2013/08/07/android-nears-80-market-share-in-

global-smartphone-shipments-as-ios-and-blackberry-share slides-per-idc/

[14] http://www.edn.com/design/automotive/4420104/Teardown–OBD-II-Bluetooth-adapter

2013 IEEE 8th International Conference on Industrial and Information Systems, ICIIS 2013, Aug. 18-20, 2013, Sri Lanka

512