Low-Cost Embedded System for Driver Drowsiness Detection

8/9/2019 Low-Cost Embedded System for Driver Drowsiness Detection

1/17

A Low-Cost Embedded System for Driver

Drowsiness Detection

Humberto Aboud

December 2014

Abstract

This report presents a low-cost approach for a driver drownsiness de-tection embedded system. The project relies on inexpensive open sourcehardware and software. The system tracks the users face and eyes todetect patterns and infere drowsiness. The hardware architecture is com-posed by a Single-Board Computer (SBC) and a USB camera, and theembedded software relies on the open source library OpenCV. This projectuses heavy image processing and depends on high optimization both onthe hardware and the software sides. The SBC chosen for this project waspowerfull enough to produce a working system but a more powerfull onewould provide better results.

1 Introduction

Brazils national fleet is composed by almost 50 million cars according to theNational Traffic Deparment[1]. This vehicle provides an efficient and convenienttransportation modality, however, traffic accidents are the second major cause ofdeaths in Brazil, as reported by the National Observatory in Traffic Safety[2].Only between 1980 and 2011 980,838 lifes were lost on the national roads[3].Behind these numbers, statistics from the Center of Multidisciplinary Studiesin Somnolence and Traffic Accidents (CEMSA)[4] of the Federal University ofSao Paulo shows that between 26% and 32% of the countrys traffic accidentsare caused by drivers that fall asleep while driving.

To address this problem, the automobile industry develops a wide rangeof safety solutions. Ford, for example, produced its own system, the Driver

Alert[6]. This system comprises a small forward-facing camera connected to anon-board computer. The camera is mounted on the back of the rear-view mirrorand is trained to identify lane markings on both sides of the vehicle. When thevehicle is on the move, the computer looks at the road ahead and predicts wherethe car should be positioned relative to the lane markings. It then measureswhere the vehicle actually is and if the difference is significant, the system issuesa warning.

Unfortunately, the current technology able to detect and prevent this kind ofaccident is still very expansive. Fords system, for example, can cost from $2,000to $12,000 for a single vehicle[7], making it very inaccessible to the averagehousehold.

1


2/17

This project uses open source hardware and software to develop an inexpan-

sive drowsiness detection system that can be easily reproduced and installed inany car with total cost less then US$ 80.

2 Objective

This projects objective is to compile and express the knowledge obtained by thestudent along his Computer Engineering undergraduate degree in University ofCampinas.

The undertake chosen was to produce a low cost driver drownsiness detectionembedded system that can be easily reproduced and installed on any car. Toachieve that, the project had four goals:

Develop an image recognition software to infere drowsiness state Set an embedded systems hardware environment

Embed the developed software on the hardware

Optimize the systems performance

Measure the systems performance and accuracy

3 Materials and Methods

3.1 Approach

There are a couple known approaches to perform driver drowsiness detection,three common are the eye tracking, the heartbeat monitoring and the drivingpattern monitoring, this paper will tackle the eye tracking approach, due to itspotential to provide consistency with low-cost equipment.

Research from the National Association of Sleep[5] relates the drowsinesswhile driving with increase on the drivers eye blinking frequency. This relationcan be explored to develop a system that tracks the drivers eyes image andanalyzes it to determine its degree of sleepiness.

To do that, a computer vision software was developed. The software readsvideo streaming input from a camera pointing to a persons face and track herface and eyes to determine her sleepiness condition. At first, the software wasdeveloped and tested in a desktop computer environment so it could be free

of performance constraints. After the software was proved to work correctly,the hardware part of the embedded system was setup. With both software andhardware tested and working correctly separated, the software was ported tothe hardware creating the embedded system. The embedded system at firstpresented a weak performance capacity leading to poor results. To overcomethat, perfomance improvements techiniques on both software and hardware sideswere used. The system operation was then tested and the results compiled.

2


3/17

3.2 Face and Eye Recognition Software

The software developed for this system was writen in Python[8] and is shown inApendix 1 at the end of this report. Its objective is to detect the users eyesopen/closed state and use this information to infere drowsiness condition.

The software uses OpenCV[11] to perform face and eye recognition and de-fines a threshold to filter the obtained information. The Cascade Classificationis used to train the software to recognize a certain object using classifiers,files with samples. The following quoted text from OpenCVs website explainits action.

First, a classifier (namely a cascade of boosted classifiers workingwith haar-like features) is trained with a few hundred sample viewsof a particular object (i.e., a face or a car), called positive examples,

that are scaled to the same size (say, 20x20), and negative examples- arbitrary images of the same size.

After a classifier is trained, it can be applied to a region of interest(of the same size as used during the training) in an input image.The classifier outputs a 1 if the region is likely to show the object(i.e., face/car), and 0 otherwise. To search for the object in thewhole image one can move the search window across the image andcheck every location using the classifier. The classifier is designed sothat it can be easily resized in order to be able to find the objectsof interest at different sizes, which is more efficient than resizing theimage itself. So, to find an ob ject of an unknown size in the imagethe scan procedure should be done several times at different scales.

The developed software uses two classification files openly distributed byOpenCVs developers community:

Haarcascade frontalface default.xml for face recognition

Haarcascade eye tree eyeglasses.xml for eye recognition

The software operation works as an infinite loop. First, it captures a framefrom the camera and analyzes it looking for the users face. If it doesnt recognizea face it captures another framework and analyzes it until it finds. Then thesoftware analyzes the face area of the frame looking for the users eyes. If itrecognizes two (or more) eyes on the face it increments a threshold variable upto a maximum of ten, otherwise it decrements it down to a minimum of zero.

Then, if the threshold variable is above five, the software outputs that the usereyes are closed infering drowsiness, otherwise it outputs that that they are openand the user is awaken.

A flowchart ilustrating the algorithm is presented below:

3


4/17

Figure 1: Algorithm for drowsiness condition detection

To analyze the frame and perform face and eye recognition the software

uses OpenCVs detectMultiScale() function. Given an image, this function usesthe classification file and two parameters to detect a certain object in it. Thefirst parameter is a scale factor related to the object expected size. The secondparameter is a minimum number of neighbors, candidates for the expectedobject, to characterize a recognition. The following images explain this conceptin more details.

For the first image, the minimum number of neighboors chosen was 1, andthe software recognized many faces in the picture. For the second image,the minimum number of neighboors was 3, making the software consider onlyobjects with three or more neighbors leading to less faces recognized and amore precise result. For the built software, the best value for the minNeighbors

4


5/17

parameter was 3, obtained through trial and error.

Figure 2: A detectMultiScale() call with minNeighbors parameter 1

5


6/17

Figure 3: A detectMultiScale() call with minNeighbors parameter 3

3.3 Hardware setup

In order to make this system a total embeddable device that can be easilyimplanted on any car the system uses a Single-Board Computer (SBC) as itscore processing unit. Single-Board Computers are small, light and low-powerconsumption computers that have powerfull computational capability. Theycan run operational systems and control and communicate with other hardwaredevices.

There are several SBCs available on the market with different processingpower, size, power consumption and prices. Some of the most common SBCson the market are displayed on table 1.

The Raspberry Pi is a popular SBC with a good cost-bennefit, however its

processing power leaves to be desired. The PandaBoardhas a very good powerprocessing, but its hardware is not open source making it very expansive andthe systems overall price impractical. Freescales RIoTboardhas a reasonableprocessing power and price, but has no support for vector processing, as theimage processing of this system requires. The BeagleBone Black, from TexasInstruments, is a low-power open source hardware SBC that costs a modestprice, has a reasonable processing power and vector processing support, fitingthe needs for this project.

6


7/17

SBC CPU Average Price

Architecture Cores FrequencyRaspberry Pi ARM11 1 700 MHz US$ 35.00BeagleBone Black ARM Cortex-A8 1 1 GHz US$ 58.00PandaBoard ES ARM Cortex-A9 2 1.2 GHz US$ 180.00RIoTboard Freescale i.MX6 1 1 GHz US$ 79.00

Table 1: Comparison of Single-Board Computers1[10]

TheBeagleBone Blackhas a strong support for vector processing, its greatestadvantage is the Neon SIMD2 integrated processor block[9]. The SIMD providesparallel operations, meaning that during the execution of one instruction the

same operation can occur on up to 16 data sets in parallel at the same time.This is a great advantage for the necessary image processing required by thecomputer vision software. The SBC BeagleBone Blackwas then chosen as thecore processing unit for the embedded system.

Figure 4: SBC BeagleBone Black

To capture image in order to track the drivers eyes a simple USB camerawas used, almost any available USB camera with VGA quality or better wouldwork. For this project, we chose Nipponics NIP-VC76Pwebcam, because itsa cheap device and it was available for this project. The average market pricefor this camera is US$20.00. This camera has a VGA quality image and USB2.0 communication.

Figure 5: USB camera used to capture streaming video

The hardware was then assembled. The camera was plugged into the SBCs

1Prices quoted from the distributors websites on the publish date of this paper2Single Instruction Multiple Data

7


8/17

USB input and the SBC was connected to the power supply being fed with 5

volts and 1 ampere DC current. The final system is shown below.

Figure 6: Final system assembled

3.4 SBC environment

The open source Linux distribution Debian 7 was installed on the SBC. Debianwas chosen because it has a strong support community that maintains and fixesmany issues, leading to a very reliable and stable operational system, a requiredproperty for this project system given the low fault-tolerance of the addressedproblem.

A serial communication through SSH protocol was then stablished betweenthe SBC and the desktop computer. The desktop computer had total access tothe SBCs super user, making it possible to install the necessary softwares, suchas drivers, compilers and interpreters.

The USB camera did not have a driver supporting the SBCs processor ARMCortex-A8 architecture, making the communication with the SBC a hard taskthat was solved setting up low level system configurations.

3.5 Optimization

Even though SBCs have a much more powerfull processing capability then mi-crocontrollers, they are normally still behind desktop computers, GPUs3 orFPGAs4. Therefore in order to execute the designed software, which requiresheavy processing, it was necessary to perform optimizations on the system. Theoptimization took place on two fronts: the software and the hardware.

3Graphics Processing Unit4Field-programmable Gate Array

8


9/17

On the software front, the frame size read as input from the camera was

shortened and the code used to interact with the SBC, such as monitoring theprocessed image, removed. The shortest frame size that still gives concise resultobtained by trail and error was 160x120. That lead to a faster performancedue to the smaller number of pixels that had to be analyzed by the softare.Removing the code used to interact with the software such as the code thatdrawed rectangles for the recognized faces and eyes areas also made the softwarefaster for executing less instructions. At the beggining, an existent hypothesiswas that converting the image to shades of gray would make the analysis fasterdue to less pixel color values, but it was confirmed false later.

On the hardware front the SBCs clock frequency was modified. Beagle-Bonesdefault clock frequency operation is 300 MHz, which at first was clearlybelow the project needs. To change that, the linux application cpufrequtils[12]was used. This tool provides an interface to control the CPU frequency. The

SBCs clock frequency was then modified to run at a 1 GHz clock frequency, itsmaximum safe clock speed according to Texas Instruments, the manufacturer.With that, the hardwares processing power was substantially increased.

3.6 Measurement

To measure the systems performance the recognition time was measured varyingthe inputs frame size. Furthermore, the softwares performance was comparedwhen running on the SBC and a desktop computer. The embedded systemsprocessor was the BeagleBones ARM Cortex-A8 and the desktop computersthe Intel R CoreTM i5.

The measuring procedure was to calculate the time required to process the

eye detection, regardless of the output, after the face was detected. The timewas measured twenty times for each case and the standard deviation was taken.To do that, the Python librarys function time.process time() was used. Thisfunction is usefull to measure the performance because it takes into accountboth system and user CPU time, as stated in the functions description fromPythons website:

time.process time()

Return the value (in fractional seconds) of the sum of the systemand user CPU time of the current process. It does not include timeelapsed during sleep. It is process-wide by definition. The referencepoint of the returned value is undefined, so that only the differencebetween the results of consecutive calls is valid.

3.7 Testing

To measure the systems functionality four mutually exclusive and completelyexhaustive states were defined:

True Open: When the users eyes are open and the system detects it open

True Closed: When the users eyes are closed and the system detects itclosed

False Open: When the users eyes are closed and the system detects itopen

9


10/17

False Closed: When the users eyes are open and the system detects it

closedThese four states cover all the systems possible states.

Figure 7: True Open state

Figure 8: True Closed state

10


11/17

Figure 9: False Open state

Figure 10: False Closed state

To measure the systems accuracy on determining the users drowsiness con-dition, it was defined an efficiency ratio being:

Efficiency = TrueOpen+TrueClosed

TrueOpen+TrueClosed+F alseOpen+F alseClosed

11


12/17

To classify each systems output into one of the four states the testing pro-

cedure was to use recorded videos instead of stream input and record the frametime associated with each output so it could be validated by a human observerlater. Therefore, a one minute video was recorded with a person alternating hereyes state between open and closed. Then, the software was executed and itsoutputs recorded. Each output was associated with a videos time frame, afterthe software finished its execution a human observer would check each outputand classify it as one of the four earlier defined states.

This measure procedure inserts a performance constrain to the software be-cause it adds execution steps that makes the software slower and changes itsaccuracy. However, it is still useful to compare the performance between theSBC and the desktop, since it introduces constraints for both platforms. Theimage below shows the softwares output associated with a time frame.

Figure 11: Softwares output associated with videos time frame

4 Results

4.1 Performance

The systems performance was measured varying the input video streams framesize both on the SBC and the desktop computer, with a total of 4 differentframe sizes: 160x120, 240x180, 320x240 and 640x480. The measured results arepresented on Table 2 and ilustrated on Chart 1 below.

12


13/17

Frame Size ARM Cortex-A8 Intel R CoreTM i5

160x120 0,18s 0, 04s 0,012s 0, 009s240x180 0,33s 0, 05s 0,016s 0, 001s320x240 0,83s 0, 04s 0,04s 0, 01s640x480 2,18s 0, 07s 0,11s 0, 02s

Table 2: Execution performance comparison between SBC and desktop com-puter

Figure 12: Performance analysis (in seconds)

4.2 Accuracy

The systems accuracy was measured as described on session 3.7. The proce-dure was performed with four different persons. For comparison reasons, themeasurement was also executed on the desktop computer, with a shorter periodof 20 seconds, due to the high number of outputs that the desktop computerwas able to emit. The results are presented in the tables below.

Test True Open True Closed False Open False Closed Total Efficiency

1 86 69 22 15 192 81%2 71 62 23 20 176 76%3 61 55 38 31 185 63%4 69 66 31 24 190 71%

Table 3: Accuracy test for the embedded system (60 seconds period)

13


14/17

Test True Open True Closed False Open False Closed Total Efficiency

1 102 92 9 11 214 91%2 101 72 14 8 195 89%3 103 74 11 15 203 87%4 92 95 21 13 221 85%

Table 4: Accuracy test for the detection software running on the desktop com-puter (20 seconds period)

5 Discussion

On the performance side, even though the instructions used to measure addeda performance constraint the output was still concise enough to provide mean-ingful data. The frame size increase affected the systems performance in aquadratic realation. That happens because the frame size is a two-dimensionpixels matrix and when the size vary the number of pixels vary quadratically,so does the necessary processing.

Another clear behavior that can be observed is the performance differencebetween the SBC (ARM Cortex-A8) and the desktop computer (Intel Core i5).On average, it can get around 20 times faster on the desktop. That happens dueto the difference on the architecture nature, the desktop for example, has a muchmore powerfull processor and hardware such as GPUs to perform graphic pro-cessing, the SBC on the other hand, has less processing capability and graphicprocessing support.

The embedded system would present a better perfomance if the hardwareused were a more powerfull or specific device, such as FPGA or GPU. This typeof hardware is normally more expansive then an SBC, but it is still possibleto find a simple reasonable priced GPU or FPGA that can perform better notmaking the systems cost unpractical.

A hardware approach that could potentially improve the systems perfor-mance considerably would be to overclock the SBC above the used 1GHz. Withthis frequency, the board was not even warm yet, giving signals that with theright cooling techniques it could reach a much higher clock frequency. Unfortu-natel, with the low budget for this project, it was not possible to risk losing theSBC by performing an overclock on it.

A software approach that could potentially improve the performance would

be to write the embedded code in a more low level language, such as assembly.This approach would make the instructions be executed faster, due to the effi-ciency difference between high and low level languages. This approach was nottaken due to time constraint reasons for the project development.

On the accuracy side, we can observe from the two tables the link betweenthe systems performance and its accuracy efficiency. This happens becausewhen the image processing gets too slow the system skip frames and use less in-formation to determine the output, leading to more mistakes. Therefore havinga more powerfull performance on the embedded system, on both hardware andsoftware sides, would provide more accurate results.

14


15/17

With the results obtained, we had a 72,75% accuracy efficiency. This is still

too small for this application, once it deals with peoples lives. Furthermore, thesystem was developed not taking into consideration conditions such as changesin the drivers head position, drivers with different height and the different lightconditions faced by a car.

A big potential for perfomance improvement was identified. Even thoughtthe system presents humble results on accuracy, it shows great potential, onceit was build with a budget less then US$80.00, in contrast with the currentmarkets systems price on the range of $2,000 to $12,000.

6 Conclusion

This project summarizes the knowledge obtained by the student during its Com-

puter Engineering undergraduate degree at University of Campinas. Knowledgeacquired from many fields of study in computer engineering were used. The fieldsthat mainly provided the necessary skills to produce this project were:

Computer Vision

Image Processing

Embedded Systems

Computer Architecture

Computer Network

Computer Vision knowledge was used to understand how to develop a soft-ware to detect drowsiness condition by analyszing the users face image. ImageProcessing knowledge was used to optimize and improve the detection software.Embedded System knowledge was used to build and setup the embedded sys-tems hardware, to develop the embedded software taking into considerationthe systems constraints, and to combine both into an integrated system. Com-puter Architecture knowledge was used to define the best fit hardware for theapplication and to set it up and optimze its performance. Computer Networkknowledge was used to understand how to develop network communication in aSBC using protocosl to set its environment and communication up.

The project demonstrated to be a product not yet ready to be used forreal life application. However, it presented promising results showing its strongpotential. Some conditions are still needed to be taken into account on its

operation and more testing techniques must be performed to comprove its realcapability of providing a working system, but the large space for improvementidentified showed the prospective of providing a working system.

This project synthesizes the students computer engineering undergraduatedegree for combining both software and hardware skills to propose a solutionfor a problem that affects society directly. Furthermore, the project proposes alow-cost technology solution, providing a wider household access to it.

15


16/17

References

[1] DENATRAN - Departamento Nacional de Transitohttp://www.denatran.gov.br

[2] ONSV - Observatorio Nacional de Seguranca Viariahttp://www.onsv.org.br

[3] CEBELA - Centro Brasileiro de Estudos Latinos Americanoshttp://www.cebela.org.br

[4] CEMSA - Centro de Estudo Multidisciplinar em Sonolencia e Acidenteshttp://www.cemsa.com.br

[5] FUNDASONO - Fundacao Nacional do Sono

http://www.fundasono.org.br[6] Fords Driver Alert

http://www.euroncap.com/rewards/ford driver alert.aspx

[7] Sleep Review: Drowsy Driving Technologieshttp://www.sleepreviewmag.com/2013/11/alert-alive

[8] Pythonhttps://www.python.org/

[9] Cortex-A8http://processors.wiki.ti.com/index.php/Cortex-A8

[10] SBC Comparisonhttp://en.wikipedia.org/wiki/Comparison of single-board computers

[11] OpenCVhttp://docs.opencv.org/modules/objdetect/doc/cascade classification.html

[12] CPU frequency scalinghttps://wiki.archlinux.org/index.php/CPU frequency scaling

16


17/17

A Drowsiness Dettection Software

import numpy as npi m po r t c v2

f a c e c a s c a d e = c v2 . C a s c a d e C l a s s i f i e r ( h a a r c a s c a d e f r o n t a l f a c e d e f a u l t . x ml )e y e c a s c a d e = c v2 . C a s c a d e C l a s s i f i e r ( h a a r c a s c a d e e y e . xml )

count =0cap = cv2 . VideoCapture (0 )ca p . s e t ( cv 2 . c v . CV CAP PROP FRAME WIDTH, 1 6 0 )ca p . s e t ( cv 2 . cv . CV CAP PROP FRAME HEIGHT, 1 2 0 )

whi le (True ):retVal , img =cap . read ()f a c e s = f a c e c a s c a d e . d e t e c t M u l t i S c a l e ( img , 1 . 3 , 3 )

f o r ( x , y , w, h ) i n f a c e s :r o i c o l o r = i mg [ y : y+h , x : x+w ]

e y e s = e y e c a s c a d e . d e t e c t M u l t i S c a l e ( r o i c o l o r , 1 . 1 , 7 )i f ( le n ( ey es ) >= 2 ) :

i f ( count < 1 0 ) :count += 1

e l s e :i f ( count > 0 ) :

count = 1

i f ( count>5):p r i n t op en

e l s e :p r i nt c l o s ed

17

Date post:	01-Jun-2018
Category:	Documents
Upload:	humberto-aboud
View:	237 times
Download:	0 times

Low-Cost Embedded System for Driver Drowsiness Detection

Documents