AndroidSensorProgramming- ArindamGhosh

SensingandSensors

• acapabilitythatcancapturemeasurementsaboutthedeviceanditsexternalenvironment

• Candetectsandrespondstosometypeofinputfromthephysicalenvironment.

• Thespecificinputcouldbelight,heat,motion,moisture,pressure,etc.

• Convertthemeasurementintoasignalthatcanberead.

ExampleSensorData

1|MPU6500AccelerationSensor|[-1.6741456,9.370906,2.6886885]|1441670212915

1|MPU6500GyroscopeSensor|[-0.02263687,-0.016777916,-0.008788432]|1441670213508

1|AK09911CMagneticfieldSensor|[16.86,-64.26,-62.7]|1441670213400

1|GPS|{"mProvider":"fused","mResults":[0.0,0.0],"mAccuracy":29.0,"mAltitude":83.0,” ,"mLatitude":40.748431”,mLongitude":73.985741 …}|1441573552851

1|WiFi|{"BSSID":"00:21:6c:87:02:d1","SSID":"eduroam","capabilities":"[WPA2-EAP-CCMP]","frequency": 2462,"level":-82}|1392465248466

AndroidOSFragmentation

http://opensignal.com/reports/2015/08/android-fragmentation/

http://opensignal.com/reports/2015/08/android-fragmentation/

http://opensignal.com/reports/2015/08/android-fragmentation/

SensorAvailability

• Variesfromdevicetodevice

• MayvarybetweenAndroidversions

NewerSensorsOverTime

Sensors

• PositionsensorsGPS,orientationsensorsandmagnetometers.

• Motionsensorsaccelerometers,gravitysensors,gyroscopes,etc.

• Environmentalsensorsbarometers,photometers,andthermometers.

SensorFramework

Accesssensorsandandacquirerawsensordata.

• Determinewhichsensorsareavailableonadevice.• Determineanindividualsensor'scapabilities,suchasitsmaximumrange,manufacturer,powerrequirements,andresolution.• Acquirerawsensordataanddefinetheminimumrateatwhichyouacquiresensordata.• Registerandunregistersensoreventlistenersthatmonitorsensorchanges.

Sensors•Hardware-based orSoftware-based•Hardware-basedsensors- physicalcomponentsbuiltintoahandsetortabletdevice• directlymeasuringspecificenvironmentalproperties.• acceleration,geomagneticfieldstrength,orangularchange.

• Software-basedsensors- mimichardware-basedsensors• derivetheirdatafromoneormoreofthehardware-basedsensorsandaresometimescalledvirtualsensorsorsyntheticsensors.• Thelinearaccelerationsensorandthegravitysensor.

SensorFramework

Accesssensorsandandacquirerawsensordata.AndroidSensorFrameworkincludesthreeclassesandoneinterface.

• SensorManager• Sensor• SensorEvent• SensorEventListener

http://developer.android.com/guide/topics/sensors/sensors_overview.html

• Identifyingsensorsandsensorcapabilities• Monitorsensorevents

1

2

3

4

5

SensorManager

• SystemServicethatmanagessensors

• FirsttheapplicationneedstogetareferencetotheSensorManager• getSystemService(SENSOR_SERVICE);

• Accessaspecificsensorwith• SensorManager.getDefaultSensor(int type)

Sensor

• Accelerometer• Sensor.TYPE_ACCELEROMETER

• MagneticField• Sensor.TYPE_MAGNETIC_FIELD

• Pressure• Sensor.TYPE_PRESSURE

GetaListofAllSensors

SensorEventListener

• ForanapplicationtoreceiveinformationfromaSensor• ItneedstoimplementaSensorEventListener• BeforestartingtoreceivesensorEvents

• TypeofSensor• Delay

SensorEventListener

SensorDelay

SensorEventListener

SensorEventListener

SensorEventListener

• ForanapplicationtoreceiveinformationfromaSensor• ItneedstoimplementaSensorEventListener

• Invokedwhenaccuracyofasensorchanges• Whenthesensoracquiresanewreading

onAccuracyChanged

• Accuracyisrepresentedbyoneoffourstatusconstants:• SENSOR_STATUS_UNRELIABLE• ConstantValue:0(0x00000000)

• SENSOR_STATUS_ACCURACY_LOW,• ConstantValue:1(0x00000001)

• SENSOR_STATUS_ACCURACY_MEDIUM,• ConstantValue:2(0x00000002)

• SENSOR_STATUS_ACCURACY_HIGH,• ConstantValue:3(0x00000003)

https://developer.android.com/reference/android/hardware/SensorManager.html

onAccuracyChanged

onSensorChanged

onSensorChanged

SensorEventListener

• OnceyouaredoneUsingtheSensor

1

2

3

4

5

SensorCoordinateSystem

• Thesensorframeworkusesastandard3-axiscoordinatesystemtoexpressdatavalues.• Xaxisishorizontalandpointstotheright• Yaxisisverticalandpointsup• Zaxispointstowardtheoutsideofthescreenface• coordinatesbehindthescreenhavenegativeZvalues

SensorCoordinateSystem

Suchacoordinatesystemisusedby:

• Accelerationsensor

• Gravitysensor

• Gyroscope

• Linearaccelerationsensor

• Geomagneticfieldsensor

PointstoRemember

• Yourapplicationmustnotassumethatadevice'snatural(default)orientationisportrait.• Thesensorcoordinatesystemisalwaysbasedonthenaturalorientationofadevice.• Thenaturalorientationformanytabletdevicesislandscape.

• Verifysensorsbeforeyouusethem• Verifythatasensorexistsonadevicebeforeyouattempttoacquiredatafromit

Thesensor'scoordinatesystemneverchangesasthedevicemoves

• Youmusttestyoursensorcodeonaphysicaldevice.• Youcurrentlycan'ttestsensorcodeontheemulatorbecausetheemulatorcannotemulatesensors.

• Thereare,however,sensorsimulatorsthatyoucanusetosimulatesensoroutput.

Verifysensorsbeforeyouusethem

IfyouarepublishingyourapplicationonGooglePlayyoucanusethe <uses-feature> elementinyourmanifestfiletofilteryourapplicationfromdevicesthatdonothavetheappropriatesensorconfigurationforyourapplication.

PointstoRemember• Unregistersensorlisteners

• whenyouaredoneusingthesensororwhenthesensoractivitypauses.• Ifasensorlistenerisregisteredanditsactivityispaused,thesensorwillcontinuetoacquiredataandusebatteryresourcesunlessyouunregisterthesensor.

• Don'tblocktheonSensorChanged()method• Sensordatacanchangeatahighrate- systemmaycalltheonSensorChanged(SensorEvent)methodquiteoften

• DoaslittleaspossiblewithintheonSensorChanged(SensorEvent)methodsoyoudon'tblockit

• Choosesensordelayswisely• Sensorscanprovidedataatveryhighrates.• Sendingextradatathatyoudon'tneedwastessystemresourcesandusesbatterypower.

OrientationSensors

• TYPE_ACCELEROMETER usestheaccelerometerandonlytheaccelerometer.Itreturnsrawaccelerometerevents,withminimalornoprocessingatall.

• TYPE_LINEAR_ACCELERATION (ifpresent)usesthegyroscopeandonlythegyroscope.Likeabove,itreturnsrawevents(angularspeedunrad/s)withnoprocessingatall(nooffset/scalecompensation).

• TYPE_ORIENTATION isdeprecated.Itreturnstheorientationasyaw/pitch/rollindegrees.• Thissensorusesacombinationoftheaccelerometerandthemagnetometer.• MarginallybetterresultscanbeobtainedusingSensorManager's helpers.• Thissensorisheavily"processed".

Orientations

Accelerometer Gyroscope

OrientationSensors

OrientationSensors.• TYPE_LINEAR_ACCELERATION,TYPE_GRAVITY,TYPE_ROTATION_VECTOR are"fused"sensorswhichreturnrespectively• thelinearacceleration,• gravityand• rotationvector(aquaternion).• Onsomedevicestheyareimplementedinh/w,• Onsomedevicestheyusetheaccelerometer+themagnetometer• Onsomeotherdevicestheyusethegyro.

TYPE_ACCELEROMETER

• MEMSaccelerometersaretinymassesontinysprings.• Theycansense• Speedinguporslowingdowninastraightline• Shakingthedevice• Earth’sgravity,whichis1gdownward

TYPE_ACCELEROMETER

TYPE_LINEAR_ACCELERATION

TYPE_ACCELEROMETER

TYPE_GYROSCOPE

Measuresrateofrotation.Youcannotdirectlymeasureangleusingagyroscope.Youcanintegratetherateofrotationovertimetogetangle

ActivityRecognition

Walking

45

Bus

46

Driving

47

Train

48

TYPE_MAGNETIC_FIELD

• HardwareSensor• MostlyHalleffectSensors• Androidreportsmagneticfieldsinmicrotesla.• Earth’smagneticfield canvaryfrom30microtesla to60microtesla• Uses• Compass• Themagneticfieldsensorcanbeinfluencedbynearbymetal,somepeoplehaveusedthesensortomakeanAndroiddeviceintoacrudemetaldetector• Duetoaneffectcalledhysteresis

TYPE_PROXIMITY•HardwareSensor•Letsyoudeterminehowfarawayanobjectisfromadevice• Someproximitysensorsprovideaboolean value(near/far).• Typically,thefarvalueisavalue>5cm,butthiscanvaryfromsensortosensor.

•Usuallyusedtodeterminehowfarawayaperson'sheadisfromthefaceofahandsetdevice• Tolockthescreenwhenauserisonacall

TYPE_PROXIMITY

TYPE_PROXIMITY

TYPE_STEP_COUNTER

• Returnsthenumberofstepstakenbytheusersincethelastrebootwhileactivated.• Resettozeroonlyonasystemreboot.• Thetimestampoftheeventissettothetimewhenthelaststepforthateventwastaken.• Thissensorisimplementedinhardwareandisexpectedtobelowpower.• Applicationneedstostayregisteredforthissensorbecausestepcounterdoesnotcountstepsifitisnotactivated.

TYPE_STEP_DETECTOR

• Triggersaneventeachtimeastepistakenbytheuser.• Theonlyallowedvaluetoreturnis1.0andaneventisgeneratedforeachstep.• Thetimestampindicateswhentheevent(herethestep)occurred•Whenthefoothitstheground- generatingahighvariationinacceleration.

ENVIRONMENTSENSORS

• TheAndroidplatformprovidesfoursensorsthatletyoumonitorvariousenvironmentalproperties.• AmbientPressure

• MeasurestheambientairpressureinhPa ormbar.• AmbientHumidity

• Ambienthumiditynearthephone(expressedas%atmospherichumidity)• Illuminance

• Usedtocontrolscreenbrightness(measuredinlux)• Ambienttemperature

• Ambienthumiditynearthephone(measuredindegreecentigrade)

• Theyareallhardwaresensors.

TYPE_HEART_RATE

• FoundinAndroidWearables.• Thereportedvalueistheheartrateinbeatsperminute.• Thissensorrequirespermissionandroid.permission.BODY_SENSORS• Itwillnotbereturned by SensorManager.getSensorsList orSensorManager.getDefaultSensor ifthepermissionismissing.

HEARTRATESENSOR

• MeasuredusingaPhotoplethysmography (PPGsensor).• Measuresthedifferentialreflectionoflightby oxygenatedanddeoxygenatedblood

HEARTRATESENSOR• Similartotheprincipleofandroidappstomeasureheartrateusingthecamera.

• SensorMovementArtifact• NervousFidgetingArtifact

RemovalofLocalFidgeting

ChangesinWristbandAcc +Physiologicalsignal=YESChangesinSmartphoneAcc/Gyro =NO

Removenext30seconds61

ARTIFACTSINHEARTRATE

ArtifactRemoval– ActivityRecognitionUseCase.

RemovalArtifactsusingFiltering

• Twocommonfilteringtechniques• Low-passfilters• Passfrequencieslowerthancutofffrequency• Deemphasizetransientforcechange(vibrations)• Emphasizeconstantforcecomponents• e.g.,forabubblelevel

• High-passfilters• Passfrequencieshigherthancutofffrequency• Emphasizetransientforcechanges• Deemphasizeconstantforcecomponents(gravity)• e.g.,foragamecontroller

SignalPreprocessing

𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)

𝐴"#$% = 𝐴() +𝐴+) +𝐴,)

𝐴.

𝐴+

𝐴,

𝐺(

𝐺+

𝐺,

Accelerometer

Gyroscope

Audio

InputStreams Preprocessing64

SignalPreprocessing

𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)

𝐴"#$% = 𝐴() +𝐴+) +𝐴,)

𝐴.

𝐴+

𝐴,

𝐺(

𝐺+

𝐺,

Accelerometer

Gyroscope

Audio

InputStreams Preprocessing

Tseconds

Nsec Nsec

N=T/10ifN<300

N=30secsOtherwise

BoundaryRemoval

65

SignalPreprocessing

𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)

𝐴"#$% = 𝐴() +𝐴+) +𝐴,)

𝐴.

𝐴+

𝐴,

𝐺(

𝐺+

𝐺,

Accelerometer

Gyroscope

Audio

InputStreams Preprocessing

Screen-oncheckingandremoval

Forcontrolledscenario: Ifscreen-unlocked>10seconds

40Hz

40Hz

8KHz

66

SignalPreprocessing

𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)

𝐴"#$% = 𝐴() +𝐴+) +𝐴,)

𝐴.

𝐴+

𝐴,

𝐺(

𝐺+

𝐺,

Accelerometer

Gyroscope

Audio

Segm

entinto3se

condwindow

InputStreams Preprocessing Segmentation

BoundaryRemoval

AndFiltering

HumanAverageStride rateisbetween80-120stepsaminute

67

𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)

𝐴"#$% = 𝐴() +𝐴+) +𝐴,)

𝐴.

𝐴+

𝐴,

𝐺(

𝐺+

𝐺,

Accelerometer

Gyroscope

Audio

Segm

entinto3se

condwindow

InputStreams Preprocessing Segmentation FeatureExtraction

ZeroCrossing rate

RMSEnergy

MFCCs

Mean

Standarddeviation

Numberofpeaks

Interpeakdistances

minimum

maximum

BoundaryRemoval

AndFiltering

SignalPreprocessing

68

SignalPreprocessing

𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)

𝐴"#$% = 𝐴() +𝐴+) +𝐴,)

𝐴.

𝐴+

𝐴,

𝐺(

𝐺+

𝐺,

Accelerometer

Gyroscope

Audio

Segm

entinto3se

condwindow

InputStreams Preprocessing Segmentation FeatureExtraction

ZeroCrossing rate

RMSEnergy

MFCCs

Mean

Standarddeviation

Numberofpeaks

Interpeakdistances

minimum

maximum

Classification

RandomForestLOSO

ACCGYRO

ACC+GYROACC+GYRO+AUDIOBoundary

RemovalAndFiltering

69

Challenge

Highsamplingratedrainsbattery

70

Effectsofloweringsamplingrates

71

PublicationRelatedtothissection

RecognizingHumanActivitiesfromSmartphoneSensorSignals.ACMMultimedia2014,Ghosh,Arindam ,andRiccardi,Giuseppe

72