Robust Real-Time Pulse Rate Estimation From Facial Video ... · Outline 1 Motivation 2 Data...

Robust Real-Time Pulse Rate Estimation From FacialVideo Using Sparse Spectral Peak Tracking

Aditya Gaonkar P1 Bhuthesh R2 Dipanjan Gope2 PrasantaKumar Ghosh1

1Department of Electrical EngineeringIndian Institute of Science

2Department of Electrical Communication EngineeringIndian Institute of Science

SPCOM, 2016

Aditya Gaonkar P, Bhuthesh R, Dipanjan Gope, Prasanta Kumar Ghosh (Indian Institute of Science)Robust Real-Time Pulse Rate Estimation From Facial Video Using Sparse Spectral Peak TrackingSPCOM, 2016 1 / 25

A challenge!


Outline

1 Motivation

2 Data collection

3 Proposed Algorithm

4 Experiments and Results

5 Conclusions

6 Future work


Motivation

Outline

1 Motivation

2 Data collection



5 Conclusions

6 Future work


Motivation

Motivation for this study

(a) A person shouting at his smartphone (b) A typical example of telemedicine

Situations where remotely monitoring pulse rate can be beneficial


Motivation

Summary of this work

Prior methods

Prior methods for computing pulse rate from facial video include ICAbased techniques, Machine Learning based techniques etc.

In all these works, the pulse rate is computed independently in eachanalysis window.

The performance on our dataset obtained by using the state of theart methods was 14.62 beats per minute (BPM) in root mean squareerror (RMSE).

Our contribution

This work explores benefits of constraining the change of the PRvalue between successive analysis windows.

We have obtained an improvement of 6.71 BPM in RMSE on ourdataset over the state of the art methods.


Data collection

Outline

1 Motivation

2 Data collection



5 Conclusions

6 Future work


Data collection

Recording setup

The resolution of the videos were 1280x720p.

The Samsung phone’s videos were in mp4 format and iPhone’s videoswere in avi format.


Data collection

The FaViP dataset

This dataset is available for download in our lab webpage 1.S U B J E C T 1 S U B J E C T 2 S U B J E C T 3 S U B J E C T 4 S U B J E C T 5

S U B J E C T 6 S U B J E C T 7 S U B J E C T 8 S U B J E C T 9 S U B J E C T 10

S U B J E C T 11 S U B J E C T 12 S U B J E C T 13 S U B J E C T 14 S U B J E C T 15

A good amount of variability across the subjects is illustrated.

1at http://spire.ee.iisc.ernet.in/spire/database.phpAditya Gaonkar P, Bhuthesh R, Dipanjan Gope, Prasanta Kumar Ghosh (Indian Institute of Science)Robust Real-Time Pulse Rate Estimation From Facial Video Using Sparse Spectral Peak TrackingSPCOM, 2016 9 / 25

Proposed Algorithm

Outline

1 Motivation

2 Data collection



5 Conclusions

6 Future work


Proposed Algorithm

Motivation

In all prior works, decision is taken independently in each analysiswindow.

Pulse rate is a slowly varying quantity and for 2 analysis windowsalmost overlapping each other, pulse rate wouldnt have changed by ahuge margin.

The above fact has been exploited in the proposed algorithm.


Proposed Algorithm

State of the art method

The above block diagram illustrates the method proposed inthe state of the art method a

aMing-Zher Poh, Daniel J McDuff, and Rosalind W Picard, “Non-contact,automated cardiac pulse measurements using video imaging and blind sourceseparation”, Optics express, vol. 18, no. 10, pp. 1076210774, 2010.


Proposed Algorithm

Illustrating the state of the art method


Proposed Algorithm



Proposed Algorithm



Proposed Algorithm



Proposed Algorithm

Additional step #1: Spectral windowing of the ICA traces

We propose spectral windowing of the ICA traces in eachsegment using various spectral windows as an additional step


Proposed Algorithm

Additional step #2: Sparsifying and adding windowed ICAtraces

We propose sparsifying the spectra of the windowed ICAtraces by preserving the lobes around the top R peaks and

then adding these spectra which potentially can magnify thepeak corresponding to the pulse rate and suppress noisy

peaks.


Proposed Algorithm

Advantage of sparsifying and adding spectra illustrated


Proposed Algorithm

Advantage of sparsifying and adding spectra illustrated


Proposed Algorithm

Additional step #3: Constructing candidate PR trajectories

Candidate pulse rate trajectories are constructed byselecting R highest peaks from the sum of the sparsified ICAspectra. The trajectory whose cumulative spectral power is

the highest is declared as the best pulse rate trajectory


Proposed Algorithm

Illustration of constructing pulse rate trajectories (R=5)

1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)

Constructing frequency trajectories


Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)



Proposed Algorithm


1 2 3 4 5 6 7 8 9 10

0.8

1

1.2

1.4

1.6

1.8

2

2.2

analysis window

frequency (

in H

z)

Frequency trajectories

Chosen trajectory


Experiments and Results

Outline

1 Motivation

2 Data collection



5 Conclusions

6 Future work



Experimental setup I

T=20 sec and ∆T=1 sec was chosen. Rectangular, Hann, Hammingand Blackman windows were used. For sparsification, R=3,5,10,15were chosen.

FFTs of length 4096 were computed in finding the power spectra.

We consider the median of the values of PRW ,R(K ) for the given Rsand W s as the final decision on the PR, henceforth denoted asC SSPT

The code for the whole algorithm was written in MATLAB R2014a.



Experimental setup II

Root Mean Square Errors (RMSE) from the groundtruths were usedas the measurers of the performance of the proposed algorithm.

As a baseline scheme for Comparison, the work by Poh et al 2 is used.

Also to compare how sparsification can help, the spectra of the ICAtraces were added and then trajectories were constructed by pickingtop R peaks.

2Ming-Zher Poh, Daniel J McDuff, and Rosalind W Picard, “Non-contact, automatedcardiac pulse measurements using video imaging and blind source separation”, Optics express,vol. 18, no. 10, pp. 1076210774, 2010.



Results I

Mean RMSE in BPMRectangular Hann Hamming Blackman

7.91 9.49 8.88 10.03

Rectangular window gives the best performance (byconsidering R=3,5,10,15 and taking C SSPT) .



Results II

Mean RMSE in BPMR Sparse Spectra Full Spectra

3 8.30 8.77

5 8.12 8.36

10 8.18 8.43

15 8.15 8.21

It can’t be said with surety which R ensures bestperformance.

Benefit of sparsification is highlighted.



Results III

Mean (std. dev.) RMSE in BPMExperi- Baseline C SSPTmental Sparse Full

Condition spectra spectra

S3 0.5 11.36 (11.04) 6.45 (6.13) 7.88 (7.17)

S3 1 10.62 (7.77) 7.56 (8.04) 9.03 (8.68)

S3 2 19.08 (17.31) 13.24 (18.41) 13.06 (18.08)

3GS 0.5 13.75 (14.04) 6.98 (7.15) 7.79 (7.93)

3GS 1 15.57 (9.07) 7.86 (9.06) 7.82 (8.74)

3GS 2 17.38 (14.58) 15.41 (15.23) 15.25 (17.29)

The proposed SSPT framework outperforms the baseline framework by∼ 6.71 BPM on an average (by considering rectangular window and

R=3,5,10,15 for C SSPT).



Results IV

60

70

80

90

S U B J E C T 3

60

70

80

90

40

60

80

40

60

80

70

80

90

Hea

rtbea

t rat

e in

bea

ts p

er m

inut

e

10 20 30 4070

80

90

40

50

60

70

S U B J E C T 9

40

50

60

70

40

60

80

40

60

80

406080

100120140

10 20 30 40406080

100120140

Time (in second)

406080

100120140

S U B J E C T 11

S3_0.5

406080

100120140

3GS_0.5

50

100

150

S3_1

50

100

150

3GS_1

100

200

S3_2

10 20 30 40

100

200

3GS_2

Ground Truth Sparse Spectra Full Spectra Baseline

The benefit of the proposed method vs baseline method highlighted


Conclusions

Outline

1 Motivation

2 Data collection



5 Conclusions

6 Future work


Conclusions

Conclusions

Pulse rate estimation from face videos benefit from sparsifying thespectra of ICA traces and by using the fact that pulse rate is a slowlyvarying quantity.

The nearer the subject is to the camera, better is the estimationaccuracy.


Future work

Outline

1 Motivation

2 Data collection



5 Conclusions

6 Future work


Future work

Case for better trajectory selection

0 5 10 150

20

40

60

80

Trajectory index

RM

SE

of

ea

ch

tra

jecto

ry in

BP

M

RMSEs of all trajectories

← Least RMSE=0.70092BPM

← RMSE of chosen tra jectory=9.0377BPM

0 5 10 150.5

1

1.5

2

2.5

3

3.5x 10

4 Tρ values for all trajectories

Trajectory index

Tρvalue

← Tρ for least RMSE trajectory

← Tρ for selected tra jectory

A case for better a trajectory selection mechanism (where Tp is the

net spectral power of a trajectory across analysis windows)


Future work

Future work

Studying effects of using different video compression schemes(codecs) is also a part of our study in this direction.


Future work

Thank You!


Date post:	20-Aug-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Robust Real-Time Pulse Rate Estimation From Facial Video ... · Outline 1 Motivation 2 Data...

Documents