Eye Contact Correction using Deep Neural Networks

Leo F. Isikdogan, Timo Gerasimow, and Gilad MichaelIntel Corporation, Santa Clara, CA

{leo.f.isikdogan, timo.gerasimow, gilad.michael}@intel.com

Abstract

In a typical video conferencing setup, it is hard to main-tain eye contact during a call since it requires looking intothe camera rather than the display. We propose an eye con-tact correction model that restores the eye contact regard-less of the relative position of the camera and display. Un-like previous solutions, our model redirects the gaze froman arbitrary direction to the center without requiring aredirection angle or camera/display/user geometry as in-puts. We use a deep convolutional neural network that in-puts a monocular image and produces a vector field and abrightness map to correct the gaze. We train this model ina bi-directional way on a large set of synthetically gener-ated photorealistic images with perfect labels. The learnedmodel is a robust eye contact corrector which also predictsthe input gaze implicitly at no additional cost.

Our system is primarily designed to improve the qualityof video conferencing experience. Therefore, we use a setof control mechanisms to prevent creepy results and to en-sure a smooth and natural video conferencing experience.The entire eye contact correction system runs end-to-end inreal-time on a commodity CPU and does not require anydedicated hardware, making our solution feasible for a va-riety of devices.

1. IntroductionEye contact can have a strong impact on the quality and

effectiveness of interpersonal communication. Previous ev-idence suggested that an increase in the amount of eye con-tact made by a speaker can significantly increase their per-ceived credibility [1]. However, a typical video conferenc-ing setup creates a gaze disparity that breaks the eye contact,resulting in unnatural interactions. This problem is causedby having a display and camera that are not aligned witheach other. During video conferences, users tend to look atthe other person on the display or even a preview of them-selves rather than looking into the camera.

Earlier solutions required specific hardware such asa pair of cameras that help synthesize gaze-corrected

Figure 1. Eye contact correction: the user is looking at the screenin the input frame (left). The gaze is corrected to look into thecamera in the output frame (right).

images [2, 22] or reflective screens similar to that ofteleprompters. A more recent solution [8] used a singlecamera to correct the gaze by 10-15 degrees upwards, as-suming that a typical placement for a camera would be atthe top-center of the device, just above the screen. How-ever, many new portable devices have their cameras locatedat the top-left and top-right corners of the displays. Such de-vices would require horizontal gaze correction as well as theupwards correction. Furthermore, many tablets and smart-phones can be rotated and used in any orientation. Differ-ent users may use their devices at different orientations andview the display from different distances. This effectivelychanges the relative position of the camera with respect tothe user and the center of the display. Therefore, a uni-versal eye contact corrector should support redirecting thegaze from an arbitrary direction to the center regardless ofthe relative camera and display positions.

A deep learning based approach [3] showed that it is pos-sible to redirect gaze towards an arbitrary direction, given aredirection angle. In a typical use case of eye contact cor-rection, however, neither a redirection angle nor the inputgaze direction is available. It is indeed possible to replaceeyes with rendered 3D models of eyes to simulate an ar-

1

arX

iv:1

906.

0537

8v1

[cs

.CV

] 1

2 Ju

n 20

19

bitrary gaze [21, 15] without having a redirection angle.However, using such a model for gaze correction in video-conferencing would be challenging since it is hard to renderdetails such as eyelashes and glasses in real-time while re-maining faithful to the original input.

We propose an eye contact correction system that is de-signed primarily to improve video conferencing experience.Our system first uses a facial landmark detector to locateand crop the eyes, and then feeds them into a deep neuralnetwork. Our proposed model architecture learns to redi-rect an arbitrary gaze to the center without requiring a redi-rection angle. We show that when a redirection angle is notgiven, the model learns to infer the input gaze implicitly. Asa side product, our model predicts the input gaze directionand magnitude at no additional cost. Finally, our eye con-tact corrector outputs frames having smooth and naturallycorrected gaze using a set of control mechanisms. Thosemechanisms control the strength of the correction, prevent‘creepiness’ from overly corrected eye contact, and ensuretemporal consistency in live applications. Our live applica-tion (Figure 1) runs in real-time on CPU, making our eyecontact corrector a feasible solution for a wide range of de-vices.

2. Related Work

Eye contact correction can be considered a specific caseof gaze manipulation where the gaze is redirected to thecenter in a video conferencing setup. Numerous solutionsthat specifically addressed the video conferencing gaze cor-rection problem required additional hardware such as stereocameras [2, 22] or depth sensors [10, 23]. Kononenko etal. [8] proposed monocular solution that solely relied onimages captured by a web camera. Their solution used en-sembles of decision trees to produce flow fields, which arelater used to warp the input images to redirect gaze upwards10 to 15 degrees. As discussed earlier, this type of verticalcorrection works well only when the camera is located atthe top center of the screen, with a predefined distance fromthe user. However, many hand-held devices can be used inboth landscape and vertical orientations and at an arbitraryviewing distance.

A more flexible approach, named DeepWarp [3], used adeep neural network to redirect the gaze towards an arbi-trary direction. DeepWarp can manipulate the gaze towardsany direction, thus can be used for gaze correction in videoconferencing regardless of device orientation and user dis-tance, given a redirection angle as input. However, such aredirection angle is usually hard to obtain in real life sce-narios. For example, even when the device type, orienta-tion, and user distance is known, a fixed redirection anglewould assume that all users look at the same point on thedisplay to properly correct the gaze. In practice, windows

that show the participants in a video call can be shown atdifferent parts of the display. Furthermore, users may evenprefer to look at the preview of themselves rather than theother person.

Wood et al. [21] proposed an approach that can redirectthe gaze to any given direction without inputting a redirec-tion angle. Their method created a 3D model of the eyeregion, recovering the shape and appearance of the eyes.Then, it redirected the gaze by warping the eyelids and ren-dering the eyeballs having a redirected gaze. However, themodel fitting step in their algorithm limited the real-timecapability of their approach.

Although some of the earlier work employed tempo-ral smoothing techniques [10], earlier gaze correction andredirection solutions overall tried to correct the gaze con-stantly, without a control mechanism. Therefore, the useof a general-purpose gaze redirector for video conferencingwould lead to unnatural results particularly when the user isnot engaged or moves away from a typical use case.

3. Data PreparationTo train and validate our system, we prepared two dif-

ferent datasets: one natural and one synthetic. The natu-ral dataset (Figure 2) consists of image pairs where a sub-ject looks into the camera and at a random point on dis-play. Similarly, the synthetic dataset (Figure 3) consists ofimage sets within which all factors of variation except forgaze stays constant. We used the natural dataset primarilyto validate our model and to refine the samples in the syn-thetic dataset to look virtually indistinguishable from thenatural ones. Being able to generate a photorealistic syn-thetic dataset allowed for generating an immense amount ofperfectly-labeled data at a minimal cost.

3.1. Natural Dataset

We created a dataset that consists of image pairs wherethe participants saccaded between the camera and randompoints on display. The gaze of the participants was guidedby displaying dots on the screen. The subjects participatedin our data collection at their convenience without being in-vited into a controlled environment, using a laptop or tabletas the data collection device. Therefore, the collected datais representative of the typical use cases of the proposed ap-plication.

Unlike the gaze datasets that are collected in a controlledenvironment [16], we did not use any apparatus to stabi-lize the participans’ face and eyes or to prevent them frommoving between frames. To locate the eyes in the capturedframes, we used a proprietary facial landmark detector de-veloped internally at Intel. The facial landmark detectorprovided a set of facial landmarks which we utilized to alignand crop the eyes in the captured frames.

Figure 2. Sample pairs from the natural dataset: the first imagein every pair looks at a random point on the display whereas thesecond one looks into the camera.

To improve the data quality, we created a routine thatautomatically deleted the frames that were likely to be erro-neous. First, the cleaning routine removed the first frames ineach sequence to compensate for any lagged response fromthe subjects. Second, it removed the frames where no faceswere detected. Finally, it removed the frames where thesubject was blinking, where the blinks were inferred fromthe distances between eye landmarks. We removed any in-complete pairs where either the input or ground truth im-ages were missing to make sure all pairs in the dataset arecomplete. The clean dataset consisted of 3125 gaze pairsequences collected from over 200 participants.

3.2. Synthetic Dataset

Our synthetic data generator used the UnityEyes plat-form [20] to render and rasterize images of eyes, which arelater refined by a generative adversarial network. UnityEyesprovides a user interface where the gaze can be moved bymoving the cursor. We created sets of eye images by pro-grammatically moving the cursor to move the gaze towardsrandom directions. We modeled the cursor movements as azero mean Gaussian random variable, where zero means acentered gaze, looking right into the camera.

To increase the diversity of samples in the dataset, werandomized subject traits, lighting, and head pose betweendifferent sets of images. We sampled 40 different gazesper set, where all images within a set had the same randomconfiguration. Randomizing the subject traits changed thecolor, shape, and texture of the face, skin, and eyes. Usingthis process, we generated 3200 sets of artificial subjectswith random traits, resulting in 128,000 images and nearly2.5 million image pairs.

We limited the range of movement in the head pose ran-domization since we would not enable eye contact correc-tion if the user is clearly looking at somewhere other thanthe camera and display. Therefore, we made sure that thehead pose was within the limits of a typical use case wherethe eye contact correction would be practical to use. To fur-

Figure 3. Samples pairs from the synthetic dataset: each imagepair belongs to a distinct randomized subject. The image pairs arealigned fixing everything except the gaze.

ther increase randomness, we also randomized the renderquality of the synthesized images. Indeed, the use of highestpossible render quality can be ideal for many applications.However, the amount of detail in those images, such as thereflection of the outside world on the surface of the eyes,can be unrealistic in some cases depending on the imagingconditions. After we captured raster images from the Uni-tyEyes platform, we superimposed glasses of different sizesand shapes on some of the image sets. The glasses used 25different designs as templates, where their size, color, andrelative position were randomized within a visually realisticrange.

UnityEyes provides facial landmarks for the eyes, whichare comparable to the ones we used for the natural dataset.Once the glasses are superimposed, we used those faciallandmarks to align and crop the eyes. Since the images aregenerated synthetically, they can be perfectly aligned be-fore the eyes are cropped. However, merely using a bound-ing box that encloses the eye landmarks leads to misalignedpairs. Cropping each image separately leads to small off-sets between the images in the same set due to landmarksshifted by the gaze. Thus, we created a bounding box thatfit all images in a given set and used a single bounding boxper set. The bounding boxes had a fixed aspect ratio of 2:1and are padded to have twice as much width as the averagewidth in a given set.

To enhance photorealism, we used a generative adversar-ial network that learned a mapping between synthetic andreal samples and brought the distribution of the syntheticdata closer to real ones. Using the trained generator, werefined all images in the synthetic dataset to create a largedataset that consists of photorealistic images having virtu-ally perfect labels. This process is detailed in Section 8.

All of the steps mentioned above are done only once asa pre-processing step. The pre-processed image pairs arealso distorted on the fly during training with additive noise,brightness and contrast shift, and Gaussian blur, in random

64x32x3

TargetAngle

Tile

64x32x5

Input Image Co

ncat

enat

e2 64x32x2

Poo

ling

Co

nvol

utio

nB

lock

Poo

ling

Co

nvol

utio

nB

lock32x16x32

Poo

ling

Co

nvol

utio

nB

lock16x8x64

Poo

ling

Co

nvol

utio

nB

lock8x4x128

Up

-con

volu

tion

Co

nvol

utio

nB

lock

4x2x256

Up

-con

volu

tion

Co

nvol

utio

nB

lock

8x4x128

Co

ncat

enat

e

16x8x64

Co

nvol

utio

nB

lock

Co

ncat

enat

e

16x8x32

BrightnessMap

64x32x1

Vector Field 64x32x2

Up

-con

volu

tion

Up

-con

volu

tion

Co

nvol

utio

nLa

yer

Sig

moi

d

Warp

Adjust LocalBrightness

OutputImage

ConvolutionLayer

BatchNorm

ReluConvolution

LayerBatchNorm

ReluConvolution

LayerBatchNorm

Relu+

Convolution Block

Figure 4. The architecture of the eye contact correction model: ECC-Net inputs a patch that contains a single eye, warps the input toredirect gaze, and adjusts the local brightness to enhance eye clarity. Blocks with trainable parameters are shown in blue.

order and magnitude. These distortions not only emulateimperfect imaging conditions but also further augment thediversity of the samples in the dataset.

4. The ECC-Net ModelOur eye contact correction model, named ECC-Net, in-

puts an image patch that contains a single eye and a targetgaze vector. The image patches are resized to 64 × 32 be-fore they are fed into the model. The target gaze vector isrepresented in the Cartesian domain with its horizontal andvertical components and is tiled to have the same spatial di-mensions as the input image. Once the training is complete,the target angle is set to zeros to redirect the gaze to center.

The core of ECC-Net is a fully-convolutional encoder-decoder network which uses U-Net style skip connectionsand channel-wise concatenations [13] to recover detailslost at the pooling layers. The model does the bulk ofprocessing in low resolution both to reduce the compu-tational cost and to improve spatial coherence of the re-sults. The convolutional blocks in the model consist ofthree depthwise-separable convolutional layers with a resid-ual connection [5] that skips over the middle layer. Theconvolutional layers use batch normalization [6] and ReLUactivations.

The model produces a flow field and a brightness mapsimilar to the methods presented in [8] and [3]. The outputlayer consists of two up-convolution layers (2× 2 convolu-tion with a stride of 1/2) followed by a convolutional layerhaving a 3-channel output. Two of these channels are useddirectly to predict the horizontal and vertical components ofa vector field that is used to warp the input image. The thirdchannel is passed through a sigmoid function and used as amap to adjust local brightness. Using such a mask is shownto be effective in improving the appearance of eye whites

after gaze warping [3]. The brightness mask enhances eyeclarity and corrects the artifacts that result from horizon-tal warping when there are not enough white pixels to re-cover the eye white. The overall architecture of the modelis shown in Figure 4.

For eye contact correction, training a model to output avector field has several advantages over training a genericencoder-decoder model that produces pixel-wise dense pre-dictions. First, the vector fields produced by the modelcan be easily modified in a meaningful way using externalsignals. For example, their magnitude can be scaled be-fore warping to control the correction strength. Those vec-tors can also be averaged over time for temporal smoothingwithout producing blurry results (Section 7). Second, pre-dicting a motion vector imposes the prior that pixels shouldmove rather than changing in an unconstrained way whenthe gaze changes. Finally, training a model to output thepixel values directly can lead to a bias towards the meanimage in the training set [3], resulting in loss of detail.

Indeed, images can be generated with a high level of de-tail using an adversarial loss [11] instead of a mean squarederror loss. A generative adversarial network (GAN) canlearn what is important to produce in the output [4]. How-ever, although generative adversarial networks are better atreconstructing details, the details they produce might origi-nate neither in the input nor the ground truth. A model thatis trained with an adversarial loss can hallucinate detailswhen the output is comprised of unrestricted pixels. Thisbehavior might be acceptable or even preferred for manyapplications. However, we would not want this type of flex-ibility to redirect gaze in a video conferencing setup. Forexample, adding eyelashes or any other traits that are hallu-cinated might lead to unnatural results. Therefore, we builta model that manipulates the location and brightness of ex-

ECC-Net

Input Image

Target Angle Correction Loss

Output Image

Target Image

ECC-Net

Input AngleReconstructed

Image

Reconstruction LossShared Weights

Figure 5. Bi-directional training: the model optimizes the correc-tion and reconstruction losses concurrently to enforce mapping re-versibility.

isting pixels. This approach ensures that any detail that is inthe output originates in the input.

5. Bi-directional Training

We trained ECC-Net in a bi-directional fashion to en-force mapping reversibility. The model is first given an in-put image and a target angle to redirect the gaze. In the firstdirection, the model is expected to minimize a correctionloss Lc, which is defined as the mean squared error betweenthe gaze-corrected and ground truth images. In the other di-rection, the model is given the gaze-corrected output imageand the input angle to redirect the gaze back to its originalstate. Although this should be the expected behavior of agaze redirection model, we found that some warping arti-facts in the output make it difficult to recover the originalimage. To address this problem, we defined a reconstruc-tion loss Lr between the reconstructed image and the orig-inal image and optimize it concurrently with the correctionloss (Figure 5).

Training the model in a bi-directional way reduced theartifacts and resulted in more natural gaze redirection re-sults. However, assigning the correction and reconstruc-tion losses the same weight during training led to a modecollapse where the model quickly converged to an identitytransform to minimize the reconstruction loss. Readjust-ing the weights of the losses in the total loss function asLtotal = 0.8Lc + 0.2Lr helped the optimizer keep a goodbalance between the loss functions in both directions.

The target angles are used only during training and setto (0, 0) during inference since the goal of the model is tomove the gaze to the center. Using target angles other thanzero during training improved the robustness of the modeland allowed for post-training calibration. For example, ifthe gaze is still off after correction on a particular devicethen the target angle can be tuned to compensate for theoffset, although this should not be necessary in a typical

Figure 6. Gaze prediction: ECC-Net predicts the input gaze as abyproduct of eye contact correction. The white circle in the figureshows the predicted gaze.

case. Using pairs of images having arbitrary gazes also in-creased the number of possible image pairs in the trainingdata. For example, using a set of 40 images for a given sub-ject,

(402

)= 780 unique pairs can be generated as compared

to 39 pairs using a single target. This effectively augmentedthe data and reduced the risk of overfitting.

6. Gaze PredictionAn intriguing phenomenon we observed is that the model

learned to predict the input gaze implicitly. We found thatcomputing the mean motion vector, negating its direction,and scaling its magnitude to fit the screen gives an estimateof the input gaze (Figure 6). Unlike a typical multi-tasklearning setup where a model is trained to perform multipletasks simultaneously, our model learns to perform two taskswhile being trained to perform only one of them. Therefore,we can arguably consider the eye contact correction prob-lem as a partial super-set of gaze prediction.

We should note that our model is not a fully-blown gazepredictor, but rather is an eye contact corrector that learnsthe input gaze to function better. This behavior is likelya byproduct of training the model to redirect gaze withoutproviding a redirection angle, which requires the input gazeangle to be inferred. The inferred gaze does not incorporatehead pose or distance from the screen and relies only onthe information extracted from eyes in isolation. Therefore,it should not be expected to be as accurate as systems thatuse dedicated sensors or models [9, 18] that are designedspecifically for gaze prediction.

The predicted gaze can still be practical to use in a va-riety of use cases where the computational cost is a con-cern, since the additional cost, i.e., mean computation andnegation, is negligible. For example, a video conferenc-ing application that uses eye contact correction would be

able to compute gaze statistics with minimal overhead. Thereal-time gaze information would also enable hands-free in-teractions, such as dimming the backlight when the user isnot engaged. Thus, the gaze prediction property of our eyecontact corrector has the potential to decrease battery con-sumption while providing additional functionality.

7. Control MechanismWe provide a set of mechanisms that control the correc-

tion strength smoothly to ensure a natural video conferenc-ing experience. The control mechanisms we use can begrouped into two blocks: a control block that reduces thecorrection strength by scaling the ECC-Net output whenneeded, and a temporal stability block that temporally fil-ters the outputs.

Eye contact correction is disabled smoothly when theuser is too far from the center, too far away from the screen,too close to the screen, or blinking. The correction is alsodisabled when the user looks somewhere other than thecamera and display (Figure 7). The control block moni-tors the face size, distance from the center, head pose (i.e.,pitch, roll, yaw), and eye opening ratio, which are inferredfrom the output of the same facial landmark detector thatwe use to align and crop the eyes. In addition to the fa-cial landmarks, the control block also factors in mean andmaximum motion vector magnitudes to limit correction forextreme gazes. Both landmark and motion vector based sig-nals produce a scaling factor between 0 and 1. The overallcorrection strength is calculated by multiplying those scal-ing factors calculated for each triggering signal.

The stability block filters the motion vectors tempo-rally using an alpha-beta filter, which is a derivative of theKalman filter [12]. The filtering is done on the vector fieldbefore warping input images rather than pixel values afterwarping. This process eliminates flicker and outlier motionvectors in an input video stream without blurring out theoutput images. When used together with the control block,the temporal stability block ensures the eye contact correc-tion operates smoothly in a video conferencing setting.

Overall, the control mechanisms prevent abrupt changesand ensure that the eye contact corrector avoids doing anycorrection when the user diverts away from a typical videoconferencing use case. Consequently, the eye contact cor-rector operates smoothly and prevents ‘creepy’ or unneededcorrections.

8. ExperimentsIn our experiments, we trained ECC-Net using only the

synthetic dataset and used the natural dataset as a valida-tion set to pick the best performing model configuration.Once the training is complete, we tested the frozen model onthe Columbia Gaze Dataset [16], which is a public bench-

Figure 7. Control mechanism: ECC is enabled for typical use cases(top) and disabled when the user diverts away from the primary usecase (bottom).

mark dataset that was originally used for eye contact detec-tion [16]. We reorganized the Columbia Gaze Dataset tohave gaze pairs similar to our natural dataset. Using datafrom entirely different sources for training, validation, andtest sets minimized the risk of overfitting, including its im-plicit forms such as information leakage from the valida-tion set due to excessive hyperparameter tuning or datasetbias [19]. We disabled the control mechanisms in all experi-ments. However, in the validation and test sets, we excludedthe images where the control block would disable ECC-Netentirely.

Initially, we trained the model on both left and right eyes,where left eyes on the synthetic dataset were generated byflipping right eyes. This resulted in a poor horizontal cor-rection since the model needed to put considerable effort todecide whether the input is a left or right eye to be able tocorrect the gaze horizontally in the right amount. To bet-ter utilize the model capacity for correction, we trained themodel on right eyes only and flipped left eyes during infer-ence. Consequently, the model learned to correct the gazebetter both horizontally and vertically.

We used the relative reduction in mean squared erroras the performance metric and modified it to be more ro-bust against minor misalignments. This misalignment-tolerant error used the minimum of errors between imagepairs shifted within a slack of 3x3 pixels. We found themisalignment-tolerant error more consistent with the visualquality of the results as compared to a rigid pixel-to-pixelsquared error.

We trained our model for about 3 million iterations, us-ing an Adam [7] solver with β1 = 0.9 β2 = 0.999, ε = 0.1,and a cyclic learning rate [17] between 0.002 and 0.01. Us-ing a relatively large ε helped stabilize training. The errorreached its minimum value at around 2 million iterations.The model at this iteration reduced the error by 62% com-

Figure 8. Samples from the synthetic dataset before (left) and after(right) they are refined using a generator network. The refined im-ages reveal some details about the distribution of data in the naturaldataset, such as reflections in the eyes and glare on glasses. Thegenerator brings the distribution of the synthetic data closer to realdata and makes eyes and their surroundings more photorealistic byadding those details among many others.

pared to identity transform (Table 1). The model also pro-duced visually good looking results. We were able to furtherdecrease the overall error by using a portion of the naturaldataset for fine-tuning and the rest for validation. It is acommon practice in deep learning applications to freeze thefirst layers and fine tune the last ones to prevent overfitting.This is because the models transfer weights from other mod-els that used similar data to accomplish different tasks. Inour case, however, the task is the same for both the naturaland synthetic dataset while the input data distribution mightdiffer. Therefore, we tuned only the first layers (layers be-fore the first skip connection) while the rest of the networkstayed frozen. Using a portion of the natural data for finetuning decreased the error marginally.

Although fine tuning on natural data helped reduce theerror, it also noticeably decreased the correction strengthand worsened the qualitative results (Figure 9). Despite themisalignment-tolerant error metric, some of the remainingerror on the natural dataset was still due to the differencesother than the gaze, such as shadows and reflections. Weobserved that a substantial decrease in the error was a resultof better gaze correction whereas smaller ‘improvements’were a result of closer-to-average results that smoothed outother factors of variation. Therefore, we used the naturaldataset as a development set and calculated the error as asanity check rather than as a benchmark, while continu-ously monitoring the results qualitatively. Overall, trainingthe model solely on synthetic data resulted in visually bet-ter results. This is likely a result of the impact of perfectlabels in the synthetic set outweighing the impact of a datadistribution closer to the real use case in the natural set.

To bring the distribution of the synthetic data closer to

(a) (b) (c) (d) (e)

Figure 9. Results on samples from the validation set: (a) input,(b) model fine-tuned on natural data, (c) model trained on unre-fined synthetic data only, (d) model trained on refined syntheticdata, (e) ground truth.

Training Data Validation Error Test ErrorUnrefined Synthetic 0.386 0.431Natural + Synthetic 0.372 0.465Refined Synthetic 0.375 0.414

Table 1. The relative mean squared error on the validation (naturaldataset) and test (Columbia Gaze) sets when the model is trainedon synthetic data before and after refinement. Training a model onrefined synthetic images achieved a similar error as training it onunrefined images followed by fine tuning on a disjoint portion ofthe natural dataset. However, the models that used synthetic dataachieved a low error via better gaze correction whereas the modelthat is fined tuned on the natural data produced closer to averageresults.

real data without sacrificing the label quality, we built agenerative adversarial network based on CycleGAN [24].CycleGAN uses a cycle-consistent training setup to learna mapping between two image sets without having a one-to-one correspondence. We modified and trained Cycle-GAN to learn a mapping between our synthetic and naturaldatasets, generating a photorealistic eye image given a syn-thetic sample. In our training setup, we used two additionalmean absolute error (L1) losses defined between the inputsand outputs of the generators to further encourage input-output similarity. This type of ‘self-regularization’ loss hasbeen previously shown to be effective for training GANs torefine synthetic images [14]. We defined the additional lossfunctions only on the luminance channel to give the modelmore flexibility to modify color while preserving the gazedirection and the overall structure of the input. We used thedefault hyperparameters for CycleGAN for training, treat-ing the additional L1 losses the same as the reconstructionlosses. The trained generator produced photorealistic im-

Figure 10. Results on a random subset of the Columbia Gaze Dataset. The model is trained using only synthetic samples which wererefined using the natural images in our dataset. The leftmost image in each group shows the input, the middle image shows the ECC-Netresult, and the rightmost image shows the ground truth which was used to compute the test set error.

ages without changing the gaze in the input (Figure 8).Training the model on the GAN-refined synthetic images

achieved a similar error as the fine tuned model without de-grading the visual quality of the outputs. The results had al-most no artifacts for the typical use cases. The artifacts wereminimal even for the challenging cases such as where thereis glare, glass frames are too close to the eye, or the sceneis too dark or blurry. Qualitative results on a random sub-set of the Columbia Gaze Dataset are shown in Figure 10.The visual examples show that some of the error betweenthe gaze-corrected and ground truth images is explained byfactors other than the gaze, such as moved glass frames andhair occlusions. The results look plausible even when theyare not very similar to the ground truth images.

9. ConclusionWe presented an eye contact correction system that redi-

rects gaze from an arbitrary angle to the center. Our eyecontact corrector consists of a deep convolutional neural

network, which we call ECC-Net, followed by a set of con-trol mechanisms. Unlike previous work, ECC-Net does notrequire a redirection angle as input, while inferring the in-put gaze as a byproduct. It supports a variety of video con-ferencing capable devices without making an assumptionabout the display size, user distance, and camera location.ECC-net preserves details, such as glasses and eyelashes,without hallucinating details that do not exist in the input.Our training setup prevents destructive artifacts by enforc-ing mapping reversibility. The trained model employs con-trol mechanisms that actively control the gaze correctionduring inference to ensure a natural video conferencing ex-perience. Our system improves the quality of video con-ferencing experience while opening up new possibilities fora variety of other applications. Those applications may in-clude software teleprompters and personal broadcasting ap-plications that provide cues on display while maintainingeye contact with the viewers.

DisclaimerNo license (express or implied, by estoppel or otherwise)

to any intellectual property rights is granted by this doc-ument. This document contains information on products,services and/or processes in development. All informationprovided here is subject to change without notice. Intel andthe Intel logo are trademarks of Intel Corporation in the U.S.and/or other countries. c© Intel Corporation.

References[1] S. A. Beebe. Eye contact: A nonverbal determinant of

speaker credibility. The Speech Teacher, 23(1):21–25, 1974.1

[2] A. Criminisi, J. Shotton, A. Blake, and P. H. Torr. Gaze ma-nipulation for one-to-one teleconferencing. In IEEE Interna-tional Conference on Computer Vision (ICCV), pages 13–16,2003. 1, 2

[3] Y. Ganin, D. Kononenko, D. Sungatullina, and V. Lempit-sky. Deepwarp: Photorealistic image resynthesis for gazemanipulation. In European Conference on Computer Vision(ECCV), pages 311–326. Springer, 2016. 1, 2, 4

[4] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Gen-erative adversarial nets. In Advances in neural informationprocessing systems, pages 2672–2680, 2014. 4

[5] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learningfor image recognition. In IEEE Conference on Computer Vi-sion and Pattern Recognition (CVPR), pages 770–778, 2016.4

[6] S. Ioffe and C. Szegedy. Batch normalization: Acceleratingdeep network training by reducing internal covariate shift.In International Conference on Machine Learning (ICML),pages 448–456, 2015. 4

[7] D. P. Kingma and J. Ba. Adam: A method for stochasticoptimization. In International Conference on Learning Rep-resentations (ICLR), 2015. 6

[8] D. Kononenko and V. Lempitsky. Learning to look up: Real-time monocular gaze correction using machine learning. InIEEE Conference on Computer Vision and Pattern Recogni-tion (CVPR), pages 4667–4675, 2015. 1, 2, 4

[9] K. Krafka, A. Khosla, P. Kellnhofer, H. Kannan, S. Bhan-darkar, W. Matusik, and A. Torralba. Eye tracking for every-one. In IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2016. 5

[10] C. Kuster, T. Popa, J.-C. Bazin, C. Gotsman, and M. Gross.Gaze correction for home video conferencing. ACM Trans-actions on Graphics (TOG), 31(6):174, 2012. 2

[11] W. Lotter, G. Kreiman, and D. Cox. Unsupervised learn-ing of visual structure using predictive generative networks.arXiv preprint arXiv:1511.06380, 2015. 4

[12] R. Penoyer. The alpha-beta filter. C Users Journal, 11(7):73–86, 1993. 6

[13] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolu-tional networks for biomedical image segmentation. In In-ternational Conference on Medical Image Computing and

Computer-Assisted Intervention, pages 234–241. Springer,2015. 4

[14] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang,and R. Webb. Learning from simulated and unsupervisedimages through adversarial training. In IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), pages2107–2116, 2017. 7

[15] Z. Shu, E. Shechtman, D. Samaras, and S. Hadap. Eye-opener: Editing eyes in the wild. ACM Transactions onGraphics (TOG), 36(1):1, 2017. 2

[16] B. A. Smith, Q. Yin, S. K. Feiner, and S. K. Nayar. Gazelocking: passive eye contact detection for human-object in-teraction. In ACM Symposium on User Interface Softwareand Technology, pages 271–280, 2013. 2, 6

[17] L. N. Smith. Cyclical learning rates for training neural net-works. In IEEE Winter Conference on Applications of Com-puter Vision (WACV), pages 464–472, 2017. 6

[18] Y. Sugano, M. Fritz, X. Andreas Bulling, et al. It’s writtenall over your face: Full-face appearance-based gaze estima-tion. In IEEE Conference on Computer Vision and PatternRecognition Workshops, pages 51–60, 2017. 5

[19] A. Torralba and A. A. Efros. Unbiased look at dataset bias.In IEEE Conference on Computer Vision and Pattern Recog-nition (CVPR), pages 1521–1528, 2011. 6

[20] E. Wood, T. Baltrusaitis, L.-P. Morency, P. Robinson, andA. Bulling. Learning an appearance-based gaze estimatorfrom one million synthesised images. In ACM Symposiumon Eye Tracking Research and Applications, pages 131–138,2016. 3

[21] E. Wood, T. Baltrusaitis, L.-P. Morency, P. Robinson, andA. Bulling. Gazedirector: Fully articulated eye gaze redi-rection in video. Computer Graphics Forum, 37:217–225,2018. 2

[22] R. Yang and Z. Zhang. Eye gaze correction with stereovi-sion for video-teleconferencing. In European Conference onComputer Vision (ECCV), pages 479–494. Springer, 2002. 1,2

[23] J. Zhu, R. Yang, and X. Xiang. Eye contact in video confer-ence via fusion of time-of-flight depth sensor and stereo. 3DResearch, 2(3):5, 2011. 2

[24] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial net-works. In IEEE International Conference on Computer Vi-sion, pages 2242–2251, 2017. 7