Research ArticlePerformance Analysis of Display Field Communication withAdvanced Receivers
Pankaj Singh ,1 Byung Wook Kim ,2 and Sung-Yoon Jung 1
1Department of Electronic Engineering, Yeungnam University, Gyeongsan, Gyeongbuk 38541, Republic of Korea2Department of Information and Communication Engineering, Changwon National University, Changwon,Gyeongsangnam 51140, Republic of Korea
Correspondence should be addressed to Sung-Yoon Jung; [email protected]
Received 20 August 2019; Revised 15 January 2020; Accepted 11 February 2020; Published 4 April 2020
Display field communication (DFC) is an imperceptible display-to-camera (D2C) communication approach that provides dual-mode, full-frame, visible light communication capabilities. Unlike conventional screen-to-camera communication approaches,DFC embeds data imperceptibly in the spectral domain of individual video frames. This paper analyzes the practicalperformance of the DFC scheme with advanced receivers, including zero forcing (ZF), minimum mean square error (MMSE),and maximum likelihood (ML). A 249 × 262 color image is used for embedding data consisting of eight individual informationvectors with their elements 2-QAM and 4-QAM modulated. The color image is separated into three individual channels, i.e., red(R), green (G), and blue (B). A lossy display-camera channel is considered in the presence of Gaussian noise, blooming, andvarious geometric distortions. Simulation results show that the ML receiver outperforms MMSE and ZF receivers. In addition,independent RGB data channels are evaluated to compare the symbol error rate of each channel. The proposed color DFCalgorithm can be a viable candidate for practical scenarios in applications like smart content transmission and for supportingrobust communication performance with advanced receivers, while the data embedded in the images remain unobtrusive to thehuman eye.
1. Introduction
Optical camera communication (OCC) [1–5] has beenrapidly emerging as a compelling technology for wirelesscommunications, generally consisting of light-emittingdiodes (LED) as a transmitter and a camera as a receiver.The widespread use of multiple cameras in handheld devices,and the explosion of related smart content-oriented displaytechnology, such as digital signage [6], gives OCC systemsextensive applications in information technology. Smartcontent data can be defined as content that is intelligibly per-sonalized to a specific person, delivering a targeted messagefor a personal experience. The smart content transmissionindustry is expected to be approximately $32 billion in thenext five years, having applications in the fields of education,hospitality, government, corporate environment, cinema,advertising, and many more [7]. Moreover, smartphone ortablet cameras are used, not only for capturing images but
also for communicating information. For example, the per-vasive quick response (QR) codes communicate a short codeto smartphone cameras, and recent research has exploredusing screen-to-camera communications for large data trans-fers. In such a scenario, display-to-camera (D2C) communi-cations will play an important role when the smart contentdata becomes more personalized, using concepts from artifi-cial intelligence, big data, and augmented reality.
The most popular D2C communication is QR codewhere information is encoded into a two-dimensional (2D)barcode [8–13]. In other words, QR codes are well-knowndigital advertising methods that send data to a camera fromdigital marks in print media. However, QR codes are typi-cally limited by their size and location. This limits theamount of information that can be encoded. On the otherhand, as the demand for communications through multime-dia services increases, interactive applications in the ubiqui-tous computing environments require a large amount of
HindawiWireless Communications and Mobile ComputingVolume 2020, Article ID 3657309, 14 pageshttps://doi.org/10.1155/2020/3657309
data to be transmitted. In the D2C communications environ-ment, an image on an electronic display (e.g., TVs, monitors,billboards, and projector screens) is a transmitter, and thecamera is a receiver [14–17]. In other words, a camera is usedas both an image sensor to capture an image and a commu-nication receiver to obtain the information transmitted fromthe display pixels. In addition, because an ordinary displaycan embed large amounts of data, it is possible for users toprovide a full-frame display while simultaneously transmit-ting data.
As mentioned above, display- (or screen-) to-cameracommunications is a technology where an LCD screen anda camera sensor can communicate via device-to-device com-munications [1, 18, 19]. First inspired by the traditional RFmodulation scheme, PIXNET [20] proposed encoding infor-mation in 2D spatial frequencies of an image. PIXNETincludes a perspective corrective algorithm, blur-adaptiveorthogonal frequency-division multiplexing (OFDM) cod-ing, and an ambient light filter. Another approach toscreen-camera communications was proposed in color bar-code streaming for smartphones (COBRA) [11]. COBRAwas designed to achieve one-way communications betweensmall-sized screens and the low-speed cameras in smart-phones using 2D color barcodes. The limited availablethroughput in screen-to-camera links was further enhancedby LightSync [12], which improves frame synchronizationbetween transmitter and receiver and can nearly double theachievable throughput. The creators of HiLight [21] intro-duced a new scheme for screen-camera communicationswithout any coded images. Leveraging the properties of theorthogonal transparency (alpha) channel, HiLight “hides”the bits by changing the pixel translucence instead of modify-ing the red-green-blue (RGB) color. Another mechanism forhigh-rate, flicker-free screen-camera communications wasproposed, which is similar to QR coding techniques, but thedata are embedded in the image spatially using a content-adaptive method [22]. The proposed scheme considers someblocks of the pixels in an image, embedding data as a texturein each pixel block (but not in the edges of the texture,because the human eye is very sensitive to changes in the tex-ture of an image). Two methods are applied to embed dataspatially in the texture of an image. The first is texton analy-sis, which is a machine learning technique to detect thedesired texture in an image. The other is pixel-based textureanalysis, which employs detection of the so-called goodregion (in which changes remain imperceptible to the humaneye) in an image to embed data.
These kinds of hidden display-camera communicationstechniques have emerged as a new paradigm that embedsdata imperceptibly into regular videos while remainingunobtrusive to human viewers. Various other studies havealso been conducted on how to embed data that can remainunobtrusive to the human eye in a displayed image [15–17,22–27]. In [15], a screen-to-camera communications systemwas proposed where data embedding is done on the relativebrightness of an image block. This was done by increasing(encode bit 1) or decreasing (encode bit 0) the brightness ofthe block of pixels. Moreover, the messages are embeddedinto selected video frames using watermarking that is not
perceptible to the human eye and is subsequently played ata high frame rate. However, the details of system imple-mentation and performance evaluation were not presented.Yuan et al. [16] presented a watermarking technique forembedding data into an image frame based on the Lapla-cian pyramid method. However, fundamental analysis ofthe error rate performance was not presented. Wanget al. [17] proposed a method called InFrame that uses acomplementary frame concept and embeds a data frameinto a pair of multiplexed video frames. Jo et al. [23] pre-sented DisCo, which enables displays and cameras to com-municate with each other while also displaying andcapturing images for human consumption. Messages weretransmitted by temporally modulating the display brightnessat high frequencies so they are imperceptible. Messages werereceived by a rolling shutter camera that converts the tempo-rally modulated incident light into a spatial flicker pattern.Zhang et al. [25] presented ChromaCode, which introduceda uniform color space for unobtrusive data embedding. Thebits are embedded into pixels using the most accurate colordifference formula, CIEDE2000, in a perceptually uniformcolor space, CIELAB. The authors also proposed a noveladaptive embedding scheme in an outcome-based philoso-phy, which accounts for both pixel lightness and frame tex-ture and ensures flicker invisibility over the full frame.Overall, most of the above methods for D2C communica-tions embed data into the spatial domain of an image (orvideo), which can directly affect image perception by thehuman eye. On the other hand, in the display field communi-cation (DFC) scheme [28, 29], the data are embedded in thespectral (or frequency) domain of an image, and they stillprovide dual-mode, full-frame, visible light communicationfunctionalities.
DFC embeds data in and extracts from the spectraldomain so that the properties associated with the frequencycoefficients of an image can be employed. In particular, thedata are embedded in the designated spectral subbands (SB)of an image. Kim et al. [28] was the first work on DFC wherethe practical performance of the DFC scheme was evaluatedin the presence of additive white Gaussian noise (AWGN)and various geometric distortions. The data are embeddedin a 256 × 256 grayscale image with 16-QAM modulationto achieve a maximum data rate of 9.5Kbps. However, thescheme considered data embedding in only one dimension(the width) of the frequency-domain image, resulting in1D-DFC. The same authors extended the concept of 1D-DFC to 2D-DFC [29] where the data are embedded in two-dimensions (width and height) of a grayscale image. It hasbeen shown that 2D-DFC achieves a higher data rate than1D-DFC, and hence is more appropriate for practical uses inthe D2C environment. However, it also uses a grayscale imageas input on the transmitter screen. The work by Kim and Jung[30] proposed color DFC, where the authors used threedifferent-colored RGB images as input on the screen. Eachindependent RGB data channel was evaluated, showing simi-lar performance for all three input images. In other words, itwas shown that, despite the different characteristics of theinput images, similar output results are observed in both gray-scale and color images, indicating that the type and channels
2 Wireless Communications and Mobile Computing
of the input image do not really affect the symbol error rate(SER) performance in screen communications [28, 30].
In this paper, we evaluate the concept of DFC for thecolor image (as input on the transmitter screen) in order toapply it to a more practical D2C environment. In addition,we mathematically evaluate and analyze the performance ofthe proposed method for different decoding schemes. Threeadvanced receivers, including zero forcing (ZF), minimummean square error (MMSE), and maximum likelihood(ML), are evaluated. Furthermore, the SER in a display-camera channel with various distortions and AWGN noiseis evaluated for all the receivers and for various other systemparameters, such as modulation order, subbands, and differ-ent RGB channels. The rest of this paper unfolds as follows.Section 2 describes the system model and the data embed-ding process in the proposed color DFC scheme. Section 3describes the display-camera channel and captures variousother channel distortions, such as blooming, that can occurduring display-camera communications. Section 4 explainsand analyzes the decoding process for all the receivers. Per-formance of the scheme is assessed and compared in Section5, and the study concludes in Section 6.
2. Color DFC Scheme
A DFC system is composed of a digital camera pointed at anelectronic screen (cf., Figure 1). On the transmitter side, theinput spatial-domain images are first converted to the fre-quency domain by applying discrete Fourier transform(DFT). Because the input image is in color, it should firstbe separated into individual RGB channels. Then, each chan-nel is converted separately to the frequency domain. At thesame time, the modulator modulates the binary input databy mapping bits to binary symbols. Both the data and theirHermitian symmetric equivalents are used in the dataembedding process to conserve the spatial property of thetransmitted image [28]. Each data channel image is then con-verted back to the spatial domain using inverse DFT, and allthe channels are combined to show (or transmit) the final
image on the screen. On the display device, the data-embedded image and the reference image are rendered alter-nately to minimize the image artifacts that may be visible tothe human eye [28].
At the receiver, the frames are received sequentially bythe camera, and the images are classified as data-embeddedand reference images. The combined spatial-domain imageis then separated into its three channels and converted tothe frequency domain. Finally, the data are decoded usingthe various advanced receivers.
2.1. Data Embedding. In the RGB image, the data can beembedded in each of the individual color channels. There-fore, the data rate is tripled in color DFC, compared to gray-scale DFC [28]. The frequency-domain image for a particularchannel can be calculated by taking the column-wise one-dimensional discrete Fourier transform (1D-DFT) of theimage:
IF,j = F · it1,j, F · it2,j,… , F · itQ ,jh i
= F · It,j, ð1Þ
where F is a P × P DFT matrix, It,j is a P ×Q spatial-domainimage, itq ,j is the column vector of It,j where q = 1, 2,⋯,Q,and j is a particular R, G, or B channel. In the frequency-domain image, each point represents a particular frequencycontained in the spatial-domain image. The result of 1D-DFT has low-frequency components on both sides of thefrequency-domain image, whereas the high-frequency com-ponents lie symmetrically in the central region.
Regarding data embedding, the data are first modulatedusing quadrature amplitude modulation (QAM); 2-QAMand 4-QAM were used in this study, as shown in the constel-lation diagram of Figure 2. By using further high-order mod-ulation, i.e., more points on the constellation, although it ispossible to transmit more bits per symbol; the points becomecloser together and are susceptible to noise. Therefore, wewill stick to only 2-QAM and 4-QAM in this study. Afterthat, the modulated data and their Hermitian symmetric
Inputdata Modulator
Hermitiansymmetry
Inputimage DFT Data
embedding IDFT
Electronicdisplay
Display-camerachannel
Camera
DFTData
retrievalDemodulatorOutputdata
d X
It(ref)
It I
FD
F
Dt
Yt
Zt
YF
ZF
d̂
Transmitter
Receiver
Figure 1: System architecture for DFC.
3Wireless Communications and Mobile Computing
equivalents are embedded in the frequency-domain image.The Hermitian symmetric equivalents of the data are embed-ded because, since the spatial-domain image has pixel valuesof real and positive numbers, the elements of 1D-DFT outputwill exhibit columnwise conjugate symmetry. Therefore, toembed data in the frequency domain and sustain the real-valued and positive properties for a data-embedded imagesimultaneously, the data sequence should also have conjugatesymmetric properties [28, 29]. Hence, the data matrix, X, canbe represented as
X j = X1,j, X2,j,… , XQ,j� �
, ð2Þ
with
Xq,j = 1|{z}1× s−1ð Þ
dq,j� �T 1|{z}
1×c
flip d∗q,j
� �� �T1|{z}
1× s−2ð Þ
264
375T
, ð3Þ
where dq,j = ½dq,jð1Þ,… , dq,jðLÞ�T is the data vector on the qth column of the data matrix, j ∈ fR,G, Bg, s is the startingpixel of the data symbol, and “flip” is an operation as definedin [28]. The starting pixel of the data symbol is given as
c = P − 2s − 2L + 3, ð4Þ
satisfying
s + L < P2
� + 2, ð5Þ
where L is the number of data symbols per column. In thisway, the data structure covers an L ×Q rectangular region,and another L ×Q conjugate symmetric region on thefrequency-domain image shown as white bands in Figure 3.These white regions in the frequency-domain image repre-sent the position of the frequency subbands.
The data embedding process is then carried out in the fre-quency domain using the multiplicative coefficients on thepixel values of an image. A data-embedded image, DF,j, inthe frequency domain is given as
DF,j = IF,j ∘ X j, ð6Þ
where ∘ is the Hadamard product operator. The above data-embedded frequency-domain image is then converted tothe spatial domain by taking the inverse DFT. Therefore,the data-embedded image in the spatial domain is repre-sented as
Dt,j = FH ·DF,j: ð7Þ
To be displayed on the screen, each channel of the colorimage has to be combined as follows:
Dt =〠j
Dt,j, ð8Þ
where j ∈ fR,G, Bg. The above data-embedded image isthen displayed on the electronic screen that is captured bythe camera. Note that the data-embedded image is placedbetween the neighboring reference images in the sequenceof image frames. This will achieve two important purposes.First, the reference image can be used for the decoding ofthe embedded data in the camera receiver. Second, it will helpan electronic display perform its original purpose, i.e., by ren-dering images at a high frame rate, the artifacts visible to thehuman eye can be greatly minimized.
As mentioned above, the frequency-domain image haslow-frequency components on both sides of the image,whereas the high-frequency components lie symmetricallyin the central region. For data embedding in the frequencydomain, we choose several frequency subbands, i.e., the coef-ficients of several groups of frequency bins. In Figure 3, wecan see that subband 3 (SB3) loads data in relatively lower
–1 –0.5 0 0.5 1In-phase
–1
–0.8
–0.6
–0.4
–0.2
0
0.2
0.4
0.6
0.8
1
Qua
drat
ure
Scatter plot
(a) 2-QAM
–1 –0.5 0 0.5 1In-phase
–1
–0.8
–0.6
–0.4
–0.2
0
0.2
0.4
0.6
0.8
1
Qua
drat
ure
Scatter plot
(b) 4-QAM
Figure 2: Modulated signal constellation.
4 Wireless Communications and Mobile Computing
frequency bands than subband 2 (SB2), and so on. The corre-sponding data-embedded images for individual RGB chan-nels are also shown. These individual RGB data-embeddedimages are combined to make final spatial-domain image tobe transmitted on the screen. Figure 4 represents the corre-sponding combined spatial-domain images. The effect ofloading data in various frequency bands can be observed intheir spatial-domain counterparts. We can see that fewerdetectable artifacts are introduced in SB1 and SB2, whilethe artifacts become high and easily visible in SB3. In partic-ular, we can observe fewer visible artifacts in Figure 4(c) andclear visible artifacts in Figure 4(d). Therefore, because thelow-frequency subbands contain the primary parts perceivedin image content by the human eye, mid- or high-frequencysubbands are preferred for embedding data.
Note that in the current DFC scenario, a smaller numberof data symbols per column (L) are chosen, which results in asmall vertical region for data embedding. However, L is a var-iable, and the value can be increased to embed more data.This may lead to an increased data rate, but a more distortedimage at the same time. Recall that the target of the proposedDFC scheme is embedding and extracting data from thespectral domain of an image while simultaneously lettingthe electronic display perform its original purpose. Hence,the proposed method uses the coefficients in a certain rangeof frequencies to embed the data in the Fourier domainimage in such a way that any artifacts introduced in the cor-responding spatial image would be invisible. Moreover, byusing reference images and rendering images at a high framerate, any artifacts still visible to the human eye can be highlyminimized.
3. Display-Camera Channel
3.1. Path Loss. In a DFC system, the pixels of the displayare the transmitter, and the camera capturing both the dis-play screen and the background is the receiver. We assumethat transmitter and receiver locations are fixed and thatthe channel characteristics are stationary in time. Thisassumption is realistic for situations in which the channelvaries slowly and can be tracked. Furthermore, it isassumed that the optical axes of the transmitter and thereceiver are aligned. When perfect alignment between thedata-transmitting screen and the camera are considered,all the light-emitting pixels from the screen are the focusof the camera. In many cases, this assumption is nearlytrue, or it can be corrected by spatial predistortion tech-niques. A commercial video-processing equipment existsthat removes projective distortion in the case of off-axisprojection. Display-camera communications involves non-equivalent attenuation between the brightness of differentpixels on the transmitter screen through the camera lensonto the image plane. Ideally, for any given aperture ofthe camera, the attenuation would be constant for everypart of the image. However, unavoidable matters of geo-metric optics result in image illuminance declining as wemove outward from the center of the frame. This phe-nomenon can be approximated by the “cosine fourth”law [31], which can be summarized as follows:
EΘ
E0= cos4Θ, ð9Þ
Red data embedded Green data embedded Blue data embedded
(a) Data embedding in the high-frequency band (subband 1)
Red data embedded Green data embedded Blue data embedded
(b) Data embedding in the mid-frequency band (subband 2)
Red data embedded Green data embedded Blue data embedded
(c) Data embedding in the low-frequency band (subband 3)
Figure 3: Data-embedded images in the frequency domain with their corresponding RGB components.
5Wireless Communications and Mobile Computing
where EΘ and E0 are the data signal energy on the pixel atthe point off-axis and point on-axis, respectively, and Θ isthe angle at which transmitted pixels are off-axis. Conse-quently, the received pixel intensity can be calculated as
~Zt i½ � = It refð Þ i½ � cos4θt,i,~Y t i½ � =Dt i½ � cos4θt,i,
ð10Þ
where i is the index for a given pixel and θi is the corre-sponding off-angle for the respective pixel.
3.2. Blooming. Another kind of distortion that can affectdisplay-camera communications is blooming. It occursdue to the charge leakage between neighboring pixels ina charge-coupled device (CCD) sensor. In particular, thecharge capacity of a CCD pixel is limited, and when apixel is full, the charge starts to leak into adjacent pixels.This process is known as blooming. The bloom effectmakes the received image look brighter and have a hazylook. In other words, blooming results in an image inwhich bright light appears to bleed beyond its natural bor-ders. Although under normal circumstances, this imperfec-tion is not noticeable, an intensely bright light could causethe imperfections to be visible. Blooming can be approxi-mated by the blurring effect followed by brightening theblurred image with reduced contrast. The spatial responseof an imaging system is described by its point-spreadfunction (PSF) [32]. The blur effect due to imperfect focushas a 2D Gaussian distribution in the spatial domain [32].Let h denote the PSF characterizing the linear and spa-tially invariant response of the imaging system. In this
case, the blurred pixels can be modeled as a convolutionof received pixels and the PSF [19]:
where ⊗ represents 2D linear convolution and ht½i� is theattenuation in the ith pixel of the corresponding images,captured by the PSF of the imaging system.
The effect of blooming distortion on an image can berecovered by using a Wiener filter [33]. In particular, theWiener filter algorithm deconvolves the PSF from the origi-nal received image, finally returning the deblurred image.Note that in addition to blurring, there is the presence ofnoise in an image. In the absence of noise, a Wiener filter isequivalent to an ideal inverse filter. Figure 5 shows the effectof blooming on the transmitted image and the correspondingrecovered image. As shown in the figure, blooming causes thereceived light energy to spread to areas outside the pixel. Theamount of spread depends on the type of lens used in thecamera. Specifically, blooming can be understood as a low-pass filtering phenomenon that distorts the high-frequencycomponents in the image. Note that blooming occurs onlyin smartphones having a CCD image sensor. Despite the factthat nearly all smartphones these days use complementarymetal-oxide semiconductor (CMOS) image sensors, a fewcamera phones still have a CCD sensor.
3.3. Noise. The signal quality in the camera receiver is alsoinfluenced by noise in the channel. Noise in camera systemsmanifests as noise current on each camera pixel and is
Reference image
(a)
Data embedded in subband 1
(b)
Data embedded in subband 2 Fewer visibleartifacts
(c)
Data embedded in subband 3 Visible artifacts
(d)
Figure 4: Spatial-domain images with data embedded in different frequency subbands.
6 Wireless Communications and Mobile Computing
generated due to the photons from environmental lighting.At the output of the camera, the noise current in eachcamera pixel is a quantized quantity and manifests as fluc-tuations in the intensity of that pixel. The noise energyaccumulated in each pixel can be quantified using themean value of the variances in pixel intensity. In thispaper, we consider noise in a camera pixel to be primarilyfrom the background and that it follows an AWGN char-acteristic [34, 35] and it is uniform over the image sensor,quantified through AWGN noise variance. Therefore, thecaptured image can then be represented as
Zt = €Zt +N t refð Þ,
Y t = €Y t +N t ,ð12Þ
where Zt is the received reference image, Y t is thereceived data-embedded image, and N t is the AWG noisematrix.
A camera is generally composed of an imaging lens, animage sensor (IS), and other image signal processing compo-nents. In order to obtain color information, a red, green, orblue filter normally covers the IS in a repeating pattern. Thispattern (or sequence) of filters can vary, but one could use awidely adopted Bayer color filter array, which is a repeating2 × 2 pattern, for digital acquisition of color images [36].Therefore, we are able to separate the RGB signal with a colorcamera. Consequently, the combined image is separated intoits individual channels as
Zt,j = €Zt,j +N t refð Þ,j,
Y t,j = €Yt,j +Nt,j,ð13Þ
where j ∈ fR,G, Bg. This will help decode each channel’s dataseparately. In addition, because the data are embedded usinga multiplicative property on the frequency-domain image,individual images are then transformed into their frequencydomain counterparts as follows:
ZF,j = F · Zt,j,Y F,j = F · Y t,j:
ð14Þ
3.4. Geometrical Distortion. Because of the nature of the cam-era imaging mechanism, the electronic screen may not befrontally aligned with the camera. This gives rise to geo-metric distortion as the screen pixels are captured at aperspective that results in shape distortion. Therefore, toobserve practical performance of the proposed colorDFC, geometric distortion based on the various visiontransformation parameters can be considered. The per-spective distortions in the DFC channel can be modeledas a composite effect of video quality reduction due toperspective scaling, rotating, and twisting of the pixel areasfrom the camera projection. The projection matrix [37]can be expressed as
P =ε cos ϕ −ε sin ϕ tε sin ϕ ε cos ϕ t
0 0 1
2664
3775, ð15Þ
where the scalar, ε, represents scaling factor, ϕ is the rotat-ing angle, and t is the twisting factor. The scale operatorperforms a geometric transformation which shrinks orzooms the size of an image.
Note that before data retrieval, the boundaries of theelectronic display should be accurately detected so that apredefined frequency range for data embedding can be iden-tified. For that, it is assumed that the electronic display isplanar. To recognize the borders of the display for precisealignment, Harris corner extraction and the Hough trans-form, which are widely used geometric correction methods,can be exploited [38]. To resize the distorted image to itsoriginal size, one can obtain the missing pixels by interpola-tion. Because the data are hidden in the intensity values ofthe spatial-domain image, a high degree of spatial resolutionresults in high accuracy in data detection. Figure 6 shows theperspective screen alignment for precise data detection whenrotating, scaling, and twisting distortions are considered.Furthermore, note that the camera should capture the entireimage area because the embedded data are spread over theentire spatial-domain image. Therefore, a large standoffdistance, which makes a camera acquire the whole imagearea, is required. The data are then decoded differently usingthe three advanced receivers described in the next section.
(a) Bloom image (b) Restored image
Figure 5: An image with bloom distortion and the corresponding restored image.
7Wireless Communications and Mobile Computing
4. Data Decoding
To retrieve data at the receiver, the ability to distinguishembedded data from the original data is required. For thisreason, a reference image is inserted between data-embedded frames in the image frame sequence.
4.1. Zero Forcing. In the zero-forcing receiver, we have to findthe inverse of the channel matrix. Considering the referenceframes’ pixel values are equivalent to the channel coefficients,the data can be decoded as
d̂q,j,ZF lð Þ = YF,j s + l − 1, qð ÞZF,j s + l − 1, qð Þ , ð16Þ
where l = 1, 2,⋯, L, and d̂q,j = ½d̂q,jð1Þ,… , d̂q,jðLÞ�T
is theestimated data column vector for the qth column and jthchannel of an image. It is clear from the above equation thata ZF receiver does not consider the noise effect. On the otherhand, the noise may get enhanced during the decodingprocess.
4.2. Minimum Mean Square Error. The MMSE receiver triesto minimize the mean square error between the transmittedsymbols and the detected symbols, and thus maximizes thesignal-to-noise ratio (SNR). Let WMMSE denote the MMSEdetector. The estimated data column vector, d̂q,j, is then com-puted as
d̂q,j,MMSE lð Þ =W q,j,MMSE lð ÞYF,j s + l − 1, qð Þ, ð17Þ
where
W q,j,MMSE lð Þ =E YF,j s + l − 1, qð Þd̂q,j lð Þn oE Y2
F,j s + l − 1, qð Þn o
=ZF,j s + l − 1, qð Þ
Z2F,j s + l − 1, qð Þ + σ2
N s + l − 1, qð Þ :ð18Þ
The final estimated symbol can then be expressed as
d̂q,j,MMSE lð Þ = ZF,j s + l − 1, qð ÞYF,j s + l − 1, qð ÞZ2F,j s + l − 1, qð Þ + σ2N s + l − 1, qð Þ , ð19Þ
where σ2N is the noise power in the received frame. We can
see from the above equation that the MMSE receiverattempts to reduce the noise at the receiver based on SNR.Therefore, at a high SNR, MMSE behaves as a ZF receiver.
4.3. Maximum Likelihood. Maximum likelihood finds theminimum distance between the received image frames andthe product of all possible transmitted image frames. Let Sand M denote the set of transmitted image signal constella-tion symbol points and the modulation order, respectively.Then, ML detection determines the estimated transmitteddata vector as
d̂q,j,ML lð Þ = mindq, j∈SM
Y F,j s + l − 1, qð Þ − ZF,j s + l − 1, qð Þ · dq,j lð Þ 2,
ð20Þ
where dq,jðlÞ is the transmitted symbol, YF is the receiveddata-embedded image, and ZF is the received referenceimage. Note that theML receiver achieves the optimal perfor-mance when the transmitted symbol generation probabilityis equal. This is because it minimizes the error probabilityat the receiver by comparing the received signal vector withall the possible combinations of transmitted signal vectorsto estimate the final symbol.
5. Simulation Results
This section presents the simulation results for the proposedcolor 1D-DFC scheme from comparing three advancedreceivers and evaluating various system parameters, such asRGB channels, subbands, and modulation order. The receiv-ing camera was assumed to be in front of the electronicscreen to ensure perfect line-of-sight communication. Thecolor image at 249 × 262 pixels (cf., Figure 4) was used asthe input image, i.e., the image on the electronic display.For the performance evaluation, two different modulationtechniques (2-QAM and 4-QAM) were exploited. Frame
(a) Distorted image (b) Reconstructed image
Figure 6: Reconstructed image according to the vision parameters of the perspective distortion.
8 Wireless Communications and Mobile Computing
synchronization in the D2C link was assumed, and a 30frame per second off-the-shelf camera for a receiver was con-sidered. The frame rate of the camera is assumed to be greaterthan the display frame rate. This is because the camerashould successfully capture the entire sequence of referenceimages to decode the transmitted data [28]. In addition, thenumber of embedded data symbols per image column, i.e.,L, was set to 20 vertical pixels. Moreover, the position ofthe subbands in the frequency-domain image was set by con-sidering the start pixel value, s, equal to 95 for SB1, 75 forSB2, and 45 for SB3. The default modulation in the simula-tion is 2-QAM unless otherwise specified.
5.1. SER for Different Receivers. The symbol error rate perfor-mance for all three receivers is compared in Figure 7. The
green data channel was evaluated for comparison, and allthe symbols were 2-QAM modulated. We can see that theZF receiver performs worse in all the subbands, and the MLreceiver is the best. This is because ZF is performed by divid-ing the received frame by the reference frame; it does not carefor the noise term and can amplify the noise in the process ofestimating the bits. In the MMSE receiver, the coefficientswere optimized and take care of the noise term amplificationwith the factor 1/SNR. Therefore, when the SNR becomeshigh, MMSE behaves like a ZF receiver. The ML receiveravoids the problem of noise enhancement, since it does notconsider equalization. Instead, it estimates the transmittedsymbol by choosing the minimum distance between thereceived image signal vector and all possible combinationsof reference image signal vectors. Note that ML is the optimal
SNR (dB)
10–2
10–1
100
SER
0 10 20 30 40 50 60 70
ZFMMSEML
(a) Subband 1
SNR (dB)
10–3
10–2
10–1
100
SER
0 10 2 30 40 50 60 70
ZFMMSEML
(b) Subband 2
0 10 20 30 40 50 60 70SNR (dB)
10–3
10–2
10–1
100
SER
ZFMMSEML
(c) Subband 3
Figure 7: SER of color DFC for different receivers over the green channel.
9Wireless Communications and Mobile Computing
receiver in our case, because the occurrence probability of allthe transmitted symbols is the same. Moreover, we can seethat SB1 and SB3 show the worst and best performances,respectively.
5.2. SER for Different Subbands. The various receivers’ per-formance according to the subbands is depicted in Figure 8.For this comparison, we evaluated the red data channel forall the receivers with 2-QAM modulated symbols. For low-SNR values (<15 dB), a similar SER performance can beobserved for all the receivers due to poor communicationslinks. However, when the SNR is increased, we can see thatthe SER performance of low-frequency bands, i.e., SB3, out-performs other bands for all receivers. This is because theenergy of the low-frequency band is higher than that ofmid- and high-frequency bands. As a result, data embedding
at a low frequency achieves better robustness against noise,compared to the other subbands.
5.3. SER for Different Modulation Order. Figure 9 presentsSER as a function of SNR for the different modulationschemes. Here, the blue channel was considered in subband1. Two modulation schemes were considered, i.e., 2-QAMand 4-QAM, having the modulation order (M) of 2 and 4,respectively. We can see that 2-QAM shows better perfor-mance than 4-QAM for all the receivers. This is because ifthe energy of the constellation plane remains the same, thepoints on the constellation plane must be closer together withan increasing modulation order (cf., Figure 2). Therefore, asmodulation order increases, data transmission becomesmore susceptible to noise. The effects of the distance betweenthe adjacent points in the constellation plane become
SNR (dB)
10–3
10–2
10–1
100
SER
0 10 20 30 40 50 60 70
SB1SB2SB3
(a) ZF receiver
SNR (dB)
10–2
10–1
100
SER
0 10 20 30 40 05 60
SB1SB2SB3
(b) MMSE receiver
0 10 20 30 40 50 60 70SNR (dB)
10–3
10–2
10–1
100
SER
SB1SB2SB3
(c) ML receiver
Figure 8: SER of color DFC for different subbands over the red channel.
10 Wireless Communications and Mobile Computing
significant as SNR increases. In addition, as the modulationorder increases, the ML receiver’s computational complexityincreases exponentially. In particular, MMSE is computa-tionally more complex than ML when the modulation is 2-QAM. However, if the modulation is 4-QAM or higher, MLbecomes more complex.
5.4. SER for Different Color Channels. Figure 10 depicts theperformance in subband 2 of different RGB channels forthe three receivers. We can see that the performance of allthe channels is similar, and there is a negligible difference.Therefore, we can say that the type of channel in the inputimage does not really affect SER performance in DFC. Onthe other hand, we can deduce that color DFC could providethree times the data rate compared to grayscale DFC, wherethere is one channel only.
5.5. Peak Signal-to-Noise Ratio. Table 1 shows PSNR valueswith regard to various subbands. We can see that as the
position of the start pixels of the subband increases, theimage quality of the data-embedded image is improved.Note that the visual characteristics of the image arelocated in the low frequencies, while the details and noiseare located in higher frequencies. Since SB1 and SB2occupy medium- to high-frequency regions, when usingSB1 and SB2 for data embedding, visual artifacts arehardly ever perceived.
6. Conclusions
This paper evaluates the performance of a colored DFCscheme with three advanced receivers, including zero forc-ing, minimum mean square error, and maximum likeli-hood. This approach utilizes an RGB image on anelectronic display as a transmitter, and a digital camerais the receiver. Because the RGB image is composed ofthree channels, the data can be embedded into each indi-vidual channel. In addition, we showed that data decoding
SNR (dB)
10–2
10–1
100
SER
0 10 20 30 40 50 60 70
M=4M=2
(a) ZF receiver
0 10 20 30 40 5 60 70SNR (dB)
10–2
10–1
100
SER
M=4M=2
(b) MMSE receiver
0 10 20 30 40 50 60 70SNR (dB)
10–2
10–1
100
SER
M=4M=2
(c) ML receiver
Figure 9: SER of color DFC for different receivers over the blue channel and subband 1.
11Wireless Communications and Mobile Computing
using the ML receiver achieves the best performance. Forthe application of smart content services, color DFC isan important step towards realizing the potential forrobust data communications while supporting the originalfunctionality of displaying image sequences without imageartifacts.
Data Availability
The simulation parameter data used to support the findingsof this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interestregarding the publication of this paper.
Acknowledgments
This work was supported by the National Research Founda-tion of Korea (NRF) grant funded by the Korean government(NRF-2018R1A2B6002204).
SNR (dB)
10–3
10–2
10–1
100
SER
0 10 20 30 40 50 60 70
RedBlueGreen
(a) ZF receiver
0 10 20 30 40 50 60 70SNR (dB)
10–3
10–2
10–1
100
SER
RedBlueGreen
(b) MMSE receiver
0 10 20 30 40 50 60 70SNR (dB)
10–3
10–2
10–1
100
SER
RedBlueGreen
(c) ML receiver
Figure 10: SER of color DFC for the different receivers over subband 2.
Table 1: PSNR performance of a data-embedded image for differentfrequency subbands.
Bands Values (2-QAM) Values (4-QAM)
High-frequency band (SB1) 43.96 dB 43.92 dB
Mid-frequency band (SB2) 40.62 dB 40.63 dB
Low-frequency band (SB3) 33.75 dB 33.70 dB
12 Wireless Communications and Mobile Computing
References
[1] N. T. Le, M. A. Hossain, and Y. M. Jang, “A survey of designand implementation for optical camera communication,” Sig-nal Processing: Image Communication, vol. 53, pp. 95–109,2017.
[2] N. Saha, N. T. Le, M. S. Ifthekhar, and Y. M. Jang, “Sur-vey on optical camera communications: challenges andopportunities,” IET Optoelectronics, vol. 9, no. 5, pp. 172–183, 2015.
[3] T. Yamazato, I. Takai, H. Okada et al., “Image-sensor-basedvisible light communication for automotive applications,”IEEE Communications Magazine, vol. 52, no. 7, pp. 88–97,2014.
[4] I. Takai, T. Harada, M. Andoh, K. Yasutomi, K. Kagawa, andS. Kawahito, “Optical vehicle-to-vehicle communication sys-tem using LED transmitter and camera receiver,” IEEE Pho-tonics Journal, vol. 6, no. 5, pp. 1–14, 2014.
[5] J. Hu, F. C. P. Shin, K. Y. Sam, and C. Yu, “LED-camera com-munication system with RGB coding,” in 2012 PhotonicsGlobal Conference (PGC), pp. 1–4, Singapore, Singapore,December 2012.
[6] J. Schaeffler, “What is digital signage?,” in Digital Signage,pp. 61–92, Focal Press, 2012.
[7] “Global digital signage market statistics, trends, size, 2018analysis and forecast to 2025,” January 2019, https://www.reuters.com/brandfeatures/venturecapital/article?id=34829.
[8] D. Wave, “Information technology-automatic identificationand data capture techniques-QR code bar code symbologyspecification,” in International Organization for Standardiza-tion, vol. 18004, ISO/IEC, 2015.
[9] K. Kamijo, N. Kamijo, and Z. Gang, “Invisible barcode withoptimized error correction,” in 2008 15th IEEE InternationalConference on Image Processing, pp. 2036–2039, San Diego,CA, USA, October 2008.
[10] A. Mohan, G. Woo, S. Hiura, Q. Smithwick, and R. Raskar,“Bokode: imperceptible visual tags for camera based interac-tion from a distance,” ACM Transactions on Graphics,vol. 28, no. 3, p. 98, 2009.
[11] T. Hao, R. Zhou, and G. Xing, “COBRA: color barcode stream-ing for smartphone systems,” inMobiSys '12: Proceedings of the10th international conference on Mobile systems, applications,and services, pp. 85–98, New York, NY, USA, June 2012.
[12] W. Hu, H. Gu, and Q. Pu, “LightSync: unsynchronized visualcommunication over screen-camera links,” in MobiCom '13:Proceedings of the 19th Annual International Conference onMobile Computing & Networking, pp. 15–26, New York, NY,USA, September 2013.
[13] A. Wang, S. Ma, C. Hu, J. Huai, C. Peng, and G. Shen,“Enhancing reliability to boost the throughput over screen-camera links,” inMobiCom '14: Proceedings of the 20th annualinternational conference on Mobile computing and networking,pp. 41–52, New York, NY, USA, September 2014.
[14] M. R. H. Mondal and J. Armstrong, “Analysis of the effect ofvignetting on MIMO optical wireless systems using spatialOFDM,” Journal of Lightwave Technology, vol. 32, no. 5,pp. 922–929, 2014.
[15] R. Carvalho, C.-H. Chu, and L.-J. Chen, “IVC: imperceptiblevideo communication,” in ACM, 2014.
[16] W. Yuan, K. Dana, A. Ashok, M. Gruteser, and N. Mandayam,“Dynamic and invisible messaging for visual MIMO,” in 2012
IEEE Workshop on the Applications of Computer Vision(WACV), Breckenridge, CO, USA, January 2012.
[17] A. Wang, C. Peng, O. Zhang, G. Shen, and B. Zeng, “InFrame:multiflexing full-frame visible communication channel forhumans and devices,” in Proceedings of the 13th ACM Work-shop on Hot Topics in Networks - HotNets-XIII, p. 23, NewYork, NY, USA, October 2014.
[18] P. H. Pathak, X. Feng, P. Hu, and P. Mohapatra, “Visible lightcommunication, networking, and sensing: a survey, potentialand challenges,” IEEE Communications Surveys Tutorials,vol. 17, no. 4, pp. 2047–2077, 2015.
[19] T. Nguyen, M. D. Thieu, and Y. M. Jang, “2D-OFDM for opti-cal camera communication: principle and implementation,”IEEE Access, vol. 7, pp. 29405–29424, 2019.
[20] S. D. Perli, N. Ahmed, and D. Katabi, “PixNet: interference-free wireless links using LCD-camera pairs,” in MobiCom'10: Proceedings of the sixteenth annual international confer-ence on Mobile computing and networking, pp. 137–148, NewYork, NY, USA, September 2010.
[21] T. Li, C. An, A. Campbell, and X. Zhou, “Hilight: hiding bits inpixel translucency changes,” in VLCS '14: Proceedings of the 1stACM MobiCom workshop on Visible light communication sys-tems, pp. 45–50, New York, NY, USA, September 2014.
[22] V. Nguyen, Y. Tang, A. Ashok et al., “High-rate flicker-freescreen-camera communication with spatially adaptive embed-ding,” in IEEE INFOCOM 2016 - The 35th Annual IEEE Inter-national Conference on Computer Communications, pp. 1–9,San Francisco, CA, USA, April 2016.
[23] K. Jo, M. Gupta, and S. K. Nayar, “DisCo: display-camera com-munication using rolling shutter sensors,” ACM Transactionson Graphics, vol. 35, no. 5, p. 150, 2016.
[24] T. Li, C. An, X. Xiao, A. T. Campbell, and X. Zhou, “Real-timescreen-camera communication behind any scene,” inMobiSys'15: Proceedings of the 13th Annual International Conferenceon Mobile Systems, Applications, and Services, pp. 197–211,New York, NY, USA, May 2015.
[25] K. Zhang, Y. Zhao, C. Wu et al., “ChromaCode: a fully imper-ceptible screen-camera communication system,” IEEE Trans-actions on Mobile Computing, p. 1, 2019.
[26] S. Shi, L. Chen, W. Hu, and M. Gruteser, “Reading betweenlines: high-rate, non-intrusive visual codes within regularvideos via implicitcode,” in UbiComp '15: Proceedings of the2015 ACM International Joint Conference on Pervasive andUbiquitous Computing, pp. 157–168, New York, NY, USA,September 2015.
[27] A. Wang, Z. Li, C. Peng, G. Shen, G. Fang, and B. Zeng,“Inframe++: achieve simultaneous screen-human viewingand hidden screen camera communication,” in MobiSys '15:Proceedings of the 13th Annual International Conference onMobile Systems, Applications, and Services, pp. 181–195, NewYork, NY, USA, May 2015.
[28] B. W. Kim, H.-C. Kim, and S.-Y. Jung, “Display field commu-nication: fundamental design and performance analysis,” Jour-nal of Lightwave Technology, vol. 33, no. 24, pp. 5269–5277,2015.
[29] S.-Y. Jung, H.-C. Kim, and B. W. Kim, “Implementation oftwo-dimensional display field communications for enhancingthe achievable data rate in smart-contents transmission,” Dis-plays, vol. 55, pp. 31–37, 2018.
[30] S.-A. Kim and S.-Y. Jung, “Screen communication schemebased on display,” The Journal of Korean Institute of
Communications and Information Sciences, vol. 44, no. 4,pp. 670–673, 2019.
[31] D. A. Kerr, Derivation of the Cosine Fourth Law for Falloff ofIlluminance Across a Camera Image, Technical Report 4, 2007.
[32] S. Hranilovic and F. R. Kschischang, “A pixelated MIMOwire-less optical communication system,” IEEE Journal of SelectedTopics in Quantum Electronics, vol. 12, no. 4, pp. 859–874,2006.
[33] E. R. Dougherty, Random Processes for Image and SignalProcessing, SPIE Optical Engineering Press, 1999.
[34] T. Komine and M. Nakagawa, “Fundamental analysis forvisible-light communication system using LED lights,” IEEETransactions on Consumer Electronics, vol. 50, no. 1,pp. 100–107, 2004.
[35] A. P. Tang, J. M. Kahn, and K.-P. Ho, “Wireless infrared com-munication links using multi-beam transmitters and imagingreceivers,” in Proceedings of ICC/SUPERCOMM '96 - Interna-tional Conference on Communications, pp. 180–186, Dallas,TX, USA, June 1996.
[36] P. Cheremkhin, V. Lesnichii, and N. Petrov, “Use of spectralcharacteristics of DSLR cameras with Bayer filter sensors,”Journal of Physics: Conference Series, vol. 536, no. 1, article012021, 2014.
[37] R. Hartley and A. Zisserman,Multiple View Geometry in Com-puter Vision, Cambridge University Press, 2003.
[38] W. Burger and M. J. Burge, Principles of Digital Image Process-ing: Fundamental Techniques, Springer Science & BusinessMedia, 2010.
