7/31/2019 Smart Cameras1 (1)

1/25

INTRODUCTION

What is a smart camera?

Different researchers and camera manufacturers offer

different definitions. There does not seem to be a well-established and agreed-upon definition in either the videosurveillance or machine vision industries, probably thetwo most active and advanced applications for smart

cameras at present.

7/31/2019 Smart Cameras1 (1)

2/25

A smart camera combines video sensing, video processing

and communication within a single device.

The idea of smart cameras is to convert data to knowledge by

processing information where it becomes available, and transmitonly results that are at a higher level of abstraction.

A smart camera is smart because it performs applicationspecific information processing (ASIP), the goal of which tounderstand and describe what is happening in the images for thepurpose of better decision-making in an automated controlsystem.

7/31/2019 Smart Cameras1 (1)

3/25

Recent technological advances are enabling a new generation ofsmart cameras that represent a quantum leap in sophistication.While today digital cameras capture images, a smart cameracapture high level descriptions of the scene and analyze what

they see.

Matrox Iris GT smart camera

Smart cameras not only capture images, they further performhigh-level image processing on-board, and transfer the data via

network.

7/31/2019 Smart Cameras1 (1)

4/25

Due to their logarithmic behavior, high dynamic range and highbit resolution the low-cost and low-power CMOS sensors acquireimages with the necessary quality for further image processingunder varying illumination conditions. The integration of these

advanced image sensors with high-performance processors intoan embedded system facilitates new applications such as motionanalysis and face recognition on-board and to transmit the(compressed) video data as well as the extracted video

information via a network.

NI 1742 Smart Camera

7/31/2019 Smart Cameras1 (1)

5/25

[1] CMOS image sensors can overcome problems like large intensitycontrast due to weather conditions or road lights and further blooming,which is an inherent weakness of existing CCD image sensors.Furthermore, noise in the video data is reduced by the capability of

video computation close to the CMOS sensor. Thus, the smart cameradelivers a new video quality and better video analysis results, if it iscompared to existing solutions. Beside these qualitative arguments andfrom a system architecture point of view, the smart camera is animportant concept in future digital and heterogeneous third generation

visual surveillance systems.

[2]. Not only image enhancement and image compression but also videocomputing algorithms for scene analysis and behavior understandingare becoming increasingly important. These algorithms have a high

demand for real-time performance and memory. Fortunately, smartcameras can support these demand as low-power, low-cost embeddedsystems with sufficient computing performance and memory capacity.Furthermore, they offer flexible video transmission and computing inscalable networks with thousands of cameras through a fully digital

interface.

7/31/2019 Smart Cameras1 (1)

6/25

Block Diagram

7/31/2019 Smart Cameras1 (1)

7/25

A smart camera usually consists of several (but not necessarilyall) of the following components:

Image sensor (matrix or linear)Image digitization circuitryImage memoryProcessor (often a DSP or suitably powerful processor)

program- and data memory (RAM, nonvolatile FLASH)Communication interface (RS232, Ethernet) I/O lines (often optoisolated)Lens holder or built in lens (usually C, CS or M-mount)

Built in illumination device (usually LED)Purpose developed real-time operating system (For exampleVCRT)

http://en.wikipedia.org/wiki/Digitizehttp://en.wikipedia.org/wiki/Central_processing_unithttp://en.wikipedia.org/wiki/Digital_signal_processorhttp://en.wikipedia.org/wiki/RS232http://en.wikipedia.org/wiki/Ethernethttp://en.wikipedia.org/wiki/Input/outputhttp://en.wikipedia.org/wiki/LEDhttp://en.wikipedia.org/wiki/LEDhttp://en.wikipedia.org/wiki/Input/outputhttp://en.wikipedia.org/wiki/Ethernethttp://en.wikipedia.org/wiki/RS232http://en.wikipedia.org/wiki/Digital_signal_processorhttp://en.wikipedia.org/wiki/Central_processing_unithttp://en.wikipedia.org/wiki/Digitize

7/31/2019 Smart Cameras1 (1)

8/25

Classification Of Vision SystemsAnd Smart Cameras

7/31/2019 Smart Cameras1 (1)

9/25

Architecture of the SmartCamera

7/31/2019 Smart Cameras1 (1)

10/25

The smart camera is divided into three majorparts:

1. Video Sensor

The video sensor represents the first stage in the smart camerasoverall data flow. The sensor captures incoming light and transforms itinto electrical signals that can be transferred to the processing unit.

2. Processing Unit

The second stage in the overall data flow is the processing unit. Dueto the high-performance on-board image and video processing therequirements on the computing performance are very high. A roughestimation results in 10 GIPS computing performance.

3. Communication Unit

The final stage of the overall data flow in our smart camera representsthe communication unit. The processing unit transfers the data to theprocessing unit via a generic interface. This interface eases theimplementation of the different network connections such as Ethernet,wireless LAN and GSM/GPRS. For the Ethernet network interface onlythe physical-layer has to be added because the media-access

control layer is already implemented on the DSP.

7/31/2019 Smart Cameras1 (1)

11/25

Dynamic Power Management

The basic idea behind DPM is that individual components can be

switched to different power modes during runtime. Each power

mode is characterized by a different functionality/performance of

the component and the corresponding power consumption. For

instance, if a specific component is not used during a certain time

period it can be switched off.

The commands to change the components power modes are

issued by a central PowerManager (PM). The commands are

issued corresponding a Power Managing Policy (PMP).

The PMP is usually implemented in the operating system of the

main processing compo-nent.

7/31/2019 Smart Cameras1 (1)

12/25

Embedded Processors

There are generally four main families of embedded processors that canbe used for smart cameras:-

Microcontrollers - Microcontrollers are cheap but have limited processingpower and are generally not suited for building demanding smart cameras.

ASICs (Application Specific Integration Circuits) - ASICs are powerful

and power-efficient processors, but the design cost and risk are high and theyare viable solutions only when volume is high and time-to-market is well-timed.

DSPs (Digital Signal Processors) - DSPs are relatively cheap andpowerful in performing image and video processing. They typically have a high-

end DSP core employing SIMD (Single Instruction Multiple Data) and VLSIarchitectures.

PLDs (Programmable Logic Devices) such as the FPGA. One of the mostimportant advantages of the FPGA is the ability to exploit the inherently parallel

nature of many vision algorithms.

7/31/2019 Smart Cameras1 (1)

13/25

An automated face recognition system forintelligence surveillance: Smart camera recognizingfaces in the crowd.

Stand-alone Smart Cameras: CCTV Applications

Research in Computer Vision and PatternRecognition Industry Machine Vision ITS (Intelligent Transport Systems) Automobiles

HCI(Human Computer Interface) Medical/Healthcare Video Conferencing Biometrics

Applications

7/31/2019 Smart Cameras1 (1)

14/25

Industry Machine Vision

Most machine vision cameras are stand-alone and autonomoussmart cameras, where communications with PC or other central

control unit is only needed for camera configuration, firmwareupgrading or in some cases output data collection. Mostalgorithms implemented in these cameras follow the similarprocessing flow described in the figure below :-

7/31/2019 Smart Cameras1 (1)

15/25

Intelligent Transport Systems and Automobiles

Generally speaking, the application and algorithmic requirements for ITS

are quite similar to those of IVSS. These requirements are different forautomobile applications, however, where high-speed imaging andprocessing are often needed. Increased robustness is also required forcar-mounted cameras to deal with varying weather conditions, speeds,road conditions, car vibrations. CMOS image sensors can overcome

problems like large intensity contrasts due to weather conditions or roadlights and further blooming, which is an inherent weakness of existingCCD image sensors.

The VIEWS system at the University of Reading is a 3D

model-based vehicle tracking system, which is capable ofdetecting potential accident situations and is designed forexisting camera setups on road networks.

7/31/2019 Smart Cameras1 (1)

16/25

Automobile Applications

Smart camera-powered intelligent vehicles will have the comprehensivecapability of monitoring the vehicle environment including the drivers

state and attention inside of the vehicle as well as detecting roads andobstacles outside the vehicle, so as to provide assistance to drivers andavoid accidents in emergencies.However, building and integrating smart cameras into vehicles is notan easy task:

On one hand the algorithms require considerable computing powerto work reliably in real-time and under a wide range of lightingconditions.On the other hand, the cost must be kept low, the package sizemust be small and the power consumption must be low.

Applications of smart cameras in intelligent vehicles include lanedeparture detection, cruise control, parking assistance, blind-spotwarning, driver fatigue detection, occupant classification andidentification, obstacle and pedestrian detection, intersection-collisionwarning, overtaking vehicle detection.

7/31/2019 Smart Cameras1 (1)

17/25

Key Issues or Challenges

1) System Design

The proprietary nature of smart cameras can limitchoices of hardware, like imagers, I/O, lighting, lens and thecommunications format. This may lead to a lack of expandability andflexibility of PC-based systems. In terms of design methodology, theeasy integration of intellectual property in the design tool and flow

can help foster product differentiation. Other important system-levelissues include smart camera operating systems, development tools.

2) CMOS Image Sensors Dynamic range is still one of the keyaspects where CMOS image sensors lag behind CCD. Improvementin this area can lead to more low-cost smart cameras using CMOSimage sensors for machine vision and surveillance applications.

7/31/2019 Smart Cameras1 (1)

18/25

3) Algorithm Development Many intelligent pattern recognitionalgorithms work well in laboratory conditions but fail when deployedand implemented in real-world conditions (occlusion, lightingcondition changes, unfavourable weather conditions), andembedded system environments (scant resources, low power, lowcost). Robustness and low complexity are among key issues facingresearchers developing algorithms for smart cameras in

surveillance, ITS and automobile applications.

4) Performance Evaluation - This is a very significant challenge insmart surveillance systems. Evaluating the performance of video

analysis systems requires significant amounts of annotated data.Typically, annotation is a very expensive and tedious process.Additionally, there can be significant errors in annotation. All of theseissues make performance evaluation a significant challenge.

7/31/2019 Smart Cameras1 (1)

19/25

5) Standards Development There is need for the development ofsome smart camera standards. In fact, the European Machine VisionAssociation (EMVA) has recently launched an initiative (EMVA 1288Standard) to define a unified method to measure, compute andpresent specification parameters for smart cameras and imagesensors used for machine vision applications. More needs to bedone in this respect.

6) Single Chip Smart Cameras Single-chip smart cameras are anattractive concept, but the manufacturing cost for the single-chipsmart cameras can be high because the feature size for makingdigital processors and memory is often different from the one usedto make image sensors, which may require relatively large pixels to

efficiently collect light. Therefore, it probably still makes sense todesign the smart camera in a multi-chip approach with a separateimage sensor chip. Separating the sensor and the processor alsomakes sense at the architectural level, given the well-understoodand simple interface between the sensor and the computation

engine.

7/31/2019 Smart Cameras1 (1)

20/25

Future Scope of Smart CamerasThe demand for smart cameras will steadily increase in traditional

industries such as surveillance and industry machine vision, and mayalso come from new industry and market segments such ashealthcare, entertainment, education and so on.Based on the discussions above, we can discern the following futuredirections for smart camera system and technologies.

At the system design level, continuous effort will be made in thedevelopment of a research strategy or design methodology for smartcameras as embedded systems.

At the ASIP algorithm development level, in order to improveperformance and robustness of existing techniques, research shouldaddress issues such as occlusion handling, fusion of 2D and 3Dtracking, anomaly detection and behavior prediction, combination ofvideo surveillance and biometrical personal identification, multi-sensory

data fusion .

7/31/2019 Smart Cameras1 (1)

21/25

Multi-modal, multi-sensory augmented video surveillance systemshave the potential to provide improved performance and robustness.Such systems should be adaptable enough to adjust automatically andcope with changes in the environment like lighting, scene geometry orscene activity.

Work on distributed (or networked) IVSS should not be limited to the

territory of computer vision laboratories, but should involvetelecommunication companies and network service providers, andshould take into account system engineering issues.

Standards development. One area which may need standardization isthe metadata format that facilitates integration and communicationbetween different cameras, sensors and modules in a distributed andaugmented video surveillance system. New communication protocolsmay be needed for better communication between different smart

camera products.

7/31/2019 Smart Cameras1 (1)

22/25

New product developments will introduce smart camera-based digitalimaging systems into existing consumer and industry products, to

increase their value and create new products.

In the machine vision arena, smart cameras will offer more and morefunctionality. The trend of distributing machine vision across the entireproduction line at points before value is added will continue. Neuralnetwork techniques seem to have become a key paradigm in machine

vision that are used either to correctly segment an image in a widevariety of operational conditions or to classify the detected object.Stereo and 3D-vision applications are also increasingly widespread.Another trend is to utilize machine vision in the non-visible spectrum.

BOA Smart Camera: Small, flexible withvision system

7/31/2019 Smart Cameras1 (1)

23/25

As the price of cameras and computing elements continue to fall itbecomes increasingly feasible to consider the deployment of smartcamera networks. Such networks would be composed of small,

networked computers equipped with inexpensive image sensors.

Consider, the proliferation of camera equipped cell phones. Suchcamera networks could be used to support a wide variety ofapplications including environmental modeling, 3D model construction

and surveillance.

7/31/2019 Smart Cameras1 (1)

24/25

A number of research efforts at a variety of institutions are currentlydirected towards realizing aspects of this vision. One criticalproblem that must be addressed in such systems is the issue oflocalization. That is, in order to take full advantage of the imagesgathered from multiple vantage points it is helpful to know wherethe cameras are located with respect to each other.

In an advanced system each of the smart cameras is equipped witha co-located controllable light source which it can use to signalother smart cameras in the vicinity. By analyzing the images that itacquires over time, each smart camera is able to locate and

identify other smart cameras in the scene. This arrangementmakes it possible to directly determine the epipolar geometry of thecamera system from image measurements and, hence, recover therelative positions and orientations of the smart camera nodes.

7/31/2019 Smart Cameras1 (1)

25/25

ConclusionA smart camera realized as an embedded system has been presented in

this paper. Our smart camera integrates a digital CMOS image sensor, a

processing unit featuring two high-performance DSPs and a network

interface. High-level video analysis algorithms in combination with state-

of-the-art video compression transform this system from a network

camera into a smart camera.There is a rapidly increasing market for smart cameras. Advances in

performance and integration will enable new and more functionalityimplemented in smart cameras. The next steps in development of smartcameras include

(i) the development of the target architecture,(ii) the implementation of further image processing algorithms,

(iii) the real-world evaluation of objects.