Technology has played a vital role in shaping the lives we are living today. Technology has improved the life’s quality, efficiency and effectiveness. It has affected the way socio-economic events are conducted in today’s world. Technology is now being used to transform cities, the center of man’s activities, to “Smart Cities”.
The term “Smart Cities” is often explained as “Interconnected, Instrumented and Intelligent” centres [1].This basically refers to the use and application of IT in urban environments to conduct the activities in a better and smarter way. It involves using of various devices and sensors integrated in a way that allows communication of information across cities’
different locations and services. Advanced analytics and modelling is then applied on this information to make the process work in a more productive manner.
Recently the term “Smart City” has been used widely in Singapore’s context. Singapore is striving to become world’s first smart nation where government and corporations are coming together to implement smart solutions across different sectors [2]. Singapore has limited resources and space. Thus sustainability is the best way forward for the country to progress and make a mark in the future.
Indoor mapping has also recently gained attraction. Prime Minister Lee Hsien Loong announced the Smart Nation Initiative in 2014. This has led to an increase in focus on developing effective 3D mapping techniques for urban solutions of the 21st century. In May 2017, Smart Nation and Digital Government Office will be set up under the Prime Minister’s office to take this initiative forward and it will be responsible for the city’s digital transformation [3].
Digital 3D maps are increasingly used for diverse tasks in our daily lives. Until recently the entertainment industry has been the leading market for virtual environments, however many other applications such as autonomous robots, military, training simulations, etc. have recently gained attention. Exploration of various locations for varied reasons like engineering design, leisure, simulations, dealing with hazardous situations, etc. could be effective if an accurate virtual 3D map could represent its physical counterpart. Currently, most of this technology has been in use for outdoor use, however various indoor applications for this technology are highly desirable [4]. Few potential applications of an indoor 3D mapping technology include tracking construction performance, facility management, security in case of emergencies and retail (in the form of innovative virtual shopping environments).
II. OBJECTIVE
The objective of the study is to develop an indoor mapping system for data collection in a building environment. Affordable and quicker alternatives to indoor 3D mapping by using new sensors and technologies are explored.
Xbox Kinect Sensor and StereoLab’s ZED Camera are tested in particular for this study. Comparisons between the
2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan
technologies will be made based on various factors like accuracy, resolution, speed, memory and lighting conditions.
Focus will be on the data collection process which is vital to realize this technology towards building a “Smart City”. This study will be discussing the possibility of extending the indoor 3D mapping technology to track activities in a cost effective and fast manner. Use of mapping tools with mobile platforms will be explored. The project will be using readily available equipment and software packages. These solutions will then be compared with the existing conventional solutions. User experiences play a vital role in implementing a new technology and will thus be documented along with the future scope and recommendations for the project.
III. BACKGROUND
A number of 3D mapping solutions are currently available in the market. FARO, Leica and Matterport are few of the popular ones. These solutions are however expensive and thus, limited to only a few people. These systems are also bulky and require technical know how to operate.
There has been various studies done to find alternative solutions in the area of indoor mapping. These mostly include research groups on robotics and computer vision [5]. RGB-D, laser scanners and stereo cameras have been used for such projects. With the release of Kinect sensor, various experimentations has surfaced using this RGB-D camera [6] [7].
Stereo vision cameras could be used to generate indoor 3D maps [8]. However little has been done to explore the possibilities of using ZED camera for indoor mapping. Additionally, none of the previous studies have compared the applicability of projected light based (like Kinect) and stereo vision based (like ZED camera) for different applications. This study will be discussing the effectiveness of the Kinect and ZED camera and comparing them in detail.
This study uses the Kinect sensor v2.0 and ZED camera as hardware devices. Kinect is a motion detection camera developed by Microsoft for Xbox. It is equipped with depth sensing technology, infrared emitter and a color camera [9]. In order to calculate depth, IR projector emits a pattern which is processed by the IR receiver. The pattern for a particular depth is then generated. With the help of triangulation method, the depth is calculated from the difference in the IR patterns [10].
ZED is a 2k Stereo camera which is used for Depth Sensing and Motion Tracking [11]. The camera uses an advanced sensing technology based on the principle of human stereo vision and can be used for depth perception, positional tracking and 3D mapping applications. It uses real time depth based visual odometry and SLAM technology. Stereo vision cameras work on the same principle as our brain works on measuring distance using our eyes. In a stereo camera, 2 cameras are used which are generally placed a short distance apart in the same plane. 3D spatial relationships like the tilt, separation and placement are known for both the cameras. 3D stereovision algorithms are then applied on the 2 images obtained by the cameras which aligns the pixels and calculates the depth information [12]. A depth map is then visualized from the available information.
The ZED camera can be connected to a computer via USB 2.0 without requiring an additional power source. The computer however needs to have a NVIDIA GPU and a minimum memory of 2GB.
This study uses the standard software packages Microsoft’s Kinect Fusion and Stereolab’s ZED Depth viewer for the experiment. In addition to them, Autodesk’s ReCap 360 Pro and open source software CloudCompare and RTAB-MAP are used for processing.
IV. EXPERIMENT AND ANALYSIS
To assess the suitability of the devices for commercial purposes, comparison is required to be made to choose the more appropriate device for different conditions.
In the following sections, the results of indoor mapping from the two devices: ZED Stereo Camera and the Xbox Kinect Sensor are compared. Comparison has been made in terms of accuracy, speed, memory usage, focus range and lighting conditions.
A. Measurement Accuracy
Both the device generate models which are visually appealing. However in order to assess the models in greater detail, the accuracy of the linear dimensions are tested. To test the accuracy of the models generated by the ZED camera and Kinect, nine points in the indoor 3D space were visually chosen and compared with those in the models. The distances between the points in the 3D models were calculated using the point picking feature in the CloudCompare software. The measurement and the error values can be seen in the figures and table as below:
Fig. 1. Actual Indoor 3D Space
Fig. 2. Kinect Model Measurements
621 mm 1901 mm
441 mm 2154 mm
537 mm
2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan
Fig. 3. ZED Camera Model Measurements
TABLE 1. Actual and 3D models distance measurements
Object Real
(mm)
Kinect
(mm)
Error
( %)
ZED
(mm)
Error
(%)
Door Height
2125 2154 1.36 2620 23.29
Cupboard Width
500 537 7.40 588 17.60
Cupboard Height
1850 1901 2.76 2209 19.4
Drawer Width
465 441 -5.16 469 0.86
Hanger Width
610 621 1.80 660 8.20
The mean error can now be calculated from the values presented in Table 1.
It can be seen that:
Mean Error for Kinect= 3.70%
Mean Error for ZED= 13.87%
As evident from table 1, Xbox Kinect is comparatively more accurate in generating 3D models when compared to ZED camera. The Kinect is relatively more accurate for longer distances on flat surfaces as compared to shorter distances, while it is vice versa for the ZED camera. It can also be noticed that nearby objects of similar color but at varying depths tend to merge at certain points for both devices as evident from the 3D models.
B. Speed of Conversion
The speed of capturing the indoor environment and generating a 3D model plays a vital role when analyzing such a technology for commercial purposes. A lot of factors affect the speed of system. Most of these factors are based on the computing machine used for the analysis. For the sake of simplicity, this study compares the performance of the Kinect and the ZED camera when used on the same computing machine to record an indoor room. All the background applications were closed on the laptop system apart from the required software so as to provide the maximum available memory.
Note: The time recordings presented in this section are only for capturing a particular frame of point cloud data. Comparisons are then made for different resolutions.
The time for 3D mapping for the whole indoor space depends on the speed of motion, quality of scan and positioning of the sensor. These factors have to be optimized in order to generate an accurate scan.
i. Data Collection
The following tables records the timing for Kinect Sensor and ZED Camera to store each point cloud as a PLY file:
TABLE 2. Timing for the Kinect (left) and the ZED (right)
For both the devices, the record time increases as the resolution is increased. The ZED camera is however faster in recording high resolution depth maps than the Kinect. This can be observed from the Chart 1 as shown below.
Chart 1. Time vs Resolution charts for the Kinect and the ZED camera
Low Lighting Conditions: It was also observed that both the devices took 1.4x more time to record the point cloud when used in low lighting conditions.
ii. Data Processing
Once the data is stored as PLY files on a hard disk, it is uploaded on software like CloudCompare and ReCap 360 for processing. The timing of uploading varies widely for both the devices. This is dependent on various factors like available RAM on the computer at the time of uploading, number background applications running, processor speed as well as the quality of the mesh amongst other factors. It was however noticed that ZED Stereo camera’s files on an average took more time to upload successfully on the system when compared to the Xbox Kinect Sensor under similar conditions.
C. Memory Usage
The amount of memory consumed by each device becomes crucial when we are looking of using it with a mobile device. The more memory the system consumes, the costlier the application becomes.
660 mm
588 mm
2620 mm
2209 mm
469 mm
2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan
Note: The memory recordings presented are only for capturing a particular frame of point cloud data. Comparisons are then made for different resolutions for one frame only.
The memory required for 3D mapping for the whole indoor space depends quality of scan and the area to be captured.
The following table records the amount of memory space acquired on the disk for the Kinect and the ZED Camera to store each point cloud as a PLY file:
TABLE 3. Memory Consumption by Kinect (left) and ZED (right)
As the resolution increases, the memory space required on the disk increases. Both the devices require nearly similar amounts of memory space to operate with the ZED optimizing the memory required for high resolution operations. The comparison can be seen in the Chart 2 as shown below.
Chart 2. Memory vs Resolution charts for the Kinect and the ZED camera
D. Focus Range
In order to compare the devices, 3D models were generated at different locations of varied space.
i. Mid-Sized Room (within 4.5m)
Both the Kinect and the ZED camera were able to furnish satisfactory 3D models when used within 4.5m range. The ZED camera was however able to record more details than the Xbox Kinect. The details too close to the sensor were not recorded in accordance to the sensor specifications.
Fig.4. Actual Space- Mid Sized Room
Fig. 5. Kinect Model (left) and ZED Model (right)
ii. Large Space
a. Corridor - Within 20m
When used in a corridor, the ZED camera gave better depth point cloud as compared to the Kinect. However it was not able to register some details which were well within the sensor’s range.
Fig. 6. Corridor a) Actual b) Kinect model c) ZED
b. Hall- More than 20m
The Kinect sensor was able to capture details within its sensor’s range (4.5m). However ZED camera delivered a blank 3D model of the entire space.
Fig. 7. Actual Space- Hall
Fig. 8. a) Kinect Model- Hall b) ZED Camera model- Hall
E. Lighting Conditions
i. Low Light
Both the ZED and the Kinect were able to produce 3D models in low lighting conditions. The ZED camera however captured more details than the Kinect.
2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan
Fig. 9. Actual Space- Low Light
Fig. 10. a) Kinect model b) ZED Camera model
ii. Excess Light/Reflective Surfaces
The ZED camera gives a poor model when subject to reflective surfaces or excess shine. The Kinect on the other hand gives a more stable model in most of the conditions.
Fig. 11. Actual Space – Reflective
Fig. 12. a) Kinect model b) ZED model
However ZED camera provides more distinct boundaries when a glass plane is part of the picture. Most of the points on the glass plane are missing for both the devices. This can be observed in figure 13 as shown below.
Fig. 13. Glass Plane a) Actual Space b) Kinect model c) ZED model
F. Device in motion
This section aims to test the sensor’s capabilities when mounted on an autonomous robot for indoor mapping purposes in a dynamic motion scenario. The quantitative analysis done with the devices in static conditions in previous sections still hold true as even in motion the device feeds results frame by frame to the system. However an error in the readings could be assumed owing to the various motion parameters. The experiment was initiated using a P3DX robot system. However due to the space restrictions, a manually pushed trolley was used instead to mimic the motion of the robot. The Sensors were mounted on a tripod and then placed on the trolley. A portable power bank was used to power the Xbox Kinect Sensor.
The setup can be seen in Figure 14 as shown below.
Fig. 14. Setup- Kinect (left) and ZED Camera (right)
The trolley was pushed at a moderate pace to ensure that the sensors do not lose their odometry while scanning. The location contains various reflective surfaces, glass panels and non-distinctive wall features. The experiment was conducted in medium lighting conditions with ample sunlight in the space and with minimum human movement. The actual space can be seen in figure 15. The entire 3D point cloud is shown in figure 16.
Fig. 15. Actual Sample Space
Fig. 16. Entire 3D Point Cloud
RTAB-MAP software package was used to conduct the 3D mapping on a Linux system. In order to analyze the differences between the sensors, the following factors were considered and observed:
2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan
i. Range
Owing to the sensors capabilities, ZED Stereo camera was able to capture information at a longer range at any given frame when compared to the Kinect Sensor. Objects too close to the sensors were not recorded as it fell outside the range of both the sensors. The differences in the range can be seen in Figures 17 and 18.
Fig. 17. Kinect Model- RTAB MAP
Fig. 18. ZED Camera Model- RTAB MAP
ii. Sensitivity to motion
The Kinect sensor was more stable during the motion when compared to the ZED Camera in an indoor environment. The ZED loses its odometry very often compared to the Kinect camera and thus had to be moved more safely and precisely.
iii. Susceptibility to Moving Objects
Both the cameras were very sensitive whenever any human motion was encountered and required a pause or a reset based on the scenario to resume mapping.
iv. Features Details
At any given frame, the ZED camera captured finer details of the surroundings than the Kinect sensor. This becomes crucial when the sensors are used for commercial indoor mapping where feature recognition is necessary. The differences in the features can be observed in the figures 19 and 20. Both the sensors were unable to capture information accurately for surfaces parallel to the sensor as the angle for viewing was not ideal.
Fig. 19. ZED Camera’s Point Cloud
Fig. 20. Kinect Sensor’s Point Cloud
v. Non-distinctive Environment Features
In the case of non-distinctive surrounding features like a plain wall or similar colored wall features, the sensors find it difficult to make out the difference between consecutive frames. Both the sensors were sensitive under such scenarios and lost their odometry. The Kinect sensor was however comparatively more accurate than the ZED camera in such conditions.
vi. Ease of Use
When considering the applicability of these sensors on a mobile system, it is important that the hardware and software of the sensor is easy to use and integrate with the existing systems. In this study, the size and wiring required for the Kinect made it difficult to use on the available autonomous robot P3DX and thus a manual trolley had to be used to complete the experimentation.
During the mapping it was realized that the ZED camera is more convenient compared to the Kinect sensor. This is due to its smaller size and weight. The Kinect sensor required additional adapters for collection along with a power source. On the other hand, the ZED just requires a USB port for power and connection.
V. COMPARISON WITH CONVENTIONAL DEVICES
3D scanning has varied industry applications. The current market is about $1.328 billion with constituting sectors like Industry and Commercial ($749M), Scientific, defense and space ($325M), Medical ($177M), Consumer ($20M) and Automotive ($56M) [13]. The volume of applications is low but the product value is high. But as of recently, the market is shifting. With the release of low cost solutions, a new consumer market has emerged in the recent years with a CAGR of 158% [13]. This has been largely due to the capabilities of mobile phones having multiple camera sensors which has led to an advent of virtual reality, augmented reality and mixed reality applications. The availability of solutions which are low cost, portable, easy to use and provide high quality results is thus desirable. Such characteristics are also vital in building towards a Smart Nation where 3D Scanning will be required to be used on a widespread basis.
In search of an affordable and efficient system, the earlier sections of this study have explored Xbox Kinect Sensor and the ZED Camera in detail, however in order to look at the bigger picture, a comparison between these devices and the conventional systems is necessary. The various available devices available in the market are compared in the table below:
2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan
TABLE 4. Comparison of Specifications [11] [14] [15] [16] [17]
Weight (g)
Size (mm)
Range (m)
Field of
View (degrees)
Price (USD)
ZED 159 175x30
x 33 0.7- 20
96x 54
$449
Kinect Sensor v2.0
1400 249x66
x67 0.5-4.5
70x 60
$99
Matterport Pro 3D
3000 22x260
x111 - 360 $3,600
FARO (Focus 3D)
5000 240x200x100
0.6-120
305x360
$15,000
LEICA (SS C10)
13000 238x358x395
0-300 270x360
$30,000
Google Tango
259 180x89
x11 -
180x180
$499
TABLE 5. Comparison of Pros and Cons
Pros Cons
ZED Stereo Camera
Open Source Software
Cheap and portable
Motion Mapping
Additional power source/equipment not required
High sensitivity to motion
High error for measurement
Difficulty with non-distinctive features areas
Kinect Open Source Software
Cheap and portable
Motion Mapping
Easy to integrate with other systems
Low Range
Low Resolution
Matterport Advanced Software Support
Fast and portable
Easy to use by the user
Monthly Cloud Subscription required
Cannot edit individual images
Stationary scans only
Faro Large Range
Very high quality
Low error
Expensive
Stationary Scans only
Leica Large Range
High Resolution
Low error
Expensive
Stationary Scans only
Tango Open Source Software
Cheap and portable
Low Range
Low resolution
Motion Mapping
Additional power source/equipment not required
Limited applications as of now
Which one is the best? – A question of integration
When considering the compatibility and growth of any technology device, it is important of how the device integrates itself with humans (the end user) and other existing technologies (for modification and varied applications). The same applies for 3D scanning systems. Different scanning techniques have come up (like stereo vision, time of flight, structured light, laser triangulation etc.) providing solutions for different applications. However every individual may have different needs. As seen in table 5 above, different devices have different characteristics. But which one is the best? Looking at it from a practical viewpoint, a mobile based 3D scanning system (such as Google Tango) seems the most viable. This is because it eliminates the need of any additional scanning device. The smartphone penetration rate is 30% worldwide with the numbers being as high as 72% in developed countries such as the US [18]. Thus, such a solution will help make 3D scanning accessible to the masses. An integrated and versatile system will be driving the technology in the future.
VI. CONCLUSION
The study has been successful in analyzing the scanning devices available and using them for indoor mapping purposes. The Kinect sensor by Windows and Stereolab’s ZED Stereo Camera both proved to be viable alternatives to the traditional laser scanners for the indoor 3D mapping. These alternative devices could be potentially used for realizing the goal of a Smart City by utilizing the generated indoor maps for varied purposes. The devices are available at a fraction of a cost and provide quality results. They come with advanced standard software packages with which the new applications can be easily developed and explored. However as of now when compared to professional devices, these devices have limitations such as the quality of scan, range and industrial compatibility. When compared to the ZED camera, the Kinect sensor is cheaper and has proved to be more accurate while recording the features. It is also better when the surroundings include reflective surfaces and non-distinctive features. It provides stable mapping when placed on a moving platform. The ZED Stereo camera on the other hand is portable and easy to use and has an extended range and resolution quality. It is also faster and better at memory optimization.
Owing to the different qualities possessed by both the Kinect sensor and the ZED Stereo camera, they can be used under different scenarios for different purposes where very high quality is not required. Post processing of the meshes can also help optimize the quality as per the requirements.
The ease of use, portability, size and cost effectiveness of the devices explored in this study shows the potential use of these technologies in the future. With further research, such indoor mapping technologies can be optimized on both the software and
2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan
hardware front. The devices are expected to get smaller and faster. Applications such as automated path finding, indoor mapping, etc. for real estate, commercial and facility management purposes can be realized.
Recommendations
In order to help transform a city to a “Smart City”, new technologies should be able to deploy autonomously at a low cost point. This study has provided a good basis of comparison and capabilities of the systems under different scenarios. The following advancements are recommended to take this technology to the next level:
1) Integration with Unmanned Aerial Vehicles (UAVs) With the recent advancements in the drone technology,
drones can autonomously plan the path around unknown spaces. Both the devices could be mounted onto UAVs to map indoor spaces autonomously. However challenges like drone movement, speed, scan registration process and limited drone flight time are required to be overcome. Such systems could be of great applications at places where the area is inaccessible by humans or ground vehicles.
2) Integration with Autonomous Ground Robots Grounded robots mounted with 3D scanners can be used to
autonomously map indoor spaces. Grounded vehicles provide more stability and control when compared to UAVs. There are however limitations to this system like inability to work on uneven surfaces, etc.
3) In built Mobile phone camera technology If 3D mapping technology is required to be implemented
using a hand held device, integrating a depth camera onto existing mobile phones is the fastest and most cost effective method. This concept is similar to the technology which is currently being developed by Google under Project Tango. This technology has wide applications where virtual reality could be integrated with 3D mapping to create advance solutions for different industries. Depth Sensor such as Occipital Structure Sensor which can be mounted on an existing mobile device also looks promising [19].
4) Memory Management As evident from the experiments conducted via this study,
the amount of memory space acquired by each 3D depth file is enormous. This leads to an acute case of memory shortage when this technology is used on a mobile device. Real time cloud data storage techniques could be explored as a future focus area with these devices.
5) Hardware The current available 3D scanners requires high
configuration hardware which might restrict their integration with other devices. Solutions requiring minimum resources could be explored. The size and power requirements for the device could also be optimized based on the application.
As said before, these alternative 3D scanning technologies hold great potential in applications across various areas of the industry. The future work should base its focus on developing a mobile, precise, easy to use and cost effective solution.
ACKNOWLEDGMENT
The authors would like to thank Nanyang Technological University (NTU), Singapore and Temasek Laboratories, NTU for providing the facilities and financial support to complete this study. The author would also like to thank Mr. Ong Eng Hui and Mr. Lee Yi Han from Temasek Laboratories, NTU for providing their assistance and technical know-how for the experimentation process.
REFERENCES
[1] B. E. R. H. P. H. J. K. J. P. P. W. C. Harrison, "Foundations for
Smarter Cities," IBM Journal of Research and Development, vol. 54,
