Kinect calibration

Ilya Afanasyev

Facoltà di Ingegneria

Trento, 2012

1/2025/01/2012

Content

2/20

1. The description.2. About the existing software.3. Tests of Nicolas Burrus software binaries.4. Open Kinect drivers installation.5. Compilation of Nicolas Burrus source code.6. Results.7. Conclusions8. References.

25/01/2012

The description.

3/20

The software for Kinect Calibration should be used for:

1. Saving the RGB images and IR-Depth data from Kinect sensor under OS Windows.

2. Calibration the Kinect sensor with obtaining [1]: - the intrinsic parameters (i.e. matrixes of intrinsic parameters for IR camera (K_ir) & RGB camera (K_rgb)).- the extrinsic parameters (i.e. extrinsic mapping between IR (depth) and RGB Kinect camera (R,T)).- distortion coefficients (kc_ir, kc_rgb). - the intrinsic and extrinsic parameters (i.e. matrixes of intrinsic parameters for IR camera (K_ir) & RGB camera (K_rgb) and extrinsic mapping between IR (depth) and RGB Kinect camera (R,T)).

3. Developing the interface of different algorithm with Kinect for different real-time object recognition applications.

16/01/2012

About the existing software.

4/20

The existing analogues of the software:1. - The author doesn’t know analogues under Windows7/64 bit . It

can be created, basing on Microsoft Kinect SDK for developers [6] under the platform .NET Framework ver. 4.0.

2. - There is the software developed by Nicolas Burrus [2] under OS Linux / Ubuntu 10.10 / 32 bit, which is claimed to be compiled also with Windows-32 bit. Can it be used under Windows7/64 bit ?? -> It seems that NO!!

25/01/2012

Microsoft Kinect SDK for developers [6]

5/20

1. SDK (Software Development Kids) includes drivers, APIs for raw sensor streams and human motion tracking, installation documents, and resource materials. It provides capabilities offered by the Microsoft Kinect device connected to computers under Windows 7 / Windows 8 to developers who build applications with C++, C#, or Visual Basic by using Microsoft Visual Studio 2010.

16/01/2012

Nicolas Burrus software [2]

6/20

Nicolas Burrus software [2]:There are different versions of the Burrus software:1. Test programs running from binaries -> Windows binaries -> Kinect RGBDemo v0.6.1. -> supports OpenNI/Nite backends [3] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit -> there is no the extrinsic mapping between IR and RGB camera at Kinect calibration.2. Compiling from source -> 2.1 Compilation on Windows -> with OpenNI/Nite backends doesn’t give the extrinsic mapping between IR and RGB camera at Kinect calibration.with Libfreenect backend -> I wasn’t succeed to compile under Windows 7/64 bit?! 2.2 Compilation on Linux (Ubuntu 10.10 - 32 bit) -> supports Libfreenect backend -> the only workable variant with getting all calibration parameters (intrinsic, extrinsic and distortion) was realized on Alberto Fornaser’s PC (UniTN)

16/01/2012

7/20

1. OpenNI backend:1. Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports OpenNI/Nite

backends [2] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit.

2. Install Sensor Kinect drivers under Windows 32 bit [4] and OpenNI/Nite modules (Win32) [3] according to the instruction [5] and verified that they work properly.

3. Use RGB-D Capture [1] to grab Kinect RGB and IR images with intensity.raw and depth.raw files. -> the software doesn’t grab IR images!! -> only RGB image, intensity.raw, depth.raw and calibration file (calibration.yml without distortion parameters)!!

4. “calibrate-openni-intrinsics --pattern-size 0.0405 grab1 calibration.yml” -> gives “openni_calibration.yml” with Intrinsic matrix and distortion coefficients for Kinect RGB camera !!

5. “calibrate-openni-depth.exe --pattern-size 0.0405 grab1” gives partly processed figures with a message about errors.

6. The software doesn’t produce a file with xyz- coordinates and RGB-colors info of every point!!!

1st a

ttem

pt

16/01/2012

The using of Nicolas Burrus software [2] with OpenNI backend under Windows

The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu 10.10 - 32 bit) - OpenNI backend (under OS Windows 32bit)

-

The same figures of Intrinsic Matrixes for Kinect RGB and

Depth Cameras

Libfreenect backend OpenNI backend

The Depth Camera Calibration for OpenNI backend is

NOT available!!!16/01/2012

The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu 10.10 - 32 bit) - OpenNI backend (under OS Windows 32bit)

- 23/11/2011

There is NO extrinsic mapping between Kinect Depth and RGB

cameras for OpenNI backend

Libfreenect backend OpenNI backend

The reprojection of 3D points from Kinect Depth Camera to

RGB image for OpenNI is NOT possible!!

Install OpenKinect for Windows 7 [7]

10/20

It is necessary to have:- Visual Studio 2010 (as a compiler on a 32 bit Windows 7 installation).- Kinect Sensor drivers for Windows 7 (OpenKinect-libfreenect ofTheo/ofxKinect) [9].- freenect windows dependencies [8]: libusb-win32 1.2.4.0 (Currently there is a breaking change in libusb 1.2.5.0, which I used!!); pthreads-win32 Glut

17/01/2012

Kinect Sensor drivers installation [7]

11/20

- Download & extract the latest GitHub repository [9]. - Make sure your Kinect sensor is plugged into a

USB port on your computer.- Open Device Manager (Start Control Panel

System and Security System Device Manager)

- Find “Human Interface Device” (Xbox NUI Motor) Update Driver Software uninstall the previous Kinect drivers (if there are) set path like C:\Special\OpenKinect_libfreenect_v0_1_1_56\platform\windows\inf\xbox nui camera\ the drivers will correspond to the processor type.

18/01/2012

Kinect Sensor drivers installation [7]

12/2019/01/2012

Check the files: libusb0.dll (x64) in C:\Windows\System32\and libusb0.sys in C:\Windows\System32\drivers\and libusb0.dll (x86) in C:\Windows\SysWOW64\

Cmake configuration & Generation of a Visual Studio 10 solution [7]

13/20

1. Open Cmake GUI and set the Browse Source path to /libfreenect/openkinect.

2. Set the browse build path to /libfreenect/openkinect/build

19/01/2012The software should be configured when Kinect turned off!!

Cmake configuration & Generation of a Visual Studio 10 solution [7]

14/20

You will now need to point certain build paths to the freenect dependencies (pthread).

19/01/2012

The software should be configured when Kinect turned off!!

Cmake configuration & Generation of a Visual Studio 10 solution [7]

15/20

You will now need to point certain build paths to the freenect dependencies (libusb).

19/01/2012

If take "msvc" instead of "msvc_x64" -> 1 fatal error of compilation less !!!It doesn't matter if choose THREADS\LIB\ pthreadVC2.lib or pthreadVSE2.lib

Cmake configuration & Generation of a Visual Studio 10 solution [7]

16/2019/01/2012

Run "Configure" and it should successfully complete. Click Generate afterwards and then open the libfreenect.sln file from the build folder the output was placed in - /openkinect/build/

Run Configure again. It should successfully complete the process this time but we still need to add the Glut references.

Visual Studio 10 solution [7]

17/2019/01/2012

RESULT: Rebuild All: 13 succeeded, 2 failed, 2 skipped.You will receive a few errors when building the solution but it should compile correctly regardless. The binaries will be placed in build/bin/Debug.

Disable the Antivirus if it disturbs the creation of the VS10 solution!Choose Visual C++ 2010 and then select Build Solution from the

Debug Menu or press F7.

The result of building the c for Windows 7 [7]

18/20

The final step is to place a few .dll files in the appropriate folder and then we can run some of the samples provided: Copy freenect.dll and freenect_sync.dll from \build\lib\Debug Copy from dependencies folders: pthreadVC2.dll, glut32.dll, libusb0.dll (choose the correct version x86/x64) Copy tiltdemo.exe, regtest.exe, glview.exe, glpclview.exe, glpclview.exe, regview.exe, hiview.exe from \build\bin\Debug It can be helpful to copy freenect_sync.lib, freenect.lib, freenect.exp from \build\lib\Debug to \lib\ Successful start of demo-samples Kinect works!

19/01/2012

Demo-samples of OpenKinect for Windows 7

19/20

tiltdemo.exe drives the Kinect motor. regview.exe shows the combined

IR&RGB video (IR up to 1 m)

regtest.exe grabs bad-quality pictures: -- registration_test_depth_registered.pgm, -- registration_test_rgb.ppm, -- registration_test_depth_raw.pgm, -- registration_test_depth_mm.pgm

glview.exe gives IR&RGB video in separate windows (IR up to 3.5 m)

19/01/2012

Demo-samples of OpenKinect for Windows 7

20/20

hiview.exe gives IR&RGB video in separate windows (IR up to 3.5 m, RGB has hi-quality & hi resolution)

glpclview.exe gives color 3D video with big shadows behind the objects. cppview.exe produces a continuous list with messagesunder the phrase:“thread execution started”

19/01/2012

Compiling RGBDemo-0.6.1 under Min GW from source with OpenKinect on Windows 7 [2]

21/20

The step-by-step procedure for Min GW, in case using libfreenect:1. Install QT opensource for Windows. This will also install Min GW. 2. Add C:\Special\QtSDK\mingw\bin\ to the Path environment variable: Start Computer (properties) Advanced System Settings Environmental Variables (button) Variable: PATH Edit Add. 3. Install and run cmake on rgbdemo Disable the NESTK_USE_OPENNI cmake variable “Could not find module FindOpenCV.cmake” “Configuring incomplete, errors occurred!”4. Ungrouped Entries OpenCV_DIR (NOT FOUND) C:/Special/opencv. QT_QMAKE_EXECUTABLE (NOT FOUND) C:\Special\QtSDK\Desktop\Qt\4.7.4\mingw\bin\qmake.exe it is better to use qmake.exe from QtSDK\mingw ??

19/01/2012

2nd a

ttem

pt

Compiling RGBDemo-0.6.1 under Min GW from source with OpenKinect on Windows 7 [2]

22/20

The step-by-step procedure for Min GW, in case using libfreenect: Open the CMakeLists.txt in Qt Creator (from C:\Special\RGBDemo-0.6.1-Source\RGBDemo-0.6.1-Source\CMakeLists.txt open with C:\Special\QtSDK\QtCreator\bin\qtcreator.exe). It creates the folder: qtcreator-build Add QT-qmake & OpenCV path to the Path environment variable:C:\Special\QtSDK\Desktop\Qt\4.7.4\mingw\bin\C:\Special\opencv\ to the Path environment variable: Start Computer (properties) Advanced System Settings Environmental Variables (button) Variable: PATH Edit Add.

20/01/2012

Compiling RGBDemo-0.6.1 under Min GW / MSVS 2010 from source with OpenKinect on Windows 7 [2]

23/20

The step-by-step procedure for Min GW / MSVS2010, in case using libfreenect: Cmake 2.8.6 Ungrouped Entries PKG_CONFIG_EXECUTABLE C:\Special\opencv\OpenCVFindPkgConfig.cmake Results: Cmake can’t compile the QT & OpenCV_DIR OpenCVConfig.cmake We should use Qt!! Qt Run Cmake “CMAKE_MODULE_PATH C:\Special\qtcreator-build\” Run Cmake “CUDA_TOOLKIT_ROOT_DIR not found or specified” [10] if set path to C:\Special\PCL_1_4_0\ which contains CUDA libraries, then we have the messages “CMake Error: The source directory "C:/Special/PCL_1_4_0/3rdParty/flann/bin" does not appear to contain CMakeLists.txt.Specify --help for usage, or press the help button on the CMake GUI”.

20/01/2012 Burrus software “RGBDemo-0.6.1” is not installed on Window7-64bit!!

Conclusions

24/20

1. Nicolas Burrus software with OpenNI binaries [2] works under Windows 7-64 bit, but the software doesn’t give the extrinsic mapping between Kinect Depth and RGB cameras and doesn’t produce a file with xyz- coordinates and RGB-colors info of every point!!!2. Although the drivers of OpenKinect has been successfully installed under Windows 7-64 bit, the Nicolas Burrus source code RGBDemo-0.6.1 (with LGPL License) for Kinect Calibration [2] is not compiled properly under Windows 7-64 bit. It should be used another software or another Kinect drivers!!3. Both Microsoft Kinect SDK for developers [6] and OpenKinect drivers [9] can be applicable to save RGB images and IR-Depth data to calibrate Kinect sensor and use it under Windows 7-64 bit, but no suitable software has not been produced yet.

25/01/2012

5. References1. Camera Calibration and 3d Reconstruction. // OpenCV (Open

Source Computer Vision) v2.1 documentation: http://opencv.willowgarage.com/documentation/cpp/camera_calibration_and_3d_reconstruction.html

2. Nicolas Burrus. Kinect. RGBDemo, calibrate and visualize Kinect output. // http://nicolas.burrus.name/, 2011.

3. OpenNI Modules. // www.openni.org, 4. SensorKinect drivers. // github social coding.

https://github.com/avin2/SensorKinect.5. How-to: Successfully install Kinect on Windows (OpenNI and

NITE). // Vangos Pterneas blog: http://studentguru.gr/b/vangos/archive/2011/01/20/how-to-successfully-install-kinect-windows-openni-nite.aspx

6. Microsoft Kinect SDK for developers. http://www.microsoft.com/en-us/kinectforwindows/

25/2016/01/2012

5. References7. Install OpenKinect for Windows 7 and XP. //

http://kinect.dashhacks.com/kinect-guides/2011/01/09/install-openkinect-windows-7-and-xp

8. Getting Started using OpenKinect. // http://openkinect.org/wiki/Getting_Started#Windows.

9. https://github.com/ofTheo/ofxKinect 10. “OpenCV: CUDA_TOOLKIT_ROOT_DIR not found or specified“.

http://xinyustudio.wordpress.com/2011/11/30/opencv-cuda_toolkit_root_dir-not-found-or-specified/

26/2016/01/2012