+ All Categories
Home > Documents > Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer...

Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer...

Date post: 02-Oct-2020
Category:
Upload: others
View: 0 times
Download: 0 times
Share this document with a friend
22
1 Horst Bischof Redundancy for aerial computer vision Exploiting redundancy for reliable aerial computer vision Horst Bischof 2 Horst Bischof Redundancy for aerial computer vision Digital Image Increase
Transcript
Page 1: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

1 Horst Bischof Redundancy for aerial computer vision

Exploiting redundancy for reliable aerial computer vision

Horst Bischof ���������� ����������������

2 Horst Bischof Redundancy for aerial computer vision

Digital Image Increase

Page 2: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

3 Horst Bischof Redundancy for aerial computer vision

Images Worldwide

� �������������� ��� ������������� ��������������� ������������� ��������������� �� !�� ��������������

�"��#�����$!��������%�����

4 Horst Bischof Redundancy for aerial computer vision

Terrestrial Image Acquisition

� �������& ��� �%�'�� (���������!�����)���*� ��� �+� �,����!� ���� �����*�� ��-����� �����

Page 3: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

5 Horst Bischof Redundancy for aerial computer vision

Aerial Photogrammetry

� -����������.�� �)#������ � �,� �������+� �&/� �01������� �� ��-�!����2�#����3���)���!��&,+�� (����!�����)(���*� ��� �+��

6 Horst Bischof Redundancy for aerial computer vision

New Sensor Platforms

� -�� ������� �� 4��3����)�� ���� ����������5����5��� �

��� ������5� ��*&�'����� ����� ��+�

6 Towards Fully Automatic Photogrammetric Reconstruction Using Digital Images Taken From UAVs./7�$�������"�%78�1�!���"�#7�8������ �"�97�:�����"���5�47�;�����7�����<�����$&*=&�����7>��

Page 4: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

7 Horst Bischof Redundancy for aerial computer vision

Airborne vs. MAVs

#������ ��� ��-�!,�

������3�?����

#/%�*�� �3�/@��@����3�������

A���7���� A����

�&,B�(C�! �3���9��B�(2��!�

�&,B�(�7��! �3���9��B�(��!�

8 Horst Bischof Redundancy for aerial computer vision

Applications

Page 5: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

9 Horst Bischof Redundancy for aerial computer vision

Application: Virtual Habitat

[Leberl et al. IEEE Computer 2010]

10Horst Bischof Redundancy for aerial computer vision

Application: Architecture and Cultural Heritage

[Zebedin 2010], [Irschara 2010]

Page 6: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

11Horst Bischof Redundancy for aerial computer vision

Application: Construction Site Monitoring

12Horst Bischof Redundancy for aerial computer vision

Application: Mining

Page 7: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

13Horst Bischof Redundancy for aerial computer vision

Outline

D 3D Reconstruction �E & �1� 1������!�#� ���E ,�����& �����E =�51�5�����

D Semantic Classification �E *3���������������� ���E =��1����� ���E 41����

D Discussion & Outlook

14Horst Bischof Redundancy for aerial computer vision

3D Reconstruction

Page 8: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

15Horst Bischof Redundancy for aerial computer vision

Structure from Motion

6,7�F� ��"�G/1 �!� ��5����������� �1� ������!�1������� �5�5�����H1�����I7�*�7,7"�����>�6#7�*���������� ���7"�G%�1���!�5�����' �������5���5���!���I7�$J-%"�����@>�6F7�&������� ���7"�G*�� ����1��!B�03�������*�� ��-����� �������,I7�&$��=/*9"�����>�6&7�/���'���� ���7"�G:1�5���=�!������,��I7�-%*="����2>�6J7K#7�4���!�� ���7"�G:1�5���=�!�������-��15�����,��I7�0--%"�����>�

5�����H1����� �1��������5�!�����

�,�!�5�����!�������� � ����L��,�� �1� 1���

$F*��� M��*���

16Horst Bischof Redundancy for aerial computer vision

Structure from Motion

-�!�����M��� � ����

�,�&������& �1� 1���

$!�����-��������5������

Page 9: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

17Horst Bischof Redundancy for aerial computer vision

Structure from Motion

X1

X4 X

3 X2

X6

X7

X5

-�!������R2, t2, [K2]

minimize g )R"t"[K]"X+�

-�!������R1, t1, [K1]

-�!������R3, t3, [K3]

6F7�&������-%*=�N�2� 1 ����>�

�7�7��������!��������7�����,���� ���

����7�����1����'���

18Horst Bischof Redundancy for aerial computer vision

�*�� �-*��&���51����K��3�

Structure from Motion

Feature Extraction

Coarse Matching

Detailed Matching

Geometric Verification

Images Pose Prior

Geometric Estimation

Epipolar Graph

Image Overlap

Local Descriptors

Camera poses 3D points Matches

Calibration Pose Prior

-�!�1 ������5�,����/��� �� 1���)-�,/+�

Page 10: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

19Horst Bischof Redundancy for aerial computer vision

Aerial Photogrammetry

� �����������!������ �����(�����!O�� C�P��������� �@"@���3�2"@����3���� (����!��&,�

20Horst Bischof Redundancy for aerial computer vision

Global Depth Map Optimization

5� �� ��!����1��� �� ��!�

���

��� ��������������

���

)),(()(

)),(( ���� 77�!� �������� ��)F--+�

�������

)( 77��� ���%��� ���)�%+����1����� ��B�

� ������������� ����!�� ������ �5���� �1 �����������

6�7�*����� ���7"�G/�����3����!1�� ��������� �1�1��!1� K������������!�I7�0--%"����C>�

-�� �1�1��0������#�!�� ���/�������

Page 11: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

21Horst Bischof Redundancy for aerial computer vision

KKEY view

22Horst Bischof Redundancy for aerial computer vision

DEPTH map

min

5� �� ��!����1��� �� ��!�

���

��� ��������������

���

)),(()(

Page 12: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

23Horst Bischof Redundancy for aerial computer vision

DEPTH map

min

5� �� ��!����1��� �� ��!�

���

��� ��������������

���

)),(()(

24Horst Bischof Redundancy for aerial computer vision

Page 13: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

25Horst Bischof Redundancy for aerial computer vision

Primal-Dual Optimization D �������1� �������3��� !�� ���������!����������5���

����� K��5�����!��K51�������� �!�6-��!�����"�*��������>�D 4�� 1���������������� !����������������������3�

������!��' �����'��� �1� 1���D -������������ �������������5����!����������������

���5'�����1�������*���

26Horst Bischof Redundancy for aerial computer vision

Distributed Visual SLAM

Set of updateable, geo-referenced virtual cameras

PPose

Low framerate, for map extension and relocalization

Full framerate, “cheap” features, for visual servoing

Very little data needed, provide only data necessary for the

current environment Maintain global map,

use “expensive” features!

Page 14: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

27Horst Bischof Redundancy for aerial computer vision

Dense Reconstruction On-the-Fly

28Horst Bischof Redundancy for aerial computer vision

Dense Reconstruction On-the-Fly

Page 15: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

29Horst Bischof Redundancy for aerial computer vision

Semantic Segmentation

30Horst Bischof Redundancy for aerial computer vision

Motivation – Semantic Interpretation

Water ?

Building ?

Tree ?Street ?

10 cm/pixel

Page 16: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

31Horst Bischof Redundancy for aerial computer vision

Motivation

�… a lot of overlapping images

3D information

32Horst Bischof Redundancy for aerial computer vision

Fusion - Model

D �%K;��5�������!�5���6F�������@>�����

- Robust against outliers - Preserves sharp edges

Page 17: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

33Horst Bischof Redundancy for aerial computer vision

Fusion - Model

D 03 ������ ��!1� ���������� ����6Q����?>��

D ��

Some observations

34Horst Bischof Redundancy for aerial computer vision

Fusion – Semantic Interpretation

D =����!�� ����� ��������������!�� ��D -�� �1�1�����!1�� ������ ���*� ��!�5���6*����2>�

� aggregated

refined

Page 18: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

35Horst Bischof Redundancy for aerial computer vision

Fusion - Color and Height

Color: Wavelet-Fusion Height: TV-Fusion

36Horst Bischof Redundancy for aerial computer vision

��

Some observations in ortho-view

Page 19: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

37Horst Bischof Redundancy for aerial computer vision

Evaluation – Redundant Interpretation

Remember evaluation on single images: Graz 89.5%, Dallas 92.5%

38Horst Bischof Redundancy for aerial computer vision

Large-Scale Results

Building

Green Area

Waterbody

Tree

Streetlayer

Dallas, 4 tiles, each 240 x 240 m, 15 cm

Page 20: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

39Horst Bischof Redundancy for aerial computer vision

Building

Green Area

Waterbody

Tree

Streetlayer

SF 10 x 3 tiles 2500 x 700 m 15 cm

40Horst Bischof Redundancy for aerial computer vision

Large-scale Results

Building

Green Area

Waterbody

Tree

Streetlayer

Graz, 7 km2, 155 images, 20 x 10 tiles, 8 cm

Page 21: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

41Horst Bischof Redundancy for aerial computer vision

Conclusions & Future Work D /���������5��,������� �1� ������� ������5�!����

� ���� ��������� ����D /������������5��,�����!� ��������

�D =�51�5����������B�8��� ���1 �!� �� ���

�D =��1� ���!�1 � ��������semantic 3D model

D 41 1��B��������#/%��E 4��3����E -�����

42Horst Bischof Redundancy for aerial computer vision

Videos/Code/Papers see

'''7��1@���7���������7��7 1����7� �

Page 22: Digital Image Increase - Universität Klagenfurt · Horst Bischof Redundancy for aerial computer vision 25 Primal-Dual Optimization D 1 3 ˆ ! ˆ ˇ ! 5 K 5 ˆ ! K51 ! 6- !ˇ "* ˙˝˘˛˘>

43Horst Bischof Redundancy for aerial computer vision

Acknowledgments 41�5������5�5���B��D 4R4�1�5���*��.�� �9�����M�5���%��� �����#� ��5������-�!�1 ���

%����D 4$�K$��*�����!��1�5�5����:#%$��1�5���*��.�� �%#�*�"�*0�/&�&"�

-MF&�=�-�"�9M;$&�$-�D F%5���

���


Recommended