Open Source Multipurpose Multimedia Annotation Tool
Joed Lopes da Silva1, Alan Naoto Tabata1,2, Lucas Cardoso Broto1,2,Marta Pereira Cocron1, Alessandro Zimmer1,2, and Thomas Brandmeier1
17th International Conference onImage Analysis and Recognition
ICIAR 202024-26 June, 2020 – VIRTUAL CONFERENCE
1 Research and Test Center CARISSMA, Technische Hochschule IngolstadtEsplanade 10, 85049 Ingolstadt, Germany
{Joed.LopesdaSilva, Marta.PereiraCocron}@carissma.eu{alt9707, luc2031, Alessandro.Zimmer, Thomas.Brandmeier}@thi.de
2 Federal University of Parana, XV de Novembro, Curitiba, Paraná 1299, Brazil
Technische Hochschule Ingolstadt | CARISSMA
- Important task
- Models trained with bad quality labeled dataset may not achieve good results
IntroductionData Annotation
2
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Bad Quality Labeling
Train
Bad Quality Detection Results
Target: Human Face
Technische Hochschule Ingolstadt | CARISSMA
- Efficient Data Labeling requires Efficient Tools
- Current Open Source Labeling Tools:
- Supports only few data types: Image or Video or Audio [1,2,3]
- Few customization
- Few/no integrated data analysis
IntroductionData Labeling
3
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Raw Data
Labeling Tool
Draw Labels(Editor) Train
Common Data Annotation Workflow
ImportData
ExportData
Technische Hochschule Ingolstadt | CARISSMA
- Current Demands: Multimedia and Multipurpose
IntroductionData Labeling
4
17th International Conference onImage Analysis and Recognition
ICIAR 2020
LiDAR
Data Labeling Toolbox
RADAR
RGB Camera
Near InfraredCamera
IMU
Environment
Video
Video
1D-Signals
Point Cloud
N-DimensionalSignal
Text, Number
Multiple Data Types
Useful Editors Customization
Analysis
Data Storage
Reports
DataFilters
TrainModels
Import and Analyze Results
DatasetApplication 1
DatasetApplication N
DatasetApplication 2
.
.
.
Multiple Applications
ImportData Export
Data
Automotive Scenario
Technische Hochschule Ingolstadt | CARISSMA
Proposal
5
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
ProposalMain Objectives
6
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Multipurpose
Open Source Project
Multimedia
Data Analysis
The users can collaborate, improve and customize the application according to their requirements.
- Multiple annotation types:- 2D Label: Bounding Boxes, Ellipse, Polygons, Custom.- General Data: Text, Number, Boolean, Arrays, States, Checklist.
- Attributes customization according the project.- Grouped Data.
Support multiple types of data: - Image, Video, Audio, Point Clouds, Signals.
- Statistics: Basic, Descriptive Analysis, Hypothesis Tests, ANOVA.- ML Metrics: Accuracy, Precision, Recall, F1-Score, mAP, etc.- Data Filtering: create subsets based on labeled data.
Export - YOLO Format, Data Frame, CSV, Excel, JSON,.
Technische Hochschule Ingolstadt | CARISSMA
ProposalSoftware Technologies
7
PyQtGraph
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Py
Technische Hochschule Ingolstadt | CARISSMA
ProposalRecommend Workflow
8
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
ProposalData Storage Model
9
17th International Conference onImage Analysis and Recognition
ICIAR 2020
- Generic Properties- Document Based is Suitable- Functions: Map and Reduce- Remote Labeling
Technische Hochschule Ingolstadt | CARISSMA
Use Cases
10
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
- Objective: Driver Distraction Monitoring [4]
- Driver Info: Name, Age, Country, Automobile Licenses
- Computer Vision: Head Pitch, Roll, and Yaw angles
- 2D Label: Custom Deformable Face Model, Eyes, Head Region
- Event Annotation: Texting, Radio, Cellphone use
Use Case 1Driver Distraction Database
11
17th International Conference onImage Analysis and Recognition
ICIAR 2020Inertial Sensor
Camera and IlluminationSimulator
Vehicle Data- Velocity- Steering Angle- Position- Break Pressure- etc.
Head Angle
Technische Hochschule Ingolstadt | CARISSMA
- Objective: Eye Study based on Infrared
Images
- Subject Info:
- Name, Age, Gender, Weight
- Additional Info
- Eye Info:
- State: Blink (Yes, No)
- Yaw and Pitch Angle
- 2D Label:
- Iris (Ellipse)
- Pupil (Ellipse)
Use Case 2Eye Movement Study - Pupillometry
12
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
- Objective: Label objects in traffic scenario
- Recording Info:
- Day, Night, Sunny, Snow, Rainy
- Environment:
- Highway, Residential Road
- Drawables:
- Bouding Boxes: Car, Truck, Traffic
Signs, Pedestrian, Bycicle,
Advertisement
Use Case 3Multi-Object Traffic Dataset
13
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
- Objective: Label Image and Point Cloud
- 3D Object Detection and Segmentation
Use Case 4Image + Point Cloud Annotation
14
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
Project Roadmap
15
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
- New Features for Implementation:
RoadmapProject Roadmap
16
3D Workflows
NLPWeb Apps
Automatic Labeling
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Current Effort
Q4 / 2020
Q2 / 2021Q1 / 2021
Technische Hochschule Ingolstadt | CARISSMA
E-Mail: [email protected]
www.thi.de/go/thi-labeling-tool
AccessDownload
17
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
Thank you!
18
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
Questions?
19
17th International Conference onImage Analysis and Recognition
ICIAR 2020
Technische Hochschule Ingolstadt | CARISSMA
[1] Ambardekar, A., Nicolescu, M., Dascalu, S.: Ground truth verification tool (GTVT) for video surveillance systems. In: 2009
Second International Conferences on Advances in Computer-Human Interactions, pp. 354–359. IEEE (2009)
[2] Dutta, A., Zisserman, A.: The VIA annotation software for images, audio and video. In: Proceedings of the 27th ACM International
Conference on Multimedia, MM 2019. ACM, New York (2019). https://doi.org/10.1145/3343031.3350535
[3] Jaynes, C., Webb, S., Steele, R., Xiong, Q.: An open development environment for evaluation of video surveillance systems. In:
PETS02, pp. 32–39 (2002)
[4] da Silva, J.L., Thomas Brandmeier, A.Z.: Automatic measurement of automobile drivers attention level via computer
vision. In: XXIV Congresso Brasileiro De Engenharia Biom´edica (2014)
References
20
17th International Conference onImage Analysis and Recognition
ICIAR 2020