Unified Benchmarking of Big Data Platforms The HOBBIT Platform€¦ · ALotofTools 2 2 Ngonga Ngomo...

Unified Benchmarking of Big Data PlatformsThe HOBBIT Platform

Axel-Cyrille Ngonga Ngomo

Horizon 2020GA No 688227

01/12/2016–30/11/2018

Apache Big DataSevilla, Spain

November 11, 2016

Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 1 / 42

A Lot of Data

1

1http://www.ibmbigdatahub.com/infographic/four-vs-big-dataNgonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 2 / 42

http://www.ibmbigdatahub.com/infographic/four-vs-big-data

A Lot of Tools

2

2https://cloudramblings.me/Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 3 / 42

https://cloudramblings.me/

A Lot ... of Tools

33http://mattturck.com/2016/02/01/big-data-landscape/


http://mattturck.com/2016/02/01/big-data-landscape/

A Lot of Views

44https://steemit.com/philosophy/@l0k1/

subjectivity-and-truth-how-blockchains-model-consensus-buildingNgonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 5 / 42

https://steemit.com/philosophy/@l0k1/subjectivity-and-truth-how-blockchains-model-consensus-building

https://steemit.com/philosophy/@l0k1/subjectivity-and-truth-how-blockchains-model-consensus-building

Core Questions

Developers: How good is my tool?Vendors: Who is my tool good for?Users: Which tool(s) should I use formy application?


Many Questions

Where are the current bottlenecks?Which steps of the data lifecycle arecritical?Which solutions are available?Which key performance indicatorsare relevant?How well do or should toolsperform?How do existing solutions performw.r.t. relevant indicators?


SolutionBenchmark

ComponentsDataset(s), e.g., Twitter stream, sensor dataTask(s), i.e., NER, NEL, ingestionKey Performance Indicators, e.g., precision, recall


SolutionBenchmark

ComponentsDataset(s), e.g., Twitter stream, sensor dataTask(s), i.e., NER, NEL, ingestionKey Performance Indicators, e.g., precision, recall


ChallengesDataset Mismatch

Year

ACE

Wiki

AQUA

INT

MSN

BC

IITB

Meij

AIDA

/CoN

LL

N3collection

KORE

50

Wiki-D

isamb3

0

Wiki-A

nnot30

Spotlight

Corpus

SemEv

al-2013task

12

SemEv

al-2007task

7

SemEv

al-2007task

17

Senseval-3

NIF-based

corpus

Micr

oposts2014

Softw

areavailable?

Webservice

available?

Cucerzan 2007 3Wikipedia 2008 3* 3MinerIllinois Wikifier 2011 3 3 3* 3 3Spotlight 2011 3 3 3AIDA 2011 3 3 3**TagMe 2 2012 3 3 3 3Dexter 2013 3 3KEA 2013 3WAT 2013 3 3AGDISTIS 2014 3 3 3 3 3 3 3 3 3 3Babelfy 2014 3 3 3 3 3 3 3NERD-ML 2014 3 3 3 3

BAT- 2013 3 3 3 3 3 3 3* 3FrameworkNERD 2014 3 3 3 3 3FrameworkGERBIL 2014 3 3 3 3 3 3 3* 3 3 3 3 3 3 3


ChallengesUnclear KPI Semantics

ExampleWhich time do we measure?

First or last result?With or without network delay?



ExampleWhich time do we measure?

First or last result?With or without network delay?



ExampleWhen is an annotation correct?

Weak or strong annotation?Semantically equivalent or exact URI?



ExampleWhen is an annotation correct?

Weak or strong annotation?Semantically equivalent or exact URI?


SolutionUnified Benchmarking Framework

Benchmark Core

Web service calls

Dataset Wrapper

Web service calls

Interface View

AnnotatorWrapper

Interface View

Open Datasets

Configuration(Model)

...

Benchmark Core

Your Annotator

Your DatasetDataHub.io

GERBIL Core Controller

Persistent Experiment Database

(Model)


GERBILOverview

Evaluation platform for NER/NEL18 reference annotation systems32 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users


GERBILOverview

Evaluation platform for NER/NEL18 reference annotation systems32 reference datasetsBenchmarking 10× faster

Archiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users


GERBILOverview

Evaluation platform for NER/NEL18 reference annotation systems32 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysis

Open-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users


GERBILOverview

Evaluation platform for NER/NEL18 reference annotation systems32 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users


GERBIL

Annotator TasksNIF-based Annotators 2519Babelfy 958DBpedia Spotlight 922TagMe 2 811WAT 787Kea 763Wikipedia Miner 714NERD-ML 639Dexter 587AGDISTIS 443Entityclassifier.eu NER 410FOX 352Cetus 1Overall 24.3K exps

50+ papers


HOBBIT

Rationale

A community-driven benchmarking framework for the community

Focus on Big (Linked) DataBuild upon 24.3K experiments performedwith GERBILCover all steps of the Linked Data lifecycle

Used by a growing number of companiesMature and maturing technologies

Open benchmarks based on industrial dataand use cases


HOBBIT

Rationale

A community-driven benchmarking framework for the community

Focus on Big (Linked) DataBuild upon 24.3K experiments performedwith GERBILCover all steps of the Linked Data lifecycle

Used by a growing number of companiesMature and maturing technologies

Open benchmarks based on industrial dataand use cases


Aims

1 Gather real requirementsPerformance indicatorsPerformance thresholds

2 Develop benchmarks based on real data3 Provide universal benchmarking platform

Standardized hardwareComparable results

4 Periodic benchmarking challenges5 Periodic reporting6 Found independent Hobbit association


Overview

Data Collection

Industrydata

Measure Collection

Benchmark Creation

Benchmark 1

KPIsTasks

KPIsTasksKPIsTasks

KPIsTasks

KPIsTasks

KPIsTasks

Benchmark 2

Benchmark n

HOBBITPlatform

Solution 1

Solution k

Solution 2

Challenges

Reports

Participants/Community


Survey

QuestionsIn what areas are organizations active?What do people expect from benchmarks?How are benchmarks being used?

Profile CountSolution providers 56Technology users 67Scientific community 65


SurveyCan your solution be benchmarked?


SurveyDo you benchmark your solution?

Own datasets and settings in many casesOwn implementations of measuresResults not comparable


SurveyApplication Areas


HOBBIT PlatformFeatures

Uses established deploymenttechnologies (Docker)

Decoupled componentsBenchmark and Systems can bewritten in different languages

Uses scalable message queues forcommunication

Open-source implementation

Supports distributed benchmarksand systemsOnline instance on server cluster


HOBBIT BenchmarksFeatures

Addresses all steps of the LinkedData Lifecycle

Benchmarks derived from industryuse casesReal data under the bechmarksScalable size of benchmarks

Open-source implementation


HOBBIT PlatformBenchmarks

Streaming and static deterministic benchmarksRealistic benchmarksControlled volume and velocity

Generation and AcquisitionConversion of XML into RDFEntity recognition and linkingRelation extraction

Analysis and ProcessingLink DiscoveryMachine LearningSupervised and unsupervised

Storage and CurationTriple storesVersioningIncl. updates

Visualization and ServicesQuestion AnsweringFaceted BrowsingUsage-based benchmarks


HOBBIT PlatformArchitecture

PlatformController

Data Generator

Task Generator

Data Generator

Data Generator

Task Generator

Task Generator

Front End

Benchmarked System

data flowcreates component

StorageAnalysis

BenchmarkController

Evaluation Module

Eval. Storage

Logging


HOBBIT PlatformBenchmark Initialization

PlatformController

Data Generator

Task Generator

Data Generator

Data Generator

Task Generator

Task Generator

Benchmarked System


Storage

BenchmarkController

Eval. Storage


HOBBIT PlatformBenchmark Execution

PlatformController


Storage

Data Generator

Task Generator

Data Generator

Data Generator

Task Generator

Task Generator

Benchmarked System

BenchmarkController

Eval. Storage

ex:Entity rdf:type ex:Class...



PlatformController


Storage

Data Generator

Task Generator

Data Generator

Data Generator

Task Generator

Task Generator

Benchmarked System

BenchmarkController

Eval. Storage

vex:Entity...

SELECT ?vWHERE { ?v a ex:Class }



PlatformController


Storage

Data Generator

Task Generator

Data Generator

Data Generator

Task Generator

Task Generator

Benchmarked System

BenchmarkController

Eval. Storage

X

vex:Entity...


HOBBIT PlatformBenchmark Evaluation


PlatformController

Storage

BenchmarkController

Evaluation Module

Eval. Storage

precision=...recall=...F1-score=... precision=...

recall=...F1-score=...

benchmark parameters: ...

vex:Entity...

vex:Entity...


DatasetsTWIG

Goal: Simulate real Twitter FirehoseRelies on 476 million tweets as training dataMimicking algorithm based on

Distribution of character frequenciesDistribution of transportation frequencyNetwork topology


DatasetsLinkedConnections

Goal: Simulate real transport networkReal transportation data from Belgium for trainingMimicking algorithm based on

Observed correlation between population density and transportationDistribution of transportation frequencyNetwork topology


DatasetsPrinting Machinery

Goal: Simulate events from printing machineryMimicking algorithm using event correlations and distributions

Changing plate

Double sheet

Early sheet

Finish job

Misaligned sheet

Missing sheet

Operation partially completed

Performance

Printing interval

Produktion Good Sheet

Side guide warning

Start job

Washing blanket

Washing impression cylinder

Washing ink rollers

with washing ink fountain roller

with washing plates

Mai 01 00:00 Mai 01 06:00 Mai 01 12:00 Mai 01 18:00 Mai 02 00:00Time

Eve

nts


DatasetsWeidmüller

Goal: Simulate events from injection molding machineryMimicking algorithm using event correlations and distributions


DatasetsSemantic Publishing

Goal: Simulate data from the BBCGenerator based on manually configurable set of correlations


Join HOBBIT

Join the HOBBIT communityProvide KPIsProvide datasetsJoin the platform developmentFollow us on Twitter

https://twitter.com/hobbit_project



HOBBIT Benchmarks

Streaming and static deterministic benchmarksRealistic benchmarksControlled volume and velocity

Generation and AcquisitionConversion of XML into RDFEntity recognition and linkingRelation extraction

Analysis and ProcessingLink DiscoveryMachine LearningSupervised and unsupervised

Storage and CurationTriple storesVersioningIncl. updates

Visualization and ServicesQuestion AnsweringFaceted BrowsingUsage-based benchmarks


HOBBIT RunRuntimes


HOBBIT RunEffectiveness

System Precision Recall F1-measureFOX 0.515 0.310 0.351Balie 0.369 0.230 0.249Illinois 0.500 0.288 0.327OpenNLP 0.442 0.241 0.285Stanford 0.486 0.303 0.335


HOBBIT RunExample results

A2KB, weak annotation match, Micro F1-measure

System AIDA

/CoN

LL-Com

p.

IITB

KORE

50

MSN

BC

Micr

op.2014-Train

N3-Re

uters-128

AIDA 0.668 0.141 0.625 0.622 0.363 0.391Babelfy 0.448 0.129 0.564 0.423 0.311 0.289DBpedia Spotlight 0.545 0.262 0.341 0.457 0.448 0.320FOX 0.512 0.100 0.268 0.127 0.309 0.518FREME NER 0.358 0.074 0.160 0.208 0.254 0.263WAT 0.673 0.137 0.543 0.631 0.403 0.480xLisa 0.363 0.233 0.352 0.365 0.322 0.274


Thank You

http://project-hobbit.eu/get-involved/

http://goo.gl/forms/1iRIoG4Xpb



http://project-hobbit.eu/get-involved/

http://goo.gl/forms/1iRIoG4Xpb


Acknowledgment

This work was supported by grants from the EU H2020 Framework Programmeprovided for the project HOBBIT (GA no. 688227).


Date post:	20-May-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Unified Benchmarking of Big Data Platforms The HOBBIT Platform€¦ · ALotofTools 2 2 Ngonga Ngomo...

Documents