DIY: Computer Vision with GWT.

transcript

DIY: computer vision with GWT

Francesca Tosi Alberto Mancinifrancesca@jooink.com alberto@jooink.com

Alberto Software Developer & Linux Sysadmin

C/C++ ( since 80’s :/ )GWT ( since 1.6 )

Javascript & Java

alberto@jooink.com+AlbertoMancini

FrancescaWeb and mobile developer,software engineer and architect

C/C++, CFD, HPCGWT since 2009

“with a passion for clean code and fine tuned details”

francesca@jooink.com@francescatosi+FrancescaTosi

Devfest’s CodeProject

A couple of years(**) ago we started working on a pet project trying to explore how mixing modern WebAPIs and an effective programming approach(*) can enable us to develop complex client-side applications running in a browser.

(*)read: GWT, i.e. a typed language compiled into js

(**)for the 2012’s GDG DevFest season

Anybody knows

Browsers are a good

“execution environnement”

for evoluted web-pages ….

Browsers are

enough ?

I Mean ...

C’mon we are in 2015:

browsers are certainly more than enough

for any kind of application !

GoogleDocs, Inbox, Plus, Scala vm in js, JsDOSBox…

BETting

The right tool(*) is enough to let us to develop

- complex - compute-intensive

browser-based applications ?

(*) GWT of course :)

The RIGHT TOOL

Should (at least) let us:● master the (quite tricky) web platform● code safely … in a type-safe language

(but provide native-like flexibility and performance)

● reuse existing code

… and of course produce a fast and reliable application

NO,no way

Our forecast was

Target

We were working on Augmented Reality and we had an expertize on it so we started trying to realize an AR webApp in order to show that this kind of applications is way too-complex for the performance of a browser.

Augmented Reality (browser based)

Marker BasedAugmented Reality

Marker Based AR

Essentially composed of 3 steps

● Marker detection and identification(finding the marker in a picture)

● Computation of the marker pose(det. six degrees of freedom w.r.t camera, f.i.)

● Syntetic objects collocation (object space to camera space transf)

Marker Based AR

NyARToolkit, do not reinvent the wheel

NyARToolKit is an ARToolKit built with 100% pure Java.

[great performance, featurefull, GPL]

http://nyatla.jp/nyartoolkit/wp/

GWT-NyARToolkit

… given an image we get a 3x4 (3x3 + a translation vector actually) matrix that identifies the marker in the camera space

nyar.update(i_sensor); ← ‘push’ an image

found = nyar.isExistMarker(marker_id); ← query if a marker is detected

mm = nyar.getMarkerMatrix(marker_id); ← compute the collocation matrix

https://code.google.com/p/gwt-nyartoolkit/http://jooink.blogspot.com

Browser Based AR

GWT … glue

Acquiring Video (and eventually Audio)

WebRTC

Acquiring Video (and eventually Audio)

Acquisition of Audio/Video stream via simple js api.

Why WebRTC ?

Why WebRTC ?Plugins for IE

e.g.Temasys provides ad ad hoc craftedreplacement ofadapter.js

Audio/Video

Acqusition Audio/Video stream via simple js apis.

navigator.getUserMedia(constraints,successCallBack,errorCallBack)

WebRTC with GWT

GWT (Elemental at the beginning)

At the beginning elemental, but right now (using jnsi and now jsinterop) we are able to do all the operation without dependencies GWT.

https://code.google.com/p/elemental-getusermedia-demo

https://github.com/jooink/gwt-getusermedia

Browser Based AR

the video stream

can (has to be) be played in a <video/> tag

“native” (jsni)

video.src = URL.createObjectURL(stream);

Browser Based AR

canvasCtx2D.drawImage(video, ...);

Browser Based AR

canvasCtx2D.drawImage(video, ...);

canvasCtx2D.getImageData(sx, sy, sw, sh)

Browser Based AR

<video/>Ny

Browser Based AR

3D models to be shown on the marker

after the recognition

Browser Based AR

WebGLWe had to reinvent the wheel :(

three.js, BabylonJS

parallax (GWT !)

awesome shading rendering and lighting

tend to mix webgl-interaction with 3D graphics

WebGL, gwt-webgl

<video/>Ny

https://code.google.com/p/gwt-webgl

based on Elemental with some modifications

damn thin wrapper around webgl standard

Tiny3D

● 3D geometry (4x4 & affine matrices)

● Lambertian and Phong lighting

● Polygons, Polyhedra, normals etc..Wavefront OBJ import

Browser Based AR

picshareWe lost the BET

http://picshare.jooink.com

Browsers are enough!

GWT is more than enough!

GWT Widgets & morefor mobile

FirefoxOS

Web developerspreferredmobile OS?

Browser Based AR

Browser Based AR

Computer Vision

As told by Alberto, after a while we needed to have some more complex features on this tool for image recognition, in order to have more information on the image we are working with.

Computer Vision

There exists a lot of CV libs (complete, tested and fine tuned) for example:

- OpenCV- VLFeat- VLX- …

Computer Vision

There exists a lot of CV libs (complete, tested and fine tuned) for example:

- OpenCV- VLFeat- VLX- …

- BoofCV

Computer Vision

Computer Vision by BoofCV

There is a great library for Computer Vision interely written in Pure Java: BoofCV → we decided to adapt BoofCV for GWT.

BoofCV

BoofCV is a complete & complex project

BoofCV is an opensource Java library for computer vision & robotics applications.

BoofCV is organized into several packages: image processing, features, geometric vision, calibration, recognition, visualize and IO.

BoofCV

Written by Peter Abeles

www.boofcv.org

Link to the source code:

https://github.com/lessthanoptimal/BoofCV

BoofCV

Adapting an entire big & complex library is not is a funny game.

BoofCV depends on other Java libs, so we must adapt all the dependencies

BoofCV

- DDogLeg: Java numerical lib for non-linear optimization, robust model-fitting, polynomial root finding, sorting and more;

- GeoRegression: open source java geometry lib for scientific computing;

- EJML: dense matrix linear algebra library written in Java

BoofCV- GWT-DDogLeg:

https://github.com/jooink/gwt-ddogleg

http://gwtddogleg.appspot.com/

- GWT-GeoRegression: https://github.com/jooink/gwt-georegressionhttp://gwtgeoregression.appspot.com/

- GWT-EJML: https://github.com/jooink/gwt-ejml

http://gwtejml.appspot.com/

- EJML: dense matrix linear algebra library written in Java

https://github.com/lessthanoptimal/ejml

GWT-EJML

is an adapter-project around EJML that let us to use the original library:

- provides .gwt.xml files - supersources many classes - contains running examples

GWT-EJML

published on github

https://github.com/jooink/gwt-ejml

GWT-EJML

org.EJML.gwt.xml

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.6.1.rc1//EN" "http://google-web-toolkit.googlecode.com/svn/tags/2.6.1.rc1/distro-source/core/src/gwt-module.dtd">

</module>

GWT-EJML

.gwt.xml

<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0//EN"

"http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">

<super-source path="jre" />

</module>

GWT-EJML

The Strategy

we decided to use the SuperSourcing in order to change/adapt the class of the basic lib and leave the ejml project unchanged

→ the maintainability is easier

GWT-EJML

SuperSourcing

We added the super source path in the gwt.xml config file

and we created the file to modify in that directory (using the right path of the file in the main project).

GWT-EJML

.gwt.xml

</module>

GWT-EJML

Supersourced parts of JRE

java.io.ByteArrayOutputStream

public class ByteArrayOutputStream extends OutputStream {

public ByteArrayOutputStream() { … }

public void write(int arg0) { … }

GWT-EJML

Supersourced parts of EJMLorg.ejml.ops.MatrixIO;

public class MatrixIO { … }

org.ejml.alg.dense.decomposition

.eig.symm.SymmetricQREigenHelper

org.ejml.alg.dense.decomposition

.svd.implicitqr.SvdImplicitQrAlgorithm

java.io.PrintStream

printf()

GWT-EJML

Supersourced parts of EJML

java.io.PrintStream does not have printf() in the gwt emulated jre

We could not supersource the PrintStream because already exist on emulated jre and we cannot super-sourcing just a method.

GWT-EJML

https://github.com/jooink/gwt-ejml

http://gwtejml.appspot.com/

DDogLeg

- DDOGLEG: Java numerical lib for non-linear optimization, robust model-fitting, polynomial root finding, sorting and more;

https://github.com/lessthanoptimal/ddogleg

GWT-DDogLeg

.gwt.xml <?xml version="1.0" encoding="UTF-8"?>

</module>

GWT-DDogLeg

Supersourced parts of DDOGLEG:

java.lang.OutOfMemoryError

Unchecked Exception …

GWT-DDogLeg

Supersourced parts of DDOGLEG:

org.ddogleg.struct.FastQueue

org.ddogleg.struct.FastQueueList

→ Reflection (not supported by gwt-jre)

GWT-DDogLeg

https://github.com/jooink/gwt-ddogleg

http://gwtddogleg.appspot.com/

GEOREGRESSION

- GEOREGRESSION: open source java geometry lib for scientific computing;

https://github.com/lessthanoptimal/georegression

GWT-GEOREGRESSION

- GWT-GEOREGRESSION: minor modifications

GWT-GEOREGRESSION

<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0//EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">

</module>

GWT-GEOREGRESSION

https://github.com/jooink/gwt-georegression

http://gwtgeoregression.appspot.com/

BoofCV

- BOOFCV: fast computer vision library written entirely in Java;

https://github.com/lessthanoptimal/boofcv

<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0//EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">

</module>

BoofCV

GWT-BOOFCV:

● Reflection

● Image managment

BoofCV

GWT-BOOFCV:

● Reflection: ○ used mostly to handle Arrays of primitive

types○ probably unneeded upstream

working with Peter to understand if can be avoided through, for instance, a few of factories.

● Image managment

BoofCV

Reflection:

java.lang.reflect.Array

Array.newInstance( … )

Array.getLength( … )

BoofCV

GWT-BOOFCV:

● Reflection

● Image managment

BoofCV

Image managment:

of course we had to work-out the different point of view about Images:

Browser → (<img/> <video/>)

BoofCV → array of values

→ we used <canvas/> (also in this case);

canvas.getContext2d().getImageData( … );

BoofCV

EXAMPLES:

- Interest Point detection

- Interest point association

BoofCV

EXAMPLE 1

Interest Point detection:

ImageFloat32

DetectDescribePoint<ImageFloat32,SurfFeature>detDesc.detect(boofImage);

BoofCV

- Interest point

detection

http://static.jooink.com/experiments/boofcv-gwt/interest-points/

BoofCV

- Interest point

association

Questions? Contact us :

Francesca TosiR&D at Jooink Teamfrancesca@jooink.com

Alberto ManciniDev at Jooink Teamalberto@jooink.com

Thanks !Please rate this presentation http://gwtcreate.com/agenda