Building droids with JavaScript

Andrew Fisher @ajfisher!MelbGeeks / Be Reponsive, September 2014

Hi! My name is Andrew Fisher and I’m an interaction researcher. Today I want to talk to you about web connected hardware and building droids with javascript for the next 30 minutes.

Video

Video of this talk is available at:!!

http://vimeo.com/107036858

A Video of this talk is available at http://vimeo.com/107036858 if you want to watch it there instead.

JS all the real things

flickr (cc) Quasimondo

Hardware for a long time was really hard. My initial interest in computing really came from an interest in electronics. But the reality was that for a kid growing up in the late 80s, early 90s, building serious hardware was prohibitively expensive and required facilities that most kids didn’t have access to in those days - even the idea of something like a hacker space or a makerspace didn’t exist then. !Doing anything non-trivial in hardware was a very difficult undertaking. As a result, I moved to software. As you’d expect, in the world of software, the effort to reward ratio was far better than for hardware, especially for the attention span of a teenager. !And this situation remained the case until the early 2000s. !!Image: http://www.flickr.com/photos/quasimondo/5203908319

Hardware is more like software

(cc) Phill Farrugia

Until this came along - this is an arduino and it has changed the way hardware is thought of. Hardware was no longer expensive - these were designed at the time to be less than 20 euros and the designs were open sourced, meaning anyone could make them. Today you can get them for $8. Sophisticated hardware became accessible to students, artists, kids and web people with little repercussion for blowing one up. With it, this community has brought ideas around design, user experience, art and software and architectural principles. !More recently, over the last 2 years, some great work has been done in the node community getting JS to work with hardware like this - to the point where working with hardware using javascript is now extremely easy. !So tonight, I want to talk to you about that and how all of you can all start working with hardware with JS and along the way we’ll bump into some robots. !http://phillfarrugia.com/post/92908852193/spent-the-day-building-robots-at-nodebotsau

TODO

1. Different ways to interface with hardware!2. The JS / hardware stack!3. Applications and examples

So to do this, we’ll look at. !How we can interact with hardware. !Some code to show what a common JS hardware stack starts to look like !Then we’ll play with some examples.

Working with hardware

flickr (cc) Oskay

Let’s start by looking conceptually at how we develop with hardware. I think about this at sort of three levels. !At the metal At the device And somewhere in between. !Image: http://www.flickr.com/photos/oskay/2310115216/

Working with the metal

flickr (cc) Wonderlane

Here we are usually writing code specifically for a chip or board at a very low level. It usually means writing C or C++. You can get a bit of abstraction with hardware libraries but even with good libraries you need to write a lot of code. !You have insane limitations. It’s like going back to the 80s - an arduino for example has TWO KILOBYTES of RAM. !Image: http://www.flickr.com/photos/wonderlane/3198166347

Working low level

flickr (cc) lisovy

Playing at the chip level gives you masses of control but your iteration cycles take time and you’re constantly thinking about things like memory and garbage collection and not frying a chip. Debugging is also generally a nightmare. !Image: http://www.flickr.com/photos/lisovy/4677688431/

Device hacking

flickr (cc) Roo Reynolds

So At the other end of the spectrum there are devices that are already made. This is hardware with APIs - maybe they are via the network, maybe via an embedded web server or some sort of serial protocol. !However it works, you’re basically interacting with the hardware given to you as a service. !You can build clever ways of working with the service but fundamentally you can’t change the way the hardware works. !IMage: http://www.flickr.com/photos/rooreynolds/9350631793

AR.Drone

flickr (cc) neeravbhatt

A good example of this is the AR Drone - you can play with using a node library to interact with the copter directly to do some really interesting things but you can’t fundamentally change the hardware. Having said that, people have done some fantastic things at this level. !So there’s plenty of very interesting devices you can grab hold of and start playing about with really easily. !http://www.flickr.com/photos/neeravbhatt/6885424870

Hardware in comfort

flickr (cc) Zack Hoeken

In between these extremes though is a new category, where you can work with and change the hardware but you can still use high level languages to do it. This is a great combo as it allows for rapid prototyping but also makes hardware more accessible to non-embedded application designers - people like us - web designers and developers to be able to tinker with. !!!Image: http://www.flickr.com/photos/hoeken/3519955473/

Sketching in hardware

flickr (cc) Camille Moussette

So this approach give you the ability to prototype rapidly as well as work almost directly with the hardware. There are some limitations around some hardware but to start off with and for some prototyping that doesn’t matter. !To show you how easy this is let’s make something here and now. !http://www.flickr.com/photos/9225693@N08/6051548279

First stepsWe’re going to focus on the arduino. !This is an arduino - they cost about $30, they are awesome and come in many different forms for different applications from small to large. They run off USB or a battery so it’s hard to blow them up and even harder to electrocute yourself. There is also huge amounts of community information about them. !Now the arduino is way too under powered to run JavasScript so we’re going to use our computer to do that. As such we just need to pass messages down the wire to the arduino over the USB cable so for that we use a protocol called Firmata. !So Firmata allows you to tell the arduino to do things like turn a pin on or off, take a reading etc. Firmata is a neat idea as it exposes nearly all of the features of the arduino but via a protocol so now you can control it from somewhere else that talks the firmata protocol… !Hang on - that sounds suspiciously like an API…

Hardware hello world

flickr (cc) pj_vanf

So we plug in an LED (this one is bigger than normal so you can see it easily) and then we’ll need to write some code. !Go interactive version here to show connecting and turning an LED on and off. !I’ve got a little bit of scaffolding here so I don’t have to type everything. But you can see it just creates a board and connects it. Now it’s connected we can tell the board we want to assign a pin as an output then we’ll tell it to make the pin go HIGH which means send it some volts - we do that with the digitalWrite command. Then we can make it go LOW and as you can see as I do this it turns the LED on and off. !http://www.flickr.com/photos/vanf/5210360116 !

Web thing hello world

http://github.com/ajfisher/nbscaffold

So we’re all web devs here so let’s not stray too far away from our comfort zone. Let’s wrap a web interface around this light so we can turn it on and off with a click of a button on a page. !We can’t do this interactively very easily so I’ll show you some code. !This is a bit of overkill but we’re going to use express and web sockets so it’s a bit more realtime but also it will lay the foundation for what I’ll show you next. !We set up the socket messages on the server to switch things on and off . On the client side all our HTML and JS is doing is just sending messages on click so nothing too interesting. !And there we go - button click to turn a light on and off via a web browser.

The JS hardware stack

Arduino (Sensors and actuators)!Firmata (Communications protocol)!NodeJS (Application logic)!WS/HTTP (Network & security protocols)!Client (UI, Input, Visualisation)

So this is what the JS hardware stack looks like. !We’ve got an arduino with sensors and actuators. Firmata which provides the communications protocol. The NodeJS application gives us application logic and integration with other libraries. !Networking is provided over http or websockets and this can give us security and encryption. !Finally the web page for the client gives as input methods, a data viz layer as well as user interface.

Easy install

npm install firmata express socket.io

So this stack can be created with pretty much just this command plus an arduino with firmata on it in about 2 minutes. !Which is pretty cool. And it’s robust enough now that it’s most likely going to work the first time you try it. !Now it’s pretty cool to be able to connect a web page to a bit of hardware right?

Hardware as objects

var led = new five.Led(13);!led.blink(1000);!led.stop();!led.on()

Wouldn’t it be great if we could do something like this rather than telling the pins to turn on and off? !Wouldn’t it be cool to have some sort of abstraction for LEDs which are super common and be able to interact with them in standard ways? !Well we can, thanks to a project called Johnny Five that was kicked off about 2 years ago by a chap called Rick Waldron.

Johnny FiveJohnny Five uses the firmata protocol we were just using but provides abstractions for common electronic devices. Things like LEDs, sensors, motors, accelerometers - all sorts of things. The list keeps growing all the time and there’s now over 60 committers to the project. !This means we can prototype really, really fast and build cool things.

Build stuffThis was a pulse oximeter that someone made with an LED, a light sensor and some signal processing in JS

Build stuffThis was the beer-o-matic 2000 - a drinks dispenser that gave you beer when light was on the hat.

Build stuffA nerf gun attached to a build system to find and then shoot the developer that broke a build.

Build stuffAnd of course, robots.

Looking for droids?

flickr (cc) solo

So let’s look at a robot that is built with web tech. !Image: http://www.flickr.com/photos/donsolo/3768623542/

NodebotThis is a bot I built from scratch - The core was just some metal and some motors. !All of the control and interface for this is web tech. Node, HTML, CSS, JavaScript

Nodebot architecture

RPiArduino

Battery

4WD motor control

Sensors Firmata

NodeJS!- Express!- Johnny Five!- Open CV!- Socket IO

WiFi

Camera

4WD motor control

Architecturally this is what it looks like: !The arduino deals with the hardware using firmata. !A raspberry pi runs a node server and communicates to the arduino over a serial connection. !The webcam provides a video feed. !A web interface shows the video and provide input control to drive the bot using the device API and the gyro in your phone or tablet or keyboard on your desktop. !And here it is here…

Go and make

Arduino.cc (start here for everything)!Johnny Five (npm install johnny-five)!Node ARDX (node-ardx.org)!Minimal examples (github.com/ajfisher/nbs)!SimpleBot (github.com/nodebotsau/simplebot)!!

http://nodebotsau.github.io!

As the web starts extending its reach into more and more physical territory, we need to translate our skills into that of physical design and engineering along with it. Hardware is still ruled by traditional electronics engineering and design - which mean we end up with interfaces on our hardware that are appalling. The skills all of you have in this room are immensely applicable to the physical world, experience design, interface design, architecture, engineering, development. !So I’d like to encourage all of you to to play around with some hardware and take your skills physical. !The stack I’ve talked about is very solid, it works really well and will allow you to experiment very efficiently. You can all do this and I’m sure build a robot much better than this one I’ve got here. !Don’t be worried about playing with hardware - it’s loads of fun - just stay away from mains electricity. !These links should get you started. !We run NodeBots events here in Melbourne and around Australia. The next one is on October 1st at the Melbourne HackerSpace but go here and you can find out more.

Building droids with JavaScript

Andrew Fisher @ajfisher!MelbGeeks / Be Reponsive, September 2014

So thanks a lot and if you have any questions now or after please let me know.