Using Clojure, NoSQL Databases and Functional-Style JavaScript to Write Gext-Generation HTML5 Apps

transcript

Mallorca, 24 September 2010

Presentation @ HackFwd Build 03, Release 0.1

TheDeadline GmbH & Co. KG

Using Clojure, NoSQL Databases and Functional-Style JavaScript to Write Next-Generation HTML5 Apps

TheDeadline GmbH & Co. KGPresentation @ HackFwd Build 03

Release 0.12

Since 1999 I have been running a successful software development company. In 2009 we started building TheDeadline with Clojure. We are a HackBox!And I am a Board Member of HackFwd: Eat your own dogfood!> whoamiStefan Richter

Apple IIPascal,C64,6502

VAX, Unix, C,4GLs,SmalltalkLisp,Scheme

Unix, C/C++,4 GLs,Windows NT,VB (!)

University studies and work as a programmer in research institutes

Working as a professional programmer

Running my own company

20102005199819951990198719831966

Linux, Java,Python, RubyObjective-C, C/C++Common Lisp, Clojure, JavaScript

Release 0.1

Please pardon the colors.

Release 0.1

The Web is dead.

Release 0.1

Okay. We all know that more and more people are accessing the internet via mobile devices (smartphones).

Release 0.1

By accident, Apple ”invented“ the app economy.

Release 0.1

People are buying specialized apps which are bound to a specific platform (iPhone/iPad, Android, etc.).

Release 0.1

The Wired magazine calls these platforms ”walled gardens.“

Release 0.1

Because: These apps use the internet for data transmission and networking, but they are not on the web.

Release 0.1

It seems like people are building more and more stuff for ”walled gardens“!?

Release 0.1

Do we really have to build for ”walled gardens“?

Release 0.1

No. Because it is a waste of time to build apps that run on just one platform.

Release 0.1

And it is a waste of time to implement the same app for different platforms.

Release 0.1

We (TheDeadline-Team) had the same problem: We needed mobile clients for iPhone/iPad and Android.

Release 0.1

But then we would have to maintain at least three different platforms: Web, iPhone, Android.

Release 0.1

So, every feature would require three times the effort.

Release 0.1

Or: Our team would become three times slower.

Release 0.1

Our solution: Build an unified HTML5 client that runs in all modern web browsers as well as on both iPhone and on Android.

Release 0.1

As you all know, HTML5 is basically a set of browser APIs and HTML extensions. Some of them can be used to build a web client that feels like a native client.

Release 0.120

Release 0.1

HTML5 can do much more (video, 2D canvas, etc.), but we don’t need it for our core app.

Release 0.1

What an ”HTML5 client“ is depends on the type of app you are building.

Release 0.1

My definition of an HTML5 client:

- App runs mostly inside the browser (JavaScript)- App can be used offline- Syncs data automatically with server- Feels more like a native app, but no WIMP interface (use CSS3)- Supports only modern browsers, no IE6/IEx fallbacks ;)

Release 0.1

So, what parts of the HTML5 standard do we need to implement this?

We need:- Client-side storing of data - Client-side caching of UI- Optional: - New stylable HTML5 elements - CSS3 media queries

Release 0.1

We are using the Local Storage API to store all user data locally on the client side.

Release 0.1

We are using the Caching API to cache the static UI parts on the browser-side.

Release 0.127

Google App Engine

Clojure

AppEngine SDK for Java

Datastore TaskQueues

Compojure

Browser

JavaScript

JQuery (mostly)

Before: a request/response style web-application.

HTTP/SSL HTML/CSS/JS

Release 0.1

Almost everything was rendered on the server. (Okay, we had some AJAX, too. But not much.)

Release 0.1

For the server-side rendering, we are using Compojure and Hiccup.

Release 0.1

On the next slide, you can see how nice it is to have your application code, HTML and CSS in one place.It is also a good example of how to use multimethods for UI rendering. (You don‘t need a web-framework when you are using Lisp).

Release 0.131

Release 0.1

You see, we are not only strong supporters of the NoSQL movement, we want to suggest a new movement:

The No Templates movement.;)

Release 0.1

Too bad that we can’t use this on the client side. So, how can we render the UI on the client side?

Release 0.1

Most people would shout: JQuery!

Release 0.1

The idea behind JQuery is to execute (chained) function calls on a set of DOM elements. This is why JQuery is so easy to learn and to use. You don’t even have to know how to program JavaScript!

Release 0.1

But can you build a large HTML5 client that consists solely of JQuery plugins which just operate on the DOM-tree?

Release 0.1

You will end up with lots of code blocks that all iterate over the same DOM tree.For a large client, we think you need to bring more structure to your code.

Release 0.1

So we tried Google Closure Library. Google Closure Libary is to JavaScript what the JDK is to Java:a mature standard library.

Release 0.1

And it helps you to structure your code by providing a namespace system.

Release 0.1

BTW, you can’t compare JQuery and Google Closure using simple examples. It isn’t a question of writing five lines of elegant code. The question is wether you can write thousands of lines of complex code ...

Release 0.1

I have to admit: 10 years ago I hated JavaScript.The code we received from web agencies was such a mess. It was slow. The JavaScript interpreters inside the browser were broken. It was impossible to see the beauty behind the beast ...

Release 0.1

Today most people are trying to structure JavaScript code like Java code.

Release 0.1

I think this is wrong: JavaScript may look better now, but it is bloated in the same way that Java is ...

Release 0.1

Let’s stop the bloat:We are already using a functional programming language on the server side (Clojure).Of course, we want to program in the same style on the client side!

Release 0.1

What do I mean by ”functional programming“*? - First class functions- Just returning values- Causing no side effects- Functions as parameters- Function combination

*(Yes, this definition is too simple)

Release 0.1

JavaScript makes it all possible.

Release 0.1

And: Google Closure comes with a library of functions that enables you to work with collections of data in a functional programming style.

Release 0.1

Check goog.functions:

Release 0.149

Release 0.1

Check goog.array:map, reduce, every, filter, flatten, some, repeat, etc.

These functions don’t alter their input.

Release 0.1

Implementation started ...

Release 0.1

First we implemented a key/value store API on top of the HTML5 Local Storage API.It behaves very much like the Google App Engine Datastore. We can store arbitrary maps identified by a :key and a :kind.

Release 0.1

This key/value store on the client side (we call it event-db) automatically syncs with the Google datastore on the server.

Like this:

Release 0.154

Google App Engine

Browser

Auto-Sync

event-db API

XHR/JSON Notification

Datastore

Local Storage API

TheDeadline HTML5 UI (rendering)

NotificationData-Access(Maps/JS-Objects)

Release 0.1

A small step for mankind, but a huge step for TheDeadline:Now we just use the event-db API to access the user data. We don’t need to worry about XHR anymore ...

Release 0.1

Cool thing: Throughout the whole TheDeadline-system we just pass around a very simple datastructure: maps of key/value pairs. No complex DB-mappings needed. NoSQL. Just filter, map, reduce.Easy transformations.

Release 0.1

The Google Datastore is a distributed key/value store.

Clojure works with key/value structures.

JSON is build on key/value structures.

Release 0.1

From this perspective is functional programming the opposite of object-oriented programming:- Simple data structures- Powerful functional operations on these data structures- No state (well, almost)

Release 0.1

We try to write as much functional code as possible. But the UI must maintain some state.

Release 0.1

So sometimes object-orientation can be useful: A Widget Library is a good example. Google Closure comes with such a library, which you can extend by building your own UI components.

Release 0.1

When you have to encapsulate state, you can write a component. For rendering the client-side UI we are using Google Closure Templates.

Like this:

Release 0.162

Release 0.1

Google Closure Templates are used like function calls: You pass the data to be rendered as parameter.

Like this:

Release 0.164

The application of the Soy template:

Release 0.1

Just a reminder: This code runs on all platforms. Computers and smartphones.

Release 0.1

Compare this with the work you have to do when writing proprietary clients for each platform. And I can tell you: This is much more fun!

Release 0.1

Now we have a nice HTML5 client. You can use it on several devices at the same time. You can use it offline and sync later.

Uhh. But what happens when you have concurrent modifications?!

Release 0.1

You need a data model that is able to store versions of your user data!

Release 0.1

Relational databases are not well suited for data versioning. Try to version a large graph of entities distributed over several tables. Good luck!

Release 0.1

Good to have a schema-free, NoSQL, key/value store!

(No matter what Michael Stonebraker says.)

Release 0.1

Our solution is almost a functional data structure: We store every change as an operation on an entity. We can replay all operations. We could easily provide undo-operations. We can restore every state change. You can see it in our todo history:

Release 0.172

The history shows all operations that were applied to a todo entity.

Release 0.1

A schema-free database like Google’s datastore helps a lot to implement such a model.

Release 0.1

And to make and keep the code really elegant, a dynamic, functional programming language like Clojure helps, too!

Release 0.1

What else are we doing on the server side with Clojure?

Release 0.1

We are trying to make modern applications more intelligent.

Release 0.1

TheDeadline contains a rule-based expert system.

Release 0.1

We are using the expert system for decision support: We want to provide the right information to the right person at the right time in real-time in a project management situation.

Release 0.1

When you have lots of rules, you don’t want to make decisions in cascading if-then-else clauses.

Release 0.1

The expert system is a Clojure-based implementation of a classical production system like OPS5 or CLIPS.

Release 0.181

Release 0.1

So, on the server side we are processing large datasets: We generate facts from the user data and run them through our expert-system, which then generates actions for the users based on our rule-set.

Release 0.1

And again: Clojure really helps us to implement stuff like this.

Release 0.1

Conclusion

Release 0.1

So when you want to start a company, it is very likely, that you have to build for mobile platforms from the beginning. Like Eric Schmidt said: Mobile first.

Release 0.1

Instead of building several apps for proprietary platforms use HTML5 as an open standard. (This saves time and money, too).

Release 0.1

If you want to be successful, stay away from relational databases. Use schema-free databases to build versioning data structures. You will need them to resolve concurrent modifications of user data.

Release 0.1

Become a JavaScript master. Use a mature library. Your HTML5 client will become big and complex.

Release 0.1

Start thinking in a functional way. Not everything is an object...

Release 0.1

And: Have fun!Starting a company can be so much fun. And maybe you sign-up for HackFwd?!

I‘d love to see more Clojure Hackers building their dreams!

Thank you.

TheDeadline:http://www.the-deadline.com

Corporate Blog:http://www.hackers-with-attitude.com

Corporate WebSite:http://www.freiheit.com

Self:http://www.smartrevolution.dehttp://www.twitter.com/smartrevolution

Using Clojure, NoSQL Databases and Functional-Style JavaScript to Write Gext-Generation HTML5 Apps

Design