Nico Ludwig (@ersatzteilchen)

(2) GUI – Drawing

2

TOC● (2) GUI – Drawing

– Vector- and Raster-based Graphics

● Idea behind Vector- and Raster-based Graphics

● Crispness

● Overview of Raster-based Drawing APIs

– Platform independent Graphics and GUIs in the Web Browser

● Bare HTML Pages

● Plugins and Problems

● From rich Content to HTML 5

– Drawing with HTML 5 Canvas

● Continuous, Event driven and free Drawing

● Basic Drawing "How does Drawing work with JavaScript?"

● Interaction with Controls

● Sources:– http://channel9.msdn.com/Niners/rbarraza

– web.archive.org, Bare_HTML_Page (www.yahoo.com, 1996)

3

Vector and Raster-based Graphics● We already discussed vector and raster displays, which bring graphics to a screen with different hardware respectively.

● Apart from how graphics are brought to screen by hardware, there are also different ways to render graphics by software.– (The term "rendering" is a more general term for "bringing graphics (incl. video) to the screen or to paper".)

● From a software perspective there exist vector-based and raster-based graphics.

4

Raster-based Graphics● In this lecture we'll only discuss raster-based graphics in depth.

● Graphics that is rendered on a screen, whereby software handles pixels individually, is called raster-based.– (Raster-based graphics are also called pixel-based graphics.)

● For example Java's "Java 2D" software provides the function setRGB() to set one pixel to a certain value/color:

● Often, only raster-based graphics are associated with the word "drawing".

● Raster-based graphics are also used for photographs.

// Java: Set the pixel at (0; 0) to white color:image.setRGB(0, 0, Color.WHITE.getRGB());

5

Basics on Raster-based Drawing● Let's recapitulate, what a picture on a screen is: a big two-dimensional array of pixels, in other words it is a matrix.

– Mind that we already discussed the needed memory depending on spatial dimension and color resolution: a picture is an array!

● To have graphics on the screen we could use functions, which draw pixels of a specific color at specific coordinates, e.g.:

● But there are some problems with this pixel-oriented approach:– It is relatively hard to use these functions to draw more complicated shapes like circles (e.g. with trigonometric functions).

– Drawing individual pixels would be very inefficient and slow, at least not very interactive.

● Therefor graphics libraries provide special functions to draw graphic primitives (rectangle, ellipse, line, bezier and text):

– This approach is very much faster than pixel-oriented drawing! Often the graphics adapter provides hardware acceleration.

● I.e. the calculation of coordinates of individual pixels to draw a circle (with trigonometry) can be done on the graphics adapter without using the CPU!

● The term graphics adapter can be reduced to the term Graphics Processing Unit (GPU).

– A set of functions and objects controlling specific aspects on a computer is called Application Programming Interface (API).

● For operating systems (OS) providing a GUI, it is mandatory to have an API for graphics programming.

// Java: Draw an ellipse of width and height at (0; 0):graphics.setColor(Color.WHITE);graphics.drawOval(0, 0, width, height);

// Java: Set the pixel at (0; 0) to white color:image.setRGB(0, 0, Color.WHITE.getRGB());

6

Crispness● We should have a look at the result of drawing a circle with a raster-based API, we'll continue using Java 2D:

● Welcome the alias-effect: It looks crisp, but...– …when we zoom the drawn image, the pixels become visible and we see how the circle is plotted by pixels mimicking an arc:

● For raster-based graphics there are two ways to get at least a bit of the crispness back:– (1) By the application of anti-aliasing, which smoothes a raster-graphics' edges with the background with colored extra pixels.

– (2) By the application of subpixel rendering (e.g. ClearType on Windows).

Zoom ZoomZoom

Zoom ZoomZoom

Subpixelrendering

// Java: Draw a circle with the diameter 100 at (0; 0):graphics.setColor(Color.WHITE);graphics.drawOval(0, 0, 100, 100);

7

Excursion – Vector-based Graphics – Part I● There are problems with raster-based graphics in general:

– When parts of the graphics are zoomed, the crispness of details gets lost. It results in pixilated graphics, not recognizable as detail.

– When the resolution increases, graphics get smaller. When the resolution decreases, graphics get larger.

● Is that a problem? Well, when we have a premium 4K display do we like smaller graphics or crisper graphics?

● These effects emerge because the rendering of raster-based graphics is dependent on the resolution.– The shown problems are esp. visible on arcs and circles and they are very annoying on text.

increase resolution

8

Excursion – Vector-based Graphics – Part II● To solve these problems, there needs to be a way to make the rendering of graphics independent from the resolution.

– The logical resolution was made independent from the physical resolution.

● E.g. this idea was put into effect with Microsoft's Windows Presentation Foundation (WPF):– WPF introduced Device Independent Units (DIUs). DIUs are independent of a device's pixel raster, a DIU is defined as 1/96 of an inch.

● On a device having 96ppi (pixels per inch), 96 pixels would consume 1 inch; on a device having 120ppi, 96DIUs would be represented by 120 pixels...

● (If logical DIUs do not exactly match to physical pixels WPF will use anti-aliasing.)

● Vector-based graphics are described as formula, which is parameterized by the physical resolution.– It means that vector-based graphics are not described as an array of pixels!

– Raster-based graphics get smaller with higher resolution and zooming leads to pixelation:

– Vector-based graphics get crisper with higher resolution and zooming, em, just zooms! Vector-based graphics are scalable!

raster-basedzoom

raster-basedzoom

raster-basedzoom

vector-basedzoom

vector-basedzoom

vector-basedzoom

9

Excursion – Vector-based Graphics – Part III● The application of vector-based graphics allowed the development some very important applications:

– crisp and scalable texts to allow WYSIWYG editing of documents,

– a new category of software for Computer Aided Design (CAD) to construct structures on a computer instead of pencil and paper,

– a new category of software for Desktop Publishing (DTP) to layout high quality documents for printout.

– => Graphics and print results of excellent quality on any scale.

– (Vector-based graphics are not suited for photographs.)

● Besides these applications, which more or less deal with design, also the desktop was influenced meanwhile:– The scalability and crispness of vector-based graphics was applied to controls, windows and the desktop.

– This was required, because of:

● (1) the availability of higher and higher physical resolutions (high definition, 4K, 8K) and

● (2) the availability of different screen sizes for the same applications (watch, mp3-player, automotive, mobile, tablet, notebook, PC screen, tv, projector).

– Examples:

● Desktop: WPF, (Graphics Device Interface+ (GDI+)), Quartz, PostScript, Cairo

● Browser: Silverlight (WPF/E), Scalable Vector Graphics (SVG)

● Printer: PostScript

10

Overview of raster-based Drawing APIs● In this presentation we'll only discuss raster-based drawing in more depth.

● Up to now we used the Java 2D API for the examples, but there exist more APIs:

● Windows– Graphics Device Interface (GDI)

● Linux– GTK+ drawing (GTK for GIMP Toolkit; GIMP for GNU Image Manipulation Program; GNU for GNU is not Unix)

● OS X– Quartz, Cocoa drawing

11

Native and platform independent Drawing APIs● The drawing APIs we've enumerated are so called native APIs. What does that mean?

– Native APIs are only usable on/compatible with a certain platform. E.g. Windows GDI code won't run on OS X.

– I.e. code written in native APIs is not portable! → Native API vs. platform independent API.

– On the other hand native APIs are fast, because they can exploit the power of the platform, e.g. directly use Windows functions.

● From a 20K miles perspective there are two ways to have platform independent (drawing) APIs:– Create and establish a common platform.

● Establish and use a common graphics API, which is translated into native graphics API calls at run time.

● E.g. Java 2D, QT

– Use a present common platform.

● Find a platform, which is existent on most OS' and implement graphics on that platform.

● E.g. web browsers

Operation System

Graphics Adapter

Native graphics API

Platform independent graphics API

Application

Operation System

Graphics Adapter

Native graphics API

Browser, interprets HTML

Website

12

Platform independent Graphics in the Web Browser● The following discussion deals with platform independent graphics via the web browser (browser).

● Browsers are an interesting platform, because they are available on all important operating systems!

● The first approach to put "graphical content" into effect on the browser was bare HTML with forms and controls.– ~mid 1990s

13

Bare HTML Pages● Bare HTML pages do of course work very good in a browser, but using them didn't feel like using an application.

– HTML pages basically look like a pages of text containing some links, form elements (controls) and images.

– Using HTML pages is more or less navigation in the browser with links and the browser history.

– HTML pages seem to have no togetherness: somehow the contents of the same web address seem to belong together somehow.

● In other words: togetherness is primarily defined by how the session with the HTML "site" is defined.

● Yes, it works, but the result is not very appealing. It is just "navigation in the browser".– When this approach was used, network throughput was low and the performance of clients was not very good.

– On the client the browser just needed to store and render pages. The web server had to handle many requests.

– The approach of bare HTML pages is the basis for the GUI of the internet!

HTMLpage 1

the "application"

HTMLpage 2

HTMLpage 3

HTML page 4

Form

Control 1

Control 2

Image

Link

One page in the browser

browser history

link

link

link

webserverC

lient

Server

delivers HTML pages

14

Rich content with Plugins● The next incarnation of "graphical content" in the browser was the usage of additional browser plugins.

– E.g. Java Applets, Adobe Flash, Apple QuickTime etc.

● The idea is to use a special control, which has very rich interactive features:– The functionality of these controls is provided by plugins, which are installed on the browser's OS.

● Typically the "object"- or "embed"-HTML-element is used.

– With plugins the browser is able to execute mini-applications, often called applets, in instances of plugin controls.

● The browser does indeed start an embedded runtime (e.g. the Java runtime) to execute applets.

– The web server delivers binary components (and HTML pages) that contain the code to be run in the plugin controls.

– (Adobe Flash and Apple QuickTime were the most important plugins in order to play video material in the browser at all!)

HTML page

Plugincomponent

One page in the browser

15

Rich content with Plugins● Browser plugin technologies allowed "web-applications" consisting of only one site without the need for navigation.

– It should be clarified that each type of plugin provides its own event system, apart from the browser's event system.

– Because events are an intrinsic part of genuine GUIs, we can say that a web-application provides a genuine GUI.

– Pages could be designed as a mosaic of plugin controls, such web-applications are called Single Page web-Applications (SPAs).

● A new aspect of SPAs is that no navigation/reloading of sites is needed. Only parts of the sites are reloaded.

● After SPAs with plugins established, another kind of SPAs emerged: those using AJAX.– AJAX (Asynchronous JavaScript And XML) uses JavaScript to load content dynamically without plugin components.

● The important point: AJAX allows to load parts of a currently displayed page w/o reloading the whole page!

– The mix of those technologies is sometimes called "Web 2.0" to have a nice marketing term.

– A web-application using Web 2.0 is sometimes called Rich Internet Application (RIA).

● It can be said that the last incarnation of RIAs were sites that consisted of one huge control making up the whole content.– E.g. JavaFX and Silverlight.

● Yes, this approach also worked and the results were appealing, but there are problems with the application of plugins!

16

Problems with Browser Plugins● The problem lies in the place where plugins are executed: in the browser and on the client side.

– Different browsers on different clients may behave differently. This also influences the effort for quality assurance!

– The required runtimes (and correct versions thereof) need to be installed on the client side.

– Because plugin components are loaded from a foreign source and executed on the client side, there is a certain security problem.

● To shield the client from vicious plugin code, plugin runtimes are executed in a so called sandbox having strong security constraints.

● The problem is that there could be a leak in the sandbox, which still allows vicious plugin code to be executed!

● => As a result the execution of plugins is often blocked, and many users don't trust these technologies (esp. Java applets).

– Running plugin code requires a lot of resources: computing time as well as battery lifetime!

17

The Solution: HTML 5● However, in the end there are still many problems with GUIs in the browser:

– With bare HTML the interaction is not appealing, e.g. navigation among HTML pages.

– Plugins need a lot of resources: computing time and battery lifetime.

– Plugin runtimes are not necessarily installed and activated on the client-side.

– Plugins provide potential security problems.

● It took years until industry addressed these problems. In the end the answer was "HTML 5", providing important updates:– (HTML 5 is not strictly the successor of HTML 4, instead it is a set of new features running in the browser.)

– New clever input controls: date pickers, email and web address controls, sliders and spinboxes etc.

– Support for offline web-applications, HTML 5 storage and support for data mapping and annotation with microdata.

– Support for mobile web-applications: geo location APIs.

– Support for playing video material without third party components like Apple QuickTime or Adobe Flash.

– Support for appealing/more desktop-like usage, e.g. with web workers for doing things in the background.

– But, potentially the most powerful new feature is the "canvas"-HTML-element, which we will discuss for the rest of this lecture.

● "canvas" is basically a way to bring interactive graphics into the browser with JavaScript without any plugins.– Therefor we can now continue our discussion about of how software implements drawing in general...

18

Excursion: Continuous, Event driven and free Drawing – Part I● Basically there exist three models for drawing on the computer screen:

– Continuous drawing.

– Event driven drawing.

– Free drawing.

● Continuous drawing:– The code, which performs the drawing, is executed automatically and continuously at a certain framerate.

● Benefit: the programming of animation and other "vivid" graphics is very simple. It's good for learning graphics basics and simple to use for "sketching".

● Downside: it needs a lot of resources.

– Example: Processing

● Event driven drawing:– The code, which performs the drawing is only executed, when specific events are raised (e.g. update, redraw, refresh, resize).

● Benefit: it doesn't need many resources.

● Downside: W/o interaction there is generally no drawing. (To have continuous drawing with an event driven drawing API, we have to fire draw events at a certain framerate or call the drawing code continuously ourselves.)

– Examples: Cocoa drawing (OS X), GTK, Java 2D, Windows Forms

19

Excursion: Continuous, Event driven and free Drawing – Part II● However, we want to discuss the third approach: free drawing.

● The idea of free drawing is that there is neither a dedicated event that initiates drawing nor any redrawing framerate.

● Free drawing means that any code, which can access the context of the graphics is able to draw!– The code can draw independently on events as well as independently on a continuous framerate.

– It should be said that this is also possible with, e.g., Windows Forms, but the result might be overdrawn by the draw-event!

● E.g. with HTML 5 canvas each function is able to draw, because the graphical context can be accessed from everywhere.– Benefits: it is simple to use from scratch.

– Downsides: it can be tricky to migrate code from other drawing approaches.

● Ok, now its time to see free drawing with HTML 5 canvas!

20

Let's go and draw!

● We'll begin with a general HTML scaffold to program canvas with these important parts:– (1) Import the file containing the needed JavaScript code (Canvas.js).

– (2) Bind the HTML page's load event to the JavaScript function draw() (defined in Canvas.js).

– (3) Mark the canvas element in the page's body with the id "myCanvas".

● In the JavaScript file "Canvas.js" we implement the function draw():– (a) Get the HTML page's canvas element by its id "myCanvas".

– (b) Get the graphical context from the canvas element by calling getContext("2d").

– (c) On the canvas' graphics context draw on the canvas programmatically with special functions (e.g. arc() and fill()).

● As can be seen, the event onload and the canvas object connect the HTML page with the JavaScript code.

<html><head>

<script src="Canvas.js"></script> <!--(1)--> <title>Canvas</title>

</head><body onload="draw();"> <!--(2)-->

<canvas id="myCanvas" width="300" height="300"></canvas><!--(3)--></body>

</html>

function draw() { var canvas = document.getElementById("myCanvas"); // (a) var context = canvas.getContext("2d"); // (b)

// (c): context.arc(canvas.width/2, canvas.height/2, 75, 0, 2 * Math.PI, false); context.fill();}

21

Excursion: Notification or Polling● When the page's load event is fired, the function draw() will be called.

– This event is fired when the page is loaded into the browser. The state of the page then changes from unloaded to loaded.

● There exist two "extreme" ways to deal with state changes.

● Communication via events:– We attach our code to the event, to which we want to react (get notified).

● The code, which is called when the event fired, is called eventhandler or eventlistener.

– => We passively assume that the event will be fired somewhen in future. When the event is fired, we're getting notified.

– (This is what we call the "hollywood principle": Don't call us, we'll call you! It's also called "callback principle".)

● Communication via polling:– We query a system function continuousely, until a certain state has changed.

– => We actively poll for a state to be changed (here with "slow busy waiting").

● JavaScript uses events for communication.– Most GUI frameworks use events as well.

// Attaching an eventlistener:// The eventlistener draw():function draw() { /* pass */ }// Attach the eventlistener draw() to the load event:document.addEventListener("load", draw);

// Polling (this is no legal JavaScript code):// The draw() function:function draw() { /* pass */ }// Wait until the page is loaded:while (!document.isLoaded) {

wait(1000); // "slow busy waiting"}// When the document is loaded call draw():draw();

22

Drawing on a Canvas Element

● Canvas allows to draw two kinds of primitives: solid primitives and paths (no official JavaScript API terms).

● Solid primitives are filled with a color, gradient or pattern.– Relevant functions and properties: fillRect(), fillStyle, fill() (to fill a drafted path (see below))

● Paths are drafted with functions, which define the skeleton of a figure. Later on, these paths are stroked to be visible.– Drafting paths is like "preliminary" drawing a figure with a very fine and hard pencil.

– Stroking is like drawing the real figure with ink.

– Relevant functions and properties: beginPath(), closePath(), moveTo(), lineTo(), rect(), arc(), strokeStyle, stroke() or fill()

– Sometimes

● (1) a path needs to be started by calling beginPath() and sometimes

● (2) after the drafting is done a path needs to be closed (connecting the first with the last "pencil stop" with a line) by calling closePath().

function drawPathAndStrokeCircle() { var canvas = document.getElementById("myCanvas"); var context = canvas.getContext("2d"); context.arc(/* pass */); context.stroke();}

function drawSolidCircle() { var canvas = document.getElementById("myCanvas"); var context = canvas.getContext("2d"); context.arc(/* pass */); context.fill();}

23

Controls

● So far we have a rough understanding of how we can draw visible contents on an HTML page.– But it would be very time consuming to draw every piece of interaction in such a way!

– Therefor we have predefined pieces of interaction: controls.

● Controls are pieces on a GUI, arranged on a window, with which the user can interact.– Users use mouse, keyboard and touch gestures to interact with the controls on the GUI.

24

Combine/implement Interaction with Controls and Canvas

● In this example we bind another eventlistener to the onclick event of a button.– The onclick event is fired, when the button is clicked.

– The eventlistener drawInnerCircle() draws a smaller inner circle into the initial circle.

– The eventlistener drawInnerCircle() sets the text of the textbox to "Button clicked!".

● Via events and JavaScript code we can have real interactivity.– Events are typically a result of user interaction, e.g. clicking a button.

● Events are fired, when the state of HTML controls changes (page loaded, button clicked etc.).

– JavaScript code is provided by developers.

● JavaScript code is able to access (getElementById()) and change the state of controls (e.g. setting a value).

function drawInnerCircle() { var canvas = document.getElementById("myCanvas"); var context = canvas.getContext("2d"); var textBox; context.beginPath(); // Initiate a new path to decouple from 1st arc! context.arc(canvas.width/2, canvas.height/2, 20, 0, 2 * Math.PI, false); context.fillStyle = "white"; context.fill(); textBox = document.getElementById("myText"); textBox.value = "Button clicked!";}

<html><head>

<script src="Canvas.js"></script> <title>Canvas</title>

</head><body onload="draw();">

<canvas id="myCanvas" width="300" height="300"></canvas> <button onclick="drawInnerCircle();">Click me!</button> <input id="myText" type="text"></input>

</body></html>

25

Events with Parameters

● We can also attach events to a canvas element, e.g. to canvas' onmousedown event:– The onmousedown event is fired, when the canvas is clicked.

– As can be seen, the onmousedown event is somewhat special: it accepts an argument!

● Here we pass the global event object, just called "event".

– The event's argument is available via the parameter e in the eventlistener canvasClicked().

– The parameter allows to read the coordinates (i.e. properties x and y) of the mouse position, on which the mouse was clicked.

● A common strategy is to pass the global event object as an argument to the eventlistener.– The passed object contains extra information about the state of the page, controls, keyboard and mouse.

– Then the eventlistener's parameter provides a lot of properties and functions, which allow almost full control of the GUI.

function canvasClicked(e) {var textBox = document.getElementById("myText");

textBox.value = "x = "+e.x+"; y = "+e.y;}

<html><head>

<script src="Canvas.js"></script> <title>Canvas</title>

</head><body onload="draw();">

<canvas id="myCanvas" onmousedown="canvasClicked(event);" width="300" height="300"></canvas>

<input id="myText" type="text"></input></body>

</html>

26

Web Apps with HTML and JavaScript in Reality● There are some important downsides of this approach:

– A browser needs to be able to render "HTML 5" in order to fully unleash the power of, e.g., canvas.

– The execution of JavaScript needs to be enabled in the browser!

– The HTML- as well as the JavaScript-code are loaded to the client side and can be read.

● I.e. the code could be copied/reused without permission. Also leaks could be easily detected and exploited.

● (However there is a way to reduce the readability of code: with "code minimizers" (i.e. the opposite of code beautifiers).)

● General acceptance of web-applications:– Most of them do not work offline.

– The acceptance of users is not always given, esp. the look and feel (l&f) is not like that of a desktop application.

– Web-applications can be a cheap but effective solution for business applications.

27

Thank you!