1

Nico Ludwig (@ersatzteilchen)

(1) GUI – History of Interactivity

2

TOC● (1) GUI – History of Interactivity

– Wires and Bulbs

– Batch Processing

– Terminal and Mainframe

– From the Command-Line to Killer Applications

– Vector Displays and Raster Displays

– Color Displays

– The Mouse and the Takeoff of Interactivity

– The Desktop Metaphor

● Sources:

– Wikipedia:

● Bubba73, ASR 33; Shieldforyoureyes/Dave Fisher, BM-3279; Oliver.obj, Panel 370-145; Boffy b, IBM PC 5150; Rama & Musée Bolo, Apple II-IMG 4217; SRI International, SRI Computer Mouse

– Video series "Triumph of the Nerds"

– Matthew MacDonald, Pro .NET 2.0 Windows Forms and Custom Controls in C#, Appendix A

– Charles Petzold, Code – The Hidden Language of Computer Hardware and Software, Chapter 25

– other sources are public domain

3

The Beginning – Data Processing

● Basically we'll discuss the user interface (ui), which is the place where computer and human meet.

● In the early days of computing, computing was a job for white collar workers.

– In the 1940ies computers were used by switching switches and plugging cables.

– The state of the switches and the wiring represented data and program "code".

– The results were displayed with an array of lightbulbs.

– Using and programming was usually the same thing.

● In the 1950ies, 1960ies and early 1970ies batch processing was applied.

– Program code and data were punched into cards or tapes("punched cards/tapes"), which were interpreted by the computer.

– The results have been printed on paper.

– The whole input was processed as one batch of punched cards.

Input

Processing

Output

Process input in batch

Batch of punched cards (data and program code)

Print the output to paper

Dire

ctio

n o

f the

pro

cess

ing

● The graphic shows an ENIAC (Electronic Numerical Integrator And Computer) (1947 – 1955).

● Punched cards and tapes have also been used for organs and looms.

4

Interactivity raises – Having a Session with a Computer

● Somewhen in the 60ies computers learned to interact with humans. (interact, from lat.: inter = "between", agere = "to act")

– Er, or, … somewhen in the 60ies humans learned to interact with computers.

● Interaction means that human input and computer response happen multiply within the same session.

– The computer and the means of input/output were separated devices connected by a network.

– The user inputs data via a teletypewriter (tty) and the computer responded via the tty as well. The tty was the so called "terminal".

– So user input can directly follow the last computer output, but not in parallel. This makes the interaction.

● The terminal prompts for input, the user sends his input, after it is processed by the computer it is resend to the terminal and another prompt appears.

– The communication is based on transmission of streams of serial data following the American Standard Code for Information Interchange (ASCII) protocol.

– The whole session goes only into one direction, it can be traced along the tty's paper roll.

Input

Processing

Output

Input

Processing

Output

Input

...

computersession

Dire

ctio

n o

f the se

ssio

n

Input & output/tty

computer

terminal network computer

● I.e. ttys were used as peripheral device and terminal.

● Input and output of ttys was just text, so it is also called "text terminal" (German: "Schreibmaschinenterminal").

● Because a paper roll was used to trace the procedure, it was also called "hard-copy terminal".● A tty is a printer and a keyboard.

● Prompt from lat. promptare: "to publish something".● There survived some terms from this era of

computing:● ASCII, control characters: LF, CR● The "TTY", as oldest serial interface, is still

available as terminal interface (shell) and to program industry plants.

5

Interactivity raises – Keyboards and Screens

● In the 70ies, separated peripherals for input (the keyboard) and output (cathode ray tube (CRT) screens) were introduced.

– Keyboard and CRT acted as terminal together, other terms: "glass ttys", Video Display Units (VDU), Video Display Terminals (VDT).

● In opposite to the term "hard-copy terminal" for the tty, which basically was a printer more than an input device.

● At this time in history two aspects are relevant to understand:

– Still the "tty-idea" is used here! VDUs just represented rolls of paper, it was just a "line by line" communication.

– VDUs were connected to a remote computer via a network. As being just terminals, VDUs had no intelligence at all!

– => This arrangement consists of a "dumb terminal" (has little logic) and an "intelligent computer", the mainframe (has a CPU).

Output/CRT

Input/keyboard computer

dump terminal networkremote, intelligent

computer (mainframe)

● "Line-by-line" communication means that we have no cursor control, but following control codes are mandatory: CR, LF and Bell.● OS' like MS DOS and C/PM uses/used monitors only that ways.● By a matter of fact today we still use this metaphor in diverse

command line interfaces/shells.● Monitors have the potential to interact with the user by applying the

whole two dimensional space of the screen. ASCII was already equipped to be extended with control codes handling two dimensional character devices (clean the whole screen or move the cursor to the left upper corner).

● Before ASCII there was the Extended Binary Coded Decimals Interchange Code (EBCDIC). EBCDIC was used by IBM's mainframes in the 1950ies and early 1960ies.● EBCDIC's character codes were not arranged sequentially in

alphabetic order (i.e. there were gaps), but optimized for punched card processing. For modern applications ASCII was much more useful, because of its alphabetic structure.

● Mainframe, German: Großrechner● The shown terminal is an IBM-3279 (~1972) a green screen terminal● The shown mainframe is an IBM System/370-145 (~1970) (the

system console).● We don't talk about applications running on the mainframe, which

were basically databases and financial processing software. The large units to store all the data are not shown (the compatible storage unit is the IBM 3850 with magnetic tape cartridges).

● The terms "dumb terminal" or "thin client" have other meanings today.

6

Terminals and Mainframes? – What the heck...?

● One moment! Why was this kind of arrangement with ttys/VDUs and mainframes and a network in between necessary?

– Because of the problems, which were solved with computers that time! Read on:

● In the beginning, computers were basically used for financial and military purposes.

– Starting in the 1960ies, people understood how computers are able to process masses of data, esp. in business.

– This problem domain required holding and managing huge amounts of data! The problem domain drove two developments:

● (1) Mass storage hardware, which holds data.

● (2) Database software, which manages data.

● In the end there are three (four) individual devices, connected by network/wire:

– ((0) User.) (1) Terminal. (2) Mainframe. (3) Mass storage device.

– It was required, because of technical possibilities that time and because folks just clung to the idea of a "big computer room".

Output/CRT

Input/keyboard computer

terminal network mainframe

database'sdata

mass storage device

● (To be frank in the late 60ies also more intelligent terminals with some microprocessor power and some memory to handle editing operations (insert a character), without sending/re-sending the whole screen to the computer were available)

7

The Evolution of the Command-Line

● With terminals and mainframes the command-line was the major means of interactivity. There were some historical steps:

● The bare command-line age:

– Full control, the user could freely use all tools in any order.

– No guidance, no structure, no metaphor, commands needed to be remembered by the user.

● The command-line prompt age:

– No control, the user needs to follow an application's usage paradigms. A program could only do a single task.

– Full guidance, each step is prompted, we had an interview-metaphor (question/answer).

● The menu age: (Let's call a program "application".)

– Compromise of control, the user was able to do a set of tasks. An application was able to do multiple tasks.

– Ergonomic guidance, menus are a compromise of prompt and free selection of the next action. We have a document-metaphor.

8

Two dimensional Interactivity – Killer Applications

● Its time to give up working with line-by-line communication.

– For terminals cursor control was not mandatory, only these control codes were required: CR, LF and Bell

– But: screens have the potential to break free line-by-line and use their full two dimensional space.

● ASCII was already ready to be extended with control codes for two dimensional character devices (clean screen, move cursor to left upper corner, etc.).

● In 1979 VisiCorp presented an application that used the screen in a two dimensional manner: VisiCalc.

– VisiCalc was a spreadsheet application using the screen as a table for numerical calculation. It brought following new aspects:

● The cursor could now be moved among columns and rows!

● The screen followed a metaphor. The "spreadsheet"-metaphor mimics a chalkboard or pencil/paper on a computer screen.

● The application had menus to guide the user and applied another metaphor: the "document"-metaphor.

● VisiCalc was the first killer application (app) for home computers!

– A killer app is an app that significantly changes the attitude of users onusing a technology. A killer app is a "game changer".

– This killer app proved that one can do useful things with a computer(here: "number crunching").

● VisiCalc was the breakthrough for the Apple II.

– It showed that the Apple II really had a business use!

● The Apple II, the first PC, needed an important application (a "killer app") to be a success: VisiCalc (designed by Dan Bricklin at the "Harvard Business School" in 1979).● Another way to characterize a killer app: vendors

and developers use it as reference. "Let's do it like it is done in VisiCalc."

● VisiCalc was also the killer app on Apple II's competitor "the IBM PC" (IBM model 5150 in 1981).

● With VisiCalc the transition of the computer from a hobby object to a business object started.● The spreadsheet concept was, however, not

patented by VisiCorp and it was copied by Microsoft (Multiplan, Excel), Lotus Development Corporation ((belongs to IBM today)Lotus 1-2-3)).

● Also text processors which were then partially replaced by word processors featuring WYSIWYG.

9

Two dimensional Interactivity – Hey Terminal, good Riddance!

● Apple II? Where did the mainframe go?

● There is a very important point: an app like VisiCalc can't be brought to the dumb terminal/mainframe arrangement:

– Fast screen updates as reaction on user input are required, and no extra communication via a slow network.

– Memory went short, because usually terminals (the better ones) could only store a few lines of text up to page.

● From terminals to Personal Computers (PC). PCs are standalone computers.

– Basically, the intelligence of computer and terminal were integrated into one unit. This unit was called Personal Computer (PC).

– Apple (!) was the first manufacturer of PCs for consumers: the Apple II reached the market in 1977 (at 1298$).

● Apple made computers white goods (like vacuum cleaners or microwaves are).

● Speed is important to make interactivity work acceptably!

– (1) PCs directly accessed the video memory without the OS or the network.

● Dumb terminals had only means for character generation in the hardware.In opposite to some PCs even having a graphics adapter.

– (2) PCs had their own memory (4KB and more).

● In parallel to the development of two dimensional interactivity the development of video display evolved...

● Virtually IBM "slept" until the early 1980ies, then they understand the meaning of PCs. They released the IBM-PC (IBM 5150) in 1981, four years after Apple's Apple II.

● PC today: IBM-compatible (IBM compatible – read: esp. the interface to video memory is standardized) This "openness" was the key to make software independent from hardware. Apple II and IBM PC were expandable "in the box" by attaching new adapters (also from 3rd party vendors) to the so called mainboard.

● Homecomputers could be connected to the TV, but early IBM-PCs could also be connected to the TV. A better way to categorize homecomputers is that they are closed in opposite to the openness of IBM compatible computers. - In the world of a homecomputer everything is proprietary (connectors and internal adapters, one could buy extensions or replacement parts only from the manufacturer of the homecomputer).

10

Video Display – Vector Displays

● To understand the next steps on the way to interactivity, the further development of video displays has to be understood.

– Up to now we only discussed systems using the terminal screen, often only with a two-color palette (e.g. a "green screen").

– The most important step was from character displays to graphical displays.

● The first graphical displays for computing were so called vector displays.

– Vector displays are CRT displays, in which an electron ray (emitted by an electron gun) directlydraws objects composed of primitives on the screen.

● This is similar to a human drawing on a canvas. Also characters are composed from primitives.

● Examples: radar screens, oszilloscopes and the first primitive computer games.

● Vector displays brought an interesting idea of interaction: the light pen.

– Light pens also brought a new killer app: "Sketchpad" (Ph.D. thesis Ivan Sutherland, 1962/1963).

– Sketchpad was the first interactive graphical app, it introducedComputer Aided Design (CAD).

● But these vector displays needed to update the drawing in short intervals.

– The time needed to redraw the display limited the amount of content (and complexity) that could be displayed.

canvas

electron gun

vector display screen CRT

electron beam

● The Apple II had up to 15 colors.● Lightpen:

● A lightpen detects the beam emitted by the electron gun. Lightpens are also usable for CRTs.

● Afterwards, lightpens were used to fill out forms on the screens (e.g. ticking check boxes or for simple multiple choice tests). Today we have the stylus for cellular phones.

11

Video Display – Raster Displays

● The next step in the evolution of CRT-based displays were the so called raster displays.

– A raster display composes the content of the display by sending a beam of electrons to the canvas in a row wise manner.

– Each row consists of column elements, the so called picture elements, or pixels. Raster displays are known from TV screens.

● A raster, or an array of pixels makes up the displayed image.

● And in software a rastered image corresponded to a block of memory in the computer (as an array).

● Technically, each pixel on the canvas is a spot of phosphor, being lighted by the electron beam.

● The benefit of raster displays is that they can also display complex graphics (in opposite to vector displays).

– But "feeding" raster displays with data needs much more memory than feeding vector displays with data.

electron gunraster with lightedand dark pixels

raster display screen CRT

canvas

electron beam

12

Monochrome Video Display

● From the perspective of raster displays a screen has a resolution, e.g. a resolution of 640 x 480 pixels.

– (The resolution is sometimes called spatial dimension.)

– This makes a total of 307200 pixels.

– If we want to represent each pixel by one bit, we need 307200b or 38400B to store the content of the screen.

– With this representation we can only have pixels in either the state "on" (white/green) or "off" (black), like the states of one bit.

– The representation of one pixel with one bit makes a raster display a monochrome display.

● In order to be able to represent more color states per pixel, we need more bits to store the state of a pixel.

– We could use 1B to store a pixel. This would of course increase the required memory to store an image.

● For a resolution of 640 x 480 at 1B 307200B of memory are required.

– If we use 1B per pixel, we can have one pixel presenting 28 (256) colors.

● E.g. we could have 256 different nuances of grey between the pixel values 0 (black) and 256 (FF16

) (white).

● Technically, the CRT of a monochrome display sends its electron beam in different intensities to have different nuances of grey.

– We could have 256 different nuances of grey between a pixel being dark (value: 0, color: black) and lighted (value: 256, color: white).

13

Video Display in Colors

● The step from a monochrome display to a colorful display is simple:

– Use three electron guns/beams for each primary color.

– Each beam can then have a different energetic intensity.

● The three primary colors of the additive color mix are red, green and blue (RGB).

– Additive color mix works with spots of color emitting light.

– The additive color mix allows the representation of any color by having combinations of different intensities of each primary color.

– Technically, each pixel on the canvas is a combination of three spots of phosphor, colorized in red, green and blue, the subpixels.

– Each subpixel can be lighted individually by the electron beam responsible for the color of that subpixel.

additive color mix

RGB electron gunsthree electron beams

pixel of acolor raster display

green subpixel (lighted)

raster with lightedand dark pixels/subpixels

● Subtractive color mix works with spots of color reflecting light, i.e. a part of the color information is absorbed. The subtractive color mix its relevant for printers.

14

Video Display in Colors – Color Resolution

● To drive this point home we have to represent a pixel in software with a combination of the three primary colors:

– A color pixel is represented by a combination of different intensities of each primary color.

– The count of intensities of each primary color defines how many different colors a pixel can represent. This is called color resolution or color depth.

● Examples of color resolution:

– Use 2B/16b per pixel.

● We have 5b per color (1b left over or 6b for green).

● Then each color can have one of 32 intensities.

● Then a pixel can represent one of 32,768 colors.

● The 2B/16b color resolution is often called high color.

– Use 3B/24b per pixel.

● We have 8b per color.

● Then each color can have one of 256 intensities.

● Then a pixel can represent one of 16,777,216 colors.

● The 3B/24b color resolution is often called full color.

● It should be clear that the required memory increases with the color depth.

– With a resolution of 640 x 480 we have a total of 307200 pixels.

– With the color depth of 3B/24b almost 1MB (307200 x 3B) is required to store an image.

General formular: number of representable colors = 2b per pixel

Lucy in high color Lucy in full color

● The color resolution of the shown picture was manipulated (in this case downgraded of course) with the program GraphicsGale.

15

The Takeoff of Interactivity – The Mouse

● In the late 1960ies Doug Engelbart showed the fruits of his research concerningcomputer input devices.

– He presented the concept of the computer mouse in the first real demo in computer industry.

● His demo, held on 1968-12-09, is sometimes called "The mother of all demos" (Steven Levy).

– => Doug Engelbart invented the mouse in 1968, but it was forgotten afterwards.

● The usage concept of the computer mouse is "point and click".

– The user controls a virtual input pointer on the screen by rolling the real mouse over the writing desk.

● In the beginning of the 1970ies, some folks of Engelbart's team left and joined the Xerox corporation.

– More exactly, they joined the Xerox PARC (Palo Alto Research Center, sometimes just called "PARC").

– Xerox PARC drove Xerox' initiative to enter computer business.

● Originally Xerox produced devices for office and document management (e.g. photocopiers and printers).

– Many more early interested engineers joined PARC that time.

● At PARC the problem of the computer-human interface was recognized soon!

– It led the engineers at PARC to revolutionary ideas...

● The Xerox corporation (founded as"Haloid Company") is the inventor of the photocopier (1949). Nowadays the term "Xerox Machine" is a synonym for "photocopier".

16

The Takeoff of Interactivity – Graphical User Interfaces

● Affordable raster CRTs became available. The result was the Xerox Alto (1972/73), a computer with revolutionary features.

● Xerox' revolutionary ideas in user interfaces – Part I:

– A high resolution CRT 606 x 808 in portrait orientation (i.e. shaped like a sheet of paper).

– The content of the main screen was laid out like a real writing desk. Hence, the main screen wasjust called "desktop".

– Applications on the desktop ran in so called windows and could be controlled viagraphical elements.

● These graphical elements were called controls: textfields, buttons, checkboxes, icons and menus.

● The user interface goes beyond bare text!

– I.e. multiple applications could run simultaneously on the same screen! This was a significantimprovement!

– The user used a mouse to interact with the controls. Xerox remembered, reused and improvedEngelbart's ideas here.

– Such a rich interface was then called Graphical User Interface (GUI) in opposite to Console User Interface (CUI).

– => The Xerox Alto was the first computer with a GUI.

● See "Xerox" video "Triumph of the Nerds" (3/3, beginning at 5:50).

● The word "control" is a general term, in Java those are called "components", in many unioxid environments those are called "widgets".

17

The Takeoff of Interactivity – Metaphors and Object-Orientation

● Xerox' revolutionary ideas in user interfaces – Part II:

– GUIs enforced the definition and application of interface metaphors (i.e. simulations of the reality):

– Desktop metaphor: a region, on which items with specific functions can be placed.

● While the mouse was rolling across the writing desk, the virtual mouse pointer moved across the virtual desktop.

– Window metaphor: sheets of papers, on which different things are handled, windows can be stacked like real sheets.

– Form metaphor: arrangement of controls on a window, the controls could be filled with data (textfields, checkboxes).

– What You See Is What You Get (WSIWYG): design documents on the screen exactly as they will be reproduced on paper.

● WYSIWG text editors: "Bravo" (1974, Butler Wright Lampson and Charles Simonyi) and its successor "Gypsy" (1975, Larry Tesler and Timothy Mott)

– Metaphors allow to use a computer based on guessing how things work, not knowledge.

● This is a key to being user friendly, sometimes it is what we call an ergonomic user interface.

● Xerox' revolutionary ideas in programming:

– The more complex GUI required a mighty programming paradigm to program interface metaphors: object oriented (oo) programming.

● The controls of Alto's GUI were represented as genuine programming objects.

● Control objects could be combined on window objects while programming at design time and were represented as graphical elements at run time.

● (Introduction and application of the Model-View-Controller (MVC) software architecture.)

– For the Alto, Alan Kay invented of the first oo programming language "Smalltalk", incl. an IDE, in the 1970ies.

– Smalltalk basically influenced all oo languages we have today!

● Ergonomics means that things can be used intuitively. I.e. users can even guess, where needed functions do reside, because the usage pattern is always the same and follows metaphors. After this definition of ergonomics, ergonomics was somewhat broken by Windows 8's so called "Modern UI" (codename "Metro").

18

The Takeoff of Interactivity – The Desktop Metaphor

● Think back: esp. on the command line interface it is difficult to remember all the commands! With the desktop metaphor we have icons with symbols, which are self explanatory.

● On touch devices we have no desktop but a home screen (called springboard on iOS).

19

The Takeoff of Interactivity – Well, the Xerox Alto...

● Isn't that great? Wait! Have you ever heard about the Xerox Alto? Hm... no!

● Xerox didn't revolutionize with its ideas, because:

– The management of Xerox did not recognize the potential of the things their engineers created.

– The Alto was too expensive: 30,000$ in 1973. (Later on the real commercial product derived from the Alto was the Xerox Star.)

– In other words: The Alto was ahead its time, or it wasn't yet the time for this revolution.

– The Alto was not yet a PC, esp. because it was not affordable!

● In 1979, Steve Jobs and some fellows from Apple visited PARC and attended a demo of the Alto.

– Jobs and his team were impressed and instantly understood the outstanding meaning of having a GUI.

– In took until 1983, but Apple made the computer a consumer product: the Apple Lisa.

– But the Apple Lisa was too expensive (10,000$). (Em, Steve's fault!)

● Only about 1000 pieces of Xerox Alto were sold.● The Xerox Star was sold at ~16500$● Only about 11,000 – 50,000 pieces of the Lisa

and about 50,000 pieces of the Lisa 2 were sold.

20

The Takeoff of Interactivity – Breakthrough

● The real breakthrough of interactivity on consumer computers was the Apple Macintosh (Mac) in 1984.

– A revolutionary TV commercial during a break of the telecast of the Super Bowl (1984-01-22) started to attract potential users.

– Affordable at ~2,500$.

– All in one computer (Motorola 68000 CPU), keyboard, mouse and screen (512×342 monochrome).

– The revolutionary system software Mac OS (the ancestor of OS X), which provided the user interactivity.

● The Mac introduced a new class of applications based on the desktop metaphor (applications beyond "number crunching").

– Desktop Publishing (DTP)

– What You See Is What You Get, WYSIWYG ("wizzy wig")

– (Before that, CUI applications were basically applications using the file system.)

● After the desktop metaphor was established it was mostly only copied by other companies for more than four decades.

– VisiCorp VisiOn (1983)

– Digital Research GEM (Graphical Environment Manager) (1984)

– Microsoft Windows 1.0 (1985); the real takeoff was in 1990 with Windows 3.0

– X Window System (1984)

● The keyboard of the Macintosh in 1984 didn't even have arrow keys! - Obviously Apple's engineers thought that all users will prefer the mouse. And it took until the 1990ies, when Apple applied the "inverted-T" layout of the arrow keys.

● Windows 2.0 was introduced 1987 and it allowed overlapping windows.

21

Not discussed Topics

● X Window System and multi user support

● Voice recognition

● Touch

22

History – Part I

● Users were white collar workers. Input: plugging cables; output: lightbulbs.

– Users were programmers!

● Users applied batch processing.

– Users needn't to be programmers any longer!

– Input: a batch of punched cards/tapes (code and data); output on a printer (continuous printing).

● Users sat in front of a terminal. A terminal was one device for input/output (tty), later separate devices (glass ttys).

– Users had a session with mainframe, which was controlled from the terminal via a network!

– One dimensional interactivity: line-by-line communication, one program for one task.

– Bare command-line: full control, less comfort. Command-line prompt: less control, some comfort.

● Two dimensional applications allowed moving the cursor in two dimensions and using menus.

– Users worked with a standalone PC.

– Metaphors (simulations of the reality) emerged in applications: document and spreadsheet. Multipurpose applications.

– Compromise of control and comfort.

● It should be said that there is a tendency to reduce the freedom of using multipurpose applications on the desktop somewhat. The idea is to guide users through using certain applications at a certain time for a certain task, along a and as part of a so called business process. ● This tendency is still ongoing and evolving,

because the technical infrastructure grew in the meantime: cheaper network throughput, clouds, applications able to access cloud services etc.

● This tendency merges up with web-applications, esp. with three-tiered applications, on which much processing power and handling multiple user sessions reside on a server. - In a sense it means a step "back" to the terminal/mainframe idea!

23

History – Part II

● The desktop metaphor. Input: keyboard, mouse; output: screen, printer/WYSIWYG.

– Users controlled applications with the mouse via point and click. Ultimate two dimensional usage with windows and controls/forms.

– Control and comfort were mixed up completely, because applications could be used by "guessing", i.e. ergonomics.

● Emerging UI aspects (read: not only graphical aspects):

– Touch-gestures (killer app: Apple iOS)

– Haptic feedback (Apple's Force Touch Trackpad)

– Physical-gestures (killer app: Nintendo EAD Wii Sports)

– Voice recognition (killer app: Apple's Siri)

– Motion detection

– Ambient detection

● Nintendo EAD, EAD for Entertainment Analysis & Development.

● Meanwhile we are confronted with further usage paradigms and emerging metaphors because of new applications: mobile devices, smart devices (e.g. TVs) support of disabled persons and the internet of things.

24

Thank you!