Learning Layout Design: Challengesand Opportunities

Janin KochAalto UniversityHelsinki, Finlandjanin.koch@aalto.fi

Daryl WeirAalto UniversityHelsinki, Finlanddaryl.weir@aalto.fi

Antti OulasvirtaAalto UniversityHelsinki, Finlandantti.oulasvirta@aalto.fi

Paste the appropriate copyright statement here. ACM now supports three differentcopyright statements:

• ACM copyright: ACM holds the copyright on the work. This is the historicalapproach.

• License: The author(s) retain copyright, but ACM receives an exclusivepublication license.

• Open Access: The author(s) wish to pay for the work to be open access. Theadditional fee must be paid to ACM.

This text field is large enough to hold the appropriate release statement assuming it issingle spaced in a sans-serif 7 point font.Every submission will be assigned their own unique DOI string to be included here.

AbstractThis position paper discusses the use of machine learningmethods in layout design. Interactive layouts are pervasiveand a central part of e.g. GUIs, Web interfaces, menus andforms. They have been hard to design algorithmically,because search spaces are large and multiple factorscontributing to design choices. We argue that in order totouch base with real design practices, machine learningapproaches should take into account the requirementsposed by user-centered design. We have identified fourtouch points to user-centered design. For each touch pointwe discuss both opportunities and challenges and showresults from our on-going work.

Author KeywordsLayout design; Machine learning

ACM Classification KeywordsH.5.m [Information interfaces and presentation (e.g., HCI)]:Miscellaneous

IntroductionInteraction design is a prevalent yet challenging activity inmodern software and product development. Despite itsprevalence, many designers still work with paper prototypesand other analogue tools when creating and evaluatingdesigns. Digital tools, such as Adobe Illustrator, are in use,

but generally function as sketching tools or automate verysimple tasks. We argue that there is substantial untappedpotential in using machine learning techniques to activelysupport the designer’s creative work. In particular, we pointout opportunities inspired by the user-centered designprocess: 1) automate requirement gathering, 2) offerinspirations for design, 3) automate the evaluation process,and 4) even automatically create whole layouts.

The time is ripe to discuss the potential of data-drivenapproaches in layout design. With the vast array of visualexamples available online, and the ease of collecting largedatasets on the preferences of users and their habits wheninteracting with software, this is an application domain ripefor the application of machine learning. In this paper weanalyse a commonly used process model of user-centereddesign, Interaction Design Lifecycle Model [5], to identifyopportunities to apply machine learning in the designprocess, and formulate challenges that these opportunitiesimply.

There is an emerging trend in this direction, both inresearch and industry. Recent work has studied automaticprediction of the aesthetic appeal of an interface [3]. TheWebzeitgeist project [2] aims to ‘design-mine’ the web tosupport data-driven web design. DesignScape [4] looks atautomatic generation of static graphical layouts such asfliers. However, a similar tool for interactive layouts does notyet exist. In industry, a startup named The Grid1 currentlybuild a automated web design tool using machine learningwhich caught a lot of public attention. Perfectbanner2

automatically redesign web ads to increase traffic, with amachine learning algorithm that learns successful adstrategies from click data.

1http://thegrid.io2http://perfectbanner.com

Figure 1: Interaction Design Lifecycle Model

However, there is still substantial scope for improvement inthis research area. We need better understanding of howdesign is created, how humans perceive layouts andinteractions, and what makes a layout good. Machinelearning offers tools to obtain this understanding.

The design process and opportunitiesTaking the interaction design lifecycle model (see Figure 1),we will introduce the steps of the model, the opportunitieswe have identified for machine learning, and the associatedchallenges. Our focus is on the design of interactive layouts,whether in software or for the web.

Establishing RequirementsThe initial step in the design process is to gather informationregarding the client, the target users, and the experienceand functionalities the design should support. Theserequirements build the fundament of the design project andall created designs have to satisfy these requirements. Inthe context of layout design, this involve the analysis ofclient side, competing products, the user group’s needs, andthe functionalities the layout should provide.

Gathering information about the client and target users iscurrently a manual process for the designer. We seeopportunities to support this process using machine

learning, especially in automatically identifying relevantonline information and mentions of the client. Furthermachine learning could help to identify and analyse the webpresence of competitors, e.g information presented, brandcolours or layout structure.

Enabling such support requires text mining of the client’scurrent web presence (e.g. using topic modelling) andautomated search to identify the sites of competingbusinesses. Tools to automatically classify and aggregaterelevant documents about the client or their business area(such as news articles, social media, and blog posts) couldalso be constructed based on the results of the text mining.

The challenges here lie in automating the search processand accurately collating the relevant information. Thisrequires the combination of techniques from informationretrieval and machine learning. The models that extract therelevant information must be accurate, and robust to thediverse methods of presenting content on the web.

Designing alternativesDesigning alternatives is the core activity of the interactiondesign process. This step can be divided into twosub-activities, which are conceptual design and physicaldesign.

The conceptual design of a layout, often referred to aswireframing, is the basic structure, which defines interactiveelements, the interaction concept involved and the generalstructure of all elements present in the layout. The physicaldesign refers to the more detailed specification of thedesign such as colour, images, menu design, and so on.

While systems which support this step exist (e.g. Adobeillustrator or Sketch), they are limited in that the designer isthe only actor. Using machine learning could enable us to

go beyond that by creating a second source of input —effectively allowing the system to participate in the design.Two examples would be generating ’inspirations’ to improvean existing design or creating new layouts based on definedrequirements.

This poses a number of machine learning challenges. Toprovide inspirations, we need to be able to classify the typeof layout and the purpose of interface elements. Then thishas to be matched against a database of mined designs inorder to find alternative ways of presenting the informationor common elements that may be missing in the currentdesign. To generate entirely new designs, we need to mineexisting layouts to create probabilities for position, size andsemantic grouping of interface elements. These must becoupled with a set of design requirements to producecandidate layouts. This is an extremely complex densityestimation task.

PrototypingIn order to test the created alternatives, the mostly staticlayouts have to be converted to interactive prototypes, whichimitates the interaction concepts used. These prototypes donot necessarily have to be a working software.

Current prototyping tools for layout design such as InVisionand Sketch have already started to support this step of theprocess. They offer possibilities to connect multiple layoutsusing simple interactions like clicks or gestures, but morecomplex interactions are not currently automaticallyintegrated. This means that prototypes have to be manuallyprogrammed by the designer, e.g. with Processing, oralternative low-fidelity prototyping methods have to beconsidered as soon as the interaction become morecomplex.

Machine learning can be used to analyse flows ofinteraction in existing applications and automaticallygenerate interactive prototypes based on the designalternatives produced in the previous phase of design.

Alternatively, patterns of user behaviour could be learnedfrom software logs and used to generate probabilisticagents which can simulate interaction with the new design.This could enable early problems with a design to beidentified without the significant resource cost of multipleuser tests, and allow more designs to be effectivelyprototyped.

Creating such agents is a difficult challenge. We need arobust model of user behaviour that can classify theelements of a new interface, and generate clicks, gestures,or other inputs in a realistic fashion based ondesigner-specified goals. Regression models predictinghow the interface will be perceived are also necessary, aswell as a way of anticipating confusions between elements.Essentially, what we need is a model of failure behaviour ininteraction, based on the visual layout. Clearly this ademanding task, but even a quasi-realistic agent would bevery effective in improving this step of the design process.

EvaluatingIn the evaluation step, the designer conducts tests in orderto determine how usable and aesthetically pleasing a designis, and whether or not it meets the requirements. Severalmethods for conducting user studies to evaluate thesefactors are available. Depending on the result, the designerreturns to an earlier stage of the process — either to rethinkthe requirements or create new alternatives — and iterates.

Involving the user in the design process and especially inthe evaluation is a fundamental idea of user-centereddesign. Nevertheless, the recruiting of users, preparation

and execution of studies all require significant time andresources within this iterative process. The end result istypically that only a small number of design alternatives areconsidered.

We see opportunities to improve the whole process oflayout design by automatically testing the usability andreadability of a layout, as well as an evaluation of how visualpleasing the created layout is. This will not eliminate thenecessity of evaluating a layout with real users, but it willresult in faster iterations and improved outcome before userevaluation. Further, it allows for checking a diverse set ofdesigns quickly, allowing designers to explore the designspace more fully.

From a machine learning standpoint, the main challengehere is automatic evaluation of readability, usability andaesthetic preference. These are all effectively complexregression problems. Given the colours, fonts, and layout ofa design, we must predict how it will be perceived by users.Previous research has attempted to do this using metricsbased on e.g. symmetry and grid alignment [3] or visualsearch models from psychology [1]. Using machine learningalgorithms and crowdsourced data, there is significantpotential for improvement in this area.

OutlookThere are many opportunities for the application of machinelearning to the interaction design process. We believe thatresearch in this area could offer significant benefits todesigners, clients, and users by increasing the quality ofdesigned layouts and producing guarantees about thequality of the results based on learned preference models.

For our own part, we see the greatest potential in the‘designing alternatives’ and ‘evaluating’ steps of the lifecyclemodel. We have begun preliminary work on a number of

tools to support this activity. First, we have developed acrawler which analyses web pages and attempts toautomatically group the HTML layout elements intosemantically meaningful groups, using the Gestalt Laws ofperceptual organization. An example output is shown inFigure 2. From this, we intend to generate probabilitydistributions for the placement and grouping of elements, asdescribed above, so that we can learn common patterns ofweb design.

These distributions will be used to improve our second tool,which is an interactive sketching application that allowsdesigners to create wireframes for web layouts. Theapplication suggests reorganisations and recolourings ofthe layout which may be more performant or aestheticallypleasing. The distributions we learn from the web crawlerwill be used to improve these suggestions. Currently weevaluate layouts using performance metrics like Fitts’ Lawand models of visual preference from the psychologyliterature. In the future we hope to supplement these withmachine learned models based on crowdsourcedpreferences for different layouts.

We hope that this paper will spur discussion on thispromising research topic, and inspire new work that usesmachine learning to support creativity and design.

Figure 2: A webpage (top) and theoutput of our page analysisalgorithm (bottom). HTMLelements are automaticallygrouped into meaningful groupsusing Gestalt Laws.

REFERENCES1. David E. Kieras and Anthony J. Hornof. 2014. Towards

Accurate and Practical Predictive Models ofActive-vision-based Visual Search. In Proceedings of

the SIGCHI Conference on Human Factors inComputing Systems (CHI ’14). ACM, New York, NY,USA, 3875–3884. DOI:http://dx.doi.org/10.1145/2556288.2557324

2. Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres,Maxine Lim, Salman Ahmad, Scott R. Klemmer, andJerry O. Talton. 2013. Webzeitgeist: Design Mining theWeb. In Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems (CHI ’13). ACM,New York, NY, USA, 3083–3092. DOI:http://dx.doi.org/10.1145/2470654.2466420

3. Aliaksei Miniukovich and Antonella De Angeli. 2015.Computation of Interface Aesthetics. In Proceedings ofthe 33rd Annual ACM Conference on Human Factors inComputing Systems (CHI ’15). ACM, New York, NY,USA, 1163–1172. DOI:http://dx.doi.org/10.1145/2702123.2702575

4. Peter O’Donovan, Aseem Agarwala, and AaronHertzmann. 2015. DesignScape: Design withInteractive Layout Suggestions. In Proceedings of the33rd Annual ACM Conference on Human Factors inComputing Systems (CHI ’15). ACM, New York, NY,USA, 1221–1224. DOI:http://dx.doi.org/10.1145/2702123.2702149

5. Jenny Preece, Helen Sharp, and Yvonne Rogers.2015. Interaction Design-beyond human-computerinteraction. John Wiley & Sons.