How novices design business processes Jan Recker n , Norizan Safrudin, Michael Rosemann Information Systems Discipline, Queensland University of Technology, 126 Margaret Street, Brisbane, QLD 4000, Australia article info Available online 22 July 2011 Keywords: Design skills Process modeling Design quality Experiment abstract Process modeling is an important design practice in organizational improvement projects. In this paper, we examine the design of business process diagrams in contexts where novice analysts only have basic design tools such as paper and pencils available, and little to no understanding of formalized modeling approaches. Based on a quasi- experimental study with 89 BPM students, we identify five distinct process design archetypes ranging from textual to hybrid and graphical representation forms. We examine the quality of the designs and identify which representation formats enable an analyst to articulate business rules, states, events, activities, temporal and geospatial information in a process model. We found that the quality of the process designs decreases with the increased use of graphics and that hybrid designs featuring appropriate text labels and abstract graphical forms appear well-suited to describe business processes. We further examine how process design preferences predict formalized process modeling ability. Our research has implications for practical process design work in industry as well as for academic curricula on process design. & 2011 Elsevier Ltd. All rights reserved. 1. Introduction When seeking to (re-) design business processes, orga- nizations often use graphical documentations of their busi- ness processes—so-called process models [1]. These models are blueprints of organizational processes that capture, in some graphical and/or textual notation, at least the tasks, events, states and business rules that constitute a business process [2]. Process models are used as a key tool in organizational re-design decisions, i.e., decisions about where, how and why changes to the processes should be enacted to warrant improved operational efficiency, cost reductions, increased compliance or better IT-based systems [3]. In such decision-making tasks, a process model is essentially a cognitive design tool that allows the process analyst to offload memory and information processing, and to promote discovery and inferences about the process at hand [4]. When the process design activity is not computer-supported (e.g., through a modeling tool), analysts use basic tools such as pencil and paper to illustrate how a business operates at present (as-is process design) or in the future (to-be process design). Our interest in this paper is in the way analysts use the affordances offered by paper and pencil to create business process design representations. Specifically, we seek to understand how novice analysts create business process design representations when they are uninformed of any process design method (such as standardized process modeling notation like BPMN [5]). Studies of process design in industry practice [6] still report on the wide- spread use of such ‘butcher paper’ process design work, where tools such as whiteboards, flip charts and post-it notes are used to capture knowledge about a current or future process. Deriving insights on the use and quality of these external representations may therefore promote an understanding of how individuals form their own cogni- tive framework in process design work [7]. Contents lists available at ScienceDirect journal homepage: www.elsevier.com/locate/infosys Information Systems 0306-4379/$ - see front matter & 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.is.2011.07.001 n Corresponding author. Tel.: þ61 07 3138 9479; fax: þ61 07 3138 9390. E-mail addresses: [email protected] (J. Recker), [email protected] (N. Safrudin), [email protected] (M. Rosemann). Information Systems 37 (2012) 557–573
Transcript
Contents lists available at ScienceDirect
Information Systems
Information Systems 37 (2012) 557–573
0306-43
doi:10.1
n Corr
fax: þ6
E-m
norizan
m.rosem
journal homepage: www.elsevier.com/locate/infosys
How novices design business processes
Jan Recker n, Norizan Safrudin, Michael Rosemann
Information Systems Discipline, Queensland University of Technology, 126 Margaret Street, Brisbane, QLD 4000, Australia
Process modeling is an important design practice in organizational improvement
projects. In this paper, we examine the design of business process diagrams in contexts
where novice analysts only have basic design tools such as paper and pencils available,
and little to no understanding of formalized modeling approaches. Based on a quasi-
experimental study with 89 BPM students, we identify five distinct process design
archetypes ranging from textual to hybrid and graphical representation forms. We
examine the quality of the designs and identify which representation formats enable an
analyst to articulate business rules, states, events, activities, temporal and geospatial
information in a process model. We found that the quality of the process designs
decreases with the increased use of graphics and that hybrid designs featuring
appropriate text labels and abstract graphical forms appear well-suited to describe
business processes. We further examine how process design preferences predict
formalized process modeling ability. Our research has implications for practical process
design work in industry as well as for academic curricula on process design.
& 2011 Elsevier Ltd. All rights reserved.
1. Introduction
When seeking to (re-) design business processes, orga-nizations often use graphical documentations of their busi-ness processes—so-called process models [1]. These modelsare blueprints of organizational processes that capture, insome graphical and/or textual notation, at least the tasks,events, states and business rules that constitute a businessprocess [2]. Process models are used as a key tool inorganizational re-design decisions, i.e., decisions aboutwhere, how and why changes to the processes shouldbe enacted to warrant improved operational efficiency,cost reductions, increased compliance or better IT-basedsystems [3].
In such decision-making tasks, a process model isessentially a cognitive design tool that allows the process
ll rights reserved.
analyst to offload memory and information processing,and to promote discovery and inferences about theprocess at hand [4]. When the process design activity isnot computer-supported (e.g., through a modeling tool),analysts use basic tools such as pencil and paper to illustratehow a business operates at present (as-is process design) orin the future (to-be process design).
Our interest in this paper is in the way analysts use theaffordances offered by paper and pencil to create businessprocess design representations. Specifically, we seek tounderstand how novice analysts create business processdesign representations when they are uninformed of anyprocess design method (such as standardized processmodeling notation like BPMN [5]). Studies of processdesign in industry practice [6] still report on the wide-spread use of such ‘butcher paper’ process design work,where tools such as whiteboards, flip charts and post-itnotes are used to capture knowledge about a current orfuture process. Deriving insights on the use and quality ofthese external representations may therefore promote anunderstanding of how individuals form their own cogni-tive framework in process design work [7].
J. Recker et al. / Information Systems 37 (2012) 557–573558
When given basic cognitive design tools without theuse of a (semi-) formalized design method such as BPMN,individuals have numerous ways to illustrate a businessprocess design. For instance, their design diagrams mayentail the use of textual descriptions, graphical icons,geometric shapes or even cartoon sketches, to name justa few. An example for such an informal design diagram,representing an airport check-in and boarding process, isgiven in Fig. 1.
The aim of our research is three-fold. First, we seek tounderstand which design forms novice analysts choose whenconceiving business process diagrams with paper and pencil.Second, we seek to establish differences between theseprocess design types in terms of their ability to representrelevant information about the business process. Third, weseek to examine whether novice analysts with a certainpreference for a process design type learn and performformalized process modeling more successfully than others.
To that end, in this paper we report on an empiricalanalysis of process design work carried out by studentanalysts as part of their university coursework. We statethe following research questions:
RQ1 How can process design representations chosenby novice analysts be characterized?
RQ2 How good are different types of process designsin describing important elements of a businessprocess?
RQ3 How do students with different preferences fortypes of process designs perform in using for-malized process modeling notations?
We proceed as follows. In Section 2, we review priorwork on process modeling as a design activity, and relatedwork from design disciplines that provide an understand-ing of the design process as such. In Section 3, we discussour research model. In Section 4, we discuss how wecollected data on informal business process designs bynovice analysts, and how we prepared this data foranalysis. Then, in Section 5 we give the results from ourstudy, and present in Section 6 a discussion of theseresults. In Section 7 we discuss implications and limita-tions of our research, before we conclude this paper inSection 8 by reviewing our contributions.
Fig. 1. Example of an informal b
2. Background
2.1. Process models as design artifacts
The common aim of process representations such asprocess models is to facilitate a shared understanding andto increase knowledge about a business process [8]. Thisrepresented process knowledge is meant to supportproblem solving in the form of (re-) design decisions—atask performed by business analysts and systems designers,for instance, in the context of organizational re-structuring,compliance management or workflow implementations [9].
Following [10], we view process modeling as a design
activity because process models are used to represent aprocess problem in order to make potential solutionsapparent. Being the most commonly employed cognitivevehicle in process (re-) design work, process models need tobe readily and intuitively understandable by the variousstakeholder groups engaged in this work to facilitate theidentification of process problems and their solutions [11].
To gauge how well process models support suchproblem solving, various approaches have been suggestedto measure the quality of a process model. Discussions ofthese approaches are provided, for instance, in [12–14].Yet, these quality measures only apply to formalized
process modeling methods such as Petri Nets, EPCs orBPMN. In contrast, these approaches are not applicable toinformal process design representations such as sketches,doodles, diagrams or free text that do not follow anexplicit meta model or well-defined graph-based rules.Still, evidence suggests that such informal representationsare still widespread in actual practice [15].
For us to be able to judge the quality of informal businessprocess design representations, we therefore turn to moregeneral diagram correctness criteria suggested by Yang et al.[16], and define the quality of a process design as its abilityto accurately represent all the important constituent factorsof a business process in context, i.e., the activities, events,states and business rule logic that constitute a businessprocess [2]. We complement these process-specific cor-rectness criteria with two criteria found to be importantin general design work, viz., temporal and geospatial
design information [4,17]. These two criteria, in a processdesign, relate to where (geographical location) and when
usiness process diagram.
Fig. 2. Example of a formalized BPMN business process diagram.
J. Recker et al. / Information Systems 37 (2012) 557–573 559
(temporal location) work tasks in a business process haveto be carried out.
Fig. 2 illustrates how typical formalized cognitivedesign vehicles used in process modeling practice, in thiscase a BPMN process model, meet these criteria. Specifi-cally, it shows that while activities, events, states andbusiness rules are graphically represented, temporal andgeospatial design information are normally absent fromthese design representations.
2.2. Process modeling as a design activity
Process modeling, as any design work, is a cognitiveactivity [18]. Regardless of the work discipline, designsbear similarities, particularly in terms of the cognitiveapproach taken by the designer. For instance, an archi-tectural student is more likely to generate multiplesolutions to a problem before arriving at a final design,whereas a science student is more likely to analyze aproblem thoroughly before drawing out only one designsolution [18].
Viewing process modeling as a design activity, there-fore, suggests the importance of prior experience indesign approaches (e.g., experience in process modelingmethods) to this activity. For instance, Wang and Brooks[19] found that novice modelers conceptualize importantdomain elements in a fairly linear process in contrast toexperts, who were found to have better analysis andcritical evaluation skills.
Looking at the artifacts created in process design work,business process diagrams, at a very simple level, typicallyentail the use of graphic icons, basic geometric shapes andtextual information [11]. Several studies highlight theimportance of these visual means to aid understanding ofthe design outcome [20]—which is the key premise under-lying process modeling as a design activity [21]. Visual
attributes function as an aid for the human mind torecognize and group objects in diagrams [22]. Work onimagery, for instance, has shown how images have parti-cular properties [23] that can affect interpretations. Thesefindings suggest that different types of visual aids used inbusiness process design will affect interpretation and under-standability of the created process models.
Investigating the notion of a process diagram as avisual aid to problem solving further, often, conceptualdesign work is carried out using informal sketching, aprocess of mental imagery with the purpose of identifyingproperties of imaged elements to enable the retrieval ofinformation from memory [24]. Like drawings, sketchingplays a consistent role in the generation, development,evaluation and communication of ideas [7], which sug-gests their applicability to process (re-) design activities.
Within sketches as well as more formal process dia-grams, the use of graphical icons and geometric shapes isoften prevalent. The types of graphics that can be found inprocess diagrams can be categorized as Concrete andAbstract. Concrete, high-imagery and high-frequency gra-phics, are often represented with freehand sketches ofobjects such as stickman figures and telephone icons (seeFig. 5c–e for examples). These graphic icons are often said tobe quicker and easier to recognize than text [25]. In someprocess diagrams we find such icons to denote special typesof events (e.g., messages that arrive) or certain actors (e.g.,human agents). Abstract graphics, on the other hand, arelow-imagery, low-frequency graphics that entail geometricshapes and arrows [26] (see Fig. 5c for an example). Informalized process models we find that many elements of abusiness process are represented through dedicated geo-metric shapes such as rectangles (for activities), circles (forevents) or diamonds (for business rule logic).
Also, process diagrams typically feature textual infor-mation in the form of labels attributed to geometric
J. Recker et al. / Information Systems 37 (2012) 557–573560
shapes (like activity boxes) or additional free-text des-criptions [11]. Textual information plays a vital role inensuring proper interpretation and association, as well asto enhance the building of a cognitive model [22]. Textualinformation further enhances the graphical information ina process diagram, because textual and graphical infor-mation can be processed in parallel through the comple-mentary receptor channels of the human brain [27].
2.3. Summary
Above we discussed prior work that suggests thatprocess modeling can be seen as a cognitive design activity,and the resulting process diagrams as cognitive designvehicles. Based on this view and the findings to date, wecan thus assert that process design work concerns at leastthe following questions, which we attempt to answer in thestudy we describe below:
�
Which representation aids are used in process design(e.g., the use of textual means, abstract geometricshapes, concrete iconic imagery and the like)? � How well do process design means enable a reader to
receive all relevant information about a businessprocess (such as important events, activities, states orbusiness rules)?
� Whether and how are temporal or geospatial informa-
tion about a business process represented in a diagram?
� How do individual experience levels, specifically with
design work, with modeling approaches or with theprocess itself, contribute to the design work?
3. Research model
Based on our review of relevant work, we conceptualizeour approach towards answering the research questions
Informal ProcesDesign Work
F: Process DesRepresentat
O: Diagram Cla
F: Process DesRepresentat
O: Semantic CoAssessment
Prior Experience
F: Method Knowledge
O: Process Modeling Experience
Data Modeling Experience
Object-Oriented Modeling Experience
F: Domain Knowledge
O: Experience with Airport Domain
F: Artistic Competency
O: Drawing skill Assessment
KEY F: Theoretical Factor O: Operationalisation of Factor
Fig. 3. Researc
that we address in this study in the research model shownin Fig. 3. This model suggests that informal process designwork is a function of prior experience, and that formalprocess modeling performance is influenced by informaldesign work.
In line with our research questions, first, we seek tounderstand the types of process design representationschosen by novice analysts. To that end, we seek toascertain to which extent prior experience determinesthe type of process design representation used. As perFig. 3, we distinguish three forms of experience: followingKhatri et al. [28], we differentiate (a) experience with amethod (a modeling approach) from (b) experience with aprocess (knowledge of the process domain). We anticipatethat novice analyst with an educational or working back-ground in any formalized modeling approach (data-, process-or object-oriented) would have a predisposition towards thediagraming representation typically associated with themodeling approach, which can be expected to affect theirpreference for such a process design representation type.Domain knowledge has been shown to affect modelingprocesses and outcomes [28], and may thus influence boththe type and quality of the process design conceived. Finally,given the importance of graphical and visual cues in con-ceptual design work [22,23], we further expect that noviceanalysts with experience in graphical design work maychoose a design representation format that is more graphi-cally than textually oriented.
Second, we seek to examine the outcome of the processdesign work. Following Fig. 3, our interest in the outcome ofthe design process is two-fold, namely the type of processdesign representation chosen by the novice analysts, andthe quality of the design types used in representing abusiness process correctly and accurately.
Third, we are interested in examining whether thepreferential design styles, which novice analysts use when
s
ign ion Type
ssification
ign ion Quality
rrectness
Formal Process Modeling Performance
F: BPMN Model Quality
O: Exam Task Score
F: Formal Process Modeling Skills
O: Overall Exam Score
h model.
J. Recker et al. / Information Systems 37 (2012) 557–573 561
uninformed of formalized methods, and the quality of thosedesigns allow us to predict how well the novices learnstandardized formalized process design notations suchas BPMN. To that end, following Fig. 3, we examine therelationships between the type of process design represen-tation chosen by the novice analysts and the quality of thedesign types used on one hand, and their performance informalized process modeling, in terms of the quality of aformal process model design, and the overall performancein a formalized modeling course, after one semester ofcoursework on the other hand. In the following, we describehow we collected data to examine our research model.
4. Research method
We collected data using a four-part quasi-experimentconducted with a group of students enrolled in a BusinessProcess Modeling subject unit as part of their universityInformation Systems course. The first three parts of theexperiment took place during opening minutes of thefirst lecture on the subject in a lecture hall, consumingapproximately 25 min. The last part of the data collectiontook place as part of the mandatory written exam at theend of the semester.
The first part of the experiment captured demographicinformation about the students, viz., their level of educa-tion (under-graduate or post-graduate), gender and Eng-lish Language as their arterial language. This data wascollected to describe a sample profile of our participants,in order to allow readers to judge the representativenessof our study population as well as the potential threats toexternal validity [29]. Additionally, we collected dataabout the participants’ experience in formalized modelingmethods (process-, data- and/or object-oriented), andtheir familiarity with the procedures at an airport, whichwas the process domain selected for our study. This datais important for us to examine the potential effects ofmethod and domain knowledge on design representationtype and quality, as per our research model (see Fig. 3).
The second part of the experiment aimed at assessingthe students’ ability to draw graphical diagrams, as aproxy measure for graphical design skills. To that end, apicture of the Sydney Opera House was projected to theparticipants, who were to draw an accurate sketch ofthe image on a blank piece of paper. The rationale behindthe Sydney Opera House image was based on the assump-tion that the majority of the participants would be familiarwith the landmark, as it represents one of Australia’s mostprominent features. Students were given 10 min to com-plete this task but task times were not recorded.
The third part of the experiment was to examinethe students’ ability to create a business process designrepresentation. A specific process scenario was portrayedin textual format to the participants as a narrative of anactor seeking to travel to Sydney. This included a detailedaccount of the arrival at the airport, followed by check-inand boarding procedures and leisurely activities taken inbetween. The rationale behind this activity was to providea business process with which both domestic and inter-national students would have some level of familiaritywith (as opposed to a business process in a specific
industry vertical – for instance, insurance – where resultscould have been significantly biased due to non-existenceof any domain knowledge). Students were instructedorally to draw a model that represents the airport processscenario as accurately and completely as possible, within10 min, using only a blank piece of paper and pencil.
The fourth and last part of the data collection occurredat the end of the teaching term, as part of the mandatorywritten exam. As part of this exam, the students were inone task asked to develop a model of the airport processscenario using the formalized process modeling notationBPMN that they had been taught over the course of thethirteen-week semester. Students were asked to draw aBPMN model that represents the airport process scenarioas accurately and completely as possible, using as manyconstructs of the BPMN notation as they deemed requiredand appropriate. All relevant measures from the datacollection are provided in Appendix A.
5. Results
5.1. Demographic statistics
Overall, 89 students participated voluntarily in thestudy. After eliminating incomplete and unusable entries,we retained complete and usable data about the firstthree parts of the experiment from 75 students (84%).Incomplete entries included those where several parts ofthe data collection (e.g., the drawing of the opera house,or the drawing of a process diagram) were not completedby the participants. Unusable entries were those whereentries were not readable (e.g., demographic data, or theprocess diagram). A likely explanation for the unusableentries is the fact that we relied on a paper and pencilsetting, where participants could not be closely monitoredfor completing the experiment tasks accurately. We werefurther able to collect the exam grades from a total of59 students (67%). Table 1 provides further descriptivestatistics about our participants.
With this dataset, first, we coded the demographicinformation obtained. Our specific interest was in students’experience with airport processes (domain knowledge), aswell as experience in formal modeling methods—processmodeling knowledge (PMK), data modeling knowledge(DMK) and object modeling knowledge (OMK). Table 1shows that, within our study population, 60% of participantshad prior experience in process modeling, 36% of partici-pants had prior experience in data modeling and 41% ofparticipants had prior experience in object modeling.
Next, we assessed the quality of the Opera housedrawings, to create a measure of graphical design skills.To that end, all drawings were provided to a professionalartist, who judged each drawing using a six-item drawingquality measure that assessed composition (COM), pro-portions (PROP), perspectives (PERS), shading (SHAD),drawing style (STY) and overall impression (IMP) of thedrawings on a 7-point scale (1¼very bad, 4¼neutral,7¼very good).
Table 1 shows that participants ranged in terms oftheir graphic design skills but the overall level of designskills was low. The standard deviation was above 1 for all
Table 1Participant demographics.
Variable Scale Mean St.dev.
Distribution
Gender Male/female N/A N/A 76% male, 24% female
Study course Under-graduate/post-
graduate
N/A N/A 44% under-graduate, 56% post-
graduate
English as an arterial language Yes/no N/A N/A 56% native speakers, 44% non-native
speakers
Experience with airport processes (domain
knowledge—DK)
0–10 6.14 2.13
Prior process modeling knowledge (PMK) Yes/no N/A N/A 60% with PMK, 40% without PMK
Prior data modeling knowledge (DMK) Yes/no N/A N/A 36% with DMK, 64% without DMK
Prior object modeling knowledge (OMK) Yes/no N/A N/A 41.3% with OMK, 58.7% without OMK
General graphic impression (IMP) 1–7 2.57 1.37
Correctness (COR) 1–7 2.33 1.02
Composition (COM) 1–7 3.36 1.74
Proportions (PROP) 1–7 2.81 1.53
Perspective and background (PERS) 1–7 2.53 1.46
Shading and shape design (SHAD) 1–7 2.51 1.30
Drawing style (STY) 1–7 2.83 1.40
Fig. 4. Process design archetypes.
J. Recker et al. / Information Systems 37 (2012) 557–573562
graphic design skill dimensions, but the mean was alwayslower than the mid-point 4, with composition skills beingrated highest (mean¼3.36) and graphical correctnessbeing rated lowest (mean¼2.33). Again we believe theseresults are realistic and sensemaking for the cohort ofInformation Systems and Information Technology stu-dents that participated in our study.
5.2. Identifying process design types
To distinguish different process design representationtypes, we categorized the various types of process designrepresentations created in the third part of the experi-ment, in accordance with their esthetic design properties.This assessment included the examination of the relativeuse of graphical icons, textual information and sequentialflow or structure of the process diagram. To ensure codingreliability, all diagrams were assessed separately by threeresearch assistants, who then, iteratively, met to discuss,defend and revise their coding work until consensus wasreached.
Our coding of the 75 process diagrams resulted in theidentification of five process design archetypes. This assess-ment was based on the esthetic representation of the processdiagrams, such as frequency of graphic use, textual informa-tion and the sequential flow of the process structured withinthe Euclidean space afforded by the piece of paper. Similar tothe Physics of Notations suggested by Moody [20], we foundthat the archetypes could be differentiated based on theiruse of text and graphics. Fig. 4 positions the five identifiedarchetypes along a continuous scale from dominantly textual(type I) to dominantly graphical (type V) representationformats, and describes key traits of each design type.
Fig. 5a–e provides examples for each design archetypeand displays information about absolute and relativefrequency of occurrence in our dataset.
The first type, Textual design (Fig. 5a), resembles veryclosely to that of an algorithm pattern. This design type doesnot utilize any form of graphical illustration but uses lines ofwords as the primary representation of process information.The second type, Flowchart design (Fig. 5b), contains textualinformation embedded within graphical shapes that are of
Fig. 5. Process design archetype examples. Process Design Type 1: Textual (1 diagram—1.3%); Process Design Type 2: Flowchart (54 diagrams—72.0%);
Process Design Type 3: Hybrid (6 diagrams—8.0%); Process Design Type 3: Hybrid (6 diagrams—8.0%) and Process Design Type 5: Canvas (3 diagrams—
4.0%).
J. Recker et al. / Information Systems 37 (2012) 557–573 563
abstract nature, i.e., lines/arrows and/or boxes and border-lines around captions, and generally have a sequential flowthat, to some extent, resembles more formal modelingtechniques used for process-, data- or object-modeling, andof course, the classical flowchart. The third type, Hybrid
designs (Fig. 5c), uses concrete graphics (such as stickmanfigures, telephone icons and the like) to supplement thetextual labels and descriptions in the presence of abstractgraphics (shapes and boxes). The Hybrid design also follows astructured process flow of information. The fourth and fifth
design types are notable due to the distinctively dominantuse of concrete graphics over and above textual representa-tions. The Storyboard design (Fig. 5d) uses a great variety ofconcrete graphics such as icons, complemented with brieftextual descriptions, typically in the form of verbs and nouns.Resembling a real ‘‘Storyboard’’, this design type furtherfeatures segmented pieces of information, some partitionedas objects within rectangular boxes (abstract graphics) orswim-lanes, and were structured in a flowing manner toaccommodate the Euclidean space and orientation of the
Table 2MDA: significance tests of equality of group means.
Factor Wilks’ Lambda F df1 df2 Sig.
PMK 0.97 0.63 4.00 70.00 0.65
DMK 0.97 0.59 4.00 70.00 0.67
OMK 0.91 1.72 4.00 70.00 0.15
DK 0.89 2.06 4.00 70.00 0.10
IMP 0.93 1.32 4.00 70.00 0.27
COR 0.98 0.30 4.00 70.00 0.88
COM 0.91 1.72 4.00 70.00 0.16
PROP 0.95 0.96 4.00 70.00 0.44
PERS 0.95 0.85 4.00 70.00 0.50
SHAD 0.94 1.10 4.00 70.00 0.36
STY 0.96 0.78 4.00 70.00 0.54
J. Recker et al. / Information Systems 37 (2012) 557–573564
paper. Last, the Canvas design (Fig. 5e) illustrates the entireprocess with concrete graphics and without any meaningfuluse of textual information, occupying the entire page of thepaper to provide a picturesque view of the scenario. Due tothe ‘‘picture-painting’’ nature of this design, the diagramlacks any precise representation of the process flow, ordetailed textual information.
5.3. Explaining the process design types
Having distinguished the five different process designrepresentation types, we then examined whether any ofthe experience factors we considered (method, domain orgraphical design experience) were significantly associatedwith any of the design representation types chosen by theparticipants. Such associations could indicate reasonswhy participants would opt for a certain process designrepresentation type.
To that end, we ran a multiple discriminant analysis(MDA), which is an approach used to classify a categoricaldependent that has more than two categories, using aspredictors a number of interval-scaled or categorical inde-pendent variables [30]. This statistical technique is used toclassify cases of a dependent variable, in our case, thedifferent categories of design types, ranging from textual(DT1) to fully graphical (DT5) designs, by estimating howwell a set of variables (in our case, personal demographicssuch as experience and graphical design skills) predict thecorrect classification into one of the categories of thedependent variable. If the multiple discriminant analysis iseffective for a set of data, the classification table of correctand incorrect estimates of the classification of cases into thefive categories will yield a high percentage correct.
Multiple discriminant analysis works by estimatingthe so-called canonical roots, which is a latent variablethat is created as a linear combination of the discriminat-ing (independent) variables. The canonical roots areestimated such that the distance between the means ofthe criterion (dependent) variable (in our case, the differ-ent types of process designs) are maximized.
Based on the maximum likelihood estimation of thenumber of canonical roots, multiple discriminant analysisconsiders two tests: first, an F test (Wilks’ lambda) is usedto test if the discriminant model of all canonical roots asa whole is significant and second, if the F test showssignificance, then the individual independent variables areassessed to see which differ significantly across the groups,and these are used to classify the dependent variable.
Like multiple regression, MDA assumes proper modelspecification (inclusion of all important independents).Based on our literature review, this condition was pre-sumed to be correct in our case. MDA, furthermore, is apreferred alternative to logistic regression, because it hasmore statistical power than logistic regression (and thusless chance of type 2 error) and is able to extend theanalysis to dependents with more than two categories,which was important in our case.
Our dependent variable was design type (with thegroups 1–5), and as independent variables we used threebinary variables PMK, DMK, OMK capturing respondents’prior experience with different modeling methods, and
six interval factor scores (IMP, COM, PROP, PERS, SHAD,STY) describing the graphical design skills as per theevaluation from a professional artist. Last, we used theinterval factor score domain knowledge (1–10) to includethe self-perceived rating of familiarity with airport pro-cedures as another independent variable.
Table 2 and Appendix B give the results from multiplediscriminant analysis, including descriptive statistics(Table B1). Specifically, Table B2 shows that the discri-mant model estimated through the MDA consists of fourcanonical roots. Table B3 shows how each of the indepen-dent variables is related to each of the four canonical roots(discriminant functions). Importantly, however, the data inTable B4 shows that the discriminant model described bythe four canonical roots has insignificant explanatory power(the significance of the canonical roots in correctly predict-ing the classification of design types into the five identifieddesign types ranges from p¼0.19 to 0¼0.57). These resultsshow that the linear model of the canonical roots is not
appropriately discriminating.Furthermore, the data in Table 2 shows that the model
of the independent factors considered fails to explain thecategorization into the design types DT1 (Textual design)through to DT5 (Canvas design). None of the independentvariables considered is a significant predictor of the classi-fication of the design types, with p-values ranging from0.10 (domain knowledge) to 0.88 (design correctness).These results suggest that, while we identified five prefer-ential styles of process design, the preference for any ofthese styles cannot be explained through either previousmodeling knowledge factors (PMK, DMK or OMK), throughgraphical design skills (IMP, COR, COM, PROP, PERS, SHAD,STY) or through experience within the domain modeled(DK). We note, however, that part of the non-significance ofthe results may be due to the limited relative sample sizefor some of the design types (e.g., DT1 or DT5), althoughequal group size is not a necessary assumption for MDA.
5.4. Evaluating process design quality
In our next analysis, we attempted to measure the qualityof each process design representation. To that end, weadapted the semantic correctness criteria suggested by Yanget al. [16] to the constituent elements of business processmodels (activities, events, states, business rules, see [2]) in asix-item 5-point scale (1¼aspect not at all represented,
Table 3Univariate ANOVA test results.
Factor df F Sig. Partial eta squared
Corrected model 49 3.55 0.001 0.87
Intercept 1 1025.45 0.00 0.98
Design type 4 12.46 0.00 0.67
Domain knowledge (groupDK) 1 9.57 0.01 0.28
Prior process modeling knowledge (PMK) 1 0.16 0.69 0.01
Prior data modeling knowledge (DMK) 1 3.96 0.06 0.14
J. Recker et al. / Information Systems 37 (2012) 557–573 565
5¼aspect fully represented). Our coding protocol noted 4activities, 4 events, 4 business rules and 6 states from thedescription of the airport process scenario. We then distrib-uted the item count throughout the 5-point scale; forinstance in the case of activities, we noted ‘1’ as activitynot at all represented, ‘2’ as 1 activity represented, ‘3’ as 2activities represented, ‘4’ as 3 activities represented and ‘5’as all 4 activities represented. Again, we used a three-member coding team and an iterative consensus-buildingprocess to ensure validity and reliability of our assessment.The protocol detailing the coding scheme is available inAppendix C.
In examining the data collected about the quality ofthe process designs, we proceeded in two steps. First, weran a Univariate Analysis of Variance (ANOVA, [31]), withDesign Quality (DQ) as an aggregate dependent variable,computed as the average total factor score of the sixsemantic correctness scale items. As independent factors,we used design type (DT), a binary grouping variabledomain knowledge (groupDK) that grouped respondentsinto two classes—participants with high levels of previousknowledge of airport procedures (domain knowledgescore 6–10) and those with low levels (domain knowledgescore 0–5). Also included as independent factors were thethree binary measures for previous modeling methodknowledge (PMK, DMK and OMK) and the graphic designscore overall impression (IMP). Table 3 gives the results.
The data in Table 3 shows that Design Type (F¼12.46,p¼0.00) and previous domain knowledge (F¼9.57, p¼
0.01) are significant predictors of the aggregate designquality measure, whilst the other independent factorswere insignificant.
We found from the ANOVA, firstly, a significant asso-ciation between higher levels of domain knowledge and
process design quality scores. Secondly, we found thedesign type to be significant indicator for process designquality scores, suggesting differences in semantic correct-ness between the textually oriented and the graphicallyoriented process design representation types (as per theclassification in Fig. 4).
To examine these results in more detail, we then ran aMultivariate Analysis of Variance (MANOVA), with the sixsemantic correctness measures as dependent variables,and the same input factors as above. Table 4 givesselected descriptive results from the MANOVA about theimpact of the Design Type, and Table 5 displays corre-sponding significance levels.
The results from Tables 4 and 5 suggest that there is arelationship between the type of design employed by thestudents to represent the business processes and thedifferent dimensions of the quality of these designs.Specifically, Table 4 shows that design type 2 (Flowchart)was associated with the highest correctness scores forrepresenting the State, Task, Event and Business Rulesaspects—under elimination of DT1, which only featuredone case. The purely graphical design, DT5 Canvas, wasassociated with the lowest aggregate scores in representingthe six factors that entail the design quality. We further notethat the highest scores for the Distance dimension were
linked to design type 4 (Storyboard). Finally, Table 5 showsthat these score differences are significant, except for thequality dimension Business Rules, where we did not identifya significant association with any of the types of designused.
Table 5 further suggests important interaction effectsstemming from the type of knowledge possessed by theparticipants. We note that participants with prior processmodeling knowledge, when exercised with their choice of
J. Recker et al. / Information Systems 37 (2012) 557–573566
design, achieved higher quality scores for their representa-tions of Time and Distance. Subjects with object modelingknowledge were found to be better in representing Timewith their design type, while those with previous domainknowledge were found to be better in representing Distance.
Perusing MANOVA we further found a number of inter-esting effects on design quality stemming from prior experi-ence of the subjects. Table 6 summarizes the significancelevels for the different types of prior experience captured.
Examination of the data displayed in Table 6 shows thatthose participants with knowledge of process modelingmethods achieved higher scores for representing BusinessRules (p¼0.04). Students with both process and datamodeling knowledge achieved higher scores for represent-ing States (p¼0.02) and Distance (p¼0.010). Time was wellrepresented by students with data modeling knowledge(p¼0.02) and those with object modeling knowledge(p¼0.02). The data also showed the existence of an inter-action effect between students with both data modeling anddomain knowledge in representing Tasks (p¼0.03), whilethose with object modeling and domain knowledge repre-sented Business Rules well (p¼0.05). Last, we found aninteraction effect concerning the representation of States(p¼0.01), for those participants with both object andprocess modeling knowledge. These findings suggest thatdifferent method knowledge, solely or when combined withother method or domain knowledge, can increase thespecific level of quality in a business process diagram.
5.5. Evaluating formal process modeling performance
In a last step, we collected the marked results from thefinal written exam task on modeling the airport process ina BPMN diagram. Students were graded on a scale of 0–10(ten being the highest score) for this task. The gradingscheme allocated up to 5 points for completeness (interms of modeling all tasks, events, actors, states andbusiness rule logic), 1 point for visual intuitiveness (howeasy to read is the model), 2 points for appropriatelabeling of events and tasks (in terms of following thelabeling guidelines as discussed in [11]) and 2 points forsyntactic correctness (in terms of equivalence to thegrammatical rules of BPMN as per specification [32]).Exam marking was completed by an experienced teachingassistant uninformed and unaware of the study.
We collected the individual marks for this task fromthe students as an approximate measure for formal
Table 6Multivariate ANOVA test results (prior experience).
process model quality. For comparison purposes, we alsocollected data on the students’ overall exam performance,on a scale from 0 to 100, to establish an approximatemeasure for the student’s performance in formal processmodeling.
Overall, we received results from fifty-nine studentsthat had undertaken the experiment prior to receivingformal education in process modeling with the formalindustry standard notation BPMN [32] over the course ofthe thirteen-week semester. The course followed thecurriculum described in detail in [33], and ended with awritten individual exam in which the students, amongstother tasks, were asked to draw a formal BPMN model ofthe airport scenario used in the initial experiment (seetask description in Appendix A).
With this data, first, we ran a non-parametric Kruskal–Wallis [34] test to examine the relationship betweenDesign Type chosen by the students at the start of thesemester plus their Task Score, and overall Exam Scores atthe end of the semester. The Kruskal–Wallis test is thegeneralization of the Mann–Whitney test when there aremore than two independent groups, like in our study(five), ranging on an ordinal scale from textual to graphi-cal designs (see Fig. 4). The distribution-free nature ofnon-parametric tests places few restrictions on the sam-ple size in contrast with parametric tests, which rely onasymptotic properties or normality of the sample distri-bution, which was not given in our distribution of processdesign types. Also, rank-based non-parametric tests arenot affected by outliers, which allows us to also considerthose design types that relatively few students selected(e.g., Design Type 5—Canvas). Table 7 gives the resultsfrom the Kruskal–Wallis test. Note that we did not receiveexam scores from students that originally selected DesignType 1—Textual.
The data in Table 7 shows that for both measures offormal process model quality (task score—TS) as well asfor our measure of formal process modeling skills (examscore—ES), there is a downward trend in scores whentraversing from more textual (Design Type 2) towardshighly graphical (Design Type 5) informal process designstyles. Mean scores as well as mean ranks for task as well asexam scores steadily decrease (except for the task score forDesign Type 5). Fig. 6 displays these results graphically.
The Kruskal–Wallis test showed the differences in taskand exam scores between the different design styles to besignificant (TS: w2
¼7.99, df¼3, p¼0.04; ES: w2¼8.89,
df¼3, p¼0.03). These results suggest that students thatinitially chose an informal design type that is character-ized by structured text with abstract graphics (such as the
Table 7Kruskal–Wallis test results.
Designtype
N TS TS ES ESMean (std.dev.)
Meanrank
Mean (std.dev)
Meanrank
2 45 7.89 (1.61) 33.22 85.78 (15.40) 33.67
3 6 7.33 (1.21) 23.83 73.33 (11.54) 19.50
4 7 6.86 (0.90) 16.57 65.14 (24.05) 18.21
5 1 7.00 (0.00) 16.00 64.00 (0.00) 10.50
Table 8Linear regression descriptive results.
Independentvariable
Dependentvariable
Mean Std.dev.
N
TS 7.70 1.51 59
ES 81.69 17.60 59
STATE 2.95 1.04 59
TASK 3.64 1.14 59
EVENT 2.46 1.27 59
BusRule 3.82 0.90 59
TIME 3.10 1.60 59
DISTANCE 3.13 1.75 59
Table 9Linear regression model results.
Independentvariable
Dependentvariable
St. beta t Sig.
STATE Task score (TS) �0.07 �0.46 0.65
Exam score (ES) 0.08 0.55 0.59
TASK Task score (TS) 0.29 1.54 0.13
Exam score (ES) 0.21 1.09 0.28
EVENT Task score (TS) 0.18 1.05 0.30
Exam score (ES) 0.13 0.74 0.47
BusRule Task score (TS) �0.17 �0.88 0.38
Exam score (ES) �0.05 �0.25 0.81
TIME Task score (TS) 0.05 0.25 0.81
Exam score (ES) 0.08 0.38 0.70
DISTANCE Task score (TS) �0.07 �0.32 0.75
Exam score (ES) �0.03 �0.12 0.90
(Constant) Task score (TS) 7.34 0.00
Exam score (ES) 5.47 0.00Fig. 6. Mean exam and task scores per design type.
J. Recker et al. / Information Systems 37 (2012) 557–573 567
one characterizing Design Type 2 in Fig. 5) performedbetter in formal process modeling than those that hadpreferred highly visual process designs featuring concretegraphics and little textual information (e.g., Design Type4 or 5).
Following our research model, in a second test, we setout to examine the relationship between the quality ofthe informal process designs (DQ) and our formal processmodeling quality and skills measures. To that end, weestimated two linear regression models, one with taskscores (TS) as a dependent variable, and the other withthe overall exam scores (ES). Both regression models usedas independent variables the five informal process designquality measures representation of states (STATES), tasks(TASK), events (EVENT), business rules (BusRule), temporalinformation (TIME) and geospatial information (DISTANCE).Table 8 shows descriptive statistics and Table 9 gives theresults from the regression model estimation.
Perusal of the data in Table 9 reveals that none of thequality measures used to gauge the quality of the infor-mal process design at the start of the semester turned outto be a significant predictor of the formal process modelquality (TS) or skills (ES) measure collected at the end of thesemester. Notably, these results indicate that the quality ofinformal process designs is not related to the quality of
formal process designs using a standardized notation suchas BPMN.
Finally, we ran additional MANOVA tests to examine therelationship between prior experience and knowledge(groupDK, PMK, DMK, OMK, IMP) and task and exam scores.These additional tests did not show significant associationbetween any of the independent factors and the dependentvariables and are therefore not reported in detail.
6. Discussion
Our study set out to examine preferential styles ofprocess design when uninformed of the formalized pro-cess modeling notations, the quality of these designs andthe ability to learn formalized process design methods.We identify three core findings from our study thatextend our understanding on the use of conceptual designtools and the quality traits of the design outcomes:
(a)
design representation forms chosen to conceptualizebusiness processes range from predominantly textual,to hybrid, to predominantly graphical types;
(b)
some of the design types, more notably the combinedgraphical and textual types, are associated with higherquality scores and
(c)
students with preferences for textual and abstract butnot concrete graphical design types are associatedwith an improved ability to apply formalized processdesign methods, once knowledge is acquired.
Aside from these general findings, our results also permita number of important specific interpretations. We turn toDual Coding theory [35] to interpret our results in moredetail. This theory stipulates that text and graphics togethercan provide a more effective representation of informationthan using either on their own. We find that Design Types 2(Flowchart) and 3 (Hybrid) both fall under this banner. Yet,our results that participants that selected Flowcharts as adesign type achieved higher semantic correctness scores in
J. Recker et al. / Information Systems 37 (2012) 557–573568
comparison to the Hybrid design type might suggest text andabstract graphics (shapes such as boxes, circles and arrows)may be more effective in displaying important domainsemantic elements than the combined use of text andconcrete graphics (icons such as stickman figures—as foundin Design Type 3, Hybrid). Our findings about the quality ofthe designs as well as the ability to perform in formalizedprocess modeling support this tentative interpretation. Also,during the three-member evaluation of the process diagrams,it was reported that certain concrete icons, when unfamiliarwith the given context, tended to create a certain level ofambiguity towards the end-users. For instance, one of thecoders mis-interpreted a sketched icon representing theutility of an online check-in facility (as per context scenario),as a public restroom. This anecdotal evidence further corro-borates our interpretation. Still, our study allows us to reasononly about the model design choices made by noviceanalysts. Further studies of how well the process informationin these design types can be interpreted by model readers[36] are required to provide conclusive evidence.
We turn to Moody’s [20] theory of effective visualnotations to further examine our findings regarding the useof abstract and concrete graphics in process designs. The useof concrete graphics such as icons can in some instancesviolate the notion of monosemy whereby a symbol shouldhave only one predefined and independent meaning. This isnot to say that all concrete graphics used in diagrams areundefined. For instance, the use of concrete graphics such asstickman figures, which clearly represent the main actor in aprocess, or a combination of a stickman with a telephoneicon, followed by a taxi vehicle, can clearly indicate therepresentation of the actor calling a taxi as described in theprocess scenario. Such icons are of a semantically immediatenature, which allows novices to establish its meaning gra-phically in a process diagram based on their appearancealone [20]. Still, the only partial and inconsistent use ofsemantically immediate concrete graphics in more graphi-cally oriented diagrams (Design Types 3, 4 or 5) may explainwhy we found the more textually oriented process diagrams,such as the Flowchart design, which employ abstract gra-phics such as geometric shapes and arrows, to be associatedwith higher semantic correctness scores. Moody [20] high-lights such symbols as being semantically opaque, in whichthe relationship between a symbol’s appearance and con-notation is merely arbitrary. Note that we found thatpredominantly students with notably high levels of domainknowledge tended to employ this design type withincreased use of text and semantically opaque symbols.This finding would suggest specifically that geometricshapes can faithfully be used to describe different constitu-ent elements of a process such as activities (typicallyrectangles), events (typically circles) or business rules (typi-cally diamond-shaped gateways). It also highlights theimportant role of appropriate textual labels and the impor-tance of conventions to guide the textual semantic specifi-cation of these labels.
Further note that the Flowchart design was also found tobe the most favored type of design by the majority ofstudents (72%), which may not only indicate preference, butperhaps also the novice’s default way of conveying processinformation (using bare minimum concrete graphics).
Turning to what appears to be the second most used typeof design (15% of students), the Storyboard design, we notethat the simultaneous use of both graphics and text, plus astructured flow of process, may imply intuitiveness ofgraphical use to emphasize representation. And indeed, thetheory of spatial contiguity [37] suggests that inclusionrather than segregation of both text and images can be moreeffective towards the end-user in terms of comprehension,regardless of spatial and verbal abilities. This theory mayalso contribute to explaining why we found only one case ofDesign Type 1, Textual design, because, as per theory, suchdiagrams lack the intuitiveness of graphics.
Therefore, we posit that concrete graphic icons, in certaininstances, may be used as an additional vehicle to describerelevant information in intuitive and easy form. They areesthetically pleasing as people generally have a preferenceon real objects rather than abstract shapes. However, ourstudy also shows that abstract icons, in conjunction with theuse of textual information, were significantly often in use bythose participants with low levels of previous modelingknowledge.
Next, we turn to the representation of the ‘‘non-stan-dard’’ contextual process elements temporal and geospatialinformation. We found that participants that employed theStoryboard design tended to achieve the highest correctnessscores in the ‘distance’ dimension. The Storyboard designcomprises of graphics, both abstract and concrete, with littletextual annotation. Notably, we found the most prominentrepresentation to be a signboard graphic icon with the unitof measure (e.g., 3 km).
The highest scores for describing temporal informa-tion, on the other hand, were found to be associated withthe Flowchart design type. In this style, we found thattemporal information was generally described using textlabels and abstract shapes such as additional timelinearrows complementary to the process flow. This findingcould suggest that it is deemed more accurate for theillustrator to use textual descriptions of time periods, asopposed to drawing a clock icon (a concrete graphic) toindicate a particular time or duration. Again we note thatthis finding presents an opportunity to study whether thedesign choice is in alignment with an interpretationchoice (i.e., whether models incorporating these designfeatures indeed improve model comprehension).
Regarding our findings about the linkages betweeninformal process designs and formalized process model-ing at the end of the course, the results in Fig. 6 show thedescending progression in both task and exam scores forthose participants that incorporated more concrete gra-phics in their initial, informal process designs. Those whoemployed a more textual-oriented design, complementedwith the use of abstract graphics (such as the Flowchartdesign), appeared to perform better also in the subse-quent formal process modeling tasks. These findings arein line with a recent study that analyzed students’ mentalconceptualizations via their drawings [38]. This studyfound that a drawing can exhibit a students’ perceptiontowards an object, including the level at which theyobserve details and present them, thereby functioning asa ‘‘window’’ to an individual’s conceptual knowledge. Thisphenomenon may reflect the mirrored trend of scores for
J. Recker et al. / Information Systems 37 (2012) 557–573 569
both formal and informal process design types as thosewho employed the Flowchart design achieved a higherscore in terms of semantic correction. This is in contrastwith subjects who adopted the Storyboard and Canvasdesigns, whose scores are lower than Flowchart andHybrid designs. Such occurrence may indicate a lack ofperception, and subsequent modeling, of detailed infor-mation by participants, which may persist also in theirlater formal process modeling.
The linear regression tests in Tables 8 and 9 reveal anon-existent relationship between the quality of informaland formal process designs, which is further corroboratedfrom the insignificant findings from the additional MAN-OVA tests. These results suggest that formal processmodeling quality or performance is not subject to theability of students to provide high-quality informal pro-cess designs, and suggests that better ‘‘sketching’’ skills donot make a better process modeler. Instead, we mayspeculate that formal process modeling skills will dependupon the students’ ability to comprehend essential pro-cess concepts (such as concurrency, repetition, conver-gence and divergence) [36], and their general aptitude tolearning in a classroom environment [39]. Notably, weobserved that the vast majority of our participants per-formed better in process modeling after having receivedthirteen weeks of process modeling education.
7. Implications and limitations
7.1. Implications for research
The work presented in this paper has important implica-tions for future research. Particularly, we believe that ourstudy provides some valuable insights towards cognitiveaspects of novice process designers, which can be the basisfor further cognitive studies in the field of business processdesign. One of the most apparent connections from our workis to the theory of cognitive fit [40] that stipulates that taskperformance is best when the mental representation of aproblem matches that of the cognitive design vehicle (e.g.,the process model) that is used to solve the problem at hand.Our study provides an initial exploration of the mentalrepresentations of processes as employed by novice analysts,and the way they would choose to externalize these mentalrepresentations when not required to use a formalizeddesign vehicle (such as BPMN). Yet, the quasi-experimentalresearch design restricts our work to a correlational study, inwhich we evidenced that process design archetypes areassociated with different levels of design quality. Still, thissetup does not allow us to reason about causal relationshipsbetween design type and design quality. In identifyingpotential explanatory theories to provide causality to ourfindings, cognitive fit theory could provide an explanatorymechanism to examine the relationships identified throughour analysis. Further work could thus be undertaken toconsider how well existing formalized modeling methodsprovide a fit to the mental representations of the processesbeing analyzed/designed, and the consequential effect on theperformance of the process (re-) design task.
A second stream of research emerges from the settingof our research with novice analysts. The core part of our
study took place prior to classroom teaching in processmodeling. Thus, our study provides important insightsinto the set of potential business analysts before theyundergo education in principles of process, managementor formalized modeling. Future research can now buildupon these insights to examine different modes of teach-ing for different types of novice analysts, following ourclassification scheme to identify the most appropriatelearning mechanisms for different types of analysts.
Last, we identify opportunities for studies to comparethe informal process design representations with typicalformalized representations (e.g., with BPMN, Petri nets orEPCs) alongside quality dimensions such as perceivedsemantic quality [41], ease of interpretation [42] orretention and transfer of domain information [43].
7.2. Implications for practice
Our findings on the various types of design generatedby students provide insights on how individuals withoutexperience in formal modeling method(s) conceptualizeand externalize business processes. These findings areimportant for both process design practice and education.Specifically, the moderate use of graphics and abstractshapes to illustrate a process is often considered moreintuitive and may in turn aid the understanding on theconcept of process modeling. This would benefit theteaching aspect of business process modeling subjects,or any process-oriented disciplines, by introducing aninformal approach before applying formal modelingmethods. This is due to the nature of graphical illustra-tions being intuitive, such as that of concrete icons andabstract symbols used in the Flowchart, Hybrid andStoryboard designs. However, there is also a trade-off inthe quality of process design when graphics are fullyincorporated, which suggests that while graphics may, toa certain extent, aid the understanding and communica-tion of a business process, they could also result in a lossof information due to ambiguity and/or misinterpretation.On the other hand, process designs that fully utilizetextual labels and descriptions, such as that in Textualdesign, may be useful in representing certain processinformation such as Business Rules, but are not entirelyintuitive.
7.3. Limitations
We acknowledge that our study bears certain limitations.First, the subjects observed were students and not businessanalysts. As such, our findings may only hold for noviceanalysts, which, however, was the desired cohort for ourstudy. Second, there could be residual subjectivity in ourcoding of data analysis. We attempted to mitigate potentialbias through a multiple coder approach. Third, our attempt toascertain the designing skills of the students could be seen asan assessment of their drawing but not their actual designskills. Fourth, our quasi-experiment was conducted usingpaper and pencil only. We note that the type of tools andnotations employed heavily influences the way in whichprocess models are constructed [9]. Nonetheless, we deemthe usage of pen and pencil to be valid due to the prevalence
Table B1Descriptive statistics.
Factor Mean St. dev. N
PMK 0.60 0.49 75
DMK 0.36 0.48 75
OMK 0.41 0.50 75
groupDK 0.61 0.49 75
IMP 2.57 1.37 75
COR 2.33 1.02 75
COM 3.36 1.74 75
PROP 2.81 1.53 75
PERS 2.53 1.46 75
SHAD 2.51 1.30 75
STY 2.83 1.40 75
QUAL 2.71 1.16 75
Table B2Summary of canonical discriminant functions.
Function Eigenvalue % ofvariance
Cumulative%
Canonicalcorrelation
J. Recker et al. / Information Systems 37 (2012) 557–573570
of such a setting not only in industry [6,44], but also ineducational classes. Still, our results are not necessarilygeneralizable beyond a paper and pencil setting, becausetools and grammars offer syntactical and perhaps evenmethodological support for modeling, which could influencethe design work by an analyst. Another limitation might behow the scenario was portrayed in a narrative way thatdescribes a specific instance of the process of going to theairport. One can argue that such a story line does not qualifyas a repeatable process due to the introduction of a fixedcharacter, location, distance and so forth. However, ourmotivation was to elicit whether or not particular aspectsof the process are represented by novices, i.e., as per semanticcorrectness criteria stated in the body of this paper. Thus,these elements, when stripped from its encompassing con-tent, are indeed and to a large extent, repeatable for anyinstances of this airport process. Furthermore, this type ofprocess information provision would be similar to thosenarratives obtainable from process stakeholders who alsodescribe processes in the form of stories about how indivi-dual cases are handled. A further limitation is the potentiallylimited explanatory power of the statistical analysis due tothe non-normal distribution of the design categories, andtheir relative sample size, which in some cases are very low.For some design types we received only few data points,which renders some conclusions about these types difficultto make. Yet, our selected data analyses are valid for theobtained distribution of our data, which increases our con-fidence in the results obtained. Still, we caution the readerabout a potential lack of external validity of our findings toour settings and invite further replications of the study toexamine the results in larger data samples.
8. Conclusions
In this paper we reported on an empirical study carriedout to examine how novices conceptualize their under-standing of a business process using paper and pencil. Ourfindings reveal that the five types of design range frombeing dominantly textual to a hybrid of text and graphics(both abstract and concrete), and to being dominantlygraphical, and that flowchart-oriented types of designwith the use of textual labels and abstract graphics havepositive relationships to design quality as well as indivi-dual skills in formalized process modeling.
In conclusion, within the boundaries of the character-istics and limitations of our study, we believe that wehave provided an important initial body of knowledgeinto the practices of process design that adds to theliterature on organizational design, conceptual modelingand individual behaviors.
Appendix A
Data collection instrument
Part One: Pre-Test Questionnaire
1 0.36 40.96 40.96 0.52
2 0.25 28.05 69.01 0.45
1)H
3 0.17 18.98 87.99 0.38
ave you ever done any process modeling (e.g., with EPCs, BPMN,
Flowcharts)?
4 0.11 12.01 100.00 0.31
& yes
& no
2)H
ave you ever done any data modeling (e.g., with ERMs, ORMs)?
& yes
& no
3)H
ave you ever done any object modeling (e.g., with UML)?
& yes
& no
4)H
ave you ever been to an airport to fly somewhere?
& Yes, for international flights to/from Australia
& Yes, for domestic flights in Australia
& No, never.
5)P
lease indicate your level of familiarity with typical airport
procedures on a scale from 0 (‘‘no knowledge at all’’) to 10 (‘‘expert
knowledge):
Familiarity:
Part Two: Drawing Skills Assessment
Table B4Canonical discriminant function significance tes
Test of function(s) Wilks’ Lambda
1 through 4 0.45
2 through 4 0.62
3 through 4 0.77
4 0.90
Table C1Coding protocol used for process design representation quality assessment.
No. Type Elements
Semantic correctness01 States (i) Taxi arrives
(ii) Arrives airport
(iii) Receives boarding pass
(iv) Proceed to security gate
(v) Walks to/from
departure gate
(vi) Walks up/down shops
Activities (i) Call taxi
(ii) Check-in online
(iii) Buy newspaper
(iv) Board plane
Events (i) Arrange transport to airp
(ii) Use online check-in or tic
counter
(iii) Check-in luggage or go t
security check
(iv) Frequent flyer lounge
or shop
Business rules (i) Check-in for boarding pa
(ii) Check-in luggage if any
(iii) Security checked
(iv) Proceed to gates prior to
departure
Table B3Canonical discriminant function structure matrix.
Factor Function
1 2 3 4
DMK 0.29 0.09 �0.06 0.01
PMK 0.27 0.00 �0.15 �0.24
OMK 0.29 �0.46 0.22 0.27
DK 0.44 �0.22 0.45 �0.15
IMP �0.04 0.41 0.36 0.31
PROP 0.20 0.36 0.14 0.20
COR 0.15 0.15 0.15 0.01
SHAD �0.11 0.28 0.48 �0.01
COM �0.26 0.39 0.41 0.26
STY �0.01 0.22 0.34 0.35
PERS �0.31 0.03 �0.09 0.35
J. Recker et al. / Information Systems 37 (2012) 557–573 571
Part Three: Process Model Context Scenario
ts.
Chi-square df Sig.
52.19 44 0.19
31.69 30 0.38
16.99 18 0.52
6.70 8 0.57
Description Measurement
� Representing actions that are
in transition or in motion
� Involves concrete icon(s)
� Supplementary use of text
labels and/or description
� 1¼states not at all
represented in model
� 2¼Z1 state represented
� 3¼3 states represented
� 4¼Z4 states represented
� 5¼All 6 states fully
represented in model
� Carrying out something or
execution of a task
� Primarily involves actor
� Supplementary use of text
� 1¼activities not at all
represented in model
� 2¼1 activity represented
� 3¼2 activities represented
� 4¼3 activities represented
� 5¼all 4 activities fully
represented in model
ort
ket
o
� Decision-making point(s)
� Shows alternative option(s)
� Use of abstract shapes
� Supplementary use of texts
3 Primarily verbs
� 1¼events not at all
represented in model
� 2¼1 event represented
� 3¼2 events represented
� 4¼3 events represented
� 5¼all 4 events fully
represented in model
ss � Mandatory for something to
proceed
� Use of text descriptions to
assist in illustrating process
concept
� Supplementary use of
graphical icons
3 Commonly concrete
� 1¼rules not at all
represented in model
� 2¼1 rule represented
� 3¼2 rules represented
� 4¼3 rules represented
� 5¼all 4 rules fully
represented in model
Table C1 (continued )
No. Type Elements Description Measurement
Time (i) 10 min
(ii) 30 min
(iii) 5 min
(iv) 15 min
(v) 10 min
� Primarily indicated with
text labels
� Also represented with
concrete graphic icons
� 1¼0 time labels noted
� 2¼Z1 time label(s) noted
� 3¼3 time labels noted
� 4¼4 time labels noted
� 5¼all 5 time labels noted
Distance (i) 20 km
(ii) 100 m
� Primary use of text label
� Supplementary use of
graphic icon
� 0¼not at all noted
� 1¼only 1 label for
time noted
� 2¼all 2 labels for time noted
Graphic Icons—What it isUse of Concrete Icons (i) Stickman figure for actor(s),
(ii) Functional objects such as
telephone and online check-
in counter,
(iii) Landmarks such as home,
airport, etc.
� Freehand sketch by modeler
that are not used as part of
standard modeling method
� Measured in conjunction
with number of verbs used,
i.e., total¼11 verbs
� High-imagery, high-
frequency graphics
(Rogers[26])
� 1¼0 graphics used
� 2¼Z1 graphics used
� 3¼5 graphics used
� 4¼Z6 graphics used
� 5¼Z11 graphics used
Graphic Icons—What it is notUse of abstract symbols (i) Rectangles,
(ii) Diamond-shapes,
(iii) Circles,
(iv) Ovals,
(v) Arrows, etc.
� Icons that use a typical object
to present a general class of
objects
� Low-imagery, low-frequency
graphics (Rogers[26])
n/a
J. Recker et al. / Information Systems 37 (2012) 557–573572
Part Four: Exam Question on Process Model ContextScenario
Consider the following scenario:
Mark is going on a trip to Sydney. He decides to call ataxi from home to the airport. The taxi arrives after10 min, and takes half an hour for the 20 km to theairport. At the airport, Mark uses the online check-incounter and receives his boarding pass. Of course, hecould have also used the ticket counter. He does nothave to check-in any luggage, and so he proceedsstraight to the security check, which is 100 m downthe hall on the right. The queue here is short and after5 min he walks up to the level with the departuregates. Mark decides not to go to the Frequent Flyerlounge and instead walks up and down the shops for15 min and buys a newspaper before he returns to thegate. After 10 min waiting, he boards the plane.
Your task is to create a BPMN process model for the abovescenario, using only constructs from the BPMN core set.
Appendix B
Multiple discriminant analysis results
See Tables B1–B4.
Appendix C
See Table C1 .
References
[1] I. Davies, P. Green, M. Rosemann, M. Indulska, S. Gallo, How dopractitioners use conceptual modeling in practice? Data & Knowl-edge Engineering 58 (2006) 358–380.
[2] B. Curtis, M.I. Kellner, J. Over, Process modeling, Communications ofthe ACM 35 (1992) 75–90.
[3] N. Kock, J. Verville, A. Danesh-Pajou, D. DeLuca, Communicationflow orientation in business process modeling and its effect onredesign success: results from a field study, Decision SupportSystems 46 (2009) 562–575.
[4] J.V. Nickerson, J.E. Corter, B. Tversky, D. Zahner, Y.J. Rho, The spatialnature of thought, in: R.J. Boland, M. Limayem, B.T. Pentland (Eds.),Proceedings of the 29th International Conference on InformationSystems, Association for Information Systems, Paris, France, 2008.
[5] J. Recker, Opportunities and constraints: the current struggle withBPMN, Business Process Management Journal 16 (2010) 181–201.
[6] A. Luebbe, M. Weske, Tangible media in process modeling—acontrolled experiment, in: H. Mouratidis, C. Rolland (Eds.), Proceed-ings of the Advanced Information Systems Engineering—CAiSE2011, vol. 6741, Springer, London, England, 2011, pp. 283–298.
[7] M. Prats, S. Lim, I. Jowers, S.W. Garner, S. Chase, Transforming shapein design: observations from studies of sketching, Design Studies 30(2009) 503–520.
[8] W. Bandara, G.G. Gable, M. Rosemann, Factors and measures ofbusiness process modelling: model building through a multiplecase study, European Journal of Information Systems 14 (2005)347–360.
[9] J. Recker, M. Indulska, M. Rosemann, P. Green, The ontologicaldeficiencies of process modeling in practice, European Journal ofInformation Systems 19 (2010) 501–525.
[10] H.A. Simon, The Sciences of the Artificial, MIT Press, Cambridge,MA, 1996.
[11] J. Mendling, H.A. Reijers, J. Recker, Activity labeling in processmodeling: empirical insights and recommendations, InformationSystems 35 (2010) 467–482.
[12] J. Recker, M. Rosemann, M. Indulska, P. Green, Business processmodeling: a comparative analysis, Journal of the Association forInformation Systems 10 (2009) 333–363.
J. Recker et al. / Information Systems 37 (2012) 557–573 573
[13] J. Krogstie, G. Sindre, H.D. Jørgensen, Process models representingknowledge for action: a revised quality framework, EuropeanJournal of Information Systems 15 (2006) 91–102.
[14] J. Mendling, H. Reijers, W.M.P. van der Aalst, Seven processmodeling guidelines (7PMG), Information and Software Technology52 (2010) 127–136.
[15] M. Rosemann, Potential pitfalls of process modeling: Part A,Business Process Management Journal 12 (2006) 249–254.
[16] Y. Yang, Q. Tan, Y. Xiao, Verifying web services composition basedon hierarchical colored Petri nets, in: A. Hahn, S. Abels, L. Haak(Eds.), Proceedings of the First International Workshop on Inter-operability of Heterogeneous Information Systems, ACM, Bremen,Germany, 2005, pp. 47–54.
[17] L. Boroditsky, Metaphoric structuring: understanding time throughspatial metaphors, Cognition 75 (2000) 1–28.
[18] W. Visser, Design: one, but in different forms, Design Studies 30(2009) 187–223.
[19] W. Wang, R.J. Brooks, Empirical investigations of conceptualmodeling and the modeling process, in: S.G. Henderson, B. Biller,M.h. Hsieh (Eds.), Proceedings of the 39th Conference on WinterSimulation, IEEE, Washington, DC, 2007, pp. 762–770.
[20] D.L. Moody, The ‘‘Physics’’ of notations: toward a scientific basis forconstructing visual notations in software engineering, IEEE Trans-actions on Software Engineering 35 (2009) 756–779.
[21] J.H. Larkin, H.A. Simon, Why a diagram is (sometimes) worth tenthousand words, Cognitive Science 11 (1987) 65–100.
[22] H. Koning, C. Dormann, H. van Vliet, Practical guidelines for thereadability of it-architecture diagrams, in: K. Haramundanis,M. Priestley (Eds.), Proceedings of the 20th Annual InternationalConference on Computer Documentation, ACM, Toronto, Ontario,2002, pp. 90–99.
[23] A.T. Purcell, J.S. Gero, Drawings and the design process: a reviewof protocol studies in design and other disciplines and relatedresearch in cognitive psychology, Design Studies 19 (1998)389–430.
[24] M. Kavakli, J.S. Gero, Sketching as mental imagery processing, DesignStudies: The International Journal for Design Research in Engineering,Architecture, Products and Systems 22 (4) (2001) 347–364.
[25] J. Ferreira, J. Noble, R. Biddle, A case for iconic icons, in:W. Piekarski (Ed.), Proceedings of the Seventh Australasian UserInterface Conference, CRPIT, Hobart, Australia, 2006, pp. 64–100.
[26] Y. Rogers, Pictorial representations of abstract concepts relating tohuman–computer interaction, ACM SIGCHI Bulletin 18 (1986)43–44.
[27] R.E. Mayer, Multimedia Learning, Cambridge University Press,Cambridge, MA, 2001.
[28] V. Khatri, I. Vessey, V. Ramesh, P. Clay, P. Sung-Jin, Understandingconceptual schemas: exploring the role of application and is domain
knowledge, Information Systems Research 17 (2006) 81–99.[29] S. Isaac, W. Michael, Handbook in Research and Evaluation:
A Collection of Principles, Methods, and Strategies Useful in thePlanning, Design, and Evaluation of Studies in Education and theBehavioral Sciences, Edits, San Diego, CA, 1995.
[30] G.J. McLachlan, Discriminant Analysis and Statistical Pattern Recog-nition, Wiley-Interscience, New York, NY, 2004.
