UsingRelationAlgebratogeneratefeedback
inanIntelligentTutoringSystemforBusinessRules
Student: Esther Hageraats Identiteitsnummer: 834972368 Datum rapport: 5 december 2016 Datum presentatie: 14 december 2016 Datum einde inschrijving: 2 mei 2017
UsingRelationAlgebratogeneratefeedback
inanIntelligentTutoringSystemforBusinessRules
Opleiding: Open Universiteit, faculteit Management, Science & Technology Masteropleiding Business Process Management & IT Programme: Open University of the Netherlands, faculty of Management, Science & Technology Master Business Process Management & IT
Cursus: IM9806 Afstudeertraject Business Process Management and IT Student: Esther Hageraats Identiteitsnummer: 834972368 Datum: 29 November 2016 Afstudeerbegeleider Lloyd Rutledge Meelezer Stef Joosten Derde beoordelaar ... Versie nummer: V8 Status: Final
4
Acknowledgements
This master thesis was an adventure. I really enjoyed my first encounter with business rules and relation alge-bra in the course Business Rules, but it was a big step outside my comfort zone to make it the subject of my the-sis. I am my own worst critic and I needed support keep looking at the bright side of things. I want to say special thanks to the following people: Lloyd Rutledge for his encouragement and patience. Stef Joosten for his open-ness. Han Joosten for responding to all my requests for help and making me feel welcome in the Ampersand group. Thanks to Hieke Keuning for that long and enlightening conversation in Starbucks about what an Intelli-gent Tutoring System is and where to start reasoning about it.
I am an archetypal Open University student, starting at the age of 45 from scratch. Now, eight year later, I conclude this endeavor with a master thesis. This did not go by unnoticed at home. I want to thank Willem Ha-bers, my love, for taking such good care of me and listening to all the stories about the things I am learning.
1
Using Relation Algebra to generate feedback
in an Intelligent Tutoring System for Business Rules
Esther Hageraats
Faculty of Management, Open Universiteit Enschede, The Netherlands [email protected]
Abstract. Intelligent Tutoring Systems are recognized as valuable aids in education. We investigate how a rule-based design can automatically be verified and whether this can be done with a rule-based approach and relation algebra. The rule-based approach is compatible with constraint-based tutors: as long as the stu-dent’s product does not reach a state that is known to be wrong, the student is on the right track. It will help recognize errors and with that stimulate the inter-nalization of knowledge. Relation algebra has already been proven to be an apt language to define business rules and our research question is: How can rela-tion algebra be applied to create feedback for students about their business rules models? The research is designed as an exploratory case study in which a feedback system is simulated using an already existing system for business rules formulated in relation algebraic expressions. We propose a classification of modeling problems based on their verifiability and show how rules can be used to compare models or verify patterns within these different classes. The conclusion is that although the rules yield useful feedback, there are limitations to relation algebra that make it less suitable for a learning system than a pro-gramming language. The most important limitation being that rules and mes-sages cannot be applied conditionally.
Keywords: Business rules, intelligent tutoring system, relation algebra, con-straint-based feedback
2
1 Introduction
A business rule is “an element of guidance that introduces an obligation or a neces-sity” [1]. Structural rules define the information the organization deals with and how it is structured. Operative rules define obligations in the behavior of people in the organization. With the business rules approach information systems are being de-signed based on business rules. Professionals in the domain of business rules are con-vinced that the essence of business processes can be captured in business rules [2].
Ampersand is a dedicated development environment that uses relation algebra to
define business rules. An Ampersand model contains enough information to build a web-based information system [3]. At the core is a model specified in an Ampersand-script. Functional specifications in natural language can be generated from such a script so it can be understood and commented on by the stakeholders. Ampersand can also generate a working prototype of the information system including database, web-interface and rule-engine. With the prototype, a model can be further analyzed and correctness of a model can be checked with the stakeholder.
The Open University (OU) uses a web-based simulation of Ampersand, called RAP2, as an e-learning environment in the course “Rule-based Design”. In this course students learn to specify a model and business rules in Ampersand-script as a way to specify a business process [4]. The students work with a book and exercises about formal logic, relation algebra and business rules. All exercises are available in the course book [4] and executed on paper or a word processor. RAP2 is used in the se-cond half of the course when a student models a real process from the student’s work environment. This assignment takes the student through all the steps of the develop-ment process from describing the context and requirements to coding the model and the business rules in an Ampersand-script. To test the model and the rules, a popula-tion of elements must be added. The student can upload the Ampersand-script in the e-learning environment to analyze the model. The code is translated into an overview of all elements in the model presented as hyperlinks. The student can click through the model and extend the population. The diagnoses page shows unpopulated rela-tions, concepts without definition and rules and relation without descriptions [5].
Besides this diagnoses, RAP2 produces three types of error messages: syntax er-rors, type errors and rule test errors. The syntax error is a message that the Amper-sand-script won’t compile. The type error states that in a certain rule the conventions of relation algebra are not applied correctly and incompatible concepts are being combined. The type checker is a major support in learning relation algebra [5]. The third type of message results from applying the rules the student has created on the population the student has added. These are the messages the code is made for, the rule violations that will guide the business process at hand. The student can use these messages to check, with changing populations, if the rules produce the intended mes-
3
sages. Although RAP2 provides important feedback to students they still find it hard to right correct code. A measurement of almost 4000 files showed that in 25% of the cases the changes made in the code resulted in semantically the same code and con-tained no improvement [5]. Research on almost 7000 Ampersand code files showed that a quarter of the code entered in the system turns out to be syntactically incorrect and another quarter is semantically incorrect, so learning to code is a major issue [6]. This is also reflected in unpublished student evaluations [5]. Students have difficulty using the messages of RAP2 to analyze their script. Maybe with the use of artificial intelligence techniques this can be improved.
An Intelligent Tutoring System (ITS) is a learning system that offers automated
feedback and instruction to the students about the quality of the current product [7]. The feedback process in an ITS is basically the same as with a human tutor: The stu-dent will do an exercise that has been published in the system by the teacher. When the student has uploaded the work, the system will interpret the input and generate feedback to help the student correct the work and take the next steps in the exercise.
The Ideas project, a collaboration of the OU and the Utrecht University, has creat-ed a generic method to build an ITS for any domain as long as it is well structured. This means it is a domain with a formal and static body of knowledge that is applica-ble in its own context only [7]. Defining the domain in the so-called domain reasoners is one of the key concepts in their solution. The domain reasoner contains all elements required to produce automated feedback, e.g. the domain’s grammar, strategies for working within this domain and rewrite rules and refine rules that the ITS will use to interpret the input from students and generate feedback. The Ideas platform, a tech-nical implementation for domain reasoners has been used to create an ITS for Haskell [8] and Java [7]. In these ITSs unfinished work can also be evaluated and student receive feedback about what should be the next step.
The same time the Ideas project was working on the ITS for Haskell, Gerard Mi-
chels built RAP2 [5]. An important aspect of RAP2 is the recursive touch in the im-plementation: the functionality of RAP2 itself, has been specified with Ampersand-script. This script contains a meta-model for the models that students create including relation algebraic rules about them. Using this meta-model, the application RAP2 has been generated with Ampersand. So besides being the subject of an e-learning system, Ampersand is also the platform for an e-learning system.
Because even with the feedback provided by RAP2, students find it difficult to cre-ate sound models, the OU aspires to develop further on this e-learning environment and extend it with ITS functionality. A logical next step is to investigate if rules can be used to create feedback, not only about syntax, but about the models themselves. One of the principles that can be applied in ITSs is constraint-based modeling, which is described as a method “that models constraints as conditions that must be met in the end result” [7]. This fits perfectly well with the rule-based approach that also puts constraints on the result, rather than on the process towards the result. A business rules language seems therefore a good language for a constraint-based ITS. The ques-
4
tion for further research will be: How can relation algebra be applied to create feed-back for students about their business rules models?
We will investigate this problem in a case study. We will analyze the educational
problem and define classes with respect to verifiability of the modeling problems that students can be presented with. A meta-model will be created based on the design of RAP2. This meta-model will be used in prototypes to compare the student’s model with a reference model in a number of exercises. We will demonstrate the results and with those results evaluate the effectiveness of relation algebra in this design.
2 Related Work
2.1 Business Rules
Business rules are statements that specify something that must be true now or in the future. The Business Rules Group defines the term business rule from two per-spectives. From the business perspective a business rule is guidance that there is an obligation to act. From the information system perspective a business rule is a state-ment “that defines or constraints some aspect of the business” [9]. Having two per-spectives on the same concept reflects the role of business rules in the infamous Busi-ness-IT-alignment: business rules exist both in business and in information systems. Not all rules in the business find their way into the information system, but the other way around the constraint certainly stands: each business rule in the information sys-tem must also exist in the business. This vital link is also expressed in the standard Semantics of Business Vocabulary and Business Rules (SBVR): a business rule is “a rule that is under business jurisdiction” [1]. Business rules in an information system are used to influence the real world by identifying violations to the rule.
In rule-based design, there is no model of a sequence of actions to change data,
which is the way processes are usually defined. Professionals in the domain of busi-ness rules are convinced that the essence of business processes can be captured in business rules alone [5]. Different types of business rules exist when applied in in-formation systems: mandatory constraints reject violations; guidelines produce a warning when violated; action-enablers initiate other events when certain criteria are met; computations create new information based on mathematical computation; infer-ences create new information based on if-then constructs; event condition action (ECA) rules evaluate certain conditions after a certain event [2]. This classification shows that not all business rules block data-entry, some just trigger warnings. One of the advantages that result from this is that the business rule approach provides work-ers freedom of choice in when and how to accomplish the goals set by the operative rules. There are more advantages of the business rules approach [2]. It makes explicit the rules the organization wants to enforce and helps them use them consistently. Further the business rules approach improves the understandability of what is imple-mented and the traceability between the rules and the implementations. But maybe the
5
most important advantage of the business rules approach is that it supports the busi-ness need for agility: business is changing all the time and the IT should change easily with it. Business rules are specified, managed and applied in a rule engine, separately from the information system, which makes it possible to change the rules without releasing new versions of the information system software [2].
The transformation of rules in the business into rules in an information system re-
quires a transformation of language. Von Halle (as cited by Boyer and Mili [2]) iden-tifies four types of languages in this transformation. The business conversation lan-guage is the starting point. Rules are phrased in the “native” language of the people in the organization. This is translated into the structured natural language. This language is understandable as natural language, but “using a predefined linguistic template that corresponds to the various rule categories” [2]. A widely-used example of this lan-guage is RuleSpeak®, which has been one of the sources for the SBVR and is fully consistent with it [10]. In a third translation step the business rules are being specified in a formal authoring language. This language must still be easy to use for people in the business, but on top of that, this language must be formal and unambiguous, be-cause this language is already executable or can be mechanically translated into the engine execution language [2].
Most Business Rules Management Systems currently on the market use a formal authoring language close to the programming language of the rule engine or a more business-oriented language. Boyer and Mili [2] present rule coding patterns for each type of business rule, using Boolean algebra to transform the rules in a disjunction of conjunctions that can be translated into if-then constructions. The language that is the main focus of this article, Ampersand-script, is also a formal authoring language be-cause it can be mechanically translated into the engine execution language.
2.2 Relation Algebra as language to specify Business Rules
Relation algebra was made into a mature theory with established fundamental con-cepts and laws by C.S. Peirce with extensions from E. Schröder in the second half of the 19th century. Tarski extends the theory with several operators and constants to create means for calculating with binary relations [11]. Important additions to relation algebra are relation constants, like the universal relation that consists of all possible pairs (x, y) between two sets. Another constant is the identity relation that consists of all pairs (x,x) of a given set. Unary operators apply to a single relation, like the con-verse-operation R~ that results in pairs (y,x). Another unary operation is the comple-ment, -R, which consists of all pairs that are not in R. The power of the extensions is that it allows for complex reasoning about relations [11]. We can, for instance, calcu-late with binary operators like the operator “;” for composition. R;S results in a new relation, that is a set of pairs (x,z) where the target-element y in xRy is the source-element y in ySz.
Apparently unaware of the link with relation algebra Dijkman, Ferreira Pires and Joosten were looking for ways to make data models more precise and the process of
6
modeling more effective [12]. The method they developed in 2001 was called Calcu-lating with Concept (CC) and comprised of defining constraints on relations between UML classes with concepts originating from set theory. The term “business rule” is nowhere to be found in the description of the CC technique, which is not surprising, since the whole business rules approach had just started to evolve in the United States and probably had not landed on the coasts of Europe yet. But nonetheless, constraints are defined in a much more formal way than UML, which makes it possible to verify consistency of the model with cycle chasing [12]. Only a few years later, the creators of the CC method chose relation algebra as language for specifying those constraints [3]. The CC method was already close to relation algebra and the rigor of relation algebra assured the validity of analysis done on a model to a point where automated generation of an information system seemed realistic.
Not long after that, parallels with rule-based design and developments in the area of Semantic Web were becoming apparent [3]. Further development resulted in the Ampersand tool, which is now presented as a rule-based development method. The tool operates between the developers and the stakeholders of an information system. On one hand, specifications in natural language can be generated. But in addition, a working prototype of an information system can be generated from the model. It is this prototyping functionality of Ampersand that makes Ampersand-script a formal authoring language. The number of element types in Ampersand-script is very lim-ited, which makes the language relatively easy to learn, but recognizing clever pat-terns and knowing how to use them is a skill that requires a lot of practice.
2.3 The design of RAP2
As is explained in the introduction, RAP2 is generated with Ampersand based on a meta-model of the e-learning system. Because we will be using this meta-model as basis for our research it will be explained to some degree in this section. The com-plete specification is available in [5]; this section is limited to some examples that give an impression of the meta-model and how it is applied.
A student’s script is parsed in RAP2 and the code is divided in chunks that are placed in the database. There is of course a concept named Concept. The relation Concept~AtomID means that a concept may be populated with atoms. There are rela-tions, modelled in several concepts, among which the concept Declaration.
Anything that is not recognized by the parser results in a parse error, which is also stored in a concept, as is the file position of the error. We then have an atom of the concept FileRef that has a relation parseerror with an atom of the concept ParseError and that atom has a relation pe_position with an atom of the concept FilePos.
Rule violations are also concepts in this meta-model. A Violation is special kind of relation; it is populated with pairs of atoms. The population of a Violation consists of those pairs that are the complement of a rule expression; that means all pairs in a rela-tion that do not comply to the rule. The concept Violation is used to create messages when the rules the student has defined in the student’s model, are violated by the pop-ulation the student has created for these concepts and relations [5].
7
2.4 Modeling, learning and scaffolding
Model construction can be found in most educational frameworks and curricula from elementary school to professional education [13]. A model is an expression in a formal language that denotes properties of a given system. Examples are mathemati-cal equations, computer-programming languages and formal diagram techniques like UML. The learning curve for modeling is steep as it contains learning the modeling language, syntax and tools before learning about the domain that is represented with the model [13]. The course Rule-based Design, which uses RAP2, is definitely a course about modeling. Coding in RAP2 is not a learning objective itself, but one has to know how to code to be able to create a model with RAP2. The steep learning curve that is often associated with modeling is also visible in this course [5].
The ACT-R theory states that there are two long-term memory stores in our brains: declarative knowledge is represented as more or less loose chunks where in the se-cond store these chunks are combined into production rules. The latter is called pro-cedural knowledge, which is goal-oriented and further optimized when a student be-comes an expert. “The fundamental assumption of ACT-R is that cognitive skills are realized by production rules” [14]. To help students with their learning, a wide range of methods can be applied. Some of these methods are called “scaffolding” tech-niques, drawing on the analogy with scaffolds around a house. These temporary aids and additions will be removed when the goal is reached, in our case when the student is skilled enough. Scaffolding techniques can be used by a teacher, but also in tutoring systems [13]. Feedback is one of many methods for scaffolding. The most obvious and widely used scaffolding is providing hints about the student’s product.
Learning means that the student closes the gap between the current knowledge and
the knowledge the student is aiming for. There are, however, strategies to close this gap that impede instead of advance the learning process: making the goal more vague or less ambitious or disengaging from the learning goal all together [15]. When trying to avoid these unwanted strategies, feedback about the task (FT) works best on simple tasks, as too much FT will put students in a trail-and-error mode without focus on the underlying principles. Students then just keep asking hints until the system seems to be satisfied without understanding the reasons behind the hints [15]. This mechanism of help abuse is also observed with modeling. Students make the mistake to be satis-fied when the model produces the right predictions with their input without internaliz-ing the knowledge [13].
To be effective the feedback must keep the student engaged and promote learning as the way to close the knowledge gap [15]. This requires that students get the oppor-tunity to develop self-monitoring skills and take control over the learning process. Effective feedback answers three questions: Where am I going? How am I going? And where to next? Giving hints on the student’s process rather than the product is also used to counter the problem of help abuse. This scaffolding with process feed-back is called meta-tutoring [13] or feedback about processing of the task (FP) [15].
8
2.5 Intelligent Tutoring Systems
Following VanLehns’s example, we will describe an ITS by describing its func-tionality. Although there are major differences in the internal structure of ITSs, their behavior is very similar and describing functionality provides us with generic words to describe this behavior [16].
Most ITSs provide the students with tasks. A task requires several steps. A step is a physical action taken by the student in the user interface of the ITS. The cognitive action that accompanies this step is a learning event in which a student constructs and applies a knowledge component. A step can invoke several learning events. A knowledge component is applied knowledge. Knowledge can be specified on different levels of detail. The granularity chosen to define the knowledge components is deter-mined by the teacher creating the tasks based on the learning goals of the ITS. Im-portant concepts are the outer loop and the inner loop [16].
The outer loop is about the selection of tasks. This decision can be delegated to the student by providing a menu of some sort. This decision can also be guided by the ITS. The most basic guidance is that a certain order is enforced in which tasks have to be executed. More complicated to implement is requiring the student to prove a cer-tain skill level before being allowed to move on to a next level. The most sophisticat-ed way of guiding the task selection is when the student’s accomplishments are meas-ured per knowledge component and mastery level and when that information is used to determine the next task.
The inner loop of an ITS is about guiding the students while performing a task. There are tutoring systems without an inner loop, but these are not called intelligent tutoring systems [16]. These systems can still be effective for learning, and building an inner loop is something to be considered carefully as it is quite expensive. Guid-ance to students is provided in the inner loop with feedback. The most basic feedback found in tutoring systems states that a certain solution is correct or not. Some systems also have the option to state that something is correct, but not optimal or that the solu-tion is not recognized. The feedback can be provided on request, after performing an entire task or immediately during typing. The second level of feedback is providing hints. Determining the right moment for providing hints has proven to be difficult. Here also an important issue is the amount of control that is given to students: are all hints available to be selected or does the system provide hints in a certain order and only after new attempts have been submitted? When an ITS is able to determine the cause of the error (which incorrect learning event has taken place), then the feedback can be more sophisticated than just giving a hint for what is the standard next step [16].
2.6 Model-tracing tutors and constraint-based tutors
Now that we have described the functionality of ITSs, we should say some more about the design. The tutor provides feedback, but how is this feedback created? We will have a look at two types of design: model-tracing tutors (also named cognitive tutors [17]) and constraint-based tutors. A model-tracing tutoring system checks
9
whether the student takes the right steps to come to a solution by comparing each student’s step with the possible correct and incorrect production steps specified for this particular problem [16, 18]. The tutor gives feedback about the student’s choice of actions. This way the student learns the production rules [14]. However, even knowing the production rules, students don’t always apply them [13].
Constraint-based tutoring systems ignore the steps the student has taken and only check the result of those actions, the student’s product. The rationale behind con-straint-based tutors is the theory that students make errors in their products because their declarative knowledge is not yet internalized [14]. The tutoring system should help recognize errors and with that stimulate the internalization of knowledge. The design of constraint-based tutors is based on the assumption that as long as the stu-dent’s product does not reach a state that is known to be wrong, the student is on the right track. The system identifies errors in the product, “which is extremely important for students lacking declarative knowledge, because they are unable to detect errors themselves” [14]. To be able to make the analysis, the tutor KERMIT has a domain model consisting of a set of state descriptions (constraints) with relevance conditions (RC) and satisfaction conditions (SC). The basic form of these constraints is if <RC> then <SC> [14]. When a student’s product is entered in the system, all RCs are first matched against the product. Then for each RC found, the corresponding SC’s are tested. If a constraint is violated, then the student is informed that something is wrong. The student has to do some form of reasoning to come to a better product. This reasoning will result in the student’s own – possibly unique - procedural knowledge about the correct steps to solve a given problem.
The strength of constraint-based tutors is analogue to rule-based design that pro-vides more freedom to users of a system: constraint-based tutors allow for students to follow their own unique – possibly inconsistent – process while internalizing declara-tive knowledge. Constraint-based tutors are, however, not well equipped for providing meta-tutoring or feedback on where to go next. A type of FP (feedback about the process) that is possible in constraint-based tutors, is about strategies for error-detection. This feedback helps students to provide themselves with feedback. This is an important step in learning self-monitoring skills [14]. A method that supports this is Point-Teach-Bottom-out (Hume, Michael, Rovick & Evans, as cited by [16]): The first hint points to the place in the students task where the problem occurs. The next hint provides theory that should be applied to solve the problem and the final hint provides the solution.
2.7 Ideas: a platform for strategy-based tutors
The Ideas framework, created by a collaborative research group of the OU and the Utrecht University, is suitable for building ITSs for well-structured domains. It has been used for several mathematical areas [19] and for the programming languages Haskell [8], Java and PHP [7]. The aim of the Ideas platform is to make the ITS cal-culates feedback based on the possible strategies within the domain instead of ‘just’ displaying feedback the author of the model has prescribed for each possible scenario.
10
Furthermore, the feedback must be about intermediate products so that the student’s strategy is also under review.
The main ingredient of an ITS in Ideas is the domain reasoner that defines the do-main. First the domain grammar must be defined; this is used for syntax checks. The intelligence comes from the strategies defined with so called “combinators”, operators to specify if steps are interchangeable, can be taken in any given order, in a prescribed order or if they cannot be combined at all, each being the start of a different path. The possible strategies for an exercise are calculated by the platform and a teacher does not have to specify this separately. The calculated strategy for an exercise becomes a tree of possible routes that a student can take. The nodes in this hint-tree have labels referring to a text file with the feedback.
As the student has quite some liberty in how to implement a solution, it’s a chal-lenge to recognize errors and acknowledge variants in implementation and strategy that are allowed. First all input from the student is rewritten into a canonical form. A few of these normalizations are: removing syntactic sugar from the student program, assigning new unique names to all variables, replacing function calls with the applica-tion of the function [7, 20].
2.8 The verifiability of educational problems
Whether or not an ITS is suitable for a particular learning goal depends on the type of problem. A well-defined problem has a solution that can be verified as either cor-rect or incorrect. Tutoring systems for these problems have been created successfully and the challenge is to handle less well-defined problems. There is no clear line be-tween a well-defined and ill-defined problem, but a great help for getting more insight in what we are working on, are the levels of “ill-definedness” based on the two di-mensions depicted in Fig. 1 [18]. For each class of educational problems other ap-proaches to computer-aided learning seem viable. The first four classes have verifia-ble solutions, where the fifth class has not.
Class 1 problems have one solution strategy and one implementation. An ITS is not
needed, as the system only has to know the correct answer. This can be handled with more simple systems like Computer Aided Instructions.
Class 2 problems have only one solution strategy, but different implementations are possible. The challenge is to recognize implementations that are different but equally appropriate. Most ITSs handle this class of problems, and the two mostly used approaches are model-tracing tutors and constraint-based tutors. To make educational problems better suited for an ITS, they are often narrowed down to a class 2 problem by allowing the student only one particular solution strategy. One limitation that stu-dents are faced with in class 2 problems is the choice of names for model elements [18]. This will solve the big problem of semantics and ontology [21]. Tutoring sys-tems have different ways to implement these restrictions. The tutor for data modeling KERMIT allows students to create their own names for tables, attributes and rela-tions, but they are required in the user interface to indicate which fragments in the problem text these elements represent [22]. Another method used in several tutors, is
11
making students choose from options presented for instance in a dropdown menu [13]. Despite of their limitations, class 2 tutors have proven to be a very useful tool in education [18].
Fig. 1 Classification of educational problems [18] Class 3 problems allow the student different strategies, but the choice is limited.
The implementation strategy is up to the student. The limited choice of strategies assures that the alternative solutions can be anticipated. This class poses more chal-lenges for an ITS than class 2 problems. The feedback has to match the strategy used, which means that the ITS has to recognize the strategy and its implementation. Data-driven tutors can be used for this class of problems. They compare the student’s prod-uct with numerous products from other students that already have been analyzed and use the one most similar as reference model [18]. The Ideas project uses a domain reasoner and strategy specifications to handle class 3 problems [7, 18, 23]. These tutors for coding transform the student’s code and the model solution into a normal-ized form so they can be compared. One of the transformations is to rename all varia-bles into simple identifiers and keep record of the link between new and original names [7, 8].
Class 4 problems are more complex than class 3 problems and the choice of strate-gies is not limited. The correctness of solutions to class 4 problems can be determined automatically, but these problems have such a great variety of solution strategies that they cannot be known in advance. ITSs for these problems are scarce.
Class 5 problems cannot be verified automatically. Research with regard to ITSs for this class of problems focuses on expert models and machine learning techniques [18].
12
2.9 Conclusion: How to change RAP2 into an Intelligent Tutoring System
To conclude the literature review we will apply the information above and describe what functionality needs to be added to (the successor of) RAP2 to make it an ITS.
First, the system will need an outer loop. Currently, students use RAP2 for only one case of their own choosing. There are no exercises in the tool [4]. The modeling problem the students are faced with in the current course is a class 5 problem, just as information system design is. There is no automated way to verify the correctness of the model on that level. This means that the educational problems must be downgrad-ed to class 2 or class 3 problems. In an ITS, exercises can be selected with different levels of well-definedness.
Second, the inner-loop needs to be extended with more levels of feedback and a request of feedback. Currently, RAP2 offers first level feedback: correct/incorrect. These messages will appear unrequested after code has been uploaded. The diagnosis page is very sophisticated but the information presented is not necessarily about errors in the model. It is up to the student to find out what these messages imply about the correctness of the model. Student don’t get support to develop effective error-detection strategies. There is no mechanism to display on request a sequence of mes-sages about the same error. Such a mechanism is needed if the Point-Teach-Bottom-out method is to be implemented.
The ITS we are aiming for could function as follows: A teacher creates exercises.
An exercise consists of an assignment in natural language, a reference model, and rules that need to be applied with this particular exercise. This whole package is up-loaded into the system, to be used when a student model calls for it. The student is working in a web-based environment. The student selects an exercise and reads the assignment. The student creates an Ampersand-script as a solution. This is the student model. The student uploads this script. This script also specifies the exercise it applies to, with a keyword EXERCISE. The compiler will use this information to obtain the reference model and rules to apply. In compile time the relation between the student model and the reference model is established. Also, the rules that need to be applied in this exercise are enabled to be executed. The feedback from the rules is then pre-sented to the student using the Point-Teach-Bottom-out method. First only the posi-tion in the script that has an error is presented, next theory is presented to help solve the problem and finally, if a student cannot fix it, instructions on how to solve the problem are given.
3 Method
The empirical part of the research will be a case study with an embedded single case design [24]. The single case is the learning objective of the course Rule Based Design [4]: to define a model and business rules with Ampersand script. The embed-ded units of analysis are simulations of exercises including student models and the rules that provide feedback if a student has made a mistake. All this will be imple-
13
mented as Ampersand prototypes. The prototypes will not be integrated with (the successor of) RAP2 during this case study.
In this section, we will scope the educational problems for our ITS to class 2 and
class 3 problems, discuss some technical issues and limitations, and describe how these prototypes will be verified. We conclude this section with issues regarding the validity and reliability of this case study.
3.1 The verifiability of Ampersand models
Ampersand script is a simple specification language, not a programming language and the language itself has few variants1. A model, however, has many variants and that is our focus. The actual model consists of concepts, relations and rules. Amper-sand-script has many additional keywords for structuring the code, formatting text in the functional specification, formatting the web-interface of the prototype and data-types for the database of the prototype [25]. We will focus on the core modeling as-pects of Ampersand and limit the current research to exercises in which students have to specify concepts, relations and rules. This limitation is mainly a practical one, to make the research effort more realistic.
As we are working in a constraint-based system. The main challenge will be to
recognize correctness in the student’s model; to avoid false negatives and false posi-tives. This is illustrated with the four models in Fig. 2 that have different concepts, relations and rules, but are all correct implementations of the assignment to “create a model that states that buyer and seller of a thing cannot be the same person”.
Fig. 2 Different models with same result
The first three models are much alike: two concepts, two relations, one rule. More
concepts, relations and rules could have been used. That is not necessary and there-
1 Ampersand-script supports for instance two different ways to declare a relation, but after
compilation they cannot be distinguished.
14
fore not preferred, but it is not wrong as long as the rule yields the same result. Stu-dent 2 uses different names for the concepts, a human tutor would recognize them as the same. Student 3 decided to stick with what’s asked for and modeled just one con-cept Person with an endo-relation. Combined with the right rule this can also be a correct model, as the assignment given did not explicitly ask for a concept for the things that are being sold.
The challenge for automatically verifying a student’s product depends on the com-
plexity of the exercise and the amount of known alternatives [18]. The simplest exer-cises, class 1 problems, do not need intelligence to verify them, so class 2 problems are the least we should be aiming for in our research. But what is a class 2 problem with regard to modeling in Ampersand? We need to define this. The most obvious driver for complexity seems to be the size of the model as more concepts create more options to define relations and rules. But the example above shows that even with the smallest model the number of alternatives is extreme when the student can freely choose the names of concepts and relations. This freedom in naming model elements will be an important factor in defining problem classes. We will propose a classifica-tion for Ampersand modeling and discuss its compliance to the original classification model [18].
We define a class 1 modeling exercise as an exercise that dictates each detail of the model. All concepts are specified with their name, all relations are specified with name, source concept, target concept and multiplicity. Rules are described literally in natural language. The reference model in Fig. 2 would have the following assignment: “Create a rule stating that the composition of sells and the converse of buys must be a subset of the complement of the identity relation.” This is not a useful exercise as it is some sort of language puzzle about the words we use for mathematical operators. It is a class 1 problem because it has only one correct implementation. The size of the model does not affect that.
Class 2 education problems have a limited number of variants that are considered a
correct solution. The way to achieve this is to limit the freedom of students when they try to solve the problem. The lower boundary of this class is easy: every problem that has more than one correct solution is of a higher class than class 1. The line between class 2 and class 3 is determined by two characteristics. First, class 2 problems allow only implementation variants where class 3 problems allow strategy variants. The second characteristic is whether or not student can choose their own names for model elements. This is of course related to the first issue, allowing free naming of variables will result in an unlimited number of variants.
The literature does not provide us with definitions of the terms ‘implementation’ and ‘strategy’, but the examples presented for programming education give some indication: The choice between an if-then-else algorithm and a for-do-loop is about strategy; the choice between i++ and i=i+1 is about implementation [7, 18]. We think that the choice of concepts is the core of an Ampersand model and should be considered a strategy choice. Choices with regard to the number of concepts and their
15
meaning have consequences for all other elements, like the meaning of the relations, the number of rules and the relations used in these rules. Choice of relations and rules, we consider implementation variants.
Based on the two issues of implementation and fixed variable names, we propose that a class 2 problem for Ampersand modeling is a problem where the concepts and their names are fixed. Within this class we see two major steps in which the number of correct variables increases and we propose two sub-classes as shown in Table 1: Class2/Rule-Only problems have prescribed concepts and relations, class2/Rule-Rel problems prescribe only the concepts and allow students to define relations.
Table 1. Problem classes for Ampersand Model ele-ment
Details of the model. (X = fiXed in exercise Gray = for student to define)
Cl. 2
Cl. 3
Cl. 4
Cl. 5
Rule-only
Rule-Rel
Con-cept
Name to use for each concepts X X The number of concepts X X X
Rela-tion
Name to use for each relation X Multiplicity of each relation X
Direction of each relation X Involved concepts of each relation X Description in natural language X The number of relations X X X Rule Operators to use Antecedence and consequence The concepts being evaluated The concept involved Name to use for each rule The number of rules X X X Rule description natural language X X X X Case Case description natural language X X X X
A class2/Rule-Only assignment for the models in Fig. 2 would be to “create a rule
that states that buyer and seller are not the same person, using the relations buys and sells”. All three models will be evaluated as incorrect because they don’t match the reference model. To avoid having to check the names of concepts and relations, it makes sense to just give the students correct code with the concepts and relations already defined. Then all focus of the students can be on the rules. This type of exer-cise is useful for students learning the relation algebra language for business rules, probably as a first step towards creating a complete model.
16
When we look at verifiability, there are a limited amount of typical correct variants in this class. A rule is composed with relations alternated with operators. The relations are known and the set of operators the student can choose from is limited by the Am-persand language [26]. Assuming the model of the student is exactly as the reference model in Fig. 2 the following rules are typical correct answers to the assignment:
• B;S~ |- -I • S;B~ |- -I • I |- -(B;S~) • I |- -(S;B~)
There are, however, also less typical correct answers: • B;S~ /\ I |- -V
And the student can extend a correct rule without adding any meaning: • (B;S~) |- (-I) • B;S~ /\ V /\ V |- -I
If one should add up the typical and not-so-typical solutions, the conclusion is that the real amount of possible answers that yield the right result is infinite. For models with more concepts and relations there are more correct answers, the line of reasoning remains the same.
Class2/Rule-Rel problems have more variants than the previous class. The student
is given the exact names of the concepts, but not the names of the relations, nor their sign. It is only stated how many relations there must be. The student is asked to speci-fy relations and use these in a rule that satisfies the rule description. If the exercise in Fig. 2 were a class2/Rule-Rel exercise, the assignment could have been “to create a relations and a rule that states that buyer and seller are not the same person”. The model made by student 1 would be correct and should be recognized as such. The system can recognize that goes_to is the reversed version of buys. The model made by student 2 has different concept names, which makes it impossible for the system to recognize the similarities with the reference model. In this tutoring system the models of students 2 and 3 will be evaluated as incorrect.
The number of correct and incorrect models that this class adds to the previous problem class is limited by definition, regardless of the case presented in the exercise. One relation between two concepts can have two directions. With two concepts and two relations there are eight variants. When endo-relations are also allowed there are in total twelve variants, a few of which are presented in Fig. 3. The number of vari-ants increases substantially with the increase of concepts and relations, but there is always a limit. For each of these models separately, the reasoning for Class2/Rule-Only applies to the possible variants of rule implementations.
Class 3 problems allow a known number of typical strategy variants [18]. We had
defined a strategy as the choice for concepts and related to that, complete freedom in naming the elements. Still the number of variants must be limited, to separate this class from the next. To limit the number of variants, we propose for class 3 those exercises in which the number of concepts and relations is known, but not their name.
17
To illustrate this, we have a look at the model in Fig. 2 again. If this were a class 3 exercise, the assignment would have been “to create a model with two concepts and two relations and one rule stating that the person buying an item must be a different person than the one selling this same item”. The models of student 1 and 2 would be correct, the model of student 3 incorrect.
Fig. 3 Models for 2 concepts and 2 relations in problem Class2/Rule-Rel
Class 3 problems have less correct and incorrect variants than the previous class,
because the names of concepts are not being evaluated and the checks will be on a more conceptual level. In Fig. 3 model 1 would not be different from model 3 with two concepts and two relations. Model 2 is different as the signs of the two relations are different. Note that although the number of variants is lower than in class 2, veri-fication is not necessarily easier.
Class 4 problems allow a great variability of verifiable variants [18]. This is the
least clear category presented. In this problem class even the core of the model is for the student to define: the number of concepts and relations. The case that needs to be translated into a model is given, but not more than that. All student models in Fig. 2 are correct and should be recognized as such. The assignment could be “to create a model stating that the person buying an item must be a different person than the one selling this same item”. Compared to class 3 problems, the number of variants in-creases to a level where they cannot be anticipated. That characteristic aligns with the classification model [18]. The classification model also states, however, that the solu-tions to class 4 problems can still be verified in an automated way. It is not evident that this is possible with these class 4 problems for Ampersand.
Class 5 problems are very similar to class 4 problems except that there are no case
or rule descriptions provided by the teacher. The assignment would be “to create a model of some real-life situation.”
18
We conclude this section about problem classes with the specification of the exer-cises that are implemented in Ampersand prototypes for this research. They are cho-sen to cover a broad range of class 2 and class 3 problems as shown in Table 2.
Table 2 Coverage of problem classes in the case study
Exercise Description Class 2/ Rule-only
Class 2/ Rule-rel
Class 3
1 Identical names X 2 Identical Source X 3 Related Concepts X 4 Antecedent X X X 5 Anonymous Model X
3.2 Technical implementation
As shown in Table 2 we will create five simulations of exercises in Ampersand prototypes. We use Ampersand modeling to create our simulations. A complete simu-lation consists of:
- a description of the exercise, the rules and the problem class, all in natural lan-guage;
- an adl-file2 with the meta-model and the rules for this exercise in Ampersand-script;
- test cases2 (student products with specific errors) each one in an xlsx-file with a population;
- the verification results for each test case. The basis for all simulations is the meta-model. Fig. 4 shows a part of the Amper-
sand-script that defines the meta-model. We will not need to design such a meta-model from the ground up. Michels has already made a model of models in Amper-sand-script that is compiled into RAP2 [5]. Our goal is to extend this model.
In section 2.9 we described the functionality of the ITS that RAP2 could become. The case study will be done with that description in mind. However, we focus on the application of relation algebraic rules and only Ampersand-script and Ampersand will be used to implement the prototypes. This approach poses constraints on the research, each of which will be discussed below.
RAP2 was generated with Ampersand, but after that, Ampersand has been im-proved and extended. The current model and syntax of Ampersand might not be ex-actly the same as the model used in RAP2. This means that we will not be able to actually bring the design in a production environment. It will remain a simulation to
2 Adl-files and xlsx-files are available on
http://is.cs.ou.nl/OWF/index.php5/Masters_Thesis_Esther_Hageraats
19
demonstrate the results. The design will stay as close as possible to the original design of Michels. It will be precisely recorded what elements have been changed, added or reused as-is (Table 5).
Fig. 4 Screenshot of code fragment for the meta-model We assume the complete system has exactly one student and one exercise. This is
obviously not the case in the real system. The limitation brings the complexity of the problem down to a more manageable level which makes it possible to focus on feed-back to students. This means our prototypes do not take into account the practical implementations of the file repository for all student files in RAP2.
Ampersand itself has been developed with the Haskell programming language [27]. The feedback system of RAP2 that shows messages to students is also imple-mented in Haskell. The RAP2 compiler applies the rules to the data in the meta-model and the result is sent to the feedback system that generates a message [5]. This feed-back system will not be used in our design. We will use the keyword VIOLATION that is available in the current version of Ampersand to generate feedback.
We know in advance that the following functionality that should be in an ITS can-not be created with relation algebraic rules:
- A repository of exercises including reference models that teachers can fill. - The selection of exercises, the outer loop of an ITS. - Presenting the exercises to students.
20
- Recognizing that a script is the implementation of an exercise and needs to be handled as such.
- Point-Teach-Bottom-out method to provide, on request, feedback about an er-ror in different levels of detail.
3.3 Verification
Our exercises and the rules used to evaluate the student’s product are a software system, as are all ITSs. To proof a software system works, it needs be verified and validated. We will first discuss methods used in other research on ITSs and then de-scribe our verification method.
Verification provides evidence whether or not the system conforms to the require-ments. Validation provides evidence whether or not the product satisfies it’s intended use and user needs [28]. In ITS research, verification tries to answer the question how accurate the system recognizes erroneous and correct answers in the student’s prod-uct. Validation is about to the contribution the ITS has an on the learning activities of students. Both types of evaluation are applied in research on ITSs, often in combina-tion.
Validating an ITS can best be done in a real-life situation where students use the system during a course. The knowledge levels of students before and after the course can show whether the tool has effectively supported the learning activities. To elimi-nate all other influences, one should have a control group [29]. A questionnaire can be used, during a trial, to measure the subjective experiences of the students [29, 30]. A questionnaire can cover all kinds of aspects from the quality and timing of feedback to issues concerning performance and user-friendliness of the ITS as a whole. The importance of the subjectivity is that what the designers consider good feedback could be different from what students perceive as good feedback.
Verification of ITSs is at least done with test cases constructed by the researcher. These test cases can be used as demonstration, but often the number of variants is so that a complete set of test cases is not realistic [7]. The best source for test cases are the students themselves. After a trial, the student products can be given to human teachers and their feedback can be compared to the feedback the ITS has produced [30]. A method that is often used in trials is analyzing the log files of the ITS. Such an analysis results in a quantitative measure of the accuracy of the ITS. We find that no ITS is 100% correct in recognizing right and wrong in student products [7, 29, 30]. A good example is the extensively described analysis of 3466 log files of student prod-ucts being uploaded in Ask-Elle, the Haskell tutor from the Ideas project. The analysis showed what type of products are not recognized as correct or incorrect by the tutor. The information is used to improve the tutor, lowering the number of not-recognized products [30].
In the current research, validation of the prototypes is not possible because they will not have enough functionality to use in a real-life course. In the prototypes, stu-dent products will be represented in population files, but not in Ampersand-scripts that can be uploaded. This also means that there will be no log files to analyze. So we
21
are left with the option to verify whether the rules in our model, trigger violation mes-sages as we expect them to.
Fig. 5 Screenshot of test case 1b showing the violation message and underlying dataset
22
Literature about Ampersand research provides us with little information about methods to verify a model. There are no test methods mentioned in the literature about the design of RAP [5, 26]. The type checker has been tested with “toy exam-ples” and “in practice”, but not systematically [31]. The course book rule-based de-sign [4] mentions a few non-systematic methods to verify a model, among which the author’s review of the model on completeness, correctness, consistency and syntax, “reasoning” with rules and tests with populations that should produce specific viola-tions. We will use the last two suggestions.
We will produce test cases to seek out limitations of the rules applied in the exer-
cise. The test cases are student models that have a specific student error or are a vari-ant that is correct. A student model is in fact a meta-model filled with data, which means that the researcher does what in a real-life situation the RAP2 compiler should do. To illustrate how test cases are applied, Fig. 5 shows the prototype output for test case 1b of the exercise Identical names. The violation message that represents the message to the student is on top of the screen. Below that, the underlying dataset used for this test case is presented.
3.4 Validity of the case study
The quality of a design can be judged with three tests: construct validity, external validity and reliability [24]. We will discuss each of them in this section. Everything has to be evaluated in light of the research question How can relation algebra be ap-plied to create feedback for students about their business rules models?
To judge external validity, we need to look at the scope of our case study in rela-tion to the research question. As we stated in the introduction, there is a recursive aspect to this research: The student’s Ampersand model is tested with an Ampersand meta-model. This will eliminate from the case study a lot of things that relation alge-bra cannot do. The student will for instance not be able to make a model that can add up license fees, because relation algebra cannot do that [4]. The external validity to-wards other ITSs with other learning goals will be limited.
To judge the construct validity, we first need to ask ourselves whether the units of analysis, the exercises chosen, represent what relation algebra can do within the scope of learning how to model in Ampersand? And second, whether the test cases we use to verify the prototypes are complete enough? The answer to the first question is that the exercises are far from complete. However, we were able to make exercises for concepts, relations and rules and for both class 2 and class 3 problems. Table 1 is used to select the exercises. Second, the test cases are constructed to cover most errors students can make. A series of verification questions that cover a lot of the pitfalls in making rule, will be answered for each exercise (Table 3).
23
Table 3 Verification questions for rules in the test cases Main question Details Is the exercise (part of) a class 2 exercise?
Reasoning with relation algebra Is the rule free of type errors? Does the rule produce information about the
concepts being investigated? Are the multiplicities defined in the model ef-
fective in this rule? Does the rule work if the student creates endo-
relations? Does the rule work if the student creates dupli-
cates? Does the rule work if not all relations are popu-
lated? Test Cases Does the rule produce a violation message for
each variant with an error? Does the rule NOT produce a message for each
variant that is correct? What are the situations in which the rule is inef-
fective? Is the feedback informative? What level of detail is possible with this rule? Can the student be informed correctly in case
the rule is practically ineffective? Any extra instructions about the reference model?
4 Results
We will first present the meta-model that accommodates all exercises. Then we will present each exercise in a separate section with the reference model, the rules that performs the verification of the student model and the verification of those rules with reasoning and test cases.
4.1 The meta-model
The meta-model of concepts and relations that makes it possible to compare a stu-dent model with a reference model is depicted in Fig. 6 and all elements are described in Table 4. Each exercise uses only part of the model. The assumption is that the model will be filled with a population just as in RAP2: the student writes a script, uploads it and the script is compiled and a database is filled according to the meta-
24
model. The meta-model is based on the original RAP2 meta-model [5]. The congru-ence of our meta-model with the original meta-model is described in Table 5.
The meta-model we have created for the student model is a subset of the original meta-model of Michels [5]. We have used different names for the elements in the model and some relations are simplified or left out because we don’t need them in this research. An example of a simplification is that the concept Sign that in the original model mediates between the relation and the source and target concepts, is left out.
Fig. 6 The meta-model for the ITS functionality
25
Another deviation from the original RAP2 model is that all relations that have the concept StudModel as source are injective and surjective. This expresses the assump-tion that there is only one student, which means there is exactly one atom in the con-cept StudModel.
The model that is created by the teacher we call the reference model. The relation refers_to will indicate that the student model is the implementation of a specific exer-cise and needs to be checked against the reference model of that exercise. This rela-tion is univalent because a student model can relate to one exercise only. The con-cepts and relations of the reference model are at the right-hand side of the meta-model in Fig. 6.
The student model and the reference model use the same concepts as much as pos-sible. A relation must be uniquely identified so that they can be used unambiguously in rules. This is done with univalent and total relations from the concept for relations with three other concepts: the name, source and target of the relation (rule 5.73 in [5]). This implies that the reference model and the student’s model cannot use the same concept for relations. Rules have to be unique too, so this also makes it neces-sary to make separate concepts for rules as well.
We have added new elements that were not available in the original RAP2 meta-model, like the relations between the concepts for Rule and Relation to be able to define rules about rules.
We also added two endo-relations nextConcept and nextRelation to be able to count the number of atoms in the concepts. The relation nextConcept will be populat-ed as follows: Connect a first concept in the student model with another concept in the student model that is not connected yet. Connect that second concept with another concept that is not connected yet. When there is no unconnected concept available the concept will be connected to the first. The result will be that all concepts in the stu-dent model are connected in one closed chain. The relation nextRelation is filled in the same way.
Table 4 Elements of the ITS meta-model in alphabetical order Model element Description Concept Concept for the concepts in both the student model and the
reference model. nextConcept An endo-relation to be able to count the number of concepts in
the student model. nextRelation An endo-relation to be able to count the number of relations in
the student model. rAnt A rule in the reference model may have relations in the ante-
cedent. rCons A rule in the reference model may have relations in the conse-
quent. refers_to A student model can be created as implementation of a specif-
ic exercise. refMod_con A reference model may have concepts.
This relation is needed to distinguish concepts in the reference
26
Model element Description model from concepts in the student model.
refMod_refRel A reference model may have relations. refMod_refRule A reference model may have rules. RefModel Concept for the ID of the reference model. refRel_name A relation in the reference model has one name. refRel_src A relation in a reference model must have one source concept. refRel_tgt A relation in a reference model must have one target concept. RefRelation Concept for the relations defined in the reference model. RefRule Concept for the rules in the reference model. RelationName Concept for the name of the relations in both the student mod-
el and the reference model. sAnt A rule in the student model may have relations in the anteced-
ent. sCons A rule in the student model may have relations in the conse-
quent. studMod_con A student model may contain concepts. studMod_studRel A student's model may have relations. studMod_studRule A student's model may have rules. StudModel Concept for the student model. This is the base for the rest. It
defines the context of the model. studRel_name A relation in the student model has a name. studRel_src A relation in the student model has a source concept studRel_tgt A relation in the student model has a target concept StudRelation Concept for the relations defined in the reference model. StudRule Concept for the rules in the student model.
Table 5 Congruence of the ITS meta-model with RAP2 Model element Congruence with RAP2 [5] Concept Similar to concept Concept (ch. 5.8.1) nextConcept - nextRelation - rAnt - rCons - refers_to - refMod_con Similar to relation 5.39 refMod_refRel Similar to the composition Ctxpats;ptdcs (relations 5.38 and
5.43) RAP2 uses a concept Pattern between Context and Declaration
refMod_refRule Similar to the composition Ctxpats;ptrls (relations 5.38 and 5.41) RAP2 uses a concept Pattern between Context and Rule
27
Model element Congruence with RAP2 [5] RefModel - refRel_name Similar to relation 5.63 refRel_src Similar to the composition decsgn;src (relations 5.64 and 5.58)
RAP2 uses a concept Sign between Declaration and Concept refRel_tgt Similar to the composition decsgn;tgt (relations 5.64 and 5.59)
RAP2 uses a concept Sign between Declaration and Concept RefRelation Similar to the concept Declaration (ch 5.10.1) RefRule Copy of the concept Rule (ch 5.12.1) RelationName Similar to the concept Varid (ch. 5.13.1) sAnt - sCons - studMod_con Copy of relation 5.39 studMod_studRel Similar to the composition Ctxpats;ptrls (relations 5.38 and
5.41) RAP2 uses a concept Pattern between Context and Declaration
studMod_studRule Similar to the composition Ctxpats;ptrls (relations 5.38 and 5.41) RAP2 uses a concept Pattern between Context and Rule
StudModel Similar to the concept Context (ch. 5.5.1) studRel_name Copy of relation 5.63 studRel_src Similar to the composition decsgn;src (relations 5.64 and 5.58)
RAP2 uses a concept Sign between Declaration and Concept studRel_tgt Similar to the composition decsgn;tgt (relations 5.64 and 5.59)
RAP2 uses a concept Sign between Declaration and Concept StudRelation Similar to the concept Declaration (ch 5.10.1) StudRule Copy of the concept Rule (ch 5.12.1)
4.2 Exercise 1: Identical element names
The Identical names exercise is described in Table 6. This exercise in the class2/Rule-only category (Table 2) shows it is possible to check the names used for elements in the model. The concepts and relations in this exercise are fixed, this is one strategy variant allowed. The ITS must check if this strategy is indeed applied. The rule checks this by comparing the student model to the reference model. This rule will therefore work with any exercise in this class; we call this a reference rule.
Table 6 Description Identical names exercise
Exercise 1: Identical names Exercise The student is given a case description and an Ampersand-
script that has the definition of concepts. The assignment is to create relations using a specific name.
28
Exercise 1: Identical names E.g. Create the relations “sells” and “buys”.
Rule in Rulespeak Relations in the student model must also be present in the reference model.
Rules in RA studRel_name~;studRel_name |- refRel_name~;refRel_name
Violation message with maximum detail
"The relation ‘selling’ is not in the reference model”
The rule in this exercise is about the relation names, but the same mechanism can
be used for concept names. The rule cannot be used for concepts that are separated in the student model and reference model, like studRelation and refRelation.
The maximum level of detail in the feedback to the student is adequate, but it gives away more than is necessary in a first hint. The message can be made less precise by leaving out the information about the specific violating pair. E.g. “The list of relations is not the same as in the reference model.”
The rule of this exercise states that the names used in the student model must also be present in the reference model. An important choice is the selection of concepts under evaluation, the concepts used in the antecedent. The following rule evaluates whether the elements in the reference model are present in the student model: refRel_name~;refRel_name |- studRel_name~;studRel_name. A violating pair will consist of elements in the reference model and a violation message will re-port a problem with these elements: they are missing in the student model. Reporting that relation X is missing in the student model is the same as a literal instruction to add relation X in the student model. That kind of message does not support the learn-ing process the way a constraint based tutor should. The strength of constraint-based tutors is their ability to help recognize errors and with that stimulate the internaliza-tion of knowledge. To stimulate learning in the area of error-correction or error-detection, a message should be about the student model: relation Y has a problem, or relation Y does not exist in the reference model. The student has to perform some analysis before knowing what to do to improve the model. To create this kind of feedback, the elements in the student model must be evaluated and must be the main ingredient of the antecedent of the rule. When we do that, an incomplete model will not be recognized as incorrect. What is not in the student model will not be evaluated. Here we are confronted with a limitation of rule-based design with relation algebra: all rules will be executed simultaneously. The ideal situation would be to execute the rule for completeness after the other rules are satisfied.
Table 7 Test cases for exercise: Identical names Test cases Description of student model Message Rule works Test case 1a Perfect model No Yes Test case 1b Correct amount of relations, but some
incorrect names Yes Yes
29
Test cases Description of student model Message Rule works Test case 1c More relations than what was asked for,
extra relation has wrong name Yes Yes
Test case 1d Correct amount of relations, but two rela-tions use the same correct name, where the reference model uses that name only once.
No No
Test case 1e Less relations than required No Yes Test case 1f More relations than required and a correct
name is used twice No No
Test case 1g No relations defined No Yes
The test cases 1d and 1f in Table 7 show that duplicates in the student model make errors less visible. A mistake with duplicate names is easily made and such an error will remain unchallenged in this exercise and the student thinks there is nothing wrong. An extra rule can be added that states that relation names must not be used twice so that at least this situation is reported to the student (Table 8).
Test case 1g shows that when the student model has no relations at all, there is no error. We described this earlier, completeness is not tested with a rule that has the student model in the antecedent.
Table 8 Remedy Identical names exercise Exercise Identical names: remedy Remedy for Test case d and f Rule in Rulespeak The relation names in the student model must all be different. Rules in RA studRel_name;studRel_name~ |- I Violation message with maximum detail
“You have used a relation name ‘sells’ more than once.”
4.3 Exercise 2: Identical source
The Identical Source exercise (Table 9) shows how to check the source concept of a relation. This is an exercise in the class2/Rule-only category because it assumes relation names and concept names are fixed. The rule uses the relation names to create a relation from concepts in the student model to concepts in the reference model. When the resulting pair is not an identical pair, relations with the same name do not have the same source concept and a violation message is given. The rule is a reference rule, meaning that the reference model is incorporated into the evaluation, so this rule will work with any exercise in this class.
30
Table 9 Description exercise: Identical Source Exercise: Identical Source Exercise The student is given a case description and an Ampersand-
script that has the definition of concepts. The assignment is to create a relation using a specific name and specific source and target concepts. E.g. Create the rela-tions “sells” from Person to Thing.
Rule in Rulespeak A relation in the student model must have the same source concept as that same relation has in the reference model
Rules in RA studRel_src~;studRel_name; refRel_name~;refRel_src |- I[Concept]
Violation message with maximum detail
“The concept ‘Person’ seems to be incorrectly used as source-concept”
The violation message mentions the concept that is in error, not the relation, be-
cause it is the concept that is being evaluated in the rule. The student has to do some error-detection and find the applicable relation before a solution can be implemented. A more general message is possible. E.g. “Have a closer look at the source-concepts of the relations.”
An alternative would be to check source, target and relation name against the refer-ence model in a single rule like: studMod_studRel |- refers_to; refMod_refRel ; (refRel_src ; studRel_src~ /\ refRel_tgt ; studRel_tgt~ /\ refRel_name ; studRel_name~). The downside of that approach is that the information which of the three aspects under evaluation has caused the violation is not available to report back to the student. The learning curve for modeling is steep [13]. The target audience of our ITS is the novice student, and small, well-chosen steps are needed to scaffold the learning process. Therefore, it seems best to create separate rules for each specific aspect of the model.
The test cases (Table 10) show that the rule works well as long as the relation
names are correct and unique. When the relation name in the student model cannot be matched with a name in the reference model, there is no way of linking the source concepts and no error is found. This can be checked separately in a remedy rule that checks the names, like the rule in the Identical names exercise. Test case 2f shows that duplicate relation names always generate error messages. The antecedent of the rule creates non-identical pairs when a relation name is used more than once. This is different from the previous exercise where duplicate relation names resulted in poten-tially cloaking errors. Nevertheless, the remedy rule that was applied in the Identical names exercise can be used to warn for such a situation here as well.
31
Table 10 Test cases for the Identical Source exercise Test cases Description of student model Message Rule works Test case 2a Perfect model No Yes Test case 2b One relation has a source that is not used
in the reference model at all. Yes Yes
Test case 2c All relations have an incorrect source. Some concepts should be the source of other relations.
Yes Yes
Test case 2d One relation has an incorrect name as well as an incorrect source.
No No
Test case 2e No errors, but the models both use dupli-cate relation names.
Yes No
Test case 2f One relation has an incorrect source; both model use duplicate relation names.
Yes* No
Test case 2g One relation has an incorrect source and there are duplicate relation names, while the reference model has no duplicates
Yes Yes
Test case 2h No relations defined No Yes * The error is reported, but also the correct duplicates are reported as errors.
4.4 Exercise 3: Concepts in a relation
The Related Concepts exercise shows a rule that checks more for patterns than for exact copies of the reference model. This exercise is for class2/Rule-Rel problems. The exercise requires the names of concepts in the student model to match the ones in the reference model, but name and direction of the relations are implementation vari-ants that are for the student to define.
The violation message is clear without giving away too much. A less detailed mes-sage would be: “You have not chosen the correct concepts for each relation.”
Table 11 Description of the Related Concepts exercise
Exercise: Related Concepts Exercise The student is given a case description and an Ampersand-
script that has the definition of concepts. The assignment is to create relations with a certain meaning. E.g. “Create relations between the given concepts to define the name, place and age of a buyer and the date of the trans-action for each item that has been bought.”
Rule in Rulespeak Two concepts that are related in the student model must also be related in the reference model.
Rules in RA studRel_src~;studRel_tgt |- (refRel_src~; refRel_tgt) \/ (refRel_tgt~; refRel_src)
Violation message “You have related the concepts ‘Place’ and ‘Person’, but this
32
Exercise: Related Concepts with maximum detail
combination is not used in the reference model”
This exercise (Table 11) shows that it is possible to check source/target-
combinations without using the name of the relation and regardless of the direction of the relation. The working of the rule is that each source/target-combination in the student model should also be in the reference model, either exactly equal or in the flipped version. This last addition acknowledges that a relation can be defined in ei-ther direction without changing its effectiveness in the model. This is a reference-rule; it can be used with any exercise of this class.
Test case 3e show that this rule has the same problem we saw earlier of being fooled by duplicates. The remedy could be a check to report this situation (Table 13).
Table 12 Test cases for exercise: Related Concepts
Test cases Description of student model Message Rule works
Test case 3a Perfect model No Yes Test case 3b A source has been used twice: in a correct and in
an incorrect combination with a target. Yes Yes
Test case 3c A concept that is not used in the reference model is used as target-concept
Yes Yes
Test case 3d A flipped version of a relation in the reference model is used.
No Yes
Test case 3e A combination of source and target is used twice, while in the reference model it is only used once. One of the relations is wrong. It is the flipped version of the correct combination.
No No
Test case 3f Different concepts from the reference model Yes Yes Test case 3g There are no relations defined No Yes Test case 3h Perfect model. Both models use a source in two
relations No Yes
Test case 3i Source is used twice, once in an endo-relation that is not in the reference model
Yes Yes
Test case 3j A student model to demonstrate the remedies Yes: Remedy rules
n/a
Table 13 Remedies for the Related Concepts exercise
Exercise Related Concepts: remedies Remedy for Test case e and g Rule in Rulespeak A pair of concepts must not be assigned to more than one
relation Rules in RA Flipped_sign:
33
Exercise Related Concepts: remedies (studRel_src;studRel_tgt~) /\ (studRel_tgt;studRel_src~) |- I Same_sign: (studRel_src;studRel_src~) /\ (studRel_tgt;studRel_tgt~) |- I
Violation message with maximum detail
“There are relations that have each other’s flipped sign: Rel3” “There are relations that have the same sign: Rel1”
4.5 Exercise 4: Antecedent/Consequent
The Antecedent exercise shows it is possible to implement rules on rules. In the current implementation of RAP2 this is not possible because rules are modeled in a single concept as one complete expression. To be able to create a rule about rules, we need to model the semantics of the rule in separate concepts and relations. As a first step in this exploration of what is possible with rules, we have chosen the crude and simple division in antecedent and consequent. We have added a relation stating that a rule can have relation terms in the antecedent and another relation stating that a rule can have relation terms in the consequent.
In the rule for this exercise we aim to look more for patterns than literal element names. A typical pattern is that a composition of relations is compared to I or V and all relation terms must be on one side of the rule. That is what we apply here. The rule requires the antecedent or consequent to have no relation expressions (Table 14).
Table 14 Description of the Antecedent exercise
Exercise 4: Antecedent Exercise The student is given a case description and an Ampersand-
script that has the definition of concepts. The assignment is to create relations and a rule. E.g. “Create a rule stating that the buyer and seller of a thing cannot be the same person. Create the relations you need for this rule.”
Rule in Rulespeak All relation expressions in the rule must be on one side, either in the consequent or the antecedent of the rule.
Rules in RA (sAnt;sAnt~) |- -(SCons;SCons~) Violation message with maximum detail
“Rule ‘different_persons’ has relation terms in both the ante-cedent and the consequent”
This rule can be applied in both class 2 and class 3 exercise, because the model can
be completely anonymous; we do not need the names of concepts, relations or rules. It is, however not applicable to each exercise in those classes. Unlike the previous exer-
34
cises, this rule does not execute a comparison of student model with the reference model. We call this a direct rule; the requirements of the reference model are directly expressed in the rule. Direct rules apply therefore to specific reference models.
The violation message refers to a rule with the name the student has chosen for it. It could be made less precise by giving a more general comment. E.g. “All relation terms should be in either the antecedent or the consequent.”
The test cases in Table 15 show that this rule works nicely. Because the rule is a di-rect rule its does not have problem with duplicates that reference rules have. The rule shares the problem of completeness with all other exercises: if there are no rules at all in the student model, this will not be recognized as incorrect.
Table 15 Test cases for the Antecedent exercise
Test cases Description of student model Message Rule works
Test case 4a Perfect model, all in antecedent No Yes Test case 4b There are different relation expressions in
the antecedent and consequent of one rule. Yes Yes
Test case 4c There are no rules No Yes Test case 4d There are two rules, one does not comply. Yes Yes Test case 4e There are two rules, both don’t comply. Yes, twice Yes Test case 4f Perfect model, all in consequent No Yes Test case 4g Antecedent and consequent have exactly
the same relations Yes Yes
4.6 Exercise 5: Anonymous model
The Anonymous Model exercise investigates what characteristics of a complete model can be specified in rules without using any element name. The set of rules applies to a class 3 problem because we still prescribe the number of concepts and relations. The assignment in this exercise is to create a model with concepts and rela-tions with a certain meaning. E.g. “Create a model to check whether the street a per-son lives in, is in the town where that person is registered.” To find patterns, we in-vestigate the possible variants in Fig. 7. The two correct variants are models a and b, they have three concepts, three relations and each concept is related to the two other concepts. As is appropriate for a class 3 exercise, the names of elements will be ig-nored. This means that if we want to say anything about the student model, we must be able to count the number of elements. We use the new endo-relations nextConcept and nextRelation for this.
Analysis of the correct and incorrect variants leads to the conclusion that with five direct rules they can all be identified. The incorrect models 5c to 5i shown in Fig. 7 cover the typical mistakes a student can make. Models 5j and 5k are less typical, but created to demonstrate the limits of the rules. Test case 5j is created to create viola-tions for all five rules and Fig. 9 shows a screenshot of the resulting messages. Fig. 8 shows the underlying dataset of test case 5j. How each rule reacts to the different models is shown in Table 18.
35
Fig. 7 Correct and incorrect variants for the Anonymous Model exercise.
All rules are direct rules, there is no comparison with a reference model. The two
counting rules are specific for the number of elements the teacher wants in the model. The rules are easy to adapt for other exercises with smaller or larger models. The counting rules do not identify that there are no elements or just one (test case 5k). In
36
the latter, the relation nextRelation becomes an endo-relation that always produces an identical pair. The other three rules, “No_endo”, “Unique_sign” and “Flipped_sign” work with every model, regardless the amount of relations and concepts.
Table 16 Description exercise 5 Anonymous Model Exercise 5: Anonymous Model Exercise The student is given a case description
The assignment is to create a model with concepts and rela-tions with a certain meaning. E.g. “Create a model with three concepts to check whether the street a person lives in, exists in the town where that person is registered.” The expected model is one with three concepts and three rela-tions. There are two correct variants because the choice to flip a relation is an implementation choice, not strategy.
Rule Fig. 7 shows two correct models and most incorrect variants. With the five rules specified in Table 17, these models are all evaluated correctly (Table 18).
The violation messages listed in Table 17 show the maximum level of detail, less
detail is easy to accomplish and might be better for the learning process.
Table 17 Rules in the Anonymous Model exercise Rule Three concepts
RuleSpeak The student model must have three concepts Relation Algebra I |- nextConcept;nextConcept;nextConcept
Violation Message “There should be three concepts in this model”. Three relations
RuleSpeak The student model must have three relations Relation Algebra I |- nextRelation;nextRelation;nextRelation
Violation Message “There should be three relations in this model”. No endo- relations
RuleSpeak The student model must not have endo-relations Relation Algebra studRel_src~;studRel_tgt |- -I Violation Message “You have defined an endo-relation with concept ‘Street’.”
Unique sign
RuleSpeak Each relation in the student model must have a unique sign Relation Algebra studRel_src;studRel_src~ /\
studRel_tgt;studRel_tgt~ |- -I Violation Message “The relation ‘is_registered_in’ has a src-tgt combination
that is used by another relation too.” No flipped sign
RuleSpeak Within the student model, the sign of a relation must not be the flipped version of another relation.
Relation Algebra (studRel_src;studRel_tgt~) /\(studRel_tgt;studRel_src~) |- I
Violation Message “The relation ‘is_registered_in’ has a sign that is the flipped version of another relation.
37
Table 18 Test cases for Anonymous Model exercise Test cases see Fig. 7
Rule: Three
concepts
Rule: Three
relations
Rule: No endo-relations
Rule: Unique
sign
Rule: No flipped
sign Test case 5a - - - - - Test case 5b - - - - - Test case 5c - - Violation - - Test case 5d Violation Violation - - - Test case 5e - Violation Violation - Test case 5f - - - Violation - Test case 5g - - - - Violation Test case 5h - - - Violation - Test case 5i - Violation - - - Test case 5j Violation Violation Violation Violation Violation Test case 5k - X1 - - - 1 the rule does not produce a violation, while the model is incorrect.
Fig. 8 test case 5j: underlying dataset as presented in the prototype
38
Fig. 9 All violations messages are visible in test case 5j
39
5 Conclusion
The OU aspires to extend the current e-learning environment RAP2 for rule based design with ITS functionality. The RAP2 system has itself been created with a rule based design in Ampersand that uses relation algebra as language for rules. In this system, student models are stored as information and rules are applied on this infor-mation. To be an ITS the system needs to automatically verify the correctness of models against a reference model. This research investigated whether the same busi-ness rules language can be used for this verification. The research question we tried to answer is: How can relation algebra be applied to create feedback for students about their business rules models? The research was designed as an exploratory case study in which a feedback system with exercises, reference models and student models is simulated. The research has two results: a categorization of business rule models based on their verifiability and an indication of the effectiveness of relation algebraic rules as verification method.
We demonstrate in five exercises that it is possible to generate feedback with rela-
tion algebra. This requires first a meta-model of the student model combined with the reference model. Second, each exercise has a reference model and rules. Reference rules compare the student model with the reference model as is done in exercises Identical names, Identical Source and Related Concepts. Another method is using only elements from the student model as is done in the exercises Antecedent and Anonymous Model. We call this a direct rule. Both rule types are checking conform-ance to the reference model. In a reference rule, the norm the student model should comply to, is taken from the reference model, where in a direct rule, the norm is di-rectly translated into the rule. There is a practical application for both types of rules. Theoretical all reference rules could be defined as direct rules, but when literal names of model elements are relevant for the rule, a reference rule has the advantage of be-ing reusable in more exercises.
The business rules approach results in a constraint-based feedback system, which means that anything that is not recognized as incorrect, is considered to be correct. Feedback for the students is meant to help them develop first error-correction skills and in a later stage error-detections skills. The rules must be constructed in such a way that the feedback can point out the problem without giving away the whole solu-tion. This leads to two requirements:
- A rule must be simple, evaluating only one possible problem at a time. - A rule must be about the student model so that elements in the student model
cause the violation. A consequence of these requirements is that completeness of the model cannot be verified. Only elements that exist in the student model will be evaluated.
The biggest problem for ITSs is that of semantics, and our system is no exception. As we have seen in the definition of problem classes, an educational problem that has some depth allows for more than one way to solve the problem. As long as the student cannot choose the names of all elements, we are dealing with a class 2 problem, as is the case in the exercise Related Concepts. Exercises that allow free choice of names
40
for model elements and evaluate the student model on a more conceptual level, are class 3 problems. The exercises Antecedent and Anonymous Model can be applied in this problem class. In a completely anonymous model, counting the number of ele-ments is essential and specific relations nextConcept and nextRelation have been created in the meta-model to do this.
Both class 2 and class 3 exercises are meaningful exercises for a learning student. It is a typical form of scaffolding to limit the problem for the student to specifying only a rule, or specifying relations and rules based on given concepts, before starting the job of creating a model from the ground up.
Relation algebra and business rules have of course limitations. A rule defined as implication - antecedent implies consequent – is a variant of the if-then construction. But there is no else-construction. A violation is triggered with every violating pair and messages are repeated where one message would suffice. Examples are most rules in the exercise Anonymous Model as shown in Fig. 9. E.g. The rule “flipped_sign” trig-gers a message that two relations are each other’s flipped version for each of these two relations. Because all rules are executed without knowledge of the history, it will not be possible to apply, with relation algebra alone, the Point-Teach-Bottom-out technique [16]. A last problem is that the rules are valid within the limited scope of the exercise. Such a mechanism is not available in the current implementation of RAP2 [5].
This case study leads to the conclusion that relation algebraic rules can be used to generate feedback for students. To be able to generate information that goes beyond syntax and type issues, the rules must be specific for an exercise or group of similar exercises. To generate feedback that scaffolds the learning process, a rule must be about the student model and verify one issue. Rules are not effective in all possible situations. There are some limitations to relation algebra that makes it less suitable for a learning system than a programming language, the most important being that rules and messages cannot be applied conditionally.
6 Discussion
The analysis of how the complexity and number of variants increase in modeling exercises for Ampersand, has resulted in a definition of problem classes in Table 1. This case study has succeeded in creating some rules for class2/rule-only, class2/rule-rel and for class 3. As the definition of the classes show, a lot of possible exercises have not been covered yet. Examples are rules for multiplicities and for the concepts that are being evaluated in rules. The exercises presented give a good idea how to go about making such rules.
Creating the right result within the multiplicity constraints on a relation, remains the most difficult aspect of specifying effective relation algebraic rules. This research has circumvented a big problem by defining some relations in the meta-model as be-ing injective and surjective (Fig. 6). In RAP2 these relations do not have these con-
41
straints because there are many students and even more student models [5]. Without adjustments, the rules that use those relations in the prototypes will not perform in the real system.
In this case study, we have used a rule engine for business rules to generate feed-back about student products. This approach by itself has not been questioned yet, but there are certainly some issues worthwhile to address. A business rule is fundamental-ly different from a feedback model for a student. First, a business process is different from a learning process. A learning process is inherently private while in a business process the actor performs a business role. Second, scaffolding a learning process is different from supporting the execution of a process. The scaffolding is temporary and aimed at reducing the need for scaffolding, where the rules in a business process are there to guarantee correct process outcomes continuously. There is a risk that incorpo-rating two incompatible goals in one system will weaken both applications.
With regard to an ITS for Ampersand we have the following recommendations for
future research: As we stated in the method section, much of the functionality that is needed to
make Ampersand into an ITS cannot be handled with relation algebra and needs to be implemented in Haskell, the language Ampersand is built with. Some of the rules we did implement with relation algebra require many interventions at compile time like the counting relations. The question is valid whether this is worth the trouble. Can the same result be achieved with less effort when it is handled completely in the compil-er? We propose to investigate the possibilities of implementing the whole feedback system in Haskell.
Regardless of the technical implementation, future research should also be aiming at the knowledge components in this domain. Two areas are relation algebra and de-sign patterns for Ampersand models, but no doubt there are more. When there is a systematic overview of these knowledge components the right scaffolding therefor can be developed.
Finally, a didactical research on the log-files that RAP2 has produced can be of great value for creating the right exercises. What are the mistakes students make? What knowledge components are not yet available to the student that makes these mistakes? This information would be the best input for defining the exercises in the ITS.
References
1. Object Management Group, Semantics of Business Vocabulary and Business Rules (SBVR) - version 1.3. 2015, Object Management Group.
2. Boyer, J. and H. Mili, Agile Business Rule Development: Process, Architecture, and JRules Examples. 2011, Berlin, Heidelberg: Springer Berlin Heidelberg.
3. Joosten, S. Why Ampersand [PREZI presentation]. Ampersand Conference 2015 [PREZI presentation] 2015 23 Juni 2015; Available from: https://prezi.com/isufoqz_cchc/why-ampersand/.
42
4. Wedemeijer, L., et al., Rule Based Design. 2014, Heerlen: Open Universiteit. 5. Michels, G., Development Environment for Rule-based Prototyping [THESIS]. 2015,
Open Universiteit Nederland: Heerlen. 6. Michels, G. and S. Joosten, Progressive Development and Teaching with RAP, in
Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research. 2013, Open Universiteit, Heerlen: Arnhem, Netherlands. p. 33-43.
7. Keuning, H., Strategy-based feedback for imperative programming exercises [MASTER THESIS], in Faculty of Management, Science and Technology. 2014, Open University The Netherlands.
8. Gerdes, A., Ask-Elle: a Haskell Tutor [THESIS]. 2012: Universiteit Utrecht. 9. The Business Rules Group. Defining "Business Rule". 2016 15 maart 2016];
Available from: http://www.businessrulesgroup.org/defnbrg.shtml. 10. Business Rule Solutions. RuleSpeak. [cited 2016 16 maart 2016]; Available from:
http://www.rulespeak.com/en/. 11. Tarski, A., On the calculus of relations. The Journal of Symbolic Logic, 1941. 6(03):
p. 73-89. 12. Dijkman, R.M., L. Ferreira Pires, and S. Joosten. Calculating with Concepts: a
Technique for the Development of Business Process Support. in Practical UML-based rigorous development methods - countering or integrating the extremists, workshop of the puml-group held together with the UML 2001. 2001. Toronto, Canada.
13. VanLehn, K., Model construction as a learning activity: a design space and review. Interactive Learning Environments 2013. 21(4).
14. Mitrovic, A., K.R. Koedinger, and B. Martin, A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling, in User Modeling 2003, P. Brusilovsky, A. Corbett, and F. de Rosis, Editors. 2003, Springer Berlin Heidelberg. p. 313-322.
15. Hattie, J. and H. Timperley, The power of feedback. Review of educational research, 2007. 77(1): p. 81-112.
16. VanLehn, K., The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 2006. 16(3): p. 227-265.
17. Kodaganallur, V., R.R. Weitz, and D. Rosenthal, A comparison of model-tracing and constraint-based intelligent tutoring paradigms. International Journal of Artificial Intelligence in Education, 2005. 15(2): p. 117-144.
18. Le, N.-T., F. Loll, and N. Pinkwart, Operationalizing the continuum between well-defined and ill-defined problems for educational technology. Learning Technologies, IEEE Transactions on, 2013. 6(3): p. 258-270.
19. Jeuring, J. IDEAS: Interactive Domain Reasoners. 2010 [cited 2015 8 December 2015]; Available from: http://ideas.cs.uu.nl/www/.
20. Keuning, H., B. Heeren, and J. Jeuring, Strategy-based feedback in a programming tutor. 2014, Department of Information and Computing Sciences Utrecht University: Utrecht.
21. Berners-Lee, T.J.O., THE SEMANTIC WEB. Scientific American, 2001. 284(5): p. 34.
43
22. Suraweera, P. and A. Mitrovic. KERMIT: A constraint-based tutor for database modeling. in International Conference on Intelligent Tutoring Systems. 2002. Springer.
23. Jeuring, J., A. Gerdes, and B. Heeren, Ask-Elle: A Haskell Tutor, in 21st Century Learning for 21st Century Skills. 2012, Springer. p. 453-458.
24. Yin, R.K., Case study research: design and methods. 2009, London: Sage Publications.
25. Team, T.A. Documentation of Ampersand. 2016 [cited 2016; Available from: https://www.gitbook.com/book/ampersandtarski/documentation/details.
26. Michels, G., et al., Ampersand - Applying relation algebra in parctice, in Relational and Algebraic Methods in Computer Science: 12th International Conference, RAMICS 2011, Rotterdam, The Netherlands, May 30 – June 3, 2011. Proceedings, H. Swart, Editor. 2011, Springer Berlin Heidelberg: Berlin, Heidelberg. p. 280-293.
27. Team, T.A., Ampersand developers hub. 2016. 28. IEEE, IEEE Standard for System and Software Verification and Validation. 2012. 29. Lodder, J., H. Passier, and S. Stuurman. Using IDEAS in teaching logic, lessons
learned. in 2008 International Conference on Computer Science and Software Engineering. 2008. IEEE.
30. Gerdes, A., et al., Ask-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback. International Journal of Artificial Intelligence in Education, 2016: p. 1-36.
31. Joosten, S.M.M. and S.J.C. Joosten, Type Checking by Domain Analysis in Ampersand, in Relational and Algebraic Methods in Computer Science: 15th International Conference, RAMiCS 2015, Braga, Portugal, September 28 - October 1, 2015, Proceedings, W. Kahl, M. Winter, and J. Oliveira, Editors. 2015, Springer International Publishing: Cham. p. 225-240.
