Competency Based Learning in the Web of Learning Data

Guillaume DurandNational Research Council

Canada100 des Aboiteaux str.

Moncton, CanadaGuillaume.Durand@nrc.gc.ca

Nabil BelacelNational Research Council

Canada100 des Aboiteaux str.

Moncton, CanadaNabil.Belacel@nrc.gc.ca

Cyril GoutteNational Research Council

Canada1200 Montreal Rd.Ottawa, Canada

Cyril.Goutte@nrc.gc.ca

ABSTRACTIn this paper, we present, discuss and summarize differentresearch works we carried out toward the exploitation ofthe Web of data for learning and training purpose (Web oflearning data). For several years now, we have conductedefforts to explore this main objective through two comple-mentary directions. The first direction is the scalabilityand particularly the need to develop methods able to pro-vide learners with adequate learning path in the world ofbig data. The second direction is related to the transitionfrom Web data to Web of learning data and particularly theextraction of cognitive attributes from Web content. Forthis purpose, we proposed different text mining techniquesas well as the development of competency framework en-gineering tools. Resulting evidence-based techniques allowus to properly evaluate and improve the relationships be-tween learning materials, performance records and studentcompetencies. Although some questions remain unansweredand challenging technology improvements are still required,promising results and developments are arising.

KeywordsWeb learning data recommendation; Web data features ex-traction; Learning skills engineering.

1. INTRODUCTIONAccording to Wiley [23], a learning object is “any digi-

tal resource that can be reused to support learning”. Assuch, a significant number of Web pages match the defini-tion of learning object and Wikipedia could be seen as theprototypical provider of learning objects. The 37 million ar-ticles contained in the online encyclopedia generate around18 billion views a month. Therefore, Web learning, eveninformal, is already happening on a large scale thanks toa constantly increasing number of quality contents, betterconnectivity and web literacy. In the meantime the toolsproviding web users with learning material have not reallyevolved since the beginning of the WWW. Wikipedia is still

Copyright 2016 Crown in Right of Canada. This article was authored by employeesof the National Research Council Canada. As such, the Canadian Government retainsall interest in the copyright to this work and grants to the publisher(s) a nonexclusive,royalty-free right to publish or reproduce this article, or to allow others to do so, pro-vided that clear attribution is given both to the NRC and the authors.

WWW’16, April 11-15, 2016, Montréal, Québec, Canada

ACM 978-1-4503-4144-8/16/04.http://dx.doi.org/10.1145/2872518.2890460

relatively flat in term of content hierarchy, search enginesare more interested in tracking users’ habits for advertis-ing purpose rather than learning needs and no tools offerlearning services similar to what can be found on virtuallearning environments. A learning path recommender sys-tem providing users guidance to select the right web materialto reach a targeted level of knowledge would benefit userslooking to mine the web for learning purpose. The sameway some big internet names have been able to determineuser’s purchase interests, we should be able to detect user’sabilities and knowledge and align their specificities to theresources available to optimize learning. This vision doesnot come without deep challenges that we discuss along thedifferent propositions presented in this paper. Among thechallenges of transforming the WWW into a real web oflearning, is the scalability of the solutions. This is exploredin the context of our investigations regarding learning pathor curriculum recommendation (Section 2). Over the years,educational data mining and recommendation technologieshave proposed significant contributions to provide learnerswith adequate learning material by recommending educa-tional papers [20] or internet links [13], using collaborativeand/or content-based filtering. Other approaches, especiallyin the course generation research community, address theneed for recommending not only the learning objects them-selves, but sequences of learning objects [22, 12]. However,none have investigated learning path recommendation forlarge repositories counting millions of learning objects likethe Web is potentially offering. We discuss and present de-velopments regarding learning path in the following sectionbefore focusing in section 3 on potential approaches to con-vert the Web to a Web of learning data and discussing cur-rent and further developments in the fourth section of thepaper.

2. BUILDING (WEB) LEARNING PATH

2.1 Dynamic DependenciesTo some extent, the Web can be seen as a graph in ex-

pansion, where edges are built according to the usage made.For instance, it is usual to build edges according to the hy-pertext links between pages but they can also be built for amore specific learning perspective considering competenciesprerequisite and gains. For this last purpose, let G = (V,E)be a directed graph representing the Web of learning data.Each vertex or node in G corresponds to a learning object.Two vertices are connected if there exists a dependency re-lation, such that one vertex satisfies the prerequisites of the

489

WWW’16 Companion, April 11–15, 2016, Montréal, Québec, Canada.

Figure 1: Illustration of the dynamic direct graph principle.If A provides competency 1, B provides competency 2, andD is accessible to a learner with competencies 1 and 2, a newedge should be created to connect B and D.

other. So, an edge−→AB between two vertices A and B means

that the learning object A is accessible from B. Buildingsuch dependencies is non-trivial especially if they are notsolely based on explicit HTTP links. We will talk moreabout this problem later in Section 3. Dependencies aside,the issue is now to build a learning path on a large graph,starting from user’s initial competencies, and ending at thetarget competencies.

If the Web was simply a big graph with static edges then,depending on the learning strategy, recommending a learn-ing path could be solved by a shortest path algorithm. How-ever, the edges are not static, but rather dependent on thelearning objects consulted previously. For example, let’sconsider a learning object D that would be accessible toa learner having reached mastery in competencies 1 and 2.Assume that competency 1 is provided by learning objectsA and C and competency 2 is provided by learning objectsB and C. D is reachable if learning objects A and B arecompleted or if learning object C is completed. If a learnercompletes learning object A at time t and learning objectB at time t + 1, the learner will have the competencies re-quired to reach D and a new edge between B and D shouldbe created (Figure 1).

2.2 Heuristic Approach and Graph theoryFinding a learning path, in our model, consists in looking

for the shortest path in a large dynamic graph. Due tothe complexity of searching such a graph, no deterministicapproach is suitable. Therefore we proposed a two-stagealgorithm that first reduces the problem state or graph size,then solves the reduced graph.

The first stage can be seen as a loop generating subgraphsor cliques1, until one such clique is generated whose prereq-uisites are a subset of the learner’s competencies. Cliquesare generated in a top-down manner. We begin with thetarget clique, which is composed of a single learning object(we create a fictitious learning object, β, whose prerequisitecompetencies correspond to the list of the learner’s targetcompetencies). Cliques are then generated by finding ev-ery vertex where at least one output competency is found

1Complete subgraphs in which all the learning objects areadjacent to each other.

in the prerequisite competencies of the clique (the union ofall prerequisite competencies of every learning object withinthe clique) to which it is a prerequisite. As such, cliquescontain the largest possible subset of vertices which satisfiesthe condition “if every learning object in the clique is com-pleted, then every learning object in the following clique isaccessible”2 while preserving dynamicity constraints.

In the second stage, a greedy algorithm attempts to finda path by considering each clique one after the other andreducing it to a minimal subset of itself which still verifiesthe condition “if every learning object in the clique is com-pleted, then every learning object in the following clique isaccessible”. For each clique, the local optimum is obtainedwhen the minimum subset of vertices with a minimum “de-gree”, being the sum of the number of prerequisite compe-tencies and output competencies of the vertex, are found. Inother words, the greedy algorithm selects in each clique a setof learning objects minimizing the number of competenciesrequired and gained in order to locally limit the cognitiveload of the selected material. Note that the degree functioncould be calculated to accommodate other learning policieslike maximizing learning gains to stimulate curiosity.

2.3 Further DevelopmentsOverall, the clique-based approach is an efficient way to

reduce the solution space and check the existence of a solu-tion. However, a greedy search may not lead to the shortestlearning path. To solve this issue, we investigated binaryinteger programming as an alternative [4]. Our implemen-tation of the branch-and-bound (B&B) algorithm solved theaccuracy issue but the performance cost is questionable. Al-ternatives as mentioned by Applegate et al. [1] like branch-and-cut could prove to be faster but likely not as fast as thegreedy approach. Moreover, as mentioned in [11], the effi-ciency of reducing the solution space with the clique mech-anism is highly dependent on the dataset topology (averagenumber of gain and prerequisite competencies per learningobject) as highlighted by Figure 2. For instance, the calcula-tion time increases differently depending on the variation ofthe number of output or prerequisite competencies3. Mixingalgorithms in order to balance computational time and ac-curacy based on the graph topology might bring interestingdevelopments.

3. WEB TO WEB OF LEARNING DATA

3.1 Competency MiningUsing the Web of learning data requires the algorithms to

be able to provide the adequate guidance to learner. Thegraph model proposed in the previous section to recommendlearning paths requires competencies to be identified for eachWeb page in order to be processed and potentially recom-mended. However, this information is usually not availablein web content. Moreover, editing each web page to man-ually define required metadata is not an option. Not onlybecause this would take a huge amount of time, but also be-cause extracting competency information is complicated and

2This condition confers also the completeness property tothe subgraphs. By extension, if every learning object is ac-cessible in the following clique then all of them are adjacent.3A more detailed performance study of the greedy algorithmis available in [11].

490

Figure 2: Computational time of the Greedy version of thetwo stage algorithm with different graph topologies counting105 learning objects and 104 competencies (Intel Core 2 Ex-treme Q9300 CPU at 2.54GHz and 8GB RAM with ApacheCassandra Database on the same machine).

prone to mistakes. Even for experts in cognitive sciences,defining the right granularity as well as the distribution ofthe competencies among the learning tests is a challengingtask [10].

Q =

Skillc1 Skillc2 Skillc3

ItemA 1 0 0ItemB 0 1 0ItemC 1 1 0ItemD 0 0 1

Figure 3: Example of a Q-matrix illustrating the competen-cies gained in the dynamic graph example of figure 1.

One promising approach comes from the psychometric andeducational data mining (EDM) community where some re-searchers are trying to automatically discover competenciesbased on learners performances. They pursue the objectiveof generating a matrix called Q-matrix that associates learn-ing items and competencies (Figure 3). Desmarais and as-sociates [7, 8] refined expert Q matrices using matrix factor-ization, with impressive successes. However as non-negativematrix factorization is sensitive to initialization and proneto local minima, a fully automated generation might be outof rich with this method. Sun et al. [19] generated binaryQ-matrices using an alternate recursive method that auto-matically estimates the number of competencies, yieldinghigh matrix coverage rates. Others [18, 6] estimated the Q-matrix under the setting of well-known psychometric modelsthat integrate guess and slip parameters to model the vari-ation between ideal and observed response patterns. Theyformulated Q-matrix extraction as a latent variable selectionproblem solved by regularized maximum likelihood, but re-quiring the number of competencies as input. Finally, Sparse

Factor Analysis [17] was recently introduced to address datasparsity in a flexible probabilistic model.

3.2 Competency DescriptionAll of these approaches address competency frameworks

generation from slightly different angles but none of thesetechniques address the problem of providing a textual de-scription of the discovered attributes. This makes them hardto interpret and understand, and may limit their practicalusability. In the mean time, it is difficult to extract a full textdescription of a latent competency fully automatically. How-ever, a lot of textual information is available in test items,whether in the text of the questions, hints or responses. Weproposed a simple probabilistic model that extracts, fromthis text, the keywords that are most relevant to each skill[14]. The intuition is that relevant keywords are not alwayshigh frequency word, which tend to be common or topicalwords. Keywords relevant to a competency are words thatare relatively frequent in items testing that competency, andrelatively infrequent in items testing other competencies.

We tested this on a small dataset from the PSLC Datashop[16] containing 823 test items, with competency frameworksranging from 44 to 108 competencies associated with at leastone item. Table 1 shows examples of keywords extracted for5 competencies with known labels. In our experiments, thelabels were removed before the keyword extraction was ap-plied to the associated test items. Note that in most cases,we extracted words from the unseen label, as well as manyother related relevant words.

We quantified this process using various metrics measur-ing coverage and specificity of keywords, on several com-petency frameworks, and compared our simple probabilisticextraction technique to the common alternative of using themost common (most frequent) words. We found that withour extraction, keywords are used to describe on average 1.2to 1.4 competencies (maximum 9), whereas the highest fre-quency keywords describe on average 3 competencies each(maximum 87). In fact some words like “correct” or “incor-rect” are highly frequent, but clearly not very informativeabout competencies.

In our proposed method we only extracted key words fromthe textual data. A straightforward improvement would beto extract longer, more descriptive information such as mul-tiword terms, short snippets from the data or more compli-cated linguistic structure such as subject-verb-object triples[2]. The data-generated descriptions can also be useful inthe generation or the refinement of Q-Matrices. Namingcompetencies can offer significant information on the con-sistency of a Q-matrix. This can be used as an alternativeor a complement to existing refinement methods based onfunctional models optimization [8].

3.3 Competency Frameworks EvaluationWhile competency generation seems to provide interest-

ing results toward the objective of automatically extractingcompetencies from observed performance patterns, the qual-ity of the extracted matrices is questionable. Unfortunately,the predictive quality of such matrices is sometimes only asgood as chance in term of predictive accuracy. In order toaddress this problem, we proposed a method that aims atspecifically evaluating the predictive quality of a Q-matrix.For this purpose, we proposed an evaluation method using adeterministic model using matrix factorization techniques.

491

Competency label #items Top 10 extracted keywordsidentify-sr 52 phishing email scam social learned indicate legitimate engineering anti-phishingp2p 27 risks mitigate applications p2p protected law file-sharing copyright illegalprint quota03 12 quota printing andrew print semester consumed printouts longer unused costvpn 11 vpn connect restricted libraries circumstances accessing need using universitydmca 9 copyright dmca party notice student digital played regard brad policiespenalties dmca 2 penalties illegal possible file-sharing fines 80,000 $ imprisonment high yearspenalties bandwidth 1 maximum limitations exceed times long bandwidth suspended network access

Table 1: Top 10 keywords extracted, for a sample of competencies, from the text of their test items. “# item” is the numberof items associated with the competency. Competency labels (left) are hidden from the extractor.

The evaluation method considers a factorization model;R = Q × S where R and S are, respectively, the “Results”(item×student) and “Student” (competencies×student) ma-trices. Assuming R observed andQ known, our method eval-uates how well Q would predict unobserved results througha classical 10-fold cross validation algorithm (injecting miss-ing values in R). For each fold, an estimated student ma-

trix Ŝ is obtained by solving the system of linear equationsQ× x = R. In our experiments, we used the weighted leastsquares method4, although we could also impose variousconstraints on the s0tudent matrix using non-negative orBoolean matrix factorization. Test observations are removedfrom the results matrix R according to the cross-validationframework, and predictions are made for these values using

the product of the Q-matrix and the estimated Ŝ. Prelimi-nary results obtained with this method are conclusive [10].To illustrate this method, let’s consider two Q-matrices pre-sented in Figure 4 and estimate the predictive quality ofeach Q-matrix. In this example, we consider a well knowndataset with its original Q-matrix and a Q-matrix variantthat was automatically improved.

(a) Tatsuoka (20x8) (b) Desmarais (20x8)

Figure 4: Graphical representation of the Q-matrices usedin our example. Black cells in the Q-matrix (ones) indicatethat one of the 20 items (rows) is associated with one of the8 skills (columns).

Both Q-matrices are related to a dataset involving 2144middle school students answering 20 items on fraction alge-bra and requiring the use of the eight following skills [21]:

1. Convert a whole number to a fraction,

2. Separate a whole number from a fraction,

4http://en.wikipedia.org/wiki/Least squares#Weighted leastsquares

3. Simplify before subtracting,

4. Find a common denominator,

5. Borrow from whole number part,

6. Column borrow to subtract the second numerator fromthe first,

7. Subtract numerators,

8. Reduce answers to simplest form.

The Q-matrix (a) on the left side in Figure 4 was proposedby Tatsuoka [21], while the one on the right (b) was refined(automatically improved from the original (a)) by Desmarais[9]. The refining process resulted in changing the mappingbetween items and skills slightly: skill 3 (“Simplify beforesubtracting”) was added to item 8 and removed from items19 and 20, and skill 8 (“Reduce answers to simplest form”)was removed for items 10 and 12. Table 2 shows Q-matrixevaluation results calculated with our evaluation method.The Root Mean Square Error (RMSE) as well as the MeanAverage Error (MAE) are smaller for Q-matrix (b), showingthat the refined matrix has better predictive ability than theoriginal one (lower reconstruction errors). Calculating theRMSE for Q-matrix (a) with another cognitive model likethe Additive Factor Model (AFM) yields an error close to.37. The values range obtained with our method tends tocorroborate that matrix factorization models lead to predic-tion errors that are comparable to other cognitive diagnosticmodels [3] while keeping interesting advantages.

As the weighted least squares algorithm handles miss-ing values without imputing them, this makes the proposedmethod usable in cases where the observed results are in-complete, such as when learners do not perform the sameitems, or progress at different paces. Uncompleted itemsshould do not necessarily prevent the Q-matrix from beingtested and iteratively improved in parallel with the testingactivity. Using a cognitive model with very few parametersis also an advantage to generate and evaluate Q-matrices.In fact, adding parameters to a cognitive model make theevaluation of the Q-matrix more difficult since the expertmisconceptions can be more easily compensated by theseextra parameters.

3.4 Multi-Relational Competency FrameworksThe generation of competency frameworks is a challeng-

ing task. So far, our research effort has focused on relativelysimple competency frameworks modeled by Q-matrices. Infact, Q-matrices do not consider different types of associa-tions between tests and competencies. Nonetheless, in thegraph model we proposed in Section 2 we considered twotypes of competencies; the competencies that are required to

492

Table 2: Results obtained by the Q-matrix evaluationmethod on expert-made Q-matrices, in root means squared(RMSE) and mean average (MAE) reconstruction errors.

Dataset (a) Tatsuoka (b) Desmarais

RMSE .4051 .3810MAE .2531 .2353

understand the learning material and the competencies thatthe learning material provides to the learner. Q-matrices ascompetency frameworks are built around tests that requiresome competencies to be passed but rarely provide learninggains. As a result, the unidirectional learning competencyrelationship investigated with Q-matrices is not sufficient tobuild the graph we discussed in Section 2 to build envisionedWeb of learning data. Extracting both competencies gainedand required while research focusing on “simple” Q-matrixmodels is still emerging, represents a very serious challenge.For instance, the question of discriminating competenciesrequired to use a web page and the competencies providedas benefits might not be simple to answer. Many web pagesmay not explicitly contain any words, metadata or moreglobally information related to required competencies. Forexample, a web page presenting techniques of matrix decom-position would require from the reader a good understandingof matrix multiplication but no reference to matrix multi-plication might be attached to the document. Even known,the relation between competencies required and competen-cies gained may not be straight forward to understand andexploit. If a person consulting the page on matrix decompo-sition does not understand matrix multiplication, the set ofcompetencies gain from reading the page may be limited. In-ferring such limitation may require a deeper understandingof competencies dependencies as well as learner knowledgeand cognitive capacities that is once again challenging toobtain in such an informal learning environment.

4. DISCUSSIONOnline learners are learning without being evaluated like

students would be by a teacher that is able to provide guid-ance based on the normative and formative assessment per-formed. Nonetheless, this might not necessarily mean thatcognitive diagnostic models cannot benefit the Web of learn-ing data research. Even so the cognitive diagnostic modelsused in competency frameworks generation require the useof test results and that typical Web pages do not measuresuccess or failure, similar metrics could be built. One of themost widely used technologies online is collaborative filter-ing, using either explicit ratings filled by users or implicitratings based on different observed behaviors (or both). Us-ing feature extraction and navigational patterns, it shouldbe possible to build metrics that could be used to build weblearning predictive models.

However, considering only the informal nature of web learn-ing might narrow the scope of the possible applications. TheWeb of learning data is composed of an increasing numberof shared educational contents. Some of them are shared forindexing purpose and are formatted with rich metadata in-formation through standards like the Learning Object Meta-data. While the explicit definitions of competency gained

and competency required are not designed in metadata for-mat, related information like learning objectives, or depen-dencies can be exploited to define competency gains. Com-petency required can be induced when a hierarchical com-petency framework is defined, or using the learning objectdependencies whenever it is correctly filled in the metadataformat. This information, when available, can provide usefulinitialization values for competency referential refinement.

Scalability is a requirement for learning path recommen-dation but also for competency frameworks extraction, re-finement and evaluation. So far, competency frameworksengineering has been conducted on relatively small datasetscompared to the amount of information that might be treatedfor a Web of learning data. Hubwieser and Mühling [15]clearly embraced this issue by proposing a method to minecompetencies in large data sets (tens of thousands of par-ticipants, after preprocessing). Their method is particularlyinteresting since it looks first at the set of items using latenttraits analysis to find a set of questions that would evalu-ate a common competency (joint psychometric construct).Their comparison to several psychometric models (Item Re-sponse Theory, IRT) confirms the validity of the competencymapping. So far, the type of competency framework builtis very simple and equivalent to a Q-matrix with one com-petency shared by n items but the authors mention thata multidimensional IRT can potentially be used. Alterna-tively, our Q-matrix evaluation algorithm could technicallybe implemented in this method to build and validate Q-matrices with several competencies per item. However, themain limitation of this method may come from scaling thelatent trait analysis, which requires a very unbalanced ratiobetween items and performance observations. As a result,the method is particularly well adapted to situations when alot of participant and results are available on a small numberof items.

Considering the high number of documents and the lim-ited performance information of web learning, promisingwork may come from the information retrieval communitywith methods based, for example, on Latent Dirichlet Al-location (LDA) [5]. Like topics in textual data analysis,competencies can be modeled as latent variables that are in-ferred rather than directly observed. Once topics and theirassociated distributions have been estimated from a corpusof documents, LDA allows the assignment of new documentsto these topics. Similarly, new test items could be associatedwith estimated latent competencies. Note that in LDA likein the related Matrix Factorization methods, the number oflatent topics/competencies must be pre-specified.

5. CONCLUSIONIn this paper we discussed ongoing efforts towards a Web

of learning data that would use the constantly growing re-sources available online to benefit web learners needs. Aprominent challenge that was initially discussed is to pro-vide learners with customized tailored learning path allow-ing them to reach target or key competencies. However, itis necessary to recognize that learning requires needs goingbeyond the navigational constraints the WWW was builtupon. Considering the scale of the Web and the dynamicnature of learning we proposed some new sets of algorithmsusing heuristics in our effort to support web learners moreadequately than traditional search engines. We also realizedthat the information required to discriminate web content

493

for learning purpose goes beyond hyper-links and traditionalmetadata keywords. Consequently, we proposed an infor-mation model in which contents are qualified regarding thecompetencies they require and the competencies they pro-vide. As latent factors, competencies are difficult to recog-nize even by knowledge domain experts. For this purpose wepresented our results on automated competency extractionproviding methods to name them and evaluate competencyframeworks predictive quality. Among future developments,we envision the extension of our work on competency frame-works to multi-relational structures including the two typesof competencies defined in our information model (compe-tencies gain and required) while taking care as discussed inSection 4, of the scalability of proposed methods.

6. ACKNOWLEDGMENTSThis work is part of the National Research Council Canada

program Learning and Performance Support Systems (LPSS),which addresses training, development and performance sup-port in all industry sectors, including education, oil and gas,policing, military and medical devices.

7. REFERENCES[1] D. Applegate, R. Bixby, V. Chvatal, and W. Cook. On

the solution of traveling salesman problems. In Int.Congress of Mathematicians, 1998.

[2] T. Atapattu, K. Falkner, and N. Falkner. Acquisitionof triples of knowledge from lecture notes: A naturallanguage processing approach. In 7th Intl. Conf. onEducational Data Mining, pages 193–196, 2014.

[3] B. Beheshti and M. C. Desmarais. Predictiveperformance of prevailing approaches to skillsassessment techniques: Insights from real vs. syntheticdata sets. In 7th Intl. Conf. on Educational DataMining, 2014.

[4] N. Belacel, G. Durand, and F. Laplante. A binaryinteger programming model for global optimization oflearning path discovery. In Proceedings of theWorkshops held at the 7th International Conferenceon Educational Data Mining (EDM 2014), London,UK, 2014.

[5] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latentdirichlet allocation. The Journal of Machine LearningResearch, 3:993–1022, Mar. 2003.

[6] Y. Chen, J. Liu, G. Xu, and Z. Ying. Statisticalanalysis of Q-matrix based diagnostic classificationmodels. Journal of the American StatisticalAssociation, 110(510), 2015.

[7] M. Desmarais. Mapping questions items to skills withnon-negative matrix factorization.ACM-KDD-Explorations, 13(2):30–36, 2011.

[8] M. Desmarais, B. Beheshti, and P. Xu. The refinementof a Q-matrix: Assessing methods to validate tasks toskills mapping. In 7th Intl. Conf. on Educational DataMining, pages 308–311, 2014.

[9] M. Desmarais and R. Naceur. A matrix factorizationmethod for mapping items to skills and for enhancingexpert-based Q-matrices. In H. C. Lane, K. Yacef,J. Mostow, and P. Pavlik, editors, ArtificialIntelligence in Education, volume 7926 of LectureNotes in Computer Science, pages 441–450. SpringerBerlin Heidelberg, 2013.

[10] G. Durand, N. Belacel, and C. Goutte. Evaluation ofexpert-based Q-matrices predictive quality in matrixfactorization models. In Design for Teaching andLearning in a Networked World, EC-TEL 2015conference, pages 56–69. Springer, 2015.

[11] G. Durand, N. Belacel, and F. Laplante. Graph theorybased model for learning path recommendation.Information Sciences, 251(10), 2013.

[12] G. Durand, F. Laplante, and R. Kop. A learningdesign recommendation system based on Markovdecision processes. In 17th ACM Conference onKnowledge Discovery and Data Mining (SIGKDD)Workshop on Knowledge Discovery in EducationalData, 2011.

[13] D. Godoy and L. Amandi. Link recommendation ine-learning systems based on content-based studentprofiles. In C. Romero, S. Ventura, M. Pechenizkiy,and R. Baker, editors, Handbook of Educational DataMining. CRC Press, 2010.

[14] C. Goutte, S. Léger, and G. Durand. A probabilisticmodel for knowledge component naming. InProceedings of the 8th International Conference onEducational Data Mining, 2015.

[15] P. Hubwieser and A. Mühling. Competency mining inlarge data sets - preparing large scale investigations incomputer science education. In Proceedings of theInternational Conference on Knowledge Discovery andInformation Retrieval (KDIR 2014), pages 315–322,2014.

[16] K. Koedinger, R. Baker, K. Cunningham,A. Skogsholm, B. Leber, and J. Stamper. A datarepository for the EDM community: The PSLCdatashop. In C. Romero, S. Ventura, M. Pechenizkiy,and R. Baker, editors, Handbook of Educational DataMining. CRC Press, 2010.

[17] A. S. Lan, A. E. Waters, C. Studer, and R. G.Baraniuk. Sparse factor analysis for learning andcontent analytics. Journal of Machine LearningResearch, 15:1959–2008, June 2014.

[18] J. Liu, G. Xu, and Z. Ying. Data-driven learning ofQ-matrix. Applied Psychological Measurement,36(7):548–564, 2012.

[19] Y. Sun, S. Ye, S. Inoue, and Y. Sun. Alternatingrecursive method for Q-matrix learning. In 7th Intl.Conf. on Educational Data Mining, pages 14–20, 2014.

[20] T. Tang and G. McCalla. Data mining for contextualeducational recommendation and evaluationstrategies. In C. Romero, S. Ventura, M. Pechenizkiy,and R. Baker, editors, Handbook of Educational DataMining. CRC Press, 2010.

[21] K. K. Tatsuoka. Analysis of errors in fraction additionand subtraction problems. Final report,Computer-based Education Research Laboratory,University of Illinois at Urbana-Champaign, 1984.

[22] C. Ullrich and E. Melis. Complex course generationadapted to pedagogical scenarios and its evaluation.Educational Technology & Society, 13(2):102–115,2010.

[23] D. A. Wiley. Connecting Learning Objects toInstructional Design Theory: A Definition, aMetaphor, and a Taxonomy. The Instructional Use ofLearning Objects. D. A. WILEY (Ed.), 2002.

494