Invited Talk 2007 International Conference on Case-Based Reasoning 13 August 2007

Addressing Perceptions of Case-Based Reasoning Addressing Perceptions of Case-Based Reasoning David W. Aha David W. Aha ICCBR-07 ICCBR-07 13 August 2007 13 August 2007 Belfast, Northern IrelandBelfast, Northern Ireland 1

1. Perceptions 2. Objectives 3. Survey 4. Findings 5. Interpretation

Invited Talk2007 International Conference on Case-Based Reasoning

13 August 2007Belfast, Northern Ireland

David W. AhaHead, Adaptive Systems Section

Navy Center for Applied Research in AINaval Research Laboratory, Code 5514

Washington, [email protected]

Addressing Perceptions of Case-Based Reasoning



Goals of this presentation

1. Raise awareness on how to assess CBR R&D methods

2. Assess CBR R&D methods we’re publishing

3. Relate CBR’s R&D methods to those used in AI

4. Beg for your forgiveness?



Outline

1.Perceptions2.Objectives3.Survey4.Findings5.Interpretation



Outline

1.Perceptions• Story: Gnats, envy, & self-doubt• Quest

2.Objectives3.Survey4.Findings5.Interpretation



What perceptions of case-based reasoning (CBR) exist?

• Among active CBR researchers/practitioners• Among others



My perception

Case-Based Reasoning

Artificial Intelligence

……



Gnats, envy, & self-doubt

Gnat UK Gnat

Observation: CBR perceived differently by others



Gnats

In CBR (Pal & Shiu, 2004; Kolodner, 1993) expertise is embodied in a library of past cases… <long, accurate description of CBR> The major problem with CBR is that it lacks a sound theoretical framework for its application and has only achieved limited success. - Anonymous senior AI researcher/proposer, 2005

“Case based reasoning is often limited to surface features that may not be relevant to the operational military situation. (There is a need for deeper underlying reasoning, including analogical reasoning.)” - Anonymous ONR Program Manager, 2007



Gnats

ML: Prevailing view is CBR = Instance-based learning ML• No (e.g., 61% of papers in ECCBR-06 not related to ML)• Yet there is a relationship

–e.g., “CBR is a technique within the field of machine learning…” (Beltrán-Ferruz et al., ECCBR-06)

Artificial Intelligence: CBR not taught?• AIMA (Russell & Norvig, 2002-)

– 90% market share (1000+ universities, 91 countries)• CBR: Not discussed• IBL (3 pages) Statistical_Learning



Gnats

Computational analogy– “CBR systems…tend to use only minimal first-principles

reasoning…[and] rely on feature-based descriptions…[or] use domain-specific and task-specific similarity metrics. This can be fine for a specific application, but being able to exploit similarity computations that are more like what people do could make such systems…more understandable to their human partners.” (Forbus et al., IAAI-02)

Episodic memory– “Episodic memory can be thought of as the mother-of-all CBR

problems – how to store and retrieve cases about everything relevant in an entity’s existence. Most CBR research has avoided these issues.” (Nuxoll & Laird, ICCM-04)

Why isn’t some AI-related CBR research published at IC/ECCBR?



Gnats

Some Stereotypical perceptions of CBRUSA funding agencies Of questionable merit (?)

Cognitive Architectures Informal/incomplete models of episodic memory

Cognitive Psychology Cognitively implausible exemplar models

Artificial Intelligence A subfield, once dominated by speculative evaluation methodologies (Hall & Kibler, AIM 1985)

Machine Learning Case-based algorithms for supervised learning

Statistics A target?

Knowledge Management A panacea (still true?)

Business A mysterious technique whose name is rarely mentioned by its practitioners (still true?)

Us A discipline worthy of research & application

How could any misperceptions be addressed?How could any misperceptions be addressed?



Not interested in giving you yet another content survey

# Grouping Title (CBR = "Case-Based Reasoning") Authors1 CBR commentaries: Introduction Aha, Marling, & Watson2 CBR foundations Richter & Aamodt3 Representation in CBR Bergmann, Kolodner, & Plaza4 Retrieval, reuse, revision, and retention in CBR López de Mántaras et al. (13 authors)5 Integrations Marling, Rissland, & Aamodt6 Advances in conversational CBR Aha, McSherry, & Yang7 Textual CBR Weber, Ashley, & Brüninghaus8 Distributed CBR Plaza & McGinty9 Soft CBR Cheetham, Shiu, & Weber

10 Design, innovation, and CBR Goel & Craw11 CBR for diagnosis applications Goker, Howlett, & Price12 Case-based planning Cox, Muñoz-Avila, & Bergmann13 Medical applications in CBR Holt, Bichindaritz, Schmidt, & Perner14 CBR and law Rissland, Ashley, & Branting15 CBR-inspired approaches to education Kolodner, Cox, & González-Calero16 Knowledge management in CBR Althoff & Weber17 Image processing in CBR Perner, Holt, & Richter18 Case-based recommender systems Bridge, Goker, McGinty, & Smyth19 Fielded applications of CBR Cheetham & Watson20 Emergent CBR applications López de Mántaras, Perner, & Cunningham

Introduction

Techniques

Applications

Topic Areas

Task Areas

We We havehave existing surveys of CBR (e.g., existing surveys of CBR (e.g., KERKER 2005 special issue) 2005 special issue)



Envy

Am I (unnecessarily) wishing for something?Am I (unnecessarily) wishing for something?

Formal foundations envy?•e.g., Bayesian, first-order logic, decision theory, COLT, …•But we have this:

– e.g., Cover & Hart, 1967; Richter, FLAIRS-07; Richter & Aamodt, 2005 KER– And we’re the ultimate chameleons, even within AI

Methodological approach envy?•e.g., Experimental study of ML (Langley & Kibler, 1991), Crafting papers on ML (Langley, ICML-00), …

•Possibly: – We haven’t had received much proselytizing on this…yet– My awareness of these issues has increased; worth reviewing



EnvyCrafting papers (e.g., on ML) (Langley, ICML-00)•Content•Evaluation strategy•Communication

Paper content recommendations

•State the research goals and evaluation criteria•Specify the component (e.g., learning) & overall perf. task•Describe rep’n and organization of knowledge & data•Explain the system components (if any)•Evaluate the approach

–Empirical, theoretical, psychological, novel functionality•Describe related work

–Explain similarities/differences with your work•State the limitations

–Propose solutions



Self-Doubt

Summary questions

• How should (royal) we respond to possible misperceptions of CBR?– i.e., Other than to survey the field’s contents and its foundations

• Why are some folks ignoring CBR?• How can we attract them? • Does this concern our research methodologies (and/or their communication) rather than our research focus?

ProposalProposal: Examine our research methodologies: Examine our research methodologies

Realization: Realization: This requires a framework for investigationThis requires a framework for investigation



Quest: Identify, characterize, & compare CBR research methods

Don Quixote (Scott Gustafson)



Outline

1.Perceptions2.Objectives

• Questions• Conjectures/Hypotheses

3.Survey4.Findings5.Interpretation



Questions

1. How should we describe CBR to others?• i.e., in the context of AI

2. What R&D methodologies are we using? 3. Does CBR R&D differ from AI R&D?



Conjectures/Hypotheses

AI research is dominated by two methodologies (Cohen, 1991)• Model-centered (neat) (i.e., proving theorems on formal models)• System-centered (scruffy)

1. CBR research is not (currently) dominated by both• Dominated only by system-centered papers, which often lack models

for deriving claims, generating predictions, and explaining behavior

2. CBR research suffers from similar methodological problems• Model- and system-centered papers differ in whether they conduct

evaluations, assess performance, and describe expectations

3. The designation of CBR conference publications are distinguished by their research methodologies• Oral vs. poster presentations• Best paper nominees from others



Outline

1.Perceptions2.Objectives3.Survey

• Case base• Retrieval• Reuse• Revision

4.Findings5.Interpretation



Case Base

Case #1

Cohen, Paul R. (1991). A Survey of the Eighth National Conference on Artificial Intelligence: Pulling together or pulling apart? AI Magazine, 12(1), 16-41.

Paul R. Cohen (circa ~1991)

Frameworks for assessing AI R&D Methods

Summary of (Cohen, 1991)

• Conclusion: AI research follows two incomplete, complementary methodologies• Proposes: MAD (Modelling, Analysis, & Design) mixed methodology

RecommendationRecommendation: Make this required reading for AI researchers: Make this required reading for AI researchers

Read 150Papers!(can you imagine?)

Paul R. Cohen(circa ~2007)



(Cohen, 1991): 40 citations (Google Scholar, 8/1/07)

The importance of this link has been highlighted by several researchers, some even going so far as to state that AI will not advance as a science until the gap between those who construct models and those who build systems is closed. (Jennings, 1995 Artificial Intelligence).

There are two ways in which the fields proceed. One is through the development and synthesis of models of aspects of perception, intelligence, or action, and the other is through the construction of demonstration systems (Brooks, 1991 Science).

As Cohen (1991) demonstrated in his analysis of the papers presented at AAAI90, we are, as a discipline, just learning how to perform real, systematic experimentation. One hears a lot of talk about AI as an experimental science, but typically the “experiments" amount merely to writing a computer program that is supposed to validate some hypothesis by its very existence. (Pollock, 1992 Artificial Intelligence)

As Cohen (1991) points out, most research papers in AI, or at least at an AAAI conference, exploit benchmark problems; yet few of them relate the benchmarks to target tasks. (Howe & Dahlman, 2002 JAIR)

Cohen (1991) discovered that only 43% of the papers that described implemented systems report any kind of analysis of their contributions. Even of the papers that do describe evaluatory experiments, very few go beyond evaluating the programs to analyzing the scientific claims that the programs were written to demonstrate . (Ram & Jones, 1995 Philosophical Psychology)

Many of our suggestions are similar to the excellent points made by Cohen (1991) in his discussion of AI, but they seem worth instantiating for the field of machine learning (Langley & Kibler, 1991 “Experimental Study of ML”)

Methodological issues are by no means resolved (Cohen, 1991), but they are much discussed and a consensus is emerging on the importance of combining theoretical and empirical investigations. (Bundy, 1998 book)



Retrieval

My query: • Identify R&D methodologies being used in CBR•Compare results with general AI and other AI subfields

Case #1: (Cohen, 1991)•Develop and apply framework for analyzing AI R&D methodologies• Identify R&D methodologies being used•Propose novel R&D methodology (MAD)

Case #1

Cohen, Paul R. (1991). A Survey of the Eighth National Conference on Artificial Intelligence: Pulling together or pulling apart? AI Magazine, 12(1), 16-41.

Framework for assessing AI R&D Methods

1. Retrieve

Case #1

My Query



MAD Framework (Cohen, 1991)

Purpose

3. Models (Define, extend, generalize, provide semantics)

4. Theorems/Proofs for models

5. Present algorithm(s)

6. Analyze algorithm(s) Complexity Formal Informal

7. Present system

8. Analyze aspects of system Complexity Formal Informal

Argument

9. Example task Natural Synthetic Abstract

10. Task type Natural Synthetic Abstract

11. Task environment Embedded Not embedded

12. Assess performance

13. Assess coverage

14. Comparison

15. Predictions, hypotheses

16. Probe results

17. Unexpected results

18. Negative results

Note: It does not Note: It does not (completely) eliminate (completely) eliminate

subjective assessments!subjective assessments!



MAD Framework: Purpose fields

Field Description3. Models (Define, extend,

generalize, provide semantics)• Abstract, typically formal description of behavior and/or

environmental factors that affect behavior.• Purpose of building a model is to analyze its properties

4. Theorems/Proofs for models • e.g., Complexity, soundness, completeness, decidability


6. Analyze algorithm(s) • e.g., Complexity, soundness, completeness, decidability

7. Present system • Describes components, control flow

8. Analyze aspects of system



MAD Framework: Argument fields

Field Description9. Example task • Natural, synthetic, or abstract

10. Task type • Used iff multiple trials are described• Natural, synthetic, or abstract

11. Task environment • Used iff multiple trials are described• Embedded or not embedded (e.g., in other s/w, env’t)

12. Assess performance • Weak criterion: One perf. measure over many examples• e.g., a bakeoff is an assessment

13. Assess coverage • Solve instances of some problems in a defined problem space • Not a demo on superficially different problems w/o justification

14. Comparison • Goal: Study relative strengths/limitations of multiple techniques• e.g., a bakeoff is not a comparison

15. Predictions, hypotheses • Indicate reason to implement/test an idea • Not “My algorithm solves this problem”, or simple perf. demos• Many papers are vague as to why empirical work is described

16. Probe results • Go beyond central results (e.g.,follow-up expt’s, explanations)

17. Unexpected results • Infrequent

18. Negative results • Rare (if ever)



Reuse

1. Retrieve

MAD Framework

(Cohen, 1991)

Case Base

2. Reuse

MAD Framework

My Query

Analyze

Metrics

Results

Hypotheses

MAD Methodology

AAAI-90 Data

AAAI-90

ECCBR-06

ECCBR-06 Data

Analyze

AdaptedHypotheses

Results

ICCBR Audience(at lunch)



ECCBR-06: 36 Papers # First Author Title

1 Gabel Multi-agent CBR for cooperative RLs2 Ros Retrieving and reusing game plays for robot soccer3 Watson Self-organizing hierarchical retrieval in a case-agent system4 Karoui COBRAS; Cooperative CBR system for bilbliographic reference recommendation5 McDonnell A knowledge-light approach to regression using CBR6 Weber CBM for CCBR-based process evolution7 Gu Evaluating CBR sytems using different data sources: A case study8 Nicholson Decision diagrams: Fast and flexible support for case retrieval and recommendation9 Hervas CBR for knowledge-intensive template selection during text generation

10 Gupta Rough set feature selection algorithms for textual case-based classification11 Minor Experience management with case-based assistant systems12 McCarthy The needs of the many: A case-based group recommender system13 Kofod-Petersen Contextualised ambient intelligence through CBR14 Recio-Garcia Improving annotation in the semantic web and case authoring in textual CBR15 Herrera Unsupervised case memory organization: Analysing computational time and soft computing capabilities16 Freyne Further expeirments in case-based collaborative web search17 Bergmann Finding similar deductive consequences: A new search-based framework for unified reasoning from cases and general knowledge18 Baccigalupo Case-based sequential ordering of songs for playlist recommendation19 Perner A comparative study of catalogue-based classification20 Gomez-Gauchia Ontology-driven development of conversational CBR systems21 Massie Complexity profiling for informed case-base editing22 Wiratunga Unsupervised feature selection for text data23 Stahl Combining case-based and similarity-based product recommendation24 Coyle On the use of selective ensembles for relevance classification in case-based web search25 Bogaerts What evaluation criteria are right for CCBR? Considering rank quality26 Chakraborti Fast case retrieval nets for textual data27 Lamontagne Combining multiple similarity metrics using a multicriteria approach28 Althoff Case factory: Maintaining experience to learn29 Beltran-Ferruz Retrieval over conceptual structures30 Kuchibatla An analysis on transformational analogy: General framework and complexity31 Funk Discovering knowledge about key sequences for indexing time series cases in medical applications32 Montani CBR for autonomous service failure diagnosis and remediation in software systems33 Mendez Tracking concept drift at feature selection stage in SpamHunting: An anti-spam instance-based reasoning system34 Bergmann Case-based support for collaborative business35 Hefke A CBR-based approach for supporting consulting agencies in successfully accompanying a customer's introduction of KM36 Goker The PwC connection machine: An adaptive expertise provider



Outline

1.Perceptions2.Objectives3.Survey4.Findings

• Results• Analysis & Patterns• Followup

5.Interpretation



Findings: Results1 4 5 7

Complexity Formal Informal Complexity Formal Informal Natural Synthetic Abstract Natural

1 1 1 1 1 12 1 1 1 1 1 13 1 1 14 1 15 1 1 16 1 1 1 1 17 1 1 18 1 1 1 1 19 1 1 1

10 1 1 1 1 111 1 1 112 1 1 1 113 1 1 114 1 1 115 1 1 116 1 1 1 117 1 1 1 118 1 1 119 1 120 1 121 1 1 122 1 1 123 1 124 1 1 125 126 1 1 1 1 127 1 1 1 128 1 129 1 130 1 1 1 131 1 132 1 133 1 1 1 134 1 135 1 136 1 1

3

Present Algorithms

Present system

Example type Task Evaluation typeAnalyze Algorithms Analyze aspect(s) of system

#

Define, Extend, Generalize,

Differentiate, Semantics for Formal Models

Theorems and proofs re: Model

Informal Model to frame the research

1096 8



Findings: ExamplesField # Example

Pu

rpo

se

3. Models (Define, extend, etc.) 9 Similarity/deductive reasoning (Bergmann & Mougouie)

4. Theorems/Proofs for models 0

5. Present algorithm(s) 21 Retrieve k cases from a DD (Nicholson et al.)

6. Analyze algorithm(s) 7 Unsupervised algs. (Fornells Herera et al.)

7. Present system 21 PwC Connection Machine (Göker et al.)

8. Analyze aspects of system 7 Fast CRNs (Chakraborti et al.)

Arg

um

ent

9. Example task 22 Song playlists (Baccigalupo & Plaza)

10. Task type 26 European skiing holidays (McCarthy et al.)

11. Task environment 4 Game plays for robot soccer (Ros et al.)

12. Assess performance 27 Integrated CCBR evaluation (Gu & Aamodt)

13. Assess coverage 1 Complexity profiling (Massie et al.)

14. Comparison 15 SpamHunting (Méndez Reboredo et al.)

15. Predictions, hypotheses 19 Rough set feature selection (Gupta et al.)

16. Probe results 7 Ave. diversity of decision trees (Coyle & Smyth)

17. Unexpected results 6 Pima’s precision results (Bogaerts & Leake)

18. Negative results 0



Categorizing Papers: AAAI-90 (Cohen, 1991)

Papers

Models(3 4)

Algs(5 6)

Systems(7 8)

Field3. Models (Define, extend, etc.)



6. Analyze algorithm(s)

7. Present system


Models Algs

Systems

25 43 36

14

3

37

Categories

Model-Centered: (M A) S Hybrid: (M A) SSystem-Centered: (M A) S

AAAI-90



Categorizing Papers: ECCBR-06

Papers

Models(3 4)

Algs(5 6)

Systems(7 8)

Field3. Models (Define, extend, etc.)



6. Analyze algorithm(s)

7. Present system


Models Algs

Systems

1 5 7

03

6

13

Categories

Model-Centered: (M A) S Hybrid: (M A) SSystem-Centered: (M A) S

ECCBR-06



Comparing MAD Categorizations of Papers

Models Algs

Systems

1 5 7

03

6

13

ECCBR-06

Models Algs

Systems

25 43 36

14

3

37

AAAI-90

Models Algs

Systems

3% 14% 19%

0%8%

17%

36%

Models Algs

Systems

17% 29% 24%

1%3%

2%

25%

25% M50% M

25% Hybrid6% Hybrid



Results: ECCBR-06

System-CenteredM M+A A M+S M+S+A S+A S Totals

Distribution by fields 3-8 1 5 7 0 3 6 13 35A Natural 1 2 3 0 1 2 7 16A Synthetic 0 0 1 0 0 1 1 3A Abstract 0 2 0 0 0 1 0 3A None 0 1 3 0 2 2 5 13B Natural 0 4 6 0 2 4 6 22B Synthetic 0 0 1 0 0 1 0 2B Abstract 0 1 0 0 0 0 0 1B None 1 0 0 0 1 1 7 10C Embedded 0 0 1 0 2 1 0 4C Not embedded 0 5 6 0 0 3 6 20C None 1 0 0 0 1 2 7 11D Demo 1 5 7 0 2 5 6 26D No demo 0 0 0 0 1 1 7 9E Expectations 0 4 4 0 1 4 6 19E No expectations 1 1 3 0 2 2 7 16

Field 11: Embedded TaskFields 12-14: DemoFields 15-18: Followup

Model-Centered Hybrid

Field 9: Example

Field 10: Eval Task

LegendA=AlgorithmsM=ModelsS=Systems

Contingency table



Results: ECCBR-06

M-C H S-CDistribution by fields 3-8 13 9 13

A Natural 6 3 7A Synthetic 1 1 1A Abstract 2 1 0A None 4 4 5B Natural 10 6 6B Synthetic 1 1 0B Abstract 1 0 0B None 1 2 7C Embedded 1 3 0C Not embedded 11 3 6C None 1 3 7D Demo 13 7 6D No demo 0 2 7E Expectations 8 5 6E No expectations 5 4 7

Field 11: Embedded TaskFields 12-14: DemoFields 15-18: Followup

Field 9: Example

Field 10: Eval Task

Legend

M-C: Model-Centered

H: Hybrid

S-C: System-Centered



Analysis: Comparing ECCBR-06 with AAAI-90

Source Model-Centered Hybrid System-CenteredECCBR-06 13 9 13AAAI-90 104 8 37

LegendA=AlgorithmsM=ModelsS=Systems

χ2(6)=24.1, p<0.006

χ2(2)=19.0, p<0.0001

Could this distribution of M-C, Hybrid, and S-C methodologies have arisen by chance, or does it reflect a real difference between ECCBR and AAAI?

• The ECCBR/AAAI distinction is not independent of the research methodology class

System-CenteredSource M M+A A M+S M+S+A S+A S

ECCBR-06 1 5 7 0 3 6 13AAAI-90 25 43 36 1 4 3 37




Analysis: Examining ECCBR-06

M-C H S-CDistribution by fields 3-8 13 9 13Any Example 9 5 8No Example 4 4 5

M-C H S-CDistribution by fields 3-8 13 9 13

Any evaluation 12 7 6No evaluation 1 2 7

χ2(2)=0.4, p>0.8

Unlike AAAI-90, the methodological choice of an example is independent of the paper’s class.

χ2(2)=7.0, p<0.003

The methodological choice of whether an evaluation was conducted is not independent of the paper’s class.

• Model-centered and hybrid papers include evaluations significantly more frequently than do system-centered papers.



Analysis: Examining ECCBR-06 (cont.)

M-C H S-CDistribution by fields 3-8 13 9 13Demonstration 13 7 6No demonstration 0 2 7

M-C H S-CDistribution by fields 3-8 13 9 13Expectations 8 5 6No expectations 5 4 7

χ2(2)=9.9, p<0.007

Like AAAI-90, Model-centered and hybrid papers are more likely than system-centered papers to include (any type of) performance assessment.

χ2(2)=0.6, p>0.7

Surprisingly, and unlike AAAI-90, model-centered and hybrid papers do not provide (any types of) expectations more frequently than do system-centered papers.

• Perhaps this warrants a follow-up analysis • Perhaps system-centered researchers make predictions not

derived from models, which would be dangerous, or perhaps they are simply not stating the models, which is more likely.



Patterns: Comparing ECCBR-06 with AAAI-90

PatternObserved frequencies

AAAI-90 ECCBR-06

p(M) 0.49 0.25

p(S) 0.30 0.61

p(SM) 0.03 0.08

p(SM) 0.89 0.86

p(No or abstract examples | M-C) 0.76 0.46

p(Test implementations | M-C) 0.33 0.92

p(Prediction/hypothesis) <0.21 0.53

p(Evaluation) 0.30 0.72

p(prediction/hypothesis | evaluation) 0.69

p(Negative, surprising, or probe) 0.16 0.25

Generous?

Generous!

Näive



ECCBR-06: Distinguishing Papers from Posters

““Now we tread on hallowed groundNow we tread on hallowed ground” - Anon” - Anon

Hypothesis: Reviewers are human and subjective. While there’s probably a trend that oral presentations show more “maturity” than do posters, exceptions exist and this trend is probably not significant.

Results of analysis: I was wrong…• …assuming the presentation/use of models is

indicative of a paper’s level of maturity



ECCBR-06

Models Algs

Systems

0 0 2

01

5

5

Posters (13)

Models Algs

Systems

1 5 5

02

1

8

Oral Papers (22)

Models Algs

Systems

0% 0% 15%

0%8%

38%

38%

Models Algs

Systems

5% 23% 23%

09%

5%

36%

15% M50% M

46% Hybrid14% Hybrid



ECCBR-06: Distinguishing Papers from Posters

System-CenteredM M+A A M+S M+S+A S+A S

Oral 1 5 5 0 2 1 8Poster 0 0 2 0 1 5 5


2(5)=9.3, p<0.1

The poster/paper designation of an accepted paper at ECCBR-06 was not independent of the paper’s class.

• Tentative conclusion: If you want your accepted paper to be an oral presentation, then present your work in the context of a model.

Distribution by fields 3-8 M-C H S-COral 11 3 8Poster 2 6 5

2(2)=6.0, p<0.05

SoSo: Will you think about this, and want to learn more? : Will you think about this, and want to learn more?

But maybe you are unconvinced…But maybe you are unconvinced…



ECCBR-06: Distinguishing Papers from PostersOral Poster

Example 14 8None 8 5

Oral PosterEval 16 9None 6 4

Oral PosterDemo 17 9None 5 4

Oral PosterDemo 11 8None 11 5

Nothing else (so far) distinguishes papers from posters

2(1)=0.02, p>0.9

2(1)=0.05, p>0.8

2(1)=0.28, p>0.5

2(1)=0.44, p>0.5



ECCBR-06: Distinguishing Best Paper Nominees?

• But there were only 5• Future work: Analyze after adding 9 ICCBR-07 nominees



Hypotheses revisited

Summary

1. Unlike AAAI-90, CBR research is not dominated by both model-centered and system-centered methodologies Dominated only by system-centered papers

2. CBR research suffers from similar methodological problems• Model- and system-centered papers differ in whether they:

Conduct evaluations Assess performance Describe expectations

3. The class of a paper in the MAD framework distinguishes Oral vs. poster presentations Best paper nominees from others



Outline

1.Perceptions2.Objectives3.Survey4.Findings5.Interpretation

• A new case• Caveats• Next steps



A new case…hopefully

1. Retrieve

MAD Framework

(Cohen, 1991)

Case Base

2. Reuse

MAD Framework

How can we assess CBR R&D Methodologies?

AAAI-90

ECCBR-06

Frameworks for assessing AI R&D Methods

MAD mixed methodology

Today’s Results

3. Revise4. Retain

(Aha, 2007?)



Many of Cohen’s (1991) Points for AI Apply to CBR

• A goal of AI research is to develop science & technology to support the design and analysis of intelligent systems

• Model- and system-centered methods are complementary– Model-centered researchers typically develop algorithms for simpler

problems, but with deeper analysis, expectations, and demos– System-centered researchers typically build large systems to solve

realistic problems, but w/o explicit expectations, analyses or demos– See the MAD methodology (Cohen, 1991)

• Models are used to derive hypotheses & expectations• Few systems merit attention on the basis of existence alone• It is impossible to evaluate a system without predictions• Creating benchmarks will not fix AI’s methodological problems



Caveat #1: No cases for ECCBR-90, AAAI-06, etc.

AAAI-90 ECCBR-06

?!

Note: CBRW-91’s R&D methods differ greatly from ECCBR-06’s



Caveats (that could influence the results)

• The case base is small to compare CBR with AI– AAAI has changed since 1990! Compare to AAAI-07 (and IAAI-07!)– ECCBR’s differences may reflect a higher acceptance rate– No analysis of other subfields; how does CBR relate?

• No reliability data: Subjective classification of papers!– e.g., I gave up distinguishing “informal” and “no” system analysis– “Volunteers welcome!” ((Cohen, 1991), which I repeat)

• Not representative of CBR community’s work?– Is ECCBR-06 an aberration? (Wait until next year?)– Perhaps we publish model-centered work elsewhere (e.g., COLT)– ECCBR readers’ expectations match ECCBR-06’s class distribution?

• Lack of page-space places limitations on what is presented– e.g., hypotheses not made explicit

• Science/research works iteratively– Earlier exploratory research (e.g., involving surprising results) resulted in

changes to the model, algorithm, or system; we see only the end result



Caveat finale: I’ve ignored many issues!

• Relation of evaluation to:– Investigating the claims, if any– The predictions, if any

• Results of formal analysis – e.g., average- vs. worst-case

• Quality of the empirical evaluation (e.g., scale)• Significance of evaluation’s results• …



Other potential applications of the MAD framework

Next Steps

• A decision aid for predicting whether an accepted paper should be categorized as an oral presentation– Or to ensure diversity among the presentations

• Selection of best paper nominees• To assist reviewers with spotting novelty and/or

expected characteristics in a submission• Explaining/characterizing CBR methodologies to others



Next Steps

Conjecture: CBR has shifted from heavily model-centered to heavily system-centered research

– Needs analysis to provide evidence, but it’s obvious– AI needs both to achieve balance– We should consider this in our reviewing processes

Milestones: We are halfway to our 25th anniversary– A time for reflection– We should make it our goal that, by the 25th, we will

achieve a better balance

Roger Schank

David Leake

Agnar Aamodt(1st ICCBR Co-Chair)



Next Steps

• Refine our research methodologies– Ensure program PCs reflects them broadly – A CBR-related journal could assist (by providing feedback)

• Change perceptions by improving communication– WWW site (maybe AAAI, but probably not, as we have

different specifications in mind)– Co-locate our conferences with others

• e.g., IJCAI, ECAI, IR conference

• Reach out in new ways (e.g., Video)



Next Steps

AAAI-07 AI Video Competition (Co-Chairs: Thrun & Aha)• Goal: Encourage the interest of prospective students

• Results (see aivideo.org)– Quick funding– 30 Submissions (in a short time period)– Large turnout for awards ceremony– Invited, and will be held, for AAAI-08

• Two CBR videos1. k-nearest Neighbor Classification (Antal van den Bosch)2. Invisible Threats (Rosina Weber & André Testa)



Takeaway Message

And now: The 25 Re’s!!!• e.g., Reuse, Regurgitate, Repulse, …• Beats previous record of 13 by 12 (Bridge, ICCBR-05)

Just kidding – I’ll spare you

Bridge Gauntlet: 13 Re’s!

Derek Bridge



Concluding remarks

1. Goal: Raise awareness of CBR R&D methodologies

4. MAD framework has several potential uses

†This presentation is dedicated to my late colleague John Urban and the late great Donald Michie, for their support.

5. We have work to do to address perceptions

3. Paper/poster distinction relates to use of models

2. Current CBR methods are unlike traditional AI’s• CBR’s is not dominated by model-centered work• We must beware system-centered limitations• But there’s much more to learn

Thanks for listening

Date post:	10-Jan-2016
Category:	Documents
Upload:	sydnee
View:	26 times
Download:	3 times

Invited Talk 2007 International Conference on Case-Based Reasoning 13 August 2007

Documents