Eliciting Formal Models From Informal Requirements

transcript

Sep 2003, Chicago,1

Some issues and an approach.

Nikhil Dinesh, David E. Arney, Aravind K. Joshi,

Owen Rambow, Martha Palmer and Insup Lee

University of Pennsylvania

Chicago, September 24 2003

Sep 2003, Chicago,2

Outline

• Introduction• Example• Specification Language for Requirements• The approach• Future work

Sep 2003, Chicago,3

Issues

• What is the right specification language for requirements ?– Natural Language– Formal Specification Language– Formal Specification Language with an NL-

looking restricted language interface• Evaluation

– What metrics can be applied in the evaluation of such a system

Sep 2003, Chicago,4

Overall ApproachNL requirements

Extended Finite State Machine

Verification& Validation

Errors

Corrections to requirementsin NL

NL-based Finite State Machine

RequirementsEngineer

PolicySpecifier

Sep 2003, Chicago,5

Approach to elicit an EFSM*

• Work is preliminary, no implementation• Outline the stages in eliciting an EFSM

– Desired output representation at each stage– How to compute from the NL requirements in

terms of :• What is available• What is needed

– What is achievable with existing tools and what research is needed for the long range

*An Extended Finite State Machine is an FSM with variables. (as in the HASTEN project)

Sep 2003, Chicago,6

Outline

• Introduction• Example• Specification Language for Requirements• The approach• Future work

Sep 2003, Chicago,7

Example from FDA CFR* 610.40

(a) Except as specified in paragraphs (c) and (d) of this section, you must test all samples for evidence of infection due to the following communicable disease agents:

(i) Human immunodeficiency virus

(ii) Hepatitis-B virus

* A policy document from Food and Drug Administration, Code of Federal Regulations. There are several volumes each of which is updated once each calendar year and issued on a quarterly basis.

Sep 2003, Chicago,8

Example from FDA CFR 610.40

(b) To test for evidence of infection due to communicable disease agents in paragraph (a), you must use a screening test approved by the FDA.

Sep 2003, Chicago,9

Example from FDA CFR 610.40

(b) To test for evidence of infection due to communicable disease agents in paragraph (a), you must use a screening test approved by the FDA. You must perform one or more such test, as necessary, to reduce adequately and appropriately the risk of transmission of communicable disease.

Sep 2003, Chicago,10

Outline

• Goals• Example• Specification Language for requirements• The approach• Future work

Specification Language for Requirements

• Natural Language (NL)– Specification is accessible to people– Properties should correspond to the requirements– Hard to compute properties

• Formal Specification Language (FSL)– Allows specification to be easily verified– Not easily accessible to people such as policy users

• Application should motivate the choice– What is the right choice for the policy domain ?

Policy

• Large number of policy documents in NL– Hand conversion to FSL is expensive and

error prone.• Human readability

– Policy is interpreted by humans.• Is this just an interface problem ?

Interfaces to Specification Languages

• Suppose policy requirements were in a formal specification language

• Interface provides NL-looking, restricted language

• How easily can it be read?• We examine one such interface

Specifying a Property• PROPEL – An approach supporting property

elucidation [Smith et al], 2002.• For our example,

– Core phrase : Exactly oneExactly one occurrence of arrival of a arrival of a samplesample eventually leads to one or moreone or more occurrences of test for diseasestest for diseases.

– Repetition Phrase : The above behavior is repeatable– Scope Phrase : This property must hold before the

first occurrence of P (where P is a state corresponding to “Exceptions in paragraphs (c) and (d)”)

How do properties relate to the NL document ?

• Core phrase : Exactly oneExactly one occurrence of arrival of a samplearrival of a sample eventually leads to one or moreone or more occurrences of test for diseasestest for diseases

• Repetition Phrase : The above behavior is repeatable

• Scope Phrase : This property must hold before the first occurrence of P

Exceptions in (c) and (d)

Test for disease

Sentence 1

One or more tests

Sentence 3

Repeatability

Arrival of a sample

• To enforce use of a screening test:– test for disease -> screening test for diseasescreening test for disease

• “test for disease” from Sentence 1• “screening” from Sentence 2

• Correspondence becomes harder• Interface starts to look like an FSL• Not easily accessible to policy users

Outline

NL Document

EFSM-likerepresentation

from NL document

Agree to perform test

Agree to use screening test

Do the test

Have you tested ?

Have you reduced risk?

NLFSM to EFSM

Clause a

Clause b

V(a) == true

NL Document

Clause Connective Dependency Structure

Temporal Ordering Of Clauses

Clauses and Connectives

(i) Human immunodeficiency Connective virus

Except

as specified in paragraphs (c) and (d) of this section

you must test all samples for evidence of infection due to the following communicable disease agents :

(i) Human immunodeficiency (ii) Hepatitis-B virus

ClausesExplicit Connective

Clauses and Connectives

(b) To test for evidence of communicable disease agents in paragraph (a), you must use a screening test approved by the FDA.

In order to

To test for evidence of communicable disease agents in (a)

You must use a screening test approved by the FDA

ClausesImplicit Connective

Predicates and Arguments

John ate the apple

Predicate: ate(Agent,Object)

(Agent)John

(Object)The apple

Dependency Structure

Arguments of Connectives

• Except– Exception– Default

• In order to– Action– Purpose

Arguments across sentences

• If X is true do action A. Otherwise do action B.

Otherwise

(Condition)X is true

(Action)Do action B

Computing the Dependency Structure

• Discourse Connectives vs Verbs– arity, Verbs (1 – 3), Connectives (2)– Locating the arguments, easier for verbs– Roles of the arguments, verb-specific

(Propbank), connective-specific (our approach)

• Chunking– [Except] [as specified in paragraphs (c) and (d) of this

section] [you must test all samples for evidence of infection due to the communicable disease agents: (1) Human immunodeficiency virus (2) Hepatitis B virus]

• Simple for the cases where there is only one connective in the sentence.

• You should test donation for evidence of all disease, except if it is a dedicated donation, you need not test for diseases (a)(5) and (a)(6).

• Three Generative, Lexicalized Models for Statistical Parsing, Michael Collins, Proceedings of the 35th Annual Meeting of the ACL, Madrid (1997)

• Statistical Parsing with an automatically-extracted tree adjoining grammar, David Chiang, Proceedings of the ACL, Hong Kong (2000)

• Discriminative Reranking for Natural Language Parsing, Michael Collins, ICML (2000)

• Resources available– Parsers compute rich dependency structure at

the sentence-level– Accuracy of approximately 90% (Collins 2000)– Efforts underway for parsers at the discourse-

level (DLTAG, Discourse Treebank)• Resources required

– Some annotation of requirements documents– Labeling of connective-specific roles

NL Document

Temporal Ordering of Clauses

• Related to programs• Cannot impose an order from an

operational perspective (there may be cycles)

• But can an order be imposed from a syntactic perspective – the way we write programs ?

Temporal Ordering of Clauses

while(i < 9){i = i + 1;

}i = i – 1;

i=i+1 i=i-1

True True

Operational FSM

while(i < 9)

i = i + 1 i = i - 1

True False

Temporal Tree

Computing the Temporal Ordering(Between Clauses in a Sentence)

Exception before the Default

Action before the check for Purpose

Computing the Temporal Ordering(Between Clauses in Different Sentences)

Agreement to use a screening test precedes agreement to testNoun-Verb “link” through the light verb (use)

Check that test has taken place should follow the testFinite Verb – Nonfinite Verb “link”

Computing the Temporal Ordering(Between Clauses in Different Sentences)

Computing the Temporal Ordering(Between Clauses in Different Sentences)“tests” and “screening tests” are related.

But the order is not quite clear from these in isolation. However these clauses are both

scoped by purpose clauses

reduce follows testDesirable to keep scopes

nested.

Computing the Temporal Ordering

• Resources available– Work on noun coreference (Morton)– New Machine Learning algorithms (Maximum

Entropy Models, Conditional Random Fields etc)

• Research needed– Granularity of annotation– Incorporating world knowledge

NL Document

From Temporal Trees to NLFSM

• Find the phrases/connectives indicative of iterative constructs– while, for (connectives)– “one or more” (phrases)

• Identify the scope and add the back-edges in the temporal tree.

• Scope given by purpose clauses or relations between verbs

Outline

Future Work: NLP

• Extending and adapting ongoing NLP work– Discourse Structure– Discourse Connectives– Temporal relations between clauses

• Adapt existing tools– Parsers– Chunkers– Shallow semantic parsers

Future Work: Verification

• Completeness– NL documents are usually underspecified– Harmless vs. Harmful underspecification

• Harmless – What happens if an FDA-approved screening test cannot be used ?

• Harmful – What happens if you cannot do the test ? (because in repeated testing one might run out of a sample)

– Relations between connectives• “if” without a corresponding “else” or “otherwise”

– Some domain specific way ?

Future Work: Verification• Consistency

– Relations between variables preventing certain transitions from being taken

• A donation shipped prior to testing cannot have been tested– Requires lexical knowledge and/or world knowledge

• A shipped donation is not longer in the possession of the establishment

• Safety– Check if the system behaves as intended

• A sample of blood should not be shipped before testing unless in the case of dire emergencies

– Associating this statement with ones in the document

Future Work: Evaluation

Evaluation– Corpus-based evaluation of NLP techniques– Comparison with models from other systems– Experts evaluation of the models generated

the end

Eliciting Formal Models From Informal Requirements

Documents