American Sign Language Natural Language Generation and Machine Translation Systems Written Preliminary Written Preliminary Examination II Examination II Computer and Information Computer and Information Science Science University of Pennsylvania University of Pennsylvania September 11, 2003 September 11, 2003 Committee: Committee: Norm Badler, Chair Norm Badler, Chair Martha Palmer Martha Palmer Mitch Marcus Mitch Marcus Matt Huenerfauth Matt Huenerfauth
Transcript
American Sign Language Natural Language Generation and Machine Translation Systems
Written Preliminary Examination IIWritten Preliminary Examination IIComputer and Information ScienceComputer and Information Science
University of PennsylvaniaUniversity of PennsylvaniaSeptember 11, 2003September 11, 2003
Martha PalmerMartha PalmerMitch MarcusMitch Marcus
Matt HuenerfauthMatt Huenerfauth
OverviewOverview
● IntroductionIntroduction● Motivations and ApplicationsMotivations and Applications● American Sign Language LinguisticsAmerican Sign Language Linguistics● English-to-ASL ProblemEnglish-to-ASL Problem
● The Four SystemsThe Four Systems● Greatest Strength of Each SystemGreatest Strength of Each System● Point-by-Point ComparisonPoint-by-Point Comparison
● Conclusions and Future DirectionsConclusions and Future Directions
Motivations and ApplicationsMotivations and Applications
● English and ASL are very different languages, but many English and ASL are very different languages, but many approaches to helping the deaf access the hearing world approaches to helping the deaf access the hearing world forget that English is their second language.forget that English is their second language.
● Half of deaf high school graduates read English below Half of deaf high school graduates read English below fourth-grade level, but the vast majority have sophisticated fourth-grade level, but the vast majority have sophisticated fluency in American sign language. fluency in American sign language.
● Applications:Applications:● TV captioning, teletype telephones.TV captioning, teletype telephones.● Human interpreters intrusive/expensive.Human interpreters intrusive/expensive.● Educational tools, access to information.Educational tools, access to information.● Storage and transmission of ASL.Storage and transmission of ASL.
Holt 1991.
ASL Linguistics IASL Linguistics I
● What is ASL? What is ASL? ● Real language? Who uses it? Real language? Who uses it? ● Different than SEE or SSE.Different than SEE or SSE.
● How is it different than English?How is it different than English?● Grammar, Vocabulary, Visual/Spatial.Grammar, Vocabulary, Visual/Spatial.● More than the Hands: Simultaneity!More than the Hands: Simultaneity!● How signs can be changed: Morphology!How signs can be changed: Morphology!● Use of Space around the Signer…Use of Space around the Signer…
ASL Linguistics IIASL Linguistics II● Discourse SpaceDiscourse Space
● Put things on “shelves” for later use.Put things on “shelves” for later use.● Spatial positioning of entities in the discourse Spatial positioning of entities in the discourse
(people or things you are talking about). (people or things you are talking about). ● ““Agreement” - Pronouns, Possessives, Standard Agreement” - Pronouns, Possessives, Standard
Verbs, Special Agreeing Verbs.Verbs, Special Agreeing Verbs.
● Three-Dimensional SpaceThree-Dimensional Space● Pretend there’s a little 3d scene in front of you.Pretend there’s a little 3d scene in front of you.● Some verbs incorporate a 3d path of motion.Some verbs incorporate a 3d path of motion.● Classifier Predicates: describe 3-D scenes.Classifier Predicates: describe 3-D scenes.
ASL Linguistics IIIASL Linguistics III
● Traditional Sentences: Traditional Sentences: ● ASL ASL withoutwithout 3-D classifier predicates. 3-D classifier predicates.
Where does Bob attend college?Where does Bob attend college? wh wh #BOB IX #BOB IXxx GO-TO UNIVERSITY WHERE GO-TO UNIVERSITY WHERE
● Spatially Complex Sentences: Spatially Complex Sentences: ● English into ASL using classifier predicates. English into ASL using classifier predicates.
I parked my car next to his truck.I parked my car next to his truck.POSSPOSSxx TRUCK ClassPred-3-{parking the truck} TRUCK ClassPred-3-{parking the truck}POSSPOSS1s1s CAR ClassPred-3-{park next to truck} CAR ClassPred-3-{park next to truck}
ASL Lacks a Writing SystemASL Lacks a Writing System
Glosses:Glosses:
Glosses used by linguists, not by signers.Glosses used by linguists, not by signers.All notations omit some details of original.All notations omit some details of original.Without writing system, generating ASL is Without writing system, generating ASL is analogous to semantics-to-speech for a analogous to semantics-to-speech for a traditional language. No boundary.traditional language. No boundary.To compensate, each system in survey To compensate, each system in survey develops their own writing formalism.develops their own writing formalism.
________ndnd
iiHELPHELPPRO.2PRO.2 PRO.2 MAN Ix PRO.2 MAN Ixii
Animated Virtual HumansAnimated Virtual Humans
● Virtual Model of the Human Form Virtual Model of the Human Form ● Can be Articulated to Produce ASLCan be Articulated to Produce ASL
● An ASL Generator produces instructions An ASL Generator produces instructions for the avatar, and the avatar performs the for the avatar, and the avatar performs the signs -- producing an animated output for signs -- producing an animated output for the user to view. the user to view.
● Our problem is how to build instructions.Our problem is how to build instructions.
Virtual Signing HumansVirtual Signing Humans
Photos: Seamless Solutions, Inc.Simon the Signer (Bangham et al. 2000.)Vcom3D
The Four Systems
TEAM SystemTEAM System
Zhao et al. 2000
TEAM SystemTEAM System
● At University of Pennsylvania:At University of Pennsylvania:● Xtag English GrammarXtag English Grammar● Synchronous TAG translationSynchronous TAG translation● HMS lab human modeling technologyHMS lab human modeling technology
● ASL morphology, aspect, and adverbials are ASL morphology, aspect, and adverbials are often expressed via subtle modifications to the often expressed via subtle modifications to the performance of signs.performance of signs.
● Specifying these changes can be laborious.Specifying these changes can be laborious.
● HMS already had “motion manner” specification HMS already had “motion manner” specification approach based on Laban Movement Analysis. approach based on Laban Movement Analysis.
● One or two input values controls the manner of One or two input values controls the manner of character’s movement.character’s movement.
● Defining ASL operators in terms of these made some Defining ASL operators in terms of these made some phenomena easier to specify and implement.phenomena easier to specify and implement.
ViSiCAST SystemViSiCAST System
Marshall & Safar 2002.
ViSiCAST SystemViSiCAST System
● University of East Anglia.University of East Anglia.● European Union Project.European Union Project.● English text to English text to BritishBritish Sign Language. Sign Language.
● Uses CMU Link Parser and Head-Driven Phrase Uses CMU Link Parser and Head-Driven Phrase Structure Rules for Generation.Structure Rules for Generation.
● Uses an expressive XML-compatible Uses an expressive XML-compatible sign-language-specific animation sign-language-specific animation control language.control language.
ViSiCAST: DRSViSiCAST: DRS
● Discourse Representation Structure:Discourse Representation Structure:● Lists all entities in the discourse.Lists all entities in the discourse.● Stores simple propositions for semantics.Stores simple propositions for semantics.
● Great for ASL:Great for ASL:● Can track all entities placed in space around signer.Can track all entities placed in space around signer.● Foundation for reference resolution algorithms.Foundation for reference resolution algorithms.
Very important for English-to-ASL!!!Very important for English-to-ASL!!!● Propositions de-aggregate information, remove Propositions de-aggregate information, remove
English syntax bias, and isolates: tense, aspect, and English syntax bias, and isolates: tense, aspect, and other modifiers expressed at varied levels of ASL.other modifiers expressed at varied levels of ASL.
Zardoz SystemZardoz System
Veale et al. 2001.
Zardoz SystemZardoz System● Trinity College Dublin.Trinity College Dublin.
● Ambitious proposal, some implemented.Ambitious proposal, some implemented.● English to many sign languages (ISL,BSL,ASL). English to many sign languages (ISL,BSL,ASL). ● Hand-coded event schemata as Hand-coded event schemata as interlinguainterlingua..● Spatial, commonsense, and metaphorical reasoning.Spatial, commonsense, and metaphorical reasoning.
● AI Focus: AI Focus: ● Metaphorical Reasoning Metaphorical Reasoning ● Knowledge RepresentationKnowledge Representation● Blackboard System ArchitectureBlackboard System Architecture
Zardoz: Reasoning for ASLZardoz: Reasoning for ASL● System designed to facilitate AI reasoning for System designed to facilitate AI reasoning for
visual spatial analysis, idiom decomposition, visual spatial analysis, idiom decomposition, creation of new signs by metaphor.creation of new signs by metaphor.
● Designers asserted that some complex ASL Designers asserted that some complex ASL constructions would require a system that constructions would require a system that models and reasons about spatial relationships.models and reasons about spatial relationships.
● Unfortunately, the schema-based approach they Unfortunately, the schema-based approach they propose is very time-consuming to implement, propose is very time-consuming to implement, and the reasoning requires sophisticated AI.and the reasoning requires sophisticated AI.
● Linguistics/Representation Focus:Linguistics/Representation Focus:● Never implemented animationNever implemented animation..● Uses LFG rules for analysis, Uses LFG rules for analysis,
f-structure transfer rules.f-structure transfer rules.● Uses modern Movement-Hold model of ASL Uses modern Movement-Hold model of ASL
phonology at basis of lexical representation.phonology at basis of lexical representation.
Workbench: ASL PhonologyWorkbench: ASL Phonology
● Movement-Hold Model.Movement-Hold Model.
● ASL defined as time ASL defined as time slices when hands move slices when hands move or pause. We specify or pause. We specify the details of each the details of each hand for each time-slice.hand for each time-slice.
● Non-manual signals Non-manual signals not captured very well.not captured very well. Speers
2001.
Workbench: ASL PhonologyWorkbench: ASL Phonology
● Highly expressive:Highly expressive:● Captures phenomena in modern linguistic studies.Captures phenomena in modern linguistic studies.● But makes lexicon building time-consuming.But makes lexicon building time-consuming.● Some details not known until generation-time.Some details not known until generation-time.
● Morphology and phonology rules for ASL can be Morphology and phonology rules for ASL can be intuitively defined when you use this intuitively defined when you use this representation system. They seem to operate on representation system. They seem to operate on MH time segments.MH time segments.
System ComparisonsSystem Comparisons
Development StatusDevelopment Status
ViSiCAST: Broadest linguistic coverage and only ViSiCAST: Broadest linguistic coverage and only system still under development.system still under development.
Workbench: ASL grammar intermediate, but few Workbench: ASL grammar intermediate, but few lexicon entries or transfer rules.lexicon entries or transfer rules.
TEAM: Demo grammar and lexicon.TEAM: Demo grammar and lexicon.
Zardoz: Limited by schema development. Zardoz: Limited by schema development. Minimal grammar and lexicon.Minimal grammar and lexicon.
System minimally implemented.System minimally implemented.
Direct: No Systems in this Survey.Direct: No Systems in this Survey.
Word for word. Sign dictionary lookup.Word for word. Sign dictionary lookup.
● As you go higher up the pyramid:As you go higher up the pyramid:● Development work increases.Development work increases.● Subtlety of divergences you can handle Subtlety of divergences you can handle
● Generalizations particularly true in non-statistical Generalizations particularly true in non-statistical systems. In non-statistical systems, we know systems. In non-statistical systems, we know better the sources of information: the linguistic better the sources of information: the linguistic artifacts available. artifacts available.
● We know the We know the limitslimits of what the system knows. of what the system knows.
● In statistical system, translation corpora data In statistical system, translation corpora data could capture could capture whateverwhatever information information humanhuman translator used. This data guides the system.translator used. This data guides the system.
● No ASL corpora No ASL corpora no statistical systems. no statistical systems.
ASL Generation Formalism (1)ASL Generation Formalism (1)
ViSiCAST: HPSGViSiCAST: HPSG● Operates on multi-level feature structures. Operates on multi-level feature structures.
Workbench: LFGWorkbench: LFG● Generator: Word order and lexical choice Generator: Word order and lexical choice
before NMS, morphology, or phonology. before NMS, morphology, or phonology.
● Both systems Phrase Structure based.Both systems Phrase Structure based.
Bad!Bad!
Bad!Bad!
Good!Good!
ASL Generation Formalism (2)ASL Generation Formalism (2)
● ASL uses 2 dimensions: time and ASL uses 2 dimensions: time and spacespace to to differentiate the roles of lexical units. differentiate the roles of lexical units. ● So word order tends to be more flexible. So word order tends to be more flexible. ● But PS best for word-ordering intensive But PS best for word-ordering intensive
generation. generation. ● Tends to narrow on particular ordering early. Tends to narrow on particular ordering early. ● Word order flexible; should 1Word order flexible; should 1stst focus on other focus on other
decisions and constraints during generation.decisions and constraints during generation.
● Also, ASL ‘word’ unit is hard to define.Also, ASL ‘word’ unit is hard to define.
ASL Generation Formalism (2)ASL Generation Formalism (2)
TEAM: Synchronous Tree Adjoining GrammarTEAM: Synchronous Tree Adjoining Grammar● Determines ASL surface tree from English one. Determines ASL surface tree from English one. ● Discourse model and deeper analysis should Discourse model and deeper analysis should
determine reference choice, topicalization, tagging, determine reference choice, topicalization, tagging, and most NMS.and most NMS.
Zardoz: Spatial Dependency GraphsZardoz: Spatial Dependency Graphs● Represents ASL signs and NMS-boundary tokens in a Represents ASL signs and NMS-boundary tokens in a
partial ordering graph structure. Can change or add partial ordering graph structure. Can change or add to the constraints. to the constraints.
● All orderings are “soft” - chose optimal linearization.All orderings are “soft” - chose optimal linearization.● Takes advantage of ASL word order flexibility. Takes advantage of ASL word order flexibility.
Workbench: Workbench: Calculates NMS from c-structure result.Calculates NMS from c-structure result.NMS merely complements the syntax.NMS merely complements the syntax.
ViSiCAST: ViSiCAST: NMS not implemented.NMS not implemented.
Zardoz: Doll Control Language.Zardoz: Doll Control Language.Lexical specs stored hierarchically. Lexical specs stored hierarchically. Good!Good!
ViSiCAST: Signing Gesture Markup Lang.ViSiCAST: Signing Gesture Markup Lang.Sign language specific – well-suited.Sign language specific – well-suited. BEST!BEST!
Workbench: MH segments are final output. Workbench: MH segments are final output. Very ASL specific, butVery ASL specific, but Good!Good! may be hard to animate. may be hard to animate. Bad!Bad!
Classifier Predicates & SpaceClassifier Predicates & Space
Workbench explained how classifier signs fit into Workbench explained how classifier signs fit into MH Model and characterized some.MH Model and characterized some.
Zardoz is only system to address how to Zardoz is only system to address how to generate them using complex spatial reasoning generate them using complex spatial reasoning on information stored in the translation on information stored in the translation schemata. But not a practical approach. schemata. But not a practical approach. Hand-coding individual schema.Hand-coding individual schema.
Bad!Bad!
Ok.Ok.
Good!Good!
User InterventionUser Intervention
TEAM & Zardoz do not allow intervention.TEAM & Zardoz do not allow intervention.
ViSiCAST allows manual intervention ViSiCAST allows manual intervention during MT to fix errors before propagation.during MT to fix errors before propagation.
Workbench requires intervention in order Workbench requires intervention in order to operate. Does not attempt any to operate. Does not attempt any reference resolution.reference resolution.
Ok.Ok.
Ok.Ok.
Bad!Bad!
ConclusionConclusion
ConclusionsConclusions
● The symbolic notation for ASL can subtly limit the The symbolic notation for ASL can subtly limit the potential expressiveness of the animation output if it is potential expressiveness of the animation output if it is insufficiently detailed.insufficiently detailed.
● The notation should make it easy to parameterize The notation should make it easy to parameterize modulations to the standard sign performance to express modulations to the standard sign performance to express adverbials and morphological operations.adverbials and morphological operations.
● ASL will require a discourse representation that has ASL will require a discourse representation that has been designed with the needs of a language which been designed with the needs of a language which unambiguously and spatially refers to entities in the unambiguously and spatially refers to entities in the discourse.discourse.
ConclusionsConclusions● Scene representation and spatial reasoning will be Scene representation and spatial reasoning will be
required to generate classifier predicates, directional required to generate classifier predicates, directional verbs, and other complex uses of the signing space.verbs, and other complex uses of the signing space.
● The generation grammar formalism should allow The generation grammar formalism should allow simultaneous access to multiple levels of ASL simultaneous access to multiple levels of ASL expression and should take advantage of ASL’s word expression and should take advantage of ASL’s word order flexibility.order flexibility.
● Limited ASL corpora means that the non-statistical MT Limited ASL corpora means that the non-statistical MT systems face the development effort vs. divergence systems face the development effort vs. divergence handling tradeoff acutely.handling tradeoff acutely.
Future DirectionsFuture Directions
● Only the ViSiCAST project still running.Only the ViSiCAST project still running.● Many projects are developing potentially useful Many projects are developing potentially useful
component technologies for ASL MT:component technologies for ASL MT:● Sign Lexicons, Signing Motion Capture, Sophisticated Sign Lexicons, Signing Motion Capture, Sophisticated
Human Hand Models, and Annotated Sign CorporaHuman Hand Models, and Annotated Sign Corpora
● Some new commercial English-to-SEE systems. Some new commercial English-to-SEE systems. (Vcom3D, iCommunicator)(Vcom3D, iCommunicator)
● Beginnings of an ASL research project here at the Beginnings of an ASL research project here at the University of Pennsylvania.University of Pennsylvania.
Questions?Questions?
Selected References (1)Selected References (1)● N. Badler, R. Bindiganavale, J. Allbeck, W. Schuler, L. Zhao, S. Lee, H. N. Badler, R. Bindiganavale, J. Allbeck, W. Schuler, L. Zhao, S. Lee, H.
Shin, and M. Palmer. 2000. Parameterized Action Representation and Shin, and M. Palmer. 2000. Parameterized Action Representation and Natural Language Instructions for Dynamic Behavior Modification of Natural Language Instructions for Dynamic Behavior Modification of Embodied Agents. AAAI Spring Symposium. Embodied Agents. AAAI Spring Symposium. ftp://ftp.cis.upenn.edu/pub/graphics/rama/papers/aaai.pdf ftp://ftp.cis.upenn.edu/pub/graphics/rama/papers/aaai.pdf
● J. A. Bangham, S. J. Cox, R. Elliot, J. R. W. Glauert, I. Marshall, S. J. A. Bangham, S. J. Cox, R. Elliot, J. R. W. Glauert, I. Marshall, S. Rankov, and M. Wells. 2000. “Virtual signing: Capture, animation, Rankov, and M. Wells. 2000. “Virtual signing: Capture, animation, storage and transmission - An overview of the ViSiCAST project.” IEEE storage and transmission - An overview of the ViSiCAST project.” IEEE Seminar on “Speech and language processing for disabled and elderly Seminar on “Speech and language processing for disabled and elderly people.” people.”
● B. Dorr, P. Jordan, and J. Benoit. 1998. “A Survey of Current Paradigms B. Dorr, P. Jordan, and J. Benoit. 1998. “A Survey of Current Paradigms in Machine Translation.” http://citeseer.nj.nec.com/555445.html in Machine Translation.” http://citeseer.nj.nec.com/555445.html
● J. Holt. 1991. Demographic, Stanford Achievement Test - 8th Edition for J. Holt. 1991. Demographic, Stanford Achievement Test - 8th Edition for Deaf and Hard of Hearing Students: Reading Comprehension Subgroup Deaf and Hard of Hearing Students: Reading Comprehension Subgroup Results. Results.
Selected References (2)Selected References (2)● iCommunicator 4.0 Website. 2003. http://www.myicommunicator.com/ iCommunicator 4.0 Website. 2003. http://www.myicommunicator.com/ ● S. Liddell and R. Johnson. “American Sign Language: The Phonological S. Liddell and R. Johnson. “American Sign Language: The Phonological
Base,” Sign Language Studies, 64, pages 195-277, 1989. In C. Valli & C. Base,” Sign Language Studies, 64, pages 195-277, 1989. In C. Valli & C. Lucas, 2000, Linguistics of American Sign Language, 3rd edition, Lucas, 2000, Linguistics of American Sign Language, 3rd edition, Washington, DC: Gallaudet University Press. Washington, DC: Gallaudet University Press.
● C. Neidle, J. Kegl, D. MacLaughlin, B. Bahan, and R. G. Lee. 2000. The C. Neidle, J. Kegl, D. MacLaughlin, B. Bahan, and R. G. Lee. 2000. The Syntax of American Sign Language: Functional Categories and Syntax of American Sign Language: Functional Categories and Hierarchical Structure. Cambridge, MA: The MIT Press. Hierarchical Structure. Cambridge, MA: The MIT Press.
● É. Sáfár and I. Marshall. 2002. “Sign language translation via DRT and É. Sáfár and I. Marshall. 2002. “Sign language translation via DRT and HPSG.” In A. Gelbukh (Ed.) Proceedings of the Third International HPSG.” In A. Gelbukh (Ed.) Proceedings of the Third International Conference on Intelligent Text Processing and Computational Linguistics, Conference on Intelligent Text Processing and Computational Linguistics, CICLing, Mexico, Lecture Notes in Computer Science 2276, pages 58-68, CICLing, Mexico, Lecture Notes in Computer Science 2276, pages 58-68, Springer Verlag, Mexico. Springer Verlag, Mexico.
Selected References (3)Selected References (3)● d'A.L. Speers. 2001. Representation of American Sign Language for d'A.L. Speers. 2001. Representation of American Sign Language for
Machine Translation. PhD Dissertation, Department of Linguistics, Machine Translation. PhD Dissertation, Department of Linguistics, Georgetown University. Georgetown University.
● T. Veale, A. Conway, B. Collins. 1998. “The challenges of cross-modal T. Veale, A. Conway, B. Collins. 1998. “The challenges of cross-modal translation: English to sign language translation in the ZARDOZ system” translation: English to sign language translation in the ZARDOZ system” in Machine Translation 13. 81-106. in Machine Translation 13. 81-106.
● L. Zhao, K. Kipper, W. Schuler, C. Vogler, N. Badler, and M. Palmer. L. Zhao, K. Kipper, W. Schuler, C. Vogler, N. Badler, and M. Palmer. 2000. “A Machine Translation System from English to American Sign 2000. “A Machine Translation System from English to American Sign Language.” Association for Machine Translation in the Americas. Language.” Association for Machine Translation in the Americas.
● See Written WPE2 Report for full references.See Written WPE2 Report for full references.
