The Concept of Connecting Ontology and itsExploitation for Knowledge and InformationPresentation for People with Special Needs

Muhammad Shuaib Karim1,2

1Institute of Software Technology and Interactive SystemsVienna University of Technology (http://www.ifs.tuwien.ac.at/)

2Computer Science DepartmentQuaid-i-Azam University, Islamabad (http://www.qau.edu.pk/)

October 09 / 2007October 09 / 2007

Technisches Universität Wien

2

Motivation

Providing a generic accessibility solution for SemanticLIFE using the Semantic Web TechnologyProviding a generic accessibility solution for SemanticLIFE using the Semantic Web Technology

3

Semantic Web – Promises

● Ability to integrate heterogeneous data sources

● Ability to formally describe the information Formal data description makes it understandable,

sharable and thus processable by software agents Automatic reasoning becomes possible

● Abundance of OS tools from modeling, storage, annotation, reasoning, & query to user interfaces

4

Semantic Web - Architecture

5

Semantic Web – Resource Description (1/5)

● Every Thing can be thought of as an Entity

● The Entity is described as a Resource

● The Resource may range from literal to concept map, and has formal Description

● The Resource is identified by a URI

● The Description has the consensus within CoP

6

Semantic Web – Resource Description (2/5)

● The Description is asserted in terms of (Subject,

Predicate, Object) triples where each of them is a Resource

● For example, <Person workingOnTask Task> <:John workingOnTask :Ontology_Template>

7

Semantic Web – Resource Description (3/5)

In principle, every “piece of information”● can be conceptualized in terms of inter-relation of entities

– Ontological (Philosophical) Level, AND● has schema (ontology) described in terms of chains of

triples as well as its instances (individuals) – Developer or Content Author Level, AND

● has implementation for those chains of triples based upon strong theoretical formalism of Description Logic – Implementation Level Allows Reasoners to make inference Deduce relationships such as containment, symmetrical,

transitive, inverse relations

8

Semantic Web – Resource Description (4/5)

● So, Semantic Web provides a platform from Conceptual Modeling to Implementation with added inference capabilities

● Not necessary to implement all of it for getting started. “Even a little semantics go a long way – James Hendler”

9

Semantic Web – Resource Description Example (5/5)

John is working on tasks “Ontology_Templates´´ and “Message_Bus´´. The task “Ontology_Templates´´ is for project “DynamOnt´´ and the task “Message_Bus´´ is for project “SemanticLIFE´´

Person

John

Shuaib

Task

Ontology_Templates

Message_Bus

Project

DynamOnt

SemanticLIFE

workingOnTask

workingOnTask

workingOnTask

partOf

partOf

workingOnProject

10

SemanticLIFE - Architecture

Message Bus Plug-in

Data Feed Processing

Repository Plug-in

PersonalRepository

Ontologies

Pipeline Plug-in

Stylesheets

User ProfilePlug-in

AnnotationPlug-in

Web ServicePlug-in

AnalysisPlug-in

Visualization Plug-in

Accessibility Plug-in

Pipelines

(Ahmed et al., 2004). SemanticLIFE - A Framework for Managing Information of A Human Lifetime. In Austrian Computer Society Book Series: iiWAS'04 Proc., (vol.183), pages 687-696.

11

Concept of SemanticLIFE UI

SemanticLIFE - x

Scratch Bags

Misc.

Control Panel

TimeLine

AbstractControls Query

Items Panel Text TableXML

AssociationPanel

RDF

4 associated mails, 5 web pages, 1 file, 12 process state data

Status Bar

Annotation

User Profile Settings HelpData Feeds

Query Trash

Information Items

User Abstract Information

System Information

PIM

EmailWeb PageProcess StateFile SystemCalendarNoteContactFile

Email Web Page FileAll Contact Note

EmailFrom ToSubject

Re: Reification withindividuals

Tim semantic-web@w3.org

Farewell Party Shehzad ATScolars@yahoogr...

FilePath SizeName

Semantic Desktop Proposal

C:\prj 20 Kbs

Text GraphRDFNativeReification withindividuals

Tim semantic-web@w3.org

A

AnnotationA

Trash

ProjectResourceTaskDeliverableLocationCost

Slife Core Ontology

Imapirments OntologyUI Ontology

Project Ontology

Khalid,modeling of n-ary relationships and reified relationships is very well described in a working draft of the Semantic Web Best Practices working group:http://www.w3.org/TR/swbp-n-aryRelations/The paper describes several patterns and examples that can help you in modeling.TimLatif, Khalid wrote:> >Hello,>I have made ontology of vehicle parts with many properties. I need to

12

Accessibility – Key Issues

● Not only an issue for people with popular disabilities and specific age group, rather it is Acc4All

● Not only related with user interface issues, rather applicable to whole system

13

Accessibility – Web Accessibility Components

User

Content

Evaluation Tools(WCAG)

Authoring Tools(ATAG, WCAG)

User Agents(UAAG)

Assistive Technology(UAAG)

Developer(WCAG)

14

Research Questions

● Can Semantic Web Technology be used for providing a Generic Accessibility solution?

● How far is it true that the investment on UI alone could provide maximum possible Accessibility?

● Is the sought-after approach exploitable towards diversity in general, and integration of Information Management Systems in particular?

15

Sample Components Requiring Connections

User Profile ↔ Assistive Technology

Assistive Technology ↔ User Agents

User Agents ↔ Contents

….

….

User Interfaces ↔ User Profile

Info. Semantics of Domain1 ↔ Info. Semantics of Domain2

Encoded in software or in data → based upon some rules → ?

Connection of heterogeneous domains => Rules

User

Content

Evaluation Tools(WCAG)

Authoring Tools(ATAG, WCAG)

User Agents(UAAG)

Assistive Technology(UAAG)

Developer(WCAG)

16

Significance of Connections (Ex. Food & Wine)

Suggests different foods and wine combinations

→ Domain expert´s knowledge encoded as OWL DL, such as;

• taste• appetite or stimulation to eat• digestion• availability

17

Ex. Food & Wine

Another Approach (“Connecting Ontology´´) Ontology of Food – Taxonomy with concepts such as;

• the calories with respect to the quantity• the tendency to get digested on ist own (Quick, Medium

duration, Long duration)• …

Ontology of Wine - Taxonomy with concepts such as;• the appetizing effects (Nil, Low, Medium, High)• the digestive effects (Nil, Low, Medium, High)• …

Above knowledge normally accessible from the available literature

18

Food & Wine (new CQ)

New CQ possible to fulfill: Food requiring no drinks Food requiring drinks of certain quality & brand Food & wine combination requiring least digestion time

19

Connecting Ontology - Characteristic Features of OC between O1 and O2

● O1 , O2 not related with same domains of discourse

● O1 , O2 developed using their own CODeP

● Similarity b / w CODeP indicate inter-connection

● Creation of new knowledge

20

Connecting Ontology - Benefits

● Useful for top-down evolution of ontologies / applications

● Incompatibilities between the two ontologies are solved at the ontological level

● Helpful in code automation● Possibility to determine the Cause - Effect

relationship between the two ontologies● Reliance on domain experts reduced due to

encoding of domain knowledge

The key is “How to capture and represent the domain knowledge ??´´

21

Connecting Ontology - Domain Knowledge

Available as;

Structured documents

Unstructured documents

Tacit knowledge with domain experts

22

Connecting Ontology – Capturing Domain Knowledge

● Text processing of knowledge about domains of CO and the two participating ontologies

● Refinement of the above by aligning participating ontologies with global standard ontologies

● Exploiting valid queries on the two ontologies

● Exploiting queries result set and data mining

● Using Semantic Web Rules

23

Connecting Ontology - Workflow

Load Ontology A

Load Ontology B

1. createModel

Ontology Model

2. addModel

Load Rules

ConnectingOntology

4. storeOntology

3. applyRules

24

Accessibility Plug-in (A scalable framework using Semantic Web Technology)

Info-VizBridge Service

Accessibility Framework

AccessibilityService

ConnectingOntology

DeviceProfiles

DomainOntology

User Profile(Impairments)

map

s maps

maps

ConnectingOntology

map

s

maps

Uses

Use

s

RepresentationOntology

Karim, S., Latif, K., and Tjoa, A. M. (2007). Providing Universal Accessibility using Connecting Ontologies: A Holistic Approach. Universal Access to Applications and Services, In Proc. of HCII’07., LNCS 4556 (vol. 7).

25

CODeP for Accessibility - Generic Pattern with Spatio-Temporal Dimensions

InterfaceElement

GenericAccessibilityPattern

Representation Capability

InformationObject

composedOf

User

CausalObject

describes has

exposedBy

ImpairmentProfile

belongsTo

dependsOn

Space-Region

affects

spatialLocation

Time-Interval

temporalLocation

26

CODeP for Accessibility – Simplified Generic Pattern

InterfaceElement

GenericAccessibilityPattern (Simplified)

Representation Capability

InformationObject

composedOf

effects

User

CausalObject

describes has

exposedBy

ImpairmentProfile

belongsTo

dependsOn

27

CODeP for Accessibility - Memory Recall Pattern

28

CODeP for Accessibility - Perception Effect Pattern

29

CODeP for Accessibility – Mobility Enhancement Pattern

30

Persistence of Patterns

Formal description of semantics for each component

=> Ontology Oi for each component

Formal description of consequences and effects of potentially interacting component on each other

=> Connecting Ontology for O1 and O2

31

Test case

Connecting Ontology for

User’s Impairments & UI Characteristics

Accessibility Guidelines

User Interface C

haracteristics Use

r Im

pairm

ents

Pro

file

Accessible User Interface

32

Role of User Impairments

Assistance to be provided = User's Needs - User's Capabilities ...(1)

Assistance to be provided = User's Needs + User's Impairments needs - User's Capabilities of (1)

Quality of Life Technology (QoLT)Ref: Kanade, T. (2007). Digital Human Modeling and Quality of Life Technology. In Keynote address in 12th International Conference on Human-Computer Interaction, Beijing, China.

33

Motivation - Ontology of Visualization Techniques

informationEntity Tree

VisualizationTechnique

visualizationAbstraction

suitableFor

Schema

hasSchema

Data

hasData

breadthFirst traversal

visualizationTransformation analyticalAbstraction

breadthFirst traversal

Layout

visualMappingTransformation

Ref: Chi, E. (2000). A Taxonomy of Visualization Techniques using the Data State Reference Model. In INFOVIS, pages 69-75.

34

Overview of Ontology Interaction for On-Demand Visualization

Information Semantics

Device Needs

Visualization Techniques

User Needs / Impairments

Connecting Ontologies

*Application Domain Ontologies

**

*

35

Human Disease Ontology

HumanDisease

Type

Treatment

Cause

Environmental

Symptom

Genetic

Physiotherapy

Chemotherapy

Surgery

DrugTherapyPsychotherapy

Ref: Hadzic, M. and Chang, E. (2005). Ontology-Based Support for Human Disease Study. In Proc. of HICSS'05, IEEE Computer Society.

36

Extension of Human Disease Ontology

HumanDisease

Type

Treatment

Cause

Environmental

Symptom

Genetic

Physiotherapy

Chemotherapy

Surgery

DrugTherapyPsychotherapy

InterfaceAdaptation

Impairment

Ref: Hadzic, M. and Chang, E. (2005). Ontology-Based Support for Human Disease Study. In Proc. of HICSS'05, IEEE Computer Society.

37

Impairment-User interface Connecting Ontology - Sample user scenarios

● Avoiding the confusing colors on an interface for a user with particular type of color blindness

● Font adjustments according to user‘s visual acuity

● Information presentation on the better part of the screen for a user suffering from Hemianopsia (absence of vision in half of visual field)

38

Impairment-User Interface Ontology

Karim, S. and Tjoa, A. M. (2006). Towards the Use of Ontologies for Improving User Interaction for People with Special Needs. In Proc. of ICCHP’06, LNCS (vol. 4061), pages 77-84.

39

Exploring Relationship between Motor Control Impairments and UI Components

Impairment Related With UI ComponentNormalMotorControl suggests ComboBox

NormalMotorControl suggests RadioButton

NormalMotorControl suggests CheckBox

NormalMotorControl suggests ScrollBar

NormalMotorControl suggests Spinner

NormalMotorControl suggests ToggleButton

VeryLowMotorControl suggests AudioFeedback

VeryLowMotorControl suggests ToggleButton

VeryLowMotorControl prohibits Spinner

VeryLowMotorControl prohibits ScrollBar

40

Representing Relationship between Motor Control Impairments and UI Components

http://www.ifs.tuwien.ac.at/ontologies/imp/Impairment

isA

.../imp/SeverityMeasure

hasSeverityMeasure

OneOf:#Normal, #High, #VeryHigh

http://www.ifs.tuwien.ac.at/ontologies/ui/UiComponent

isA

.../ui/ConvenienceMeasure

OneOf:#Normal, #High, #VeryHigh

.../ui/ComboBox#

.../ui/Spinner#

.../ui/ToggleButton#

.../ui/ScrollBar# .../ui/RadioButton# .../ui/CheckBox#

isA

isA

isA isA isA

.../imp/MotorControlImapirment#

hasMotoricConvenience

41

Representing Relationship in DL

● MotorControlImpairment with SeverityMeasure “VeryHigh” suggests UiComponent with MotoricConvenienceMeasure “VeryHigh”

VeryHighasureseverityMeyMeasurehasSeveritorControlVeryLowMotntolImpairmeMotorControrControlVeryLowMot

.

● MotorControlImpairment with SeverityMeasure “Normal” suggests UiComponent with MotoricConvenienceMeasure “Normal”

NormalasureseverityMeyMeasurehasSeveritrControlNormalMotontolImpairmeMotorContrrControlNormalMoto

.

and also;

VeryHigheMeasureconvenienceConvenienchasMotoricnentnientCompotoricConveVeryHighMo

tUiComponennentnientCompotoricConveVeryHighMo

.

NormaleMeasureconvenienceConvenienchasMotoricntentComponericConveniNormalMoto

tUiComponenntentComponericConveniNormalMoto

.

42

Impairment ontology - competency questions

For a given impairment name;● What is / are the related body parts ?

● What is the impaired side (right, left,...) ?

● What is its severity (on a predefined scale) ?

● What are the perception cues which are affected, and up to what degree (on a predefined scale) ?

● What is the effect of one impairment on another w.r.t. affected perception ?

43

Impairment Ontology

44

Some derived concepts (1 / 2)

● LeftSidedImpairment

LeftnhasPositioyPartrelatedBodmpairmentLeftSidedIImpairmentmpairmentLeftSidedI

.

● RightSidedImpairment

RightnhasPositioyPartrelatedBodImpairmentRightSidedImpairmentImpairmentRightSided

.

● BothSidedImpairment

RightnhasPositioLeftnhasPositioyPartrelatedBodmpairmentBothSidedI

ImpairmentmpairmentBothSidedI

.

.

45

Derived concepts (2 / 2)

● AnySidedImpairment

RightnhasPositioLeftnhasPositioyPartrelatedBodpairmentAnySidedIm

ImpairmentpairmentAnySidedIm

.

.

● AnySidedImpairment

ImpairmentRightSided ImpairmentLeftSidedpairmentAnySidedIm

46

User interface Ontology - competency questions

● Find the part-whole relationship of UI components

● Find the attributes of a component and their values (according to predefined usability scale for a normal user in normal conditions)

● For a given attribute, find the related UI components

47

User Interface Ontology

48

Some derived concepts (1 / 2)

● GoodUsabilityComponent

GoodtyhasUsabilitUiComponenntityComponeGoodUsabiltUiComponenntityComponeGoodUsabil

.

● FairUsabilityComponent

FairtyhasUsabilitUiComponenntityComponeFairUsabiltUiComponenntityComponeFairUsabil

.

● FairUserControlComponent

FairluserControtUiComponennentntrolCompoFairUserCotUiComponennentntrolCompoFairUserCo

.

49

Rules to Connect Impairments and UI

Low perception implies suggesting high usability components (VisualAcuityLow UI LegibilityGood)

Karim, S. and Tjoa, A. M. (2007). Connecting User Interfaces and User Impairments for Semantically Optimized Information Flow in Hospital Information Systems. Journal of Universal Computer Science: In Proc. of I-MEDIA'07 and I-SEMANTICS'07, pages 372-379.

(?x rdf:type imp:VisualAcuity) (?x imp:perceptionMeasure imp:Low)(?y rdf:type ui:UiComponent) (?y ui:hasLegibility ui:Good) (?x eg:suggests ?y).

50

Rules to Connect Impairments and UI

High perception implies suggesting fair usability components (VisualAcuityHigh UI LegibilityFair)

(?x rdf:type imp:VisualAcuity) (?x imp:perceptionMeasure imp:High)(?y rdf:type ui:UiComponent) (?y ui:hasLegibility ui:Fair) (?x eg:suggests ?y).

51

Rules to Connect Impairments and UI

High rheumatism implies suggesting easily operatable components (RheumatismHigh UI UserControlGood)

(?x rdf:type imp:Rheumatism) (?x imp:impairmentMeasure imp:High)(?y rdf:type ui:UiComponent) (?y ui:userControl ui:Good) (?x eg:suggests ?y).

CO Workflow

52

Generated RDF triples

<rdf:RDF xmlns:co="http://www.ifs.tuwien.ac.at/ontologies/co#" ...> <rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#VisualAcuity_High"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_09"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_26"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextStyle_Italic"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextStyle_BoldItalic"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_08"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_24"/> </rdf:Description> <rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#Rheumatism_High"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#LabelledButton"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#ComboBox"/> </rdf:Description> <rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#VisualAcuity_Low"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_18"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_10"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextFont_TimesNewRoman"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextStyle_Bold"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_22"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_11"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_20"/> </rdf:Description><rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#ColorBlindness_RG"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Grey"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Orange"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Cyan"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Blue"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Yellow"/> </rdf:Description> <rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#ColorBlindness_YB"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Grey"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Orange"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Cyan"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Green"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Red"/> </rdf:Description></rdf:RDF>

53

54

55

56

Visualization and Accessibility Plug-ins

Triple store

Ontology

Storage

Data Preparation for UI

UI Mapping Transformation

Data Transformation

Grouping Aggregation Rendering

Transformation for Image

Formatted Data

User InterfaceQuery

XMLTreeRDF

Graph

Filtering

Visualization Plug-in

Accessibility Plug-in

Info-VizBridge Service

AccessibilityService

57

Grouping and Aggregation of Items Arranged on a Timeline

58

Consequences of Impairments-user Interface Connecting Ontology

• Helpful in automatically adapting UI for the user• Helpful in deducing the best match of UI characteristics

for a user with multiple impairments• Possibility to use the ontology for diversity• Historical data for studying the cause-effect relationship

b/w the impairments and the computer interfaces• Useful for rehabilitation purposes• Possibility to extract impairment related semantics from

user´s information stored in SemanticLIFE repository, and modify the impairments ontology accordingly

59

Concluding Remarks – Contributions (1/5)

● Ontology for Impairments and Usability introduced● A number of Accessibility CODeP introduced● The concept of Connecting Ontology introduced● Demonstration of Semantic Web Rule Layer for

developing higher level ontologies ● The Connecting Ontology concept using the ontological

rule-based approach paves the way for a generic solution● Introduced the integration of heterogeneous domains by

persisting the tacit knowledge using Connecting Ontology● Application of the approach in the motivating scenarios

60

Concluding Remarks - Future directions (2/5)

● User testing● Semantic Web Service for the Accessibility and

the Info-Viz Bridge modules● Instantiating the Impairments and UI ontologies● Integrating capability measuring tools (MMSE)● Realization of the remaining CODeP for

Accessibility● Integration for ontology of visualization techniques● Elevation / Lifting of Connecting Ontology

61

Concluding Remarks – Goals Evaluation (3/5)

● Can Semantic Web Technology be used for providing a Generic Accessibility solution?Generic foundation, not the complete generic solutionUI adaptation possible Visualization toolkits still not ready

62

Concluding Remarks – Goals Evaluation (4/5)

● How far is it true that the investment on UI alone could provide maximum possible Accessibility?Annotation of resources, ontologies of life events, tasks

UI & Applications

Accessib

ility

63

Concluding Remarks – Goals Evaluation (5/5)

● Is the sought-after approach exploitable towards diversity1 in general, and integration of Information Management Systems2 in particular? 1Modeling of user specific attributes (impairments) 2Domain Knowledge, Business rules 2Motivating scenarios

64

Thanks!

http://www.ifs.tuwien.ac.at/~skarimSemanticLIFE Project http://storm.ifs.tuwien.ac.at/

ASEA-UNINET(Asean European University Network) http://www.uibk.ac.at/asea-uninet/

HEC (Higher Education Commission of Pakistan)http://www.hec.gov.pk/

Quaid-i-Azam University, Islamabadhttp://www.qau.edu.pk/

Technisches Universität Wien