Homeland Security• Obviously since 9/11, homeland security has been brought to the

forefront of public concern and national research– in AI, there are numerous applications

• The Dept of HS has identified the following problems as critical:– intelligence and warning – surveillance, monitoring, detection of deception– border and transportation security – traveler identification, vehicle

identification location and tracking– domestic counterterrorism – tying crimes at the local/state/federal level to

terrorist cells and organizations, includes tracking organized crime– protection of key assets – similar to transportation security, here the targets

are fixed, security might be aided through camera-based surveillance and recording, this can include the Internet and web sites as assets

– defending against catastrophic terrorism – guarding against weapons of mass destruction being brought into the country, tracking such events around the world

– emergency preparedness and response – includes infrastructure to accomplish this such as wireless networks, information sources, rescue robots

The Dark Web• Goal: collect relevant web pages from terrorism web sites and

make them accessible for specific terrorism-related queries and inferences– Starting from reliable URLs, use a web spider to accumulate related web

pages• link analysis and human input are both applied to prune irrelevant pages

– Automatically collect the pages from the URLs and annotate the pages (including those with multimedia and multilingual content)

– Content analysis performed by humans using domain specific attributes of interest

Clustering on the Dark Web

Domestic web sites of US hate groups

Middle East terror organizations sites

Clustering and classification algorithmsare run on web site data – to date, much ofthe data is manually annotated from the web sites before clustering/classification algorithms are run

Here are some results

Clustering performed usingstatistical hierarchical clustering,features include those derived throughsocial analysis, link analysis, andpatterns derived through groups of linksand sites

Dark Web Continued• With web pages

annotated, the Dark Web can now be used for document retrieval based on search queries (including cross-lingual searches)– SOM mapping– link analysis– content analysis

• Activity scale(c, d) =

• c = cluster• d = attribute of

interest• wi,j = 1 if attribute i is

in site j, 0 otherwise• m = total number of

web sites, n = total number of attributes

nmwn

i

m

j ji */,

Dynamic Interoperability of 1st Responders• Test implemented on Philadelphia Area Urban Wireless

Network– Idea is to test ability of mobile communication network and devices

(PDA, laptop, tablet computers, etc) to register agents and services for first responders who might be mobile as well to acquire information during an emergency

– Need: • service registration• service discovery• service choreography

– Solution: use OWL-S service registry for manets (mobile ad hoc networks)

Simulated Manet topology

Agent x is looking for a host with service e but x is only aware of hosts A, B and D

Experiment• Researchers ran two experiments, first on a simulation and second

on a small portion of a network in Philly using mobile devices (in about a 2 block area)– agents had various options of when to consult host registries and how to

migrate to another host (as packets or bundles)• Findings:

– cross-layered design where agents reason about network and service dynamics worked well

– using static lists, performing random walks, using inertia (following the path of the last known location of a service) all had problems leading to poor performance from either too many “hops” or not succeeding in locating the desired service

– p-early binding – agent would consult local host’s registry to identify nearest host that contained needed service and would migrate to that host as a packet

– b-early binding – agent would consult local host’s registry to identify nearest host that contained needed service and would migrate to that host as a bundle

– late binding – agent would consult a local registry at each step as it migrated, always selecting the nearest host with the available service

Ontology for Bioterrorism Surveillance

• This research deals with an ontology and problem solving agents to monitor health threats– the system, BioStorm, includes a central ontology of domain-specific and

task-specific knowledge for reasoning about bioterrorism events and two-tier medication to reconcile heterogeneous data

– problem solving methods include statistical (counting) approaches for low-level analysis, knowledge-based approaches for qualitative reasoning, disease-specific knowledge reasoners and temporal reasoners

Surveillance of San Francisco 911 Emergency Dispatch Data

Deception Detection• Analyze hand and face motion and orientation (via video) to determine degree

of truth or deception during an interview• Capture video features:

– head position, head velocity– left/right hand position, left/right hand velocity– distance from left/right hand to head, from hand to hand

• Use features as input to classifier on four classifiers– trained neural network– support vector

machine (perform non-linear classification using linear classification techniques)

– alternative decision trees (an ensemble)

– discriminant analysis• NN and SVM had

highest accuracy in experiments

Authorship Identification• Determine who the author was of some Internet web forum message in Arabic

or English– Attributes:

• lexical – word (or character for Arabic) choice• syntactic – choice of grammar• structural – organization layout of the message• content-specific – topic/domain

• Approaches were to perform classification using C4.5 and SVMs– English identification uses 301 features used from a test set (87 lexical, 158

syntactic, 45 structural and 11 content-specific)– Arabic identification uses 418 features used from a test set (79 lexical, 262

syntactic, 62 structural and 15 content-specific)– used web spider to collect test set of documents from various Internet forums

• Results are given below– feature sets F1, F2, F3, F4 are the use of lexical, lexical, syntactic, structural and

content-specific respectively

Notice how accuracyimproves significantlyas higher level featuresare added

Arabic Feature Set

Elongation – words > 10 characters are rare in Arabic, so words that were overly long wereconsidered to be no more than 10 characters so that word length distribution was not distorted

Wearable AI• Wearable computer hardware is becoming more prevalent in

society– we want to enhance the hardware with software that can supply a variety

of AI-like services• The approach is called humanistic intelligence (HI)

– HI includes the human in the processing such that the human is not the instigator of the process but the beneficiary of the results of the process

– HI embodies three operational modes:• Constancy – the HI device is always operational (no

sleep mode) with information always being projected (unlike say a wrist watch where you have to look at it)

• Augmentation – the HI augments the human’s performance by doing tasks by itself and presenting the results to the human

• Mediation – the HI encapsulates the human, that is, the human becomes part of the apparatus for instance by wearing special purpose glasses or headphones (but the HI does not enclose the human)

– These systems should be unmonopolizing, unrestrictive, observable, controllable, attentive, communicative

HI Applications• Filtering out unwanted information and alerting

– imagine specialized glasses that hide advertisements or replace the content with meaningful information (e.g., billboards replaced with news)

– alerting a driver of an approaching siren

• Recording perceptions– example uses are to have the wearable record hand motions while the user

plays piano or to record foot motions while the user dances to capture choreography, or for computer animation models

• Military applications– aiming missiles or making menu selections in an airplane so that the pilot

doesn’t have to move his hands from the controls– reconnaissance by tracking soldiers in the field, seeing what they are seeing

• Minimizing distractions– using on-board computing to determine what a distraction might be to you

and to prevent it from arising or blocking it out

• Helping the disabled– HI hearing aids, HI glasses for filtering, internal HI for medication delivery,

reminding and monitoring systems for the elderly

Beyond AI Wearables• As the figure below

shows, these devices may be more intimately wound with the human body– We are currently attaching

ID/GPS mechanisms to children and animals

– Machine-based tattoos are currently being researched • What about

underneath the skin?– Nano-technology– Hardware inside

the human body (artificial hearts, prosthetic device interfaces, etc)

Sensor Interpretation• Given sensor readings, interpret what they are telling you

– come to an understanding of the state of the device or environment

• robot/autonomous vehicle, power plant or factory, automobile engine, biological weapons sensors, etc

– given the readings, what conclusions can be drawn?• this is typical a process of credit assignment – finding the cause of the

sensor readings• in a simple situation, the reading(s) points to a simple conclusion – that

is, a one-to-one mapping from sensor value to conclusion• but more often, each sensor reading is explained by a different

hypothesis, and the hypotheses have to be combined into a single, coherent and consistent explanation

– this can be accomplished through abduction• many approaches have been tried: knowledge-based, Bayesian

probabilities or Bayesian network, hidden Markov model, logic-based, case-based, even neural network approaches

Abduction Example• Given a set of findings, generate some hypotheses that could potential

explain the findings• Evaluate the hypotheses (how plausible are they?)• Select those hypotheses that best explain the findings while

maintaining a consistent and coherent explanation– Best includes such factors as maximally plausible, simplest (most

parsimonious), most complete

Edges indicate which data ahypothesis can explain

The dotted line indicatesmutually incompatible hypotheses

Sensor Fusion for Autonomous Car• Fusion – coming to a high-level understanding of the sensor input

values and how this relates to (or threatens) goal(s)– Obstacle detection– Environmental state (road conditions in this case)– Sensor malfunctions

• Sensors and sensor interpretation are often distributed, but sensor fusion is performed by a central controller/reasoner

Figure of the Alice Land-based Autonomous Vehicle

Automated Highways• Features

– Provide guidance information for cooperative (autonomous) vehicles– Monitor and detect non-cooperative vehicles and obstacles– Plan optimum traffic flow

• Architecture– Network of short-range hi-resolution radar sensors on elevated poles– Additional equipment in

vehicles (transponders for instance for location and identification)

– Sensors on the road for road conditions and on the vehicles for traction information

– Sensors for other obstacles (e.g., animals)

– Computer network– Roadway blocked off from

sidewalk and pedestrian traffic

Example AV Control Architecture

Automated Vehicles• Three different forms

– Remote controlled – largely of no interest since it involves little or no AI– Semi-autonomous – decision making performed remotely (by humans) but

the vehicle must still plan paths and avoid obstacles– Autonomous – all aspects controlled by computer (usually on-board)

• path planning, sensor interpretation, obstacle avoidance• goal/mission directed planning (achieving objectives while maintaining safety)• failure handling, on-board diagnostics

• Sensors depend on the type of vehicle/robot– Cameras are difficult unless used merely to determine obstacle/no obstacle– Radar, ladar (laser radar), sonar, microphones, ultrasound, other

• A fully autonomous vehicle will employ a number of AI techniques– Mathematical modeling for path planning– Rules or case-based reasoning for failure handling and goal planning– Bayesian probabilities, HMMs or neural networks for low level sensor

interpretation– Some form of abduction (possibly Bayesian) for sensor interpretation– Distributed control throughout the robot with a centralized controller to

issue commands and make high level decisions

Autonomous Land Vehicles• Types

– Mobile robots (indoor and level surface robots, outdoor/terrain robots)– Autonomous automobiles

• Indoor robots tend to be easier to implement– environment is “benevolent”– most interaction/obstacles involve walls and furniture (static) and humans

(the robot is usually programmed to stop and let the human pass)

• Autonomous cars have an even surface to work on but because of other drivers and the speed, they are very challenging – its easier if we can assume all cars are autonomous – then you don’t have unexpected vehicle movements and all vehicles can

directly communicate to each other

• Terrain vehicles have to deal with different types of surfaces and grades making the actual movement difficult and it requires more complex path planning– terrain vehicles are often used for exploration and might be used for

warfare in which case they also have to worry about enemy fire

Failure Handling: No Progress Forward

Other Autonomous Vehicles• Water-based (untethered) vehicles

– there are fewer obstacles to worry about so in fact these vehicles are easier to construct however, navigation is more problematic

• fewer (or no) landmarks, and currents can cause the vehicle to move in unanticipated ways

– these vehicles rely on sonar more than cameras– an underwater (submarine) vehicle deals with depth as well as <x, y>

coordinates• the greater range of mobility could make path planning easier in spite of it

being more complex• failure handling could be as simple as surfacing and moving in a circle, sending

out a signal that a problem has arisen

– most forms of UVs (underwater vehicles) are remote controlled as there is less call for autonomous UVs than land-based vehicles

• Flying vehicles (probably the least researched)– although they are similar to underwater vehicles in that they have current

(air current) and depth to worry about, but fewer obstacles– most flying vehicles are remote controlled (predator drones) but

autonomous helicopters are being researched

Autonomous Space Probes• To date, space probes (including the Mars rovers) have been semi-

autonomous at best– Mars rovers are partially controlled from Earth, but because of the time lag,

path planning and failure handling is performed on-board– other probes (Galileo, Cassini) are largely remote controlled with only on-

board diagnosis and other simple tasks done on-board• for instance, rotating the probe to point in the desired direction

• NASA wants more autonomy in their probes– their first attempt at this was the 2003 Earth Observing-1 satellite with

onboard continuous planning, robust task and goal-based execution, and onboard machine learning and pattern recognition

– to perform onboard decision-making to detect, analyze, and respond to science events, and to downlink only the highest value science data

– to manage event-driven processing/low-level autonomy, such as housekeeping routines

– to continuously schedule/plan, execute and replan for various actions as downlinking

Example• Here is an image from the

EO-1 satellite– As can be seen, the satellite’s

software automatically performs a variety of software tasks to detect events via image processing and feature detection, and plan to take a new image from the results

• EO-1’s science algorithms perform several operations:– analyze image data – image feature detection

including cloud detection– detect trigger conditions of

science events and changes relative to previous observations

– on-board image editing

Other Uses of AI in Space/NASA• Planning/scheduling:

– Manned mission planning– Multi-agent planning, distributed and shared scheduling, adaptive planning– Planning for geological surveys (for rovers)– Scheduling for observations (e.g., telescope usage)– Deliberation vs. reactive control/planning– Plan recovery (failure handling)– Conflict resolution for multiple spacecraft missions

• Life support monitoring and control– Simulations of life support systems– On-board diagnosis and repair– Safety (for humans, systems, rovers, probes)

• Science– Weather forecasting and warning, disaster assessment, disaster reduction

(for floods)– Feature detection from autonomous probes (e.g., crater detection by

satellites)– Other forms of visual recognition and discovery

Smart Environments• Sometimes referred to as smart rooms

– although these can be any environment (autonomous highway for instance)

• Components: collection of computer(s), sensors, networks, AI and other software, actuators (or robots) to control devices

• Goal: the environment can modify itself based on user preferences or goals and safety concerns– A smart building might monitor for break-ins, fire, flood, alert

people to problems, control traffic (such as elevator usage), etc

– A smart house might alter the A/C when people are away, adjust lighting, volume, perform household chores (starting/stopping the oven, turn on the dishwasher), determine when (or if) to run the sprinkler system for the lawn

– A smart restaurant might seat people automatically, have robot waiters, automatically order food stock as items are getting low (but not actually cook anything!)

Environment 1: Smart City Block

Environment 2: Smart Highway

Environment 3: Smart Room