Chapter 9 Negnevitsky

8/2/2019 Chapter 9 Negnevitsky

1/72

Knowledge Engineering and Data Mining

Jiajuan LinJohn LarimerLopa NathNick RemishSean Ruck

Tyler Purdom


2/72

What Is Knowledge Engineering?

John Larimer


3/72

Knowledge Engineering

It is the process of building intelligent systems.

1. Problem Assessment

2. Data & knowledge acquisition

3. Prototype

4. Complete System

5. Evaluation & Revise6. Integration & Maintain System


4/72


1. Problem Assessment

Determine the problem's characteristics Identify the main participants in the project

Specify the project's objectives

Determine the resources need for building the systemTypes of problems: Diagnosis, selection, prediction,classification, clustering, optimization, control

E.g. Diagnosis -> domain knowledge, explanation facilities


5/72


2. Data and Knowledge Acquisition

Collect and analyse data and knowledge Make key concepts of the system design more explicit

Intelligent System


6/72


3. Development of a Prototype System

Choose a tool for building an intelligent system Transform data and represent knowledge Design and implement a prototype

Test the prototype


7/72


4. Development of a complete system

Prepare a detailed design for a full-scale system Collect additional data and knowledge Develop the user interface

Implement the complete System


8/72


5. Evaluation and revision of the system

Evaluate the system against the performance criteria Revise the system as necessary


9/72


6. Integration and Maintenance of the System

Make arrangements for technology transfer Establish an effective maintenance program


10/72

Case Study 1: Diagnostic Expert System

I want to develop an Intelligent system that can help me to fix malfunctions of my Mac computer.

Will an expert system work for this problem?


11/72


Phone Call Rule (Firebaugh, 1988) - "Any problem that can be solved by your in-house expert in a 10-30 minute phone call can be developed in an expert system."

Troubleshooting manuals Series of visual inspections Rule structure with domain knowledge


12/72


Taken from N. Ch9 Pg 309 & 310


13/72


14/72

Case Study 2: Classification E.S.

Problem: Identify different classes of sailboat (typical classification problem)o Handled well by both expert systems and NNs

Collect informationo In this case, the sail plan can help identify the class of boat.

Issues with the expert system approacho What if the information is incomplete or inexact? (Rough weather obscuring sails)

Manage incrementally acquired evidence with certainty factors.


15/72

9.3 Will a fuzzy expert system work for my problem?

A Fuzzy Solution?o Useful when you cannot define a set of exact rules.o Great for inherent imprecise properties and modeling human decision making.

Sometimes parameters are imprecise (a doctor dealing with a patient)o Mainly used in engineering, but has applications in any sector that relies on human

experience or is too complex or uncertain (ex: finance)


16/72

Case Study 3: Decision-Support Fuzzy System


17/72

Decision-Support Fuzzy System

Problem: assessing mortgage applicationso Use a Decision-Support Fuzzy System

Steps:o Represent the concept in fuzzy termso Implement the concept in a prototypeo Test and optimize


18/72


Represent the concept in fuzzy terms:

Triangular and trapezoidal fuzzy membership functions are used to represent knowledge.


19/72


Obtain the fuzzy rules:Based off of Von Altrock's

fuzzy model and appliedto mortgages.


20/72


Evaluate and analyse performance:

Despite having 100+ rules, decision-support fuzzy systems can be developed, tested andimplemented relatively quickly.


21/72

9.4 Will a neural network work for my problem?

Sean Ruck


22/72

Neural Network Overview

Very powerful, general purpose tools Successfully applied to prediction, classification, and clustering problems Quite popular due to the versatility of neural networks


23/72

Case Study 4: Character Recognition Neural Networks

Suppose you want to copy a document onto your computer without retyping the whole thing.o How?

Optical Character Recognitiono The ability of a computer to translate character images into a text file using software

o Capture the character images by scanning the document Converts the scanned document into a bit map


24/72

Choosing The Neural Network Architecture

Architecture and size of neural newtork dependent upon complexity of the problemo Handwritten character recognition is far more complex than computer printed

A 3-layer network will suffice for printed digit recognition


25/72

Determining an optimal number of hidden neurons

More neurons leads to a more accurate network, but takes longer to train Too many neurons may actually prevent the network from generalising or working for anything

other than training exampleso Overfitting

How to prevent overfittingo Choose the smallest number of neurons that give good results and generalisation


26/72

cont'd.

We should test out the training of the network with various numbers of hidden neuronso Performance rated by sum of squared errors

The training runs that have a good enough sum of squared errors result have anumber of hidden neurons to consider using


27/72

Test Examples?

The test set should be entirely independent of the training exampleso Only use the training runs that passed the previous test

Test examples should also contain "noise"o Distortion of the input

The training runs that give us a reasonable error in recognition even with noise have a goodenough number of hidden neurons to useo Use the lowest number for practical purposes


28/72

Improving Performance

A neural network is only as good as the examples used to train it Improve the network by training it with noisy examples


29/72

Case Study 5: Prediction Neural Networks

Neural networks are useful in prediction situations such as predicting the market value of a house Using a neural network creates a black box around how the results were reached

o The result is more important than the how anyway

For prediction training examples are critically importanto We need a wide array of examples to cover all possible inputs


30/72

Determining The Size Of A Training Set

Can be estimated with "Widrow's rule of Thumb": N = nw/eo Where N is the number of training examples, nw is the number of synaptic weights in the

network, and e is the network error permitted


31/72

Dealing With The Data

Neural networks work best with inputs in the 0 to 1 range, but in cases such as with determiningthe value of a house, our inputs are not all in that rangeo Number of bedrooms, square footage, etc.

So we need to "massage" the data to this rangeo massaged value = (actual value - minimum value) / (maximum value - minimum value)o Good for up to a dozen possible values


32/72

cont'd

We can also utilize 1 of N codingo Each possible value is taken as its own input each with a value of 0 or 1


33/72

Dealing With The Results

To validate the results we test the network with never before seen examples, as before Our network is working with values between 0 and 1. We need to convert back to actual values

o We can reverse the "massaging" we did before

To test the importance of certin inputs we can test the network's sensitivity to them: "SensitivityAnalysis"o Set each input one at a time to its minimum and maximum values and measure the results


34/72

Case Study 6: Classification Neural Networks With Competitive Learning

Using a neural network we can discover significant features of input patterns and separate thedata into different classes

Using competitive learning a single layer neural network can perform clusteringo Combining similar data into groups or clusterso Uses 1 input neuron for each input and 1 competitive neuron for each cluster


35/72

When Is The Learning Process Completed?

In a competitive neural network, there is no obvious way to know if the network is done learningo We do not know the desired output, so we cannot use the sum of squared errors

Use Euclidean Distance criterion insteado When there has been no noticeable change in the weights of the competitive neurons, the

network can be considered to have converged


36/72

How Can We Associate Neurons to Specific Classes or Clusters?

Competitive neural networks enable us to identify clusters in input data, but does nothing to labelthe clusterso We can connect a competitive neuron with a cluster/class by analyzing its weights

We can identify exactly which cluster is which by feeding the network test data corresponding toone particular cluster

o The output neuron that most often is utilized is labeled as that class


37/72

9.5 Will genetic algorithms work for my problem?

Lopa Nath


38/72

Genetic Algorithm Review

Most applicable to optimization problemso Process of finding a better solution to a problem

More than one solution not of equal quality

Generates a population of competing candidate solutions Causes candidates to evolve through process of natural selection

o Poor solutions die out while better solutions survive and reproduce

Process repetition breeds an optimal solution


39/72

Case Study 7: The Traveling Salesman Problem

I want to develop an intelligent system that can produce anoptimal itinerary. I am going to travel by car and I want to visitall major cities in Western and Central Europe and then returnhome. Will a genetic algorithm work for this problem?o Known as the traveling salesman problem (TSP)

o Given a finite number of cities, and the cost of travel (or the distance) betweeneach pair of cities, we need to find the cheapest way (or shortest route) forvisiting each city exactly once and returning to the starting point.

o TSP naturally represented in numerous transportation and logisticsapplications.

Arranging routes, scheduling drilling of holes in a circuit board (timeefficient - shortest distance)

Although we can not be completely sure if the selected routeis the best one, after several runs we can be sure that theroute obtained is a good one.


40/72

How does a genetic algorithm solve the TSP?

Representation

Chromosome where order of integersrepresents order in which cities will bevisited.


41/72

Genetic Operators in the TSP

Genetic operators used to create new routes Crossover Operatoro Classical form cannot be directly applied because a simple exchange

of parts between parents would contain duplicates and omissions.

Clearly classical crossover with single crossover point does not work.


42/72

How the Crossover Operator Works


43/72

Genetic Operators in the TSP Continued

Mutation Operatoro Reciprocal Exchange

Simply swaps two randomly selected cities in the chromosome

o Inversion Selects two random points along the chromosome string and reverses order of cities

between these points


44/72

Fitness Function in the TSP

Evaluate total length of the routeo Fitness of each individual chromosome is determined as the reciprocal of the route length

Shorter the route, fitter the chromosome


45/72

9.6 Will a hybrid intelligent system work for my problem?


46/72

Hybrid Intelligent Systems

Solving complex real-world problems require an application of complex intelligent systems thatcombine the advantages of expert systems, fuzzy logic, neural networks, and evolutionarycomputation.

Such systems can integrate human-like expertise in a specific domain with abilities to learn andadapt to a rapidly changing environment.


47/72

Case Study 8: Neuro-fuzzy decision-support systems

I want to develop an intelligent system for diagnosing myocardialperfusion from cardiac images. I have a set of cardiac images aswell as the clinical notes and physician's interpretation. Will ahybrid system work for this problem?o Analysis of two SPECT images must be done

One stress image taken 10-15 minutes after injection with radioactive tracer

One rest image taken 2-5 hours after the injectiono Brighter patches on image correspond to well-perfused areas while darker patches

may indicate the presence of an ischemia.

o Visual inspection is highly subjective--intelligent system can help a cardiologistdiagnose.

o One binary feature assigns an overall diagnosis--normal or abnormal

The neuro-fuzzy system in this example has a heterogeneous

structure - the neural network and fuzzy system will work asindependent components but cooperate in solving the problem.


48/72

Back-Propagation Neural Network to Classifythe SPECT Images into Normal and Abnormal

Each image is divided into 22 regions, so we need 44 input neurons. Since SPECT images are to be classified as either normal or abnormal, we should use two output

neurons.

Good generalization in this study can be obtained with 5 to 7 neurons in the hidden layer.


49/72

Testing the Neural Network

Testing the network, we find the network's performance is rather pooro 25% normal are misclassified as abnormalo Over 35% abnormal are misclassified as normalo Indicates that the training set may lack some important examples

Can improve this still


50/72

Neural Network Output

Two outputso First - possibility that the SPECT image belongs to class

normalo Second - possibility that the SPECT image belongs to class

abnormal Examples:

o NORMAL OUTPUT HIGH AND ABNORMAL OUTPUT LOWFirst (normal) output is 0.92 and second (abnormal) is 0.16 -image classified as normal - risk for heart attack is low

o NORMAL OUTPUT LOW AND ABNORMAL OUTPUT HIGHFirst (normal) output is 0.17 and second (abnormal) is 0.51 -image classified as abnormal 0 risk for heart attack is high

o BOTH OUTPUTS ARE CLOSE First (normal) output is 0.51and second (abnormal) is 0.49 - we cannot confidently classifythe image.


51/72

Adding Fuzzy Logic for Decision-Making in Medical Diagnosis

Fuzzy logic provides us with a means of modeling how thecardiologist asses the risk of a heart attack. Need to determine input and output variables, define fuzzy sets,

and construct fuzzy rules.o Two inputs (NN output 1 and NN output 2) and one output (the risk of

a heart attack).

Inputs [0, 1] and output vary between 0 and 100 percent.o Fuzzy sets shown in Negnevitsky page 342 and 343 - Figure 9.33,

Figure 9.34, and Figure 9.35o Fuzzy rules in Negnevitsky page 343 - Figure 9.36

Examples:1. If (NN-output1 is Low) and (NN_output2 is Low) then (Risk is Moderate)

2. If (NN-output1 is Low) and (NN_output2 is Medium) then (Risk is High)3. If (NN-output1 is Low) and (NN_output2 is High) then (Risk is Very_High)4. If (NN-output1 is Medium) and (NN_output2 is Low) then (Risk is Low)


52/72

More Certainty

Risk between 30 and 50 percent cannot be classified as either normal or abnormal - uncertain. Apply the following heuristics known by experienced cardiologists to all corresponding regions (22in each image)

1. If perfusion inside region iat stress is higher than perfusion inside the sameregion at rest, then then risk of a heart attack should be decreased.

2. If perfusion inside region iis not higher than perfusion inside the same region atrest, then the risk of a heart attack should be increased.


53/72

Three Heuristics Implemented In the Diagnostic System

Step 1 Present the neuro-fuzzy system with the cardiac case.

Step 2 If the system's output is less than 30, classify the presented case as normal and then stop. If the outputis greater than 50, classify the case as abnormal and stop. Otherwise go to step 3.

Step 3 For region 1, subtract perfusion at rest fro perfusion at stress. If the result is positive, decrease the

current risk by multiplying its value by 0.99. Otherwise, increase the risk by multiplying its value by1.01. Repeat this procedure for all 22 regions then go to Step 4.

Step 4 If the new risk value is less than 30, classify the case as normal; if the risk is greater than 50, classifythe case as abnormal; otherwise, classify the case as uncertain.

When we now apply the test set to the neuro-fuzzy system, we find that theaccuracy of diagnosis has dramatically improved - the overall diagnostic error doesnot exceed 5 percent, while only 3 percent of abnormal cases are misclassified asnormal.

Although we have not improved the system's performance on normal cases (over 30percent of normal cases are misclassified as abnormal), and up to 20 percent of thetotal number of cases are classified as uncertain, the neuro-fuzzy system canactually achieve even better results in classifying SPECT images than a cardiologistcan.


54/72

Homogeneous Structure of Neuro-Fuzzy Systems

A typical example of a neuro-fuzzy system with a homogeneous structure is an Adaptive Neuro-Fuzzy Inference System (ANFIS).o It cannot be divided into two independent distinct parts.o An ANFIS is a multilayer neural network that performs fuzzy inferencing.

Case Study 9: Time series predictiono Page 346 of Negnevitsky


55/72

Data Mining and Knowledge Discovery

Tyler Purdom


56/72

Data Mining

Definition:o The extraction of knowledge from datao The exploration and analysis of large quantities of data to to discover patterns.

Ultimate goal is to discover knowledge Amount of data doubles every year Important to have fast algorithms to process data


57/72

Data Warehouses

Definition:o Large databases that store historical data.o Contain millions and in some cases billions of data records.

The data stored is time dependent and integrated Used to help support decision making Query tools are used to discover relationships in the data.


58/72

Query Tools vs. Data Mining

Query tools are assumption-basedo User must ask the right questions to get resulto User must make assumptionso Can select a specific variable that affects the outcome

Data Mining tools determine the most significant factorso No assumptions are necessaryo Discovers patterns automatically

The representation of data in data warehouses helps facilitate the data mining process


59/72

Data Mining Practice

Data Mining is a new and evolving field Very popular in the banking, finance, marketing, and telecommunications industries Data Mining uses:

o Determine trends in marketso Detect fraudso Target people most likely to buy a product/use a service


60/72

Data Mining Tools

People used to use query tools and statistics to solve data mining problemso These techniques are not very efficient for large amounts of datao Can only correlate a few variables at a time

Now, tools are based off of intelligent technologies:o Neural networks, neuro-fuzzy systems, and decision trees

Decision trees are currently the most popular tool used for data mining


61/72

Decision Tree

A map of the reasoning process These trees do not allow for the use of noisy or incomplete data Uses tree structure to describe the data set Very effective in solving classification problems Popular because they help you visualize the problem Nodes are separated by predictors

o In the book example, homeownership was used to split the tree


62/72

Decision Tree Example


63/72

Gini Coefficient

A measure of how well the predictor separates the classes contained in the parent node Introduced by Corrado Gini, an Italian economist He used it to measure the inequality in Italy's income distribution


64/72

Calculating the Gini Coefficient

Top curve represents the real economy Bottom line represents equal distribution ofwealth

Coefficient:o (shaded area)/ area below bottom

line


65/72

Gini Split Example


66/72

Summary

Jiajuan Lin


67/72


68/72


69/72

Summary - Data & Knowledge acquisition

Questions about the datao Range? Continues? Discrete? Precise? Noise Tolerance? Numerical? Symbolic? Data Mining

o analyze data, finding pattern & rules, extracting knowledge from large quantities of datao decision tree

easy to follow

visualization of solution makes clear sets of rules


70/72

Summary - Prototype

shows understanding ofo the problemo problem-solving strategy

o tool selected

Testo Throw it away if needed

o Forcing wrong tool leads to more time waste in the later development process

o Prototype is there for discovering any inappropriate/wrong decisions made


71/72

Summary -

Complete System,Evaluation ,Revision, Integration & maintenance

Complete System Developmento plan, schedule, budget Evaluation

o no clear right/wrongo user satisfaction = measurement

Revisiono Modify as limitation & weaknesses discovered

Maintenanceo Knowledge evolves over timeo keep modifying and updating to maintain efficiency and accuracy


72/72

Date post:	05-Apr-2018
Category:	Documents
Upload:	lopanath
View:	231 times
Download:	0 times

Chapter 9 Negnevitsky

Documents