Knapp 1

MATLAB Project I Single Layer Perceptrons

Michael J. Knapp

CAP6615, Neural Networks for Computing Department of Computer and Information Science and

Engineering University of Florida

Instructor: Dr. G. Ritter Department of Computer and Information Science and

Engineering University of Florida, Gainesville, FL 32611

Date: 4 October 2006

Knapp 2

I. Introduction

The overriding theme of this project is to train and test single layer perceptrons using Rosenblatt’s training algorithm. Since the project is very intensive on vector and matrix mathematics, MATLAB was chosen since it was originally designed as a matrix laboratory. MATLAB also includes powerful built-in graphing features that make it ideal for visualizing data. This also removes any cumbersome code from the project and allows the focus to be placed on the SLP itself.

Since Rosenblatt’s algorithm can be generalized from a single output, single layer perceptron to a multi-output, single layer perceptron I wanted to write a core SLP function that would accept the input data X and the desired output D and produce the matrix of weights based on the dimensions of the parameters. Since each part requires its own program, by using this centralized function each of the three parts of this problem becomes more focused on the process that part is asking for as opposed to being inundated with having three disparate incarnations of the SLP algorithm.

I also wanted to explore how varying α affected the number of epochs required by the algorithm. Therefore, before collecting data for the results section, I stepped through the algorithm from 1.0 down to 0.01 to gauge where the algorithm decreased in time to where the time began increasing again. I then intelligently chose α values to show this phenomenon.

For the run-time metric, I specifically chose CPU time over system time. This was done since in today’s multi-tasking computing environments there is no guarantee on how much CPU time an application will receive from the kernel. As such, MATLAB’s cputime feature was used to keep track of how much CPU time the MATLAB process received during an interval and not just the elapsed system time.

For the error during a training session, I looked at two interesting measures of error. The first was the general mean squared error for the ΔW, which is an indicator of how much, on average squared, the weights were adjusted for that epoch. This is a good measure of how far we are from convergence since it indicates the level of activity that epoch. The next error information I look at is the average of the M.S.E. for all epochs. This is a strong indicator of how fast we converged since if we spend a lot of time with a small M.S.E. the average M.S.E. gets smaller.

Depending on the behavior of each trial, different information was chosen to be important and only the important information is presented in this report. However, detailed information for each run of each trial at each epoch is available and a soft-copy can be provided to the reader if requested. Also, the complete code listing of the program from each part can be provided as well. II. Part A. a) Statement of the Problem

This part is to design a single layer, single output perceptron and to train it with Rosenblatt’s algorithm, that is able to classify the binary image representation of the character ‘A’ into one class and characters ‘B’, ‘C’ and ‘D’ into the class not ‘A’. After training is completed, we must verify the input weights correctly classify the data by verifying each of the characters classify correctly.

Knapp 3

b) Approach & Algorithm

To approach this problem, I decided to break the problem into two parts. The first part is general logical flow of the program as follows:

a. Translate the uni-polar matrices of ‘A’, ‘B’, ‘C’ and ‘D’ into vectors by a row-major translation.

b. Set the input to be A, B, C and D and the desired output to 1, 0, 0, 0 respectively

c. Take the current CPU time and start the SLP function d. Output the time after the function returns e. Verify the weights classify the data by setting the calculated

output Y equal to one if the dot product of the weight vector and the input vector is greater than or equal to zero, zero otherwise.

The second part is the generic SLP training algorithm function that was

described in the introduction and covered in detail in Appendix I. c) Results

For this portion, the important information chosen was just the total number of epochs the algorithm ran, the CPU time it took to run and which characters were successfully classified by the network after training. Since all of these trial run for so few epochs, and the algorithm converging guarantees a final error of zero, error was not deemed important for this part. If detailed information about all data from each epoch is desired. For α = 0.85 Trial # | Epochs | Elapsed Time (s) | Avg. MSE | Characters Classified ---------+--------+------------------+----------+----------------------- 1 | 9 | 0.046875 | 0.080772 | ‘A’ ‘B’ ‘C’ ‘D’ 2 | 9 | 0.046875 | 0.080772 | ‘A’ ‘B’ ‘C’ ‘D’ 3 | 6 | 0.046875 | 0.072991 | ‘A’ ‘B’ ‘C’ ‘D’ 4 | 10 | 0.015625 | 0.116489 | ‘A’ ‘B’ ‘C’ ‘D’ 5 | 12 | 0.093750 | 0.092258 | ‘A’ ‘B’ ‘C’ ‘D’ ---------+--------+------------------+----------+----------------------- Average | 9.2 | 0.0500 | 0.0887 | ‘A’ ‘B’ ‘C’ ‘D’ For α = 0.75 Trial # | Epochs | Elapsed Time (s) | Avg. MSE | Characters Classified ---------+--------+------------------+----------+----------------------- 1 | 6 | 0.031250 | 0.056827 | ‘A’ ‘B’ ‘C’ ‘D’ 2 | 6 | 0.015625 | 0.056827 | ‘A’ ‘B’ ‘C’ ‘D’ 3 | 6 | 0.046875 | 0.056827 | ‘A’ ‘B’ ‘C’ ‘D’ 4 | 6 | 0.046875 | 0.056827 | ‘A’ ‘B’ ‘C’ ‘D’ 5 | 6 | 0.046875 | 0.056827 | ‘A’ ‘B’ ‘C’ ‘D’ ---------+--------+------------------+----------+----------------------- Average | 6 | 0.0375 | 0.0568 | ‘A’ ‘B’ ‘C’ ‘D’

Knapp 4

For α = 0.5 Trial # | Epochs | Elapsed Time (s) | Avg. MSE | Characters Classified ---------+--------+------------------+----------+----------------------- 1 | 7 | 0.062500 | 0.037582 | ‘A’ ‘B’ ‘C’ ‘D’ 2 | 8 | 0.046875 | 0.035385 | ‘A’ ‘B’ ‘C’ ‘D’ 3 | 7 | 0.031250 | 0.037582 | ‘A’ ‘B’ ‘C’ ‘D’ 4 | 7 | 0.046875 | 0.037582 | ‘A’ ‘B’ ‘C’ ‘D’ 5 | 9 | 0.062500 | 0.040342 | ‘A’ ‘B’ ‘C’ ‘D’ ---------+--------+------------------+----------+----------------------- Average | 7.6 | 0.0500 | 0.0377 | ‘A’ ‘B’ ‘C’ ‘D’ For α = 0.2 Trial # | Epochs | Elapsed Time (s) | Avg. MSE | Characters Classified ---------+--------+------------------+----------+----------------------- 1 | 8 | 0.046875 | 0.007892 | ‘A’ ‘B’ ‘C’ ‘D’ 2 | 10 | 0.062500 | 0.008738 | ‘A’ ‘B’ ‘C’ ‘D’ 3 | 8 | 0.046875 | 0.007892 | ‘A’ ‘B’ ‘C’ ‘D’ 4 | 10 | 0.046875 | 0.008738 | ‘A’ ‘B’ ‘C’ ‘D’ 5 | 8 | 0.062500 | 0.007892 | ‘A’ ‘B’ ‘C’ ‘D’ ---------+--------+------------------+----------+----------------------- Average | 8.8 | 0.0531 | 0.0082 | ‘A’ ‘B’ ‘C’ ‘D’ For α = 0.1 Trial # | Epochs | Elapsed Time (s) | Avg. MSE | Characters Classified ---------+--------+------------------+----------+----------------------- 1 | 14 | 0.046875 | 0.002688 | ‘A’ ‘B’ ‘C’ ‘D’ 2 | 12 | 0.062500 | 0.002631 | ‘A’ ‘B’ ‘C’ ‘D’ 3 | 14 | 0.093750 | 0.002688 | ‘A’ ‘B’ ‘C’ ‘D’ 4 | 14 | 0.078125 | 0.002688 | ‘A’ ‘B’ ‘C’ ‘D’ 5 | 12 | 0.062500 | 0.002631 | ‘A’ ‘B’ ‘C’ ‘D’ ---------+--------+------------------+----------+----------------------- Average | 13.2 | 0.0688 | 0.0027 | ‘A’ ‘B’ ‘C’ ‘D’ d) Conclusion

The first point to note is how short all of the trials lasted. In the best case, where α = 0.75, only 1.5 passes were required through the input data and in the worst case where α = 0.1, barely over 3 passes were needed through the input data. This does however correspond to a factor of two increase in epochs ran. As one would expect, the elapsed time increased by a factor of just over 1.8 here as well. This is in line with epochs and elapsed time being linearly related.

One thing that easily derived from the algorithm, but worth mentioning is how the mean squared error of ΔW is always zero for a training point the last time the algorithm passes through it. This is intuitive since the algorithm has converged once a full pass can be made through the training points with no change in weights and hence no error. Here this meant that upwards of half the epochs in this part had no error since the trials were so short in duration.

Here, the error proved to be of particular interest. For each α, every case where the number of epochs to converge was the same, the average mean squared error was the same. This is counter-intuitive since the random initialization of the weights is supposed to add non-determinism to the algorithm, but this seems to be a sign of some determinism. I hypothesize this is due to the total number of epochs being so few that if the weights

Knapp 5

fall within a certain range it determines the number of epochs from the observed totals and forces the error to the range observed and if a higher precision were used, then the differences would be seen.

Finally, the idea introduced in class that varying α is somewhat of an art is reinforced here. The optimal α was found to be ~0.75 with a sharper increase varying α larger and a gradual increase α smaller. III. Part B. a) Statement of the Problem

This part asks us to generate a relatively large set of points that are separated by, and thus not falling on, a line. Since a line is linear, the two classes are linearly separable. Hence, we are designing a three-input, one output single layer perceptron, where one of this input is, as usual, x0 = 1. Once the network is trained, we are to generate a series of test points and make sure they are appropriately classified. If not, we add them to our training set and repeat. b) Approach & Algorithm

My approach was to first generate the 1000 points by generating the x values ~uniform [0,1] and then mapping them to [b1,b2). Once the x values were generated, I then generate the y values uniformly about the separating plane. This yields a healthy random population enclosed in a rectangle about the separating place. I then test the pairs with the equation of the separating plane to classify them as one if they are above the plane and zero if they are below the plane, and finally train the network.

Once the network is trained, I generate a set of test points ~uniform [b1,b2) and then generate their corresponding y value ~uniform [separating plane - 0.1, separating plane + 0.1] since the most difficult points to classify are near the plane. If at least one points fails classification, I add the entire test set to the training data and repeat the entire training process, if all classify I display the results.

To display the results, I employed MATLAB’s plot feature. I first plot all of the original training points. I then plot all of the test points disparately regardless of whether they failed classification and became training points. I then plot the separating plane and finally plot the learned separating plane according to the definition w0 + w1*x + w2*y = 0.

For this run, I felt that the mean squared error was not as important an error measure as the number of training points. Since 1000 points are used as per the project specification, any point over 1000 is a test point that was misclassified and forced the network to have to be retrained with the new set of points, which gives a real feel for the error as opposed to just listing a numerical mean squared error. c) Results

For reference, I have included an example output from trial 1 where α = 0.20. This is a good example because it not only shows how the data was represented visually, but it also shows how when more than 5 test points are needed how they are kept distinct from the original training data in the display.

Knapp 6

For α = 0.50 Trial # | Epochs | Elapsed Time (s) | Trn. Pts. | Equation Learned ---------+---------+------------------+-----------+-------------------------- 1 | 1445560 | 55.86 | 1000 | y = 0.508015*x + 1.517851 2 |16056640 | 565.33 | 1000 | y = 0.500945*x + 1.952659 3 | 617589 | 26.58 | 1000 | y = 0.500915*x + 1.928097 ---------+---------+------------------+---------- +-------------------------- Average | 6039929 | 224.78 | 1000.00 | y = 0.503292*x + 1.799536

Knapp 7

For α = 0.30 Trial # | Epochs | Elapsed Time (s) | Trn. Pts. | Equation Learned ---------+---------+------------------+-----------+-------------------------- 1 |29803146 | 1037.84 | 1000 | y = 0.500858*x + 1.958729 2 | 2087857 | 78.02 | 1000 | y = 0.500538*x + 1.980784 3 |10634854 | 373.00 | 1000 | y = 0.501288*x + 1.956244 ---------+---------+------------------+---------- +-------------------------- Average |14175286 | 496.29 | 1000.00 | y = 0.500895*x + 1.965252 For α = 0.20 Trial # | Epochs | Elapsed Time (s) | Trn. Pts. | Equation Learned ---------+---------+------------------+-----------+-------------------------- 1 |18960794 | 678.81 | 1005 | y = 0.501190*x + 1.887784 2 |13613850 | 485.11 | 1000 | y = 0.501521*x + 1.893265 3 | 8795781 | 314.70 | 1000 | y = 0.500414*x + 1.986367 ---------+---------+------------------+---------- +-------------------------- Average |13790142 | 492.87 | 1001.67 | y = 0.501042*x + 1.922472 For α = 0.10 Trial # | Epochs | Elapsed Time (s) | Trn. Pts. | Equation Learned ---------+---------+------------------+-----------+-------------------------- 1 |12409023 | 440.02 | 1000 | y = 0.502540*x + 1.873927 2 |10242057 | 365.23 | 1000 | y = 0.502869*x + 1.826852 3 | 2954272 | 109.16 | 1000 | y = 0.505318*x + 1.754559 ---------+---------+------------------+---------- +-------------------------- Average | 8535117 | 304.80 | 1000.00 | y = 0.503576*x + 1.818446 For α = 0.05 Trial # | Epochs | Elapsed Time (s) | Trn. Pts. | Equation Learned ---------+---------+------------------+-----------+-------------------------- 1 | 6086848 | 218.53 | 1000 | y = 0.502416*x + 1.839083 2 | 4968506 | 177.94 | 1000 | y = 0.502180*x + 1.847443 3 | 1005184 | 40.60 | 1000 | y = 0.507828*x + 1.509208 ---------+---------+------------------+---------- +-------------------------- Average | 4020179 | 145.69 | 1000.00 | y = 0.504141*x + 1.731911 d) Conclusion

This part really illuminated the effect of α on the network. Depending on the α chosen, the effect was making the network converge in longer times on average. Once again, we see what appears to be a non-linear effect of α on the run time. Also, based on the varying times for each α it appears that the initial values of W do effect the convergence as well; This was something not seen previously. The relation between α and the y-intercept seems to be parabolic, with a maxima when α = 0.30. The converse appears true for the slope of the line, where while it still appears parabolic, the minima is when α = 0.30.

Regarding the error, the most surprising thing was how few of the networks had to be retrained because of failing to classify the test points. At first I thought this to be an error with my code, so I tried only having 100 points to start with and found that indeed my code was correct. Given this, I feel that for a problem of this nature where there are only two linearly separable classes, 1000 training points are far more than needed. Also, although it is obvious, having to retrain a network is undesirable as it will then approximately double the training time since the original run was fruitless.

Knapp 8

Possibly a genetic algorithm could be used in these cases to attempt to use the resultant weights as the input to the next training attempt.

Finally, this is an excellent exercise to see visually, in two dimensions, how SLPs classify data. The neural network initially starts with random weights, having no idea where the separating plane lies and after some finite number of epochs, it is able to choose a plane that linearly separates the data. While simply finding a line may not seem impressive, I find it quite remarkable that knowing nothing other than some data points, nearly the exact function used to generate the data can be found. IV. Part C. a) Statement of the Problem

The problem here is to design a single layer, multi-output perceptron that classifies the characters ‘A’, ‘B’, ‘C’ and ‘D’ into four disparate classes using the generalized Rosenblatt algorithm. Once our network converged for the four classes, we were to add noise by flipping bits on an increasing percentage of bits until the algorithm no longer classified the data. b) Approach & Algorithm

For my approach, I studied the generalized Rosenblatt algorithm and read it as training one set of inputs fed to m, where m is the number of output nodes, single layer, single output perceptrons. For this problem I chose to stay with the same representation of the data outlines in section I. I also chose to take the problem at face value and have one output node per class and not employ and encoding to define the classes, this left the network with four output nodes.

For testing the data I felt it would better show the weaknesses of an artificial neural network (ANN) if instead of stopping the tests of the noise tolerance when one class failed, I stopped when all classes failed. I felt this would show if the ANN had a particular weakness and may bring to light class encoding to circumvent these weaknesses. The overall flow of the program was:

a. Create row-major vectors of uni-polar character matrices and create class vectors for desired output D.

b. Obtain the weight vector by running the SLP function on the aforementioned input, output pairs.

c. Initialize noise level to 0.00% d. Create a random mask, ~uniform [0,1], of the input vector’s size and

any value above the noise level sets the bit to one, zero otherwise e. Exclusive-or (XOR) the vector with the mask to toggle the bits, i.e.

simulate noise at the specified level. f. Perform a dot product and apply the hard-limiter to weights with the

noisy input to arrive at the output class values. g. Verify pattern classification for each character and display which

characters classified. If no characters classified correctly then exit, otherwise increase the noise level by 5.0% and go to d. Note the noise mask is kept constant at each noise level.

Knapp 9

c) Results For α = 0.80 Maximum Noise Tolerance Trial # | Epochs | Elapsed Time (s) | Avg. MSE | ‘A’ | ‘B’ | ‘C’ | ‘D’ ---------+--------+------------------+----------+------+------+------+------ 1 | 23 | 0.140625 | 0.083542 | 0.30 | 0.45 | 0.35 | 0.20 2 | 23 | 0.171875 | 0.083542 | 0.45 | 0.45 | 0.40 | 0.35 3 | 19 | 0.109375 | 0.092943 | 0.15 | 0.45 | 0.35 | 0.00 4 | 19 | 0.140625 | 0.092943 | 0.50 | 0.40 | 0.35 | 0.35 5 | 23 | 0.140625 | 0.083542 | 0.40 | 0.45 | 0.40 | 0.05 ---------+--------+------------------+----------+------+------+------+------ Average | 21.4 | 0.140625 | 0.087302 | 0.36 | 0.44 | 0.37 | 0.19 For α = 0.65 Maximum Noise Tolerance Trial # | Epochs | Elapsed Time (s) | Avg. MSE | ‘A’ | ‘B’ | ‘C’ | ‘D’ ---------+--------+------------------+----------+------+------+------+------ 1 | 19 | 0.140625 | 0.060912 | 0.30 | 0.25 | 0.25 | 0.15 2 | 19 | 0.093750 | 0.060912 | 0.35 | 0.35 | 0.15 | 0.15 3 | 19 | 0.140625 | 0.060912 | 0.40 | 0.30 | 0.15 | 0.15 4 | 19 | 0.109375 | 0.060912 | 0.20 | 0.30 | 0.20 | 0.15 5 | 19 | 0.125000 | 0.060912 | 0.30 | 0.30 | 0.20 | 0.15 ---------+--------+------------------+----------+------+------+------+------ Average | 19.0 | 0.121875 | 0.060912 | 0.31 | 0.30 | 0.19 | 0.15 For α = 0.50 Maximum Noise Tolerance Trial # | Epochs | Elapsed Time (s) | Avg. MSE | ‘A’ | ‘B’ | ‘C’ | ‘D’ ---------+--------+------------------+----------+------+------+------+------ 1 | 23 | 0.109375 | 0.031421 | 0.10 | 0.40 | 0.25 | 0.10 2 | 27 | 0.125000 | 0.029017 | 0.10 | 0.40 | 0.30 | 0.10 3 | 23 | 0.140625 | 0.031421 | 0.05 | 0.30 | 0.30 | 0.05 4 | 23 | 0.187500 | 0.028344 | 0.00 | 0.10 | 0.15 | 0.05 5 | 27 | 0.156250 | 0.023960 | 0.25 | 0.40 | 0.25 | 0.00 ---------+--------+------------------+----------+------+------+------+------ Average | 24.6 | 0.143750 | 0.028833 | 0.10 | 0.32 | 0.25 | 0.06 For α = 0.40 Maximum Noise Tolerance Trial # | Epochs | Elapsed Time (s) | Avg. MSE | ‘A’ | ‘B’ | ‘C’ | ‘D’ ---------+--------+------------------+----------+------+------+------+------ 1 | 17 | 0.093750 | 0.017759 | 0.25 | 0.20 | 0.00 | 0.00 2 | 13 | 0.046875 | 0.025723 | 0.15 | 0.05 | 0.00 | 0.15 3 | 27 | 0.156250 | 0.017605 | 0.30 | 0.30 | 0.30 | 0.20 4 | 17 | 0.109375 | 0.020047 | 0.20 | 0.15 | 0.00 | 0.20 5 | 26 | 0.140625 | 0.021889 | 0.10 | 0.25 | 0.20 | 0.00 ---------+--------+------------------+----------+------+------+------+------ Average | 20.0 | 0.109375 | 0.020605 | 0.20 | 0.19 | 0.10 | 0.11 For α = 0.30 Maximum Noise Tolerance Trial # | Epochs | Elapsed Time (s) | Avg. MSE | ‘A’ | ‘B’ | ‘C’ | ‘D’ ---------+--------+------------------+----------+------+------+------+------ 1 | 23 | 0.093750 | 0.015026 | 0.05 | 0.10 | 0.45 | 0.00 2 | 27 | 0.140625 | 0.014992 | 0.05 | 0.25 | 0.05 | 0.00 3 | 23 | 0.171875 | 0.015417 | 0.35 | 0.40 | 0.35 | 0.35 4 | 23 | 0.140625 | 0.015541 | 0.00 | 0.05 | 0.05 | 0.00 5 | 19 | 0.140625 | 0.017038 | 0.00 | 0.00 | 0.00 | 0.00 ---------+--------+------------------+----------+------+------+------+------ Average | 23.0 | 0.1375 | 0.015603 | 0.09 | 0.16 | 0.18 | 0.07

Knapp 10

d) Conclusion

The most interesting point I found was how the last element in the training set was the most susceptible to noise. Meaning, this was usually the first element to fail classification with increasing levels of noise. I surmise this is due to it being the last element to be accounted for on the first pass and that fact propagates through and manifests itself as stated. Also of note is that the second element is consistently the least susceptible to noise, which does not follow the converse of the logic used for the last element’s susceptibility.

One very interesting thing I found was that sometimes a character would not classify at one noise level and then later classify at a higher noise level. I would suspect this is due to where the noise is located. Looking at the matrices for the four characters, we see that many of the points stay the same between some classes. This would intuitively mean that the neural network has to classify on less than the full 18 x 18 points. So, if the noise is clustered in areas that are distinct between classes, as opposed to areas that are the same between classes.

I also noticed that these trials, even more so than in part A, had a tendency to take the same number of epochs given the α, however as α decreases the variance in epochs between trials increases.

I also noticed that the more epochs in a trial tended to make the classes less susceptible to error. This follows intuitively since each epoch brings us closer to the solution so if more epochs bring us closer to the solution, then we have more room for noise. Specifically looking at α = 0.40 trials 1 and 2 have low noise tolerance, but trial 3 has a much higher noise tolerance.

Once again, run-time and epochs seem linearly related as one would expect. Also as α decreases the average mean squared error decreases and not in correlation with the number of epochs. This seems counter-intuitive and will need further investigation. V. Conclusion

Neural networks bring something completely new to the table, non-determinism as well as non-linearity. They are able to start with no knowledge of a function used to generate training points and train to behave as a linear approximation to that function. Also, once the network is trained only a vector dot product and then Boolean condition is required to generate the output from the input. This is extremely powerful as a multiply-accumulate unit (MAC) can be driven at extreme speed, while implementing the actual function in more traditional methods can not. This allows data to flow forward in real-time. VI. References [1] Ritter, Gerhard X. (personal communication, August 30, 2006)

Knapp 11

VII. Appendix I The core SLP function’s MATLAB source code is provided here as a reference. Since this is the core functionality, it should be trivial to design the calling programs to use this function, which implements Rosenblatt’s training algorithm. % This is a function to train a SLP. It takes as input a unipolar vector X, % which is an [n,k] matrix where n is the number of inputs to the SLP and k % is the number of training sets. Y is [1,k] unipolar for the output class % and alpha is the learning parameter function W = slp(X, D, alpha) % Add x0 = 1 to input and get values for n, k, m [n,k] = size(X); X = [ones(1,k); X]; [n,K] = size(X); [m,k] = size(D); if (K ~= k) disp('Error: X and D do not match.'); return; end % 1. Initialize t=0, learning rate parameter a, and set weights W(0)=arbitrary values % i.e. wi(0)=arbitrary for i=0,...,n; t = 0; W = rand(n,m); % n dimensional column vector % Prepare for the loop (2-5) disp(sprintf('Epoch#\tM.S.E.\t\tElapsed Time (s)\tIterations w/o change')); not_done = k; % not_done = k since we need k iterations where weights dont change % 2. For each pair (xk,dk) from the training set, do Steps 3–5; time = cputime; % Get the current CPU time totalmse = 0; % total the M.S.E. maxrun = 0; % longest run without updating W while (not_done > 0) i = mod(t,k) + 1; % current index % 3. Compute ?W(t)=aekxk, i.e. ?wi(t)=aekxik for i=0,...,n, where % ek=dk-yk and yk=f(?i=0...nwi(t)xik); if (m > 1) y = ((W' * X(:,i)) >= 0); e = (D(:,i) - y); dW = (alpha * e * X(:,i)')'; else y = (dot(W, X(:,i)) >= 0); % optimized for speed dW = alpha * (D(i) - y) * X(:,i); end

Knapp 12

% 4. Increment t=t+1; t = t + 1; % 5. Update W(t)=W(t-1)+?W(t-1), i.e. wi(t)=wi(t-1)+?wi(t-1) for i=0,...,n; W = W + dW; % 6. If no weight changes occurred during last epoch (iteration of Steps 3–5), % or other stopping condition is true, then stop; otherwise, repeat from Step 2. if (nnz(dW) == 0) not_done = not_done - 1; else not_done = k; end if (cputime - time) > 3600 disp(sprintf('Giving up, over one hour of CPU time used.\n')); not_done = 0; end % Display data about this iteration thismse = mse(dW); % !!! Comment these out if runtime too long totalmse = totalmse + thismse; showstats = false; if (t <= 25) showstats = true; end if (maxrun < (k - not_done)) maxrun = (k - not_done); showstats = true; end if (showstats == true) disp(sprintf('%d\t\t%f\t\t%f\t\t%d', t, mse(dW), (cputime - time), (k - not_done))); end end disp(sprintf('Average M.S.E. is %f for %d epochs', (totalmse / t), t)); % Display mean M.S.E. disp(sprintf('Total CPU Time: %f (seconds).', (cputime - time))); % Display elapsed time

Knapp 13

VIII. Appendix II Experimental test bench specifications, which were partially generated using CPUID’s CPU-Z 1.37 available at http://www.cpuid.com/cpuz.php: - Processor(s) Number of processors 1 Number of cores 2 per processor Number of threads 2 (max 2) per processor Name Intel Core 2 Duo E6600 Code Name Conroe Specification Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz Package Socket 775 LGA Family/Model/Stepping 6.F.6 Extended Family/Model 6.F Core Stepping B2 Technology 65 nm Core Speed 2397.6 MHz Multiplier x Bus speed 9.0 x 266.4 MHz Rated Bus speed 1065.6 MHz Stock frequency 2400 MHz Instruction sets MMX, SSE, SSE2, SSE3, SSSE3, EM64T L1 Data cache 2 x 32 KBytes, 8-way set associative, 64-byte line size L1 Instruction cache 2 x 32 KBytes, 8-way set associative, 64-byte line size L2 cache 4096 KBytes, 16-way set associative, 64-byte line size - Chipset & Memory Northbridge Intel P965/G965 rev. C2 Southbridge Intel 82801HB (ICH8) rev. 02 Memory Type DDR2 Memory Size 1024 MBytes - System Mainboard Vendor Intel Corporation Mainboard Model DG965SS BIOS Vendor Intel Corp. BIOS Version MQ96510J.86A.1176.2006.0906.1633 BIOS Date 09/06/2006 - Memory SPD Module 1 DDR2, PC2-4300 (266 MHz), 512 MBytes, Mushkin Module 2 DDR2, PC2-4300 (266 MHz), 512 MBytes, Mushkin - Software Windows Version Microsoft Windows XP Professional Service Pack 2 (Build 2600) DirectX Version 9.0c MATLAB Version 7.2.0.232 (R2006a)