10/7/09

1

Lecture 20: String Processing

Overview •  Reading lines from text files

•  Processing strings – Breaking a string into tokens

– Comparing strings for equality

– Converting strings to numerals

Reading lines from text files •  Matlab's implementation of the fscanf function is

not really very useful –  all the values read have to be stored as data of the same

type within the returned array.

•  This limitation somewhat defeats the idea of being allowed to specify a format string.

•  The alternative approach to reading a text file is

–  to read each line of the file as a long string,

–  to break the string into tokens (the words, numbers, or punctuation in the string)

–  to process the tokens individually.

Reading lines of text • Matlab provides two functions for

reading lines of text from a file: % Read a line from a file *excluding* the end of

% line characters.

>> line = fgetl(fid)

% Read a line from a file *including* the end of

% line characters.

>> line = fgets(fid)

•  If the end of file is encountered, -1 is returned.

10/7/09

2

An example •  For example:

% DISPLAYFILE: A function to display the contents

% of a file.

%

% Usage: displayFile(filename)

%

% Arguments: filename - The name of the file to

% display.

%

% Returns: Nil.

%

% Author: Lecturers

% Date: Continuously Updated

Example (cont.) function displayFile(filename)

[fid, msg] = fopen(filename, 'rt'); error(msg);

line = fgets(fid); % Get the first line from % the file. while line ~= -1 fprintf('%s', line); % Print the line on % the screen. line = fgets(fid); % Get the next line % from the file. end

fclose(fid);

Processing Strings • Text: Chapter 6.2.

• Matlab's string processing functions are also closely modelled on their C equivalents.

Breaking a string into tokens •  Tokens are bits of a string - for example, the numbers, words, and

punctuation. •  Which “bits” are important depends on the context or application.

•  Eg. If you are parsing an English sentence you may just be interested in the words and not the punctuation (such as full stops):

•  Given “Life wasn’t meant to be easy. But should it be this hard?”

•  The tokens of interest are: Life wasn’t meant to be easy But should it be this hard

•  On the other hand, if you are parsing numbers, you may want to keep the full stops:

•  Given “3.245, 7.683, -24.7”

The desired tokens are: 3.247 7.683 -24.7, not: 3 245 7 683 24 7

10/7/09

3

Delimiters •  The things that are used to break up the text into tokens are

called delimiters.

E.g. If the delimiters are “, ” (comma and space) we get

3.247 7.683 -24.7

whereas if they are “, .-” (or all punctuation characters) we get 3 245 7 683 24 7

•  Tokens are typically delimited by spaces, tabs, or commas.

e.g. Given a string 'A string of 5 tokens + three more',

tokens are

'A' 'string' 'of' '5' 'tokens' '+' 'three' 'more'

•  Example: reading and writing text files with Excel...

The strtok function •  The strtok function is used to extract tokens from a string.

•  The general syntax of the strtok function is:

[token, remainder] = strtok(string, delim)

–  The variable string is the string to be "tokenised".

–  Returns the first token delimited by one of the characters in the delim string (the delimiters).

•  If delim is omitted, the tokens are delimited by any white space character by default.

•  After extracting the first token of string, the remainder of the string is returned as the separate string array remainder.

•  The extracted token is stored in the output variable token.

An example •  For example, imagine breaking a line of text into a cell array of tokens:

line = 'A string of 5 tokens + three more'; % Initialise or read a

% line from a file.

token={}; ii = 1;

while any(line) [token{ii}, line] = strtok(line);

% Repeatedly apply the

ii = ii + 1; % strtok function. end

•  The any function returns True if any element of a vector is non-zero.

•  At the end of the while loop, the token cell array contains all the tokens in the string.

>> token =

'A' 'string' 'of' '5' 'tokens' '+' 'three' 'more'

Comparing strings for equality

• Two strings should never be compared for equality using the == operator.

• The == operator will return an error if its two arguments are not of the same length.

•  Even if the strings are the same length, the == operator will return an array of boolean values depending on how individual characters match, not a single boolean answer.

10/7/09

4

String Comparison Functions •  Strings in Matlab should be compared using the following functions:

>> c = strcmp(str1, str2) % Returns True if str1 is

% identical to str2.

>> c = strcmpi(str1, str2) % Compares strings ignoring % case differences.

>> c = strncmp(str1, str2, N) % Compares the first N % characters of the strings.

>> c = strncmpi(str1, str2, N) % Compares the first N % characters of the strings % ignoring case differences.

•  Note that the standard C/Java language functions differ in that they return 0 if the strings are identical, a negative value if the first string is alphabetically less than the second string, and a positive value otherwise.

Converting strings to numerals •  The function str2num will evaluate a string as a

Matlab expression, converting it to a number.

•  For example:

>> x = str2num('21')

x = 21

>> x = str2num('1+2')

x =

3

Str2num and eval •  Note however, that spaces can be significant.

>> x = str2num('1 +2') % This is interpreted as

% an array containing two % numbers: 1 and +2. x = 1 2

•  The eval function is similar, but perhaps more predictable. >> x = eval('1 +2') x = 3

>> x = eval('[1 2]') x = 1 2

Summary • The operations described in this lecture form

the basic building blocks that allow you to process text files:

– Reading values from files.

– Reading lines from files.

– Breaking lines into tokens.

– Comparing strings.

– Converting strings to values.

10/7/09

5

Putting it altogether in one example

• Here is a text datafile that specifies the property of 3 beams.

CITS1005BEAM % Magic ID value % % This is a datafile to define a series of beams % File must start with the appropriate Magic value % Comments are indicated by `%' % Number of beams is specified by the keyword `Nbeams' % followed by a value. % % Each beam definition is started by the keyword 'beam' % Fields of `length', `section_area' and `material' % must be specified, % but can appear in any order

Example (cont.)

Nbeams 3 % Number of beams

beam % Start of beam definition length 100 section_area .5 material steel

beam % Start of beam definition length 8 section_area .04 material wood

beam % Start of beam definition

% Oddly formatted, but valid, beam specification material rubber length 1 section_area .3

Design your solution •  The structure of this file has a couple of features

–  The data file starts with a Magic ID value which is used to uniquely identify the type of data file. Any program written to read these kinds of `beam files‘ should check that the first line starts with this Magic ID. This way the program can reject non beam files.

–  The file contains some keywords that establish the context as to how subsequent groups of data should be read. This allows errors to be detected in the reading process.

•  The reading states we expect are: 1.  Start with expecting to see a Magic ID. 2. Then expect to see Nbeams sepcified. 3. Then expect to see a `beam' keyword. 4. Then expect to have length, section_area and material specified. 5. Then expect to see another `beam' keyword etc, or the end of file.

Operations to use • We use the basic operations of:

– Reading lines from files – Breaking lines into tokens – Processing tokens by string comparison or

conversion into values

% readbeamfile - reads textfile specifying beam structures % % Usage: beam = readbeamfile(filename) % % Returns an array of beam structures each having fields of % `length', `section_area', and `material'

10/7/09

6

Example (cont.) function beam = readbeamfile(filename)

% Define some values to indicate reading states - this allows one % to structure the code logically

waitingForID = 0; % These numbers are arbitrary waitingForNbeams = 1; % - they just have to be unique. waitingForBeam = 2; waitingForData = 3;

% State transitions can only be as follows: % % waitingForID -> waitingForNbeams -> waitingForBeam -> waitingForData -- % ^ / % |______________________________/

[fid, msg] = fopen(filename, 'rt'); error(msg);

Example (cont.) state = waitingForID; % Initialise state beamNo = 0; % Initialise beam index line = fgets(fid); % Get first line while line ~= -1 % While there is still data to read

remainder = line;

while any(remainder) [token, remainder] = strtok(remainder);

if isempty(token) % No tokens left on this line break; % break out of while loop and % go to next line

elseif strncmp(token, '%', 1) % This is a comment, skip what is left break % and go to next line.

elseif state == waitingForID

Example (cont.) if ~strcmpi(token, 'CITS1005BEAM') % Check for file type fclose(fid); error('This is not a CITS1005BEAM data file'); end state = waitingForNbeams; % State transition

elseif state == waitingForNbeams

if strcmpi(token, 'Nbeams') [token, remainder] = strtok(remainder); % Next token is the value Nbeams = str2num(token);

% Allocate memory for struct array beam = struct('length', cell(1,Nbeams), ... 'section_area', cell(1,Nbeams), ... 'material', cell(1,Nbeams));

state = waitingForBeam; % State transition else fclose(fid); error('Unexpected data in file'); end

Example (cont.) elseif state == waitingForBeam

if strcmpi(token, 'beam') beamNo = beamNo + 1; % Increment beam count state = waitingForData; % State transition else fclose(fid); error('Unexpected data in file'); end

elseif state == waitingForData % Fill in the beam data fields

if strcmpi(token, 'length') [token, remainder] = strtok(remainder); % Next token is the value beam(beamNo).length = str2num(token);

elseif strcmpi(token, 'section_area') [token, remainder] = strtok(remainder); % Next token is the value beam(beamNo).section_area = str2num(token);

elseif strcmpi(token, 'material') [token, remainder] = strtok(remainder); % Next token is the value beam(beamNo).material = token;

else fclose(fid); error('Incomplete beam data, or unexpected data in file'); end

10/7/09

7

if beamSpecified(beam(beamNo)) % We have all the data state = waitingForBeam; % Go onto next beam else state = waitingForData; % Keep looking for data for this beam end

end end

line = fgets(fid); % Get next line end

% Check we got all the beams

if beamNo ~= Nbeams fprintf('Data for %d beams were read, %d were expected', beamNo, Nbeams); end

fclose(fid);

% Internal function to check that all fields of a beam structure have been set

function v = beamSpecified(b) v = ~isempty(b.length) & ~isempty(b.section_area) & ~isempty(b.material);

Example (cont.)