Rajesh Ramasubramanian MSc in Information...

Dissertation by

Rajesh Ramasubramanian

MSc in Information Systems

Identifying Issues in Quality of Management Processes in the Development of Information Systems

Supervisor: Dr. Miguel Nunes

Department of Computer Science, University of Sheffield

This report is submitted in partial fulfillment of the requirement for the degree of MSc

in Information Systems

1

ABSTRACT

This dissertation focussed on evaluating quality practices for an information

systems project. A case study approach was followed for the purpose and it was

undertaken with the South Yorkshire office of British Telecom (BT) based in

Sheffield. The system under review was called Training and Development Live,

which was developed in-house to monitor, track, and plan training for field

engineers with BT Field Service.

The research began with a detailed literature review on quality in information

systems projects. This was presented in two chapters; the first gave a detailed

view of quality and the second concentrated on the framework used for the

purpose of this dissertation i.e. ISO 9000 framework.

Once the above was done, the empirical research was conducted through a

series of interviews with key stakeholders and members of the project team. The

interview questionnaires were prepared using the knowledge gained through the

literature and through the ISO 9000 framework.

The information gained through these interviews were analysed using concept

maps performing a vertical and horizontal analysis to gain a broader

understanding of the pitfalls associated with the development process.

Finally, the recommendations were presented after a comparison of the process

followed in the company with that recommended by the ISO 9000 framework. It is

sincerely hoped that the company would benefit out of the results that arose from

the process assessment

2

ACKNOWLEDGEMENTS

I am obliged to those who have provided support and guidance in this project. This report would be incomplete without mentioning about those people who, with their help have made this task successful. First and foremost I would like to express my deepest sense of gratitude to my supervisor, Dr. Miguel Nunes, for providing me with valuable guidance, esteemed suggestion and constant encouragement, which led to the successful completion of this project. I would also like to thank my friends back home for providing support and encouragement throughout my stay in Sheffield.

3

TABLE OF CONTENTS 1. Introduction

1.1 Background 6

1.2 Research Topic 7

1.3 Objectives 8

1.4 Ethical Issues 8

1.5 Methodology 9

1.6 Research Strategy 9

1.7 Limitations 10

2. Quality in Information Systems Projects 2.1 Introduction 11

2.2 Importance of Quality 14

2.3 Quality Management 16

2.4 Quality Planning 18

2.5 Quality Assurance 20

2.5.1 Testing 21

2.6 Conclusion 28

3. The ISO Standards for Quality Management & Quality Assurance 3.0 Introduction 29

3.1 ISO 9000 30

3.2 Importance of ISO 9000 30 3.3 ISO 9000-3 – A Tool for Software Product & Process

Improvement 32

3.4 ISO 9000-3 Scope and Overview 34

4. Methodology 4.0 Methodology 39

4.1 Qualitative Research Method 39

4.1.2 Motivation 40

4

4.2 Research Strategy 41

4.2.1 Interview Strategy 42

4.2.2 The Case Study 43

4.3 Training and Development Live 45

5. Data Analysis

5.0 Analysis of Data 47

5.1 Interview One – Developer 52

5.2 Interview Two – Project Manager 61

5.3 Interview Three – User One 69

5.4 Interview Four – User 2 76

5.5 Conclusion 82

6. Discussion of Results and Recommendations

6.0 Discussion of Results & Recommendations 84

6.1 Product Specification and Preliminary Planning 85

6.2 Engineering Specification and Planning 90

6.3 Product Design and Implementation 96

6.4 Verification and Validation 102

6.5 Product Release 110

6.6 Conclusion 116

7. Conclusions and Future Work

7.0 Summary 117

7.1 Future Work 119

Bibliography 120

5

Chapter One: Introduction

Chapter One Introduction

1.1 Background

The need to manufacture high quality products poses a challenge to every

industry today and hence affects all kinds of projects and project management.

Today, companies use quality as an important differentiator. (Lockyer and

James, 1996)

However Card and Glass (Card and Glass, 1990) argue that quality is not merely

about having checks to control the finished product. Many companies employ

quality assurance techniques aimed at identifying and resolving problems in

software products but very few practice quality controls which are aimed at

comparing observed quality with expected quality and correcting the source of

those problems i.e. the process problems.

It is widely accepted that manufacturing a good quality product would be easier if

the process used to manufacture that product maintained a certain acceptable

standard. Thus maintaining the quality of the process is a very important step in

achieving reliable and error free end products. Thus emphasis should be

provided on controlling the production process rather than only testing the end

product. (Lockyer and Gordon, 1996: 35)

Lock (Lock, 1997: 6) takes this argument further when he points out that over the

years, the emergence of practices such as total quality management (TQM) has

resulted in severe competition amongst companies in every domain and they are

now seeking to get themselves quality certified in standards such as ISO 9000,

6


BS: 5750 etc to prove that quality principles for project management are being

applied correctly.

This dissertation focuses on evaluating those quality practices in an Information

System project. The emphasis of the dissertation is on the quality practices of

the process rather than the product. This would be determined through a process

assessment using the ISO standards.

1.2 Research Topic

This thesis is primarily concerned with the evaluation of quality practices in

Information Systems projects. This is believed to be a particularly valuable and

somewhat under developed area of academic research. The research topic is

formulated as:

“Identifying issues in quality of management processes in the development of

Information Systems”

The reason for choosing the above topic stems from the challenge of finding the

right organizational framework, the right process model, the right methodology,

the right supporting methods and techniques and the right mix of skills for a

development team.

The researcher’s personal experience of working as a programmer for two years

showed that the right mix of the above factors was very difficult to achieve for any

company.

It is however an exciting and satisfying area of research to find out the issues

related to the software development process in companies.

7


Software development is a people process and due consideration must be given

to all the players involved. Process improvement and implementation concerns

people and needs to take into account all people related aspects.

Companies should be able to translate their software engineering practices into

their competitive advantage.

1.3 Objectives The key research objectives are identified as follows:

• Undertake a thorough study and analysis of the literature relating to quality

management

• Undertake a thorough study of the literature relating to the ISO standards

since this happens to be the framework for this dissertation.

• Learn about the organisational context of quality processes with respect to

the case study in question

• Understand the quality processes involved in the project;

• Identify and understand the issues (gaps and loopholes) including

associated with each of the phases in the project.

• Provide recommendations to prevent the quality issues occurring in the

future; and

• Contribute to the academic knowledge about quality management in

information systems projects.

1.4 Ethical Issues

The proposed research involves human participants or data relating to humans.

This would be through in-depth individual interviews, and if possible, focus

groups. It may also be necessary to spend time with people who are involved in

setting up and maintaining the concerned process. . Access to the company staff

8


at different levels has been granted. I confirm that I have read and will abide by

the University’s Ethics Policy for Research Involving Human Participants, Data

and Tissue.

1.5 Methodology

This dissertation focuses on the qualitative research methods in order to carry

out the required research and analyse the data collected.

Judith Preissle states that qualitative research is a loosely defined category of

research designs or models, all of which elicit verbal, visual, tactile, olfactory, and

gustatory data in the form of descriptive narratives like field notes, recordings, or

other transcriptions from audio- and videotapes and other written records and

pictures or films (Vanguard University, 2004).

The motivation for qualitative research stems from the observation that in

research involving ‘human participants’, the ability of human beings to ‘talk’ will

enable in understanding a phenomenon better. This point is emphasised by

Kaplan and Maxwell (Kaplan and Maxwell, 1994) who suggest that the task of

understanding a phenomenon from the point of view of the people who are a part

of that process and the social and economic context that they live in is largely

lost when data is analysed using other techniques.

A case study approach was followed for this dissertation and this was carried out

with British Telecom (BT) at their South Yorkshire office based in Sheffield.

1.6 Research Strategy

The initial stages of the research began with studying the literature related to

‘quality’ in Information Systems Project Management. This helped the researcher

understand the significance of quality as a maker or breaker for the success or

9


failure of projects respectively. Key stakeholders of the project were then

interviewed in detail with questions that were prepared using the literature and

the ISO 9000-3 framework. The responses obtained were then analysed using

concept maps to get a detailed understanding of the quality practices.

1.7 Limitations

As with all research projects, this dissertation is also subjected to constraints, the

two significant of which are given below:

Time

The researcher felt that the time duration of three months was insufficient for this

kind of project. Quality is a very huge domain and it took a lot of time for the

researcher to understand the literature and the ISO framework.

The researcher wishes to state that it was extremely difficult in obtaining

literature on ISO 9000-3 in terms of books and reliable internet sources. Most of

the materials available on the internet were not free and the ones that were free

were not worth quoting in the literature review. This took away a lot of time due to

which the time to prepare for the interviews was lessened.

Staff Availability

The availability of stakeholders was a major concern to the researcher right from

the beginning. Due to the unavailability of the staff, the researcher had to wait for

over a fortnight and then finally conduct all the interviews within a span of two

days. This did not give enough time for the researcher to prepare for the

interviews.

10

Chapter Two: Quality in Information Systems Project

Chapter Two Quality in Information

Systems Projects

2.0 Quality in IS Projects

This chapter talks in detail about the importance of quality in information systems

projects. It starts off with an introduction about quality, tracing the history of its

role in today’s industry and how it is perceived differently according to the usage.

It then discusses the importance of good quality practices as a key attribute in

delivering a high class end product. The chapter then throws light on some

important quality concepts with a significant emphasis on quality assurance.

Quality assurance covers areas such as testing, walkthroughs and inspections.

The chapter finally concludes with a brief discussion on configuration

management.

2.1 Introduction

Enterprises all over the world, not necessarily in the software sector are

becoming increasingly dependent on quality software IT systems (Haug, Olsen et

al, 2001:3). As the information age develops, software will continue to grow at a

geometrical rate. As a consequence, the quality of the software product produced

by companies will be a big factor in their survival through the next decade (Haug,

Olsen et al, 2001:3)

11


The Merriam Webster Online Dictionary (Merriam Webster Online Dictionary,

2004) defines quality to be a peculiar and essential character, an inherent

feature, a degree of excellence or superiority in kind.

The Oxford Dictionary (Oxford Dictionary, 2004) defines quality to be a character,

disposition or nature, a capacity, ability or skill in some respect, etc

The above two definitions are very general and do not define quality for any

specific area. But we do understand from the above that quality is associated

with traits like excellence, superiority, etc.

The Longman Dictionary of Contemporary English (The Longman Dictionary of

Contemporary English, 1998) mentions that quality is a concept which can often

be confused in daily usage. Thus a layman would regard quality as a ‘good thing’

or a ‘durable product’. Quality in such a usage is defined as something that is

very good especially in the case of a product that is being sold

North, Blackburn et al (North, Blackburn et al, 1998) suggest that quality is often

equated with high standards of production, delivery and presentation of a product

whose cost of production are seen as immaterial. However cost of production is

an important consideration for every product and sometimes, quality is

deliberately used in the absolute sense to paint a picture where price and cost

have been taken out of the equation to represent pure quality.

Another interesting aspect of the word quality provided by Wilkinson and Willmott

(Wilkinson and Willmott, 1995:2) is that it provides a suggestion of something

that is not easily quantifiable – something that is not easily measurable. It thus

seems to generate an idea of something that is vague; however the notion of a

‘positive’ attribute gives it a positive and extensive appeal.

12


The above argument of the author is very significant when measuring quality of a

product for an industry other than the manufacturing industry. As noted by

Gronroos (Gronroos, 1984), the meaning of quality is more difficult to define for

the services industry due to the non-material character of the goods.

The above point was further emphasised by Bowen and Schneider (Bowen and

Schneider, 1988) who talked about the differences between services and

manufactured goods; the key distinction being that services are intangible, they

cannot be easily stored or transferred and they do not last till a definite point in

time. Thus services are difficult to standardize and quality is often difficult to

determine.

The above premise is further extended by Hughes and Cotterell (Hughes and

Cotterell, 1999) who suggested that the difficulty in standardizing quality as a

result of differences in the nature of the service and manufacturing could be

applied for a software product as well. Since software does not have a physical

existence, quality can be a very vague and undefined attribute if we are not

careful. Hence they suggested that the general usage of the word ‘quality’ cannot

be applied for computer software.

Probably the definition provided by BS 4778 would be ideal for judging quality in

computer software products. This is provided by Maylor (Maylor, 1996) in his

book as the totality of features and characteristics of a product or service which

bear on its ability to satisfy a stated or implied need. Thus we could talk of the

ability of the software product to meet the needs of the customer in terms of

functionality, ease of use, etc.

Keith and James (Keith and James, 1996:34) further comment that today quality

is becoming important in each and every industry and not just in computer

software. Thus all projects and project management personnel have to keep

quality issues in mind when undertaking project planning. Often project

13


managers claim that good project management is good quality management.

Though this may work in some projects, the aspect of the impact of quality

requirements on software projects needs to be formalised.

As Yeates (Yeates, 1991:140) rightly points out, formalisation of quality for other

sectors (mainly manufacturing) did however begin post Second World War

mainly with the Japanese approach to manufacturing. He argues that the origins

of many of the Japanese ideas can be traced to American quality experts like

Edwards Deming. His ideas were receptive to Japanese managers as in contrast

to western managers who thought that quality increased the cost of a product

and thus affected its productivity. Writers like Joseph Juran also stressed on

building a quality culture within companies with emphasis on areas such as

identification of customer needs.

Lock (Lock, 1997: 6) on the other hand argues that there has been continuous

paradigm shift in the perception of quality and it is beginning to be considered as

a serious issue in other sectors as well. He says that there is no longer a

misconception that quality management is the sole responsibility of the quality

control department. This was the case earlier where many managers were

adverse to the idea of quality since they thought it increased the cost of the

product and slowed down productivity and in many cases there was no

formalisation of the quality procedures for project management processes.

However with the emergence of total quality management (TQM) and with the

severe competition amongst companies building up (in every domain) in the last

few years, companies are now seeking to get themselves quality certified in

standards such as ISO 9000, BS: 5750 etc to prove that quality principles for

project management are being applied correctly.

2.2 Importance of Quality Yeates (Yeates, 1991:192) gives a very practical example highlighting the

importance of quality. He says that for any project, if the final product delivered

14


contains bugs or does not meet user requirements, the customer will be

dissatisfied. Depending on the terms of the contract, he could enforce the

developing company to rectify the errors free of cost. This would be an

unnecessary overhead for the company and also affect their position for future

contracts. As Yeates (Yeates, 1991:192) points out, this issue is so serious that

80% of the total lifetime cost of a system – excluding the running costs – is spent

on product maintenance. Any layman would agree that steps taken to reduce this

maintenance cost will result in a huge savings to the developing company.

This concept is further expounded by Haug, Olsen et al (Haug, Olsen et al,

2001:5). They mention that the satisfaction obtained from a good quality product

will have an increasing effect on the customer and the effects of this would trickle

to other areas of business. Thus any organization, not just a software company,

but any company where the generation of software would constitute a significant

component of its operation (Example Dell) would ultimately benefit from good

quality practices.

If quality is so important and benefits organizations significantly, then how do we

identify quality? How do we say, ‘This is of good quality’ or ‘This is poor quality’

What is ‘good quality’ in terms of a project? According to Turner (Turner, 1999:

149), a project is considered to be of a good quality if

• It meets the specification

• It is fit for the intended purpose

• It meets the customer’s requirements

• It satisfies the customer.

The author has argued that the above four statements do not mean the same

thing and that they are clearly different in their meaning. He also writes that

satisfaction of the four statements will almost guarantee a good quality project.

15


However, a project often houses a set of complex interlinked problems and

satisfaction of these rules is anything but straightforward.

However, Turner (Turner, 1999: 149) does argue that everything could be left to

uncertainty and chance. Companies should strive to achieve quality in their

processes. Most companies take this issue seriously and include a quality plan

along with a project plan. A quality plan highlights how the company intends on

meeting the standards that are specified by the quality policy.

The following diagram shows the importance of quality in terms of savings for the

company over a period of time:

Figure: 2.1 The cost of quality

2.3 Quality Management

Quality Management is a very important concept in the industry. ISO Easy (ISO

Easy, 2004) states that modern quality management requires the control of all

management activities that determine the quality policy, objectives, and

responsibilities, and implement them by means such as quality planning, quality

control, quality assurance, and quality improvement within the quality system.

16


The website thus points out that quality management deals with all aspects of the

overall management function that determines and implements the quality policy.

The desire to attain high quality requires the commitment of all the employees of

the organization; however the responsibility for quality management rests with

the top management. The diagram provided below gives an account of the

quality management process.

Figure: 2.2 Quality Management Process. (Office of Government Commerce,

2004)

Schwalbe (Schwalbe, 2000:179) argues that over the years, several eminent

personalities like Deming, Juran, Crosby, Ishikawa, and Malcolm Bridge have

17


helped develop the modern quality management principles. Some of their

principles as outlined by have evolved into solid quality standards.

The article on Tech Republic (Tech Republic, 2003) however says that one

should remember that implementation of the quality standards should be scaled

to the size of the project. In general, large projects go though a series of steps

such as metrics capture, quality management plan, awareness and training, and

process improvements. Smaller projects often identify specific quality activities

due to lack of time and resources.

The article further points out that quality practices need not be ignored for small

companies. All it means is that small companies would find it more beneficial to

integrate quality practices in their processes directly rather than have a formal

plan. This is important since there is a cost to managing quality as well as a

benefit. The time and resources required to manage quality must not exceed the

value that the company hopes to gain from the process, else it is not worth the

effort.

2.4 Quality Planning

The Quality Portal (The Quality Portal, 2004) states that quality planning is the

first step in ensuring project quality management. Quality planning is one of the

most important phases in information systems project management. It deals with

the task of identifying the relevant quality standards for the appropriate projects

and determining the means to satisfy them. It focuses on issues such as defect

prevention and continuous improvement instead of defect detection

Schwalbe (Schwalbe, 2000:184) takes the argument further when he mentions

that for an information systems project, quality planning would typically include

setting the response time of systems, system growth and the accuracy level of

information stored and retrieved. Thus he mentions that focus should also be on

18


the prevention of defects through the selection of proper materials, resources

(material and labour) and training the workforce in quality. Some of the

techniques used in quality planning as outlined by Schwalbe are as follows:

• Design of experiments

This technique (originally a statistical method) when applied to project

management helps us identify the most important variables that affect the project

outcome. Design of experiments issues are often applied to project management

scenarios for example to determine the appropriate mix of junior and senior

personnel who may be required for the project (Schwalbe, 2000:184)

• Functionality

Functionality is often the capabilities or behaviours of a system in the sense of

being able to perform the intended functions. Care should be taken to

differentiate the mandatory functionality from optional. This would help to

understand the system better and prevent loopholes (Schwalbe, 2000:184)

• System Outputs

System outputs are the screens, reports and the user interfaces that will be

displayed on the system. It is important to analyse whether the users will be

satisfied with the system outputs and whether they get all the relevant

information they need (Schwalbe, 2000:184)

• Performance

Performance addresses how well a system can perform for its intended use.

Performance parameters are often multiple and complex and should be

considered seriously at the time of designing the system. For example, in an

information systems project, performance parameters could be influenced by the

volumes of data that need to be processed by the system, the number of users

who could simultaneously log in, etc. (Schwalbe, 2000:184)

19


2.5 Quality Assurance (QA)

The Longman Dictionary of Contemporary English (The Longman Dictionary of

Contemporary English, 1998) defines quality assurance as the management of

the quality of goods and services so that they stay at a good standard.

Ince, Sharp et al (Ince, Sharp et al, 1993) summarise the activities conducted in

this phase. They suggest that the main activity in this phase is to monitor the

system specification against the customer’s specification. Thus they say that in

simple terms, quality assurance is the action taken to provide confidence that the

product meets the customer’s expectations. It also serves as a means to ensure

that the relevant quality standards have been adhered to. They point out that

quality assurance is not just a check on the system after it is built; it is a detailed

plan which is presented along with the project plan.

The quality assurance cycle can be summarised by the following diagram:

20


Figure: 2.3 Quality Assurance Model – (Project Management Methodology, 1997)

From the above diagram, we can observe that quality assurance is the bridge

between the quality plan and the quality assessment procedures. The quality

process is verified through assessment techniques such as verification and

validation, acceptance testing and quality audits. While the quality plan sets out

the performance standards of the system, a quality assurance plan checks that

the system when built performs exactly as specified in the plan. The following

section will concentrate on some of the several assessment techniques used in

quality assurance:

2.5.1 Testing

What is software testing? A classical definition provided by Haug, Olsen et al

(Haug, Olsen et al, 2001:46) is as follows:

21


“Testing is the process of executing a program with the intent of finding errors”

Ould and Unwin (Ould and Unwin, 1989) point out as early as 1989 that with the

increasing application of software in all kinds of companies, especially real time

and life-critical systems, it was not enough if test was conducted only on the

code. They suggest that testing be conducted in all the phases of the software

lifecycle development, especially requirements, design and implementation and

that each of these phases requires a different testing strategy.

Haug, Olsen et al (Haug, Olsen et al, 2001:46) further suggest that for

manufacturing a quality product, it is necessary to test the product against the

factors that affect its quality i.e. reliability, integrity, safety, security, ease of use,

portability, performance, ease of use, and so on. They further suggest that

testing can be classified into the following types:

• Static Analysis covering

o Program inspections

o Walkthroughs

o Inspections

• Dynamic Analysis covering

o Unit testing

o Integration testing

o System testing

o Acceptance testing

o White box testing

o Black box testing

Yeates and Cadle (Yeates and Cadle, 1996) also point out that software quality

may be evaluated by the above two principal analysis; static and dynamic. Static

analysis mainly covers inspections and dynamic analysis is concerned with

22


testing. This can be summarised by the diagram below. Both methods are aimed

at detecting and correcting errors. However, they argue that inspection is

concerned with verifying the information created at the head of the line with the

information available at the tail of the line. Most of the development environments

require a correct combination of testing techniques. The different types of testing

techniques are summarised by them in the following diagram:

Figure 2.4 Testing Techniques (Yeates and Cadle, 1996)

Thus, as an example in the above diagram, we can state that Integration testing

has to verify that design has been carried out satisfactorily.

Unit Testing

Ould and Unwin (Ould and Unwin, 1989), members of the British Computer

Society (BCA), state that unit testing essentially implies testing to ensure that a

module correctly implements its design and functionality and that it is ready to be

integrated into a bigger system or subsystem. They suggest that unit testing must

be conducted only when the code is free of all errors detectable by a compiler.

When such a state is reached, unit testing is conducted which can further be split

on two categories; the first being examination of code using external general

criteria and the second being the verification of the coded units according to the

unit test plan.

23


Yeates (Yeates, 1991) further extends the importance of unit testing. He explains

that in the industry, often in the case of large projects, there are different

professionals working on different areas at the same time. Each of them would

work on a part of the system and this, as we noted earlier is called a unit or a

module. Assuming that a good design is used for the project, these modules

would be integrated to cohesively form larger modules which turn would merge

with other modules and finally become one single system. He points out that unit

testing become very important because it is the stepping stone for a good quality

system. The primary aim of unit testing is to remove all syntactical and

semantically discovered errors through thorough testing so that subsequent

integration and system testing become much easier.

Integration Testing

Yeates (Yeates, 1991) suggests that the purpose of integration testing is to

ensure that the lower level modules are able to coordinate with themselves

seamlessly. He points out that it is not a complement to unit testing. If unit testing

has been thoroughly performed on individual modules, then integration testing

becomes much simpler. On the other hand, badly developed and tested

individual modules create all sorts of bottlenecks for an effective integration

testing.

The MSDN (MSDN, 2004) describes several ways to perform an integration

testing; the two common methods being incremental testing and top down

testing. In the case of incremental testing, two modules are compared and

tested. If they integrate efficiently, then a third module is added and the three

modules are tested with each other. If there is a problem, it can be assumed that

it is with the third module, hence pin pointing problems becomes much easier. In

top down testing, the highest level module is tested first and then the next level

24


down, working down the tree. This does not mean that the lowest level modules

are tested last; every module has to be tested during the phase of unit testing.

Systems Testing

This is the next phase after integration testing. System testing ensures that the

system does what is required of it. However it is a very generic term and can vary

from organization to organization. Yeates (Yeates, 1991) argues that this is the

live processing of the system to ensure that all the programs are linked together

and are working correctly. He stresses that unit testing and integration testing lay

the foundations for a successful system test. If the syntactic and logical errors

are ironed out during unit test and if coordination between modules are achieved

smoothly after thorough integration test, then system test is all about proving that

the system meets the desired requirements. There are several methods for

system testing and these vary depending on the type of project or organization.

Operational Testing Operational testing is defined as the formal testing conducted prior to deployment

to evaluate the operational effectiveness and suitability of the system with

respect to its mission (Computing Dictionary, 2004)

Operational testing tests the system under different hardware and software

environments. Often the development environment and the deployment

environment may differ. To iron out any problems that may arise because of such

a difference, a series of tests is performed under a live environment.

Acceptance Testing

The article on software testing (Software Testing, 2004) states that acceptance

testing is the formal testing conducted to enable a user, customer or other

authorized entity to determine whether to accept a system or component. The

25


article states that usually acceptance testing is carried out at the end of the

software development lifecycle. It further states that acceptance testing can be

divided into two types; Business Acceptance and User Acceptance.

User Acceptance Test (UAT) is very popular in the industry these days. The

concept is explained by Yeates (Yeates, 1991) who states that once the earlier

tests are done, the system is handed to the user. The user then usually signs

some form of document stating that the system conforms to the requirements

mentioned in the contract. However before doing so, the users often test the

system with their own test data and in their own conditions. Such a testing is

called acceptance testing. The extent of acceptance testing differs for each

project though it depends a lot on the relationship between the client and the

developers. If the client has been heavily involved in the development of the

system, then the acceptance testing is generally not very rigorous.

Black Box Testing

Black box testing, which is also called functional testing is a software testing

technique whereby the internal working of the item being tested is not known by

the researcher (Webopedia, 2004).

As an example, we can say that in a black box testing, the tester is concerned

only with the input and the corresponding output. He would not be concerned

with how those inputs are arrived at or with the quality of the programming code,

etc.

Thus we can say that black box testing is concerned with testing the system on

its expected behaviour and ensuring that the resulting functionality is satisfactory

(Beizer, 1995).

The following figure presented below summarises the concept of black box

testing:

26


Figure: 2.5 Black box testing. (University of Calgary, 2004)

The concept is further elaborated by Cechich and Polo in Oivo and Sirvio’s (Oivo

and Sirvio, 2002) publication wherein they state that in many cases, black box

testing (also called behavioural testing) is performed without the knowledge of

the source code or design components and the tester is concerned only with

inputs and outputs. They further state that black box testing can be classified into

two methods; binary reverse engineering and interface probing.

White Box Testing

White box testing, also known as glass box, structural, clear box and open box

testing is a testing technique whereby explicit knowledge of the internal working

of the system is used to test the system. Thus specific knowledge of the

programming code is required to test the system (Webopedia, 2004).

The closeness of this testing with the code of the system makes it a part of a

testing process called component testing. The analysis and the running of

complex code means that the testing process can succeed only if the developer

has an inherent knowledge of the working of the program and that he is able to

observe and identify if the program diverges from its intended goal (Software

Testing, 2004).

27


The concept of white box testing can be summarised in the following diagram

provided below:

Figure: 2.6 White box testing. (University of Calgary, 2004)

2.6 Conclusions

Managing quality involves setting up a clear quality culture with conformance to

accepted standards. The implementation of the standards varies through industry

and the type of project. However, they do help us in identifying gaps in quality

that exist in the management processes. Quality planning and quality assurances

are important stepping stones to modern quality management. Each of them

includes several steps covering a wide range of areas. All the stages would be

necessary only for large or complex projects. However careful planning is

required for all stages which will avoid wastage of time, resources and cost.

28

Chapter Three: The ISO Standards for Quality Management & Quality Assurance

Chapter Three The ISO Standards for Quality

Management & Quality Assurance

3.0 Introduction

Standards make a very important contribution to our lives, though most of this

contribution is invisible and is taken for granted when products meet

expectations. Their importance is felt only in the case of absence of standards

which result in poor quality goods that do not meet customer requirements

(International Organization for Standardization, 2004).

The International Organisation for Standardisation (ISO), based in Geneva, is a

world-wide federation of national standards bodies (Vanguard Education, 2001).

ISO thus represents the International Organization for Standardization. It is not

an acronym for the International Organization for Standardization as many

people think (iSixSigma, 2004).

ISO is the world's largest developer of standards. Although ISO’s main activity is

the development of standards for a wide range of disciplines, it does have

economic and social implications as well. ISO standards make a positive

contribution to society since they help in solving problems of production and

distribution faced by engineers and manufacturers (International Organization for

Standardization, 2004).

29


3.1 ISO 9000

ISO 9000/ISO 9001 is amongst ISO’s most widely known and successful

standards ever. Today, ISO 9000 is the most popular international standard for

conforming to quality requirements in business dealings (International

Organization for Standardization, 2004).

The ISO 9000 standards form a quality management system and are applicable

to any organization regardless of product, service, organizational size, or whether

it's a public or private company. With the disappearance of geographical barriers,

ISO 9000 has become a key differentiator between a company and its

competitor. It thus plays a key role in winning new contracts and satisfying

existing customers (iSixSigma, 2004).

An important differentiator of ISO 9000 from other standards is the fact that it is a

process standard and not a product standard. It is not concerned with what is

made but how it is made (Vanguard Education, 2001).

Obtaining official copies The researcher wishes to point out that official copies of the ISO standards are

not available free. They have to be purchased from the International Organization

for Standardization (ISO) website. This is because ISO documents are protected

by copyright and hence any form of reproduction is prohibited. Hence the ISO

related literature for this dissertation is obtained through books and websites.

3.2 Importance of ISO 9000

ISO has a member body of 148 countries, on the basis of one member per

country, with a Central Secretariat in Geneva, Switzerland, that coordinates the

system (International Organization for Standardization, 2004).

30


Over the last decade, ISO 9000, which is one of ISO’s most popular standards,

has seen an explosive growth. As we can see from the chart below, there has

been a continuous growth in the number of ISO registrations, though the growth

rate has fluctuated over the last few years.

Obviously there would be clear advantages to a company in getting ISO certified.

These advantages are discussed in the next section.

Figure: 3.1 Growth of ISO 9000. (International Organization for Standardization,

2004)

31


Advantages of ISO certification

The benefits of becoming certified are numerous, but some of the main

advantages are as follows (International Organization for Standardization, 2004):

• To improve business processes and save money

• To qualify for new customers.

• To enter global markets. ISO 9000 standards are required in many

countries.

• To enhance customer responsiveness for products and services.

3.3 ISO 9000-3 – A Tool for Software Product & Process Improvement

The researcher wishes to state that it was extremely difficult in obtaining

literature on ISO 9000-3 in terms of books and reliable internet sources. Most of

the materials available on the internet were not free and the ones that were free

were not worth quoting in the literature review. Hence all text in this section is

obtained from a single book “ISO 9000-3 A tool for Software Product and

Process Improvement” by Raymond Kehoe and Alka Jarvis unless stated

otherwise.

There is a strong similarity between good software engineering techniques and

the ISO 9000-3 practices. Thus companies that honestly implement ISO

standards will often find themselves adhering to ideal software engineering

practices. ISO is based on the premise that companies following the standards

are more likely to create reliable and error free products that meet customer

needs than those companies that do not follow any accepted standards or

procedures.

32


The ISO standards were developed by the International Organization for

Standardization in Geneva, Switzerland to create a set of common standards for

both quality management and quality assurance. The organization has chapters

in Germany, the United States, France and the United Kingdom. Since its

introduction, almost 31,000 companies in 60 different countries have been ISO

certified. The ISO has five related components that are numbered from 9000 to

9004 out of which this study will primarily be concerned with ISO 9000-3.

ISO 9000-3 is a guideline for the application of ISO 9001 to the development,

supply and maintenance of software – 1991. It is a near 100% expansion of the

ISO 9000-1 standard and was essentially developed to provide guidelines for the

specification, development, installation and support of software. ISO 9000-3 does

not state the rules to design, analyze or implement a system. Instead it lays

emphasis on a clearly defined and documented process.

The ISO standards form the basis of an audit process where independent

auditors assure customers that the company in question does follow the ISO

standards in manufacturing their products. Companies that do not follow any

industry standard engineering process to manufacture their products may soon

find a limited market for their products, an expensive and time consuming

maintenance process for software already shipped.

33


Management

Development Team

Updated and Edited System

Status Request for Funds and Resources

Error Reporting, Feedback, etc

Engineers

Analysis and Design

Code and Debug

Deploy & obtain feedback

Improve the system

Figure 3.2: The Business of Software Development at British Telecom –

Adapted for the case study

The ISO 9000-3 guideline clearly spells out duties and responsibilities that need

to be followed by the user and the supplier if it has to be a success. The entire

process is customizable and can be implemented in more than a single particular

way.

3.4 ISO 9000-3 Scope and Overview The ISO 9000-3 guidelines do not advocate any policies or procedures but

identify sound engineering and management practices that are suggested for the

use in product development. A company is free to develop its own policies and

procedures and the ISO guidelines serve as an excellent benchmark. The ISO

guidelines can also be used on the in-house software development process,

something which is very pertinent to this study.

34


Definitions This section defines several terms (exactly as used in the guideline). The

meaning of these terms would hold value when the researcher performs data

analysis and discusses the findings from the interviews

Verification (for software)

The process of evaluating the products of a given phase to ensure correctness

and consistency with respect to the products and standards provided as inputs to

the phase.

Validation (for software) The process of evaluating software to ensure compliance with specified

requirements

Software Intellectual creation comprising the programs, procedures, rules and any

associated documentation pertaining to the operation of a data processing

system

Software product Complete set of computer programs, procedures and associated documentation

and data designed for delivery to a user

Development All activities to be carried out to create a software product

Phase Defined segment of work

35


Quality System – Life cycle activities The ISO 9000-3 guideline states that all software development projects should

have a clearly defined and documented process. Companies are free to use any

lifecycle they deem fit as long as due consideration is given to various steps that

are followed in a software development process. The process details are

captured in a software process handbook (SPH). The SPH provides guidelines

that need to be followed in each phase along with a list of templates. The phases

can be customised according to the requirements of the project and the policies

of the company. The following gives a brief idea about the software development

process and the deliverables expected from each phase as mentioned in the ISO 9000-3 guideline.

The researcher wishes to state that these phases would be looked at in more

detail in the data analysis and discussion of findings and recommendations

chapters

Phase One: Product specification and planning

The purpose of this phase is to identify the user needs through a feasibility study

to gain an understanding of user requirements. It is also concerned with

preparing an estimate of the resources needed for the project.

Phase Two: Engineering specification and detailed planning The purpose of this phase is to identify the software and hardware requirements

needed to implement the product features identified in the product specification

and planning phase. A proper account of the resources (man hours, hardware,

software, etc) also needs to be considered.

36


Phase Three: Product design The purpose of product design is to define the functional design, interface

design, database design and error handling design and also to identify the test

cases and create an outline for all product documents

Phase Four: Product Implementation

The purpose of product implementation is to implement the software,

documentation and system test designs. It is also concerned with proving that the

software developed works with the minimum level of integration.

Phase Five: System test

The purpose of this phase is to provide management with reliable information

concerning the degree to which the software meets its functional and

performance requirements

Phase Six: Product evaluation

The purpose of this phase is to evaluate the product from a user’s viewpoint to

determine the product’s usability

Phase Seven: Product release

The purpose of this phase is to ensure that the product masters delivered to the

production vendor represent the product baseline and to assure that the

production versions reflect the product baseline

37


Phase Eight: Maintenance

Product enhancements require at least a tailored version of the SPH based on

the size and complexity of the enhancement.

38

Chapter Four: Methodology

Chapter Four

Methodology

4.0 Methodology

This chapter will describe the methodology that was followed in order to

undertake the investigation for the dissertation. The chapter would provide a

detailed overview of the qualitative research methods with a particular emphasis

on the case study analysis. It would describe the strategy that was used by the

researcher to conduct the interviews. Finally, it would provide details about the

case study organization, British Telecom. It would conclude by very briefly

providing an overview of the Training and Development Live, the system which

would be used by the researcher for the purpose of this dissertation.

4.1 Qualitative Research Method

This section explains qualitative research in Information Systems which is the

foundation stone for this research.

Information systems, as a research field is relatively new without a research

tradition that it can claim to be its own. Many of the IS research frameworks,

techniques and theories are borrowed from older and well established

disciplines; essentially social sciences. (Bikson, 1991).

Garcia and Quek (Garcia and Quek, 1997) extend the above argument further,

pointing out that due to the availability of research frameworks from other

disciplines, the starting point for a researcher’s methodological choice, whether

he employs qualitative, quantitative or some other technique is not so much of a

39


problem as it is to identify and choose the appropriate methodology for the

research problem in question.

This dissertation will focus on the qualitative research methods in order to carry

out the required research and analyse the data collected.

Judith Preissle states that qualitative research is a loosely defined category of

research designs or models, all of which elicit verbal, visual, tactile, olfactory, and

gustatory data in the form of descriptive narratives like field notes, recordings, or

other transcriptions from audio- and videotapes and other written records and

pictures or films (Vanguard University, 2004).

Garcia and Quek (Garcia and Quek, 1997) further point out that qualitative

research thus includes obtaining data though observation and participant

observation (fieldwork), interviews and questionnaires, documents and texts, and

the researcher’s impressions and reactions. Qualitative research techniques are

employed in a variety of disciplines and fields using many approaches, methods

and techniques.

4.1.2 Motivation

The motivation for qualitative research stems from the observation that in

research involving ‘human participants’, the ability of human beings to ‘talk’ will

enable in understanding a phenomenon better. This point is emphasised by

Kaplan and Maxwell (Kaplan and Maxwell, 1994) who suggest that the task of

understanding a phenomenon from the point of view of the people who are a part

of that process and the social and economic context that they live in is largely

lost when data is analysed using other techniques.

This view is further extended by Garcia and Quek (Garcia and Quek, 1997) who

point out how a respondent achieved more satisfaction in using the qualitative

40


approach as it gave a more ‘human touch’ to his study. The responses were not

just numbers, but thoughts and opinions of real people.

A note on perspectives

Myers (Myers, 1997) states that all qualitative research is essentially based

on three major perspectives which are the corner stones of the research

method. They influence or guide the research process. This can be

summarized by the following diagram:

Figure 4.1 Qualitative Research

4.2 Research Strategy

The initial stages of the research began with studying the literature related to

‘quality’ in Information Systems Project Management. This helped the researcher

understand the significance of quality as a maker or breaker for the success or

failure of projects respectively. Key stakeholders of the project were then

interviewed in detail with questions that were prepared using the literature and

the ISO 9000-3 framework. The responses obtained were then analysed using

concept maps to get a detailed understanding of the quality practices. The

strategy is summarised in the diagram below:

41


Evaluate Quality Practices

Conduct Interviews

ISO 9000-3 Framework

Literature on Quality

4.2.1 Interview Strategy

As mentioned, interviews were conducted with key stakeholders of the system.

The people chosen for the interview were those who played an important role in

the development of the system and those who interacted with the system in

some form or the other during their daily work. During the interviews, the

comments made were recorded on tape for the purpose of future reference of the

researcher. All interviewees were assured of high standards of confidentiality so

that they could be free in sharing their thoughts.

It was also mentioned to them that the researcher was a university student and

he was not officially representing the company or the university in any way and

that the dissertation was purely an academic exercise. It was also mentioned that

the researcher would be happy to provide a detailed transcript of the entire

interview to the interviewee if he/she was interested. For this reason, the

researcher also gave the interviewees his email address if they ever wanted to

contact him with any queries. Further it was added that if at any point of time,

they wanted to share something about the system or the process which they did

not want to go on record, then they were free to press the stop button on the tape

recorder.

The questions for the interview were devised essentially based on the ISO 9000-

3 framework supported with literature relating to quality in IS project

management. It is worthwhile to mention here that the researcher did initially

spent some time asking informal questions to the interviewee. This was to build a

42


relationship with the interviewee and gradually guide him into the more important

questions. Follow up questions were used by the researcher to ensure that the

interviewee did not drift away from the key concept and also to reinforce what the

interviewee said. In addition to the questions, quick notes were also made by the

researcher to re-emphasise on key points made by the interviewee. To be able to

obtain the maximum information out of the note taking process, the researcher

followed the following strategy:

• Did not worry too much about spellings and punctuation marks.

• Took notes as fast as possible trying to keep pace with the interviewee

rather than begin to write after he finished a whole sentence.

• Looked for specific words that were repeatedly mentioned by the

interviewee and highlighted them.

• Did not think too much about whether something was significant or not.

• Drew quick diagrams to summarize what the interviewee said in a

sentence.

• Tried not to mix personal bias with what the interviewee mentioned.

At the end of the interview, the interviewees were asked if they were satisfied

with the interview. They were once again assured of confidentiality and

mentioned that data collected through the interview would be used for academic

and related purposes only.

4.2.2 The Case Study

The faculty at the North Carolina Wesleyan College (North Carolina Wesleyan

College (2004) provide a very good summary of the kind of methods available

under qualitative analysis. This is given below. A case study approach was

followed for this dissertation.

43


1. Participant-Observation

2. Ethnography

3. Photography

4. Ethno methodology

5. Dramaturgical Interviewing

6. Sociometry

7. Natural Experiment

8. Case Study

9. Unobtrusive Measures

10. Content Analysis

11. Historiography

12. Secondary Analysis of Data

Figure: 4.2 Case Study Approach

The term "case study" has multiple meanings. It can be used to describe a unit of

analysis (e.g. a case study of a particular organisation) or to describe a research

method. The discussion here concerns the use of the case study as a research

method.

Orlikowski and Baroudi (Orlikowski and Baroudi, 1991) argue that case study

research is the most common qualitative method used in information systems

because it is the best method available to study organizational issues.

British Telecom (BT)

The dissertation followed a case study approach and this was carried out in

association with British Telecom (BT) who provided the case study.

BT is a UK wide communications solutions provider with a global reach in the

business market. BT’s origins can be traced to the first telecom company in the

UK beginning with the first commercial telegraph service in the early nineteenth

century. With time, many of these companies were taken over, amalgamated or

dissolved. The surviving companies were them transferred to the state control

under the Post Office and ultimately to the privatised British Telecommunications

plc. (British Telecom plc, 2004)

44


The system in BT studied for the purpose of this dissertation is called Training

and Development Live.

The researcher wishes to point out at this stage that the company had only two

documents for the system, a user process document and a power point

presentation on the benefits of the system. Hence it was difficult to obtain

information regarding the functionality and technicalities of the system.

4.3 Training and Development Live

The purpose of the system is to monitor, track, and plan training for field

engineers with BT Field Service. BT employs more than 20,000 field engineers

across the UK to maintain networks, repair faults, and provide service to

customers.

Training and Development Live aims to balance training needs with the

engineers’ availability using pre-set shrinkage targets. Shrinkage is the amount of

people who can be trained on any given day.

Engineers in BT undergo training in several technological areas related to their

work. This could range from activities such as broadband wiring, routing, etc. to

wiring, pole climbing, etc. Training is a continuous activity and the more training

courses they take, they keep acquiring more and more skills. Recording the

training activities of engineers is important because they get jobs according to

the skills that they possess. For example, for a broadband wiring problem, it has

to assign the job to that engineer who has been trained in a course that involves

fixing up broadband wiring problems.

To conclude, the working of the system can be summarised as follows:

“There is a training plan in which departmental representatives submit their

training needs for the following year. These needs are then built into the system.

45


The people/engineers requiring this training are also built into the system, i.e. put

onto the waiting list. The training plan and the waiting list are then put together

into training courses using a diary (against the shrinkage as mentioned before)”

46

Chapter Five: Data Analysis

Chapter Five

Data Analysis

5.0 Analysis of Data

Chapter four described the methodology that was used to perform research on

this dissertation and thereby collect the relevant data. This chapter focuses on a

detailed analysis of the data. The data collected by the researcher through

interviews are analysed using cognitive maps. A concept map is prepared for

every interview conducted by the researcher. This identifies the methods

employed and the issues and problems if any, faced by each of the interviewees

during the development process.

The research as mentioned was conducted by carrying out extensive interviews

with four different staff members, each of whom are associated with the system

in a different role. They can be regarded as probably the ideal individuals to

obtain information about the system from different perspectives due to their

differing job functions. Different viewpoints have been obtained within the

interviews from a development, management and analytical perspective that

have provided in depth accounts of the issues which have faced this project.

The questions for this interview were formulated using the knowledge that was

gained by reading the literature relating to quality in Information Systems project

management and also with the ISO 9000-3 standard.

It is important to remember that the ISO 9000-3 guideline only spells out duties

and responsibilities that need to be followed by the user and the supplier. The

entire process is customizable and can be implemented in more than one way. A

47


company is free to develop its own policies and procedures using the ISO

standard as a benchmark. Due to this autonomy, a particular process may exist

with a different name or would be done in a way that is very particular for the

company. Similar scenario may exist for the documentation required. This

objectivity has been kept in mind throughout the dissertation.

The ISO Framework

For this dissertation, the researcher has identified five important stages which

would be used as a framework to analyse the interviews:

• Product specification and preliminary planning.

• Engineering specification and planning.

• Product design and implementation.

• Verification and Validation.

• Product release.

Rationale

Usually, product design and product implementation are considered as two

separate phases. The researcher merged it into a single phase after studying the

development process in the company as it was felt that such a merger would

help in obtaining more detailed and related information since the interviewees

were not able to provide relevant information when asked specifically about

product design and product implementation. Also, since the system was

developed in-house, product evaluation and product release were merged into a

single phase called product release.

Since the ISO standards provide the flexibility to develop customisable

frameworks, the researcher took the opportunity to develop a framework which

would enable him to get maximum relevant information.

48


Due to confidentiality issues, the interviewees are not identified by their names.

They are classified as Interview 1 – Developer, Interview 2 – Project Manager,

Interview 3 – User One and Interview 4 – User Two.

It is important to point out that the interviewees differed significantly in their

knowledge of the system. One particular interviewee provided much more

information than the others. Some other interviewees were too general, not

willing to share specific information despite assuring all confidentiality. The

researcher feels that this has impacted the data accumulated for the purpose of

analysis. This is evident from the fact that the concept maps for some

interviewees are much more detailed than others.

It is also worth noting again that since the software was developed for in-house

purposes, the customer is the BT management; however they are not the users.

Hence there is a clear distinction between the users and the customer. This

understanding will be useful to appreciate the difference between user

requirements and customer requirements.

As mentioned earlier, the five phases are represented as I, II, III, IV and V in the

concept maps.

• Product specification and preliminary planning. (I in the concept maps)

• Engineering specification and planning. (II in the concept maps)

• Product design and implementation. (III in the concept maps)

• Verification and Validation. (IV in the concept maps)

• Product release. (V in the concept maps)

The concept maps are explained as follows:

49


• Phases are represented by the orange circle. These are each of the above

mentioned phases numbered I through V.

• Methods employed by the company for each phase are represented by a

blue rectangle.

• Problems encountered in the execution of each phase are represented by

the yellow oval.

• Consequences of problems arising in the execution of each phase is

represented by the green rectangle

• Conflicts if any, between different interviewees are represented by the

grey oval.

50


Figure 5.1 Legend

51


5.1 Interview One - Developer

This interview took place on July 21, 2004. The meeting lasted approximately two

hours. The issues arising from the interview are analysed in detail using the

techniques discussed in the last chapter, i.e. ISO framework coupled with the

literature on quality in information systems project management.

The concept map for this interview is represented below:

52


Figure 5.2 Concept Map 1

53


Phase One: Product specification and preliminary planning

The interviewee commented that most of the systems that are developed in-

house stem from a sighting that a particular process needs to be automated. If

the management feel that some process could be automated and that it would be

cost efficient for the company, then a preliminary planning is undertaken which is

followed by a detailed cost benefit analysis.

“A team member comes to the group and says ‘we’ve got this problem and we

think it can be automated”

A feasibility study was undertaken similarly for the Training and Development

Live. A report that showed the measurable benefits in terms of time, cost and

ease of use was produced in consultation with the managers and this was

presented to a board member. It was later signoff by the board member which

was an exit criterion to proceed ahead with the next phase of system

development.

Problems identified by the interviewee in this phase

Problem: No lifecycle defined

The interviewee admitted that no lifecycle was defined at the beginning of the

project. A lifecycle if specified right at the beginning of the project helps in

smoothly passing through the stages of specification, analysis, design,

implementation, test and maintenance.

“We are asked to develop and release the products very quickly and cheaply.

Now the time span does not permit me to specify a particular lifecycle and follow

all the steps in it”.

54


Consequence: The two main consequences identified are, a problem in the

succeeding stages and a conflict with the management.

Since no lifecycle is specified, the personnel associated with the project would

want to do the things the way each of them deem correct. This could result in a

chaotic software development process which could affect the quality of the end

product. Also, regular updates to the management would be difficult to present

(due to the chaos) and this could result in a conflict between the development

team and the management.

Phase Two: Engineering specification and planning

For the Training and Development Live, the hardware was already in place. It

was a matter of using the existing hardware facilities like server, router, etc. At

this stage, the developer, the project manager and the server manager worked

up the hardware requirements for the system and this was presented as a

business case document to the finance department for approval. The interviewee

commented that the exit criterion was whether it could be developed for a

reasonable cost.

“For all the systems that we develop, it is always the question of whether the

benefits outweigh the costs”

Phase Three: Product design and implementation

A statement Of Requirements (SOR) was written out by the development team.

This was then taken to the customer (in this case, the BT Management) to obtain

an approval from them. A prototype of the software was then developed in MS

Word and this was shown to the customer and to a select group of users (the end

users) to obtain comments and suggestions.

55


“We made a model in HTML or Word or whatever, which replicated the

functionality and once that was done, we took it to the select group of users and

ask them what do you think of that…and obviously the customer would see that

as well”

The developers then designed the database input and output entries on paper. A

revised SOR was prepared which included a specification of the database

entries. A technical specification document was also prepared containing

functionality details. This was followed by the actual implementation.

“We prepared the revised statement of requirements and a technical specification

document which described on certain business functionality, which bits of code to

run to get which output. That done, we began coding”

There was no methodology followed for implementation as it was felt that it was

unnecessary to have a methodology for all projects.

“In an ideal world, all stages need to be followed for all products. This doesn’t

happen in reality. For my products, I follow my own methodology”


Problem: Inaccurate team formations

The interviewee felt that a lot of problems arose because of the inability to

correlate the amount of work with the number of people required. He also felt that

due to this, the development team had to continuously chase timelines.

“We had a project manager, a developer and more often than not, we have a

support person but I sort the time, resource, documentation with the support, sort

the coding, trial groups, roll out etc.”

56


“I have colleagues who start coding 5 minutes after they start getting a project

because our managers push us with the deliverables”

Consequence: The above approach could result in a situation where the quality

of the end product could greatly be affected. It is an important part of project

management to draw up an accurate estimate of human resources for any

project. Furthermore, the development process would be chaotic since there is

no efficient delegation of responsibility.

Problem: No methodology defined

Though the interviewee initially stated that a methodological approach was not

necessary for every project, he did admit later that a good methodology would

solve a lot of problems that were faced during the product development.

“Well some developers do have a methodology. There are a lot of separate

teams and each developer has his own methodology and it so happens that

there’s no one methodology”

Consequence: Not having a methodology could make the developers do things

the way they want besides enabling them to evade responsibility. One of the

biggest advantages of having clearly defined phases is that it allows for accurate

costing, scheduling and statusing. Other consequences could be frequent delays

and a conflict with the customer due to the inability to provide accurate project

status.

Problem: Lack of training in system design

The interviewee felt that not enough training was provided to the IT staff in

system design.

“Most of the software people here haven’t had any formal training in system

design; they all have had training on how to write code, but system design is

57


something that… I have 13 of us in my team and only 3 have had some sort of

training on it. So I suppose it is due to lack of training really, lack of any

standardised training”

Consequence: If there is no training to the IT staff in system design, they would

not be able to appreciate the benefits of following a methodology from conception

to roll-out which would make the entire process much smoother and systematic.

Problem: No standard technical documentation or no documentation

Not having a standardised documentation policy resulted in a number of

difficulties for the development team.

“One of the biggest problems is that there is no standard technical

documentation. I couldn’t pick up somebody’s work half way through. We had a

meeting a few weeks ago to try and get some uniform documentation but there

was no consensus on it.”

Consequences: If the document was non existent, it would pose great difficulty

to the customer as well as to personnel who would be associated with the system

on a future date. As the interviewee rightly pointed out, it would difficult to

continue working on someone else’s code. Debugging would also be an issue if

there is no standard documentation. It could also result in errors in the

documentation due to carelessness.

Phase Four: Verification and Validation

This phase was virtually non existent in the developmental process. There were

no code or document inspections. Testing was conducted during the trial period

of the software. There were no test plans drawn up or test results recorded.

Everything was debugged and patched up on the spot.

58


“I like to think that I write quality code. I know that because the code I write is

better than the code I wrote last year. No one sees my code but me”

“User documentation is compulsory and we have a nice little template for that.

The technical documentation that we do is good, not good, or non existent.

Infact, there’s no enforcement for technical documentation”

“We do testing during the trial, I know trial should not be used for testing, but

that’s the way it is”


Problem: No separate testing phase

The interviewee acknowledged the fact that there was no separate phase for

testing. Most of the errors in the code were debugged on the spot during trial.

There were no test plans drawn up nor was there a record of the errors that

cropped up in the system.

“If we saw some problem, we would debug it on the spot and then test it again.

There’s no separate testing phase. If the software has a problem, then the

database manager would call me and inform me and I have to debug my area of

the code”

Consequence: Testing a product only during the trial period is a very dangerous

thing to do. This could result in several complications for the company in the long

run. To begin with, since the system is not tested thoroughly with a proper test

phase (test plan, test results, etc), the quality of the end product would be

affected. There is a danger of the company spending a large amount of

resources on maintenance later on. Furthermore, the test conducted during trial

would result in a chaos and send a wrong signal to the customer and the end

users thus causing resistance to the product.

59


Problem: Non technical personnel

The interviewee expressed concern that sometimes it was difficult to relate to

non technical personnel.

“Some of my managers are all non technical, I could write a page of code and

she would have no clue as to what it means. The support people are the same.

So the only people who are qualified to see my code are my colleagues and they

are busy doing their own work. Yes it is a big gaping hole”

Consequence: The biggest impact would probably be the fact that there is no

authority to provide guidance on technical issues to the development team. This

is a serious issue since the developers would never know whether they are on

the right track. This might affect the quality of the end product and the whole

project might end in a disaster.

Phase Five: Product release

The product is evaluated to a standard where the customer and the (trial) end

users are happy with the performance of the product. It is released after

obtaining feedback from the users and the customer. User training is provided

immediately after releasing the product.

“The product was evaluated to a standard that the customers and the trial users

are happy with it. I guess it can never be 100% ready. It is released after we

obtain feedback at the end of the trial”

Interview Summary

The interviewee did come up with genuine concerns regarding the development

process. He was concerned about the fact that there was no lifecycle defined at

the beginning of the development process which complicated the succeeding

60


stages. Not having a balanced team in terms of technical knowledge also

frustrated the interviewee since he felt that a lot of additional tasks had to be

undertaken. Due to this, the team was always under extra stress which affected

their working. This was further compounded by the fact that there was no

standardised documentation which made it difficult for the developer to work on

someone else’s code or even understand the technicalities of a system. He was

also concerned that the company was not investing enough in training users in

areas such as system design.

5.2 Interview Two – Project Manager

This interview with the project manager took place on July 22, 2004. The meeting

lasted approximately one and a half hours. The issues arising from the interview

are analysed in detail using the techniques discussed in the last chapter, i.e. ISO

framework coupled with the literature on quality in information systems project

management.


61



62



In the case of the Training and Development Live, the interviewee commented

that the product specification and planning stage began with a cost benefit

analysis of whether the product should be developed in the first place to

determine what was in it for the company.

“The first step was why we are actually going to do this and the benefits from it.

And when we had to weigh out those benefits into the business; how wide across

the business was it”

Studies were then carried out to understand what the end users exactly wanted.

Since it was an in-house system developed for the company, the primary end

users were the engineers and the trainers and they were consulted through a

brain storming session.

“We talked to a number of people, primarily the engineers, trainers…about what

they would want from the system…something like a brain storming session”

He reiterated the point that it was extremely important to consider the benefits

that would arise out of the system if implemented. In the case of Training and

Development Live, he pointed out that such a proposal to automate the system

was provided by one of his colleagues who saw a cost saver there.

“In training and development live, someone had an idea, that’s how it all started.

Of course the idea was not mine. But it was proposed as a cost saving idea”

The phase ended with the preparation of a proposal document which was

presented to the leadership team. He commented that the signoff is usually by a

member of the leadership team though in some situations, it could be heads of

multiple departments.

63


“For most in- house projects, probably someone from the leadership team will

sign it. So we make a presentation and then they pass it or reject it. On a larger

scale where the project affects many departments, the heads of those

departments sit together and work out the details and make a decision”


Problem: Inaccuracy in the phase

The interviewee was concerned that the preliminary planning was not being

executed accurately and that the development team were facing hurdles in the

succeeding stages.

“Feasibility studies are undertaken but in some systems, we got it wrong. These

are the kind of things that we are trying to improve. So what we think as a

feasibility study many not necessarily be true and this gap can often be painful”

Consequence: If the product specification and preliminary planning is not

undertaken correctly, the succeeding stages of development would be affected.

In fact, the rationale for developing a product should be clear at the end of this

phase. If the company with a faulty analysis came to the conclusion that some

particular software would be cost beneficial, the consequence would mean a

large amount of money spent in developing a worthless product and more money

spent on its maintenance. It is thus extremely important to spend a considerable

amount of resources to understand the worth of developing a particular.


The interviewee did not have much clue as to what this phase exactly meant.

Hence he was discussing issues more generally. He however did say that once

the earlier stage was completed, a business case document would be drawn up

64


by the team which would contain detailed specifications of how the system would

be developed. This document would contain intricate particulars including

hardware and software requirements, the resources, etc. An independent body

would conduct an analysis of the business case document.

“There will be specifications drawn up, something like a business case that

needs to go ahead. This will be in more detail regarding the specifications

outlining what you just said. Then an independent body does the analysis”

Phase Three: Product design and development

Once the business case document was cleared, a project manager was

appointed in charge of the project. The entire development process was then

broken down into chunks of smaller work packages. This enabled the project

manager to delegate work to his team more efficiently. The Gantt chart was then

prepared to graphically represent the duration of tasks against the progression of

time.

“Once we got the green light from the board, a project manager was appointed

for such a case who broke the projects into elements or work packages. Once

those were ready, we made something in Microsoft called a Gantt chart”

There was no definite methodology followed though the interviewee suggested

that their methodology is called a pilot methodology wherein the product was

tested on some trial users; their feedback recorded and then tested on another

batch of users and so on.

“For a software system, we do the basics first, test it on people here and then get

feedback. And then we might look for different audience”


65


Problem: Rigid Project Timelines

The interviewee felt that sometimes checks that are made to ensure that the

project timescales are adhered to; backfire on the project management lifecycle.

“I am going to be honest here; sometimes a project manager can overcomplicate

things and deliver little. So I have always challenged project managers on what

they say because I am one of the persons who is a doer of his own actions. So I

don’t follow the Gantt chart completely”

He reemphasised by stating that often a manager has to use his own conviction

and not just adhere to project management techniques. He felt that for some

projects, it was more useful to directly start work rather than prepare several

charts; the main complaint being flexibility is lost in trying to adhere to project

timelines.

“I don’t need a Gantt chart for every project, especially for the smaller projects”

Phase Four: Verification and Validation

The interviewee commented that there was no separate phase for verification

and validation and that these activities were integrated into the implementation

process. Hence there were no code checks or inspections conducted; feedback

was obtained from the trial users regarding the performance of the system.

“In most cases we rely on feedback rather than quality inspect”

He mentioned that maintaining code quality was not a problem since most of the

developers knew each other and they were able to provide feedbacks to their

colleagues over the phone, email, etc.

66


“We have a team where we know everyone. I know all the programmers. There

could be mistakes but then that can be corrected through a phone call or

something”

Regarding documentation, the interviewee said that they followed an approach of

documenting in the end since they could incorporate whatever they could learn

all along the way. There was no check to ensure accuracy in the documentation.

“If we followed the first approach, we would have to change the documentation

everyday or we have the option to store all the things that we have learnt along

the way and then write the documentation in the end”

A policy of unit testing was followed whereby each work package was individually

tested by the developers. This work package was then tested by the users to

check if it was satisfactory.

“Chunk one does this, let’s test that and get it out of the way. Lets then get

people on board and this is part of the testing. If I am convinced that it is being

used to the maximum, we will then start looking at stage two”


Problem: Lack of user training

The interviewee was concerned that not enough initiative was being taken to

train the end users partly because of hectic work schedules.

“Time is one big issue because we ask too much of people sometimes. And

when time is less, they tend to prioritise and then this (IT training) obviously goes

to the back seat”

Consequence: There would be a resistance to use the system due to the IT

illiteracy. Users would fail to understand the long term implications of not being

67


familiar with basic system usage. However they cannot be blamed since they are

forced to shy away due to work commitments. This point has to be taken note by

the management and significant time and motivation needs to be provided to

create a learning environment.


The interviewee admitted that a product could never be fully ready. He suggested

that it needed to be deployed eventually in order to gain feedback and then

improved accordingly. Thus this phase essentially revolved round obtaining

feedback on the system from the users through focus groups, websites,

conference calls or chat. Once the product was ready, a presentation was given

to the steering group to officially launch it.

“You can always say that an IT product is never fully ready it’s nearly enough, so

let’s chuck at it and see what happens you know. And then you wait for

feedbacks through focus groups and conference calls. You might have a chat

about it. Or you might have a website where you might give a feedback”


Problem: No evaluation and assessment post product launch

The interviewee felt that first and the middle phases are executed quite well

where managers came up with proposals to automate the system and gave

business plans. There was cost benefit analysis, talk about how the system was

going to benefit the company, etc. But there was no emphasis on evaluation and

assessment i.e. no follow up of how a particular product fared, did it really prove

cost beneficial to the company? Did it measure up, etc?

“We divide our development process into nine steps. The first and the middle

phases are executed quite well. The two main things that we miss out are

68


‘evaluate’ and ‘assess’. Have you done it right? Evaluation is feedback for what

you have done. You have done something, was it good, did it feel right? Those

sort of things”

Consequence: The above problem may have manifold impacts. It may result in

a situation where the users are not satisfied with the system because of system

bugs arising as a result of not monitoring the system post launch. The careless

attitude of the company may give an excuse to the development team to evade

their responsibilities and mistakes.

Interview Summary

The interviewee raised several issues during the course of the interview. Being a

project manager, he was particularly worried that the preliminary planning phase

could sometimes lead to inaccurate results and this could affect the entire

project. His concern was compounded when he talked about the disadvantages

of rigidly following project timelines which he felt provided absolutely no room for

flexibility. Another area of concern seemed the lacklustre attitude of the company

when it came to user training. Lastly, he also expressed his displeasure about

not having an evaluation and assessment post launch. He stressed on the

importance of justifying what was done.

5.3 Interview Three – User One

This interview with the user took place on July 21, 2004. The meeting lasted

approximately one hour. It may be worth noting that the end users are not

actively involved in all of the phases in the development process and this can be

observed in the analysis.


69



70



The interviewee commented that he was not involved in the preliminary planning

phase. Furthermore, he was not aware how the activity was carried on in the

company.

“As far as I am aware… I can’t remember there been anything specific about

asking me to comment on requirements before the product was made”


The interviewee did not have a concrete idea of the way this phase is typically

carried out and the same had to be explained to him by the researcher. Upon

explanation, the interviewee identified the following problems:

Problem: Lack of user input

The interviewee felt that there was no user input at this phase which was

annoying because as end users, their comments and suggestions were not

considered before the system was developed.

“It was felt as if there was no real input from the engineering side for us as the

end users”

Consequences: An important point to be considered while developing a system

is user needs and preferences because they would be using the system

eventually and it is important to know what's on their mind. Failing to do this

could affect the succeeding stages of development. It could also result in the

system not being user friendly when it is eventually rolled out.

71



The interviewee was not involved in this developmental phase. This phase is

concerned with identifying the hardware and software requirements needed to

implement the product and the procurement of the resources. The system in

question was developed for in-house purposes. Upon investigation, the

researcher found out that the hardware and software needed to develop this

product was already in place. In this context, the user’s role in this phase does

become limited though he could have been consulted to get a better idea of the

requirements.


The interviewee was not involved in the product design and development

process. Again, he had to be explained by the researcher about what the phase

meant and how it was executed. He could not identify problems in this phase

since he was not involved at all. He seemed satisfied when asked about the

performance of the system.

“Yes the system is pretty much good. I mean that’s what it is really for”

He also felt that this stage was very crucial in getting user input since a good

design ultimately leads to a good end product. So it was vital to know what the

users exactly wanted

“If they had come and said, you know this is what we have got at the moment,

this is what we are looking at, then people could have said, I don’t like that, we

can alter that, but its come along and we had to run for so long and slowly but

surely people start to take notice of what people were saying and things started

to change slowly”

72


Phase Four: Verification and validation

The researcher wanted to verify whether users were involved in the documenting

and testing process. The interviewee seemed happy that his inputs were

considered at this stage. He commented that the company used techniques such

as focus groups, emails, questionnaire or even personal interviews to gain an

understanding of what the users wanted. Furthermore, he said that this process

was iterative and the development team used to come to users very often to take

their inputs.

“Sometimes they do online surveys, sometimes they come and ask face to face,

sometimes team meetings and stuff like that. This goes on again and again”

The interviewee said that the system had good user documentation and he was

involved in creating the documentation. This brought out the premise that users

were involved in creating the documentation and help files.

“Yes we had quite a big hand in that one, as you said before it was rolled out and

put into the system”

Regarding testing, the user commented that he was a part of the team who was

involved in testing when the system was put through trial. This helped him

understand the system better and provide feedbacks.

“Well, they came along and told us what their brief was; they had already made

some initial starts on what they thought that we wanted. We then told them how

we wanted it. They then went away and tried to make it as much as how we

wanted it rather than they just saying…right that’s how you use it. They actually

came back and showed us how it fared and whether it was going the right way.

We then knocked it up a bit more and they then came back with what was

73


something like a trial and we went and tried that for them and we then used it for

a while and gave them feedback on what is good and bad”

“I think many companies are now beginning to realize that users can also come

up with some good ideas”


Problem: Majority of users do not get a chance to test the system

The interviewee commented that a very small percentage of the end users are

involved in testing the system and providing feedback during the trial period.

“Some of us get a chance to play around with the system but a lot many aren’t

that fortunate early on”

Consequence: This would result in a situation where some users would be able

to use the system much better than the others. This may probably impact their

work. Also, it could create a situation where users could develop a resistance to

the system simply because they aren’t able to use it well enough. This point was

also echoed by the interviewee.

“Of course there were help desks to help you out and you could ring them up or

some lads who were obviously a little more sharp but you did feel a little bit left to

struggle on a bit. So some people hated it and there are still a few that really

despise the thing”


The interviewee was not involved in this phase and was not much aware of how

this phase was conducted in companies. Again, since the product was developed

74


for in-house purposes, his role does become limited in a sense. However post

release; the interviewee did identify some problems which are mentioned below.


Problem: Lack of user training

The interviewee seemed concerned about the fact that the company does not

invest enough in getting all the users to a certain standard in computer usage. He

admitted the fact that some users were way more tech savvy than the others.

“Sometimes people don’t have an idea of even emptying their emails on a regular

basis. Suddenly they get an email saying that your mailbox is full, please empty

your emails”

Consequence: As discussed earlier, the lack of investment in the area of user

training is a cause of concern. There is no point investing time and resources in

developing a system when the end users are not in a position to use the system

effectively. The disparity in the knowledge level of users may create frustration in

the minds of some users which could impact their work. Due to the low IT literacy

levels, the development team would end up getting lesser feedbacks from the

users and hence find it difficult to recognize and fix bugs.

Interview Summary

Most of the concerns of the user revolved round two distinct areas; user training

and user involvement in the development process. He was concerned of the fact

that the company was not investing enough in IT training. He also seemed to

suggest that users should be involved in the development process right from

concept rather than only during testing. The argument was extended further

when he felt that some users did not even have a chance to participate during

75


product testing. He was not involved in all the stages of development and hence

he could not identify problems in those stages.

5.4 Interview Four – User 2

This interview with the user took place on July 21, 2004. The meeting lasted

approximately one hour. As already mentioned earlier, the end users are not

actively involved in all of the phases in the development process and this can be

observed in the analysis.

The concept map for the interview is given below:

76



77



When the researcher specifically asked the interviewee if he was consulted

before the system was developed, the immediate answer was a NO. When the

researcher explained this phase to the interviewee, he did express his regret that

they were not asked for their inputs. He suggested that a lot of niggling issues

which keep cropping up after the system is released could have been avoided if

the users were consulted before the system was developed. However when the

researcher asked the following question:

“If you were given an opportunity to provide an inputs before it was made, what

exactly would you like to have in the system?”

The interviewee could not identify clear areas where he felt the system could be

improved. He replied:

“It is hard to say, it is a paperless world now but our guys are still very

comfortable with pen and paper than computer. Anything that is complicated,

they will stay away from it”

Thus though the interviewee said that a lot of problems could have been solved

by obtaining inputs from the users at an early stage, he could not identify

problems when specifically asked.


As in the case with the other user, this user was also not involved in this

developmental phase. This phase is concerned with identifying the hardware and

software requirements needed to implement the product and the procurement of

78


the resources. The system in question was developed for in-house purposes.

Upon investigation, the researcher found out that the hardware and software

needed to develop this product was already in place. In this context, the user’s

role in this phase does become limited though he could have been consulted to

get a better idea of the requirements.


The interviewee was not involved in the product design and development

process. Again, he had to be explained by the researcher about what the phase

meant and how it was executed. He could not identify problems in this phase

since he was not involved at all. Like the other user, he too seemed satisfied

when asked about the performance of the system.

“Yes pretty much happy about performance. Obviously we can get a lot more out

of the system if we want to. I don’t think there’s anything missing in the system as

such. It’s just the extra bit of knowledge about how the system works”

Phase Four: Verification and validation

The researcher was interested in finding out whether the interviewee’s inputs

were considered in two main areas; documentation and testing. The interviewee

had a lot of information to share regarding this phase. The researcher thus took

extra time to discuss all the concerns one by one.

In the case of documentation, the interviewee commented that though some

users were consulted, his inputs were not taken.

“So what you are saying is that they did take feedback from the users for

documentation but they didn’t ask you?”

79


“Yes you are right in what you are saying. I knew it only after it came out.

Everything was well thought out and targeted”

He also emphasised on the point that even though he was not consulted for the

inputs, the documentation was of a good standard. He had no problems with

using it. He however reiterated that he was able to use it efficiently because he

was IT savvy whereas majority of the users were not. He felt that the company

should invest more resources to get the users to some standard level.

“Yes the documentation is very easy to use. They are like diagram files and they

give you everything diagrammatically. Personally I have benefited with a lot more

IT training than others. I think others have not been that fortunate. The company

has to target those people”

The interviewee added that the company also obtained regular feedbacks from

the users though again, he was not personally involved.

“Mostly it is through email. Drop whoever it is an email. Jot it down through an

email and send it through does the program work well? Is it easy to use or

whatever”

As regards testing, he said that the users were not involved in the testing

process. But he did not agree that it was a problem since the system was more

or less able to satisfy all the needs.

“Before it comes to us, it’s all done at the shop floor at our main place and it is

nice and neat. So I don’t think I have a problem with that”.

Problems faced by the interviewee in this phase

Problem: No sufficient investment in user training

80


The interviewee seemed perturbed by the fact that the company is not spending

enough in training users. He talked about this problem again and again which

gave an indication that it is really a serious issue within the company. He

however agreed and seemed happy of the fact that though the company did not

invest much resource in user training, preference (whenever training happened)

was always given to people who had no IT skills.

“Generally they pick on users who are not very smart; those engineers who don’t

have great IT skills so that they can also benefit from the training”

Problem: Hectic work schedules

The interviewee felt that the very hectic job schedule meant as a deterrent to

users to explore and learn IT on their own. He felt that the managers chase them

all the time which leaves them with no leisure time to explore the training

modules online and learn from them.

“It is difficult for the users because out there, we are pushed from pillar to post to

get more jobs done. The manager’s task is to get more work done; I don’t mean

to be nasty here. So there’s just no time to do some IT training or whatever it is”

Consequences for the above problems: As already discussed earlier, the

impact of not training users with IT skills could be disastrous in the long run. The

problem was further compounded with the hectic work schedules. This concern

was echoed by both the users interviewed which obviously meant that it was a

serious issue in the company. A software system can succeed only if the end

users are trained to use it efficiently. The interviewee felt that often development

teams do not understand the fact that all end users are not IT savvy.

“Lack of knowledge is an issue for us, not for the programmer”

81



The interviewee did not have much idea about what this phase meant. When

explained by the researcher, he said he was not involved in this phase and was

not much aware of how this phase was conducted in companies. He however

stressed on the same problems that are mentioned in the earlier phases. This

has not been discussed again in this phase.

Interview Summary

Throughout the interview, the interviewee seemed concerned about the fact that

some users were not trained up to the basic IT standard. A similar thought was

shared by the other user when interviewed. He however mentioned that this

approach was beginning to change and the company had started to invest time

and resources into training users, especially with poor IT skills. He also stressed

on the need to be involved right from the beginning and that changes proposed

to be made in the system needed to be presented to the users to gain an

understanding of their thoughts.

“I would like them to personally ask us what we would like. I want us to be more

involved with the changes that are going to be made and put forward those

changes to us before they are actually implemented, see how we can use them

before they are implemented and obviously any changes if need to be done”

5.5 Conclusion

The extensive interviews carried out with four different staff members each of

whom are associated with the system in a different role was very fruitful in

understanding the methods employed by the company, the problems faced by

each of the interviewees and the consequences that might arise out of those

82


problems. Viewpoints from the development, management and analytical

perspective were obtained which provided an in depth view to the researcher of

the issues which have faced this project. However, each of the interviewees

differed significantly in the knowledge of the system and this did have an impact

on the collection of data.

83

Chapter Six: Discussion of Results and Recommendations

Chapter Six Discussion of Results and

Recommendations

6.0 Discussion of Results & Recommendations

Chapter four discussed the methodology that was used to collect the data for the

purpose of this dissertation. Chapter five analysed in detail the data collected

through the four interviews using the technique of concept mapping. The

methods employed by the company and the problems in each of the phases of

software development were looked at from the viewpoint of each interviewee.

The data that was collected through this approach will be discussed with respect

to the ISO 9000-3 standards.

In other words, there would be a comparison between what should have been

done and what was actually done. A series of recommendations for each phase

would be provided to the company for recognising their pitfalls in the

development process. These recommendations could be adopted for similar

future projects which are conducted in the company and the researcher feels that

this will surely help the company to have a smoother development process.

A single concept map was prepared for every interviewee interviewed in the

previous chapter. This concept map distinctly showed the methods employed by

the company for every phase and the problems faced in every phase from the

viewpoint of each interviewee. This chapter will focus on preparing a single

concept map for every phase which would contain all the methods and problems

stated by all interviewees. The methods employed by the company would be

discussed as a comparison with the methods that need to be employed as per

84


the ISO 9000-3 standard. The problems faced are not discussed in detail as

these were already done in the earlier section. The emphasis is to enable the

researcher to perform a horizontal analysis and look at each phase with more

depth.

The researcher wants to state that the ISO 9000-3 framework is extremely

comprehensive and hence it has been carefully analysed and only those issues

relevant to the company and to the system developed are considered for the

purpose of this discussion.

It is also worth noting again that since the software was developed for in-house

purposes, the customer is the BT management; however they are not the users.

Hence there is a clear distinction between the end users and the customer. This

understanding will be useful to appreciate the difference between user

requirements and customer requirements. Also note that supplier in this case

means the development team in the company.

The concept maps are explained as follows:

• Methods employed by the company for each phase are represented by a

blue rectangle.

• Problems encountered in the execution of each phase are represented by

the yellow oval.

• Consequences of problems arising in the execution of each phase is

represented by the green rectangle

• Conflicts if any, between different interviewees are represented by the

grey oval.

6.1 Product Specification and Preliminary Planning

The ISO 9000-3 Software Process Handbook (SPH) states the purpose of this

phase as follows:

85


“To identify the customer’s needs, product features to meet those needs, and a

preliminary estimate of the resources required to create the product features”

The deliverables out of this phase as specified in the Software Process

Handbook (SPH) which is used as the quality manual for this dissertation is

stated as follows:

Deliverables: 1. Marketing requirements document OR equivalent document

2. Preliminary budgets and schedules OR equivalent document

3. Detailed budget and schedule for phase two OR equivalent document

Exit criteria: 1. Marketing requirements document

Signature of any competent authority like V.P engineering, marketing,

sales, etc

2. Preliminary budgets and schedules

This is reviewed by the concerned department, often the marketing and

the engineering department and the sign off is obtained

3. Detailed budget and schedule for phase two

This is reviewed by the concerned department, e.g. the engineering

department and the sign off is obtained.

86


Figure 6.1 Product Specification

87


Analysis of the phase

This phase is essentially concerned with identifying in detail, the customer’s

requirements and the proposed product’s functional, interface and technical

requirements so that the customer’s identified needs could be met. Both, the

development team and the customer have a responsibility to ensure that the

requirements are complete so that the succeeding stages of development are

smooth.

Summary of the methods employed by the company

A cost benefit analysis is undertaken by the company to ensure that the product

that is proposed to be developed would yield a measurable benefit to the

company in terms of time, cost, ease of use, etc. The feasibility study is

conducted and the phase is terminated with the preparation of a proposal

document. The team then goes through a brain storming session (according to

the project manager) with the end users to determine their requirements.

Discussion of issues and problems in the current methodology with a comparison to the ISO framework requirements

During the interview with the project manager, the researcher observed that he

was concerned only about the cost benefit of the product to the company. This is

of course a very valid issue, but it should not be the only consideration. Care

should also be taken to obtain an understanding of the user requirements so that

there is no problem in the succeeding stages. The aspect of considering user

requirements is completely missing in this phase. All activity is centred on the

cost benefit of the product. The customer might realise later on that it would have

been worthwhile to discuss functional requirements also along with cost benefits

of the product.

88


Thus ideally, the supplier should take care to obtain a complete understanding of

the customer’s requirements. The requirements must be formulated so as to

obtain an unambiguous and complete set of the functional requirements. The

requirements must be stated precisely so that there is no problem during later

phases. It is understandable that requirement analysis can never be fully

complete, but a high level understanding of the customer’s requirements needs

to be obtained.

The most important activity of this phase is the preparation of the purchaser

requirements documents (also called as the Marketing Requirements Document).

This document was missing in the company. Infact there was absolutely no

documentation at all to understand customer’s specifications. As mentioned

earlier, this is a serious issue and needs to be addressed immediately. It often

happens that the supplier and the customer fail to sit together and work out the

requirements specification and this causes confusions and misunderstandings

later on.

There was a cost benefit analysis undertaken and a proposal document prepared

but this related more to a feasibility study of the system rather than determining

the customer requirements.

There was a conflict when the manager suggested that there was a

brainstorming session with the users to understand their needs but this was not

accepted by the user when interviewed.

In order to smoothly proceed with the development work, the company should

take care that the development team has a complete and unambiguous set of

functional requirements. These should be made in consultation with the customer

and must include everything to satisfy his needs namely i.e. performance, safety,

reliability, security and privacy. It is very important that at the end of this phase,

89


the development team is able to obtain an unambiguous set of high level

requirements.

Other areas that need consideration are definition of terms to prevent

misunderstandings, recording and reviewing discussions between the

development team and the customer, methods for agreeing on requirements and

approving changes, etc

6.2 Engineering Specification and Planning


phase as follows:

“To identify the software and hardware requirements needed to implement the

product features specified in the Marketing Requirements document and to

identify and obtain the resources (e.g. engineering hours, hardware, software

and schedule) needed to construct and test the system that implements the

requirements”



stated as follows:

Deliverables 1. Feasibility prototype (optional) OR equivalent document

2. Software requirements specification (SRS) OR equivalent document

3. Software development plan (SDP) OR equivalent document

• Development costs and schedules

• Documentation plan (important)

• System test plan (important)

90


• Alpha evaluation plan

• Beta plan

Exit Criteria 1. Feasibility prototype (optional)

This is reviewed and signed off by the engineering department

2. Software requirements specification (SRS)

This is reviewed by various departments. Appropriate signatures should be

obtained (V.P engineering, V.P Marketing, project manager and engineering

heads).

3. Software development plan (SDP)

This is reviewed by various departments. Appropriate signatures should be

obtained (V.P engineering, V.P Marketing, project manager and engineering

heads).

91


Figure 6.2 Engineering Specification

92



As discussed before, the main goal of this phase is to identify the hardware and

software environment needed to implement the product and to arrange for the

resources needed to construct the system.

In the last phase, we identified the requirements of the customer. In this phase,

we are concerned with chalking out a development plan to deliver the product

that would meet all the requirements of the customer. The development plan

depends upon several factors such as the customer’s requirements, engineering

practices used by the development team, date the customer requires the product,

etc.

Methods employed by the company

All the interviewees had almost no idea of the objectives of such a phase. They

confused this phase with the objectives of the last phase. The researcher had to

go into great lengths of detail to justify the existence of this phase. The absolute

lack of knowledge of this phase can be seen from the very small concept map

prepared for this phase.

The company performed a more detailed cost benefit analysis to get a final go

ahead from the management. The findings of the analysis were summarised in a

document called the business case document which was presented to the

management. Since the Training and Development System was developed in-

house for in-house purposes, the hardware and software required to run the

system were already in place and it was just the question of using the existing

hardware. These issues were discussed by the project manager, the developer,

and the server manager in a meeting. However, the researcher could obtain no

documentation of the above meeting.

93


The business case document and the results of the meeting between the project

manager, developer and the server manager were presented to an independent

body that then performed an analysis and decided the next step of action, go

ahead or scrap. Obviously in this case, the benefits of automation were much

more than doing the task manually, hence the project was decided to be

pursued.

Throughout, the researcher observed that the company has no documentation

for this phase. Infact, most of the development phases have little or no

documentation. This shows the company’s laid back policy on documenting what

they do. The effects of such a careless attitude could prove disastrous in the long

run.


First of all, the company needs to understand the importance of this phase and

take steps to treat this as a separate phase instead of confusing the deliverables

of this phase with the earlier phase.

The company should prepare a development plan which would show how each

of the succeeding phases would be implemented explicitly identifying the

following:

• Inputs and outputs arising out of each phase The development plan should identify the inputs that are required in order

for the activities to be performed and the outputs that will come out as

deliverables which will be used in the succeeding phases. These are

identified in the Software Process Handbook (SPH); however, the

company needs to spend time to understand the inputs and outputs that

could be tailored for its own product.

94


• Schedule and resources required for executing each phase The development plan should identify how the project would be managed.

Milestones and action if any based on the status of the project should be

identified. It should also identify the reviews that would take place and the

methods of obtaining those reviews and obtaining status. Detailed

schedules need to be worked out and the resources required in creating

deliverables within the time frame need to be planned. A special thought

needs to be given to external dependencies; i.e. factors that are beyond

the control of the company because these often have a significant impact

on the success or failure of the project.

• Tools and methodology to be used to conduct each phase The development plan should also consider the tools that would be used.

A popular trend in the industry today is the use of CASE (Computer Aided

Software Engineering) tools in analysis, design, implementation and

testing. Their use would affect the project costs and schedules. If they are

proposed to be used, the development plan should identify the training

requirements for using those tools and techniques. An emphasis should

also be placed to identify the conventions, agreed upon rules and

practices. This would ensure that there is no confusion amongst the

development team, especially when it comes to producing the deliverables

associated with every stage.

• Verification procedures An important area in this phase is the identification of the verification

procedures which would ensure that the by product is correctly tested.

This can be done only when the practices and procedures that are used to

verify the product are correctly thought out.

95


As stated by the researcher all throughout, since the ISO standards can be

completely customised according to the policies and procedures prevalent, the

company needs to carefully assess the current scenario and accordingly prepare

their customised development plan. For example, there may be separate plans

for development, testing and quality assurance. It does not matter whether there

is just one comprehensive plan or several modular plans as long as the company

identifies the basic issues that need to be tackled.

Thus essentially, it is the methodology for transferring the customer’s

requirement specification into a finished product that the company needs to

concentrate on doing.

6.3 Product Design and Implementation

The ISO 9000-3 Software Process Handbook (SPH) has two separate phases;

product design and product implementation from where this phase is adapted.

The justification for such an adaptation is already stated in the earlier chapter.


phase as follows:

The purpose of product design is to define the functional design, interface

design, database design and error handling design and also to identify the test

cases and create an outline for all product documents

The purpose of product implementation is to implement the software,

documentation and system test designs. It is also concerned with proving that the

software developed works with the minimum level of integration

96




stated as follows:

Deliverables (design)

1. Software design document OR equivalent document

2. System level test case definition OR equivalent document

Exit Criteria (design) 1. Software design document

This is reviewed and signed off by the project engineering lead

2. System level test case definition

This is reviewed and the appropriate sign off from the VP Engineering,

director of marketing, project manager and the various other leads is

obtained.

Deliverables (implementation) 1. Base lined software product OR equivalent document

2. Unit tests OR equivalent document

3. Unit tests results OR equivalent document

4. Function integration tests OR equivalent document

5. Function integration tests results OR equivalent document

6. System level test cases OR equivalent document

Exit Criteria (Implementation) 1. Base lined software product

2. Unit tests, Unit tests results (optional under configuration control)

3. Function integration tests, function integration test results

4. Product documentation

5. System level test case description

This is reviewed and signed off by the concerned authority.

97


Figure 6.3 Product Design

98



As noted earlier, the purpose of this phase is to define the design criteria and

then implement the product according to the design criteria. It is also concerned

with proving that the software developed works with the minimum level of

integration.

Design is a very important activity and often the work put in this phase has a

significant effect on the final product. The quality of the product’s design impacts

its usability and maintainability. Hence the guideline suggests careful

consideration of design methodologies, design rules and guidelines. Design and

code reviews are an important part of this process.

Methods employed by the company The researcher obtained comprehensive views about the methods employed in

the company from the project manager and the developer. The two users did not

have much to say about this phase. However, when the researcher explained to

them the importance of the phase, they were anxious that their inputs had not

been considered in this phase.

A Statement of Requirements (SOR) was prepared which contained the

functionality of the product as requested by the customer. According to the

developer, user feedback was then obtained and their inputs were incorporated

into the SOR. However when the users were interviewed, they said that they had

not been consulted though they were happy with the performance of the system

and it didn’t matter to them whether they were consulted or not as long as the

system worked well.

A model of the system was prepared and shown to the customer and the users to

obtain their feedback. Their feedback was incorporated. The developer then

99


designed the database requirements which the project manager worked on

dividing the project into sub packages or elements. A technical specification

document was prepared and presented to the project manager by the developer.

The project manager then delegated responsibilities to the developers and

instructed them to begin coding.


ISO standards state that design and implementation activities are those which

transform the customer’s requirement specification into a complete software

product. Often software products and development lifecycles are complicated

and it is up to the company to ensure that this phase is correctly carried out so

that the end product meets the specification of the customer. Thus the emphasis

should be on monitoring the process and not just depending on the test and

validation activities for the assurance of quality.

One of the huge challenges faced by the company in this phase is the lack of a

methodology. Both the project manager and the developer seem to think that

following a methodology was a hindrance rather than an advantage. The

developer was at least a bit forthcoming when he mentioned that a methodology

was probably needed to make the activities more systematic but only as long as

it did not interfere with the product delivery.

Developer:

“In an ideal world, all stages need to be followed for all products. This doesn’t

happen in reality. For my products, I follow my own methodology”

Project Manager:

“Not having any methodology is probably our biggest advantage”

100


The company should thus immediately adopt a systematic design methodology

which would be appropriate to the type of software product developed.

Everybody understands that one methodology cannot be applicable for all

products, but this does not mean that there should be no methodology followed.

The company should also keep record of their earlier project so that similar

issues and problems faced earlier can be avoided. Thus it can avoid ‘reinventing

the wheel’.

it should be taken care to ensure that the design policies are framed in such as

manner to as to facilitate testing, maintenance and use. These are very important

considerations because a product could be rejected if it cannot be tested or used

in the operational environment.

Regarding implementation, the guideline suggests that the supplier use

appropriate conventions for mundane subjects such as naming convention,

coding and commentary. In this context, it may be worth pointing out that the

developer was not satisfied with the team formation. He felt that since the project

manager was non technical, he could not offer him any inputs and majority of the

development work had to be planned by the developer. In such a scenario, it

becomes difficult to have any naming convention for coding, commentary, etc as

developers may do things the way they want. It would also result in too much

pressure on developers apart from resulting in a chaotic software development

process.

“We had a project manager, a developer and more often than not, we have a

support person but I sort the time, resource, documentation with the support, sort

the coding, trial groups, roll out etc.”

Finally, the company must formulate implementation methodologies and tools to

satisfy purchaser requirements. Failure to identify the methods and tools and

101


train engineers in their use may affect the succeeding stages and ultimately

affect the quality of the product. However, the developer stated that most of the

members in the team had no exposure to system design techniques. This he

said, affected the way people approached the whole development process.

“Most of the software people here haven’t had any formal training in system

design; they all have had training on how to write code, but system design is

something that… I have 13 of us in my team and only 3 have had some sort of

training on it. So I suppose it is due to lack of training really, lack of any

standardised training”

The company must realise that training in system design is a must if the

development team have to follow good software engineering practices. Failure to

train personnel will obstruct the development effort and will have a damaging

effect in the succeeding stages, ultimately affecting product quality.

6.4 Verification and Validation

The ISO 9000-3 Software Process Handbook (SPH) has a phase called System

Test from where this phase is adapted. The justification for such an adaptation is

already stated in the earlier chapter.

The principle objectives of this phase are to determine whether there are any

defects in the system. This is achieved through testing. It is also concerned with

assessing whether the system is usable in an operational situation. There is also

a significant emphasis on creating a good documentation (user and technical).



stated as follows:

102


Deliverables: 1. System test results OR equivalent document

2. System test reports OR equivalent document

3. Updated software design document OR equivalent document

4. Base lined product OR equivalent document

5. Product documentation

6. User documentation

Exit criteria This phase is complete when the software product has passed all system level

tests and the software design document reflects the as built design of the

product.

103


Figure 6.4 Product Verification and Validation

104



As mentioned above, the primary objective of this phase is to ensure that system

test has been carried out effectively and that the documentation (technical and

user) has been prepared to the satisfaction of the concerned parties.

Testing is a very time consuming activity and it must be well planned so that it

can be executed without any problems. Within testing, there are various levels

each of which focuses on a different objective. It is for supplier and the

customers to decide areas that can be covered in this phase, the management of

resources, test environments, etc.

Documentation forms the basis of every development life cycle and there must

be a conscious effort to ensure that documentation is exhaustive and accurate.

Any proposed changes to the documentation must be reviewed by a senior

committee, authorized and then carried out.


The researcher obtained contrasting views about the methods employed by the

company in this phase. Obviously each of the four interviewees play different

roles in the development of the system and thus their perception of the activities

carried out differ. This has all been summarised in the concept map. Conflicts in

the views of interviewees have also been highlighted to get a better

understanding. All the interviewees agreed that this stage was essentially

concerned with testing and documentation. There were conflicting views from the

developer and the project manager about the documentation process. The

project manager mentioned that documentation was done only one time, in the

end.

105


“If we followed the approach of documenting as we go along, we would have to

change the documentation everyday. We have the option to store all the things

that we have learnt along the way and then write the documentation in the end”

The developer mentioned that documentation was done on a regular basis so

that things could be more systematic.

“No standard way of doing it. Mostly we do documentation as we go along and

show everything to the client in the end but there's no standard way”

A conflict was also observed in the case of testing policies. The project manager

mentioned that the project was broken down into work packages and each of that

was tested individually.

“Chunk one does this, let’s test that and get it out of the way. Lets then get

people on board and this is part of the testing. If I am convinced that it is being

used to the maximum, we will then start looking at stage two”

No such methodology was stated by the developer. He said that there was no

policy for testing and the product was tested only during the trial period.

“We do testing during the trial, I know trial should not be used for testing, but

that’s the way it is”

“If we saw some problem, we would debug it on the spot and then test it again.

There's no separate testing phase. If the software has a problem, then the

database manager would call me and inform me and I have to debug my area of

the code”

The users interviewed primarily disagreed on the methods used to gather input

from them. One user was more involved than the other and this affected his

perception of the stage. The other user was not involved much in the stage. More

106


of this information can be seen in the earlier section. Furthermore, one of the

users stated that input was obtained only once whereas the other said that

obtaining input was an iterative process.

“I tend to think that they take inputs from us only once or twice”

“Sometimes they do online surveys, sometimes they come and ask face to face,

sometimes team meetings and stuff like that. This goes on again and again”


Testing is considered to be an extremely important activity in the ISO

specifications and it is stressed that a great deal of planning needs to be

undertaken so that the phase is executed without any problems. Infact, it is

stated that testing activity needs to be conducted at every stage to ensure that

the end product is reliable and is delivered within the time constraints specified

in the contract and problems experienced in one phase do not trickle down to the

succeeding phases.

In the case of the company, to begin with, there was neither a testing phase nor

a policy on testing. This is a big loophole and could have disastrous

consequences in the long run. There should be a discussion immediately

undertaken regarding this matter and a consensus should be reached. A

framework or policy needs to be established to measure up to the ISO framework

which describes in detail the issues that need to be considered in each phase

like requirements, design, code and test with separate deliverables. The

researcher wishes to reiterate the flexibility of the ISO standard wherein it is

mentioned that depending on the size and complexity of the product, the test

plan can be a standalone document or part of the Software Development

107


Process (SDP). This flexibility to customise the process is one of ISO’s biggest

merits.

Once a policy regarding testing is established, the company needs to plan out

their test activities. Examples of these are provided in the ISO framework which

is as follows:

• Types of testing

• Test cases

• Test environment

• Resources and schedules required to create the test

• Resource and schedule required to execute the test

• Entry and exit criteria for each phase of testing

• External dependencies

• Test tools

• Technical, budget and schedule risks

This is important and needs to be conducted to support the process. It would

provide a very good overview of the resources required to perform the testing

and would help in the orderly creation and execution of the test cases. All these

need to be reviewed and approved by the senior management with a high

priority.

There should then be a more detailed review of the test plans with a special

consideration to plans relating to the software item, integration, system test and

acceptance test. Care should also be taken to carefully select test cases and the

method of recording test data and test results. Test results should be recorded as

a part of permanent documentation. This will enable to company to check if they

meet the expected results, provide a proof to the customer, and also give a

benchmark to the company that can be used to compare after future tests.

108


A careful thought has to be given on the testing team, the test environment, tools

and test software. At the moment the company has no tools or environments for

testing. Everything is conducted during trial and user feedback is obtained.

However, software testing may require a separate test environment with its own

hardware and software. Such a possibility should be considered and discussed

with the management tem. The project manager should ensure that the

environment needed to carry out the tests has been identified and put in place.

Also, the number of persons who will constitute the part of the test team should

be clearly identified and trained to carry out the testing activities identified.

The guideline considers validation to be a test that is performed by the supplier

on a version of the product that is intended to be delivered to the purchaser.

Validation thus needs to be performed by the development team on the version

of the product that is intended to go Live. Software testing can occur before this

phase, but there should be a complete system level testing to ensure that the

product meets the functional and technical requirements.

Regarding documentation, there has to be a conscious effort that it is done all

along and not hurriedly in the end. Documentation for most of the company’s in-

house IT products is divided into user documentation and technical

documentation. It is the same for the Training and Development system. The

company has no policy in store for documentation. As pointed out by all the

interviewees, this sometimes leads them to a very difficult situation in their

everyday work. The company’s laid back attitude on documentation can be

summed up by the engineer’s words:

“Infact, there's no enforcement for documentation. We do good documentation,

some documentation or no documentation. However the user documentation is

good since we follow a standard template and it does turn out to be good. The

technical documentation is different or sometimes non existent”

109


For the user documentation, the company should take care to describe how the

product is used as well as any bugs, etc in that particular version. Test cases

may be created to ensure that all functionality is incorporated in the

documentation. The test plan should schedule the resources required to create

the test cases on the user documentation and then execute the test cases.

The product documentation is described in the framework as planning

documents that describe the progress of all the activities of the supplier

(development team) and his interaction with the purchaser (BT management).

This issues a lot of issues and the company has to make a decision as to what

should be included and what should not. But in general, inclusion of the following

must be considered:

• Development phase inputs

• Development phase outputs

• Verification and validation plans and results

• Documentation for purchaser and user

• Maintenance documentation

The above documents form the basis for the development activity and hence

there should be a strong process to support their maintenance and distribution.

6.5 Product Release

The ISO 9000-3 Software Process Handbook (SPH) has a separate phase for

product release. The SPH defines the objective of the phase as follows:

The principal objective of product release is to ensure that the product masters

delivered to the production vendor represent the product baseline and to assure

that the production versions reflect the product baselines.

110




stated as follows:

Deliverables: The output for this phase is the actual product which is prepared for release.

This is further divided into:

• Release to manufacturing OR equivalent document

• Release to customer OR equivalent document

111


Figure 6.5 Product Release

112



As mentioned above, the objective of this phase is to ensure that the product that

is released to the customer meets the standards that were set by the supplier

and the customer at the beginning of the contract. Also that system test has been

carried out effectively and that the documentation is complete and accurate.

An important activity that is carried out to meet the objectives of this phase is

called acceptance testing. An acceptance plan identifies key issues such as

product delivery, installation and availability of key personnel needed to deal with

the installation and testing effort as well as any problems arising out of the

activity.


The company does not have a separate phase for this activity. Infact, the

researcher could gain very limited knowledge about the manner in which these

activities were executed in the company for the Training and Development Live.

The interviewees seemed to have no clue about this phase and were not ready

to accept that there had to be a process to tackle activities covered in this stage.

Most of the interviewees had a very superficial knowledge of this phase and the

researcher had to explain really hard to make the interviewees understand the

objectives of this phase.

The company conducted an evaluation of the product to ensure that it is made

according to the standards specified with the customer. Feedback through focus

groups, interviews, questionnaire etc is perfumed to gain an understanding of

customer and user standards. If the product does not meet the criteria specified,

it is put back to test. It is finally launched when the product meets the criteria.

The above three activities seem to suggest that the process is very iterative. This

however could not be completely confirmed since each researcher gave a

113


contrasting view and the methods only seem to suggest that an iterative process

exists.


As mentioned earlier, the core activity in this phase is the acceptance testing.

The company has no major activity in this phase except obtaining feedback from

the users regarding the system. This is a gaping hole and needs to be corrected

immediately. There may not be a separate phase for product release, but the

activities need to be carried on to ensure that the product released meets the

customer’s requirements.

To begin with, the development team should have methods to ensure that the

product being delivered meets requirements and operational use goals.

The company should also take care to ensure that the contract correctly states

purchaser and supplier contract separately. It is very important to note that in this

phase, supplier and customer responsibilities are equally important. The

company also needs to implement a plan for problem solving procedures and

solve these problems during acceptance testing. What the company is doing at

present has nothing to do with this phase at all. There are no formal procedures

to record this phase.

The acceptance test plan must ensure that the product is tested accurately so

that any bugs discovered relate to the product and not to the faulty test cases.

For this reason, a special emphasis needs to be provided to the following areas:

• Time schedules

This is important to manage resources and synchronise any possible

activity.

114


• Procedures for evaluation The supplier and the customer must agree to the test cases. There must

be a consensus that any procedures decided by the supplier and the

customer would be a legitimate test for the product.

• Software and hardware environment and resources The acceptance planning should identify the hardware and software

resources to test the product and the activities required to prepare that

environment for testing the product

• Acceptance criteria The supplier and the purchaser must work out the acceptance criteria

together. This would ensure that there are no misunderstandings later on

between both the parties.

The company also needs to take a serious note of other concepts mentioned

herein. The researcher feels that these activities are up to the company to decide

if they want them or not though following these activities will surely prove

beneficial to the company in the long run. The important ones are called

replication, delivery and installation. Replication and delivery are activities that

are performed after a product has been developed and tested and must be sent

to the customer. Installation obviously follows the above activity and may require

coordination between the supplier and the customer.

Replication

Replication is a very important quality assurance activity. It should be

remembered that a great deal of resources is spent by the company to arrive at

the finished product. It is thus important that everything goes well in the last

phase. Issues such as shipping the right version of the product, a complete and

fully working product, etc should be carefully monitored. Consideration must be

given to other areas such as number of copies of each software item, thee type

of media, custody of master and backup copies, etc.

115


Delivery

Delivery as the name suggests is concerned with delivering the actual physical

and complete product to the customer. Care should be exercised to ensure the

correctness and the completeness of the product delivered.

Installation

Sometimes the product developed should be installed in the customer’s

premises. This is an important activity and would need utmost cooperation and

understanding from both the customer and the supplier. The roles and

responsibilities of both the parties must be clearly established. Other areas such

as suitable time for installing the product, access to purchaser’s computer

systems (including username and passwords), skilled personnel, procedure to

obtain customer’s approval, etc need to be clearly worked out.

6.6 Conclusion

The chapter provided a detailed discussion on the findings of the four interviews

conducted by the researcher. These findings were compared with the ISO 9000-

3 framework and recommendations were provided in each of the five ISO 9000-3

software development phases highlighting the shortcomings in the company’s

methods. Thus the chapter exposed the pitfalls in the company’s software

development policies with a successful comparison of what was actually done

against what should have been done.

116

Chapter Seven: Conclusions and Future Work

Chapter Seven Conclusions and Future Work

7.0 Summary

The goal of this dissertation was to identify issues in quality of management

processes in the development of Information Systems. The dissertation was

carried out in collaboration with British Telecom (BT) at their South Yorkshire

office in Sheffield. The system chosen for the purpose of this dissertation was

called Training and Development Live, which was developed in-house to track,

and plan training for field engineers with the BT Field Service. A significant aim of

this research was to evaluate quality practices in the development of Training

and Development Live. The evaluation was done by comparing the methodology

used by the company with that prescribed by the ISO 9000-3 framework. Thus

the emphasis was to analyse the process and not the end product. This process

assessment was thought as being useful to the company to get a comprehensive

overview of their development process.

The research was begun by gaining knowledge through the literature review on

quality in information systems projects. Once the researcher had a solid

understanding of the issues involved in maintaining quality in software

processes, questionnaires were prepared and detailed interviews were

conducted with key stakeholders of Training and Development Live.

However, all along the exercise, there were several problems both with the

company’s approach towards software development and with the lack of facilities

to conduct such a research. Though a majority of these have been identified as

117


problems and discussed in detail in the last chapter, there are some issues that

the researcher wishes to highlight again for the benefit of the company.

Firstly, no project management techniques were followed. It was really surprising

that such a big company did not follow basic project management techniques

while developing a system. There were no objectives defined, no clear schedules

or budgets laid out, no project timelines, and no project plan. There was a strong

feeling within the development team that project management techniques were

not always practical.

The company had an extremely lack lustre policy regarding documentation. This

made it very difficult to understand the system. Training and Development Live

had only two documents, a user process document and a power point

presentation on the benefits of the system. The ISO standards place a very high

emphasis on documentation. The company could have disastrous consequences

of not having systematic documentation of their development process. These

consequences have been pointed out in chapter four in detail.

The above premise applies to the company in the case of a methodology as well.

The lack of methodology could result in the project resulting in a mishap or it

could affect the quality of the end product. ISO has all the phases clearly defined.

The lack of a development methodology in the company made it difficult to

evaluate the activities conducted in the company with the corresponding

framework specified in the ISO framework. ISO suggests that a particular activity

needs to be done in a particular phase and the company never had those

activities in place.

As with the documentation, the company had no test policies in place. ISO

requires some form of verification or validation in every phase. Testing was not

considered a separate phase at all. There was no concept of code or document

inspections. Thus the company had no quality assurance strategies in place.

118


7.1 Future Work

The detailed analysis and recommendations provided to the company should be

useful to the company in improving their development process for future IT

projects. The company could take note of the mistakes committed in the

development of this system and can be careful not to repeat them in future.

There seems to be a potential for process improvement not only for future IT

projects, but also for the current project. Now that the company has a

comprehensive overview of the issues concerning the development process of

the Training and Development Live, measures could be adopted to formally

improve the existing practices followed.

The company could conduct a similar exercise (process analysis) with projects

from other departments and monitor the results. Measures can then be taken to

improve processes in other departments as well.

119


Bibliography

Avison D. E. and Fitzgerald G. (1995) - Information Systems Development:

methodologies, techniques and tools.

London: McGraw-Hill.

Bowen D.E. and Schneider B. (1988) - Services Marketing and Management;

Implications for Organizational Behaviour

Research in Organizational Behaviour JAI Press

Merriam Webster Online Dictionary (2004)

http://www.m-w.com

[Accessed: 29th July, 2004]

North J., Blackburn A. B., et al (1998) - The Quality Business – Quality issues

in smaller firms.

Routledge London

The Oxford Dictionary (2004)


The Longman Dictionary of Contemporary English (1998)

Thomson Press (India) Ltd.

Gronroos C. (1984) - A service quality model and its marketing implications

European journal of marketing

Wilkinson A. and Willmott H. (1995) - Making Quality Critical - New

perspectives on organizational change

Routledge London

Hughes B. and Cotterell M. (1999) - Software Project Management, Second

Edition

London: McGraw-Hill

120

http://www.m-w.com/


Lockyer K. and James G. (1996) - Project management and project network

techniques

Pitman publishing

Yeates D. (1991) – Project Management for Information Systems

Pitman publishing

Maylor H. (1996) – Project Management

Pitman publishing

Project Management Methodology (1997) – Project Management Planning:

Quality Planning

Ould M. A. and Unwin C. (1989) – Testing in Software Development

Cambridge University Press

Card D. N. and Glass R. L. (1990) – Measuring Software Design Quality

Prentice Hall

Yeates D. and Cadle J. (1996) – Project Management for Information

Systems

Prentice Hall

Turner R.J. (1999) – The handbook of project based management – Second

edition

McGraw Hill

Webopedia (2004) – The encyclopaedia dedicated to computer technology

http://www.webopedia.com

[Accessed: 17th August, 2004]

Beizer B. (1995) – Black box testing: Techniques for functional testing of

software and systems

John Wiley and Sons

121

http://www.webopedia.com/


Software Testing (2004) – White box testing

http://www.softwaretesting.nildram.co.uk/cont459.htm


University of Calgary (2004) – Practical Software Engineering

http://sern.ucalgary.ca/courses/cpsc/451/F00/testing/testing.html


Bikson (1991) - Relevance versus Rigor in Information Systems Research: An

Issue of Quality

IFIP TC8 WG8.2 Working Conference on Information Systems

Garcia L. and Quek F. (1996) - Qualitative Research in Information Systems:

Time to be Subjective?

IFIP WG8.2 Working Conference on Information Systems & Qualitative

Research

Vanguard University (2004) – Qualitative Research Methods

http://www.vanguard.edu/faculty/dratcliff/index.cfm?doc_id=4254


Kaplan B. and Maxwell J. A. (1994) - Qualitative Research Methods for

evaluating Computer Information Systems

Thousand Oaks

Myers M.D. (1997) - Qualitative Research in Information Systems

http://www.qual.auckland.ac.nz/


North Carolina Wesleyan College (2004) – Qualitative Social Science

Research Methodology.

http://faculty.ncwc.edu/toconnor/308/308lect09.htm

[Accessed: 21st August, 2004]

122


http://sern.ucalgary.ca/courses/cpsc/451/F00/testing/testing.html

http://www.vanguard.edu/faculty/dratcliff/index.cfm?doc_id=4254

http://www.qual.auckland.ac.nz/

http://faculty.ncwc.edu/toconnor/308/308lect09.htm


ISO Management Systems (2004) - The international review of ISO 9000 and

ISO 14000

Central Secretariat of ISO, Volume 4, No. 4

Orlikowski W. J. and Baroudi J. J. (1991) - Studying Information Technology

in Organizations: Research Approaches and Assumptions

Information Systems Journal

British Telecom (2004)

http://www.bt.com

[Accessed: 12th June, 2004]

iSixSigma (2004) - An Introduction to ISO 9000, 9001, 9002, ISO 9000:2000


Vanguard Education (2001) - A Brief History of ISO 9000


Coallier, F. (1994) - How ISO 9001 Fits Into the Software World

IEEE Software, Vol. 11, No. 1. pp 98-100.

Durand, I.G. and Marquardt, D.W. (1993) - Updating the ISO 9000

Quality Standards: Responding to Marketplace Needs

ASQC Quality Progress, Vol. 26, No. 7. pp. 23-30.

Paulk, M.C (1994) - A Comparison of ISO 9001 and the Capability Maturity

Model for Software.


Kehoe R. and Jarvis A. (1995) – A tool for Software Product and Process

Improvement.

Springer – Verlag

123

http://www.bt.com/


International Organization for Standardization (2004) – Introduction

http://www.iso.org/iso/en/aboutiso/introduction/index.html#fourteen


MSDN (2004) – Visual Studio Integration Testing

http://msdn.microsoft.com/library/default.asp?url=/library/en-

us/vsent7/html/vxconintegrationtesting.asp

[Accessed: 21st August, 2004]

Computing Dictionary (2004) http://computing-freedictionary.com/operational%20test%20and%20evaluation


Software Testing (2004) – Acceptance Testing



Schwalbe K. (2000) - Information Technology Project Management

Thomson Learning

Lock D. (1997) – Project Management, sixth edition

Gower paperback

ISO 9001 (2004)

www.isoeasy.org/std_cmpn/glossary.htm


Office of Government Commerce (2004) http://www.ogc.gov.uk/sdtoolkit/reference/deliverylifecycle/quality_mgmt.html#success

[Accessed: 2nd June, 2004]

Tech Republic (2003)

(http://techrepublic.com.com/5100-6315_11-5053978.html

[Accessed: 3rd June, 2004]

124

http://www.iso.org/iso/en/aboutiso/introduction/index.html

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vsent7/html/vxconintegrationtesting.asp

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vsent7/html/vxconintegrationtesting.asp

http://computing-freedictionary.com/operational test and evaluation


http://www.isoeasy.org/std_cmpn/glossary.htm

http://www.ogc.gov.uk/sdtoolkit/reference/deliverylifecycle/quality_mgmt.html

http://techrepublic.com.com/5100-6315_11-5053978.html


125

The Quality Portal (2004)

http://thequalityportal.com/glossary/q.htm

[Accessed: 3rd June, 2004]

http://thequalityportal.com/glossary/q.htm

Date post:	26-Jun-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Rajesh Ramasubramanian MSc in Information...

Documents