Testing is a process used to help identify the correctness, completeness and quality of developed computer software. With that in mind, testing can never completely establish the correctness of computer software.

There are many approaches to software testing, but effective testing of complex products is essentially a process of investigation, not merely a matter of creating and following rote procedure. One definition of testing is "the process of questioning a product in order to evaluate it", where the "questions" are things the tester tries to do with the product, and the product answers with its behavior in reaction to the probing of the tester. Although most of the intellectual processes of testing are nearly identical to that of review or inspection, the word testing is connoted to mean the dynamic analysis of the product—putting the product through its paces.

The quality of the application can and normally does vary widely from system to system but some of the common quality attributes include reliability, stability, portability, maintainability and usability. Refer to the ISO standard ISO 9126 for a more complete list of attributes and criteria.

Testing helps is Verifying and Validating if the Software is working as it is intended to be working. Thins involves using Static and Dynamic methodologies to Test the application.

Because of the fallibility of its human designers and its own abstract, complex nature, software development must be accompanied by quality assurance activities. It is not unusual for developers to spend 40% of the total project time on testing. For life-critical software (e.g. flight control, reactor monitoring), testing can cost 3 to 5 times as much as all other activities combined. The destructive nature of testing requires that the developer discard preconceived notions of the correctness of his/her developed software.

Software Testing Fundamentals

Testing objectives include

1. Testing is a process of executing a program with the intent of finding an error.2. A good test case is one that has a high probability of finding an as yet undiscovered error.3. A successful test is one that uncovers an as yet undiscovered error.

Testing should systematically uncover different classes of errors in a minimum amount of time and with a minimum amount of effort. A secondary benefit of testing is that it demonstrates that the software appears to be working as stated in the specifications. The data collected through testing can also provide an indication of the software's reliability and quality. But, testing cannot show the absence of defect -- it can only show that software defects are present.

Testing Start Process

Testing is sometimes incorrectly thought as an after-the-fact activity; performed after programming is done for a product. Instead, testing should be performed at every development stage of the product. Test data sets must be derived and their correctness and consistency should be monitored throughout the development process.

If we divide the lifecycle of software development into “Requirements Analysis”, “Design”, “Programming/Construction” and “Operation and Maintenance”, then testing should accompany each of the above phases. If testing is isolated as a single phase late in the cycle, errors in the problem statement or design may incur exorbitant costs. Not only must the original error be corrected, but the entire structure built upon it must also be changed. Therefore, testing should not be isolated as an inspection activity. Rather testing should be involved throughout the SDLC in order to bring out a quality product.

The following testing activities should be performed during the phases:

1. Requirements Analysis- Determine correctness - Generate functional test data.

2. Design - Determine correctness and consistency - Generate structural and functional test data.

1. Programming/Construction - Determine correctness and consistency - Generate structural and functional test data - Apply test data - Refine test data

2. Operation and Maintenance - Retest

The following test activities should be performed during this stage:

1.1 Invest in analysis at the beginning of the project - Having a clear, concise and formal statement of the requirements facilitates programming, communication, error analysis and test data generation.

The requirements statement should record the following information and decisions:

a. Program function - What the program must do?

b. The form, format, data types and units for input.

c. The form, format, data types and units for output.

d. How exceptions, errors and deviations are to be handled.

e. For scientific computations, the numerical method or at least the required accuracy of the solution.

f. The hardware/software environment required or assumed (e.g. the machine, the operating system, and the implementation language).

Deciding the above issues is one of the activities related to testing that should be performed during this stage.1.2 Start developing the test set at the requirements analysis phase - Data should be generated that can be used to determine whether the requirements have been met. To do this, the input domain should be partitioned into classes of values that the program will treat in a similar manner and for each class a representative element should be included in the test data.

In addition, following should also be included in the data set:

(1) boundary values

(2) any non-extreme input values that would require special handling.

The output domain should be treated similarly.

Invalid input requires the same analysis as valid input.

1.3 The correctness, consistency and completeness of the requirements should also be analyzed - Consider whether the correct problem is being solved, check for conflicts and inconsistencies among the requirements and consider the possibility of missing cases.

Design

The design document aids in programming, communication, and error analysis and test data generation. The requirements statement and the design document should together give the problem and the organization of the solution i.e. what the program will do and how it will be done.

The design document should contain:

Principal data structures. Functions, algorithms, heuristics or special techniques used for processing. The program organization, how it will be modularized and categorized into

external and internal interfaces. Any additional information.

Here the testing activities should consist of:

- Analysis of design to check its completeness and consistency - the total process should be analyzed to determine that no steps or special cases have been overlooked. Internal interfaces, I/O handling and data structures should specially be checked for inconsistencies.

- Analysis of design to check whether it satisfies the requirements - check whether both requirements and design document contain the same form, format, units used for input and output and also that all functions listed in the requirement document have been included in the design document. Selected test data which is generated during the requirements analysis phase should be manually simulated to determine whether the design will yield the expected values.

- Generation of test data based on the design - The tests generated should cover the structure as well as the internal functions of the design like the data structures, algorithm, functions, heuristics and general program structure etc. Standard extreme and special values should be included and expected output should be recorded in the test data.

- Re-examination and refinement of the test data set generated at the requirements analysis phase.

The first two steps should also be performed by some colleague and not only the designer/developer.

Here the main testing points are:

- Check the code for consistency with design - the areas to check include modular structure, module interfaces, data structures, functions, algorithms and I/O handling.

- Perform the Testing process in an organized and systematic manner with test runs dated, annotated and saved. A plan or schedule can be used as a checklist to help the programmer organize testing efforts. If errors are found and changes made to the program, all tests involving the erroneous segment (including those which resulted in success previously) must be rerun and recorded.

- Asks some colleague for assistance - Some independent party, other than the programmer of the specific part of the code, should analyze the development product at each phase. The programmer should explain the product to the party who will then question the logic and search for errors with a checklist to guide the search. This is needed to locate errors the programmer has overlooked.

- Use available tools - the programmer should be familiar with various compilers and interpreters available on the system for the implementation language being used because they differ in their error analysis and code generation capabilities.

- Apply Stress to the Program - Testing should exercise and stress the program structure, the data structures, the internal functions and the externally visible functions or functionality. Both valid and invalid data should be included in the test set.- Test one at a time - Pieces of code, individual modules and small collections of modules should be exercised separately before they are integrated into the total program, one by one. Errors are easier to isolate when the no. of potential interactions should be kept small. Instrumentation-insertion of some code into the program solely to measure various program characteristics – can be useful here. A tester should perform array bound checks, check loop control variables, determine whether key data values are within permissible ranges, trace program execution, and count the no. of times a group of statements is executed.

- Measure testing coverage/When should testing stop? - If errors are still found every time the program is executed, testing should continue. Because errors tend to cluster, modules appearing particularly error-prone require special scrutiny.

The metrics used to measure testing thoroughness include statement testing (whether each statement in the program has been executed at least once), branch testing (whether each exit from each branch has been executed at least once) and path testing (whether all logical paths, which may involve repeated execution of various segments, have been executed at least once). Statement testing is the coverage metric most frequently used as it is relatively simple to implement.

The amount of testing depends on the cost of an error. Critical programs or functions require more thorough testing than the less significant functions.

Operations and Maintenance

Corrections, modifications and extensions are bound to occur even for small programs and testing is required every time there is a change. Testing during maintenance is termed regression testing. The test set, the test plan, and the test results for the original program should exist.

Modifications must be made to accommodate the program changes, and then all portions of the program affected by the modifications must be re-tested. After regression testing is complete, the program and test documentation must be updated to reflect the changes.

Testing Stop Process

This can be difficult to determine. Many modern software applications are so complex, and run in such as interdependent environment, that complete testing can never be done. "When to stop testing" is one of the most difficult questions to a test engineer. Common factors in deciding when to stop are:

Deadlines ( release deadlines,testing deadlines.) Test cases completed with certain percentages passed Test budget depleted Coverage of code/functionality/requirements reaches a specified point The rate at which Bugs can be found is too small Beta or Alpha Testing period ends The risk in the project is under acceptable limit.

Practically, we feel that the decision of stopping testing is based on the level of the risk acceptable to the management. As testing is a never ending process we can never assume that 100 % testing has been done, we can only minimize the risk of shipping the product to client with X testing done. The risk can be measured by Risk analysis but for small duration / low budget / low resources project, risk can be deduced by simply: -

Measuring Test Coverage. Number of test cycles. Number of high priority bugs.

The Software Assurance Technology Center (SATC) in the Systems Reliability and Safety Office at Goddard Space Flight Center (GSFC) is investigating the use of software error data as an indicator of testing status. Items of interest for determining the status of testing include projections of the number of errors remaining in the software and the expected amount of time to find some percentage of the remaining errors.

To project the number of errors remaining in software, one needs an estimate of the total number of errors in the software at the start of testing and a count of the errors found and corrected throughout testing. There are a number of models that reasonably fit the rate at which errors are found in software, the most commonly used is referred to in this paper as the Musa model. This model is not easily applicable at GSFC, however, due to the availability and the quality of the error data.

At GSFC, useful error data is not easy to obtain for projects not in the Software Engineering Laboratory. Of the projects studied by the SATC, only a few had an organized accounting scheme for tracking errors, but they often did not have a consistent format for recording errors. Some projects record errors that were found but did not record any information about resources applied to testing. The error data frequently contained the date of entry of the error data rather than the actual date of error discovery.

In order to use traditional models such as the Musa model for estimating the cumulative number of errors, one needs fairly precise data on the time of discovery of errors and the level of resources applied to testing. Real world software projects are generally not very accommodating when it comes to either accuracy or completeness of error data. The models developed by the SATC to perform trending and prediction on error data attempt to compensate for these shortcomings in the quantity and availability of project data.

In order to compensate for the quality of the error data, the SATC developed a software error trending models using two techniques, each based on the basic Musa model, but with the constant in the exponential term replaced by a function of time that describes the 'intensity' of the testing effort. The shape and the parameters for this function can be estimated using measures such as CPU time or staff hours devoted to testing. The first technique involves fitting cumulative error data to the modified Musa model using a least squares fit that is based on gradient methods. This technique requires data on errors found and the number of staff hours devoted to testing each week of the testing activity. The second technique uses a Kalman filter to estimate both the total number of errors in the software and the level of testing being performed. This technique requires error data and initial estimates of the total number of errors and the initial amount of effort applied to testing.

The SATC has currently examined and modeled error data from a limited number of projects. Generally, only the date on which an error was entered into the error tracking system was available, not the date of discovery of the error. No useful data was available on human or computer resources expended for testing.

What is needed for the most accurate model is the total time expended for testing, even if the times are approximate. Using the sum of reported times to find/fix individual errors did not produce any reasonable correlation with the resource function required. Some indirect attempts to estimate resource usage, however, led to some very good fits.On one project errors were reported along with the name of the person that found the error. Resource usage for testing was estimated as follows: A person was estimated to be working on the testing effort over a period beginning with the first error that they reported and ending with the last error that they reported. The percentage of time that each person worked during that period was assumed to be an unknown constant that did not differ from person to person. Using this technique led to a resource curve that closely resembled the Rayleigh curve (Figure 1).

Figure 1: Test Resource Levels for Project A

On most of the projects, there was good conformity between the trend model and the reported error data. More importantly, estimates of the total number of errors and the error discovery parameter, made fairly early in the testing activity, seemed to provide reliable indicators of the total number of errors actually found and the time it took to find future errors.

Figures 2 shows the relationship between reported errors and the SATC trend model for one project. The graph represents data available at the conclusion of the project. This close fit was also found on other projects when sufficient data was available.

Figure 2: Cumulative Software Errors for Project A

On another project, different estimates of the total number of errors were obtained when estimates were made over different testing time intervals. That is, there was inconsistent

agreement between the trend model and the error data over different time intervals. Through subsequent discussion with the project manager it was learned that the rate of error reporting by the project went from approximately 100% during integration testing to 40% during acceptance testing. Furthermore, there was a significant amount of code rework, and testing of the software involved a sequential strategy of completely testing a single functional area before proceeding to test the next functional area of the code.

Thus, the instability of the estimates of the total errors was a useful indicator of the fact that there was a significant change in either the project's testing and reporting process. Figure 3 shows the results for this project. Note the change in slope of the reported number of errors occurring around 150 days. The data curve flattens at the right end of the curve due to a pause in testing, rather than a lack of error detection. This project is still undergoing testing.

Figure 3: Cumulative S/W Errors for Project B - Flight S/W

If error data is broken into the distinct testing phases of the life cycle (e.g., unit, system, integration), the projected error curve using the SATC model closely fits the rate at which errors are found in each phase.

Some points need to be clarified about the SATC error trend model. The formulation of the SATC equation is the direct result of assuming that at any instant of time, the rate of discovery of errors is proportional to the number of errors remaining in the software and to the resources applied to finding errors. Additional conditions needed in order for the SATC trending model to be a valid are:

1. The code being tested is not being substantially altered during the testing process, especially through the addition or rework of large amounts of code.

2. All errors found are reported.3. All of the software is tested, and testing of the software is uniform throughout the

time of the testing activity.

Condition 1 is present to ensure that the total number of errors is a relatively stable number throughout the testing activity. Conditions 2 and 3 are present to ensure that the estimate of the total number of errors is in fact an estimate of the total errors present in

the software at the start of testing - no new errors are introduced during testing. If testing is not "uniform" then the rate of error discovery will not necessarily be proportional to the number of errors remaining in the software and so the equation will not be an appropriate model for errors found. No attempt will be made here to make precise the meaning of the word "uniform".

The SATC developed this model rather than using the standard Musa model because it seems less sensitive to data inaccuracy and provides for non-constant testing resource levels. An additional benefit from this work is the application of the Rayleigh curve for estimating resource usage. Future work by the SATC will continue to seek a practical balance between available trend analysis theory and the constraints imposed by the limited accuracy and availability of data from real-world projects.

Testing StrategyTest Strategy

How we plan to cover the product so as to develop an adequate assessment of quality.

A good test strategy is:

Specific Practical Justified

The purpose of a test strategy is to clarify the major tasks and challenges of the test project.

Test Approach and Test Architecture are other terms commonly used to describe what I’m calling test strategy.

Example of a poorly stated (and probably poorly conceived) test strategy:

"We will use black box testing, cause-effect graphing, boundary testing, and white box testing to test this product against its specification."

Contents: Creating Test Strategy Defining Test Strategy Requirements in Test Strategy Key points to remember

Creating a Test Strategy

The test strategy is a formal description of how a software product will be tested. A test strategy is developed for all levels of testing, as required. The test team analyzes the requirements, writes the test strategy and reviews the plan with the project team. The test plan may include test cases, conditions, the test environment, a list of related tasks, pass/fail criteria and risk assessment.

Inputs for this process: A description of the required hardware and software components, including test

tools. This information comes from the test environment, including test tool data. A description of roles and responsibilities of the resources required for the test

and schedule constraints. This information comes from man-hours and schedules.

Testing methodology. This is based on known standards. Functional and technical requirements of the application. This information comes

from requirements, change request, technical and functional design documents. Requirements that the system can not provide, e.g. system limitations.

Outputs for this process: An approved and signed off test strategy document, test plan, including test cases. Testing issues requiring resolution. Usually this requires additional negotiation at

the project management level.

Defining a Test Strategy

A solid testing strategy provides the framework necessary to implement your testing methodology. A separate strategy should be developed for each system being developed taking into account the development methodology being used and the specific application architecture.

The heart of any testing strategy is the master testing strategy document. It aggregates all the information from the requirements, system design and acceptance criteria into a detailed plan for testing. A detailed master strategy should cover the following:

Project Scope

Restate the business objective of the application and define the scope of the testing. The statement should be a list of activities that will be in scope or out of scope. A sample list would include:

* List of software to be tested * Software configurations to be tested * Documentation to be validated * Hardware to be tested

Test Objectives

The system under test should be measured by its compliance to the requirements and the user acceptance criteria. Each requirement and acceptance criteria must be mapped to specific test plans that validate and measure the expected results for each test being performed. The objectives should be listed in order of importance and weighted by Risk. Features and Functions to be Tested

Every feature and function must be listed for test inclusion or exclusion, along with a description of the exceptions. Some features may not be testable due to a lack of hardware or lack of control etc. The list should be grouped by functional area to add clarity. The following is a basic list of functional areas:

* Backup and recovery * Workflow * Interface design * Installation * Procedures (users, operational, installation) * Requirements and design * Messaging * Notifications * Error handling * System exceptions and third-party application faults

Testing Approach

The approach provides the detail necessary to describe the levels and types of testing. The basic V-Model shows what types of testing are needed to validate the system. More specific test types include functionality, performance testing, backup and recovery, security testing, environmental testing, conversion testing, usability testing, installation and regression testing. The specific testing methodology should be described and the entry/exit criteria for each phase noted in a matrix by phase. A project plan that list the resources and schedule for each testing cycle should also be created that maps the specific testing task to the overall development project plan.

Testing Process and Procedures

The order of test execution and the steps necessary to perform each type of test should be described in sufficient detail to provide clear input into the creation of test plans and test cases. Procedures should include how test data is created, managed and

loaded. Test cycles should be planned and scheduled based on system availability and deliverable dates from development. All application and environmental dependencies should be identified along with the procedures necessary to gain access to all the dependent systems.

Test Compliance

Every level of testing must have a defined set of entry/exit criteria which is used to validate that all prerequisites for a valid test have been met. All mainstream software testing methodologies provide an extensive list of entry/exit criteria and checklist. In addition to the standard list, additional items should be added based on specific testing needs. Some common additions are, environmental availability, data availability, and validated code which is ready to be tested.

Each level of testing should define specific pass/fail acceptance criteria, to ensure to ensure that all quality gates have been validated and that the test plan focuses on developing test that validate the specific criteria defined by the user acceptance plan.

Testing Tools

All testing tools should be identified and their use, ownership and dependencies defined. The tools category includes manual tools, such as templates in spreadsheets and documents as well as automated tools for test management, defect tracking, regression testing and performance/load testing. Any specific skill sets should be identified and compared against the existing skills identified for the project to highlight any training needs.

Defect Resolution

A plan to address the resolution of failed tests needs to be created that lists the escalation procedures to seek correction and retest of the failed tests along with a risk mitigation plan for high-risk test. Defect tracking should include basic metrics for compliance based on number and type of defect found.

Roles and Responsibilities

A matrix listing the roles and responsibilities of everyone involved in the testing activities, along with the anticipated amount of their time allocated to the project, must be prepared.

Process Improvement

The entire testing process should be focused on process improvement. The strategy should list ways to monitor progress and provide constant feedback. This feedback can serve to enhance the process, deliverables and metrics used in the testing. Root cause analysis should be performed on all reported defects to help isolate the true

nature of the problem and prevent unnecessary repeat offenses.

Deliverables

All deliverables should be defined and their location specified. Common deliverables are test plans, test cases, test scripts, test matrix and a defect log.

Schedule

All testing activities should be combined into one master testing schedule. The schedule should include an estimate of time for each task and the dependences for each. Testing resources should be assigned to each task and quality gates should be listed to insure oversight of the entire process.

Environmental Needs

All the requirements of the testing environment need to be listed. Common ones include a description of the environment's use, management, hardware and software, specific tools needed, data loading and security requirements.

Resource Management

The skills of all personnel involved in the testing effort need to be assessed and the gaps noted so that a comprehensive training program can be designed. Specialty skills that will not be filled with in-house staff will require job descriptions and budgeting.

Risk and Contingencies

Planning for risk in advance and ways to mitigate it are essential for a robust strategy. A risk assessment that is prioritized by severity of risk and covers technology, resource, schedule and environmental issues should feed a detailed plan to mitigate each red flag.

Approvals and Workflow

All items on the critical path must go through an approval cycle. The procedures for approval and escalation must be well defined and assigned to resources prior to the start of the testing.

The above covers the main sections of a well-drafted and documented testing strategy. The more detail that you include in the strategy document, the less ambiguity and chance for deviation there will be throughout the project

The completion of the strategy signals the beginning of the test planning phase. For each type of testing identified in the master test strategy there should be a test plan identifying the components to be tested, the location of the test data, the test

environment needs, the test procedures, resources required, and the tests schedule. For each plan a series of test conditions should be identified so that test cases with expected results can be generated for later execution.

What to include in the Testing Strategy

During the analysis phase, you gather and validate the business requirements for the solution. It makes sense that the Testing Strategy is completed during this phase as well. In a sense, you are defining the overall testing requirements.

The purpose of the Testing Strategy is to define the overall context for the entire testing process. The process is different depending on the specific characteristics of your solution. In many respects, this is the most important part of the testing process, since all future testing decisions will be made within the context of the strategy. Here are the basic parts of the testing strategy:

  Project Overview: You can copy this from the Project Definition.

Business Risks: These are high-level risks of the project that will affect the overall testing strategy. For instance, the risk of doing business on the Internet may drive the need for rigorous system tests of firewalls, technical architecture, and security. The risks can be classified as high, medium, and low, depending on the nature and impact of the problem. For each high and medium risk, identify what elements in the overall testing approach will help ensure that the potential problem does not occur.

o Testing Milestones: This section gives the reader a preliminary overview of the testing timelines. Obviously, since this document is created in the analysis phase, these dates are subject to later revision.

o Testing Approach: This describes the testing process at a high level, including how you will conduct unit testing, integration testing, system testing, and acceptance testing. (If your project is large enough, each of these might be its own section.) This is where you make fundamental decisions regarding the type of testing that makes sense for your project. For instance, if you are implementing a packaged solution, the approach may start in system testing, with the vendor providing close support. If you are doing iterative development cycles, the testing approach will reflect this overall development life cycle. For system testing, define the major testing events, such as stress testing, security testing, disaster recovery testing, usability testing, and response time testing.

o Testing Environment: Think through the technologies and facilities needed for the testing process. If the overall testing environment needs are understood up front, it will be easier to break out the specific activities required to put the environment in place. In addition, you may need to plan for and acquire some parts of the environment well in advance.

Depending on your project, there may be other high-level sections to include, such as testing objectives, testing assumptions, testing organization, and testing tools, along with effort and cost estimates.

Key points to remember

The developer’s role here is not trivial. As you contribute to this strategy document, keep these three points in mind:

If you are working on a large project, you need to formulate an overall Testing Strategy during the analysis phase. The Testing Strategy defines the overall approach to testing and describes how the testing process will ensure that the solution has the appropriate level of quality and reliability.

The Testing Strategy provides the overall guidelines from which all future testing decisions are made. A well-crafted Testing Strategy allows the rest of the testing process to be defined more effectively.

The Testing Strategy needs to be understood and approved by the sponsor. If the strategy is accepted, there is a much better likelihood that the final solution will meet the customer’s expectations.

Risk Analysis:

A risk is a potential for loss or damage to an Organization from materialized threats. Risk Analysis attempts to identify all the risks and then quantify the severity of the risks.A threat as we have seen is a possible damaging event. If it occurs, it exploits vulnerability in the security of a computer based system.

Contents:

Risk Identification Schedule Risk Risk Assessment Risk Assessment in Public Health

Risk Assessment in Auditing Criticisms of quantitative risk assessment Risk Management Create a risk management plan Implementation Review and evaluation of the plan Limitations Areas of Risk Management Risk management and business continuity

Risk Identification

1. Software Risks:

Knowledge of the most common risks associated with Software development, and the platform you are working on.

2. Business Risks: Most common risks associated with the business using the Software

3. Testing Risks: Knowledge of the most common risks associated with Software Testing for the platform you are working on, tools being used, and test methods being applied.

4. Premature Release Risk: Ability to determine the risk associated with releasing unsatisfactory or untested Software Prodicts.

5. Risk Methods: Strategies and approaches for identifying risks or problems associated with implementing and operating information technology, products and process; assessing their likelihood, and initiating strategies to test those risks.

What is Schedule Risk?

In your project, you have to estimate how long it takes to complete a certain task.

You estimate that it usually takes 15 days to complete. If things go well it may take 12 days but if things go badly it may take 20 days.

In your project plan you enter 15 days against the task.

The other information, the best case estimate of 12 days and the worst case estimate of 20 days, is not entered into the project at all.

If this seems familiar, then you already go through the process of identifying uncertainty or risk. By entering only the most likely duration a great deal of additional information is lost. But with Schedule Risk this extra information is used to help produce a much more realistic project.

And you are not just limited to durations. Uncertainty in resources and costs can also be modeled in your project to produce an even greater depth and accuracy to the information available to you.

Who should use Schedule Risk Analysis?

The simple answer is - anyone who manages a project! If you are running projects that are time and/or cost critical, risk analysis will help you manage your projects more effectively and help reduce the chances of your project being late and over budget.

Pertmaster is used by project planners of all levels, from those just entering into the Schedule Risk arena to the world's leading risk experts.

How easy is it to use?

Very easy. You do not need to be an expert in risk and statistics to be able to use schedule risk. With normal project planning, the level of detail and complexity that you build into the project is up to you and your requirements. This is the same with Schedule Risk. Very little extra information is required as a minimum but you have the ability to provide a great deal of very specific additional information if you require it.

Pertmaster is acclaimed as being very easy to use. By simply following the tutorials and examples you will be able to incorporate risk into your project with ease. Pertmaster includes a Quick Risk (link) facility that lets you add risk to your project in seconds.

Risk Assessment

Risk assessment may be the most important step in the risk management process, and may also be the most difficult and prone to error. Once risks have been identified and assessed, the steps to properly deal with them are much more programmatical.

Part of the difficulty of risk management is that measurement of both of the quantities in which risk assessment is concerned can be very difficult itself. Uncertainty in the measurement is often large in both cases. Also, risk management would be simpler if a

single metric could embody all of the information in the measurement. However, since two quantities are being measured, this is not possible. A risk with a large potential loss and a low probability of occurring must be treated differently than one with a low potential loss but a high likelihood of occurring. In theory both are of nearly equal priority in dealing with first, but in practice it can be very difficult to manage when faced with the scarcity of resources, especially time, in which to conduct the risk management process. Expressed mathematically,

Financial decisions, such as insurance, often express loss terms in dollars. When risk assessment is used for public health or environmental decisions, there are differences of opinions as to whether the loss can be quantified in a common metric such as dollar values or some numerical measure of quality of life. Often for public health and environmental decisions, the loss term is simply a verbal description of the outcome, such as increased cancer incidence or incidence of birth defects. In that case, the "risk" is expressed as:

If the risk estimate takes into account information on the number of individuals exposed, it is termed a "population risk" and is in units of expected increased cases per a time period. If the risk estimate does not take into account the number of individuals exposed, it is termed an "individual risk" and is in units of incidence rate per a time period. Population risks are of more use for cost/benefit analysis; individual risks are of more use for evaluating whether risks to individuals are "acceptable".

Risk assessment in public health

In the context of public health, risk assessment is the process of quantifying the probability of a harmful effect to individuals or populations from certain human activities.

In most countries, the use of specific chemicals, or the operations of specific facilities (e.g. power plants, manufacturing plants) is not allowed unless it can be shown that they do not increase the risk of death or illness above a specific threshold. For example, the American Food and Drug Administration (FDA) regulates food safety through risk assessment. The FDA required in 1973 that cancer-causing compounds must not be present in meat at concentrations that would cause a cancer risk greater than 1 in a million lifetimes.

How the risk is determined

In the estimation of the risks, three or more steps are involved, requiring the inputs of different disciplines.

The first step, Hazard Identification, aims to determine the qualitative nature of the potential adverse consequences of the contaminant (chemical, radiation, noise, etc.) and the strength of the evidence it can have that effect. This is done, for chemical hazards, by drawing from the results of the sciences of toxicology and epidemiology. For other kinds of hazard, engineering or other disciplines are involved.

The second step for chemical risk assessment, Dose-Response Analysis, is determining the relationship between dose and the probability or the incidence of effect (dose-response assessment). The complexity of this step in many contexts derives mainly from the need to extrapolate results from experimental animals (e.g. mouse, rat) to humans, and/or from high to lower doses. In addition, the differences between individuals due to genetics or other factors mean that the hazard may be higher for particular groups, called susceptible populations. An alternative to dose-response estimation is to determine an effect unlikely to yield observable effects. In developing such a dose, to account for the largely unknown effects of animal to human extrapolations, increased variability in humans, or missing data, a prudent approach is often adopted by including safety factors in the estimate of the "safe" dose, typically a factor of 10 for each unknown step.

The third step, Exposure Quantification, aims to determine the amount of a contaminant (dose) that individuals and populations will receive. This is done by examining the results of the discipline of exposure assessment. As different location, lifestyles and other factors likely influence the amount of contaminant that is received, a range or distribution of possible values is generated in this step. Particular care is taken to determine the exposure of the susceptible population(s).

Finally, the results of the three steps above are then combined to produce an estimate of risk. Because of the different susceptibilities and exposures, this risk will vary within a population. The decisions based on the application of risk assessment are sometimes based on a standard of protecting those most at risk. This problem raises the question of how small a segment of a population must be protected. What if a risk is very low for everyone but 0.1% of the population? A difference exists whether this 0.1% is represented by

all infants younger than X days or recreational users of a particular product.

If the risk is higher for a particular sub-population because of abnormal exposure rather than susceptibility, there is a potential to consider strategies to further reduce the exposure of that subgroup. If an identifiable sub-population is more susceptible due to inherent genetic or other factors, there is a policy choice whether to set policies for protecting the general population that are protective of such groups (as is currently done

for children when data exists, or is done under the Clean Air Act for populations such as asthmatics) or whether if the group is too small, or the costs to high. Sometimes, a suitable position is to at least limit the risk of the more susceptible to some risk level above which it seems too inequitable to leave them out of the risk

Acceptable Risk Increase

The idea of not increasing lifetime risk by more than one in a million has become common place in public health discourse and policy. How consensus settled on this particular figure is unclear. In some respects, this figure has the characteristics of a mythical number. In another sense, the figure provides a numerical basis for what to consider a negligible increase in risk.

In part, the one in a million benchmark arose early in public health risk assessment history when risk assessment was a tempering analysis to existing statutory language such as the Delaney Clause prohibition on any use of introduced carcinogens or where environmental statutes were using a "best technology" decision rule. Some current environmental decision making allows some discretion to deem individual risks potentially "acceptable" if below one in ten thousand increased lifetime risk. Low risk criteria such as these do provide some protection for the case that individuals may be exposed to multiple chemicals (whether pollutants or food additives, or other chemicals). But both of these benchmarks are clearly small relative to the typical one in four lifetime risk of death by cancer (due to all causes combined) in developed countries.

Individuals may be tempted to advocate the adoption of a zero-risk policy. After all the 1 in a million policy would still cause the death of hundreds or thousands, of people in a large enough population. In practice however, a true zero-risk is possible only with the suppression of the risk-causing activity. More stringent requirements, or even the 1 in a million one, may not be technologically feasible at a given time, or so expensive as to render the risk-causing activity unsustainable.

In the interest of public health, the risks vs. benefits of the possible alternatives must be carefully considered. For example, it might well be that the emissions from hospital incinerators result in a certain number of deaths per year. However, this risk must be balanced against the available alternatives. In some unusual cases, there are significant public health risks, as well as economic costs, associated with all options. For example, there are risks associated with no incineration (with the potential risk for spread of infectious diseases) or even no hospitals. But, often further investigation identifies further options, such as separating noninfectious from infectious wastes, or air pollution controls on a medical incinerator, that provide a broad range of options of acceptable risk - though with varying practical implications and varying economic costs. Intelligent thought about a reasonably full set of options is essential. Thus, it is not unusual for there to be an iterative process between analysis, consideration of options, and then further analysis.

Risk assessment in auditing

In auditing, risk assessment is a very crucial stage before accepting an audit engagement. According to ISA315 Understanding the Entity and its Environment and Assessing the Risks of Material Misstatement, "the auditor should perform risk assessment procedures to obtain an understanding of the entity and its environment, including its internal control."

The main purpose of risk assessment procedures is to help the auditor understand the audit client. Aspects like client's business nature, management structure and internal control system are good examples. The procedures will provide audit evidence relating to the auditor’s risk assessment of a material misstatement in the client’s financial statements. Then, auditor obtains initial evidence regarding the classes of transactions at the client and the operating effectiveness of the client’s internal controls.

In auditing, audit risk includes inherent risk, control risk and detection risk.

Criticisms of quantitative risk assessment

Barry Commoner and other critics have expressed concerns that risk assessment tends to be overly quantitative and reductive. For example, they argue that risk assessments ignore qualitative differences among risks.

Some charge that assessments may drop out important non-quantifiable or inaccessible information, such as variations among the classes of people exposed to hazards. O'Brien further claims that quantitative approaches divert attention from precautionary or preventative measures.

Risk Management

Risk management is a structured approach to managing uncertainty through, risk assessment, developing strategies to manage it, and mitigation of risk using managerial resources.

The strategies include transferring the risk to another party, avoiding the risk, reducing the negative effect of the risk, and accepting some or all of the consequences of a particular risk.

Some traditional risk managements are focused on risks stemming from physical or legal causes (e.g. natural disasters or fires, accidents, death and lawsuits). Financial risk management, on the other hand, focuses on risks that can be managed using traded financial instruments.

Objective of risk management is to reduce different risks related to a preselected domain to the level accepted by society. It may refer to numerous types of threats caused by environment, technology, humans, organizations and politics. On the other hand it involves all means available for humans, or in particular, for a risk management entity (person, staff, organization).

Establishing the context involves

1. Identification of risk in a selected domain of interest2. Planning the remainder of the process.3. Mapping out4. Defining a framework for the activity and an agenda for identification.5. Developing an analysis of risks involved in the process.6. Mitigation of risks using available technological, human and organizational

resources.

Identification

After establishing the context, the next step in the process of managing risk is to identify potential risks. Risks are about events that, when triggered, cause problems. Hence, risk identification can start with the source of problems, or with the problem itself.

1. Source analysis:Risk sources may be internal or external to the system that is the target of risk management. Examples of risk sources are: stakeholders of a project, employees of a company or the weather over an airport.

2. Problem analysis:Risks are related to identified threats. For example: the threat of losing money, the threat of abuse of privacy information or the threat of accidents and casualties. The threats may exist with various entities, most important with shareholders, customers and legislative bodies such as the government.

When either source or problem is known, the events that a source may trigger or the events that can lead to a problem can be investigated. For example: stakeholders withdrawing during a project may endanger funding of the project; privacy information may be stolen by employees even within a closed network; lightning striking a Boeing 747 during takeoff may make all people onboard immediate casualties.

The chosen method of identifying risks may depend on culture, industry practice and compliance. The identification methods are formed by templates or the development of templates for identifying source, problem or event. Common risk identification methods are:

1. Objectives-based risk identification :Organizations and project teams have objectives. Any event that may endanger achieving an objective partly or completely is identified as risk. Objective-based risk identification is at the basis of COSO's Enterprise Risk Management - Integrated Framework

2. Scenario-based risk identification:In scenario analysis different scenarios are created. The scenarios may be the alternative ways to achieve an objective, or an analysis of the interaction of forces in, for example, a market or battle. Any event that triggers an undesired scenario alternative is identified as risk - see Futures Studies for methodology used by Futurists.

3. Taxonomy-based risk identification:The taxonomy in taxonomy-based risk identification is a breakdown of possible risk sources. Based on the taxonomy and knowledge of best practices, a questionnaire is compiled. The answers to the questions reveal risks. Taxonomy-based risk identification in software industry can be found in CMU/SEI-93-TR-6.

4. Common-risk Checking:In several industries lists with known risks are available. Each risk in the list can be checked for application to a particular situation. An example of known risks in the software industry is the Common Vulnerability and Exposures.

5. Risk Charting:This method combines the above approaches by listing Resources at risk, Threats to those resources Modifying Factors which may increase or reduce the risk and Consequences it is wished to avoid. Creating a matrix under these headings enables a variety of approaches. One can begin with resources and consider the threats they are exposed to and the consequences of each. Alternatively one can start with the threats and examine which resources they would affect, or one can begin with the consequences and determine which combination of threats and resources would be involved to bring them about.

Assessment

Once risks have been identified, they must then be assessed as to their potential severity of loss and to the probability of occurrence. These quantities can be either simple to measure, in the case of the value of a lost building, or impossible to know for sure in the case of the probability of an unlikely event occurring. Therefore, in the assessment process it is critical to make the best educated guesses possible in order to properly prioritize the implementation of the risk management plan.

The fundamental difficulty in risk assessment is determining the rate of occurrence since statistical information is not available on all kinds of past incidents. Furthermore, evaluating the severity of the consequences (impact) is often quite difficult for immaterial assets. Asset valuation is another question that needs to be addressed. Thus, best educated opinions and available statistics are the primary sources of information. Nevertheless, risk assessment should produce such information for the management of the organization that the primary risks are easy to understand and that the risk management decisions may be prioritized. Thus, there have been several theories and attempts to quantify risks. Numerous different risk formulae exist, but perhaps the most widely accepted formula for risk quantification is:

Rate of occurrence multiplied by the impact of the event equals risk

Later research has shown that the financial benefits of risk management are less dependent on the formula used but are more dependent on the frequency and how risk assessment is performed.

In business it is imperative to be able to present the findings of risk assessments in financial terms. Robert Courtney Jr. (IBM, 1970) proposed a formula for presenting risks in financial terms. The Courtney formula was accepted as the official risk analysis method for the US governmental agencies. The formula proposes calculation of ALE (annualised loss expectancy) and compares the expected loss value to the security control implementation costs (cost-benefit analysis).

Potential risk treatments

Once risks have been identified and assessed, all techniques to manage the risk fall into one or more of these four major categories: (Dorfman, 1997)

Avoidance (aka elimination) Reduction (aka mitigation) Retention Transfer (aka buying insurance)

Ideal use of these strategies may not be possible. Some of them may involve trade-offs that are not acceptable to the organization or person making the risk management decisions. Another source, from the US Department of Defense, Defense Acquisition University, calls these ACAT, for Avoid, Control, Accept, or Transfer. The ACAT acronym is reminiscent of the term ACAT (for Acquisition Category) used in US Defense industry procurements.

Risk Avoidance

Once risks have been identified and assessed, all techniques to manage the risk fall into one or more of these four major categories: (Dorfman, 1997)

Avoidance (aka elimination) Reduction (aka mitigation) Retention Transfer (aka buying insurance)

Includes not performing an activity that could carry risk. An example would be not buying a property or business in order to not take on the liability that comes with it. Another would be not flying in order to not take the risk that the airplane were to be hijacked. Avoidance may seem the answer to all risks, but avoiding risks also means losing out on the potential gain that accepting (retaining) the risk may have allowed. Not entering a business to avoid the risk of loss also avoids the possibility of earning profits.

Risk Reduction

Involves methods that reduce the severity of the loss or the risk of the loss from occurring. Examples include sprinklers designed to put out a fire to reduce the risk of loss by fire. This method may cause a greater loss by water damage and therefore may not be suitable. Halon fire suppression systems may mitigate that risk, but the cost may be prohibitive as a strategy.

Modern software development methodologies reduce risk by developing and delivering software incrementally. Early methodologies suffered from the fact that they only delivered software in the final phase of development; any problems encountered in earlier phases meant costly rework and often jeopardized the whole project. By developing in iterations, software projects can limit effort wasted to a single iteration.

Outsourcing could be an example of risk reduction if the outsourcer can demonstrate higher capability at managing or reducing risks. In this case companies outsource only some of their departmental needs. For example, a company may outsource only its software development, the manufacturing of hard goods, or customer support needs to another company, while handling the business management itself. This way, the company can concentrate more on business development without having to worry as much about the manufacturing process, managing the development team, or finding a physical location for a call center.

Risk Retention

Involves accepting the loss when it occurs. True self insurance falls in this category. Risk retention is a viable strategy for small risks where the cost of insuring against the risk would be greater over time than the total losses sustained. All risks that are not avoided or transferred are retained by default.

This includes risks that are so large or catastrophic that they either cannot be insured against or the premiums would be infeasible. War is an example since most property and risks are not insured against war, so the loss attributed by war is retained by the insured.

Also any amounts of potential loss (risk) over the amount insured is retained risk. This may also be acceptable if the chance of a very large loss is small or if the cost to insure for greater coverage amounts is so great it would hinder the goals of the organization too much.

Risk Transfer

Means causing another party to accept the risk, typically by contract or by hedging. Insurance is one type of risk transfer that uses contracts. Other times it may involve contract language that transfers a risk to another party without the payment of an insurance premium. Liability among construction or other contractors is very often transferred this way. On the other hand, taking offsetting positions in derivatives is typically how firms use hedging to financially manage risk.

Some ways of managing risk fall into multiple categories. Risk retention pools are technically retaining the risk for the group, but spreading it over the whole group involves transfer among individual members of the group. This is different from traditional insurance, in that no premium is exchanged between members of the group up front, but instead losses are assessed to all members of the group.

Create a risk management plan

Select appropriate controls or countermeasures to measure each risk. Risk mitigation needs to be approved by the appropriate level of management. For example, a risk concerning the image of the organization should have top management decision behind it whereas IT management would have the authority to decide on computer virus risks.

The risk management plan should propose applicable and effective security controls for managing the risks. For example, an observed high risk of computer viruses could be mitigated by acquiring and implementing antivirus software. A good risk management plan should contain a schedule for control implementation and responsible persons for those actions.

According to ISO/IEC 27001, the stage immediately after completion of the Risk Assessment phase consists of preparing a Risk Treatment Plan, which should document the decisions about how each of the identified risks should be handled. Mitigation of risks often means selection of Security Controls, which should be documented in a Statement of Applicability, which identifies which particular control objectives and controls from the standard have been selected, and why.

Implementation

Follow all of the planned methods for mitigating the effect of the risks.

Purchase insurance policies for the risks that have been decided to be transferred to an insurer,avoid all risks that can be avoided without sacrificing the entity's goals, reduce others, and retain the rest.

Review and evaluation of the plan

Initial risk management plans will never be perfect. Practice, experience, and actual loss results will necessitate changes in the plan and contribute information to allow possible different decisions to be made in dealing with the risks being faced.

Risk analysis results and management plans should be updated periodically. There are two primary reasons for this:

to evaluate whether the previously selected security controls are still applicable and effective, and

to evaluate the possible risk level changes in the business environment. For example, information risks are a good example of rapidly changing business environment.

Limitations

If risks are improperly assessed and prioritized, time can be wasted in dealing with risk of losses that are not likely to occur. Spending too much time assessing and managing unlikely risks can divert resources that could be used more profitably. Unlikely events do occur but if the risk is unlikely enough to occur it may be better to simply retain the risk and deal with the result if the loss does in fact occur.

Prioritizing too highly the risk management processes could keep an organization from ever completing a project or even getting started. This is especially true if other work is suspended until the risk management process is considered complete.

It is also important to keep in mind the distinction between risk and uncertainty. Risk can be measured by impacts x probability.

Areas of risk management

As applied to corporate finance, risk management is the technique for measuring, monitoring and controlling the financial or operational risk on a firm's balance sheet.

The Basel II framework breaks risks into market risk (price risk), credit risk and operational risk and also specifies methods for calculating capital requirements for each of these components.

Enterprise risk management

In enterprise risk management, a risk is defined as a possible event or circumstance that can have negative influences on the Enterprise in question. Its impact can be on the very existence, the resources (human and capital), the products and services, or the customers of the enterprise, as well as external impacts on society, markets, or the environment. In a financial institution, enterprise risk management is normally thought of as the combination of credit risk, interest rate risk or asset liability management, market risk, and operational risk.

In the more general case, every probable risk can have a preformulated plan to deal with its possible consequences (to ensure contingency if the risk becomes a liability).

From the information above and the average cost per employee over time, or cost accrual ratio, a project manager can estimate

the cost associated with the risk if it arises, estimated by multiplying employee costs per unit time by the estimated time lost (cost impact, C where C = cost accrual ratio * S).

the probable increase in time associated with a risk (schedule variance due to risk, Rs where Rs = P * S):

o Sorting on this value puts the highest risks to the schedule first. This is intended to cause the greatest risks to the project to be attempted first so that risk is minimized as quickly as possible.

o This is slightly misleading as schedule variances with a large P and small S and vice versa are not equivalent. (The risk of the RMS Titanic sinking vs. the passengers' meals being served at slightly the wrong time).

the probable increase in cost associated with a risk (cost variance due to risk, Rc where Rc = P*C = P*CAR*S = P*S*CAR)

o sorting on this value puts the highest risks to the budget first.o see concerns about schedule variance as this is a function of it, as

illustrated in the equation above.

Risk in a project or process can be due either to Special Cause Variation or Common Cause Variation and requires appropriate treatment. That is to re-iterate the concern about extremal cases not being equivalent in the list immediately above.

Risk management activities as applied to project management

In project management, risk management includes the following activities:

Planning how risk management will be held in the particular project. Plan should include risk management tasks, responsibilities, activities and budget.

Assigning a risk officer - a team member other than a project manager who is responsible for foreseeing potential project problems. Typical characteristic of risk officer is a healthy skepticism.

Maintaining live project risk database. Each risk should have the following attributes: opening date, title, short description, probability and importance. Optionally a risk may have an assigned person responsible for its resolution and a date by which the risk must be resolved.

Creating anonymous risk reporting channel. Each team member should have possibility to report risk that he foresees in the project.

Preparing mitigation plans for risks that are chosen to be mitigated. The purpose of the mitigation plan is to describe how this particular risk will be handled – what, when, by who and how will it be done to avoid it or minimize consequences if it becomes a liability.

Summarizing planned and faced risks, effectiveness of mitigation activities, and effort spent for the risk management.

Risk management and business continuity

Risk management is simply a practice of systematically selecting cost effective approaches for minimising the effect of threat realization to the organization. All risks can never be fully avoided or mitigated simply because of financial and practical limitations. Therefore all organizations have to accept some level of residual risks.

Whereas risk management tends to be preemptive, business continuity planning (BCP) was invented to deal with the consequences of realised residual risks. The necessity to have BCP in place arises because even very unlikely events will occur if given enough time. Risk management and BCP are often mistakenly seen as rivals or overlapping practices. In fact these processes are so tightly tied together that such separation seems artificial. For example, the risk management process creates important inputs for the BCP (assets, impact assessments, cost estimates etc). Risk management also proposes applicable controls for the observed risks. Therefore, risk management covers several areas that are vital for the BCP process. However, the BCP process goes beyond risk management's preemptive approach and moves on from the assumption that the disaster will realize at some point.

Release Life Cycle

Introduction

A software release is the distribution, whether public or private, of an initial or new and upgraded version of a computer software product. Each time a software program or

system is changed, the software engineers and company doing the work decide on how to distribute the program or system, or changes to that program or system.

Software patches are one method of distributing the changes, as are downloads and compact discs.

Software release stages End of life

Release Life Cycle Stages

The software release life cycle is composed of different stages that describe the stability of a piece of software and the amount of development it requires before final release. Each major version of a product usually goes through a stage when new features are added, or the alpha stage; a stage when it is being actively debugged, or the beta stage; and finally a stage when all important bugs have been removed, or the stable stage.

Intermediate stages may also be recognized. The stages may be formally announced and regulated by the project's developers, but sometimes the terms are used informally to describe the state of a product. Conventionally, code names are often used by many companies for versions prior to the release of the product, though the actual product and features are rarely secret.

Pre-alpha

Sometimes a build known as pre-alpha is issued, before the release of an alpha or beta. In contrast to alpha and beta versions, the pre-alpha is not "feature complete". When it is used, it refers to all activities performed during the software project prior to software testing. These activities can include requirements analysis, software design, software development and unit testing.

In Open Source world, there are several types of pre-alpha versions. Milestone versions include specific sets of functionality and are released as soon as the functionality is complete. Nightly builds are versions that are usually automatically checked out from the revision control system and built, typically over night; these versions allow the testers to test the recently implemented functionality immediately, and find the new bugs.

Alpha

The alpha build of the software is the build delivered to the software testers, that is persons different from the software engineers, but usually internal to the organization or community that develops the software. In a rush to market, more and more companies are engaging external customers or value-chain partners in their alpha testing phase. This allows more extensive usability testing during the alpha phase.

In the first phase of testing, developers generally test the software using white box techniques. Additional validation is then performed using black box or grey box techniques, by another dedicated testing team, sometimes concurrently. Moving to black box testing inside the organization is known as alpha release.

Beta

A beta version is the first version released outside the organization or community that develops the software, for the purpose of evaluation or real-world black/grey-box testing. The process of delivering a beta version to the users is called beta release. Beta level software generally includes all features, but may also include known issues and bugs of a less serious variety.

The users of a beta version are called beta testers. They are usually customers or prospective customers of the organization that develops the software. They receive the software for free or for a reduced price, but act as free testers.

Beta versions test the supportability of the product, the go-to-market messaging (while recruiting Beta customers), the manufacturability of the product, and the overall channel flow or channel reach.

Beta version software is likely to be useful for internal demonstrations and previews to select customers, but unstable and not yet ready for release. Some developers refer to this stage as a preview, a prototype, a technical preview (TP) or as an early access. As the second major stage in the release lifecycle, following the alpha stage, it is named after the Greek letter beta, the second letter in the Greek alphabet.

Often this stage begins when the developers announce a feature freeze on the product, indicating that no more feature requirements will be accepted for this version of the product. Only software issues, or bugs and unimplemented features will be addressed.

Developers release either a closed beta or an open beta; closed beta versions are released to a select group of individuals for a user test, while open betas are to a larger community group, usually the general public. The testers report any bugs that they found and sometimes minor features they would like to see in the final version.

An example of a major public beta test was when Microsoft started releasing regular Windows Vista Community Technology Previews (CTP) to beta testers starting in January 2005. The first of these was build 5219. Subsequent CTPs introduced most of the planned features, as well as a number of changes to the user interface, based in large part on feedback from beta testers. Windows Vista was deemed feature complete with the release of build 5308 CTP, released on February 22, 2006, and much of the remainder of work between that build and the final release of the product focused on stability, performance, application and driver compatibility, and documentation.

When a beta becomes available to the general public it is often widely used by the technologically savvy and those familiar with previous versions as though it were the finished product. Usually developers of freeware or open-source betas release them to the general public while proprietary betas go to a relatively small group of testers. Recipients of highly proprietary betas may have to sign a non-disclosure agreement. A release is called feature complete when the product team agrees that functional requirements of the system are met and no new features will be put into the release, but significant software bugs may still exist. Companies with a formal software process will tend to enter the beta period with a list of known bugs that must be fixed to exit the beta period, and some companies make this list available to customers and testers.

As the Internet has allowed for rapid and inexpensive distribution of software, companies have begun to take a more flexible approach to use of the word "beta". Netscape Communications was infamous for releasing alpha level versions of its Netscape web browser to the public and calling them “beta” releases. In February 2005, ZDNet published an article about the recent phenomenon of a beta version often staying for years and being used as if it were in production-level. It noted that Gmail and Google News, for example, had been in beta for a long period of time and were not expected to drop the beta status despite the fact that they were widely used; however, Google News did leave beta in January 2006. This technique may also allow a developer to delay offering full support and/or responsibility for remaining issues. In the context of Web 2.0, people even talk of perpetual betas to signify that some software is meant to stay in beta state. Also, "beta" is sometimes used to indicate something more like a release candidate such as the Halo 3 public beta.

Origin of 'alpha' and 'beta'

The term beta test applied to software comes from an early IBM hardware product test convention dating back to punched card tabulating and sorting machines. Hardware first went through an alpha test for preliminary functionality and small scale manufacturing feasibility. Then came a beta test to verify that it actually correctly performed the functions it was supposed to and could be manufactured at scales necessary for the market, and then a c test to verify safety.

With the advent of programmable computers and the first shareable software programs, IBM used the same terminology for testing software. Beta tests were conducted by people or groups other than the developers. As other companies began developing software for their own use, and for distribution to others, the terminology stuck and now is part of our common vocabulary.

Seigo Stage

The Seigo Stage is the stage in software development when the libraries for the product are ready (or very close to being ready) but the rest of the product is still in need of polishing up. The stage comes between the Beta and Release Candidate stages of software development because the software product is still not ready to be a potential final product but libraries of the software are release quality.

Origin of the 'Seigo Stage'

The term Seigo Stage was invented after a heated discussion on The Linux Action Show! podcast regarding the naming of the KDE 4 second release candidate not being up to the standard of what a release candidate should be. The discussion took place between the two presenters of the show and Aaron Seigo via Skype. It was finally agreed that the release candidate had been termed as such because it was beyond beta quality as the libraries were of release candidate quality. At this point one of the presenters said that they should rename the stage KDE 4 had reached, the name they choose was the Seigo Stage.

Release candidate

The term release candidate refers to a version with potential to be a final product, ready to release unless fatal bugs emerge. In this stage, the product features all designed functionalities and no known showstopper-class bugs. At this phase the product is usually code complete.

Microsoft Corporation often uses the term release candidate. During the 1990s, Apple Inc. used the term "golden master" for its release candidates, and the final golden master was the general availability release. Other terms include gamma (and occasionally also delta, and perhaps even more Greek letters) for versions that are substantially complete, but still under test, and omega for final testing of versions that are believed to be bug-free, and may go into production at any time. (Gamma, delta, and omega are, respectively, the third, fourth, and last letters of the Greek alphabet.) Some users disparagingly refer to release candidates and even final "point oh" releases as "gamma

test" software, suggesting that the developer has chosen to use its customers to test software that is not truly ready for general release. Often, beta testers, if privately selected, will be billed for using the release candidate as though it were a finished product.

A release is called code complete when the development team agrees that no entirely new source code will be added to this release. There may still be source code changes to fix defects. There may still be changes to documentation and data files, and to the code for test cases or utilities. New code may be added in a future release.

Gold or general availability release

The gold or general availability release version is the final version of a particular product. It is typically almost identical to the final release candidate, with only last-minute bugs fixed. A gold release is considered to be very stable and relatively bug-free with a quality suitable for wide distribution and use by end users. In commercial software releases, this version may also be signed (used to allow end-users to verify that code has not been modified since the release). The expression that a software product "has gone gold" means that the code has been completed and "is being mass-produced and will be for sale soon." Other terms for the version include gold master, golden master, “GM”, gold release, or gold build.

The term gold anecdotally refers to the use of "gold master disc" which was commonly used to send the final version to manufacturers who use it to create the mass-produced retail copies. It may in this context be a hold-over from music production. In some cases, however, the master disc is still actually made of gold, for both aesthetic appeal and resistance to corrosion.

RTM or RTW

Microsoft and others use the term "release to manufacturing" (RTM) to refer to this version (for commercial products, like Windows XP, as in, "Build 2600 is the Windows XP RTM release"), and "release to Web" (RTW) for freely downloadable products. Typically, RTM is weeks or months before General Availability (GA) because the RTM version must be stamped to disc and boxed etc.

Box copy

A box copy is the final product, printed on a disc that is included in the actual release, complete with disc graphic art. This term is used mostly by reviewers to differentiate from gold master discs. A box copy does not necessarily come enclosed in the actual boxed product - it refers to the disc itself.

Stable or Unstable

In open source programming, version numbers or the terms stable and unstable commonly distinguish the stage of development. The term stable refers to a version of software that is substantially identical to a version that has been through enough real-world testing to reasonably assume there are no showstopper problems, or at least that any problems are known and documented.

On the other hand, the term unstable does not necessarily mean that there are problems - rather, that enhancements or changes have been made to the software that have not undergone rigorous testing and that more changes are expected to be imminent. Users of such software are advised to use the stable version if it meets their needs, and to only use the unstable version if the new functionality is of interest that exceeds the risk that something might simply not work right.

In the Linux kernel, version numbers are composed of three numbers, separated by a period. Between versions 1.0.0 and 2.6.x, stable releases had an even second number and unstable release an odd one. As of Linux 2.6.x, the even or odd status of the second number no longer holds any significance. The practice of using even and odd numbers to indicate the stability of a release has been used by other open and closed source projects.

End of life

Sometimes software companies stop selling or supporting their software products (e.g., not releasing new patches).

At this point the product will be said to be in the status of "legacy", "vintage" or "end of life". For instance, on August 15, 2007, Apple announced that AppleWorks reached "end-of-life status".