Independent Consultant QAQC Review of the Natural ... · Independent Consultant QAQC Review of the...

INDEPENDENT CONSULTANT QAQCREVIEW OF THE NATURAL ENVIRONMENT ERA CITY OF PORT COLBORNE COMMUNITYBASED RISK ASSESSMENT

CONFIDENTIAL

INDEPENDENT CONSULTANT QAQC REVIEW OF THE NATURAL ENVIRONMENT ERA CITY OF PORT COLBORNE COMMUNITY BASED RISK ASSESSMENT Prepared for: PUBLIC LIAISON COMMITTEE & CITY OF PORT COLBORNE c/o City of Port Colborne 66 Charlotte Street Port Colborne, Ontario L3K 3C8 Prepared by: WATTERS ENVIRONMENTAL GROUP INC. 8800 Dufferin Street, Suite 303 Concord, Ontario L4K 0C5

Revised November 2010 Reference No. 04-0007

Independent Consultant QAQC Review of the Natural Environment ERA City of Port Colborne CBRA Page i

Watters Environmental Group Inc. CONFIDENTIAL Reference No. 04-0007 November 2010

TABLE OF CONTENTS

SECTION PAGE

TABLE OF CONTENTS ................................................................................................................. i

1.0 INTRODUCTION ...............................................................................................................1

1.1 QUALITY ASSURANCE AND THE CBRA PROCESS ......................................2

1.2 QUALITY ASSURANCE AND QUALITY CONTROL .......................................3

1.3 QA/QC ACTIVITIES AND THE ERA-NE ............................................................3

1.4 THE ROLE OF THE INDEPENDENT CONSULTANT IN QA/QC ....................7

1.5 ERA-NE DATA QUALITY AND QUANTITY ISSUES ......................................7

1.6 REVIEW OF KEY QA COMPONENTS IN THE ERA-NE ................................10

1.7 SCOPE OF THE QA/QC REPORT ......................................................................11

2.0 PLANNING .......................................................................................................................12

2.1 INTRODUCTION .................................................................................................12

2.2 TECHNICAL SCOPE OF WORK ........................................................................13

2.3 FIELD WORK .......................................................................................................14

3.0 SAMPLING .......................................................................................................................16

3.1. PLANNING AND DATA QUALITY OBJECTIVES ..........................................16

3.2 SAMPLING AND LAB STANDARD OPERATING PROCEDURES ...............17

3.3. INCOMPLETE OR INADEQUATE PROTOCOLS ............................................18

3.4. ERA SAMPLING BEFORE PROTOCOLS DEVELOPED OR WITHOUT PROTOCOLS ........................................................................................................19

3.5. USE OF DRAFT AND UN-REVIEWED PROTOCOLS AND THE FAILURE TO ADHERE TO SAMPLING PROTOCOLS .....................................................19

3.6. THIRD PARTY OVERSIGHT OF QA/QC ..........................................................20

4.0 ANALYSIS OF SAMPLES ...............................................................................................21

4.1. SELECTION OF ANALYTICAL LABORATORY.............................................21

4.2. SAMPLING DATA BASE ....................................................................................22

4.3. QA/QC ASSESSMENT OF ERA-NE LABORATORY DATA ..........................22

Independent Consultant QAQC Review of the Natural Environment ERA City of Port Colborne CBRA Page ii

Watters Environmental Group Inc. CONFIDENTIAL Reference No. 04-0007 Revised November 2010

TABLE OF CONTENTS (Continued)

SECTION PAGE

4.4. INDEPENDENT AUDITS OF ERA SAMPLING AND ANALYTICAL DATA23

4.4.1 Species Distribution and Abundance Surveys .........................................24

4.4.2 Independent Duplicate QA/QC Sampling and Testing of ERA Samples24

4.5. RESULTS ..............................................................................................................24

4.6. DISCUSSION ........................................................................................................26

4.6.1 Maple Sap ................................................................................................26

4.6.2 Maple Tree Soil .......................................................................................26

4.6.3 Soft Maple Leaf .......................................................................................27

4.6.4 Soil ...........................................................................................................27

4.6.5 Earthworms ..............................................................................................27

4.6.6 Earthworm Soil ........................................................................................28

4.6.7 Frogs ........................................................................................................28

4.6.8 Tadpoles ..................................................................................................30

5.0 ERA DATA QUALITY .....................................................................................................33

5.1 INTRODUCTION .................................................................................................33

5.2 STUDY AREA SIZE AND ERA-NE SAMPLES.................................................33

5.3. ERA-NE DATA: ISSUES AROUND SINGLE POINT-IN-TIME SAMPLING 35

5.4 ERA-NE SURVEY ISSUES..................................................................................36

5.5 ERA-NE DATA VARIABILITY AND LACK OF DQOS ..................................37

5.6 THE EFFECT OF “AVERAGING” ON ERA-NE DATA QUALITY .................40

5.7 SAMPLING ISSUES AS A “LIMITATION” .......................................................45

5.8 CONCLUSIONS: ERA DATA QUALITY...........................................................46

6.0 REFERENCES ..................................................................................................................48

Independent Consultant QAQC Review of the Natural Environment ERA City of Port Colborne CBRA Page iii


TABLE OF CONTENTS (Continued)

LIST OF APPENDICES

Appendix A: The Independent Consultant’s Review of Quality Assurance and Quality Control for the Port Colborne CBRA, Environmental Risk Assessment – Natural Environment

Appendix B QA/QC Analytical Data for Maple Sap

Appendix C QA/QC Analytical Data for Maple Tree Soil

Appendix D QA/QC Analytical Data for Soft Maple Leaves

Appendix E QA/QC Analytical Data for Earthworm Tissue

Appendix F QA/QC Analytical Data for Earthworm Soil

Appendix G QA/QC Analytical Data for Frog Carcass, Liver and GI Tract

Appendix H QA/QC Analytical Data for Tadpole Carcass, Liver and GI Tract

Independent Consultant QAQC Review of the Natural Environment - ERA City of Port Colborne CBRA Page 1


1.0 INTRODUCTION

A final Ecological Risk Assessment report concerning the impacts of emissions from a former Inco nickel refinery on the natural environment within the City of Port Colborne, Ontario was prepared by Jacques Whitford (JW), on behalf of their client, Vale Inco (Inco). This five-volume report is entitled, “Community Based Risk Assessment Port Colborne, Ontario; Ecological Risk Assessment Natural Environment” and dated September 2004 (the Natural Environment Report), and is one component of a Community Based Risk Assessment (CBRA) that is attempting to address potential impacts from former Inco emissions on agricultural crops, the natural environment, and human health within the City of Port Colborne.

Watters Environmental Group Inc. (Watters Environmental) is the Independent Consultant to the City of Port Colborne and the Public Liaison Committee (PLC) for the CBRA and has prepared this report to document, review and comment on the overall quality of the individual technical studies that were carried out by JW and which formed the basis for the Natural Environment Report.

Reliable analytical measurements of environmental samples are an essential ingredient in the process of understanding the environment, and in determining how best to safeguard it and/or to clean it up. Environmental sampling and analysis are disciplines that have only existed as distinct areas of expertise for about thirty years, and it is only in the very recent past that there has been general agreement over what constitutes “reliable” environmental data. While most people would agree with the viewpoint that environmental measurements should be reliable, there is often debate as to what this actually means, how to achieve it, and how to balance reliability against cost.

Data reliability can only be achieved via a process of quality assurance (QA), and an environmental program with QA is more expensive than programs with little or no QA. The unfortunate fact of the matter is that many environmental projects do not receive adequate QA because of costs or ignorance of its importance, and there are environmental studies that arrived at incorrect conclusions because the environmental data was not adequate. When such conclusions are used for environmental decision-making, such as cleanup of a contaminated site, the costs of making poor decisions can be enormous and far outweigh the cost associated with conducting effective QA.

As the various modules of the CBRA evolved, QA was built into every step of the data gathering component. It is important to note that QA applies to every aspect of the CBRA, from field sampling to laboratory analysis to data assessment to final report preparation.

Independent Consultant QAQC Review of the Natural Environment ERA City of Port Colborne CBRA Page 2


These activities are distinctly separate but intimately interrelated, and errors, biases and blunders in any activity will affect the other activities. Good analytical accuracy in the laboratory can never compensate for errors made during sample collection. Great care to ensure representative sample collection in the field will not indemnify against poor precision in the laboratory or incorrect statistical treatment of sample data. In order to produce a reliable, trustworthy environmental study, all of the components must be properly planned, executed, documented, and reported.

1.1 QUALITY ASSURANCE AND THE CBRA PROCESS

The Port Colborne CBRA is not a typical environmental assessment, and the QA applied to the CBRA is similarly not typical of most environmental assessments. The process is different from the more typical “top-down” standard site assessment through the inclusion of many groups in addition to the original “parties”; i.e., Inco, the City of Port Colborne and the Ontario Ministry of the Environment (MOE), the very public nature of the process and the way that the CBRA process evolved over time.

In the case of most environmental assessments, the elements of QA are established right at the outset of the assessment allowing inclusion of the elements of QA to be present from the beginning to the end of the process. In the case of the Port Colborne CBRA, QA was not in place at the beginning of the studies and was part of a changing process.

The CBRA process was deliberately structured to involve the community (as indicated by the name “Community Based Risk Assessment”) and community involvement was part of the process from the planning stage through to the development and release of the final CBRA results, documentation, conclusions and recommendations. This approach provided strength in ensuring that the scope satisfied (or at least considered) most community concerns and was able to secure strong community support at the beginning of the process. The downside was that studies continually evolved taking public input into account, resulting in delays in protocol preparation, including the QA/QC protocols. This problem was exacerbated in many instances when, as a consequence of delays, there was a rush to get into the field to meet “windows” when sampling or observation must be done or miss an entire season. This resulted in some of the field studies being undertaken without finalized protocols to clearly describe the objectives of the study and data quality.

Regular public meetings were held, as well as several Open Houses, to ensure that the public was apprised of the development of the various activities of the project. In order to ensure adequate discussion of complex technical matters, a Technical Sub-Committee (TSC) was formed with



membership from the City of Port Colborne, PLC members, MOE, JW, the Independent Consultant, Inco and the Regional Department of Health. Members of the public were invited to attend TSC meetings as observers. The TSC meetings were chaired by the Independent Consultant. TSC meetings were intended to foster a free exchange of ideas respecting the science of conducting the CBRA and associated studies. Consequently, formal minutes were not taken of TSC meetings. Notes of key discussion points and agreed actions were maintained by the Chair of the TSC (issued through “Chairman’s Notes”).

This process of holding PLC and TSC meetings allowed for regular reporting of progress, communication of coming events, and an opportunity for study elements to be regularly reviewed, assessed and challenged. This process also helped in the timely identification of problems.

1.2 QUALITY ASSURANCE AND QUALITY CONTROL

In order to make sound environmental decisions, it is necessary to obtain adequate information about the issue at hand, and this information must be known to be accurate, precise and reliable. The application of a sound quality assurance program to the gathering of data is an essential aspect of environmental data gathering and of assessing, correcting and maintaining quality control (Keith, 1991; Clark, 2000; Mesley, 1991; Rose and Smith, 1992).

Quality control is a planned system of activities whose purpose is to provide a quality product. Quality assurance is a planned system of activities whose purpose is to provide assurance that the quality control program is actually effective. The purpose of this report is to assess the quality control activities and the overall quality assurance processes used in the ERA-NE portion of the Port Colborne CBRA that would allow an objective reviewer to form an opinion as to the accuracy, precision and quality of the data on which the conclusions in the ERA-NE are founded.

1.3 QA/QC ACTIVITIES AND THE ERA-NE

The ERA-NE study for the Port Colborne CBRA comprises many separate studies addressing different aspects of the natural environment of Port Colborne. As these studies are, necessarily, quite different in approach, the QA (and QC) activities for the various studies take many different forms. The more formal aspects of QA/QC for the ERA-NE included the planning process, which sets the goals and the decision criteria that will be needed for all subsequent activities, documentation of study objectives, training of personnel, and so on. These activities are consistent with those considered to be “the key elements of the QA component of environmental analysis studies” (Clark, 2000) i.e.:



Comprehensive planning;

Documentation of study objectives;

Documentation of study methodologies;

Thorough training of personnel;

Comprehensive record keeping;

Timely identification and resolution of problems;

Regular reporting of results;

Routine challenge of study elements; and

Regular independent audits to ascertain whether study elements are in control.

The Technical Scope of Work (TSOW) for the CBRA was developed over numerous meetings between JW, the Independent Consultant, the MOE and others and, after several drafts; the TSOW included a plan for the conduct of the studies and incomplete documentation of study objectives (TSOW, November 2000).

The development of individual study protocols was undertaken by JW and these were presented to the TSC for its review and comment. Although not consistently followed, the process required that JW finalize the various protocols only after review, discussion and approval at the TSC. JW and the Independent Consultant provided training to field staff while their sub-contracted analytical service, PSC Analytical, provided training to laboratory staff. Comprehensive record keeping was primarily provided by JW and PSC Analytical. The Independent Consultant also maintained records of its activities relating to quality control and assurance during the conduct of fieldwork. JW and PSC Analytical provided regular reports of conduct of field work and analytical results.

The dynamic and structure of the CBRA were such that study elements were routinely challenged by the PLC and its consultants, as well as by various other entities and the public at large.



The Independent Consultant and JW developed a system of regular independent audits designed to ascertain whether study elements were being conducted in accordance with the agreed protocols. This involved staff from the Independent Consultant attending JW sampling events to observe and monitor adherence to protocols and in situations where it was seen to add value, to collect duplicate samples for independent analyses.

Table 1 shows how the key QA elements are incorporated into the CBRA.

Table 1: Elements of Quality Assurance in the Port Colborne CBRA ERA-NE

Comprehensive Planning Establishment of the TSOW by JW, with review and concurrence from the PLC and TSC

Other planning meetings involving JW, the Independent Consultant, PLC and MOE

Documentation of Study Objectives

TSOW and scoping documents for the various studies and assessments for the ERA NE

Documentation of Study Methodologies

Sampling and analytical protocols

QA/QC protocols

Data interpretation protocols

Thorough Training of Personnel

Ongoing training by JW and the Independent Consultant, including field and safety training

Comprehensive Record Keeping

Program databases maintained by JW

QA/QC databases (maintained by the Independent Consultant)

Analytical databases maintained by PSC Analytical

Chairman’s notes of TSC meetings taken by the Independent Consultant

Meeting notes of PLC meetings taken by the City of Port Colborne

Timely Identification and Resolution of Problems

Identified during regular meetings of the PLC and TSC

Consultant’s meetings involving JW, other Inco consultants, the Independent Consultant, Inco and MOE. (Note that reports on these meetings and key outcomes were discussed at subsequent PLC meetings)



Table 1: Elements of Quality Assurance in the Port Colborne CBRA ERA-NE (Continued)

Regular Reporting of Results Data assessment was completed by JW, with input and review by the TSC and PLC

Interim results were reported at PLC and TSC meetings, as well as public Open Houses

Routine Challenge of Study Elements

Challenge was provided by the PLC, members of the public at PLC meetings, the TSC and the Independent Consultant

Response to challenges was provided by JW and Inco

Public input was achieved through email, letters, Open House discussions, PLC and TSC meetings, and the local newspapers

Regular Independent Audits to Ascertain Whether Study Elements are In Control

Audits of studies were conducted by the Independent Consultant, PLC and MOE

Independent peer review was sought for some elements of the study and for Draft and Final Reports.

The Independent Consultant attended and provided oversight during JW ERA-NE sampling

In some instances, independent duplicate sampling alongside JW ERA samples was conducted by the Independent Consultant.

Elements that demonstrated incomplete or inadequate QA implementation in the Port Colborne ERA-NE included Comprehensive Planning, Documentation of Study Objectives, Documentation of Study Methodologies and the Timely Identification and Resolution of Problems.

An inescapable aspect of environmental monitoring is that, even if all the key QA elements are followed and all the QC checks are in place, the analytical results will not necessarily be accurate, precise or reliable, and incorrect values will show up. However, by applying thorough QA and QC procedures throughout the process, the number of unreliable data points can be minimized and the extent of errors can be controlled.



1.4 THE ROLE OF THE INDEPENDENT CONSULTANT IN QA/QC

The Independent Consultant’s role was primarily to assist the City of Port Colborne and the PLC in understanding the science within the CBRA. The Independent Consultant was also required to help ensure that quality assurance was an integral part of the sampling, analytical, assessment and reporting stages of the studies carried out for the ERA. The Independent Consultant, with input from the PLC, critiqued the study elements of these projects to sharpen the focus of the project and to ensure that proper planning and sampling was carried out, and to ensure that QA/QC was in place and could be documented for the various projects.

The Independent Consultant also developed and led the “third party” independent check on many of the field programs undertaken by JW through replicate sampling and analysis of samples parallel to JW in the field and in the laboratory.

The Independent Consultant’s role was to help ensure that quality assurance was incorporated into the ERA NE by working with JW to ensure that the QA/QC necessary to ensure proper data production were incorporated into the study plan; and to work with the PLC to ensure that its members understood the purpose, sequencing and timing of the various studies.

1.5 ERA-NE DATA QUALITY AND QUANTITY ISSUES

The Port Colborne CBRA ERA consists of a large number of sub-projects that were designed to provide data that would be capable of permitting an assessment of the ecological risks associated with the past operation of the nickel refinery in Port Colborne. For some specialist sub-projects, sub-contracted consultants were retained. The sub-projects ranged from sampling and analyzing soil samples around Port Colborne, through analysis of CoCs in plant tissue, to determining the level of various metals in spiders, worms and other animals, to estimating the populations of local birds and amphibians.

A large number of soil, vegetal and animal samples were collected for chemical analysis. For such samples, it is relatively easy to establish measurable QA/QC indicators such as analytical precision, or the precision of field duplicates. However, it is much more difficult to determine the actual number of samples required to provide an acceptably accurate, precise estimate of the level of CoCs, for example, in spiders from a field, or to be able to state with acceptable certainty that spiders from one field have higher levels of a CoC than spiders from another field.



The difficulty arises from the unknown nature of the normal variability in the composition of the natural world. This variability might be different from one geographic area (an agricultural field, for example) to another (such as an urban area or a woodlot), and the variability may also be different for different CoCs in the same location. This variability makes it very difficult to ascertain, at the onset of a field survey, what is the appropriate number of samples required for determination of the level of CoCs in the soil, or in vegetation growing on the soil, or in insects living on the vegetation growing on the soil (or in animals or birds that eat these insects).

The natural environment is an untidy, capricious, and variable place. Nutrients, including micro-nutrients, are in limited supply and are cycled tightly amongst the resident populations. The uptake of different nutrients or CoCs may occur on highly variable timescales. The growth of bacteria (essential to many of the biological processes in the natural environment) can be affected by changes in temperature or even by clouds moving to cover the sun, perhaps on a timescale of minutes. The depuration or uptake of CoCs and other contaminants by an insect or a small mammal or bird can occur over a few days, and the entire life cycle of some insects can be completed over days or weeks.

All of these fluctuations conspire to make it difficult to properly predict how many samples, over how many days, weeks, or years will be necessary (or available) to unequivocally determine the level of CoCs and other components of living systems. Given the natural fluctuation of animal populations in the wild and the levels of CoCs in such populations (and in vegetation, to a lesser extent), samples taken in one month or year can yield results that may not agree with sample results from a different season or year. In some instances, missing a sampling window by a matter of days, e.g. in conducting frog calling surveys, can result in misleading results.

The consequence of this is that sampling projects were undertaken without a clear plan as to how many specimens of each sample type would be required, and without establishing the degree of certainty that would be required. It is acknowledged that there is never enough money, time, or laboratory capacity to ensure complete certainty. However, there were concerns expressed by PLC members and by the Independent Consultant that insufficient planning and debate had occurred to reach agreement on the numbers and types of sample that would be required for the ERA sampling to be able to meet its objectives.

The TSC discussed the need to clearly define what constituted a “reasonable” degree of certainty (60% certainty, or 90%, or 95%, and so on), and then to translate that degree of certainty into a Data Quality Objective (DQO), which would determine the number of samples required in order for each valued ecosystem component (VEC) or soil type. However, these were not incorporated into the study designs.



DQOs are suggested by the MOE as “statements of the required quality of data obtained from analytical results and from the overall field program” (MOE, 1996). While these may be difficult to establish, the scientific literature does provide useful information regarding the concentration of metals and other chemicals in various compartments of the environment. Generally, most environmental data is considered “reasonable” if it is in agreement with the generally accepted literature values, and if appropriate QA/QC is in place. It is a weakness of the ERA-NE that no comparison of the ERA sample values for the CoCs to those in the world literature has been performed and presented in the ERA-NE report.

There is no single answer as to what constitutes the appropriate number of samples required for a proper evaluation of potential impacts, as each environmental compartment or species will have different quality objectives and these might change for different zones within the CBRA study areas. However, techniques are available to provide guidance on this matter, but they were not included in the protocols or in the Scope of Work for the CBRA.

The number of samples taken for a specific environmental project is often limited by the availability of the specimens to be sampled. For example, during the ERA it was found that earthworm populations were much more plentiful in some woodlots than in others. Such situations are unavoidable, but when they occur the result may be that one population cannot be meaningfully compared with others, or as was the case with the ERA-NE Report, mathematically added to the others. A further limitation that occurred in the CBRA was that some property owners refused to give Inco’s consultants access to lands and habitat that had been deemed to be important locations to sample. The result of the lack of these data cannot, of course, be easily estimated.

While considerable quantities of data were collected during the Port Colborne CBRA, there were constraints. The main constraint was time, because of public and other demands that the study portion of the CBRA extend longer than necessary in order to meet important time windows. While many members of the public were of the view that the CBRA has taken far too long to complete, others may feel that more samples and analyses should have been taken. These conflicting attitudes reflect the difficult balance between the need for scientific certainty and the need for timely delivery that exists in studies such as the CBRA.



1.6 REVIEW OF KEY QA COMPONENTS IN THE ERA-NE

For most environmental monitoring, planning is undertaken before samples are taken and analyses carried out. Planning is usually a collaborative effort between the proponent and their technical consultants, and for a large undertaking such as the CBRA, with input from the public and other stakeholders.

In the case of the Port Colborne CBRA, some of the planning was done with public participation, input from the PLC, the Independent Consultant and the MOE, and from groups such as the local health unit. However, in many instances, by the time the planning stage reached a point where there was general agreement regarding the conduct of field work, Inco’s consultant had already been into the field collecting samples.

Sub-projects followed a normal course of sampling, analysis, data assessment and report writing, with the conclusions and data becoming part of the overall ERA output.

As previously noted, measurable, quantifiable QA/QC indicators are available for some, but not all, parts of the ERA. Examples of these include the analytical precision for field duplicates and for inter-laboratory replicates, accuracy estimates of laboratory testing based on standard or certified reference materials (SRMs), and laboratory spike recovery measurements. These provide results that can be measured and assessed. However, some activities, such as planning, or writing and following protocols, are more qualitative and assessment and verification for these elements of the CBRA studies was much more subjectively carried out through third party observation of the collection of samples by JW personnel and by taking duplicate portions of some samples for independent testing.

Third party independent QA/QC was provided by having a portion of certain field samples sub-sampled by the Independent Consultant, who sent these samples for testing at the same analytical laboratory as JW, but under a double blind numbering system.

For important but less easily measurable activities such as planning, data assessment and reporting the appropriateness of the direction taken or the quality of decisions could not be known until after the final product, in this case, the ERA-NE Report, was prepared and the final conclusions, and their rationale, was available for examination. This is generally the case in this type of scientific study but was particularly so in the CBRA studies, where successive draft versions of reports varied quite considerably in the information presented as being pertinent and in the inferences drawn in the conclusions.



1.7 SCOPE OF THE QA/QC REPORT

Studies such as those comprising the CBRA have a number of key elements:

Planning;

Sampling;

Analysis;

Data Assessment; and

Reporting.

In this review of QA/QC for the CBRA, we consider the planning of the studies, sampling methodology and sample-taking, and analysis of samples.



2.0 PLANNING

2.1 INTRODUCTION

It is axiomatic that, for an undertaking as complex as the Port Colborne CBRA, adequate planning is essential; all subsequent work depends upon proper and effective planning. Proper planning implies that consideration has been given to ensuring that adequate numbers of samples are taken at the right locations and that these locations represent generally agreed upon compartments of the environment. Thought has to be given to the types of samples that need to be taken, including sensitive receptors, how representative these are of other receptors in the environment, place in the food web, etc. Planning needs to consider the timing and timeframe over which the samples are taken and that adequate analytical techniques can be applied so as to properly determine the levels of CoCs in each sample taking consideration of factors such as anticipated concentrations and detection limits.

The Port Colborne CBRA was the first full CBRA carried out in Ontario, and it came about after the MOE had developed the Site Specific Risk Assessment (SSRA) approach as an alternative to the application of the MOE generic guidelines (now standards). In developing the SSRA approach, the MOE prepared a number of guidance documents for use in directing the implementation of risk assessments in Ontario. These documents provide guidance and describe procedures to be followed by those wishing to undertake site assessment activities in the province, and they provide explicit guidance regarding the need for planning before sampling is carried out for a site.

The MOE document “Guidance on Sampling and Analytical Methods for Use at Contaminated Sites in Ontario” (MOE, December 1996) states:

“Prior to sampling a site, the purpose and objectives of the sampling program should be clearly stated.”

The MOE’s report goes on to emphasize the need for planning, as follows:

“Adequate planning of the sampling program must occur in order to assure that samples represent the areas and depths desired, that sampling variability is properly determined and accounted for, and that there are sufficient number of samples at the appropriate locations to fulfill the purposes of the sampling. These considerations can be accounted for if specific objectives are defined early in the planning. Such objectives should also include statements of



the required quality of data obtained from analytical results and from the overall field program; these are referred to as Data Quality Objective (DQOs).” (MOE, 1996, p. 18)

2.2 TECHNICAL SCOPE OF WORK

The geography of each location, the biological and non-biological entities within each location, the manner and degree to which each will be sampled and analyzed, are all part of an intricate web that can only be held together by careful, thoughtful planning. The key document outlining these plans is the Scope of Work. For the Port Colborne CBRA, this became the Technical Scope of Work (TSOW) and it took the better part of a year for Inco and its consultants to have the TSOW in place.

The initial CBRA planning process involved only the proponent (Inco) and its consultants (JW). Very early in the process (in 1998), the MOE became more directly involved and, by 1999, the City of Port Colborne was engaged in the process. The PLC was formed in May 2000, but by that time, a scope of work had already been prepared by JW and Inco, (see PLC Minutes, May 4 and June 1, 2000).

This preliminary Scope of Work lacked sufficient detail to be an effective planning document for the CBRA. In June 2000, the PLC, recognizing the complexity of issues it was being asked to comment on, retained an expert Independent Consultant. The first assignment for the Independent Consultant was to review the scope of work for the CBRA (PLC Minutes, June 29, 2000). It was clear at that time that there were significant shortcomings in the TSOW. Seven or eight iterations were required before the TSOW was accepted by the PLC on November 30, 2000. As the final TSOW (“JW “Technical Scope of Work – Community Based Risk Assessment for Port Colborne, Ontario, November 30, 2000”) evolved, study objectives were discussed and refined at numerous public meetings and with input from the MOE, the public, the PLC and the Independent Consultant. By the time the final TSOW was tabled, initial planning of the ERA was in place, as were many of the basic ERA study objectives.

However, the slow development of the TSOW was a cause for concern on the part of many, including the public and the PLC. The minutes of PLC public meetings during 2000 reflect the growing frustration caused by the quality of the initial Scope of Work.

The frustration of the communities of interest in the CBRA, resulting from the length of time taken to develop an adequate TSOW, resulted in outcomes that were to have continuing negative influence on the conduct of the studies with important implications for QA/QC:



Public attitudes hardened regarding the role of JW and Inco in the CBRA;

Increased pressure was placed on JW to complete its tasks quickly; and

Sound planning became secondary to demonstrating action.

2.3 FIELD WORK

The field work for the CBRA got underway quickly once the TSOW was accepted and, the process became a very public event. While this speaks well to the high level of interest on the part of Port Colborne’s residents, it also put additional pressure on the JW to demonstrate progress. As an observation of how planning and protocol development can be overtaken by events, in some instances the need to show progress resulted in undertaking sampling activity before protocols were developed and agreed upon by the PLC and its consultants.

In April 2002, the Independent Consultant prepared a report specifically related to ERA-NE data quality concerns, entitled “Ecological Risk Assessment QA/QC”. This was prepared with the objective of maintaining order in the planning of sampling events. It more clearly defined the respective roles of JW and the Independent Consultant and provided a means of tracking progress on various concerns regarding QA/QC as sampling and analysis was carried out.

The role of JW was to:

Design and propose study work programs to the proponent;

Develop protocols for each component of the field program;

Submit protocols to the Independent Consultant for review and to amend protocols as necessary;

Inform the Independent Consultant in a timely manner when field work was scheduled to be undertaken;

Provide the Independent Consultant with adequate specimens to conduct independent QA/QC analysis (typically 20% of the samples for a given undertaking); and

Provide results of work to the proponent and the CBRA.



The role of the Independent Consultant was to:

Review the proposed protocols prepared by JW;

Provide suggestions and advice to JW regarding proposed protocols;

Be present for QA/QC purposes during field studies;

Obtain samples for independent QA/QC testing from JW, where appropriate;

Provide information to client (City of Port Colborne) as required; and

Undertake QA/QC assessments of work undertaken by JW and others working on the CBRA.

The agreement of roles was prepared only after much of the ERA sampling had been carried out. As a result, the respective roles of JW and of the Independent Consultant eventually evolved as indicated above but they did not necessarily reflect the actual situation in place at the beginning of the sample collection in 2001.

Various meetings were held between JW, the PLC and the Independent Consultant where concerns were raised over various aspects of the sampling program carried out in 2001. Concerns over inadequate notice to allow proper QA/QC oversight by the Independent Consultant, the number and types of various samples being taken, sample handling (cooling, freezing, dissecting, storage, preservation, etc.) were among the topics brought to JW’s attention by the Independent Consultant.

However, JW’s need to gather samples quickly or miss another sampling season took precedent, with the result being that, while many ERA samples were collected for certain VECs, a significant number of them are not capable of supporting meaningful conclusions for reasons explained in the following section (“Sampling”).

Considerable sampling had been carried out by the MOE in the Port Colborne area before the CBRA began, so Inco had the advantage of knowing the general areas within the City of Port Colborne with soil nickel (Ni) levels above MOE’s generic guideline of 200 mg/kg (designated the Secondary Study Area) and within this zone, those soils with more than 500 mg/kg Ni (designated the Primary Study Area). Once the ERA was underway, the sample planning process should have enabled the establishment of clear goals as to the number of samples required to support the ERA objectives. However, this did not occur.



3.0 SAMPLING

3.1. PLANNING AND DATA QUALITY OBJECTIVES

The objective in collecting environmental samples for analysis is to obtain a small but adequately informative portion of the population being investigated.

For the sample data to be informative, the samples should adequately represent the larger population from which the samples are drawn; that is, the sample should be representative of the whole population within a specified study site. Samples that do not provide representative information about that site are not worth the time and expense of collecting, let alone analysis. “Therefore, planning for informative sampling must be an integral part of any study” (Keith, 1991).

In order to properly determine the type and numbers of samples, it is of utmost importance “that the specific purpose of environmental analysis programs be defined before any routine process of sample collection and sample analysis is established” (M. Clark, 2000). In order to determine the number of samples of each type (e.g., soil, vegetation, insects, amphibians, mammals, etc.) it is essential to determine the degree of total variability (uncertainty or error) that can be tolerated in the data, so that limits of variability can be incorporated into the sampling and analytical plan and achieved with detailed sampling and analysis protocols (L. Keith, 1991). The critical definitions of the level of confidence that will be required when drawing conclusions from the entire project data are expressed by the Data Quality Objectives (DQOs). Without DQOs, the likelihood of obtaining adequate study site data is greatly decreased, as is the likelihood of coming to correct conclusions based on the data.

The specific elements of sample collection that require documentation and protocols are provided in authoritative papers and texts such as those by Keith (1991) and Clark (2000), and they include:

Developing Data Quality Objectives (DQOs) to establish the degree of data variability that can be tolerated for each data set;

Sampling protocols specifying how, when, and how much to collect each type of sample, whether water, soil, biota, air, etc., as well as packaging, preservation, storage and documentation of samples, in order to achieve the DQOs;



Operational protocols covering on-site maintenance and calibration of sampling/field measuring equipment, including equipment cleaning and blanks;

Protocols to include proactive QA such as blank samples, replicate samples, and other QA/QC samples;

Blanks, including travelling and field blank samples (prepared by the laboratory and deployed by samplers);

Details of site location and condition recorded;

Photographs and detailed field records documented; and

Deviations from protocols to be documented.

However, DQOs are not referenced in the ERA-NE Report (2004) or the TSOW, greatly complicating the QA/QC process and eroding the confidence that can be placed in results and consequent conclusions.

3.2 SAMPLING AND LAB STANDARD OPERATING PROCEDURES

The TSOW and the Standard Operating Procedures (protocols) demonstrate that some planning was generally carried out for ERA sampling. However, given the constraints on time, the broad scope and the ground-breaking nature of the CBRA, it may have been unavoidable that some ERA sampling was carried out without the benefit of protocols that had been reviewed and agreed on by the PLC and its consultants.

Often, sampling was dictated by events that precluded the development of protocols and agreement amongst all parties – for example, the sampling of homegrown fruit and vegetables took place in the fall of 2000. When the samples were ready for harvest, they needed to be sampled, with our without a protocol.

The net result of the pressure to make progress on the studies was that often more emphasis was placed on obtaining samples than ensuring that adequate planning had been undertaken regarding sample numbers, types, and protocols.



3.3. INCOMPLETE OR INADEQUATE PROTOCOLS

Most (but not all) of the ERA-NE sampling protocols, were subject to the agreed approval process and consequently there was proper documentation of the study methodologies. In some cases where shortcomings or technical problems were noted for these protocols, these remained unresolved. For example, the final 2004 version of a key protocol, the “Quality Assurance and Quality Control for Field Sampling and Laboratory Procedures”, contains the same technical errors as were noted in comments provided on the original first draft of the protocol dated November 2002.

Similarly, the “Surface Water Sampling Protocol” of the September 2004 ERA-NE Report, retains incorrect sample preservation procedures, which, if followed, would have resulted in improperly preserved water samples which would not provide accurate analytical data. This document also had incorrect information regarding the number and specific types of parameters analyzed, and information which conflicted with the “Quality Assurance and Quality Control for Field Sampling and Laboratory Procedures” referred to above. Neither of these documents accurately describes the actual analytical work performed, or the specific parameters analyzed in water and in other media.

The failure to correct key QA/QC documents that were known to contain technical errors indicates: (i) that the problem resolution component of the ERA-NE process did not work at times, (ii) the authors of these documents either didn’t understand the importance of the errors that were pointed out in review comments, or they chose not to follow the advice. The result is that the final ERA-NE report still contains flawed QA/QC protocols for the sampling and analysis components of the ERA, which can only serve to undermine the credibility of the results and conclusions reached.

Notwithstanding the concerns raised above, actually, many of the protocols underwent various improvements and revisions and in many cases, where problems were identified, they were corrected. The net result is that eventual documentation of the majority of the study sampling methodologies was adequate although many sampling protocols were only deemed to be adequate after the samples had actually been taken.

The processes of protocol preparation, review, correction of protocols, final review and final edits was necessary if protocols of sufficient quality were to be developed and followed. The process of preparing the sampling protocols took much longer than had been anticipated and first, second and often, subsequent drafts of these protocols were incomplete or otherwise



inadequate. Even at the end of the CBRA process, protocols exist that contain errors and do not reflect the actual work that was conducted.

3.4. ERA SAMPLING BEFORE PROTOCOLS DEVELOPED OR WITHOUT PROTOCOLS

As mentioned above, some of the sampling carried out in 2000 and 2001 had to be done without the benefit of a final peer-reviewed protocol that had undergone the agreed approval process. However, the final ERA-NE Report (September 2004) shows that some sampling was carried out without the benefit of having protocols of any kind. Some sampling was carried out with protocols that were generally inadequate and had not been approved, and some sampling was carried out with protocols that had been reviewed by the PLC and its consultant, but which were not finalized and did not include key suggestions and improvements. In fact, it has proven difficult to establish whether any ERA-NE samples were taken after a final, mutually agreed to protocol was in place.

3.5. USE OF DRAFT AND UN-REVIEWED PROTOCOLS AND THE FAILURE TO ADHERE TO SAMPLING PROTOCOLS

By 2001, when most of the sampling took place, JW had begun work on the first drafts of the field and sampling protocols. In some instances, sampling occurred before protocols were written (for example, for tree core sample collection and for corn leaf and kernel collection). In other cases, first drafts of protocols were provided to the Independent Consultant hours or days before sampling was scheduled, which precluded any independent review or reduced the review to a cursory one. This was the case for the bird surveys of May 24 and 31, 2001 (the first protocol was received by the Independent Consultant on May 22, 2001) and for the insect collections in July 2001.

In other cases, the actual sampling was carried out in a manner not consistent with the draft sampling protocol. Examples include the vole sample surveys of September 11 and 14, 2001 and the maple leaf collections of September 11-14, 2001.

For the leaf litter collection, the Independent Consultants were not informed in time to comment on the value of the work. Protocols were not received until most of sampling had taken place, and when comments were provided, some were retroactively incorporated into the study. As a result, most of this work had no QA/QC or independent oversight in either the planning stage or the sampling stage.



As discussed above, Chapter 5 of the final Natural Environment Report (September 2004) is entitled “Data Collection Methods”, and it states that “field data collection protocols were developed, which were reviewed for comment by the TSC and PLC prior to the collection of field data”. This statement is not accurate; several field collection protocols were not reviewed prior to collection of field data. While some protocols were reviewed prior to sample collection, samples were collected before the protocols were corrected based on the reviewers’ comments. The result was that many ERA samples were taken before agreed-upon methods were in place, without the benefit of review by the PLC, its Consultant, and the public in general.

3.6. THIRD PARTY OVERSIGHT OF QA/QC

Third party or “arm’s length” QA/QC was built into ERA-NE sampling and analytical undertakings in the Port Colborne CBRA. This form of QA/QC was applied in two ways:

When sampling was undertaken, the Independent Consultant would attend the sampling, observe the activities, and make note of deviations or modifications to the way samples were taken.

Duplicate QA/QC samples would be taken by the Independent Consultant at the same time that the JW took their samples. These QA/QC samples represented up to 20% of the samples taken for the ERA-NE, depending on the type and availability of samples. The Independent Consultant sent these samples to the laboratory under a confidential identification system. The results of testing by the laboratory could then be used to determine the precision of the sampling and analytical processes, so as to provide additional control over the quality of the sample taking and analysis.

The results of the Duplicate QA/QC Sampling and Testing are reported in Section 4.5 of this report.



4.0 ANALYSIS OF SAMPLES

4.1. SELECTION OF ANALYTICAL LABORATORY

For the Port Colborne CBRA ERA-NE analytical work, JW and the Independent Consultant agreed on the selection criteria for a properly qualified laboratory. The technical requirements and qualifications for the laboratory included:

The need for Quality Assurance Manual, available to laboratory clients;

An Independent Quality Assurance Officer responsible for the laboratory QA program;

Calibration of analytical instrumentation (five point weekly, verification daily)and documentation as requested;

Proper certification and accreditation by Standards Council of Canada, Canadian Association for Environmental Analytical Laboratories, or equivalents;

Formal audits of methods;

Consistent development and application of protocols;

A Training and Certification Program;

Corrective action reports (CARs) as required; and

Data archives with ability to retrieve past data (up to three years).

JW and the Independent Consultant agreed on Philip Analytical Services (PAS) as the laboratory for the ERA testing. Besides meeting the above qualifications, PAS had been the laboratory of choice for both JW and the Independent Consultant for several years before the CBRA was underway and it was regarded as having an excellent track record for service and accurate, defensible analytical results.



4.2. SAMPLING DATA BASE

Several thousand samples were taken for the CBRA, and most were analyzed for a suite of about twenty to thirty metals, which means that the total number of data points is probably around 100,000. Accompanying these data would be several thousand QA/QC data points from replicate tests and samples, blanks, SRMs, spikes, and so on.

The database for the Port Colborne CBRA ERA-NE was produced from 700 samples taken by JW and analyzed by PAS. A portion of some of the ERA samples was split with the Independent Consultant, who had them independently analyzed by PAS under separate sample codes. Together, these are the data that was primarily assessed to determine the overall acceptability of data used in the CBRA.

The data generated by the analytical laboratory has been examined to assess the overall QA and degree of reliability of the large database as it relates specifically to the analysis of the samples taken by JW.

4.3. QA/QC ASSESSMENT OF ERA-NE LABORATORY DATA

A full review of the laboratory data for the ERA was carried out by the Independent Consultant in March 2003, and is appended (Appendix A: “The Independent Consultant’s Review of Quality Assurance and Quality Control for the Port Colborne CBRA Ecological Risk Assessment – Natural Environment”).

The review was based on the examination of the analytical laboratory Certificates of Analysis and Certificates of Quality Control. The report “Quality Assurance/Quality Control and Laboratory Analytical Data”, Vol. V of the ERA-NE Report (September 2004) was also reviewed. The specific QA/QC indicators that were assessed were:

SRM data;

Matrix spike data;

Spike data; and

Replicate testing data.



The ERA-NE Report includes a review of the QA/QC activities carried out by the analytical laboratory (PAS) and of the performance of the laboratory in determining the concentrations of CoCs and other metals in the ERA-NE samples. The report also restates the 30% data acceptability limit for biomaterials and adds a 70% limit for soil and surface water. As previously noted, many of the combined study site data showed variances greater than 70% and almost all of the combined data sets had variance greater than 30%.

The ERA-NE assessment of the laboratory data makes it clear that ERA data variability is not due to the laboratory; leaving the only other source of variability as the manner in which the sample collection were carried out.

The Independent Consultant’s review of the laboratory data generally agrees with JW’s review and our conclusion is that “The QA/QC for the laboratory analysis of the Port Colborne ERA samples is extensive and of high quality. Considerable QA/QC testing was performed and the results of the testing indicated that the data base is generally precise, accurate and reliable”.

It should be noted that, despite the above conclusion, it does not mean that the data set is entirely free from problems. It means that the laboratory data had adequate QA/QC and is of acceptable quality. Up to the point where the samples have been collected and analyzed at the laboratory, there are only two possible sources of error in the ERA database: field sampling and laboratory analysis.

4.4. INDEPENDENT AUDITS OF ERA SAMPLING AND ANALYTICAL DATA

The Independent Consultant undertook independent audits of JW’s field surveys, including sampling surveys. There were basically two types of sample surveys:

To identify the distribution and abundance of certain VECs such as amphibians and birds; and

To collect samples so as to be able to perform analytical chemical testing to determine the concentrations of the CoCs in the tissues of various VECs and other biota.

The Independent Consultant provided review of both of the above types of surveys.



4.4.1 Species Distribution and Abundance Surveys

The Independent Consultant accompanied Inco’s Consultant on a number of surveys whose purpose was to evaluate the distribution and the abundance of various VECs. Amphibians and birds were evaluated in such a fashion.

The Independent Consultant concluded that, while some aspects of the amphibian survey were acceptable, some sampling points were not close to breeding sites and no sampling was carried out in wetland areas outside the study area. This would have provided a comparison set of data that could remove confounding effects of weather and other seasonal anomalies.

The bird survey was conducted in a satisfactory manner and the data derived from the field work is acceptable to establish the general avifauna of diurnal species, but not nocturnal or crepuscular species in the Study Area.

4.4.2 Independent Duplicate QA/QC Sampling and Testing of ERA Samples

The Independent Consultant recognized the value of having a system in place to ensure that duplicate samples were taken by the Independent Consultant, of samples being taken by Inco’s consultant.

The PLC recognized this as well and, at the May 3, 2001 PLC meeting, it specified that 20% to 50% of the ERA samples taken by JW be also taken by the Independent Consultant as a QA/QC measure. The PLC further specified that a portion of the duplicate samples could be submitted to the laboratory for testing, and that the results would be compared with those obtained on the samples taken by Inco’s consultant. The issue of sampling without proper, agreed-upon protocols has been previously discussed and it continued to be a shortcoming throughout the period in which the ERA samples were taken. Problems with incomplete or unavailable sampling protocols were noted at various PLC meetings in 2001 and 2002.

In any case, sampling proceeded without the benefit of proper protocols in 2001, and the Independent Consultant accompanied JW on most ERA sampling events and collected 20-50% duplicate samples, where possible.

4.5. RESULTS

For many types of environmental samples, it is not reasonable to expect a high level of agreement between duplicate samples. In fact, experience in testing metals in soil indicates that duplicate soil samples often show differences of 2-fold or even 3-fold, depending on the nature



of the soil. As more replicate samples are taken and analyzed, the mean value becomes increasingly useful, but when only two samples are taken to represent a given soil quality (as was the case for the duplicate QA/QC program), the analytical results can vary considerably. This fact is primarily due to the relative non-homogeneity of soil, in addition to other environmental factors.

Duplicate ERA QA/QC sampling was planned for several environmental media and VECs deemed to be important to the ERA, as follows:

Maple tree sap, leaves and soil;

Game (rabbit, deer) and fish;

Soil;

Amphibians (frogs and tadpoles);

Earthworms and surrounding soils;

Voles; and

Insect, arthropods.

It should also be noted that duplicate samples were also taken of crops (corn), private well water, and local food basket, garden produce, and supermarket food basket materials. The testing results for these samples were not used in the ERA.

Of the environmental sample media listed above, only the following provided adequate sampling to permit meaningful QA/QC assessments:

Maple tree sap, leaves and soil;

Soil;

Earthworm and surrounding soil; and

Amphibians (frogs and tadpoles).



The other environmental samples were taken by JW, along with duplicate samples as available, but there were insufficient data to permit an assessment of the data quality based on independent sample collection and analysis, for fish, game, and voles. In the case of insects/arthropods, adequate numbers of samples were taken (at least for plentiful insects such as grasshoppers and tent caterpillars) but problems with homogenization of samples (due to non-homogenization of exoskeletons, legs, antennae and other body parts) precluded meaningful “duplicate” testing.

Review of the analytical results for the duplicate samples of fish, game, voles, insects, and arthropods indicated that the data had such an extreme degree of variability that it cannot be considered as meaningful for use as an indicator of receptor exposure to the COCs, or even for comparison to other types of samples. The degree of uncertainty for these data is probably due to the same reasons above; insufficient numbers of samples and inadequate homogenization.

4.6. DISCUSSION

4.6.1 Maple Sap

No JW or Independent Consultant samples had arsenic (As) concentrations above the laboratory EQL of 2 μg/L (ppb). While this precludes a statistical assessment of the two data sets, it does indicate agreement that arsenic levels are too low to be detected.

Acceptable agreement was observed between the JW and the Independent Consultant samples for nickel, copper (Cu) and cobalt (Co).

Regression analysis of the two data sets for Ni, Cu and Co (JW and the Independent Consultant) showed no statistically significant differences.

Copies of the QA/QC analytical data for the maple sap samples are provided in Appendix B.

4.6.2 Maple Tree Soil

For As, Ni, Cu and Co, the results indicate the non-homogenous nature of soil. Plots of JW data vs. the Independent Consultant data for the samples indicate greater variability at concentrations closer to the EQLs, as expected. Regression analysis indicates small but significant differences between the JW and the Independent Consultant data sets, likely due to the influence of one or two data points with poor agreement, out of a total of ten duplicate samples. This variability is not unexpected for soil samples.



Copies of the QA/QC analytical data for the soil samples collected near the maple tree are provided in Appendix C.

4.6.3 Soft Maple Leaf

Arsenic concentrations were all below the EQL for the JW and the Independent Consultant replicate QA/QC samples. While this doesn’t allow for a statistical evaluation of the agreement between the two data sets, it does indicate agreement that As levels are not likely a concern.

For Co, Cu and Ni, the results indicated good agreement in 80 to 90% of the duplicate results. The regression analysis shows no significant difference from a 1:1 ratio for Co and Ni, and for Cu there was a minor difference that would not be of concern based on the end use of the data.

Copies of the QA/QC analytical data for the soft maple leaf samples are provided in Appendix D.

4.6.4 Soil

The JW and the Independent Consultant’s As results agreed well, with 10 out of 12 agreeing within ±40%. The JW vs. the Independent Consultant linear plot was not significantly different from a 1:1 ratio.

For Co, JW and the Independent Consultant’s results were with ±25% on all QA samples.

Cu and Ni results showed greater variability, but even for these two CoCs, the agreement was within ±30% in most cases (10 out of 12 for Cu, 8 out of 12 for Ni).

For each of As, Co, Cu and Ni the JW vs. the Independent Consultant linear data plot was not statistically different from a 1:1 ratio.

These results indicate good agreement between the two QA soil duplicate sets.

4.6.5 Earthworms

Arsenic data shows reasonable agreement between JW and the Independent Consultant duplicate QA sample sets (8 of 12 samples within ±30% of another, with a mean percent difference of 4.8%).

Co and Cu results show even better agreement, with 10 out of 12 Cu and Co results within 30% for the JW and the Independent Consultant data sets.



Ni results indicate greater variability, especially at the lower concentration levels.

The linear QA plots were acceptable for As, Co, Cu and Ni. Regression analysis of the slopes of the linear plots of As, Cu and Ni were not significantly different from the 1:1 trend line.

The trend line for Co was different from the 1:1 trend line, but only marginally and not different enough to have an effect on the end use of the data.

Copies of the QA/QC analytical data for the earthworm tissue samples are provided in Appendix E.

4.6.6 Earthworm Soil

Arsenic data shows reasonable linearity and acceptable percent difference between the JW and the Independent Consultant results.

Co and Cu data indicates poor agreement for three of the seven samples tested as duplicate samples by JW and the Independent Consultant. The three samples with poor agreement were those with the highest Co levels, which is unexpected. The linear plots are not close to a 1:1 ratio and are significantly different.

For Ni, the results show serious disagreement between JW and the Independent Consultant on one of eight samples. This is considered serious because it is a sample with an elevated Ni level. The linear plot appears to be acceptable but cannot be confirmed due to the loss of the electronic data set.

Overall, the earthworm soil results indicate poor agreement between the JW and the Independent Consultant data sets.

Copies of the QA/QC analytical data for the soil samples collected near the sampling location for the earthworms are provided in Appendix F.

4.6.7 Frogs

Some sites within the CBRA boundaries had green frogs, others had leopard frogs, and one had both species. Frogs and tadpoles were collected at one site (M-1) which JW counted as a Secondary (Medium) site, but this site is actually located in a Primary zone within the CBRA boundaries.



Data from Primary and Secondary sites were blended which had the effect of creating mean values with such large variances as to be environmentally meaningless. For example, the mean values for Ni (as well as the other CoCs, generally) in frog GI tracts and carcasses are exceeded by their standard deviations. In other words, the standard deviation is greater than the average value.

Statistically speaking, about 68% of the measurements will be between the mean minus the standard deviation and the mean plus the standard deviation. Scientific studies generally require a greater degree of certainty than 68%, and usually 95% certainty is considered acceptable. Ninety-five percent of the measurements will fall between the mean plus or minus two standard deviations.

Based on the existing ERA data, one can be 95% confident that the actual Ni content of frog gastrointestinal tracts is 21 mg/kg ± 2 x 26.87, which means the Ni content is anywhere from 32.4 mg/kg to 74.4 mg/kg. As it is impossible for a sample to contain less than 0 mg/kg of anything, the Ni concentration is anywhere from 0 to 74.4 mg/kg. Use of the mean value in such an example is scientifically meaningless, and is not capable of supporting valid conclusions.

Unacceptable large deviations exist for the entire frog data set, for each of the four CoCs. Frogs were analyzed as three separate body parts: gastrointestinal (GI) tracts, livers, and the remaining carcasses. These individual body parts were excised, frozen for up to 18 months, analyzed, and then the results were added together to create a calculated value for each of the CoCs for each frog. The reasons for doing this instead of simply analyzing each frog as a whole entity are unclear. This approach may have introduced a bias into the frog data set as was observed for tadpoles. Data based on tadpole GI tract plus carcass was found to be significantly and consistently lower for each CoC compared with the duplicate results presented by the Independent Consultant. As no whole frogs were analyzed, it cannot be confirmed that the approach of analyzing separate body parts biased the frog data. However, given the tadpole data, which showed consistently higher values for each CoC, for each duplicate QA/QC sample for whole tadpole samples compared to the “adding up” approach, an uncertainty is introduced for the frog data set.

Copies of the QA/QC analytical data for the frog carcass, liver and GI Tract samples are provided in Appendix G.



4.6.8 Tadpoles

JW’s data was developed by combining the separate results of tadpole GI tract and tadpole carcass analyses, while the Independent Consultant’s duplicate testing was performed directly on whole tadpoles.

JW’s tadpole data is significantly and consistently lower than data obtained by the Independent Consultant, for each CoC (see Table 2).

Copies of the QA/QC analytical data for the tadpole carcass, liver and GI Tract samples are provided in Appendix H.



Table 2: Comparison of JW and the Independent Consultant Tadpole QA/QC Duplicate Sample Testing

Arsenic

Sample JW(mg/kg) Ind. Con. (mg/kg) JW/ Ind. Con.

H-1-T 4.6 5.5 84%

H-2-T 2.2 3.4 65%

M-1-T 3.5 4.2 83%

M-3-T 8.2 9.8 84%

C-1-T 8.2 11.8 69%

Mean 73.0%

Cobalt

Sample JW (mg/kg) Ind. Con. (mg/kg) JW/ Ind. Con.

H-1-T 6.0 6.5 92%

H-2-T 2.2 3.62 61%

M-1-T 10.0 13.4 75%

M-3-T 19.0 24.4 77%

C-1-T 7.6 10.1 76%

Mean 76.2%



Copper


H-1-T 42.4 59.6 71%

H-2-T 22.0 33.7 65%

M-1-T 60.1 82.5 73%

M-3-T 53.0 70.6 75%

C-1-T 42.3 58.0 73%

Mean 71.4%

Nickel


H-1-T 119 166 72%

H-2-T 24 49 49%

M-1-T 97 129 75%

M-3-T 111 140 79%

C-1-T 30 47 64%

Mean 67.8%

It should also be noted that the sample designated M-1-T (the “M” means “medium” or “secondary” contamination zone in the CBRA mapping system) is incorrectly placed in a secondary study zone. These samples were located in a Primary Study Area (a “high” contamination zone).

The results in Table 2 indicate that CoC concentrations in JW’s tadpole data set are 25-32% lower than the Independent Consultant’s QA/QC control data set for tadpoles.



5.0 ERA DATA QUALITY

5.1 INTRODUCTION

Preceding sections of this report have covered the need for careful planning, adequate sampling, and proper analysis of samples. As well, information was provided regarding some shortcomings, especially in the planning and sampling components. This section examines the effects of some of the shortcomings on the overall ERA database.

As noted previously, proper planning is essential to effective sampling, and effective sampling leads to sound data, which in turn provides a basis for scientifically sound conclusions to be drawn. While it may be possible to obtain accurate analytical results from improperly planned and ineffectively executed sampling, the results will likely not be capable of supporting meaningful conclusions. According to Standard Methods (i.e., Standard Methods for the Examination of Water and Wastewater, 2005), “It is axiomatic that the result of any testing can be no better than the sample on which it is performed”. Each link in the chain of activities from planning, to sampling, to analysis, to assessment, to conclusion-drawing could be similarly described, as the reliability of each successive component is dependent upon the reliability of its precedent.

5.2 STUDY AREA SIZE AND ERA-NE SAMPLES

The diversity of the environmental compartments sampled and the extensive parameter list for which the ERA-NE samples were tested make the Port Colborne CBRA one of the larger environmental studies undertaken in Canada. Environmental databases include various soil types, water, sediments, and various species identified as Valued Ecological Components (VECs), including lifecycle elements for amphibians (tadpole, various sizes of adult frogs) and food chain components for some VECs. Over 700 samples were taken for analysis by multi-elemental analysis. Twenty to thirty elements were determined, but only the four CoCs (As, Cu, Co and Ni) are discussed in the ERA-NE report. Besides the sampling and analytical component, limited species inventories and assessments of woodlot health were carried out.

The geographic area covered by the CBRA is also large. The study area consisted of the Primary Study Area (105 hectares), the Secondary Study Area (462 hectares) and a Reference Area, consisting of woodlands, wetlands, and conservation areas within the general vicinity. In fact, such a large area could not be properly evaluated without taking a very large number of samples for each environmental compartment of interest.



The original drafts of the ERA-NE Report followed the original study plan as laid out in the TSOW (November 30, 2000) and provided the CoC data for the Primary Study Area, the Secondary Study Area, and the Reference Area. However, the final report (2004) combines the Primary and Secondary data into one “Study Area” (which excluded residential areas, the Inco Refinery property and a large quarry northeast of the Inco property). The size of the Study Area is 22 km2, according to the ERA-NE, September 2004, page 1-17. This is a large area. Seven-hundred samples, including trees, leaves and other vegetation, voles, tadpoles, frogs, earthworms and so on, is actually very sparse coverage for such a large land mass, relating to one sample for every 31,000 m2 for the 22 km2 of the Study Area.

The 700 samples were not distributed evenly throughout the Study Area, which resulted in some areas within the Study Area having more extensive sampling coverage. In addition, there were many large areas in the Study Area that had no sampling coverage.

Review of the data for sampling stations in the Study and Reference Areas indicates that, given the large variability in tissue levels for given locations, there may not have been adequate samples taken. For example, examination of the specific points within the Reference Secondary and Primary Areas indicates that, at the sampling locations where amphibian samples were collected, five samples of frog were taken (or in the case of tadpoles, one or two composite samples). These low numbers of samples per station raise questions, such as:

Are two samples or less for a given type of sample (in this case, for a tadpole) from a given location sufficient? Is it worth submitting for analysis?; and

Does the data, often having a more than ten-fold variability, provide meaningful insights into the potential risks posed by elevated metal loadings to amphibians?

For example, the five Ni results at one control site for five samples of frogs are 1.8, 3.9, 8.6, 12.7 and 47.8 μg/g. The mean of these five values is 15.0 μg/g. This is higher than the mean concentrations for three of five secondary sites and two of five primary sites. Should the data be adjusted to account for outliers? Is simple “averaging” an appropriate data approach?

Examples like the above are common throughout the database, and they indicate that such small numbers of samples at individual stations may not be capable of providing meaningful ERA data.



It is possible that concern over such large data variations formed part of the decision to combine the Secondary and Primary databases into one “average” Study Area database. While the objective of this approach may have been to create larger databases for each environmental component, much greater data variability accompanied the “average” results thereby obtained. This is due to the reality of the CoC concentrations in the Primary and Secondary Area sample data sets, where the CoC concentrations are generally higher in the Primary Area (for frogs, leaves, etc.) and lower in the Secondary Area (see ERA-NE Report, September 2000, Tables 6.3-6.13). When high numbers are mixed with low numbers, the calculated average value is more likely to have a large degree of variability.

In the end, the geographical size of the Study Area appears to have overcome the size of the database, since the combined database for many sample types provides numerous examples of standard deviations that are larger than the mean for the same data set.

5.3. ERA-NE DATA: ISSUES AROUND SINGLE POINT-IN-TIME SAMPLING

The biological samples collected by JW in 2001 for the purpose of determining potential risk due to CoC contamination covered one point in time, resulting in these samples to be the equivalent of “grab” samples, which are defined by Standard Methods (2005) as, “single samples collected at a specific spot at a site over a short period of time …”. The voles captured on a given day in September 2001 or the tadpoles collected on a day in June, 2001 can only represent the conditions for that time, at that location. Seasonal, feeding, mating, spawning, or other periodic activities may influence the concentration or the location of metals and other chemicals within an organism (Keith, 1991). As well, the levels of the CoCs or other contaminants that voles, tadpoles and other organisms may be exposed to are subject to change over time. The rates of processes that determine chemical transport, bioavailability, accumulation and depuration may change over time.

There are several benefits to sampling biological tissues more than once. A practical benefit is that data uncertainties that were discovered during the first sampling program can be clarified or corroborated. Unusual findings can be confirmed or challenged. Critical findings for earlier sampling can be used to guide more intensive or extensive sampling, or to minimize sampling in other areas. VECs or other species whose data yielded unacceptable data (high variability, or levels below method detection limits, for example) could be replaced by other VECs/species that might provide more useful information.



Other benefits of repeated or cyclical sampling include the possibility of determining the effects of seasonal, mating, eating and other periodic biological activities on the level of CoC in biota tissue(s). Bartell et al. (1992) indicate that for between-year variations, and biological cycles that exceed one year, longer-term monitoring is essential in order to even detect that a cyclical variation may be occurring. This can be important because “The cycle itself might be a sensitive indicator of changes in system function in response to chemical exposure” (Bartell et al., 1992).

5.4 ERA-NE SURVEY ISSUES

There are concerns regarding the ERA-NE database based on survey data for amphibians and birds that were identified by biologists from the Independent Consultant who accompanied the surveys:

The amphibian and bird surveys did not include a population or richness component;

The amphibian survey did provide an indication of the distribution and abundance of anurans in the Study Area;

Various deviations from the survey protocol were noted by the Independent Consultant personnel;

No amphibian surveys were carried out on wetland areas outside of the Study Area. Thus there is no opportunity to compare potentially affected sites in the Study Area to rural areas outside of the Study Area, or even in the Reference Area; and

Frog, toad and bird populations vary over time for a large number of reasons, including exposure to various chemicals. Because of this variation in populations, the surveys conducted on April 23, May 7-8 and May 17-18, 2001 amount to a “one-time” survey, and do not provide information relevant to a timeframe other than a three-week period in the spring of 2001. Definitive conclusions regarding the long term condition of amphibian populations or the possible effects of CoCs or other variables on the populations cannot be drawn from a one-time survey.



5.5 ERA-NE DATA VARIABILITY AND LACK OF DQOS

Because of the high degree of variability in the data, the level of uncertainty caused by the variability for many components of the ERA-NE is so high that meaningful conclusions cannot be reached. Table 3 provides some examples of data variability covering a range of sample matrices, based on Tables 6.3, 6.4, 6.5, 6.7, 6.12 and 6.13 in the ERA-NE September 2004 Report.

The data in Table 3 provide many examples where the variability (measured as the standard deviation of measurements for each CoC and various sample types) is greater than the mean result for each CoC and each sample type with a relative standard deviation (RSD) often greater than 100%.

Table 3: Variability of CoC Measurements in Organisms Sampled in the Study Area

Sample Type

Combined Study Area (Primary Plus Secondary)

CoC (mg/kg)

Mean (mg/kg)

Std. Dev. (mg/kg)

RSD (%)

Tadpole:

GI Tract Ni 192 98 51

Co 18 11 61

Carcass Ni 30 38 126

Co 3.0 3.0 100

Whole Body Ni 85 53 62

Co 8.6 6.3 73

Frog:

GI Tract Ni 21 27 129

Co 1.7 2.2 129

Liver Ni 0.5 0.5 100

Co 0.67 0.39 58

Carcass Ni 0.8 1.0 125

Co 0.14 0.08 57

Whole Body Ni 3.0 3.6 120

Co 0.33 0.29 97



Table 3: Variability of CoC Measurements in Organisms Sampled in the Study Area (Continued)

Sample Type

Combined Study Area (Primary Plus Secondary)

CoC (mg/kg)

Mean (mg/kg)

Std. Dev. (mg/kg)

RSD (%)

Maple Tissue:

Leaves Ni 10 6.8 68

Co 0.29 0.25 86

Sap Ni 0.05 0.08 160

Co 0.0017 0.0015 88

Cotyledons Ni 10 0.4 4.0

Co 0.23 0.01 4.3

Earthworms (whole): Ni 305 263 86

Co 13.4 8.95 67

Arthropods (total): Ni 9.3 7.2 74

Co 0.33 0.31 94

Voles:

Liver Ni 0.4 0.3 75

Co 1.13 0.63 56

Carcass Ni 14.8 7.8 53

Co 0.98 0.52 53

Total Body Ni 14.2 7.4 52

Co 0.98 0.51 56

RSD = Relative Standard Deviation, calculated as , expressed as a

percentage.

The variability information in Table 3 was taken from the tables in Chapter 6 of the ERA-NE Report (2004) and includes the entire Study Area data sets for each environmental compartment (frog, voles, etc.), that is, it is based on JW’s approach of combining all of the Primary Study Area CoC data for each sample type with all of the Secondary Study Area CoC data for the same sample types. The Independent Consultant does not agree with this approach, as it results in “averaging” the results for organisms sampled at locations across the 2,200 hectare study area,



instead of performing separate risk characterizations of potential effects of CoCs for plant and animal populations from the more heavily contaminated areas within the Port Colborne Natural Environment areas. Averaging the high and low data across the entire Study Area will obscure any trends that might have been observed in the data from various sample locations within the large Study Area, as seen in the Table 3 Relative Standard Deviation (RSD) data.

It would have been very interesting to see the data presented for each of the separate study areas in order to see if there is a significant difference and whether there is, in fact, an indication of a problem in the high zone.

In the absence of DQOs specifying the acceptable variation for data relating to each of the sample types that make up the ERA risk calculations, it is not possible to quantitatively assess the very high variability in the CoC database against a specific ERA yardstick. However, these large variabilities can be compared against some of the usual expectations in environmental sampling, and to some experience-based considerations.

For example, DQOs for various U.S. EPA project measurements, expressed as “Quality Assurance Objectives” include values of 100 ±20%, 100 ±25% and 100 ±30% for various chemical measurements in landfill bioreactors (J.T. Markarese et al., Annual Conference on Managing Environmental Quality Systems, 2001). Field duplicates from single study results indicate variability around ±33 to ±56% for mercury in tributaries (median = 4.5 ng/L) and open lakes (median = 0.3 ng/L), and ±40% for orthophosphate in tributaries (median = 8.6 ng/L) (L. Blume, et al., Annual Conference on Managing Environmental Quality Systems, 2001).

Generally, based on broad experience, environmental sample analytical data demonstrates typical data variability in the range of 20-50%, with the lower range for water and a mid-higher range for biological materials and soil.

When variability is greater than 50%, it generally indicates problems with insufficient samples or with sampling/analytical bias. Given the QA/QC duplicate sample program during the sampling and analysis phase of the ERA-NE, there is no indication of laboratory analytical bias (although certain individual data sets might have had a low sample handling bias, as seen in Section 5.0).



Tables 6.3-6.13 in the ERA-NA Report (2004) and Table 3 in this report demonstrate that the CoC database has very large variability. In several instances (tadpoles, frogs, and maple sap) the variability is more than 100% of the mean value. In all other cases but one, the variability is more than 50%. The maple cotyledon results provide an exception; the results are remarkably consistent.

The ERA-NE CoC database has a number of problems, mainly related to the large variability of the database for CoCs in individual sample types (see Table 3).

In the planning of the ERA-NE, considerable CoC data was in place, primarily from MOE studies and other work done from the 1970s to the 1990s. These data provide an indication of the variability of data sets for various sample types, and should have indicated the need for thorough planning to ensure that enough samples were taken to yield robust, reliable final results. As it stands, the ERA-NE database yields more questions than answers.

5.6 THE EFFECT OF “AVERAGING” ON ERA-NE DATA QUALITY

The data presented in Table 3 was derived from the 2004 ERA-NE Report, and they are based on average (mean) values for environmental samples from the entire Study Area. Earlier versions of the ERA-NE report (JW Draft CBRA ERA – Natural Environment, January 2003) provided tables with some of the same data sets averaged in the 2004 ERA-NE report, but separated into Primary, Secondary and Reference Areas.

Table 4 provides the means, standard deviations and relative standard deviations for the same ERA samples given in Table 3, but broken down into samples taken in the Primary Study Area and in the Secondary Study Area. It is instructive to compare the degree of variability between the two approaches.

RSDs greater than 50% indicate a large degree of variability in a data set while an RSD of 100% indicates a very large degree of variability in a measurement.

This indicates that the original approach of comparing ERA sample data from highly contaminated areas to data from secondarily contaminated areas yields less data variability (although the data in each set is still highly variable) and that averaging over the entire Study Area tends to increase the overall data variability (higher RSDs) and obscures valuable information on local trends in the data.



Table 4: Variability of CoC Measurements in Organisms Sampled in the Study Area (mg/kg except as stated otherwise)

Sample Type

Combined Study Area Primary Study Area Secondary Study Area

CoC Mean Std Dev RSD (%) CoC Mean Std Dev RSD

(%) CoC Mean Std Dev RSD (%)

Tadpole:

GI Tract Ni 192 98 51 Ni 219 104 47 Ni 166 106 64

Co 18 11 61 Co 13.8 3.94 29 Co 24.4 14.8 61

Carcass Ni 30 38 126 Ni 40.4 55.3 137 Ni 18.9 15.8 84

Co 3.0 3.0 100 Co 3.3 4.3 129 Co 2.8 1.9 67

Whole Body Ni 85 53 62 Ni NA NA - Ni NA NA -

Co 8.6 6.3 73 Co NA NA - Co NA NA -

Frog:

GI Tract Ni 21 27 129 Ni 26.9 30.2 112 Ni 14.8 21.5 145

Co 1.7 2.2 129 Co 1.48 1.82 123 Co 1.98 2.46 124

Liver Ni 0.5 0.5 100 Ni 0.5 0.37 74 Ni 0.49 0.43 88

Co 0.67 0.39 58 Co 0.7 0.35 50 Co 0.63 0.44 70

Carcass Ni 0.8 1.0 125 Ni 1.08 1.3 120 Ni 0.48 0.5 110

Co 0.14 0.08 57 Co 0.15 0.07 47 Co 0.13 0.09 69

Whole Body Ni 3.0 3.6 120 Ni 3.7 3.94 107 Ni NA NA -

Co 0.33 0.29 97 Co 0.33 0.24 73 Co NA NA -



Table 4: Variability of CoC Measurements in Organisms Sampled in the Study Area (mg/kg except as stated otherwise) (Continued)

Sample Type




Maple Tissue:

Leaves Ni 10 6.8 68 Ni 10.2 6.3 62 Ni NA NA -

Co 0.29 0.25 86 Co 0.31 0.21 68 Co NA NA -

Sap Ni 0.05 0.08 160 Ni 0.06 0.12 200 Ni NA NA -

Co 0.0017 0.0015 88 Co 0.01 0.01 100 Co NA NA -

Cotyledons Ni 10 0.4 4.0 Ni 10.1 0.4 4.0 Ni NA NA -

Co 0.23 0.01 4.3 Co 0.23 0.01 4.3 Co NA NA -

Earthworms (whole): Ni 305 263 86 Ni 380 141 37 Ni 132 89.1 68

Co 13.4 8.95 67 Co 18.1 10 55 Co 10.6 3.43 32

Arthropods (total): Ni 9.3 7.2 74 Ni Ni

Co 0.33 0.31 94 Co Co



Table 4: Variability of CoC Measurements in Organisms Sampled in the Study Area (mg/kg except as stated otherwise) (Continued)

Sample Type




Voles:

Liver Ni 0.4 0.3 75 Ni 0.44 0.28 64 Ni * *

Co 1.13 0.63 56 Co 1.21 0.59 49 Co * *

Carcass Ni 14.8 7.8 53 Ni 16 7.12 44 Ni * *

Co 0.98 0.52 53 Co 1.05 0.47 45 Co * *

Total Body Ni 14.2 7.4 52 Ni NA NA - Ni * *

Co 0.98 0.51 56 Co NA NA - Co * *

RSD = Reactive Standard Deviation

NA = data not available

* = only one sample taken, insufficient data



Local ERA sample site data sets are available for examination of CoC concentrations at the actual sample location level. Several data subsets were briefly examined and they show some interesting features:

Surface water CoC concentrations frequently exceed Ontario’s Provincial Water Quality Objectives (PWQOs), especially surface waters taken from ponds in the Primary Area (7 out of 11 samples fail to meet one or more PWQO for Co, Cu, Ni). This means that most of the sampled ponds failed to meet MOE water quality objectives, which are set at a level of water quality which is protective of all forms of aquatic life;

The average surface water Ni concentration for ponds in the Primary Area exceeds the Ni PWQO by more than 6 times;

One or more CoC concentrations in pond sediment exceed the relevant Sediment Lowest Effect Level (LEL) (MOE, 1994);

Every sample taken from Primary Area ponds exceeded the Severe Effect Level (SEL) for Ni, and two of them were about 14 times as high as the SEL;

One-third of the sediments sampled in ponds from the Primary Area exceeded the CuSEL; and

[Note: The SEL is defined by MOE as “the level at which pronounced disturbance of the sediment dwelling community can be expected. This is the sediment concentration of a compound that would be detrimental to the majority of benthic species” (MOE, June 1992, p. 2).]

Spiders collected from sites in the Primary Area had higher average levels of Ni, Co and Cu than those from the control sites. Ni concentrations were almost six times higher in the Primary Area spiders. The spider data inexplicably has been blended with the data for other arthropods, thus obscuring any ability to detect trends between CoC levels for predatory or vegetal arthropods, or between sample sites.

The above are only a few examples of information that does not emerge from the ERA Report, mainly because of the collapsing of local data sets within the Secondary and Primary Areas into one large “average” data set for the entire Study Area.



5.7 SAMPLING ISSUES AS A “LIMITATION”

Ecological risk assessments have an important feature that distinguishes them from simple environmental assessments, and that is the explicit, quantitative consideration of uncertainties in the analysis and the expression of the final estimated effect as a probability (S.M. Bartell et al., 1992). Oak Ridge National Laboratory has long been a leader in ERAs, and three of its leading researchers recognized the importance of accuracy and precision in developing the estimated effect, and stated, “The magnitude of the estimated effect if important; however, just as important are the accuracy and precision of the estimates”.

Experience has shown that environmental measurements tend to be less accurate or precise than for most other types of analyses. This imprecision and inaccuracy is primarily due to the natural variability of ecosystems and their functions.

In the ERA-NE Report (September, 2004), a number of uncertainties are listed respecting sample and data collection methods:

Short period of time for sample collection;

Difficulty collecting enough biomass to allow for adequate QA/QC such as duplicate testing;

Number of sites sampled was reduced due to “seasonality”; and

A high degree of data variability in the collected data due to the differing nature of the habitats that were sampled.

As stated in the Report, “this fact undoubtedly contributed to the high degree of variability within the collected data”. However, instead of incorporating these limitations into the risk characterization by attempting to identify the effect they would have on the accuracy and precision of the estimated impact of CoCs, the ERA-NE Report (2004) attempts to rationalize the limitations. The rationalizations occur in Table 10-4, which are not convincing. For example, after stating “many biological receptors could only be collected for a short period of time due to seasonality, thus affecting the number of sites that could be sampled”, the shortcoming is rationalized as follows: “However, the data collection methods satisfy the need for obtaining adequate samples from a certain (although sometimes limited) number of sites, over a specified period of time”. This rationalization is scientifically illogical.



The foregoing demonstrates that there was awareness on the part of the investigators of the problem of “a high degree of variability in the collected data”, and of the limitations of a one-time sample collection that limited the number of sites to be sampled. The expected response, when conducting a scientific study, when sample data is incomplete as well as highly variable, is to take more samples. It is likely that there were time pressures (and possibly financial constraints) that might have made additional sample collection difficult. The choices at that time were either to try to develop an ERA based on an incomplete, highly variable database or to fill in the data gaps and reduce the data variability by taking more samples. As no additional samples were taken, the ERA-NE database has a high level of uncertainty and is too variable to be able to draw from it scientifically meaningful conclusions.

5.8 CONCLUSIONS: ERA DATA QUALITY

The following conclusions are provided with regards to the ERA data quality:

Data Quality Objectives were not developed, so it was not known how many samples would be required to obtain reliable, precise CoC values for the various sample types;

Data averaging across the entire Study Area of 22 km2 contributed to data variability;

Seven hundred samples were taken for the ERA-NE, which is insufficient to properly characterize the large number of organisms, habitats and soil types in the study area, for example:

o Only three maple key samples from the Primary Area;

o Only three tadpoles from the Primary Area, three from the Secondary Area and two controls; and

o A total of 11 voles for the entire Study Area.

The low numbers of biological samples contributed to the very high variability in the database.

The CoC concentrations in frogs and tadpoles in the ERA-NE Report (2004) were determined by dissecting the animals, measuring the COCs in various body parts, then arithmetically adding up the various concentrations to obtain a CoC value for the entire animal. The Independent Consultant, for verification purposes, more simply measured the total CoC concentrations in each sample of tadpoles. The Independent Consultant’s direct CoC results were consistently and significantly higher (31% to



47% higher, depending on the CoC) than reported in the ERA-NE Report. Similar under-reporting may also have taken place for frogs and voles, where CoC levels were determined using dissection plus analysis of two or three individual body parts plus statistical data treatment. In any case, the tadpole data indicates that some of the data in the ERA-NE Report may be understated;

The arthropod data set is based on averaging the CoC in tissue values for spiders and various insects including grasshoppers, caterpillars, etc. Not surprisingly, the “plus or minus” values for the mean CoC concentrations are in the range of 70-90%. Differences in CoC concentrations between predatory and vegetarian arthropods were not possible to assess with this averaging approach; and

The sparse sampling for most of the VECs and other organisms, and the generally very large variability of the data renders it incapable of supporting scientifically meaningful conclusions.

The overall conclusion that can be drawn from examination of the ERA-NE database is that it has such sparse geographic coverage and such large variability that little confidence can be accorded to conclusions drawn from it.



6.0 REFERENCES

Bartell, S.M., Gardner, R.H., and O'Neill, R.V. 1992. Ecological Risk Estimation. Lewis Publishers, Chelsea, Michigan.

Jacques Whitford. TSOW (Technical Scope of Work) (November 2000)

Keith, L. Ed. Principles of Environmental Sampling: a Practical Guide; American Chemical Society: Washington, 1991. An Introduction to Environmental Sampling and Analysis.

Keith, L. H., In: Dynamic, Exposure And Hazard Assessment Of Toxic Chemicals, R Hague, Ed. Woburn, Ma, Ann Arbour Science Publishers, 1980, Pages 41-45.

Malcolm J.R. Clark (Ministry of Environment, Lands and Parks, Victoria, Canada), Quality Assurance in Environmental Analysis, Encyclopaedia of Analytical Chemistry 2000, John Wiley & Sons, Ltd.

Mesley, R.J., W.D. Pocklington and R.F. Walker (1991). Analytical Quality Assurance - A Review. In: The Analyst; October, Vol. 116, No. 10, Pages 975-990.

Ministry of Environment and Energy, “Guideline for the Protection and Management of Aquatic Sediment Quality in Ontario”, PIB 1962, June 1994.

Rose, K.A., and E.P. Smith. 1992. Experimental Design: "The Neglected Aspect of Environmental Monitoring". Environmental Management 16(6), 691-700.

APPENDIX A

The Independent Consultant’s Review of Quality Assurance and Quality

Control for the Port Colborne CBRA, Environmental Risk Assessment –

Natural Environment

Independent Consultants QAQC Review of the Natural Environment ERA Port Colborne CBRA Page A1


APPENDIX A

THE INDEPENDENT CONSULTANT’S MARCH 2003 REVIEW OF QUALITY ASSURANCE AND QUALITY CONTROL FOR THE PORT

COLBORNE CBRA, ENVIRONMENTAL RISK ASSESSMENT – NATURAL ENVIRONMENT

A-1 INTRODUCTION

This review is specifically for the analytical data report on biomaterials in Volume V “Ecological Risk Assessment, Natural Environment, Community Based Risk Assessment, Port Colborne, Ontario”, November 2002, prepared by Jacques Whitford Environmental Limited (JW), entitled “Laboratory and Analytical Data and Quality Assurance/Quality Control”.

This review follows a detailed review of the “Laboratory Protocol for Analysis of Biological Tissues”, Volume II “Field Data Collection and Analysis Protocols”, which constituted pages 1-53 of Volume II.

The laboratory protocols describe how the samples were prepared and analyzed in the laboratory; the laboratory data provides the analytical results themselves.

This review covers the QA/QC related to these analytical results, reported in Volume V.

A-2 SCOPE OF QA/QC DATA REVIEW

The review was carried out by examining the actual analytical data and comparing the QA/QC data to the appropriate parts of the protocols and procedures mentioned in Section 1.0 “Introduction”. Much of the QA/QC data was understandable and rationalized by this preliminary review; specific points were clarified by telephone calls with JW and the lab, Philip Analytical Services (PAS). The Independent Consultant review team met with JW on February 6, 2003 to discuss questions and concerns, and with PAS on February 25, 2003 to obtain information regarding some remaining technical concerns.

The conclusion reached after the above activities were carried out was that there was a good understanding and adequate information to complete the review of the QA/QC data for the ERA.



A-3 QA/QC INDICATORS

The usual indicators of laboratory QA/QC on a specific project are those discussed in the JWEL report “Laboratory Protocol for Analysis of Biological Tissues”, November 2002:

Standard Reference Materials (SRM);

Matrix Spikes;

Spikes; and

Replicate Testing.

In addition, a proficient laboratory will routinely carry out the QA/QC procedures outlined in the MOE’s 1996 guideline “MOE Guidance on Sampling and Analytical Methods for Use at Contaminated Sites in Ontario”, as follows:

Pre-run QC:

o Lab ware and reagent blanks;

o Instrument setup standard;

o Reference standard to validate in-house standards; and

o Instrument detection limits (IDLs) and detector linearity curves (minimum of 5-point calibration).

In-run QC:

o Baseline drift blanks;

o Standards; and

o Instrument checks.

Run QC:

o Method recovery blanks;



o Method blanks;

o In-house matrix check materials;

o Duplicates (minimum of one set per run of 30 samples). As mentioned in the MOE (1996) guidance document, a duplicate sample is defined as a second aliquot from the same sample container;

o Surrogates (added prior to organic extraction);

o Spiked samples, if applicable;

o Certified standard reference materials (SRMs) to validate method recovery; and

o Method Detection Limits (MDLs) for each parameter.

With respect to the procedures in the MOE guidance document, PAS routinely carry out all of the requirements as part of their ongoing internal QA/QC program, and they have the records to demonstrate this. Much of this information was reviewed with senior PAS staff and it was found that the laboratory is adhering to the MOE requirements as a minimum standard of performance. They also perform numerous other QA/QC tests to maintain their accreditation with agencies such as CAEAL, State of New York, and other U.S. and Canadian organizations.

The only aspects of the MOE guidance that were not routinely carried out were as follows:

5-point calibration for ICP-ES detector linearity curves, because the ICP-ES technique is linear over a very large analytical range and does not require 5-point calibration; and

Surrogates were not analyzed because they are only required for organic testing, and as all of the ERA testing was for inorganics, surrogate testing was not applicable.

The next three sections deal with the lab’s performance on the Port Colborne ERA testing and address:

SRMs;

Matrix Spikes;

Spikes; and



Replicate Testing.

A-4 STANDARD REFERENCE MATERIALS (SRMs)

SRMs are materials analyzed by a laboratory in conjunction with actual samples, in order to estimate analytical bias. These are homogenous materials whose composition is certified by a central agency, after the material has been analyzed by a large number of laboratories using a number of different analytical techniques.

In the case of the ERA, the laboratory (PAS) analyzed two SRMs whose make-up was similar to that of the subject samples. The two SRMs were bovine liver and oyster tissue, each of which is certified by the National Institute of Standards and Technology (NIST) to contain specific, published levels of 18 or more parameters including many metals of interest in the ERA study.

PAS performs SRM testing on all of its tissue testing samples, including and in addition to those reported in Volume V. The laboratory uses the data to maintain Control Charts for overall laboratory QA/QC monitoring. A review of their Control Charts for nickel, copper and lead indicated that the laboratory is performing consistent, accurate data on these samples.

With respect to the Port Colborne ERA testing, JW arranged for SRM testing in addition to that described above, for virtually every sample submission of tissue samples. The laboratory data compared to the SRM values were typically within 90% of the certified value, for nickel, arsenic, cobalt, copper, and lead, as well as for numerous other metals. In many cases the laboratory data is in 98% or better agreement with the certified values.

While the focus of this review was on QA/QC related to the analysis of biomaterials the QA/QC associated with analysis of related materials such as soil and/or water from certain areas where biological material had been taken, was also reviewed.

The review determined the following:

SRMs appear not to have been analyzed for soil and water samples. On the other hand, soil samples often had “certified” samples and “spikes” for QA/QC, and water samples were accompanied by spiked travel blanks, and the reported values for Ni, Co, Cu and Pb were within an acceptable range of the prepared spike values;

Vegetation samples appear to have scant SRM analyses. For several types of vegetation samples, no SRMs were analyzed, even though acceptable SRMs exist for



such purposes. The Port Colborne ERA sample types for which no SRMs were determined were:

o Corn kernels;

o Maple keys;

o Leaves; and

o Tree cores.

Several other groups of biomaterials had no SRMs for QA/QC purposes, including maple sap, maple syrup, spinach, leaves, grasshoppers and spiders; and

Worms had one set of SRMs; at least one set per batch submission would ideally have been reported, which would have resulted in three sets of SRMs.

A-5 SPIKES AND MATRIX SPIKES

PAS performed spikes and matrix spikes with every ERA sample set. Matrix spikes involve the addition of a known concentration of an analyte to a sample, so that subsequent analysis for the analyte will indicate bias related to sample preparation and analysis. A spiked sample involves the addition of a known amount of an analyte to a digested, prepared sample. Subsequent analysis of the “spiked” sample indicates bias due to the analytical procedure. Typically, analysis of spiked samples and of matrix spikes helps the analyst determine whether the sample itself, or the chemicals used to prepare it, either enhances, suppresses or otherwise interferes with the measurement of the analyte.

Review of the PAS “Certificate of Quality Control” shows that analysis of matrix and process spike samples yielded acceptable results for every parameter, including the chemicals of concern. The Certificates of Quality Control are located in Tabs 2-24 of Volume V, “Laboratory and Analytical Data and Quality Assurance and Quality Control”, November 2002.

A-6 REPLICATE ANALYSES

Replicate analysis of a sample means that two or more portions of the same sample are taken, prepared, and analyzed. Comparison of the results provides an estimate of the precision, or variability, of the measurements. In much environmental work, the acceptable limit of variability for biomaterials is ±50%. For soil testing the limits are often higher than ±50%.



In the case of the Port Colborne ERA study, the acceptable limit was tighter at ±30%. The ERA report, Volume V, “Laboratory and Analytical Data and Quality Assurance and Quality Control”, November 2002, provides a thorough listing of all replicate testing in Tables 1 through 47, pages 1 to 19.

The data shows the following:

Arsenic levels are too low to be assessed. Most data is “not detectable”, which indicates it is either absent or if present is below the detection limit of the analytical equipment;

For all tissue types with the exception of one earthworm duplicate test, the copper and cobalt values are well within the 30% limit, and many are within 5%; and

For nickel, there are three examples of the variability being higher than 30%, in the analysis of frog samples. These higher variabilities were likely caused by the presence of skin and bone matter which made complete homogenization of samples difficult. For all other tissues (voles, vole liver, insects, frog liver, etc.) the Ni variability was under the 30% limit and was often within 5%.

A-7 CONCLUSIONS

The QA/QC for the laboratory analysis of the Port Colborne ERA samples is extensive and of high quality Considerable QA/QC testing was performed and the results of the testing indicated that the data base is generally precise, accurate and reliable. Some specific conclusions:

The ERA database had significant QA/QC applied to it.

The QA/QC data generally demonstrates good agreement between replicate samples, little evidence of matrix interference, and acceptable recovery of matrix spikes.

Analysis of SRMs for the metals of concern shows that agreement between certified values and observed values is good to excellent.

SRMs do not exist for some sample types, and for these as well as for certain other samples, no SRMs were analyzed.

The laboratory confirmed that the ERA project followed established laboratory sample preparation and analytical protocols.

APPENDIX B

QA/QC ANALYTICAL DATA FOR MAPLE SAP

APPENDIX C

QA/QC ANALYTICAL DATA FOR MAPLE TREE SOIL

APPENDIX D

QA/QC ANALYTICAL DATA FOR SOFT MAPLE LEAVES

APPENDIX E

QA/QC ANALYTICAL DATA FOR EARTHWORM TISSUE

APPENDIX F

QA/QC ANALYTICAL DATA FOR EARTHWORM SOIL

APPENDIX G

QA/QC ANALYTICAL DATA FOR FROG CARCASS, LIVER AND GI

TRACT

APPENDIX H

QA/QC ANALYTICAL DATA FOR TADPOLE CARCASS, LIVER AND

GI TRACT

Date post:	23-Aug-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Independent Consultant QAQC Review of the Natural ... · Independent Consultant QAQC Review of the...

Documents