Evaluating Growth Models: A Case Study Using Prognosis BC Evaluating Growth Models: A Case Study...

transcript

Evaluating Growth Models: Evaluating Growth Models: A Case Study Using PrognosisA Case Study Using PrognosisBCBC

Peter Marshall, University of British Columbia, Peter Marshall, University of British Columbia, Vancouver, BCVancouver, BC

Pablo Parysow, Northern Arizona University, Pablo Parysow, Northern Arizona University, Flagstaff, AZFlagstaff, AZ

Shadrach Akindele, Federal University of Shadrach Akindele, Federal University of Technology, Akure, NigeriaTechnology, Akure, Nigeria

Presented at the Third FVS Conference, Feb. 13-15, Fort Collins, CO

OutlineOutline

BackgroundBackground- Prognosis- PrognosisBCBC

- Validation- Validation

- Study area- Study area Testing Against DataTesting Against Data Simulation TestingSimulation Testing ConclusionsConclusions

Within a framework of general observations about validation techniques and processes.

Validation Observation #1

Validation is important … but it tends to be much more of interest to the person doing it than it is to the person hearing about it.

PrognosisPrognosisBCBC

An adaptation of the northern Idaho (NI) version of theAn adaptation of the northern Idaho (NI) version of the original Prognosis modeloriginal Prognosis model

The architecture of the original model remains but many The architecture of the original model remains but many of the internal equations have been reformulated and the of the internal equations have been reformulated and the remainder have been recalibratedremainder have been recalibrated

Habitat types have been replaced with appropriate units Habitat types have been replaced with appropriate units within the BC Biogeoclimatic Ecosystem Classification within the BC Biogeoclimatic Ecosystem Classification (BEC) system(BEC) system

All inputs and outputs are in metric unitsAll inputs and outputs are in metric units Different versions have been developed for various BEC Different versions have been developed for various BEC

subzonessubzones Additional information is available at the following URL:Additional information is available at the following URL:

http://www.for.gov.bc.ca/hre/gymodels/progbc/http://www.for.gov.bc.ca/hre/gymodels/progbc/

ValidationValidation

There are various definitions of “validation” in use and a There are various definitions of “validation” in use and a growing literature on different approaches to use in growing literature on different approaches to use in validationvalidation

For the purposes of this presentation, I will define For the purposes of this presentation, I will define validation as:validation as:

““The process of evaluating model outputs forThe process of evaluating model outputs for consistency and usefulness.” consistency and usefulness.”

Under this definition, validation is very much context Under this definition, validation is very much context dependentdependent- which model outputs are being evaluated - which model outputs are being evaluated - in what location(s)- in what location(s)- for what purposes- for what purposes

Study AreaStudy Area

British Columbia

Victoria

Vancouver

Study Site Location

Interior Douglas-Fir

CANADA

Validation is most effective if several different approaches to validation are used – there are gains from added perspective.

Testing Against Independent DataTesting Against Independent Data Data were from two research installations established in the Data were from two research installations established in the

late 1980s in stands which were uneven-aged and primarily late 1980s in stands which were uneven-aged and primarily interior Douglas-firinterior Douglas-fir

One installation, consisting of 6 plots measured on 4 One installation, consisting of 6 plots measured on 4 occasions, was set up to follow stand dynamics under occasions, was set up to follow stand dynamics under different structural conditionsdifferent structural conditions(1) predominance of large older trees (dbh > 30 cm)(1) predominance of large older trees (dbh > 30 cm)(2) predominance of pole-sized trees (dbh 15-30 cm) (2) predominance of pole-sized trees (dbh 15-30 cm) (3) predominance of saplings (dbh < 15 cm)(3) predominance of saplings (dbh < 15 cm)

The second installation, consisting of 24 plots measured on 3 The second installation, consisting of 24 plots measured on 3 occasions, was set up as a precommercial thinning occasions, was set up as a precommercial thinning experiment in stands which were diameter-limit logged in experiment in stands which were diameter-limit logged in the 1960sthe 1960s

- 3 blocks consisting of three thinning treatments and - 3 blocks consisting of three thinning treatments and a a

control, with two plots in each block/treatment control, with two plots in each block/treatment Projections made for 11 years to match one of the possible Projections made for 11 years to match one of the possible

remeasurement intervals (closest match to the 10-year remeasurement intervals (closest match to the 10-year projections of Prognosisprojections of PrognosisBCBC) )

Projected stand and tree level attributes

Stand-level attributes Tree-level attributes

Stems per hectare

Basal area

Total volume

Merchantable Volume

Tree height

If at all possible, try to look at both individual tree projections as well as stand-level projections. Joint comparisons might well highlight issues that otherwise would not be apparent.

Even apparently well-tested models may well still contain hidden coding errors that have subtle impacts. They are worth looking for carefully. This process is known by some as “verification”.

(We found a few such errors by running various of the component equations both within and outside the model environment looking for “oddities”.)

It is best to look for coding errors early on in the validation process. Otherwise, you may have to re-do some of your previous work.

Individuals who are at arm’s length from the model development process are often more likely to spot errors, since they don’t usually assume that they “know” what is going on within the model.

Summary of individual tree dbh residuals by diameter classes

DBH Classes (cm)

All 10 15 20 25 30 40 50 60 > 60

Trees 3078 1467 969 398 113 41 49 28 9 4

Residual

- 0.009

- 0.154

- 0.055

- 0.036

- 0.978

- 0.925

Summary of individual tree DBH residuals by spacing treatment

Spacing Treatment

All C1 C2 CTRL STD N/A

Stems* 3078 626 335 801 729 587

Residual

-0.009

-0.135

-0.018

-0.069

C1= 3m-clump, C2=5m-clump, CTRL=Control, STD= Standard Spacing, N/ A= Plots not in the spacing study * Trees > 5 cm in DBH

Regression-based equivalence tests (Robinson et al. 2005) provide a convenient means of examining model predictions versus observations. The routine for equivalence testing in R produces nice pictures.

Robinson, A.P., R.A. Duursma, and J.D. Marshall. 2005. A regression-based equivalence test for model validation: shifting the burden of proof. Tree Physiology 25: 903-913

Predicted Dbh (cm)

0 20 40 60 80

Overall equivalence test for the tree-level DBH predictions.

Predicted Dbh (cm)

0 10 20 30 40 50

Equivalence tests (by spacing treatment) for the tree-level DBH predictions.

What you see depends on what you look at.

For example, the relationship between observed and predicted DBH will appear considerably stronger than the relationship between observed and predicted DBH growth, which is actually what is estimated within PrognosisBC.

Predicted Stems/per ha

2000 4000 6000

SimulationSimulation

Feasible and infeasible combinations of BDq. Combinations that produced at least one tree/ha for the largest dbh class are identified as , whereas combinations that did not meet that standard are shown as . (Units for B: m2/ha, D: cm, and q: no units).

B=10 B=30 B=50 B=70

q=1.5 q=2.0 q=2.5 q=3.0 q=1.5 q=2.0 q=2.5 q=3.0 q=1.5 q=2.0 q=2.5 q=3.0 q=1.5 q=2.0 q=2.5 q=3.0 D=20

0 10 20 30 40 50

Projected Time (years)

0 10 20 30 40 50

(a) (b)

0 10 20 30 40 50

(c) (d)

Projected basal area for various initial levels of initial basal area (B) with q = 1.5 and maximum diameter (D) at (a) 20 cm; (b) 40 cm; (c) 60 cm; and (d) 80 cm.

0 10 20 30 40 50

(a) (b)

0 10 20 30 40 50

(c) (d)

Projected change in quadratic mean diameter (qmd) for various initial levels of initial basal area (B) with q = 1.5 and maximum diameter (D) at (a) 20 cm; (b) 40 cm; (c) 60 cm; and (d) 80 cm.

0 10 20 30 40 50

Projected Times (years)

0 10 20 30 40 50

(a) (b)

0 10 20 30 40 50

Projected Times (years)F

(c) (d)

Projected change in stems per ha for various initial levels of initial basal area (B) with q = 1.5 and maximum diameter (D) at (a) 20 cm; (b) 40 cm; (c) 60 cm; and (d) 80 cm.

Impact of modifications to initial stand structure components on changes to particular stand attributes over a 50 year projection period, based on projections using PrognosisBC. Other stand structure components are assumed to remain constant.

Increase in Stand Structure Component Attribute B D q

Basal Area Movement towards “carrying capacity”. If B is ~53 or below, approaches from below; if B is >53 approaches from above.

Slightly slower increase (for lower values of B) or slightly slower decrease (for higher values of B).

Very slight increase (for lower values of B) or very slight decrease (for higher values of B).

Quadratic Mean Dbh Decrease in the rate of increase.

Higher initial value and higher final value after 50 years.

Lower initial and lower final value after 50 years.

Stems Per Ha Higher initial value and final value after 50 years. Increase in the mortality rate.

Lower initial value and final value after 50 years. Decrease in the mortality rate.

Higher initial value and final value after 50 years. Increase in the mortality rate.

Dbh Growth Decrease in overall growth rate.

Slight decrease in growth rate.

ConclusionsConclusions

The validation exercise allowed us to identify and The validation exercise allowed us to identify and repair minor errors in the coding.repair minor errors in the coding.

Once these errors were fixed, the model Once these errors were fixed, the model performed well against data at both the single performed well against data at both the single tree and stand level.tree and stand level.

The model produced results under a wide variety The model produced results under a wide variety of stand structures which were consistent with of stand structures which were consistent with our understanding of stand dynamics.our understanding of stand dynamics.

When preparing validation reports, remember Observation #1:

“Validation is important … but it tends to be much more of interest to the person doing it than it is to the person hearing about it.”

It is easy to bury readers with an avalanche of results. However, syntheses and summaries are much more likely to be read and understood.

This project was funded by the BC Ministry of Forests and Range, using funds provided for continuing work on PrognosisBC by the Forest Investment Account (FIA). We are grateful for this support.

Thank you! Are there any questions?

Evaluating Growth Models: A Case Study Using Prognosis BC Evaluating Growth Models: A Case Study...

Documents