Version 1 rev.3 – 25-26/02/2015
REFERENCE DOCUMENT FOR NAS NETWORK
WORKING GROUP - AUDIT EVIDENCE
MEETING 25 AND 26 FEBRUARY 2015
FVO, GRANGE
NAS Audit Evidence subgroup - working document Page 1
Version 1 rev.3 – 25-26/02/2015
The National Audit Systems (NAS) Network
The NAS network is a network of officials (auditors) from national competent authorities, responsible for the performance of audits of official control systems as provided for by article 4(6) of Regulation (EC) No 882/20041. The networks meet regularly, under the chairmanship of, and facilitated by, the FVO to exchange experiences in implementing national audit systems on official control activities. During the course of these exchanges; discussions, workshops etc. good principles and practices are identified and agreed by the network.
To enable dissemination of information the network, working in plenary session and through sub-groups, facilitated by the FVO, consolidate agreed principles and good practices on specific topics into reference documents. These reference documents may be used as guidance documents, however, they do not constitute an audit standard and are not legally binding.
Audit Evidence
OBJECTIVES
The objective of this document is to guide and support Competent Authorities (CA) and audit bodies in managing audit evidence.
The aim is:
To provide principles and definitions regarding audit evidence
To identify characteristics, types, and sources of audit evidence
To discuss evidence collection planning
To give principles/discuss verification of audit evidence
This document is intended to assist in the implementation of Section 6 of the Annex to Commission Decision 2006/677/EC.
SCOPE AND INTENDED AUDIENCE
This guidance applies to planning of audits as required by Article 4(6) of Regulation (EC) No 882/2004.
It is intended for use by CAs / audit bodies that carry out audits on official control (systems) according to the requirements of Article 4(6) of Regulation (EC) No 882/2004.
It supports the development of good practice in audit evidence collection and verification in the area of official control activities e.g. feed, food, animal health and welfare and plant health.
1 OJ L 191, 28.5.2004
NAS Audit Evidence subgroup - working document Page 2
Version 1 rev.3 – 25-26/02/2015
I. BACKGROUND AND CONTEXT
{where does evidence fit in the audit cycle – evidence and evidence collection plan}
{The objective of collecting evidences : measure, compare, evaluate to meet the objective, in order to support audit conclusions}
{audit criteria and how it links with audit objectives}
The collection of audit evidence is a familiar but important step in the audit process. The quality of the evidence collected has a direct and significant effect on the audit findings and conclusions.
The audit team should, at the audit planning stage of an audit, consider what audit evidence should be required. During the audit process, the audit team should verify the audit evidence collected and ensure it is appropriate and sufficient to achieve the audit objectives. Audit evidence needs to be compared to the audit criteria and the audit objectives to allow the audit team produce audit findings and present persuasive audit conclusions.
Only audit evidence that is appropriate and sufficient will effectively support audit findings and conclusions which are capable of withstanding challenge and satisfy internal and external scrutiny.
{Audit objectives: (IIA) 2210.A1- Internal auditors must conduct a preliminary assessment of the risks relevant to the activity under review. Engagement objectives must reflect the results of this assessment}
{Audit criteria: means the set of policies, procedures or requirements used as a reference against which audit evidence is compared, i.e. the standard against which the auditee’s
activities are assessed.}
. (ISO) – (IIA) 2210.A3- Adequate criteria are needed to evaluate controls. Internal auditors must ascertain the extent to which management has established adequate criteria to
determine whether objectives and goals have been accomplished. If adequate, internal auditors must use such criteria in their evaluation. If inadequate, internal auditors must work
with management to develop appropriate evaluation criteria. For me the approach seems different, the criteria are linked for the IIA to the measure of the organisation’s goals, the
criteria used for the appreciation of the evidences come from the key internal control points of the procedures (in the risk matrix of the audit team)}
{Different terminology may be used by different MS for the same processes}
{reference to ISO to help with language versions}
NAS Audit Evidence subgroup - working document Page 3
Version 1 rev.3 – 25-26/02/2015
{Evaluation of Evidence: Is this within the scope of this document or does it belong to the next phase ? Possibly dealt with in a separate reference document on “Recommendations”}
[II.] DEFINITION(S)
This document should be read in conjunction with the definitions contained in Regulation (EC) No 882/2004 and Commission Decision 2006/677/EC bearing in mind that the definitions of those documents apply.
Audit Evidence: records, statements of fact or other information which are relevant to the audit criteria and verifiable. (ISO 19011:2011 from ISO 9000:2005)
Effectiveness: is the extent to which official controls produce an (intended) effect / achieve an objective2. In this particular context the objectives are those of Regulation (EC) 882/2004. Effectiveness is not to be confused with efficiency, which is normally used when we want to refer to input-output ratio i.e. cost and/or resources required to produce an output. (“Auditing effectiveness of official control systems” NAS network document)
Findings: results of the evaluation of the evidence collected during the audit against the applicable standard (e.g. legislation), described in an objective manner. (FVO SOP Audit Performance)
Conclusions: statements made by the audit team concerning the outcome of the audit which are based on and after consideration of all the findings and the audit objectives but which do not propose any course of action. (FVO SOP Audit Performance)
{Note: characteristics not mentioned in this section are better described in the next chapter, less prescriptive and explaining the concept}
II.[III.] AUDIT EVIDENCE
Include statement on usefulness of evidence, i.e. when it helps to reach goals of the audit "An effective audit has persuasive findings and conclusions. The quality of audit findings and conclusions relies on the judgements the auditor makes and these judgements are directly dependent on the quality of the audit evidence collected and the competence of the auditor collecting it”
Audit Evidence: information used by the auditor in arriving at the conclusion on which the auditor's opinion is based…."(international Standard on Auditing (UK and Ireland)
Audit Evidence: Audit evidence is the information internal auditors obtain through observing conditions, interviewing people, and examining records. Audit evidence should provide a factual basis for audit opinions, conclusions, and recommendations. (IIA - SAWYER) / Audit evidence is
the information that supports or refutes an audit objective (IIA – David O’Regan)”.
The nature of audit evidence in systems audits
{particularities of systems audits vs financial or compliance audits}
2 Objectives may be at a strategic or operational level.
NAS Audit Evidence subgroup - working document Page 4
Version 1 rev.3 – 25-26/02/2015
For financial or compliance audits, evidence only needs to be collected to demonstrate activities are being carried out to planned arrangements. For systems audits, evidence needs to be collected to verify the effective implementation of planned arrangements
Quantitative versus qualitative
A. Characteristics of audit evidence
{Some text?}
Description
PersuasiveThe persuasiveness of evidence is linked to its appropriateness (relevant and reliable) and sufficiency.
(Linked also to target audience and findings)
Appropriateness / Usefulness
The appropriateness of the evidence is the measure of the quality of the evidence determined by its reliability and relevance.
Sufficient
When there is enough evidence to persuade a reasonable person that the audit findings and conclusions are valid, and that the recommendations are appropriate. [IIA]
Amount of evidence considered enough: [Scoping paper]
i) for the auditor to form a reasonable opinion (sample size, representativeness)
ii) to convince interested parties/stakeholders of validity of auditors opinions (persuasive)
Relevant
When the evidence is clearly and logically related to the audit questions, audit criteria and audit findings. [IIA]
Extent to which the information bears a clear and logical relationship to the audit findings (and audit objectives). [Scoping paper]
Reliable
When evidence is obtained through the use of appropriate techniques. When the same findings arise when alternative techniques are used or when information is obtained from different sources.
The best obtainable information through the use of appropriate engagement techniques. [IIA]
The degree to which evidence can be considered trustworthy (accurate and credible), the likelihood of coming up with the same answers if audit test is repeated or information is obtained from a different source or test. [Scoping paper]
Continuity and integrity of evidence. e.g. in a laboratory the reliability of results could be in question if sample identification, documentation and/or security is suspect.
Verifiable MA
Objective MA
Representative Representative of the audit universe and … time… {UK’s example}
Logical/ Rational/
Linked to persuasiveness MA
NAS Audit Evidence subgroup - working document Page 5
Version 1 rev.3 – 25-26/02/2015
Description
Reasonable/ Sound
Reference to Annex I – Audit Evidence Mind Map
B.[A.] Types
Type Description Examples of evidence/Techniques Considerations
Observed (or Physical)
Information gathered by the auditor through personal observation of people, events and physical.
Examples:
Visual control
Photo?
Sample
Techniques:
Direct inspection or observation of people, property or events.
Listening, smelling?
On-site verification
Shadow inspection / Witness audit
Review audit3
Whilst usually the most persuasive evidence, the auditor must be aware that a risk exists that his/her presence may distort or prejudice what would normally occur, thus reducing the quality of the evidence.*
Ways to record this type of evidence – photo, notes? Cross reference with Section V. Verification of Evidence.
Documentary
Information prepared by others than the auditor. Documentary information can exist both in paper and electronic form.
Exemples :
Documents containing routines website information, etc.
Photos
Internal/external
Paper/electronic
Legal/work
ISO definition of “document”?
Techniques:
Review of documents, reports, manuals, literature, external and internal websites, postal or web-based surveys.
This evidence may be in electronic or hardcopy format. *
However, useful information may not always be documented, thus necessitating the use of other approaches also.*
Be sure to record the date on which the information was gathered as the information may change later on.
Oral / Inquiry
Information gathered from people through
Examples:
Oral / written
Oral evidence is generally important in performance audits, as information obtained in this manner is up-to-date
3 Audis of the FBO without the presence of the inspector
NAS Audit Evidence subgroup - working document Page 6
Version 1 rev.3 – 25-26/02/2015
Type Description Examples of evidence/Techniques Considerations
interviews and focus groups. Such information may take the form of written or oral statements.
interview
Single / group
Techniques:
Interviews
Presentations
Questionnaires?
Knowledge/facts?
and may not be available elsewhere.*
However, information should be corroborated and statements confirmed if they are being used as evidence.*
Analytical
Indirect or derived evidence / information constructed by the auditor combining information from different sources and analysing that information to reach a conclusion.
Examples:
Comparison
Computation
Ratio
Crossing
Techniques
Analysis through reasoning, reclassification, computation and comparison
Such evidence is obtained by using professional judgement to evaluate physical, documentary and oral evidence. *
Be aware of importance of Audit experience and skills
Based on page 96 of “Internal Audit Practice”, chapter on “Gathering and analysing information”* based on page 60 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase
Note: Types are not related to description of ways to record evidence as this aspect may be covered by internal procedures.
C.[B.] Sources
Source Type of Evidence Examples / Techniques Considerations
Obtained directly by the auditors
Observed (and Physical)
Oral / Inquiry
Analytical
Direct inspection,On-site verificationObservation
Interviews,Preparation of questionnaires
Previous audit reports from the audit bodies,Analysis
The auditors can determine the methods that will provide the best quality of evidence for the particular audit. However, their skills in designing and applying the methods will determine the quality of the evidence. *
Provided by the auditee
Documentary Information from databases, documents, activity statements and files (e.g. procedures, instructions, legal acts, inspection reports,
Auditors must determine the reliability of data that is significant to the audit questions by review and corroboration, and by testing the auditee's internal controls over information, including general and application controls over
NAS Audit Evidence subgroup - working document Page 7
Version 1 rev.3 – 25-26/02/2015
Source Type of Evidence Examples / Techniques Considerations
Oral / Inquiry
management reviews, organisational and planning documents, certifications).*
Answers to questionnairesOral replies during interviews
computer-processed data. *
Provided by third parties
Documentary
Oral / Inquiry
Information which may have been verified by others or whose quality is well known, e.g. national statistical data.*Information belonging to third parties (Business Operators, Customs, Stakeholder representatives, other CAs, etc.)Third parties audit reportsWebsites
Answers to questionnairesOral replies during interviews
The degree to which such information can be used as audit evidence depends on the extent to which its quality can be established and its significance in relation to the audit findings. *
* based on page 59 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase
{Maybe we could add a point for the common deficiencies: failing to scrutinize important point, failing to maintain auditor independence, failing to supervise work}
III. EVIDENCE COLLECTION PLANNING
Why do we need it?
Main purpose is to allow a targeted evidence gathering to support the audit findings. This should focus on the audit objective and scope.
Reference to Annex II – Diagram of Audit Process
What is the benefit?
To gather enough evidence and not more than needed.
Plan the audit so that enough (sufficient) evidence can be obtained to be able to draw conclusions that have a bearing on the object of the audit.
(RdH - link evidence sufficiency to the audit objectives)
How do we do it?
Which methodologies are used? Is there a “good practice” that we can identify? Iterative approach? Use of external experts (e.g. in data analysis)?
NAS Audit Evidence subgroup - working document Page 8
Version 1 rev.3 – 25-26/02/2015
Knowledge of, and information available to, internal auditors vs. external auditors. Bias of auditors?
Importance of on-site
Note: important to link with characteristics of evidence (how to ensure we get useful information)
Factors to consider when judging the quality and quantity of audit evidence:
the purpose for which the evidence will be used
a higher standard is required for evidence supporting audit findings than for background information provided in the audit report
the level of the significance of the audit finding
in general, the higher the level of significance, the higher the standard of evidence that is required
the degree of independence of the source of the evidence
greater reliance can be placed on evidence which emanates from independent sources
the cost (money or time) of obtaining additional evidence relative to likely benefits in terms of supporting findings and conclusions
at some point, the cost of obtaining more evidence will outweigh the improved persuasiveness of the total body of evidence
the risk involved in making incorrect findings or reaching invalid conclusions
the greater the risk of legal action, controversy or surprise from reporting an audit finding, the higher the standard of evidence needed
the care taken in collecting and analysing the data Including the extent of the auditors' skills in these areas
* based on page 58 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase
Reference to Annex III – example of evidence matrix
When does it take place?
Create a timeline with audit steps and where evidence collection planning takes place:
Planning - Audit Objectives + scope - Phase 1 (desk study) – Risk analysis/desk study results – Phase 2 (test – on-site activities) – Audit report? Diagram?
Planning – preparation – execution – reporting? Diagram?
Evidence collection planning may take place at different stages, depending on the audit planning approach (Desk-based / on-site). Refined, adapted and developed along the audit process. On-site evidence collection is particularly important where the audit is being used to confirm/verify the effective implementation of planned arrangements.
Retention of Audit Evidence.
This would have particular significance in respect of independent scrutiny, evaluation of or challenge to an audit system and /or its findings.
Should be kept during a period described by the audit body or national rules.
Link to Section III, table “A. Types”, type “Observed (and Physical)”, ways to record this type of evidence.
NAS Audit Evidence subgroup - working document Page 9
Version 1 rev.3 – 25-26/02/2015
IV. VERIFICATION OF AUDIT EVIDENCE
A. Verification: (ISO 9000 definition?)
Is the evidence really “evidence” The information collected is not audit evidence until it has been verified.
(meeting its characteristics – described above in Section III.A)? Root-cause-analysis – link with the evidence, runs along with the collection and verification of evidence.4Reference to Annex IV – to be developed.
Who does it?
Auditors (and their managers?).
When to do it?
Along with evidence gathering. Importance of on-site (do we need to emphasise here ; need additional text, refer to the document on effectiveness)
How to verify?
Cross-checking; / Review of auditor’s work (own review or supervision);) / Quality checks;/ Peer review.
[in BTSF CB-D3-P04]
4 Reference to “Root-cause analysis of non-compliance – outcome of the workshop (MANCP WG-meeting 21-22/11/2012)”
NAS Audit Evidence subgroup - working document Page 10
Version 1 rev.3 – 25-26/02/2015
B. Validation: (ISO 9000 definition?)
Who validates?
Auditors (and their managers?).
Importance of competence and roles/responsibilities when validating evidence.
When to validate?
How to validate?
Supervision of auditor’s work; Peer review
NAS Audit Evidence subgroup - working document Page 11
Version 1 rev.3 – 25-26/02/2015
Annex I - Example of a mind map on Audit Evidence (to be adapted)
NAS Audit Evidence subgroup - working document Page 12
Version 1 rev.3 – 25-26/02/2015
Annex II – Diagram of Audit Process
NAS Audit Evidence subgroup - working document Page 13
Annex III - Example of an evidence matrix5:
1 - From the risks cartography of the competent authority showing a high level of criticality on the subject of “species substitution”, the audit team planned a mission
2 – The audit team analyses the process of the CA to deal with that subject. The audit team elaborates a risk matrix of the process to identify the key points and the criteria showing they are under control. The audit team also reduces the scope of the mission to the horse meat, because it appears to be, one of the easiest products to substitute with a cheaper one, hard to detect and enables quick and strong profits.
3 – The process of audit can be summarized with the following matrix
Audit Objective
Steps and criteria Audit evidence
Type of evidence Level of evidencedocument
aryobservat
iontestimonia
lAnalytical enough Too
weakToo
muchIs the CA efficient in horse meat species substitution controls?
Planning
All the country is covered, all year long
The annual control plan
The Y-1 synthesis
Interview of the meat board manager
Data registered in the information system related to the meat control plan
Yes
From the production to the distribution chain
Yes
the orders are efficiently transmitted to the agents
Message and instructions given to the agent through the country
Local interview of meat agents
Yes
Execution
The control plan is respected (quantity, quality, data recorded)
Local planning declining the national instructions
Interviews of management and agents
Local results vs local objectives in the information system
Interview of agents is useless
The agent knows how to make a sampling
Training records of agents on the subject
Agent evaluations
Interviews of agents
Need to add an on-site observation to conclude
5 This matrix is linked to a specific kind of audit and can be adapted to other cases.
NAS Audit Evidence subgroup - working document 14
Audit Objective
Steps and criteria Audit evidence
Type of evidence Level of evidencedocument
aryobservat
iontestimonia
lAnalytical enough Too
weakToo
muchThe agent knows the product, the law, the internal procedures,
Quality of local records
Yes
Analyse The labs used a the right equipment
The analysts have the competencies
The lab is referenced
Prosecution
The level correspond to the level of the fraud
The rate of prosecution is homogeneous on the territory
The rate of validation by the court is high
We can also add a column to the matrix to write the findings and another one for conclusions
NAS Audit Evidence subgroup - working document 15
Annex IV – Audit map and root-cause analysis
Alternative diagram
NAS Audit Evidence subgroup - working document 16