NSAA Peer Review Program:What to Know Before You Go!National State Auditors AssociationJanuary 21, 2015

1

2

Opening Remarks

MODERATORR. Kinney PoynterExecutive Director,

NASACT

SPEAKERJohn Buyce

Director of State Audits,Office of the State Comptroller (NY)

SPEAKERWilliam (Brad) Blake

Chief Auditor, Center for Audit Excellence, Office of State Auditor (OH)

SPEAKERGreg Fugate

Performance Audit Manager, Office of the State Auditor (CO)

SPEAKERStaci Henshaw

Deputy Auditor, Virginia Office of the Auditor of Public Accounts (VA)

Agenda

• Overview, Objectives, and General Considerations

• Organization and Qualifications of the Review Team

• Performing Peer Reviews

• Finalizing the Peer Review

• Tips & Insights

3

Overview, Objectives, and General Considerations

Overview

• Purpose of peer review program – to provide independent assessments of state audit organizations to determine whether they have an adequately designed internal quality control system and are in compliance with that system

5

Overview

• Yellow Book began requiring peer reviews in 1989

• NSAA program started in 1985

• Currently, 54 eligible organizations participate in the program

6

Overview – Role of Committee

• Peer Review Committee has overall responsibility for the program– Provide guidance in the form of policies

and procedures– Resolve potential disputes that may arise

in the review process and ensure consistency of reviews

– Coordinate with other groups to ensure the adequacy of the review process

7

Overview – Role of NASACT

• Administer the review process on a daily basis

• Coordinate and assign review teams

• Train review team members

• Update policies and procedures for review by the Peer Review Committee

8

Objectives of the Review

• Evaluate whether the State Audit Organization’s system of quality control is…– suitably designed, adequately

documented and communicated; and– sufficiently complied with to provide

reasonable assurance of compliance with government auditing standards

9

Objectives of the Review

• Upon completion of the peer review, the peer review team issues a report

• Types of peer review reports– Pass– Pass with deficiencies– Fail

10

General Considerations

• Maintain confidentiality of state’s working papers and peer review work papers

• Be independent of state audit organization being reviewed

• Possess adequate knowledge and proficiency to perform review

11

Organization and Qualifications of the Review Team

Organization of Review Team

• Concurring reviewer– Most senior member of team

• Team leader– Overall responsibility for planning and

performing the review

• Team member– Responsible for performing tasks

assigned by team leader

13

Organization of Review Team

• Team leader and concurring reviewer identified 4-5 months prior to the actual date of the review

• Team members identified 2-3 months prior to the actual date of the review

14

Qualifications – Concurring Reviewer

• Must have served as team leader on a least one review under NSAA’s program

• Must have been recommended by former concurring reviewer

15

Qualifications – Team Leader

• Must have served as team member on a least one review under NSAA’s program

• Must have been recommended by former team leader and concurring reviewer

16

Qualifications – Team Member

• Must be recommended by audit organization head

• Must be in a supervisory or managerial role

• Must review audit documentation as part of job responsibilities

• Minimum of three years in supervisory role

17

Qualifications for Review Team Member – Other Issues

• Ensuring a proper match– NASACT tailors team skills and

experience to the SAO’s needs• Type of work performed (Financial,

Performance, Attest)• Size of the SAO and number of audits• Government Audit Quality Center (AICPA-

GACQ) membership• Federal reviewer for Single Audit

• Usually up to six reviewers selected

18

Performing the Peer Review

Performing the Peer Review

• Preliminary phase

• Field work phase

• Completion phase

20

Preliminary Phase

• Understanding the Peer Review Program

• Obtaining/reviewing necessary SAO info including prior work papers

• Determining scope of review

• Sending audit staff questionnaire

• Selecting engagements

• Finalizing planning

21

Documents Used in a Peer Review

• Audit Organization Questionnaire

• Audit Staff Questionnaire

• Audit Organization’s Policies and Procedures and Review Guide

• Guide for Review of Audit Engagements

• Matters for Further Consideration (MFC)

• Findings for Further Consideration (FFC)

• Conclusions

22

Team Preparation: Understanding the Review Process

• Ensure each team member obtains an understanding of the process – NSAA External Peer Review Manual– Questions and Answers for All Team

Members – Administrative Matters

• Timing of review procedures• Travel arrangements• General expectations

23

Team Leaders: Review of Prior Work Papers

• Issues identified in prior peer review report– Prior peer review rating

• Read notes from exit conference

• Review other issues identified (MFCs and FFCs)

24

Audit Organization Questionnaire

• Completed by the SAO and provided to the Team Leader approximately 12 weeks prior to the review

• Provides the population of audits and audit hours within the review period

• Provides information about the state office’s staff, structure, and quality control system

25

Determining the Scope of the Review

• Determine whether the SAO performs– Financial Audits– Attestation Engagements– Performance Audits

• Engagements the SAO has stated to have been performed in accordance with government auditing standards are subject to the review

26

Determining the Scope of the Review

• Review should cover a current period of one year to be mutually agreed upon by the SAO and the review team

• Covers the quality control policies and procedures in effect and compliance for the year under review

• Includes reports issued during the year under review, or immediately thereafter if the work was substantially completed during the period

27

Determining the Scope of the Review

• The scope of the review does not cover– Non-GAGAS engagements– Audits done by others (e.g., contracted)

– Audit efficiency– State statute compliance– Administrative aspects, unless directly

required to satisfy applicable professional standards (e.g., Independence, CPE)

28

Other Administrative Matters

• Team Leader – Work Program– Prepare and send the engagement letter – Work with the SAO to establish deadlines

for receiving key documents– Initial communication with Concurring

Reviewer and Team Members

• NASACT – Get signed contract to Team Leader

29

Pre-Visit Work

• Team Leader– Select Audit Engagements for Review– Audit Staff Questionnaire– Preliminary Site Visit– Policy & Procedure Review– Finalizing the Review Plan– Administrative Logistics

30

Selecting Audit Engagements for Review

• Based on listing provided by the SAO• Factors to consider include:

– Spread across various supervisors, regions, etc.

– Spread across the review period– Different types of engagements (Cash

Basis vs GAAP, Single Audits, CAFRs, Performance, Attest)

• Select one “surprise” audit

31

Other Sample Selections

• Select a sample of audit staff to complete the Audit Staff Questionnaire

• While onsite, select a sample of audit staff for verification of CPE records and independence documentation

32

Audit Staff Questionnaire

• Work with the SAO to administer the questionnaire about 8 weeks prior to the review– Responses submitted directly to the Team

Leader about 6 weeks before the review

• Team Leader compiles the responses and communicates a summary to the team

• Evaluate responses for potential areas to concentrate review efforts

33

Preliminary Site Visit and Other Considerations

• Team Leader consults with the Concurring Reviewer to determine if a preliminary site visit is needed– Very rare– If one occurs, the preliminary site visit is

4-6 weeks in advance

• Team Leader has option to arrive on site at the SAO a few days prior to the rest of the team members

34

Team Members: Evaluation of the SAO’s Policies and Procedures

• Policies and Procedures Checklist – Separate checklists for financial audits,

performance audits, and attestation engagements

– Completed by the SAO and provided to the Team Leader along with the SAO’s policy and procedure manual, usually about 8 weeks prior to the review

35

Policies and Procedures Checklist

36

Policies and Procedures Checklist

• Team Leader assigns each team member a section or sections of the policies and procedures to review

• Assignments are made to the team members approximately 8 weeks prior to the review

• Team Members are responsible for completing this work before arriving on site for the review

37

Finalizing the Review Plan

• Team Leader discusses plan for the review with the Concurring Reviewer

• Team Leader informs the SAO about sampled audit engagements, usually about 4-6 weeks prior to the review

• Team Leader only informs the SAO about the sampled “surprise audit” when the team arrives on site

38

Sampled Audits/Engagements

• Team Leader assigns each team member one or more audits/ engagements to review

• Team Members are responsible for completing some work before arriving on site for the review– Download and review reports– Complete reporting sections of the Audit/

Engagement Review Checklist

39

Audit/Engagement Review Guide

40

Administrative Logistics

• Team Leader will communicate about administrative logistics in the 1-2 weeks leading up to the review– Confirm travel, car, and hotel– Set time/place for initial meeting– Work plan/schedule for the review– Dress code and other information

• If something is unclear, be sure to ask!

41

Field Work Phase

• Complete the study and evaluation of QC policies and procedures

• Review compliance with policies and procedures

• Identify matters, findings, deficiencies, and significant deficiencies

• Aggregate and evaluate matters• Form conclusion on type of report to issue• Communicate conclusions at exit conference

42

Overall Fieldwork Schedule

• Five Day Reviews– Generally run Monday - Friday

• Ten Day Reviews– May run Monday to Wednesday or

Wednesday to Friday– Allows a few more days for review of

sampled audits

43

Time Frames for Large Reviews

44

Time Frames for Small Reviews

45

Arrival – Day Before the Review Begins

• Team Leaders often hold a team meeting at the hotel in the evening– Answer any questions, logistics, etc.– Results of policy & procedure reviews

• Team Members– Discuss policies noted in their assigned

area and their reviews of reports

• Some teams do a conference call during the week before the review

46

Day 1 – Let’s Get Started

• Entrance Conference

• Compliance testing for Policies & Procedures

• Get oriented to the audit workpapers, software, network, etc.

• Begin detailed review of first sampled audits

47

Team Leader – Day 1

• Run the Entrance Conference

• Discuss any outstanding issues with SAO officials

• Compliance testing for P&P

• Get organized

• Assist Team Members in starting their audit reviews

• Status meeting at day’s end

48

Team Members – Audit/Engagement Reviews

• Standard review guide for each type of audit/engagement– Financial, Performance, Attest

– Covers all phases of the audit/engagement from planning through reporting

– Cross referenced to individual audit standards, AU citations, etc.

• Use to test compliance with procedures by teams

49

Audit/Engagement Review Guide

50

Audit/Engagement Reviews

• Complete checklist for each audit/engagement– Seek clarification from audit teams– Start your second review if waiting for

information from others

• “NO” answers generate an MFC – given to Team Leader

51

MFC Form

52

MFC Form

• “Matters” generally will originate from “no” answers on checklist (either with design of, or compliance with, system of quality control)

• Can be cleared, discussed verbally with the audit organization, or carried forward to the Conclusions document

53

Day 2 – More Audit Reviews

• Team Leader– Answer questions, provide guidance, ensure

consistency– Organize MFCs, review completed work,

test items to workpapers– Get ready for the Concurring Reviewer to

arrive• Team Members

– Discuss issues with audit teams – In most cases, you should be on your

second audit in the afternoon

54

Day 3 – Start Wrapping it up

• Team Leader– Brief the Concurring Reviewer on

progress and issues– Continue organizing and grouping MFCs– Assist Team Members in completing

audit reviews

55

Day 3 – Start Wrapping it up

• Team Members– Continue completing review guides and

MFCs– Second audits should be all but done– If there’s a third, it should be well

underway in the afternoon• Prioritize the areas you want to review• Key on planning, execution, documentation

and common issues

56

Day 3 – Start Wrapping it up

• End of the Day– Status meeting

• Initial discussion of issues that may warrant additional work or be potential Findings

– Should have a sense of the overall rating

• Plan out work that needs to be completed and when

– Reassign work and assist each other as necessary

57

Day 4 – Pull it all together

• Team Leader– Organize MFCs for discussion– Manage completion of reviews by mid-

day– Lead team discussion to come to a

consensus on disposition of MFCs, Findings and rating

• Contact and consult with NSAA if necessary

58

Day 4 – Pull it all together

• Team Members– Complete all review guides and MFCs– Assist others to tie up loose ends

• Afternoon – Team Discussion– Review all MFCs and discuss the ones

you prepared– Work to consensus on items to elevate to

Findings based on significance and pervasiveness

59

Day 4 – End of the Day

• Decide on form of report and issues to discuss at exit conference– Complete the relevant Conclusion

documents and finalize a draft of the report

– Consult with NSAA as appropriate if report rating is other than “pass”

• Brief officials about the general conclusions of the report and any findings

60

How do you decide what type of report to issue?

• Based on professional judgment considering nature, causes, pattern and pervasiveness of matters and their relative importance to the organization’s system of quality control taken as a whole

61

Forms Used in Documenting Results

• Matters for Further Consideration (MFC)

• Conclusions Document

• Findings for Further Consideration (FFC)

62

Reporting Matrix

• Section II of the Peer Review Manual includes a reporting matrix to provide guidance on various reporting considerations when evaluating results

63

Reporting Matrix

64

Types of Peer Review Reports

• Pass

• Pass with Deficiencies

• Fail

65

Pass

• Audit organization’s system of quality control has been suitably designed and complied with to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects

66

Pass

• Design– Adequate, or inadequate for parts of one

or more standards

• Compliance– Sufficient, or insufficient for parts of one or

more standards

• Severity – Insignificant or Moderate

• Frequency – Isolated, but considered a Finding if recurring and pervasive

67

Reporting Matrix

68

Pass with Deficiencies

• Audit organization’s system of quality control has been suitably designed and complied with to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects with the exception of a certain deficiency or deficiencies that are described in the report

69

Pass with Deficiencies

• Design– Adequate overall, but inadequate for

substantially all of one standard or parts of several

• Compliance– Sufficient overall, but insufficient for one

standard or parts of several

• Severity – Serious• Frequency – Recurring & Pervasive

70

Fail

• Based on significant deficiencies described in report, audit organization’s system of quality control is not suitably designed to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects

71

Fail

• Or based on significant deficiencies described in report, audit organization has not complied with its system of quality control to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects

72

Fail

• Design– Inadequate to provide reasonable

assurance – several standards

• Compliance– Insufficient overall – several standards

• Severity – Severe

• Frequency – Recurring & Pervasive

73

Conclusions Document

• Used to determine the appropriate reporting for matters– FFC Form– Deficiency– Significant Deficiency

74

Conclusions Document

75

Findings for Further Consideration Form (FFC)

• Used to report findings not rising to the level of a deficiency or significant deficiency

• This becomes part of the working papers, but not part of the reporting process

76

FFC Form

77

Completion Phase

• Finalize the peer review report

• If rating is other than pass, audit organization should respond

78

Tips for a Successful Peer Review

Communication

• A successful review depends on good communication at all levels before and during the review

• Avoid making assumptions

• Ask questions

• Be responsive when asked a question or information is requested

80

Preparedness

• Understand the peer review process

• Establish and adhere to deadlines for pre-review and on-site work

• Review the state office’s website and work products

81

Audit Reviews – Planning, Execution and Documentation

• Work from the end inward (documentation)– Examine the report and supporting work

• Also work from the start (planning)– Look at the application of general

standards and the planning

• Meet in the middle (execution)– Did they follow procedures? – Do what they planned? – Properly report what they found?

82

Learn Something

• There is more than one way to comply with auditing standards

• The peer review process should be value-added– What can the state being reviewed learn

from the review team?– What can the review team learn from the

state being reviewed?

83

How can you get involved?

• Must have at least three years of supervisory experience

• Complete Team Member Qualifications form online

https://www2.nasact.org/QCR/2015_Team_Member_Qualifications.asp

– Info on your background, experience and the types of audits you supervise, as well as you availability and preferences for where you want to go

• Must update form each year• Currently about 300 participants

84

Don’t Get Discouraged

• Factors can limit your chance of selection – especially the first time– Number of reviews scheduled – Type of audits those states perform– Usually only one person per state on a

team– Usually no more than one first-time

reviewer– Your state’s balance in time credits

85

Any Other Questions?

Talk to your office’s peer review liaison or contact the Peer Review Coordinator, Kathleen Young, at kyoung@nasact.org or (859) 276-1147

87

Questions

MODERATORR. Kinney PoynterExecutive Director,

NASACT

SPEAKERJohn Buyce

Director of State Audits,Office of the State Comptroller (NY)

SPEAKERWilliam (Brad) Blake

Chief Auditor, Center for Audit Excellence, Office of State Auditor (OH)

SPEAKERGreg Fugate

Performance Audit Manager, Office of the State Auditor (CO)

SPEAKERStaci Henshaw

Deputy Auditor, Virginia Office of the Auditor of Public Accounts (VA)