American Council of Engineering Companiesof Metropolitan Washington

DDOT/ACEC QUALITY FORUM. May 26, 2011

Consultant Performance Evaluation Procedure 

• Establish and implement a uniform program for systematic evaluation and assessment of consultant performance.

• Enhance the Department’s ability in the consultant selection process.

• Provide an objective and consistent method for measuring Consultant performance and benefits both the Department and the Consultant.

• Provides the Department with a means for rewarding those Consultants who perform good work, provides essential data to be used in the selection process for future projects

• Provides Consultants with the opportunity to improve job performance from one rating period to the next.

PURPOSE

PROCEDURE

•  The Project Leader/Project Manager  (PLPM) will rate consultant performance in the applicable general categories (i.e. Schedule, Quality & Project Management) Evaluations may also include work specific checklists. 

The rating criteria for each discipline below are specifically defined under each category.

• Design • Engineering Services during Construction • Bridge Management and Inspection • Construction Management and Inspection

PROCEDURE         • The evaluations for Design contracts will be project specific.  All other evaluations may be 

project specific or agreement specific.

•  Design project evaluations will include the work for Preliminary design, Intermediate Design, Final Design and Construction Phase. 

• Design evaluations may also include project work for Concept Development and Feasibility Assessment. 

• Design Phase related work such as; Environmental Investigations and/or Permitting (Wetlands, Cultural Resources, Hazardous Waste, Investigations/Permits, etc.), performed by a Consultant through a standalone Task Order Agreement will be rated according to the Design Consultant Evaluation (Design Phase) criteria. 

• Subconsultant work will not be rated independently but will be included in the prime Consultant’s rating.

 

RATING SCALE

The rating scale for consultant performance evaluations shall be as follows: 

             5 = EXCELLENT performance4 = ABOVE SATISFACTORY performance3 = SATISFACTORY performance2 = MARGINAL performance1 = UNACCEPTABLE performance

RATING CATEGORIES AND WEIGHT FACTORS

The following rating categories and weight factors will be used for the appropriate work discipline. 

Design                         Rating Category Weight Percentage 

                     I. Schedule                                               30%                          II. Quality

              Errors & Omissions                        60%                            III. Project Management                        10%       

Post construction evaluation of design will be based on 100% quality.

  Final Design Rating evaluation will be based on 70% weight during design phase and 30% post construction. 

     

RATING CATEGORIES AND WEIGHT FACTORS

Engineering Services during Construction:         Rating Category Weight

Percentage       I.  Schedule                 30%      II.  Overall Quality                      60%

 III. Project Management             10%

RATING CATEGORIES AND WEIGHT FACTORS

Bridge Management and Inspection 

Rating Category Weight Percentage                                    I.    Schedule                                  30%                                  II.    Overall Quality

Errors & Omissions                  50%                                 III.    Project Management                  20% 

RATING CATEGORIES AND WEIGHT FACTORS

Construction Management & Inspection            

                         Rating Category Weight Percentage                       I.     Overall Quality                      90%                     II.     Project Management 10%

Design Phase Example

 Consultant: A & E Consultants, Inc., Project:  Reconstruction of Street/Bridge                     Agreement No./Contract No. 000000

Schedule Quality                   Project ManagementCategory Rating 3                       4            4 Weight Factor 30% 60%       10%                        Weighted CategoryRating 0.9        2.4       0.4                       Total Weighted Category Rating =   3.7 

Design- Final Rating :

 Consultant: A & E Consultants, Inc., Project:  Reconstruction of Street/Bridge   Agreement No./ Contract No. 000000

Schedule Quality Project Management30% 60%            10%

WEIGHTED AVERAGEDesign Phase Ratings Design Preliminary:             3x 0.3=0.9      4x 0.6 =2.4           5x 0.1 =0.5           No. 1 

= 3.8Design Intermediate:  4x 0.3=1.2     3x 0.6 =1.8           5x 0.1 =0.5           No. 

2 = 3.5Design Final:             5x 0.3=1.5     5x 0.6 =3.0           4x 0.1 =0.4           No. 3 = 4.9 

QualityWEIGHTED AVERAGE 100%Construction Phase Ratings     Construction Phase 4 Rating No.:                       

                    Average Rating for Design Phase:  (3.8 + 3.5 + 4.9) /3 = 4.1Rating of Design post Construction:           4.0   FINAL DESIGN PROJECT Rating (4.1 x 70% = 2.9) + (4.0 x 30% = 1.2) = 4.1

Example of a FINAL rating for a design project

AVERAGE DISCIPLINE RATING

An average discipline rating for each consultant firm will be computed for all his projects within a discipline. Utilizing the example of the Deign Project, the Average Design Discipline rating of consultant XYZ above will be as follows:

 AVERAGE DESIGN DISCIPLINE RATING:

 Design Project No. 1 - Average Rating = 4.6Design Project No. 2 - Average Rating = 3.0Design Project No. 3 – Average Rating = 2.7*

  Average Design Discipline Rating  = 3.4 (Consultant XYZ)

* CAUTION: Any single Project Average Rating below SATISFACTORY may affect consideration of the consultant for future consulting services with the Department.

USE OF THE RATING SYSTEM

• Consulting firm’s performance with the Department over a three-year period will be assessed. 

• The ratings for past performance will be taken into account during Department review of the Expressions of Interest (EOI) for a new project.

   Ratings will be generated for each Consultant based on the firm’s ratings 

in the system: 

Final Project Average - a rating for each project. Discipline Average - an average rating of all projects within a discipline

USE OF THE RATING SYSTEM- (Continued)

   If a Consulting firm has no established ratings, the Department will apply 

the consultant average rating in other disciplines past performance during Department review of the Expressions of Interest (EOI).  The Consultant’s current rating will be used until the firm establishes other Discipline ratings.

 If a Consulting firm has not acquired ratings with the Department, he/she may receive the average of the ratings from the raters of his/her past work performance in other states or agencies for the applicable discipline until ratings by the firm are established.

CONSULTANT DEBRIEFING

• PLPM and the reviewer/supervisor will discuss the rating reports with the Consultant’s Project Manager (CPM) before they are submitted to the Chief Engineer. The consultant or his/her representative is not required to sign the Consultant Performance Evaluation Form (CPEF). 

• The consultant may request a rating debriefing within 30 days from receipt of the rating.  A written request by the Consultant should be made to the Chief Engineer, IPMA. 

• A debriefing meeting will be scheduled by the Chief Engineer or the Program Manager to discuss the rating. The Chief Engineer, the PLPM and/or his/her Supervisor, the CPM and/or Consultant Principal in Charge will be required to attend the meeting.

•  Any single Project Average Rating below SATISFACTORY may affect consideration of the consultant for future consulting services with the Department.

• For any rating lower than Satisfactory, the Consultant may be required to provide a letter indicating the manner in which the firm intends to resolve problems and issues concerning the rating and how it intends to prevent their recurrence on future projects.

Design Rating (Design Phase)Schedule (30%)

(5) Work under the phase is completed in advance of the agreed scheduled date. Projected schedule for deliverables or project completion can be accelerated.

(4)  Work under the phase is completed in advance of the agreed scheduled date. Projected schedule for deliverables or project completion cannot be accelerated.

(3)  Work under the phase is completed at the agreed scheduled date. Projected schedule for deliverables or project completion maintained but cannot be accelerated.

(2) Work under the phase is completed after the agreed scheduled date. Projected scheduled for deliverables or project completion maintained.

(3) Work is completed after the agreed scheduled date for critical date for critical path items. Projected schedule for deliverables or project completion delayed.

Design RatingDesign Phase (Cont.)

Quality (weighted 60% during design phase)

(5)  Changes were required for clarity of document presentation only. No technical errors and omissions that influenced the quality of the work.

(4) Documented errors and omissions were corrected upon notification. Not considered major error. A resubmission was not required.

(3)  Documented errors and omissions. Few critical errors identified that required resubmission.

(2) Documented errors and omissions. Significant errors identified. Resubmissions were required to correct the work. 

(1) Documented errors and omissions. Fatal critical errors. Required resubmission of the work and/or reassignment of work by the DDOT.

Design RatingDesign Phase (Cont.)PROJECT MANAGEMENT (10%)• Is organized and proficient with administrative, procedural and technical

 skills.• Performs the work of the project as required in the Scope of Services and as directed by the 

PLPM.• Supervises the progress of the work of his staff and that his Sub consultants.• Is proficient with verbal and written communication skills.• Is cooperative with the Department and /or joint operating agencies involved with the project.• Keeps the PLPM advised of general matters and also identifies and works to resolve problems 

that arise.• Is available for Department phone calls, e-mails and meetings.• Receives Department approval prior to making any changes to Consultant Contract Management 

or team structure established through the agency.

(5)  Has met all of the above requirements. No improvements needed.(4)  Above average performance, does not meet one of the above requirements.(3)  Average performance, does not meet two of the above requirements.(2) Below average performance, does not meet three of the above requirements.(1) Does not meet three of the above requirements and/or a change of the Consultant’s Contract 

Management is required by the Department.

DESIGN RATING (POST CONSTRUCTION PHASE)

(5)   There were no design errors or omissions in the plans and/or specifications that impacted the project schedule and/or cost.

 

(4)   There were minor design errors or omissions in the plans and/or specifications, which did not significantly impact the project schedule and/or cost (cost impact less than 0.5% of the project cost; schedule impact less than 2%).

 

(3)  There was one design error or omission in the plans and/or specifications, which impacted the project schedule and/or cost.  Cost impact less than 1% of the project cost; schedule impact less than 4%.

 

(2) There were one or more design error or omission in the plans and/or specifications, which impacted the project schedule and/or cost.  Cost impact less than 2.0% of the project cost; schedule impact less than 6%.

 

(1)  There were two or more errors or omissions in the plans and/or specifications, which impacted the project schedule and/or cost. Cost impact greater than 2.0% of the project cost; schedule impact greater than 6%.

 Category Weight 100%

 

PERFORMANCE EVALUATION(ENGINEERING SERVICES DURING CONSTRUCTION)

Consultants Responsiveness to the Needs of Construction:

The Consultant perform the engineering services for design related support functions such as, shop drawing review, response to construction questions on design issues, required paperwork, preparation of change of plans for unforeseen conditions, and other pertinent requests of the PLPM in a timely and professional manner.  Each support function requested of the Consultant shall be considered a “Task Assignment”.

  

PERFORMANCE RATING(ENGINEERING SERVICES DURING CONSTRUCTION Cont.)

Schedule 

The Department PLPM and the Consultant will mutually establish reasonable schedules for submissions/responses to each Task Assignment.  Each Task Assignment will be tracked (assignment date to completion date) by the PLPM.

 The Consultant’s deliverables should be completed on or before the agreed upon scheduled date(s). The rating will be based on ON-TIME submission of each assignment based on total assignments (i.e., if there are a total of 10 Task Assignments, of which 8 (80%) were completed by the Consultant on or before the agreed upon scheduled date, then the rating would be 4).

PERFORMANCE RATING(ENGINEERING SERVICES DURING CONSTRUCTION Cont.)

Schedule

5 = 90 to 100% of the time 

4 = 80 to 89% of the time 

3 = 70 to 79% of the time 

2 = 60 to 69% of the time 

1 = 0 to 59% of the time  

Category Weight 60%

PERFORMANCE RATING(ENGINEERING SERVICES DURING CONSTRUCTION Cont.)

QUALITY

(5) There were NO inaccuracies in technical review or presentation NO errors or omissions that influenced the quality of the work.

(4) There were MINOR documented inaccuracies in technical reviews, presentation, or errors and omissions that required clarification only. A resubmission of the documents was NOT required.

(3) There were documented inaccuracies in technical reviews, presentation, or errors and omissions.  One resubmission was required to correct the work.

 (2) There were documented inaccuracies in technical reviews, presentation, or errors and omissions.  Two resubmissions were required to correct the work.

 (1) There were documented inaccuracies in technical reviews, presentation, or errors and omissions after three resubmissions of the work and/or reassignment of work by the Department was  required.

  Category Weight 30%

PERFORMANCE RATING(ENGINEERING SERVICES DURING CONSTRUCTION Cont.)

Project Management

• is organized and proficient with administrative, procedural and technical skills.  • performs the work of the project as required in the Scope of Services and as directed 

by the PLPM.  • supervises the progress of the work of his staff and that of his Subconsultants.• is proficient with verbal and written communications skills.• is cooperative with the Department and/or joint operating agencies involved with the 

project.• keeps the PLPM advised of general matters and also identifies and works to resolve 

problems that arise.  • is available for Department phone calls and meetings.   • receives Department approval prior to making any changes to the Consultant’s Contract 

Management or team structure established through the agreement.• Completes work at the time frame noted in the Agreement with no impact on the 

construction cost or schedule.  

PERFORMANCE RATING(ENGINEERING SERVICES DURING CONSTRUCTION Cont.)

Project Management (Cont.)

5 Has met all of the above requirements.  No improvement needed. 

4 Above average performance, does not meet one of the above requirements. 

3 Average performance does not meet two of the above requirements. 

2 Below average performance, does not meet three of the above  requirements. 

1  Does not meet three of the above requirements and/or a change of the Consultant’s Contract Management is required by the Department.

   Category Weight 10% 

PERFORMANCE EVALUATIONBRIDGE MANAGEMENT AND INSPECTION CONSULTANT 

SCHEDULE:

  The work performed shall be in accordance with the various schedules as stated in the scope of work and the agreement. The rating will not reflect factors outside the Consultant’s control, such as delays by various authorities, which require right-of way permits etc.      

 RATING 

(5)  Field inspections completed in conformance with the approved schedule,  100% of the reports are submitted in conformance with the submittal schedule.

 (4)  Field inspections completed in conformance with the approved schedule, greater than 90% of the reports are submitted in conformance with the submittal schedule.

 (3) Field inspections completed in conformance with the approved schedule, greater than 80% of the reports are submitted in conformance with schedule.

 (2)  Field inspections are completed later than approved schedule and/or greater than 70% of the reports are submitted in conformance with schedule.

 (1)  Field inspections are completed later than approved schedule and inspection dates are not in compliance with NBIS. Final Reports are not accepted since bridge inspections were in violation of NBIS compliance.

Category Weight 30%

PERFORMANCE EVALUATIONBRIDGE MANAGEMENT AND INSPECTION CONSULTANT OVERALL QUALITY: Consultant Errors And Omissions/Corrective Work.

Based on the performance checklists on the PROJECT SUMMARY FORM, reports submitted should not require changes due to inaccuracies in Technical areas of the report or Consultant errors or omissions on the Structural Inventory and Appraisal Sheets and Pont is data. Corrective work should not require repeated submissions to the Department.

RATING  (5) Documented errors and omissions do not exceed 10% of the reports that were reviewed by the 

Department. No resubmission of final reports to the Department was required.  

(4) Documented errors and omissions do not exceed 20% of the reports that were reviewed by the Department. No resubmission of final reports to the Department was required.

 (3) Documented errors and omissions do not exceed 30% of the reports that were reviewed by the Department and these errors did not affect  the assessment of the structural condition of the structure. No resubmission of final reports to the Department was required.

 (2) Documented errors and omissions did exceed 30% of the reports that were reviewed by the Department and no more than 10% of the report contained errors that have affected the assessment of the structural condition of the structure. No resubmission of final report to the Department was required.

 (1) Documented errors and omissions affected the assessment of the structural condition more than 10% of the report. Resubmission of final reports to the Department was required.

Weight of Category 50%

PERFORMANCE EVALUATIONBRIDGE MANAGEMENT AND INSPECTION CONSULTANT PROJECT MANAGEMENT

This category rates the management and not the individual(s) serving in the position. Manages and reviews the work of his staff and sub-consultants to assure compliance with D.DOT requirements. The Consultant Project Management: The Consultant:- • is organized and proficient with administrative, procedural and technical skills.  • has properly staffed personnel in a cost effective manner to save money for the Department.• has properly managed the project by matching skills of personnel with the work assignments.• is proficient with verbal and written communications skills.• is cooperative with the Department and/or joint operating agencies involved with the project.• keeps the DCPM advised of general matters and also identifies and works to resolve problems that arise.  • is available for Department phone calls and meetings.   • receives Department approval prior to making any changes to the Consultant’s Contract Management or 

team structure established through the agreement.  • responds to emergency situations in a timely manner.

PERFORMANCE EVALUATIONBRIDGE MANAGEMENT AND INSPECTION CONSULTANT PROJECT MANAGEMENT (Cont.)

(5) Has met all of the above requirements.  No improvement needed.

 (4) Above average performance, does not meet one of the above requirements.

 (3) Average performance, does not meet two of the above requirements.

 (2) Below average performance, does not meet three of the above requirements.

 (1) Does not meet three of the above requirements and/or a change of the Consultant’s Contract Management is required by the Department.

  

Category Weight 20%

PERFORMANCE EVALUATION CONSTRUCTION MANAGEMENT /INSPECTION CONSULTANT

OVERALL QUALITY:

Based on the CONSTRUCTION MANAGEMENT /INSPECTION - Overall Quality Checklist responses and using the rating system noted below an overall rating is determined.

 RATING

 (5) Consultant received 100% “yes” or N/A ratings.

 (4) Consultant received between 85% and 99.9% “yes” ratings.

 (3)  Consultant received between 84.9% and 75% “yes” ratings.

 (2) Consultant received between 74.9% and 65% “yes” ratings.

 (1) Consultant received 64.9% and below “yes” ratings.

  

Category Weight 90%

Project:__________________________________________________D.DOT Manager:_______________________ Date: ____________Reviewer:_______________________________ Date: ___________

Consultant Name:_______________________________Period Covered:_________________________________Report No.______

Yes No N/A 

1. Has Consultant properly staffed the project? ________ overstaffed _________ understaffed 

2. Does the Consultant Resident/Chief Inspector/ Project Manager adequately supervise his staff? (No direct supervision of staff required by the Department’s representative on the project.)

 

3 Does the Consultant Resident/Chief Inspector/Project Manager check work of his staff for quality review? 

4. Are Consultant staff, assigned to the project, adhering to the applicable construction inspection procedures? 

5. Does Consultant enforce safety requirements for his staff?  6. Are Consultant staff, assigned to the project, familiar with the plans / specifications for the project?

 

7. Are Consultant staff, assigned to the project, adhering to D.DOT materials testing procedures? 

8. Does the Consultant Resident/ Chief Inspector, Project Manager enforce contract requirements? 

9. Does Consultant review claims, extra work, time extensions?  10. Does Consultant recommend sound technical solutions to problems?  11. Are Daily Reports current (entered within 1 day)?  12. Are Weekly Reports submitted on time (within two days)?  13. Are field diaries being maintained on site and available for review?  14. Does Consultant monitor environmental concerns?  15. Does Consultant monitor DBE/MBE/EEO and subcontract requirements?  16. Is Consultant staff equipped with appropriate skills for inspection?  17. Are correspondences and written reports prepared within the specified time and DOES NO require corrections?

 

18. Does Consultant review plans in advance and anticipate problems?  19. Are meeting minutes promptly prepared (within 3 days) and complete?  20. Are Contractor Payment estimates submitted within 2 days of specified date?  21. Are quantity measurements accurate and submitted within the specified time ?  22. Does the Consultant ensure that proper traffic devices are used?  23. Does Consultant coordinate the work of utilities and/or adjacent construction with the contractor?

 

24. Is project close-out performed within the specified time?  25. No construction errors or omissions occurred in the last six months.  

TOTAL # OF RATINGS  

  # of Yes Ratings x 100 = PERCENTAGE YES ____%  

  25 - # of N/A Ratings  

CONSTRUCTION MANAGEMENT / INSPECTIONOverall Quality Checklist

PERFORMANCE EVALUATION CONSTRUCTION MANAGEMENT /INSPECTION CONSULTANT

Project Management:

This category rates the management and not the individual(s) serving in the position.

Manages and reviews the work of his staff and sub-consultants to assure compliance with DDOT requirements.  The Consultant:- • Is organized and proficient with administrative, procedural and technical skills.• Is proficient with verbal and written communications skills.• Has properly staffed personnel in a cost effective manner to save money for the Department.• Has properly managed the project by matching skills of personnel with the work assignments.• Is cooperative with the Department and/or joint operating agencies involved with the project.•  Is available for Department phone calls and meetings.• Keeps the Department Field Manager or Resident Engineer advised of general matters and also identifies and works 

to resolve problems that arise.• Receives Department approval prior to making any changes to the Consultant’s Contract Management or team 

structure established through the agreement.• Responds to emergency situations in a timely manner.

PERFORMANCE EVALUATION CONSTRUCTION MANAGEMENT /INSPECTION

PROJECT MANAGEMENT RATING (Cont.)

5 Has met all of the above requirements. 

4 Above average performance, does not meet one of the above requirements. 

3          Average performances, does not meet two of the above requirements. 

2 Below average performance, does not meet three of the above requirements. Inspection personnel required improvement and/or training to meet the  requirements of the agreement.

 1 Does not meet three of the above requirements and/or a change of the Consultant’s Contract Management is required by the Department.  CPM had to  be called more than two times to resolve issue(s) concerning performance of  inspection personnel.

 Category Weight 10%

Contact information

Muhammed Khalid, P.E.Program ManagerDistrict Department of Transportation 55 M Street, S.E.4th FloorWashington, DC 20003202-671-4577

Questions !