ICT Programme Operations Unit

Information and Communications Technologies

How to fill in the IER formICT Calls 2013

This tutorial gives guidelines on how to carry out the “remote” stage of an ICT evaluation

• How to assess the proposal• Scoring the proposal • Filling in and returning the Individual

Evaluation Form (IER)

Introduction

You have signed a non-disclosure agreement• Before, during and after this evaluation you do not

disclose any information about the proposals which you have seen

• Keep any printed copies secure when not in use; bring them with you to Brussels

You are the evaluator• Do not pass this responsibility to anybody else

• Do not discuss the proposals with anybody else

But first, remember your responsibilities

Do not contact the proposers• Evaluate only the proposal as it was submitted to

us at the call deadline, without any later additions or clarifications from the proposer

• The identities of evaluating experts are never revealed to the proposers concerned

Declare any potential conflict of interest • If there is a situation which might prevent you

evaluating a proposal impartially, click on “I cannot evaluate this proposal” (see Rivet manual page 9)

Your responsibilities

For the remote stage of this evaluation we are using a software tool “borrowed” from elsewhere in the Framework programme – Rivet

• In Rivet the Individual Evaluation Report (IER) is called the Individual Assessment Report (IAR), but the meaning is the same

• Save your work regularly, unsaved work is lost if you are idle too long and Rivet times you out. If you are reading, thinking or even typing Rivet believes you are idle. Only saving is considered to be an activity. Save your form at least every 20 minutes !

The Rivet tool

Proposals are evaluated on three criteria onlyScientific and technical quality ImplementationImpact

Assess the proposal in terms of all three criteria• Each criterion is more fully defined by descriptive “bullet

points”, adapted to the project type. These are shown on the evaluation form

• Provide a comment for each of the “bullet points” (but the bullet points themselves are not scored individually or separately)

The Evaluation Criteria

First develop your comments on each criterion……then select scores accordingly • Each criterion is given a score out of five, corresponding to

the explanatory comments• A threshold of 3/5 is applied on each criterion• An overall score is calculated for each proposal by simple

addition• A threshold of 10/15 is applied on the overall score

Out of scope proposals are given low scores on Criterion 1 “Scientific and technical quality”

(FET objectives have different thresholds and a weighting scheme; See the FET Proactive evaluation forms in the “Guidance notes for evaluators”)

The scoring scale

The scoring scaleUse the full scale! Half marks may be given

0 -The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information

1 -Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses

2 -Fair. While the proposal broadly addresses the criterion, there are significant weaknesses 1

3 -Good. The proposal addresses the criterion well, although improvements would be necessary 2

4 -Very good. The proposal addresses the criterion very well, although certain improvements are still possible 2

5 -Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor

1 Problems which can’t be solved in grant agreement negotiations2 Your comments must identify the required improvements

How to fill in IER - General

Your comments• Comments are confined only to the criterion

concerned• Comments describe only your final view of the

proposal• Comments are clear and unambiguous. Try to avoid

obscure acronyms and technical terms• Comments are of adequate length and cover all the

bullet points under each criterion• Comments provide full justification for the score

given

Comments are substantial; do not write generic criticisms; be specific, explain

Comments are facts not opinions, don’t show doubt or indecision

• not “I don’t understand why....”

• but “The proposers do not make clear why.. .”

Poor comments include words like:• “Perhaps, think, seems, assume, probably, …”

Good comments include words like:• “Because, percent, specifically, for example, …”

IER – Be factual

Poor comments are vague - Good ones are precise

•“I think the consortium management plan is probably inadequate.”

•“The consortium management plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners.”

•“The resources for the project seem unrealistic.”•“The resources in Workpackages 4 and 6 are seriously

underestimated given the complexity of the activities involved.”

IER – Give Clear Messages

IER – Avoiding a dispute

Poor comments provide an opening for a debate - Good comments close the issue

• “There is no discussion of dissemination activities.”• “Dissemination activities are not adequately

discussed.”

• “There is only one end-user organisation in the consortium.”

• “The consortium lacks a sufficient participation of end-users.”

• “The proposal coordinator is not adequately experienced.”

• “The proposal coordinator does not demonstrate in this proposal an adequate level of experience of work in this field.”

IER – Varying the Vocabulary

Why say “Poor” when you can say:• Insufficient, minimal, fails to describe, unacceptable,

inadequate, very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile…

Why say “Excellent” when you can say:• Extremely relevant, credible, very clear, precisely specified,

realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …

IER – Final Check

• Have you fully explained the proposal’s strengths and weaknesses on each of the criteria ?

• Do your scores match your comments (high scores = positive comments, low scores = negative comments)?

• Have you double-checked any matters-of-fact which you have quoted?

• Have you written at adequate length?

• If this was your proposal, would you find this report fair, accurate, clear and complete?

ICT Evaluations

Submitting your IERs

Submitting your IERs

• First, read over all your proposals to get an idea of the general standard and content

• When you have completed the evaluation of each proposal, submit your IER (IAR) for that proposal in the Rivet tool - Don’t wait until you have evaluated all your proposals before starting to submit them

• You will see that the Rivet tool has the facility to create Consensus Reports CRs and Evaluation Summary Reports ESRs. We will not use this in this evaluation; you stop at the creation of IERs (IARs)

Submitting your IERs

• Please respect the deadline you have been given for completing and submitting your forms

• Don’t worry if, after submitting your IER, you would like to add to or modify your comments. You will anyway get a chance to discuss your opinions with the other experts in the meetings in Brussels

• Bring your copies of the proposals and a copy of your IERs with you to Brussels. In the meantime, keep them safe!

Give a fair and clear opinion on each proposal. You are:• Independent : you represent yourself, not your employer,

not your country…..)• Objective : you evaluate the proposal as written• Accurate : you use the official evaluation criteria only• Consistent : you apply the same standard of judgment to

each proposal• Incommunicado : you do not disclose to anybody the

contents of the proposals which you see

Finally

Finally

Thank you very much for your help!

- The ICT evaluation team