Practice Applying a Published Criterion

The Practical Check

Calibration helps reviewers interpret the same criterion consistently. It does not make judgment mechanical or guarantee identical ratings. Start from the actual published wording and any defined descriptors. Identify evidence, gaps and the reason for a judgment. Do not add a new preference after seeing an application. This exercise uses no numeric scale: its task is to compare the reasons, not invent a universal scoring rubric. In a real process, follow the permitted clarification, conflict and confidentiality rules and obtain specialist advice for matters outside reviewer competence.

Your Starting Facts

Fictional call criterion: The approach is feasible with the proposed staff, timetable, budget and access arrangements. The call does not reward larger reach in this criterion and supplies no preference for a particular organizational size. Its process permits equivalent factual clarifications through the secretariat. The excerpts below are complete for this exercise; missing details are genuinely not supplied. No real application, organization or award is represented.

Worked Example

Excerpt A: Two trained staff will run six sessions over three months. Their time and travel are budgeted; the sequence fits the supplied timetable. The proposed venue permission is pending. A named backup venue is mentioned, but its availability is not evidenced. Excerpt B: Our organization has ten years of experience and is highly trusted. We intend to reach many people quickly. No staffing, budget, timetable or access plan is supplied in the excerpt. Excerpt C: One trained staff member will run three sessions over three months at a location with documented permission. Staff time, travel, materials and the sequence are shown and internally consistent. The smaller quantity is the applicant's actual proposed scope, not a shortfall against a call target.

Try the Judgment Exercise

Which calibration note best applies the supplied criterion?

C has clearer feasibility evidence; A has several supported elements with a consequential access uncertainty; B's experience claim does not supply a delivery plan. Clarify consistently where the real process permits it.B should lead because ten years of experience guarantees feasibility; C should lose marks because it reaches fewer people.A must be rejected automatically and C must receive the maximum score because confirmed access resolves every risk.

Use the Checking Prompt

Copy this prompt into your own AI tool with non-sensitive inputs, then check the answer yourself. OpenAid does not send this text to an AI service.

Manual calibration practice: read the actual published criterion and approved non-sensitive application excerpts. For each excerpt, identify supporting evidence, missing information, consequential uncertainty and any clarification permitted by the process. Explain a reasoned judgment without adding criteria, inventing facts or assuming an award. Distinguish factual correction, needed specialist expertise and defensible interpretive disagreement. If using an approved AI tool, use public or fictional material only and independently check the output; the tool does not decide the rating.

CRITERION AND DESCRIPTORS:
[exact approved text]

EXCERPTS:
[fictional or approved non-sensitive text]

PERMITTED PROCESS:
[actual scope and limits]

Continue With the Resource

Understand What Scores Can Tell a Panel

Optional Facilitator Note

Ask participants first to compare A and C independently, then state the exact reason for any difference. Discuss what would change if A supplied a dated venue permission, and what would change if C's staff time were only an unconfirmed intention. Use the existing evidence rubric to retain criterion wording and reasons; this exercise adds contrasting interpretation practice rather than a new scoring form.

Work Product: A short evidence-based comparison for the three excerpts and one unresolved question for A. There is no required form or professional-competence score. Preserve differing defensible views on A's risk rather than forcing identical labels.

Debrief and Reasons

Can two reviewers disagree reasonably about A?
Yes, where the real descriptors allow judgment about a pending but potentially resolvable access dependency. Both must identify the same supplied facts and explain the consequence. A factual claim that permission is already confirmed would be wrong, not merely a different tolerance.
May the panel add a preference for more sessions during this discussion?
No such preference is supplied for this criterion. A genuine concern about the criterion belongs in an authorized process review; it cannot become a private current-round scoring rule.
What does a clarified venue permission establish?
It may resolve the specified access uncertainty for its actual period and conditions. It does not verify all staffing, cost, technical-safety or outcome claims. Reassess the affected judgment without treating one document as a universal guarantee.

Source and Scope

First-Level Peer Review
Peer Review Criteria and Considerations, updated April 28, 2026. US biomedical review example only; its scale and regulations are not adopted. Fictional criterion, excerpts and feedback are original OpenAid teaching material.
Application Review Process
Programmatic review paragraph. US agency context; it does not prescribe this fictional call's rating or clarification process.

Fictional examples for practice. Completing an exercise does not certify competence.

From OpenAid · www.opendevelopmentaid.org/learn/calibrate-a-published-criterion-using-contrasting-applications · free to use with credit

Calibrate a Published Criterion Using Contrasting Applications