View on GitHub

CS357

Foundations of Artificial Intelligence

CS357: Foundations of Artificial Intelligence - Lab: Rubric Pipeline, Direction 0: The promptfoo Route

Purpose

To meet the core Rubric Pipeline Lab objectives (batch rubric scoring, human-agreement validation, bias measurement, and regression testing) with promptfoo configuration files and local Ollama instead of Python.

Background Reading and References

This page is Direction 0 of the Rubric Pipeline Lab, the low-code route through the lab. You meet the same core objectives (batch rubric scoring, human-agreement validation, bias measurement, and regression testing) by writing YAML configuration for promptfoo instead of Python code. promptfoo is an evaluation harness: you describe prompts, a model, test cases, and assertions in a file, and it runs them all with one command. Its llm-rubric assertion is an LLM as judge, a language model prompted to decide whether an answer meets a stated criterion rather than to write the answer itself.

Direction 0 replaces the coding in core Parts 1-4. If you choose this route, you do not build the Python pipeline; you build the same measurements out of promptfoo configs. Core Part 5 (expressing the judge as a versioned, declarative harness) is satisfied by the route itself, because the entire route is a declarative harness. You are graded under the same 100-point rubric on the core lab page. That rubric’s wording is pathway-neutral, and this page tells you what each row means on this route. As with the core lab, you complete this route in pairs using driver/navigator roles, with swaps at least every 30 minutes and a swap log; the blind human-scoring step in Part C requires both partners.

See the course schedule for the assigned and due dates. Expect 7-9 hours total for this route. It replaces the core coding, so this is your entire Rubric Pipeline Lab time, not an add-on.

What this direction requires

  • Node.js and npm (install from https://nodejs.org); promptfoo runs via npx, so nothing else needs a global install
  • Local Ollama with the llama3.2 model, as in the earlier labs
  • No API key. Everything on this route runs locally.

Part A: Install promptfoo and Point It at Ollama

Estimated time: 30-45 min

Initialize a project directory:

mkdir cs357-lab5-d0
cd cs357-lab5-d0
npx promptfoo@latest init

The init command scaffolds a promptfooconfig.yaml. Confirm promptfoo can reach your local Ollama with a minimal smoke-test config. Replace the scaffolded file’s contents with:

# smoke-test promptfooconfig.yaml - verify Ollama connectivity
description: "Rubric Pipeline Lab Direction 0 smoke test"

prompts:
  - "Reply with exactly one word: the capital of France."

providers:
  - ollama:chat:llama3.2

tests:
  - assert:
      - type: contains
        value: "Paris"

Run it:

npx promptfoo@latest eval
npx promptfoo@latest view   # opens the results viewer in your browser

Expected: 1 test, 1 pass. If promptfoo cannot reach Ollama, confirm ollama serve is running and that curl http://localhost:11434/api/tags responds. If your Ollama runs elsewhere, promptfoo reads the base URL from the OLLAMA_BASE_URL environment variable.

Part B: Batch Rubric Scoring with llm-rubric

Estimated time: 2-2.5 h (most of it authoring the rubric and the 15 answers)

This part is the declarative counterpart of the pipeline in core Parts 1-2: a rubric, a corpus of sample answers, and a judge that scores every answer against every criterion in one command. A rubric is a list of criteria, each with observable levels, written so that two graders reading the same text land on the same score.

Step 1: Write your rubric. Exactly as in the core lab, write a rubric with four criteria and observable level descriptors for a short artifact type of your choosing (persuasive paragraph, function docstring, lab abstract). You will embed each criterion’s text into an llm-rubric assertion below. For each criterion the judge’s job is binary: PASS if the answer sits at level 3 or 4 of that criterion, FAIL if at level 1 or 2. Write each criterion’s rubric text so that this pass line is explicit and observable.

Step 2: Write your dataset, 15 sample answers. Create dataset.csv with one row per answer, spanning the full quality range. This is like the core lab’s 12 synthetic submissions plus three more: include at least one empty-ish answer, one off-topic answer, and one verbose-but-weak answer. All answers are synthetic; no real student work.

id,answer
a01,"<your excellent answer here>"
a02,"<your good answer here>"
a03,"<your adequate answer>"
...
a15,"<your off-topic answer>"

Before running anything, record privately which criteria you intend each answer to pass. You need this for Part C.

Step 3: Build the baseline scoring config. This skeleton is complete except for your rubric text. Copy it to promptfooconfig-baseline.yaml and fill in the four value: blocks with your criterion descriptors:

# promptfooconfig-baseline.yaml
# Validated against: ollama llama3.2, temperature 0
description: "Rubric Pipeline Lab Direction 0: baseline rubric-as-judge batch scoring"

prompts:
  # The echo provider passes each stored answer straight through as the "output",
  # so the llm-rubric judge grades your dataset answers, not a freshly generated one.
  - "{{answer}}"

providers:
  - echo

defaultTest:
  options:
    # The judge model that grades every llm-rubric assertion:
    provider: ollama:chat:llama3.2
  assert:
    - type: llm-rubric
      value: >
        Criterion C1 (<your criterion name>): PASS only if <your level 3-4
        descriptor, stated observably, what must visibly be present in the
        answer>. FAIL otherwise.
    - type: llm-rubric
      value: >
        Criterion C2 (<name>): PASS only if <level 3-4 descriptor>. FAIL otherwise.
    - type: llm-rubric
      value: >
        Criterion C3 (<name>): PASS only if <level 3-4 descriptor>. FAIL otherwise.
    - type: llm-rubric
      value: >
        Criterion C4 (<name>): PASS only if <level 3-4 descriptor>. FAIL otherwise.

tests: file://dataset.csv

Step 4: Run the batch and capture the output.

npx promptfoo@latest eval -c promptfooconfig-baseline.yaml --output run_baseline.json
npx promptfoo@latest view

Expected: 15 rows × 4 assertions = 60 judge verdicts. Save run_baseline.json; it is a deliverable, and Part E diffs against it. In the viewer, spot-check three items: does the judge’s reasoning for each verdict reference the actual answer text? Note one example where its reasoning is weak or generic. That note is this route’s version of the core lab’s evidence-faithfulness concern, and it feeds your Part C disagreement analysis.

Troubleshooting: if every assertion passes (or fails), your rubric text is not discriminating. Sharpen the PASS line with countable features (“names an opposing view in a full sentence AND rebuts it with a reason”); this is the observability lesson from the core lab. If the judge output is erratic, re-run. llm-rubric verdicts from a small local model are noisy, and observing that noise is a legitimate finding for your writeup.

Part C: Human Agreement, Blind Scoring and Percent Agreement

Estimated time: 1.5-2 h

Human agreement is the rate at which independent scores match: human to human, or human to judge. Before looking at any judge output, each partner independently hand-scores all 15 answers, criterion by criterion, as PASS (level 3-4) or FAIL (level 1-2). The order matters for the same reason as in the core lab: seeing the judge’s scores first anchors yours.

The score spreadsheet. Build a spreadsheet (Google Sheets, Excel, or CSV) with exactly these columns:

Column Contents
item_id a01 … a15
C1_human_A, C2_human_A, C3_human_A, C4_human_A Partner A’s blind P/F per criterion
C1_human_B, C2_human_B, C3_human_B, C4_human_B Partner B’s blind P/F per criterion
C1_judge, C2_judge, C3_judge, C4_judge The judge’s P/F per criterion (filled in after both humans finish, from run_baseline.json)
mismatches count of criteria where the human-consensus verdict differs from the judge
notes one line on any disputed item

Where you and your partner disagree with each other, record both marks and discuss until you reach a consensus verdict for the judge comparison. Do not erase the original marks; human-to-human disagreement is a finding, just as in the core lab.

Compute percent agreement (no code needed; a spreadsheet formula or a calculator is fine):

percent agreement = (number of criterion-cells where human consensus == judge verdict)
                    ÷ (15 items × 4 criteria = 60 cells) × 100

Report percent agreement overall and per criterion (each criterion has 15 cells).

Kappa replacement. The code route computes Cohen’s kappa, which is percent agreement corrected for the matches two scorers would reach by chance. On this route, kappa is not required. Two things replace it: (1) the percent-agreement numbers above, and (2) a written disagreement analysis of the 3 worst mismatches, meaning the three items with the highest mismatches count. For each of the three, quote the answer, state the human verdict and the judge verdict per disputed criterion, quote the judge’s stated reasoning from the promptfoo output, and diagnose why they diverged: is the criterion’s wording ambiguous, is the judge pattern-matching on surface features, or did the humans read something into the answer that is not on the page? Then revise the wording of the single worst criterion’s rubric text, re-run the baseline config, and report that criterion’s per-criterion agreement before and after. This is the same revise-and-remeasure discipline as core Part 2.

Part D: Bias Tests as Config Variants

Estimated time: 1-1.5 h

A bias measurement is a controlled experiment that changes one feature the rubric does not care about and records how far the verdicts move. The core lab measures one judge bias with a Python experiment; here you measure two, with two more config files over the same dataset.

Variant (i), reordering (position/order effects). Create dataset_reordered.csv: the same 15 answers with the row order reversed (a15 first, a01 last). Copy the baseline config to promptfooconfig-reordered.yaml, changing only the last line to tests: file://dataset_reordered.csv. Each answer is judged in isolation, so a trustworthy judge should give identical verdicts regardless of order. Any per-item verdict that flips between runs is evidence of order sensitivity or judge instability, which in a real grading deployment is position bias against whoever gets graded late.

Variant (ii), padding (verbosity bias). Create dataset_padded.csv: the same 15 answers, each with three filler sentences appended (substantive-sounding but vacuous, e.g., “Furthermore, it is important to consider the various aspects of this topic from multiple perspectives.”). Copy the baseline config to promptfooconfig-padded.yaml, changing only the tests line. Padding adds no rubric-relevant content, so verdicts should not improve; any criterion that flips FAIL->PASS under padding is verbosity bias.

Run both and keep the outputs:

npx promptfoo@latest eval -c promptfooconfig-reordered.yaml --output run_reordered.json
npx promptfoo@latest eval -c promptfooconfig-padded.yaml --output run_padded.json

Compare the three runs in this table (one row per item; a deliverable in your writeup):

item_id baseline passes (of 4) reordered passes (of 4) padded passes (of 4) reorder delta padding delta
a01          
…          
Total flips          

Then do four things. Summarize each bias in one sentence with a number (e.g., “padding flipped 5 of 60 verdicts FAIL->PASS, an 8% verbosity effect”). Propose one countermeasure you can express in the rubric text of the config (e.g., “Length and repetition must not be credited; judge only the content that addresses the criterion”). Apply it and re-run the padded config. Report the residual effect. This mirrors core Part 4’s countermeasure-and-residual-risk discipline.

Part E: Regression, Change the Judge, Diff the Runs

Estimated time: 45-60 min

This route satisfies the harness discipline of core Part 5 by construction; now demonstrate it. Make a deliberate, plausible-seeming degradation to the judge prompt: in a copy of the baseline config (promptfooconfig-regressed.yaml), weaken one criterion’s rubric text (for example, delete the observable PASS line and leave only the criterion name). Re-run and diff:

npx promptfoo@latest eval -c promptfooconfig-regressed.yaml --output run_regressed.json

Compare run_baseline.json and run_regressed.json; the viewer’s side-by-side, a text diff, or a hand-built table of the 60 verdicts all work. Identify which items’ verdicts changed and in which direction. In two or three sentences in your writeup, interpret the result. This is what a versioned eval configuration buys you: a tripwire that catches a silent judge-quality regression before it grades anything real. Then revert the change and confirm the baseline verdicts recover.

Part F: Writeup

Same writeup as the core lab. Complete the core lab page’s Learning Log (all six prompts plus the lab-specific prompts) in your readme, citing your percent-agreement numbers, bias flip counts, and regression diff as the specific evidence. Approximately two pages.

Deliverables (Direction 0)

Submit a ZIP containing:

How the rubric reads on this route

You are graded under the same rubric as everyone else, on the core lab page. On this route, each row maps as follows. Pipeline Implementation is your baseline config scoring all 15 items end-to-end. Human Agreement Validation is the blind spreadsheet plus percent agreement and the criterion revision. Bias Measurement is the two config variants with the comparison table and countermeasure. Evidence Verification is the judge-reasoning faithfulness check inside your 3-worst-mismatches analysis. Reproducible Eval Harness is the regression demonstration of Part E. Writeup is Part F.

Troubleshooting

Symptom Likely cause Fix
npx promptfoo cannot find a provider The provider string is wrong, or Ollama is not running Confirm curl -s http://localhost:11434/api/tags answers, then use ollama:chat:llama3.2 exactly
Every case fails with an empty output The prompt template references a variable that is not in the test case promptfoo does not error on an unknown variable; it substitutes nothing. Check that every {{ }} name matches a vars key
llm-rubric verdicts look random The grading model and the model under test are the same, and the rubric is vague Sharpen the rubric’s level descriptors first. If verdicts stay unstable, say so and quantify it; that instability is a real finding about LLM-as-judge
Percent agreement is suspiciously high The partners scored together, or one saw the judge’s output first Rescore blind. The value of this part is entirely in the independence
Results change between identical runs Temperature is not pinned in the config Set temperature: 0 under the provider config, not only in the prompt
The regression diff shows everything changed The two runs used different datasets or provider settings Change exactly one thing between runs; that is the whole method
A YAML config will not load An unquoted colon or a tab character Run the file through a YAML validator. A bare : inside an unquoted string is the usual culprit

Self-Check Before You Submit

Held against the core lab’s rubric, read through this route.