CS357: Foundations of Artificial Intelligence - Lab 2 Checkpoint: RAG Quality Checkup (100 Points)
Purpose, Task, and Criteria
Purpose: To run a structured mid-flight diagnostic on your in-progress RAG pipeline — retrieval quality measured, citations audited, one failure understood — and to freeze that diagnostic into a rerunnable regression harness that follows you into Lab 5.
Task: Complete the studio worksheet on your own Lab 2 pipeline — recall@k across two chunking configurations, a citation audit, and one analyzed failure — then pin a golden set into a rerunnable regression harness and demonstrate reproducibility.
Criteria: Assessed on a completed worksheet with real measurements from your own pipeline, and a regression harness that reruns identically; see the rubric below for the full breakdown.
Assignment Goals
The goals of this assignment are:- To measure retrieval quality (recall@k) empirically across two chunking configurations on your own corpus
- To audit citations by hand, classifying each answer's claims as supported or unsupported by the cited chunk
- To capture and analyze one concrete pipeline failure with a hypothesis and planned fix
- To freeze an evaluation into a rerunnable regression harness and demonstrate its reproducibility
Background Reading and References
Please refer to the following readings and examples offering templates to help get you started:- Hallucinations and Evaluating Agent Outputs Activity
- RAG Quality: Chunking, Clustering, and Reranking Activity
- Required prep for Lab 5: Testing Agents — Evaluation, Regression, and the Non-Determinism Problem
- Reference: promptfoo — declarative LLM/agent eval and red-teaming (harness option, runs against Ollama)
The Assignment
This small lab is a structured checkup on the RAG pipeline you are building in Lab 2, begun during the studio session and finished on your own — budget two to three hours total, most of it in class. It is deliberately a scaffold: everything you produce here is work Lab 2’s evaluation asks for anyway, done earlier and under supervision, plus one forward investment — a regression harness you will be glad to already have when Lab 5 asks you to build a reproducible evaluation pipeline.
Work on your own pipeline and corpus (this lab is individual, though the studio makes collaboration on debugging natural). Bring your Lab 2 repository to the studio session running.
See the course schedule for the assigned and due dates.
Part 1: The Checkup Worksheet (60 points)
Complete checkup.md against your in-progress Lab 2 pipeline:
- Retrieval quality. For five queries representative of your corpus, measure recall@k under your current chunking configuration and one alternative (different chunk size or overlap). Record the table and name the winner — and carry the winning configuration back into Lab 2.
- Citation audit. For five answered queries, one row each: the answer’s central claim, the chunk it cites, and your verdict — supported (the chunk actually contains the claim) or unsupported. The Lab 2 rubric weights citation quality heavily; this is that muscle, built early.
- One failure. Capture one concrete misbehavior you observed — a wrong retrieval, an unsupported citation, a failed abstention — with the query, the output, a one-paragraph hypothesis for the mechanism, and your planned fix.
Part 2: The Regression Harness (40 points)
Freeze your evaluation so it can be rerun forever:
- Pin a golden set: start from your Golden-Set Benchmark items, replace or extend with at least five corpus-specific questions (including at least one that should trigger abstention).
- Build the harness in your choice of medium: plain Python grown from the class evaluation harness, or declarative promptfoo YAML run against your local Ollama — the same tool that anchors Lab 5’s low-code direction, so either choice pays forward.
- Pin the protocol (temperature 0.0, fixed seed, model recorded), run it twice, and show the runs agree.
- Commit the harness and golden set inside your Lab 2 repository.
Deliverables
Submit checkup.md (worksheet with real measurements), the harness (script or YAML) with its golden set, and the two-run agreement log — all committed in your Lab 2 repository, with paths noted in the submission.
Grading Breakdown
| Component | Points |
|---|---|
| Part 1: Checkup Worksheet | 60 |
| Part 2: Regression Harness | 40 |
| Total | 100 |
Reflection Prompts
- Which chunking configuration won on your corpus, and did the margin surprise you?
- What did the citation audit reveal that the recall@k numbers alone would have hidden?
- AI disclosure: list any generative-AI tools you used, for what, and how you verified the results (or state ‘none’).
- Approximately how many hours it took you to finish this lab (I will not judge you for this at all — I am simply using it to gauge if the labs are too easy or hard)?
Submission
In your submission, please include answers to any questions asked on the assignment page, as well as the questions listed below, in your README file. If you wrote code as part of this assignment, please describe your design, approach, and implementation in a separate document prepared using a word processor or typesetting program such as LaTeX. This document should include specific instructions on how to build and run your code, and a description of each code module or function that you created suitable for re-use by a colleague. In your README, please include answers to the following questions:- Describe what you did, how you did it, what challenges you encountered, and how you solved them.
- Please answer any questions found throughout the narrative of this assignment.
- If collaboration with a buddy was permitted, did you work with a buddy on this assignment? If so, who? If not, do you certify that this submission represents your own original work?
- Please identify any and all portions of your submission that were not originally written by you (for example, code originally written by your buddy, or anything taken or adapted from a non-classroom resource). It is always OK to use your textbook and instructor notes; however, you are certifying that any portions not designated as coming from an outside person or source are your own original work.
- Approximately how many hours it took you to finish this assignment (I will not judge you for this at all...I am simply using it to gauge if the assignments are too easy or hard)?
- Your overall impression of the assignment. Did you love it, hate it, or were you neutral? One word answers are fine, but if you have any suggestions for the future let me know.
- Using the grading specifications on this page, discuss briefly the grade you would give yourself and why. Discuss each item in the grading specification.
- Any other concerns that you have. For instance, if you have a bug that you were unable to solve but you made progress, write that here. The more you articulate the problem the more partial credit you will receive (it is fine to leave this blank).
Assignment Rubric
| Description | Pre-Emerging (< 50%) | Beginning (50%) | Progressing (85%) | Proficient (100%) |
|---|---|---|---|---|
| The Checkup Worksheet (Goals 1-3) (60%) | The worksheet is empty or filled with invented numbers not produced by the student's pipeline | Some measurements exist but only one chunking configuration was tested, or the citation audit is missing | recall@k for both configurations and a five-row citation audit are complete, but the failure case is missing or has no hypothesis | recall@k is measured for both chunking configurations on the student's own corpus with the winner identified; the five-row citation audit classifies each claim as supported or unsupported with chunk references; and one observed failure is recorded with a plausible hypothesis and a planned fix |
| The Regression Harness (Goal 4) (40%) | No harness exists, or it cannot be rerun | A harness exists but the golden set is not pinned (questions or scoring change between runs), or it was run only once | The harness reruns with a pinned golden set and protocol, but the two runs' outputs were not compared, or the harness is not committed alongside the Lab 2 work | The harness — plain Python built on the class evaluation harness, or declarative promptfoo YAML, student's choice — pins a golden set (seeded from the Golden-Set Benchmark lab, extended with corpus-specific items) and a fixed protocol; two runs are shown to agree; and the harness lives in the Lab 2 repository where Lab 5's pipeline work can pick it up |
Please refer to the Style Guide for code quality examples and guidelines.