CS357: Foundations of Artificial Intelligence - Lab 2: A RAG Knowledge Base of Your Own (100 Points)

Purpose, Task, and Criteria

Purpose: To give an agent you run grounded, citable memory over a corpus you care about, including the honesty to abstain when the answer is not there.

Task: Build a RAG pipeline over your own corpus with Chroma and a local model, compare chunking strategies by recall@k, and audit its citations and abstention.

Criteria: Assessed on the citing, abstaining pipeline, an empirical chunking comparison, and a hand-audited evaluation of retrieval and citations; see the rubric below for the full breakdown.

Assignment Goals

The goals of this assignment are:

To construct a complete retrieval-augmented generation pipeline over a personal document corpus using Chroma and a local model
To make and defend chunking decisions empirically by comparing at least two strategies with recall@k metrics
To evaluate retrieval quality with recall at k and grounding quality with a citation audit
To implement and demonstrate honest abstention when the corpus does not contain an answer
To apply parameter-efficient fine-tuning (LoRA/QLoRA) to a local model using a real dataset
To instrument training with loss curves and evaluate output quality before and after fine-tuning
To understand the trade-offs between fine-tuning, RAG, and prompting for knowledge injection
To document a fine-tuned model with a model card and identify potential bias shifts
To implement a Monte Carlo retirement simulation that draws annual returns from a configurable normal distribution and records portfolio paths across 1,000 simulations
To generate a labeled two-panel visualization showing simulated paths with median and percentile bands, and a histogram of final balances
To construct a multimodal API request that encodes a PNG image as a base64 string, packages it in the correct Ollama JSON payload, and parses the structured text response
To conduct a two-turn conversation with a multimodal model using a structured prompt that specifies role, required response sections, and audience assumptions
To compare AI-generated quantitative claims against ground-truth statistics and identify specific numerical errors with verbatim AI excerpts
To evaluate the sensitivity of simulation outcomes to parameter changes and explain the compounding effect of return mean and volatility on the outcome distribution
To propose a user-facing guardrail that limits over-reliance on AI numerical claims in financial contexts

Background Reading and References

Please refer to the following readings and examples offering templates to help get you started:

The Assignment

In this lab, you and your partner will build a question-answering system over a corpus that matters to you: your own course notes, a student organization’s documents, a hobby wiki you maintain, or a set of public campus documents. The system must answer questions with citations when the corpus supports an answer, and say so honestly when it does not. This lab is completed in pairs using driver/navigator roles with swaps at least every 30 minutes and a swap log.

Before You Start

Prerequisite concepts — complete these activities before writing any code:

RAG Activity — the index/retrieve/generate pipeline
RAG Quality Activity — recall@k, faithfulness, and abstention

Tools to install:

# Chroma vector database (pure Python, no server needed for this lab)
pip install chromadb

# Sentence transformers for local embeddings (no API key required)
pip install sentence-transformers

# Requests for Ollama calls (already installed if you did Lab 1)
pip install requests

Health check — run this after installation. If you see ok on each line, you are ready:

python -c "import chromadb; print('chromadb ok')"
python -c "from sentence_transformers import SentenceTransformer; print('sentence_transformers ok')"
python -c "import requests; r = requests.get('http://localhost:11434/api/tags'); print('ollama ok' if r.ok else 'ollama NOT running')"

Expected output:

chromadb ok
sentence_transformers ok
ollama ok

If ollama NOT running, start the server with ollama serve in a separate terminal.

Estimated time budget:

Part	Task	Estimated time
Part 1	Curate and document corpus	45–60 min
Part 2	Index with intent	60–90 min
Part 3	Grounded generation	60–75 min
Part 4	Citation audit	30–45 min
Writeup	Readme, datasheet, reflection	30–45 min

Part 1: Curate and Document Your Corpus

Assemble a corpus of at least 15 documents or pages (markdown, text, or extracted PDF text). You may not use any document containing another person’s private information; if your corpus involves anything sensitive, the local-only nature of this pipeline is your friend, and your writeup must say so explicitly. Write a half-page datasheet: sources, time range, who and what is represented, who and what is absent, and known limitations.

Step-by-step guide

Step 1: Collect your documents.

Good corpus ideas:

Your notes from another class (15+ lecture note files)
A student club’s public meeting minutes
Wikipedia articles on a hobby topic you know well (so you can verify answers)
Ursinus College’s public website pages, saved as text

Place all files in a folder, e.g., corpus/. Each file should be plain text or markdown. If you have PDFs, extract their text first:

pip install pymupdf
python -c "
import fitz, pathlib
for pdf in pathlib.Path('corpus_pdfs').glob('*.pdf'):
    doc = fitz.open(pdf)
    text = '\n'.join(page.get_text() for page in doc)
    pathlib.Path('corpus/' + pdf.stem + '.txt').write_text(text)
    print(f'Extracted {pdf.name}: {len(text)} chars')
"

Step 2: Verify your corpus size.

python -c "
import pathlib
files = list(pathlib.Path('corpus').glob('*.txt')) + list(pathlib.Path('corpus').glob('*.md'))
total_chars = sum(f.stat().st_size for f in files)
print(f'Files: {len(files)} | Total characters: {total_chars:,}')
"

Expected output (your numbers will differ):

Files: 18 | Total characters: 142,350

You need at least 15 files. If you have fewer, add more documents before proceeding.

Step 3: Write your corpus datasheet (half page in your readme).

Answer these questions in prose:

Sources: Where did each document come from? What URL or file path?
Time range: When were these documents written or last updated?
Who and what is represented: What topics, people, or events appear?
Who and what is absent: What related topics are NOT covered? Who would find this corpus unhelpful?
Known limitations: Are any documents incomplete, low-quality, or potentially outdated?

Troubleshooting — Part 1

PDF extraction produces garbled text (especially from scanned PDFs) Scanned PDFs require OCR; pymupdf only extracts embedded text. Try pip install pytesseract and Tesseract OCR for scanned documents, or choose different source documents.

Fewer than 15 usable files after extraction Wikipedia is a reliable fallback. Use pip install wikipedia-api and download 15+ related articles programmatically.

File encoding errors when reading Always open files with encoding="utf-8", errors="replace" to handle non-ASCII characters gracefully.

Checkpoint: Before moving to Part 2, make sure you can answer:

How many documents are in your corpus, and what is their total character count?

Name one topic that your corpus covers well and one topic a user might ask about that your corpus cannot answer. (You will use both in Part 3.)

What does your corpus datasheet reveal about potential blind spots in the answers your system will give?

Part 2: Index with Intent

Implement indexing with two chunking strategies (for example, fixed-size with overlap versus paragraph-structural), with chunk parameters externalized in a JSON configuration file. Build a question set of at least ten questions whose answers you have located by hand (note the source chunk for each). Report recall@k for $k \in {1, 3, 5}$ under both strategies, and choose your shipped configuration with a quantitative defense.

Step-by-step guide

Step 1: Create your configuration file.

{
  "corpus_dir": "corpus",
  "model": "llama3.2",
  "embed_model": "all-MiniLM-L6-v2",
  "temperature": 0.1,
  "seed": 42,
  "top_k": 3,
  "abstention_phrase": "I don't have enough information in my knowledge base to answer that.",
  "chunking": {
    "strategy": "fixed",
    "chunk_size": 500,
    "overlap": 50
  }
}

Step 2: Implement the two chunking strategies.

def chunk_fixed(text, chunk_size=500, overlap=50):
    """
    Split text into overlapping fixed-size chunks.
    Returns a list of (chunk_text, start_char) tuples.
    """
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append((text[start:end], start))
        start += chunk_size - overlap
    return chunks

def chunk_by_paragraph(text, min_size=100, max_size=1000):
    """
    Split text on blank lines, then merge short paragraphs until min_size is met.
    Returns a list of (chunk_text, paragraph_index) tuples.
    """
    raw_paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks = []
    current = ""
    idx = 0
    for para in raw_paragraphs:
        current = (current + "\n\n" + para).strip() if current else para
        if len(current) >= min_size:
            chunks.append((current[:max_size], idx))
            current = current[max_size:]
            idx += 1
    if current:
        chunks.append((current, idx))
    return chunks

Step 3: Build the Chroma index.

import chromadb
from sentence_transformers import SentenceTransformer
import pathlib
import json

def build_index(config, strategy="fixed"):
    """
    Load all documents from corpus_dir, chunk them, embed them, and store in Chroma.
    Returns (collection, embed_model, chunk_metadata_list).
    """
    embed_model = SentenceTransformer(config["embed_model"])
    client = chromadb.Client()

    collection_name = f"lab2_{strategy}"
    # Delete existing collection if re-running
    try:
        client.delete_collection(collection_name)
    except Exception:
        pass
    collection = client.create_collection(collection_name)

    all_chunks = []
    all_ids = []
    all_metadatas = []

    corpus_path = pathlib.Path(config["corpus_dir"])
    doc_files = list(corpus_path.glob("*.txt")) + list(corpus_path.glob("*.md"))

    for doc_file in doc_files:
        text = doc_file.read_text(encoding="utf-8", errors="replace")

        if strategy == "fixed":
            cfg = config["chunking"]
            chunks = chunk_fixed(text, cfg["chunk_size"], cfg["overlap"])
        else:
            chunks = chunk_by_paragraph(text)

        for i, (chunk_text, position) in enumerate(chunks):
            chunk_id = f"{doc_file.stem}_{strategy}_{i}"
            all_chunks.append(chunk_text)
            all_ids.append(chunk_id)
            all_metadatas.append({
                "source": doc_file.name,
                "position": position,
                "strategy": strategy
            })

    # Embed in batches to avoid memory issues
    batch_size = 64
    all_embeddings = []
    for i in range(0, len(all_chunks), batch_size):
        batch = all_chunks[i:i+batch_size]
        embeddings = embed_model.encode(batch).tolist()
        all_embeddings.extend(embeddings)
        print(f"  Embedded {min(i+batch_size, len(all_chunks))}/{len(all_chunks)} chunks")

    collection.add(
        documents=all_chunks,
        embeddings=all_embeddings,
        ids=all_ids,
        metadatas=all_metadatas
    )

    print(f"Indexed {len(all_chunks)} chunks using strategy='{strategy}'")
    return collection, embed_model, list(zip(all_ids, all_chunks, all_metadatas))

Expected output:

  Embedded 64/312 chunks
  Embedded 128/312 chunks
  ...
  Embedded 312/312 chunks
Indexed 312 chunks using strategy='fixed'

Step 4: Define your ten questions and their ground-truth source chunks.

Before running any retrieval, locate each answer by hand in your corpus. Record the source file and approximate position. This is your ground truth.

# question_set.py
QUESTIONS = [
    {
        "id": "Q01",
        "question": "What is the main topic covered in lecture 3?",
        "answer_in_source": "lecture03.txt",  # file where the answer lives
        "answer_text_snippet": "gradient descent"  # short string that appears in the answer chunk
    },
    # TODO: Add Q02 through Q10 covering different documents
    # Include at least 2 questions your corpus CANNOT answer (for abstention testing in Part 3)
]

Step 5: Compute recall@k for both strategies.

def recall_at_k(collection, embed_model, questions, k=3):
    """
    For each question, retrieve top-k chunks and check if the ground-truth
    snippet appears in any of the retrieved chunks.
    Returns recall as a float.
    """
    hits = 0
    for q in questions:
        if q.get("unanswerable"):
            continue  # skip abstention questions for recall measurement
        query_embedding = embed_model.encode([q["question"]]).tolist()
        results = collection.query(query_embeddings=query_embedding, n_results=k)
        retrieved_docs = results["documents"][0]
        found = any(q["answer_text_snippet"].lower() in doc.lower() for doc in retrieved_docs)
        if found:
            hits += 1
    answerable = [q for q in questions if not q.get("unanswerable")]
    return hits / len(answerable) if answerable else 0.0

# Run for both strategies and all three k values
config = json.load(open("config.json"))
for strategy in ["fixed", "paragraph"]:
    print(f"\nStrategy: {strategy}")
    collection, embed_model, _ = build_index(config, strategy=strategy)
    for k in [1, 3, 5]:
        recall = recall_at_k(collection, embed_model, QUESTIONS, k=k)
        print(f"  recall@{k} = {recall:.2f}")

Expected output (your numbers will differ):

Strategy: fixed
  recall@1 = 0.50
  recall@3 = 0.80
  recall@5 = 0.90

Strategy: paragraph
  recall@1 = 0.60
  recall@3 = 0.70
  recall@5 = 0.80

In your readme, present these numbers in a table and defend your choice of strategy with a specific quantitative comparison.

Troubleshooting — Part 2

chromadb raises ValueError: Embedding function is required You are passing embeddings manually via collection.add(embeddings=...), which is correct. Make sure you are NOT also passing embedding_function when creating the collection — one or the other, not both.

recall@1 = 0.0 for every question with fixed chunking Your chunk_size may be too small, causing answers to be split across chunks. Try chunk_size=800 or chunk_size=1000. Also verify your answer_text_snippet actually appears in the source file (run grep -n "your snippet" corpus/yourfile.txt).

SentenceTransformer download is slow or fails The all-MiniLM-L6-v2 model (~80 MB) downloads from HuggingFace on first run. If your internet is slow, pre-download it: python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')" while on a fast connection, and it will be cached for later runs.

Checkpoint: Before moving to Part 3, make sure you can answer:

What is a vector embedding and why is it more useful for semantic search than exact keyword matching?

How many chunks did your corpus produce under each strategy? Why might one strategy produce more chunks than the other?

What happens if you query the collection before adding any documents? (Test it and record the error.)

Part 3: Grounded Generation

Implement the query path: embed the question, retrieve top-k, assemble a prompt that instructs the model to answer only from the provided context, to cite bracketed source numbers, and to reply with a designated abstention phrase when the context is insufficient. Demonstrate:

Five answered questions with correct citations.
Two honest abstentions on questions your corpus cannot answer.
One before/after comparison showing the bare model hallucinating where your RAG system either answers correctly or abstains.

Step-by-step guide

Step 1: Implement the query-and-generate function.

import requests
import traceback

def query_rag(question, collection, embed_model, config, ollama_url="http://localhost:11434/api/chat"):
    """
    Retrieve top-k chunks for question, build a grounded prompt, and generate an answer.
    Returns (answer_text, retrieved_chunks_with_metadata).
    """
    # Step 1: Embed the question
    q_embedding = embed_model.encode([question]).tolist()

    # Step 2: Retrieve top-k
    k = config["top_k"]
    results = collection.query(query_embeddings=q_embedding, n_results=k)
    chunks = results["documents"][0]
    metadatas = results["metadatas"][0]

    # Step 3: Build the grounded prompt
    context_parts = []
    for i, (chunk, meta) in enumerate(zip(chunks, metadatas), start=1):
        context_parts.append(f"[{i}] (source: {meta['source']})\n{chunk}")
    context = "\n\n".join(context_parts)

    abstention_phrase = config["abstention_phrase"]

    system_prompt = f"""You are a helpful assistant that answers questions strictly from the provided context.

RULES:
1. Answer ONLY using information from the numbered context passages below.
2. Cite the source number in brackets after every claim, like this: "The sky is blue [1]."
3. If the context does not contain enough information to answer the question, respond with exactly:
   {abstention_phrase}
4. Do not add information from your training data. Do not speculate.

CONTEXT:
{context}"""

    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": question}
    ]

    payload = {
        "model": config["model"],
        "messages": messages,
        "stream": False,
        "options": {"temperature": config["temperature"], "seed": config["seed"]}
    }

    try:
        response = requests.post(ollama_url, json=payload, timeout=60)
        response.raise_for_status()
        answer = response.json()["message"]["content"]
    except Exception as e:
        print(f"[lab2:query_rag] {e}")
        traceback.print_exc()
        raise

    return answer, list(zip(chunks, metadatas))

Step 2: Run and display results for five answerable questions.

config = json.load(open("config.json"))
collection, embed_model, _ = build_index(config, strategy="fixed")  # use your chosen strategy

answerable_questions = [q for q in QUESTIONS if not q.get("unanswerable")][:5]

for q_item in answerable_questions:
    print(f"\nQ: {q_item['question']}")
    answer, retrieved = query_rag(q_item["question"], collection, embed_model, config)
    print(f"A: {answer}")
    print(f"Retrieved from: {[m['source'] for _, m in retrieved]}")

Expected output format (your content will differ):

Q: What is the main topic covered in lecture 3?
A: Lecture 3 covers gradient descent and its role in optimizing neural network weights [1].
Retrieved from: ['lecture03.txt', 'lecture02.txt', 'lecture04.txt']

Step 3: Run two abstention questions.

unanswerable = [q for q in QUESTIONS if q.get("unanswerable")][:2]
for q_item in unanswerable:
    print(f"\nQ: {q_item['question']}")
    answer, _ = query_rag(q_item["question"], collection, embed_model, config)
    print(f"A: {answer}")
    abstained = config["abstention_phrase"] in answer
    print(f"Abstained correctly: {abstained}")

Expected output:

Q: What year was the Ursinus College library built?
A: I don't have enough information in my knowledge base to answer that.
Abstained correctly: True

Step 4: Show the bare-model hallucination contrast.

Ask the bare model (no context) the same question where your RAG system gives a correct answer:

def query_bare_model(question, config, ollama_url="http://localhost:11434/api/chat"):
    messages = [{"role": "user", "content": question}]
    payload = {"model": config["model"], "messages": messages, "stream": False,
               "options": {"temperature": config["temperature"], "seed": config["seed"]}}
    r = requests.post(ollama_url, json=payload, timeout=60)
    return r.json()["message"]["content"]

example_q = answerable_questions[0]["question"]
print(f"Question: {example_q}")
print(f"\nBare model answer:\n{query_bare_model(example_q, config)}")
print(f"\nRAG answer:\n{query_rag(example_q, collection, embed_model, config)[0]}")

Include this comparison verbatim in your readme.

Troubleshooting — Part 3

The model ignores the abstention instruction and answers anyway Smaller models sometimes override instructions when they have training-data knowledge. Make the instruction more direct: add “You MUST respond with exactly that phrase and nothing else if the context is insufficient.” If it still fails, add a post-processing check in Python: if no [1], [2], etc. bracket appears in the answer and the abstention phrase is absent, override the answer with the abstention phrase and log a warning.

Citations like [1] appear in the answer but the wrong chunk is cited This is a retrieval quality issue, not a generation issue. Check your recall@k from Part 2 — if recall@3 is below 0.5, the relevant chunk is not reaching the model. Try increasing top_k to 5 or switching chunking strategy.

The model’s answer is cut off mid-sentence The context may be too long for the model’s context window. Reduce top_k from 3 to 2, or reduce chunk_size in your config. You can also add "num_ctx": 4096 to the Ollama options dict to explicitly set the context window size.

Checkpoint: Before moving to Part 4, make sure you can answer:

What is the role of the system prompt’s “RULES” section in preventing hallucination? What would happen if you removed Rule 4?

In your before/after comparison, what specific false claim did the bare model make? What training-data pattern likely caused it?

How would you modify your pipeline to support a follow-up question (“Tell me more about that”) while maintaining citation grounding?

Part 4: Audit

For at least ten answered questions, audit each citation by hand: does the cited chunk actually support the claim? Report a faithfulness rate and show any failures verbatim, classified using the hallucination taxonomy from class.

Step-by-step guide

Step 1: Build an audit table.

For each of your ten questions, record:

Q#	Question	Answer excerpt	Citation #	Chunk cited	Faithful?	Failure type
Q01	What is …?	“Gradient descent is …” [1]	1	“lecture03: Gradient descent is a method…”	Yes	—
Q02	…	…	…	…	No	Fabrication

Step 2: Classify any failures using the taxonomy from class.

The taxonomy has four categories:

Fabrication: The cited chunk exists but does not contain the claimed fact; the model invented it.
Conflation: The cited chunk is about a related but different topic; the model merged two concepts.
Extrapolation: The chunk implies but does not state the claim; the model over-inferred.
Wrong citation: The fact is correct and appears in the corpus, but it is in a different chunk than cited.

Step 3: Compute and report your faithfulness rate.

# Example audit results — fill this in from your hand-audit table
audit_results = [
    {"q_id": "Q01", "faithful": True, "failure_type": None},
    {"q_id": "Q02", "faithful": False, "failure_type": "Extrapolation"},
    # ... add all 10
]

faithful_count = sum(1 for r in audit_results if r["faithful"])
print(f"Faithfulness rate: {faithful_count}/{len(audit_results)} = {faithful_count/len(audit_results):.1%}")
failures = [r for r in audit_results if not r["faithful"]]
for f in failures:
    print(f"  {f['q_id']}: {f['failure_type']}")

Include your completed audit table and faithfulness rate in your readme.

Troubleshooting — Part 4

Every citation appears faithful, making the audit trivial Your question set may be too simple. Add at least two questions that require synthesizing information from multiple chunks — these are most likely to produce conflation or extrapolation errors.

You cannot find the cited chunk in your collection Use collection.get(ids=["chunk_id"]) to retrieve a specific chunk by its ID. The chunk IDs are in the metadatas list returned by query_rag.

The model sometimes cites [1] when the relevant information was in [2] This is a “wrong citation” failure. It is worth noting separately from fabrication — the answer may be correct even though the citation number is wrong. Count these as failures in your faithfulness rate but classify them accurately.

Checkpoint: Before writing your deliverables, make sure you can answer:

What was your faithfulness rate? Is it higher or lower than you expected?

Which failure type (fabrication, conflation, extrapolation, wrong citation) appeared most often in your audit, and what does that suggest about where to add safeguards?

Which failure did you find more often overall: retrieval fetching the wrong chunk, or generation misusing a correct chunk?

Deliverables

Submit a ZIP containing your code, JSON configuration, corpus (or a pointer plus a sample if it is large), datasheet, question set with labels, evaluation results (CSV or table), audit results, pair log, and a readme writeup of approximately two pages. Ensure reproducibility by fixing random seeds and listing software version information.

Learning Log

Keep a metacognitive learning log for this lab in your readme: in the spirit of multiple means of action and expression, you may respond to each prompt in prose, in bullet points, or with an annotated diagram — whichever best conveys your thinking. (Prompt 4 adapts the AI-Assisted Learning Template by Marc Watkins.)

What I built. One paragraph, in plain language that a friend outside of computer science could follow (this is deliberate practice in writing for multiple audiences).
What surprised me.
What I verified and how. Evidence, not vibes.
How I used AI during this lab, and what I learned from that use.
What I’d tell the next student before they start.
One open question I still have.

Lab-specific prompts

Which failure did you find more often: retrieval fetching the wrong chunk, or generation misusing a correct chunk? What does that imply about where to invest next — better retrieval, or a stricter generation prompt?
Your corpus datasheet names who is absent from your documents. Give a concrete example of a question where that absence would cause your system to either abstain incorrectly (the answer exists somewhere but not in your corpus) or answer incorrectly (the corpus contains a biased or incomplete view). What would you add to the corpus to fix it?
If collaboration beyond your pair occurred, identify it. Do you certify that this submission represents your pair’s original work? Please identify any and all portions of your submission that were not originally written by you.
Approximately how many hours did this lab take (I will not judge you for this at all…I am simply using it to gauge if the assignments are too easy or hard)?

Extension Challenges

These are optional and carry no extra credit.

Challenge 1 (moderate): Add re-ranking. After retrieving top-k=10 chunks, re-rank them by asking the model: “Does this passage answer the question: [question]? Answer yes or no.” Keep only the top 3 “yes” passages. Measure whether re-ranking improves your recall@3 on your question set.

Challenge 2 (harder): Implement hybrid search. Combine your embedding-based retrieval with BM25 keyword search (install rank_bm25). For each query, retrieve the top-5 from each method, merge the lists (deduplicating by chunk ID), and pass the union to the model. Report whether hybrid search improves recall@3 over either method alone.

Challenge 3 (hardest): Add source freshness metadata. Add a last_modified timestamp to each chunk’s metadata (from the file’s mtime). Modify your generation prompt to prefer recent sources when two chunks conflict. Demonstrate this on a question where you have two versions of a document with different information.

Choose Your Direction

Everyone builds the core RAG knowledge base above. Once that pipeline is working, cited, and audited, you extend it in one direction of your choosing from the two below. You do not do both. Pick the direction that most interests you, and carry your Lab 2 corpus, config discipline, and evaluation habits into it.

The single 100-point grade for this lab covers your core RAG work plus your chosen direction together — the graded rubric above still governs your score. The direction is where you push the ideas further; treat the “What proficient work looks like” bullets in your chosen direction as the standard your extension work should meet, and fold your direction’s deliverables into the same submission ZIP and readme as the core lab.

Choose one:

Direction 1: Hands-On Fine-Tuning with LoRA and QLoRA — instead of retrieving knowledge at query time, bake domain knowledge into the weights, and decide from evidence whether that was worth it compared to your RAG pipeline.
Direction 2: Multimodal AI and Monte Carlo Simulation — turn from text retrieval to images, and probe where a multimodal model reads a chart confidently but wrongly, using a simulation you build as ground truth.

Direction 1: Hands-On Fine-Tuning with LoRA and QLoRA

This direction takes the opposite approach to knowledge injection from the one you just built. In the core lab, your RAG system kept knowledge outside the model and retrieved it at query time. Here you will adapt a local model by baking domain knowledge into a small set of trainable weights using LoRA (Low-Rank Adaptation — a technique that adds a tiny number of trainable parameters to a frozen base model, making fine-tuning feasible on consumer hardware). You will use a real domain-specific dataset, instrument training with loss tracking, evaluate output quality, and document the result with a model card — and then, crucially, decide whether fine-tuning earned its keep versus the RAG pipeline from the core lab.

This direction requires GPU access. Use your own hardware (if you have a compatible GPU), Google Colab (free tier with a T4 GPU is sufficient for a 7B model with QLoRA), or a provisioned cloud instance. Time budget: expect 2–3 hours of active work, with training running in the background.

Before You Start

Prerequisite Checklist

GPU access confirmed: your own GPU, Google Colab free tier (T4), or a cloud VM
Python 3.10 or later (python --version)
If using a Llama model: HuggingFace account and accepted model license at meta-llama/Meta-Llama-3-8B-Instruct
HuggingFace CLI installed and logged in (if downloading gated models)

Environment Setup

If using Google Colab: Create a new notebook and run this setup cell first:

# Cell 1: Google Colab Setup
# Run this cell first. Runtime > Change runtime type > T4 GPU

import subprocess
import sys

# Install required packages
packages = [
    "transformers>=4.40.0",
    "peft>=0.10.0",
    "datasets>=2.18.0",
    "bitsandbytes>=0.43.0",
    "accelerate>=0.29.0",
    "trl>=0.8.0",
]
subprocess.check_call([sys.executable, "-m", "pip", "install", "-q"] + packages)

# Verify GPU is available
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'None'}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB" if torch.cuda.is_available() else "")
print("Setup complete!")

Expected output:

CUDA available: True
GPU: Tesla T4
VRAM: 15.8 GB
Setup complete!

If using local hardware: Run in your terminal:

pip install transformers peft datasets bitsandbytes accelerate trl
python -c "import torch; print('GPU:', torch.cuda.get_device_name(0))"

Quick Sanity Check — Confirm Model Downloads Work

# Run this before starting the first step to confirm your HuggingFace access
from transformers import AutoTokenizer

# Use a small, freely available model for the sanity check
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-3-mini-4k-instruct", trust_remote_code=True)
tokens = tokenizer("Hello, world!")
print(f"Tokenizer OK. Token IDs: {tokens['input_ids']}")

Expected output:

Tokenizer OK. Token IDs: [1, 15043, 29892, 3186, 29991]

If you see OSError: Can't load tokenizer, check your internet connection and that you are logged into HuggingFace (huggingface-cli login).

Recommended base models (choose one based on your hardware):

Model	Size	Min VRAM	Notes
`microsoft/phi-3-mini-4k-instruct`	3.8B	6 GB	Best for free Colab T4
`meta-llama/Meta-Llama-3-8B-Instruct`	8B	8 GB with QLoRA	Requires HuggingFace token
`mistralai/Mistral-7B-Instruct-v0.3`	7B	8 GB with QLoRA	Good quality, open weights

Step A: Choose a Domain Dataset

Why this matters: The dataset you choose determines everything — what your model learns, what biases it might amplify, and how you can measure success. A natural choice is a dataset in the same domain as your Lab 2 corpus, so your before/after comparison speaks directly to the RAG-versus-fine-tuning question. Choosing a well-structured dataset is the difference between training that converges and training that produces nonsense.

Choose one of the following datasets, or propose your own (get instructor approval first):

Dataset	HuggingFace ID	Domain	Format
Medical Q&A	`medalpaca/medical_meadow_medical_flashcards`	Medical	instruction/output
Python codegen	`iamtarun/python_code_instructions_18k_alpaca`	Code	instruction/input/output
Legal reasoning	`nguha/legalbench`	Legal	varies by subset
Science questions	`sciq`	Science	question/answer/support

Load and inspect your dataset:

# dataset_inspect.py
from datasets import load_dataset

# TODO: replace with your chosen dataset ID
DATASET_ID = "sciq"  # change this

dataset = load_dataset(DATASET_ID)
print(f"Dataset splits: {list(dataset.keys())}")
print(f"Train size: {len(dataset['train'])}")
print(f"\nFirst example:\n{dataset['train'][0]}")

Expected output (for sciq):

Dataset splits: ['train', 'validation', 'test']
Train size: 11679

First example:
{'question': 'What type of force keeps planets in orbit?', 
 'distractor3': 'electromagnetic', 'distractor1': 'nuclear', 
 'distractor2': 'friction', 'correct_answer': 'gravitational',
 'support': 'Gravitational force keeps planets in orbit around the Sun.'}

Format the dataset for instruction tuning. Models trained with SFTTrainer expect a single string per example in a consistent instruction format. Here is a starter formatter — adapt it for your dataset’s field names:

# dataset_format.py
from datasets import load_dataset

# TODO: replace with your dataset ID
DATASET_ID = "sciq"

dataset = load_dataset(DATASET_ID)

def format_example(example: dict) -> dict:
    """Convert a raw dataset row into an instruction-tuning string.
    TODO: adapt the field names below to match your chosen dataset."""
    # Example for sciq:
    instruction = example.get("question", "")
    # TODO: replace 'correct_answer' with the answer field in your dataset
    answer = example.get("correct_answer", example.get("output", ""))
    # TODO: include supporting context if available in your dataset
    context = example.get("support", "")

    if context:
        text = f"### Instruction:\n{instruction}\n\n### Context:\n{context}\n\n### Response:\n{answer}"
    else:
        text = f"### Instruction:\n{instruction}\n\n### Response:\n{answer}"

    return {"text": text}

# Apply formatting and create train/validation split
formatted = dataset["train"].map(format_example)

# TODO: adjust split sizes based on your dataset size
# Use 90% train, 10% validation if no validation split exists
if "validation" not in dataset:
    split = formatted.train_test_split(test_size=0.1, seed=42)
    train_data = split["train"]
    val_data = split["test"]
else:
    train_data = formatted
    val_data = dataset["validation"].map(format_example)

print(f"Train examples: {len(train_data)}")
print(f"Validation examples: {len(val_data)}")
print(f"\nFormatted example:\n{train_data[0]['text'][:300]}")

Write the dataset section of your model card (model_card.md). Include: source URL, number of examples, format (instruction/input/output or chat turns), your train/validation split sizes, and one known limitation of the dataset.

Checkpoint: Before moving on, verify that your formatted dataset has a text field, that the formatted string contains ### Instruction: and ### Response: sections, and that you have a validation split with at least 100 examples.

Troubleshooting: If load_dataset hangs, you may be behind a firewall that blocks HuggingFace CDN — try load_dataset(..., cache_dir="/tmp/hf_cache") or download the dataset manually. If the field names in format_example don’t match your dataset, print dataset['train'][0].keys() to see what fields are available. If the formatted text is empty for some rows, those rows likely have None values — add if not instruction or not answer: return None and call .filter(lambda x: x is not None) after mapping.

Step B: Fine-Tune with LoRA / QLoRA

Why this matters: LoRA does not modify the original model weights at all — it learns two small matrices (called A and B, with rank r) that approximate the weight update. This means you can fine-tune a 7B model on a consumer GPU with 8–16 GB of VRAM. QLoRA adds 4-bit quantization on top, cutting VRAM usage roughly in half again.

Create your training script (train.py or a Colab notebook cell). Fill in every # TODO:

# train.py — LoRA/QLoRA fine-tuning with SFTTrainer
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
from datasets import load_dataset

# ── Configuration ─────────────────────────────────────────────────────────────
# TODO: set to your chosen model
MODEL_ID = "microsoft/phi-3-mini-4k-instruct"
# TODO: set to your chosen dataset (or "local" if you saved it)
DATASET_ID = "sciq"
OUTPUT_DIR = "./lora-finetuned"

# LoRA hyperparameters — these are your starting point, justify your choices
LORA_R = 8           # rank: higher = more capacity, more VRAM. Try 8 or 16.
LORA_ALPHA = 16      # typically 2× rank
LORA_DROPOUT = 0.05  # regularization
# TODO: adjust target_modules for your model family
# Phi-3 / LLaMA family: ["q_proj", "v_proj"] or ["q_proj", "k_proj", "v_proj", "o_proj"]
TARGET_MODULES = ["q_proj", "v_proj"]

# ── 4-bit Quantization (QLoRA) — comment out if you have enough VRAM ─────────
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

# ── Load model and tokenizer ──────────────────────────────────────────────────
print(f"Loading {MODEL_ID}...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token  # required for batch training

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,  # remove this line if not using QLoRA
    device_map="auto",
    trust_remote_code=True,
)

# ── Apply LoRA adapters ───────────────────────────────────────────────────────
lora_config = LoraConfig(
    r=LORA_R,
    lora_alpha=LORA_ALPHA,
    target_modules=TARGET_MODULES,
    lora_dropout=LORA_DROPOUT,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# ── Load and format dataset ───────────────────────────────────────────────────
# TODO: replace with your formatted dataset loading
# If you saved it locally: dataset = load_from_disk("./formatted_dataset")
dataset = load_dataset(DATASET_ID)

def format_example(example):
    # TODO: adapt field names to your dataset
    instruction = example.get("question", "")
    answer = example.get("correct_answer", example.get("output", ""))
    return {"text": f"### Instruction:\n{instruction}\n\n### Response:\n{answer}"}

train_data = dataset["train"].map(format_example)
val_data = dataset.get("validation", dataset["train"].select(range(500))).map(format_example)

# ── Training arguments ────────────────────────────────────────────────────────
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=1,          # TODO: increase to 2-3 if you have time/VRAM
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,          # TODO: justify your choice in the writeup
    fp16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_steps=100,
    warmup_ratio=0.03,
    report_to="none",            # change to "wandb" if you want W&B tracking
)

# ── Train ─────────────────────────────────────────────────────────────────────
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=val_data,
    dataset_text_field="text",
    max_seq_length=512,
)

print("Starting training...")
trainer.train()
trainer.save_model(OUTPUT_DIR)
print(f"Model saved to {OUTPUT_DIR}")

Run training and watch the loss output:

python train.py

Expected training output (your exact numbers will differ, but the pattern should show decreasing loss):

trainable params: 4,194,304 || all params: 3,825,160,192 || trainable%: 0.1097
Starting training...
{'loss': 2.3421, 'learning_rate': 0.0002, 'epoch': 0.01}
{'loss': 1.8834, 'learning_rate': 0.00019, 'epoch': 0.05}
{'loss': 1.5201, 'learning_rate': 0.00018, 'epoch': 0.10}
{'loss': 1.3012, 'learning_rate': 0.00015, 'epoch': 0.20}
{'loss': 1.1478, 'learning_rate': 0.00010, 'epoch': 0.40}
{'eval_loss': 1.2341, 'epoch': 0.40}
...
Model saved to ./lora-finetuned

A healthy training run shows loss decreasing from roughly 2.3 toward 1.1 over 200 steps. If loss is stuck above 2.0 or oscillates wildly, see troubleshooting below.

Plot your loss curve. After training, add this cell to your notebook or script:

# plot_loss.py
import json
import matplotlib.pyplot as plt

# Load training logs (saved by Trainer in output_dir/trainer_state.json)
with open("./lora-finetuned/trainer_state.json") as f:
    state = json.load(f)

train_steps = [e["step"] for e in state["log_history"] if "loss" in e]
train_loss  = [e["loss"] for e in state["log_history"] if "loss" in e]
eval_steps  = [e["step"] for e in state["log_history"] if "eval_loss" in e]
eval_loss   = [e["eval_loss"] for e in state["log_history"] if "eval_loss" in e]

plt.figure(figsize=(10, 5))
plt.plot(train_steps, train_loss, label="Training loss", color="blue")
plt.plot(eval_steps, eval_loss, label="Validation loss", color="orange", linestyle="--")
plt.xlabel("Steps")
plt.ylabel("Loss")
plt.title("Training vs. Validation Loss")
plt.legend()
plt.grid(True)
plt.savefig("loss_curve.png", dpi=150)
plt.show()
print("Saved loss_curve.png")

Annotate the curve in your writeup: Does loss converge? Is there overfitting (training loss keeps falling but validation loss rises or plateaus)? Justify at least one hyperparameter choice (why you chose your value of r, number of epochs, or learning rate).

Checkpoint: Before moving on, verify that ./lora-finetuned/ directory exists and contains adapter files, that loss_curve.png was saved, and that training loss decreased from the first logged step to the last.

Troubleshooting: If you get CUDA out of memory, reduce per_device_train_batch_size to 2 or 1, and increase gradient_accumulation_steps to keep effective batch size the same. If loss is nan from step 1, your learning rate is too high — try 1e-4 or 5e-5. If target_modules raises a ValueError saying the module does not exist, print [name for name, _ in model.named_modules()] to see the actual module names in your model. If training takes longer than 2 hours on Colab, reduce the dataset to 2000 examples with train_data = train_data.select(range(2000)).

Step C: Evaluate Before and After

Why this matters: Without systematic evaluation, fine-tuning is a black box — you spent hours training, but do you actually know if the model improved? This step builds the habit of measuring before you ship, exactly as you did with recall@k and the citation audit in the core lab.

Load both the base model and your fine-tuned model for comparison:

# evaluate_models.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# TODO: set to your model ID
MODEL_ID = "microsoft/phi-3-mini-4k-instruct"
LORA_PATH = "./lora-finetuned"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

def load_base_model():
    return AutoModelForCausalLM.from_pretrained(
        MODEL_ID, device_map="auto", trust_remote_code=True, torch_dtype=torch.float16
    )

def load_finetuned_model(base_model):
    return PeftModel.from_pretrained(base_model, LORA_PATH)

def generate(model, prompt: str, max_new_tokens: int = 200) -> str:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.1,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    # Return only the newly generated tokens, not the prompt
    new_tokens = output[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(new_tokens, skip_special_tokens=True)

Run 10 test prompts (not seen during training) through both models:

# Add to evaluate_models.py

# TODO: write 10 test prompts relevant to your chosen domain
# These must NOT be from the training set
TEST_PROMPTS = [
    "### Instruction:\nWhat is the primary mechanism of action of beta-blockers?\n\n### Response:",
    "### Instruction:\nExplain the difference between supervised and unsupervised learning.\n\n### Response:",
    # TODO: add 8 more prompts
]

import csv

base_model = load_base_model()
ft_model = load_finetuned_model(load_base_model())

rows = []
for i, prompt in enumerate(TEST_PROMPTS):
    print(f"Prompt {i+1}/{len(TEST_PROMPTS)}...")
    base_out = generate(base_model, prompt)
    ft_out = generate(ft_model, prompt)

    # TODO: manually rate each pair and fill in 'improvement' and 'notes'
    rows.append({
        "prompt": prompt[:80],
        "base_output": base_out[:200],
        "finetuned_output": ft_out[:200],
        "improvement": "?",  # fill in: Y / N / Partial
        "notes": ""
    })

with open("eval_comparison.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["prompt","base_output","finetuned_output","improvement","notes"])
    writer.writeheader()
    writer.writerows(rows)

print("Saved eval_comparison.csv — open it and fill in the improvement and notes columns.")

Compute at least one quantitative metric. Choose one:

Option A — Perplexity on held-out test set (lower is better):

# perplexity.py
import torch
import math

def compute_perplexity(model, tokenizer, texts: list[str], max_length: int = 512) -> float:
    model.eval()
    total_loss = 0
    total_tokens = 0
    for text in texts:
        inputs = tokenizer(text, return_tensors="pt", max_length=max_length, truncation=True).to(model.device)
        with torch.no_grad():
            outputs = model(**inputs, labels=inputs["input_ids"])
        total_loss += outputs.loss.item() * inputs["input_ids"].shape[1]
        total_tokens += inputs["input_ids"].shape[1]
    return math.exp(total_loss / total_tokens)

# TODO: load test texts from your dataset
test_texts = ["### Instruction:\n...\n\n### Response:\n..."]  # replace with real test set

base_ppl = compute_perplexity(base_model, tokenizer, test_texts)
ft_ppl   = compute_perplexity(ft_model,   tokenizer, test_texts)
print(f"Base model perplexity:        {base_ppl:.2f}")
print(f"Fine-tuned model perplexity:  {ft_ppl:.2f}")
print(f"Improvement: {((base_ppl - ft_ppl) / base_ppl * 100):.1f}%")

Expected output (numbers will vary):

Base model perplexity:        24.31
Fine-tuned model perplexity:  11.87
Improvement: 51.2%

Option B — Task accuracy (for MCQ datasets):

# For sciq or similar: compare model's top predicted answer to correct_answer
# TODO: implement for your dataset

Option C — LLM-as-judge score (1–5):

# Use an LLM-as-judge, or write a simple one here
# TODO: call your judge on each of the 10 test prompts

Document at least one regression in your writeup — a prompt where the base model was better. This is expected and important to be honest about.
Compare against your RAG pipeline. Take two or three of the questions your Lab 2 RAG system answered from your corpus, and ask the fine-tuned model the same questions with no retrieval. Which approach answered more faithfully? Which hallucinated? Record the head-to-head so your recommendation (below) rests on evidence, not intuition.

Checkpoint: Before moving on, verify that eval_comparison.csv has 10 rows, that you have manually filled in the improvement column for each row, and that your quantitative metric (perplexity, accuracy, or judge score) has been computed for both base and fine-tuned models.

Troubleshooting: If the fine-tuned model produces repetitive output (same phrase repeated over and over), add repetition_penalty=1.2 to the generate() call. If both models produce identical output, the LoRA adapter may not have loaded correctly — check that PeftModel.from_pretrained is pointing to the correct directory and that adapter_config.json exists there. If perplexity of the fine-tuned model is higher than the base model, the model may have overfit — check your loss curve for a rising validation loss.

Step D: Model Card and Reflection

Why this matters: A model without documentation is a liability. The model card format (from Mitchell et al. 2019) is the industry standard for responsible AI deployment — every major model on HuggingFace uses it. Writing one forces you to articulate what your model does, what it does not do, and what could go wrong — the same discipline as the corpus datasheet in the core lab.

Create model_card.md with all eight required sections. Use this template:

# Model Card: [Your Model Name]

## Model Details
- **Base model:** [model ID]
- **Fine-tuning method:** LoRA (r=[your r], alpha=[your alpha], target_modules=[your modules])
- **Training dataset:** [dataset name and size]
- **Training duration:** [steps, epochs, wall-clock time]
- **Developer:** [your name]
- **Date:** [date]

## Intended Use
**Primary use case:** [what this model is for, e.g., "answering medical flashcard questions"]

**Out-of-scope use cases:**
- TODO: list at least 2 uses this model should NOT be used for
- Example: "Clinical diagnosis or treatment decisions"

## Factors
- **Relevant factors:** [language, domain, question type, etc.]
- **Evaluation factors:** [what variables you held constant and which you varied]

## Metrics
- **Evaluation metric(s):** [perplexity / accuracy / LLM-as-judge]
- **Threshold for acceptable performance:** [what number would you be happy with?]

## Training Data
[Reference Step A. Include: source, size, format, train/val split, known limitations]

## Quantitative Analyses
[Reference Step C table. Include your before/after metric values.]

| Metric | Base Model | Fine-Tuned Model | Change |
|--------|-----------|------------------|--------|
| Perplexity | X | Y | -Z% |
| Qualitative improvement rate | X/10 | Y/10 | +N |

## Ethical Considerations
**Bias risks:**
- TODO: identify at least one bias that your chosen dataset might introduce or amplify
- Example: "The medical_meadow dataset skews toward Western clinical practice. The fine-tuned model may perform poorly on questions about traditional or non-Western medicine."

**Privacy risks:** [does the training data contain PII?]

## Caveats
- TODO: list at least 2 known limitations of your fine-tuned model
- Example: "Performance degrades on questions longer than 200 tokens"

Answer the reflection prompts in your writeup, including the fine-tuning-versus-RAG recommendation below.

Checkpoint: Before submitting, verify that model_card.md has all eight sections and that the Ethical Considerations section names a specific bias risk tied to your chosen dataset.

Troubleshooting: If you are unsure what biases your dataset introduces, search for published papers or datasheets about your chosen dataset — most HuggingFace datasets have a “Dataset Card” tab that discusses known biases and limitations.

Direction 1 Extension Challenges (optional)

These challenges push this direction from a working fine-tune to a research-grade experiment.

Extension 1: LoRA rank ablation. Train three versions of your model with r=4, r=8, and r=16. Plot all three loss curves on the same axes. Does increasing rank improve final loss? Does it improve perplexity on the test set? How does it affect VRAM usage? Report all three in a table.

Extension 2: LoRA vs QLoRA memory comparison. Train once with 4-bit quantization (QLoRA) and once without. Use torch.cuda.max_memory_allocated() after training to measure peak VRAM. How much memory did quantization save? Did it hurt final model quality?

Extension 3: Few-shot prompting vs. fine-tuning. Take 10 of your training examples and put them directly into the base model’s context window as few-shot examples. Compare the few-shot base model’s output quality to your fine-tuned model on the test set. When does fine-tuning win, and when does few-shot prompting match it with far less effort?

Direction 1 Deliverables

Fold these into your Lab 2 submission ZIP and readme:

train.py or Colab notebook (.ipynb) — runnable training script
dataset_format.py — dataset loading and formatting code
evaluate_models.py — comparison script
loss_curve.png — annotated training/validation loss plot
eval_comparison.csv — before/after comparison table (10 rows)
Quantitative metric results (printed output, screenshot, or CSV)
model_card.md — complete model card with all 8 sections
A section in your writeup.md covering reflection answers, hyperparameter justifications, and your fine-tuning-versus-RAG recommendation

What proficient work looks like (Direction 1)

Training runs with a loss curve, a quantitative before/after metric (perplexity or a task-specific metric), a before/after comparison table, and at least one hyperparameter choice justified with evidence.
The dataset is described with source, size, format, and any cleaning applied; a validation set is used to catch overfitting; and at least one dataset limitation is named.
Evaluation is systematic with a defined metric, honestly reports at least one regression (something the fine-tuned model does worse), and delivers a defended recommendation for whether fine-tuning was worth it versus RAG and prompting.
The model card is complete across all eight sections, the bias-risk section identifies at least one bias shift the fine-tuning introduced or amplified, and the reflection answers are substantive and grounded in your own results.

Direction 1 Reflection Prompts

You spent hours fine-tuning a model on 1,000 examples. A colleague says “just put those examples in the system prompt instead.” Evaluate that suggestion: when would they be right, and when would fine-tuning be worth the effort?
You built a RAG system in the core lab and a fine-tuned model here for related knowledge. For your specific domain, which one would you deploy, and what evidence from your two evaluations drives that choice?
Your fine-tuned model may now perform better in your domain but worse on general questions. Who is responsible for communicating that trade-off to users?
How many hours did this direction take?

Direction 2: Multimodal AI and Monte Carlo Simulation

This direction turns from text retrieval to images. In the core lab you audited whether a model faithfully used text you retrieved; here you will audit whether a multimodal model faithfully reads a chart. You will build a Monte Carlo retirement simulation that you generate, send its chart to a local vision model, and discover that AI image analysis is impressively capable at pattern recognition but surprisingly fragile on numerical precision — and that the difference matters enormously when the output might influence someone’s financial decisions. The ground-truth-versus-AI-claim audit is the same muscle you built in the core lab’s citation audit, applied to pixels instead of passages.

This direction is completed in pairs using driver/navigator roles: the driver types while the navigator reviews, questions, and consults documentation, and you must swap roles at least every 30 minutes, keeping a brief log of swap times and who held each role.

Before You Start

Why Monte Carlo? A spreadsheet gives you one future. Monte Carlo simulation gives you a thousand. Instead of projecting a single “expected” outcome, we draw thousands of possible annual returns from a statistical distribution, let each one play out over a 40-year career, and look at the spread of endings. That spread — not the center — is what matters when you are making a decision whose consequences will compound for decades. This is why financial planners use simulation rather than a single formula, and it is why the visualization you generate will be more informative than any average.

Prerequisite concepts — make sure you have completed these activities before writing any code:

Sampling, Temperature, and Generation Activity — stochastic sampling and output distributions
Evaluating Agent Outputs Activity — how to critically assess AI-generated content
Multimodal Agents Activity — sending images to local vision models

Tools to install:

pip install numpy matplotlib requests

Verify your multimodal model is available:

ollama pull llava
ollama list

Expected output:

NAME               ID              SIZE    MODIFIED
llava:latest       8dd30f6b0cb1    4.7 GB  2 minutes ago

If llava is not available on your machine, any of the following will work: moondream, bakllava, llava-phi3. Update the "model" key in your config file to match whichever you pull.

Verify the API responds:

curl http://localhost:11434/api/tags

Expected output (abbreviated):

{"models":[{"name":"llava:latest", ...}]}

Estimated time budget:

Part	Task	Estimated time
Step A	Simulation Engine	50–70 min
Step B	Multimodal Integration	40–60 min
Step C	Parameter Sensitivity	25–35 min
Step D	Critical Analysis	20–30 min
Writeup	Readme and reflection	30–45 min

Step A: The Simulation Engine

You will write a Python script that simulates 1,000 possible futures for a person who starts saving at age 25 and retires at 65. Each simulated year draws a random annual return from a normal distribution, applies it to the portfolio, and records the resulting balance. The output is a two-panel chart saved to disk. Keep the same config-file discipline you used in the core lab — externalizing parameters is what makes the sensitivity analysis in Step C a one-file edit.

Step A.1: Create your configuration file.

Create config.json in your project root. Externalizing parameters here means you can run Step C’s sensitivity analysis by editing one file rather than hunting through your code.

{
  "starting_age": 25,
  "retirement_age": 65,
  "life_expectancy": 90,
  "starting_savings": 10000,
  "monthly_contribution": 500,
  "annual_return_mean": 0.07,
  "annual_return_std": 0.12,
  "inflation_rate": 0.025,
  "num_simulations": 1000,
  "model": "llava",
  "ollama_url": "http://localhost:11434"
}

Step A.2: Write the simulation function.

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
import base64
import io
import requests
import json
import traceback


def load_config(path="config.json"):
    """Load simulation and model parameters from a JSON config file."""
    with open(path) as f:
        return json.load(f)


def simulate_retirement(cfg):
    """
    Run a Monte Carlo retirement simulation.

    Draws annual returns from N(annual_return_mean, annual_return_std) for each
    simulated year of a career, applying monthly contributions each year.

    Returns:
        np.ndarray of shape (num_simulations, years_to_retirement), where each
        row is one simulated portfolio path and each column is the end-of-year balance.
    """
    years = cfg["retirement_age"] - cfg["starting_age"]
    results = np.zeros((cfg["num_simulations"], years))

    for sim in range(cfg["num_simulations"]):
        balance = cfg["starting_savings"]
        for year in range(years):
            # TODO: Add annual contribution (monthly_contribution * 12)
            balance += cfg["monthly_contribution"] * 12

            # TODO: Draw a random annual return from a normal distribution
            #       using annual_return_mean and annual_return_std.
            #       Hint: np.random.normal(mean, std)
            annual_return = np.random.normal(
                cfg["annual_return_mean"], cfg["annual_return_std"]
            )

            # TODO: Apply the return to the balance.
            balance *= (1 + annual_return)

            # TODO: Clip balance at 0 — a portfolio cannot go negative.
            balance = max(balance, 0)

            results[sim, year] = balance

    return results

Step A.3: Write the visualization function.

def plot_simulation(balances, cfg):
    """
    Create a labeled two-panel visualization of simulation results.

    Left panel: all simulated portfolio paths (light gray) plus median (blue)
    and 10th/90th percentile bands (red dashed).

    Right panel: histogram of final balances at retirement, with vertical
    lines at the median and at $1 million.

    Saves the figure to retirement_simulation.png and also returns a
    base64-encoded PNG string for sending to the multimodal model.

    Returns:
        str: base64-encoded PNG image.
    """
    ages = list(range(cfg["starting_age"] + 1, cfg["retirement_age"] + 1))
    median_path = np.median(balances, axis=0)
    p10_path = np.percentile(balances, 10, axis=0)
    p90_path = np.percentile(balances, 90, axis=0)
    final_balances = balances[:, -1]

    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 6))

    # --- Left panel: portfolio paths ---
    # TODO: Plot each simulated path in light gray with alpha=0.05
    for path in balances:
        ax1.plot(ages, path, color="gray", alpha=0.05, linewidth=0.5)

    # TODO: Plot median in solid blue, labeled "Median"
    ax1.plot(ages, median_path, color="blue", linewidth=2, label="Median")

    # TODO: Plot 10th percentile in red dashed, labeled "10th / 90th percentile"
    ax1.plot(ages, p10_path, color="red", linewidth=1.5, linestyle="--",
             label="10th / 90th percentile")

    # TODO: Plot 90th percentile in red dashed (same label so it shares the legend entry)
    ax1.plot(ages, p90_path, color="red", linewidth=1.5, linestyle="--")

    # TODO: Label x-axis ("Age"), y-axis ("Portfolio Balance"), add title and legend
    ax1.set_xlabel("Age", fontsize=12)
    ax1.set_ylabel("Portfolio Balance", fontsize=12)
    ax1.set_title(
        f"Monte Carlo Retirement Simulation\n"
        f"{cfg['num_simulations']:,} paths | "
        f"${cfg['monthly_contribution']:,}/mo contribution | "
        f"Mean return {cfg['annual_return_mean']:.0%}",
        fontsize=11
    )
    ax1.legend(fontsize=10)

    # TODO: Format the y-axis as currency using mticker.FuncFormatter
    ax1.yaxis.set_major_formatter(
        mticker.FuncFormatter(lambda x, _: f"${x:,.0f}")
    )

    # --- Right panel: final balance histogram ---
    # TODO: Plot a histogram of final_balances with 50 bins
    ax2.hist(final_balances, bins=50, color="steelblue", edgecolor="white", alpha=0.8)

    # TODO: Add a vertical line at the median final balance (blue, solid)
    ax2.axvline(np.median(final_balances), color="blue", linewidth=2,
                label=f"Median: ${np.median(final_balances):,.0f}")

    # TODO: Add a vertical line at $1,000,000 (green, dashed)
    ax2.axvline(1_000_000, color="green", linewidth=1.5, linestyle="--",
                label="$1 Million milestone")

    ax2.set_xlabel("Final Balance at Retirement", fontsize=12)
    ax2.set_ylabel("Number of Simulations", fontsize=12)
    ax2.set_title("Distribution of Final Balances at Age 65", fontsize=11)
    ax2.legend(fontsize=10)
    ax2.xaxis.set_major_formatter(
        mticker.FuncFormatter(lambda x, _: f"${x/1e6:.1f}M")
    )

    plt.tight_layout()
    plt.savefig("retirement_simulation.png", dpi=150, bbox_inches="tight")
    print("Saved: retirement_simulation.png")

    # Encode as base64 for the multimodal model API
    buf = io.BytesIO()
    fig.savefig(buf, format="png", dpi=150, bbox_inches="tight")
    plt.close(fig)
    buf.seek(0)
    return base64.b64encode(buf.getvalue()).decode("utf-8")

Step A.4: Add a statistics summary.

def save_statistics(balances, cfg, path="simulation_stats.txt"):
    """
    Compute and save key statistics from the simulation to a text file.
    """
    final = balances[:, -1]
    prob_million = (final >= 1_000_000).mean()

    lines = [
        f"Simulation parameters:",
        f"  Monthly contribution: ${cfg['monthly_contribution']:,}",
        f"  Mean annual return:   {cfg['annual_return_mean']:.1%}",
        f"  Return std dev:       {cfg['annual_return_std']:.1%}",
        f"  Number of paths:      {cfg['num_simulations']:,}",
        f"",
        f"Final balance at retirement (age {cfg['retirement_age']}):",
        f"  10th percentile: ${np.percentile(final, 10):>12,.0f}",
        f"  Median (50th):   ${np.median(final):>12,.0f}",
        f"  90th percentile: ${np.percentile(final, 90):>12,.0f}",
        f"  Mean:            ${final.mean():>12,.0f}",
        f"",
        f"Probability of reaching $1 million: {prob_million:.1%}",
    ]

    text = "\n".join(lines)
    print(text)
    with open(path, "w") as f:
        f.write(text)
    print(f"Saved: {path}")
    return text

Step A.5: Wire Step A together and run a smoke test.

if __name__ == "__main__":
    cfg = load_config()
    np.random.seed(42)

    print("Running simulation...")
    balances = simulate_retirement(cfg)

    print("Generating visualization...")
    image_b64 = plot_simulation(balances, cfg)

    print("Computing statistics...")
    stats_text = save_statistics(balances, cfg)

Expected output when Step A is complete:

Running simulation...
Generating visualization...
Saved: retirement_simulation.png
Computing statistics...
Simulation parameters:
  Monthly contribution: $500
  Mean annual return:   7.0%
  Return std dev:       12.0%
  Number of paths:      1,000

Final balance at retirement (age 65):
  10th percentile: $      341,208
  Median (50th):   $    1,042,577
  90th percentile: $    2,847,031
  Mean:            $    1,298,451

Probability of reaching $1 million: 53.2%
Saved: simulation_stats.txt

Your exact numbers will vary by random seed. The two-panel PNG should show: (left) a fan of gray paths narrowing to a few wide-spread outcomes at age 65, with a visible median line and two outer dashed bands; (right) a right-skewed histogram of final balances with a vertical median line and a $1M milestone line.

Troubleshooting — Step A

ValueError: could not broadcast input array from shape... in simulate_retirement Check that results is indexed as results[sim, year] and that year runs from 0 to years - 1. Off-by-one errors here cause shape mismatches.

Chart y-axis shows scientific notation instead of dollar amounts The FuncFormatter must be assigned after ax1 is populated. If you call it before plotting, matplotlib may overwrite it. Move the formatter call to just before plt.tight_layout().

Simulation runs but all paths converge to zero The balance clip at zero combined with a very negative return draw can zero out a portfolio early. Check that you are adding the contribution before applying the return, and that annual_return_std is not set unreasonably high in your config.

Checkpoint: Before moving to Step B, make sure you can answer:

What does the width of the fan (the gap between the 10th and 90th percentile lines) represent in plain English? How would you expect it to change if you doubled num_simulations?

Why does the histogram in the right panel skew right rather than form a symmetric bell curve?

If starting_savings were $0, what would change in the simulation? In the chart? Test it.

Step B: Connecting to a Multimodal Model

You will send the PNG you just generated to a local multimodal model via the Ollama API and conduct a two-turn conversation about the chart.

Step B.1: Write the API call function.

def ask_multimodal_model(image_b64, question, cfg):
    """
    Send an image and a text question to a local multimodal model via Ollama.

    Args:
        image_b64: base64-encoded PNG string (from plot_simulation).
        question:  text prompt to accompany the image.
        cfg:       config dict containing 'model' and 'ollama_url'.

    Returns:
        str: the model's text response, or an error message.
    """
    payload = {
        "model": cfg["model"],
        "prompt": question,
        "images": [image_b64],
        "stream": False,
    }
    try:
        url = cfg["ollama_url"] + "/api/generate"
        response = requests.post(url, json=payload, timeout=120)
        response.raise_for_status()
        return response.json()["response"]
    except Exception as e:
        print(f"[montecarlo:ask_multimodal_model] {e}")
        traceback.print_exc()
        raise

Step B.2: Write the two-turn conversation.

The first turn sends a structured prompt that tells the model its role, what to look at, how to format its response, and what audience to assume. The second turn presses it on a specific quantitative claim.

def run_analysis_conversation(image_b64, cfg):
    """
    Conduct a two-turn conversation with the multimodal model about the chart.

    Turn 1: structured analysis with four required sections.
    Turn 2: follow-up pressing the model to estimate a specific probability.

    Returns:
        tuple: (response_turn1: str, response_turn2: str)
    """
    initial_prompt = (
        "You are a financial educator analyzing a Monte Carlo retirement simulation "
        "chart for a college student audience with no prior finance background.\n\n"
        "The chart has two panels:\n"
        "- Left panel: 1,000 simulated portfolio paths from age 25 to 65, with a "
        "solid blue median line and two red dashed lines showing the 10th and 90th "
        "percentile bounds.\n"
        "- Right panel: a histogram of final portfolio balances at age 65, with a "
        "vertical blue line at the median and a vertical green dashed line at $1 million.\n\n"
        "Please analyze the chart and respond in exactly four numbered sections:\n"
        "1. What the spread of paths (the gap between the dashed red lines) tells us "
        "about retirement savings risk.\n"
        "2. Your estimate of what percentage of simulations ended above $1 million, "
        "based on the histogram.\n"
        "3. One specific, actionable insight for a 25-year-old starting their career.\n"
        "4. One limitation of this simulation that a tool deployer should disclose to users."
    )

    print("=== Turn 1: Initial Analysis ===")
    response1 = ask_multimodal_model(image_b64, initial_prompt, cfg)
    print(response1)

    # TODO: Formulate a follow-up question that presses the model on a specific
    # quantitative claim from its Turn 1 response.
    # The suggested follow-up below asks it to show its reasoning for the percentage
    # estimate — this is where numerical precision often breaks down.
    followup = (
        "Based specifically on the histogram in the right panel, walk me through your "
        "reasoning for the percentage estimate you gave in section 2. What visual "
        "features of the histogram did you use, and how confident are you in that number?"
    )

    print("\n=== Turn 2: Follow-Up ===")
    response2 = ask_multimodal_model(image_b64, followup, cfg)
    print(response2)

    # Save both turns to a file for your writeup
    with open("model_responses.txt", "w") as f:
        f.write("=== Turn 1 ===\n")
        f.write(response1 + "\n\n")
        f.write("=== Turn 2 ===\n")
        f.write(response2 + "\n")
    print("\nSaved: model_responses.txt")

    return response1, response2

Step B.3: Add Step B to your main block.

Extend the if __name__ == "__main__": block from Step A:

    print("\nRunning multimodal analysis...")
    response1, response2 = run_analysis_conversation(image_b64, cfg)

Example of a good AI response (Turn 1): The model might correctly observe that the fan widens dramatically after age 40, that the histogram is right-skewed indicating many paths cluster below the median, and offer a concrete suggestion like “increasing monthly contributions by even $100 reduces your worst-case (10th percentile) outcome significantly.” These are pattern-level observations that vision models handle well.

Example of a flawed AI response (Turn 1, section 2): A model might state: “Based on the histogram, approximately 68% of simulations reached $1 million by retirement.” If your statistics file shows the true probability is 53%, this is a fabricated number — the model estimated from visual impression rather than counting. This is the kind of specific, confident numerical claim that looks authoritative but is wrong, and is exactly what Step D asks you to analyze — the same hallucination-hunting you did against text in the core lab’s citation audit, now against pixels.

Troubleshooting — Step B

KeyError: 'response' from the API call The /api/generate endpoint returns {"response": "..."} for non-chat completions. The /api/chat endpoint returns {"message": {"content": "..."}}. Make sure you are using /api/generate in ask_multimodal_model, not /api/chat.

Model returns an empty string or very short response Add "Describe the image in detail before analyzing it." as the first sentence of your prompt. Some vision models require an explicit grounding instruction before they will engage with the analytical questions.

Image is too large and the request times out Add a resize step before encoding: fig.savefig(buf, ...) then from PIL import Image; img = Image.open(buf); img = img.resize((800, 600)); ... and re-encode. Alternatively, lower the dpi parameter in fig.savefig to 100.

ollama pull llava fails or the model is not recognized Try ollama pull moondream (smaller, faster) or ollama pull bakllava. Update "model" in config.json to match.

Checkpoint: Before moving to Step C, make sure you can answer:

What specific percentage did the model report for simulations reaching $1 million? What does your statistics file say the true value is? Are they the same?

In Turn 2, did the model’s confidence in its estimate increase, decrease, or stay the same? What does this tell you about using follow-up questioning as a verification strategy?

Look at the model’s response to section 4 (simulation limitations). Did it identify any limitation that you had not already considered?

Step C: Parameter Sensitivity

Run your simulation under three configurations and record the results. Edit config.json between runs, or write a loop that overrides specific keys programmatically.

Configuration	`annual_return_mean`	`annual_return_std`	Label
Pessimistic	0.04	0.15	Poor market, high volatility
Baseline	0.07	0.12	Historical average (default)
Optimistic	0.10	0.08	Strong market, lower volatility

For each configuration, record in your writeup:

Median final balance
Probability of reaching $1 million
10th percentile (worst-case) balance
90th percentile (best-case) balance

Then answer: which parameter change had a larger effect on the median — raising the mean return from 0.07 to 0.10, or lowering the standard deviation from 0.12 to 0.08? Use numbers from your runs, not intuition.

To automate the three runs:

def run_sensitivity_analysis(base_cfg):
    """Run the simulation under three parameter configurations and print a comparison table."""
    scenarios = [
        {"label": "Pessimistic",  "annual_return_mean": 0.04, "annual_return_std": 0.15},
        {"label": "Baseline",     "annual_return_mean": 0.07, "annual_return_std": 0.12},
        {"label": "Optimistic",   "annual_return_mean": 0.10, "annual_return_std": 0.08},
    ]

    print(f"\n{'Scenario':<14} {'Median':>14} {'P(>$1M)':>10} {'10th pct':>14} {'90th pct':>14}")
    print("-" * 70)

    for scenario in scenarios:
        cfg = {**base_cfg, **scenario}
        np.random.seed(42)
        balances = simulate_retirement(cfg)
        final = balances[:, -1]
        print(
            f"{scenario['label']:<14} "
            f"${np.median(final):>13,.0f} "
            f"{(final >= 1_000_000).mean():>9.1%} "
            f"${np.percentile(final, 10):>13,.0f} "
            f"${np.percentile(final, 90):>13,.0f}"
        )

Expected output (your numbers will vary slightly):

Scenario       Median   P(>$1M)       10th pct       90th pct
----------------------------------------------------------------------
Pessimistic    $  314,042       4.2%   $   73,501   $  877,209
Baseline       $1,042,577      53.2%   $  341,208   $2,847,031
Optimistic     $2,891,044      89.7%   $1,201,330   $5,912,448

Checkpoint: Before moving to Step D, make sure you can answer:

Between the pessimistic and baseline scenarios, the median more than tripled. What does this tell you about the compounding effect of even a moderate improvement in average returns over 40 years?

The pessimistic scenario has a higher annual_return_std than the baseline. How does higher volatility affect the 10th percentile differently than the median? Why?

If a user only saw the optimistic chart and made contribution decisions based on it, what harm could result?

Step D: Critical Analysis

This is the most important part of this direction. In Steps A–C you built a tool. Now you evaluate what happens when AI interprets that tool’s output — the same audit discipline as the core lab, moved from retrieved text to a rendered chart.

Step D.1: Read the chart yourself first.

Before re-reading the model’s responses, look at your baseline chart and note:

Approximately what percentage of paths appear to end above $1 million (your best visual estimate)
Where roughly the median falls
Whether the 10th percentile line ever reaches zero during the accumulation years
Whether the histogram distribution is symmetric, right-skewed, or left-skewed

Write down your estimates. Do not change them after reading the model’s responses.

Step D.2: Compare to the model’s Turn 1 response.

Identify three specific differences between what you read from the chart and what the model reported. For each difference, record:

The exact AI output excerpt (copy-paste from model_responses.txt)
Whether the AI was correct, approximately correct, or wrong
Your best explanation for why the discrepancy occurred

At least one of your three differences must be a case where the model was wrong or imprecise about a number, not just phrased something differently.

Step D.3: Propose a prompt engineering improvement.

For the one numerical error you identified, propose a single change to the initial_prompt in run_analysis_conversation that would have reduced the chance of that error. Implement it, re-run Step B, and record whether the response improved.

Useful strategies to try:

Adding an explicit disclaimer: “If you cannot read a precise number from the chart, say ‘approximately’ and give a range.”
Asking the model to reason step by step before stating a number: “Before giving a percentage, describe what you see in the histogram bin by bin.”
Restricting the scope: “Only comment on what is visually unambiguous. Flag anything that requires precise numerical reading as uncertain.”

Step D.4: Address the deployment question.

Write a 2–3 sentence guardrail statement you would add to a financial planning tool that uses this AI chart interpretation feature. The statement should appear to the user before they see the AI’s analysis, and should protect against over-reliance on numerical claims that the AI cannot read precisely.

Checkpoint: You have succeeded at this direction when:

Your simulation produces a labeled two-panel PNG and a statistics text file

The multimodal model provides a four-section analysis with at least one follow-up exchange

You have identified at least one specific numerical error in the AI’s response with an exact AI output excerpt

You have proposed and tested a prompt engineering change

Your sensitivity analysis table covers all three configurations with four statistics each

Step E: The Simulation as a Tool — Function-Calling Extension

In Steps A–D, the model only interpreted an experiment you designed. This extension inverts the relationship, bridging this direction to the Tool Use and Function Calling session: you wrap your simulation as a tool with a JSON schema, and the model chooses the parameters, asks your code to invoke the tool, and then interprets the visualization the tool produced. The model never executes anything — it can only request; your code runs the simulation and returns the results.

A fully worked, runnable version of this part (including canned offline responses for machines without Ollama) is in the companion notebook.

Note on models: llava does not support function calling. Use a tool-capable model for the parameter-selection turn (ollama pull llama3.1, or qwen2.5), and keep llava for the vision turn.

Step E.1: Wrap the simulation as a tool.

Refactor your Step A code into a single callable with an agent-friendly signature. The new stock_allocation parameter blends an equity-like return distribution (mean 8%, std 15%) with a bond-like one (mean 3%, std 5%), giving the agent a genuinely meaningful lever to reason about.

def run_retirement_sim(years=40, annual_contribution=6000, stock_allocation=0.8,
                       n_paths=1000, seed=42):
    """
    Run a Monte Carlo retirement simulation and return summary statistics
    plus the path to a saved two-panel chart.

    stock_allocation: fraction 0.0-1.0 in stocks; the remainder in bonds.
    Blend: mean = alloc*0.08 + (1-alloc)*0.03, std = alloc*0.15 + (1-alloc)*0.05.

    Returns:
        dict with keys: median, p10, p90, mean, prob_million, years,
        annual_contribution, stock_allocation, n_paths, seed, chart_path.
    """
    # TODO: build a cfg dict from these arguments (reuse your Step A functions),
    #       call simulate_retirement and plot_simulation, and return the stats dict.

Step E.2: Write the tool’s JSON schema.

The schema — not your Python code — is the tool’s entire interface from the agent’s point of view. Every name, description, and bound you write here shapes what parameters the model will choose.

RETIREMENT_TOOL_SCHEMA = {
    "type": "function",
    "function": {
        "name": "run_retirement_sim",
        "description": ("Run a Monte Carlo retirement savings simulation and return "
                        "summary statistics plus a saved two-panel chart."),
        "parameters": {
            "type": "object",
            "properties": {
                "years": {"type": "integer", "minimum": 1, "maximum": 60,
                          "description": "Years of saving before retirement."},
                "annual_contribution": {"type": "number", "minimum": 0,
                          "description": "Dollars contributed per year."},
                "stock_allocation": {"type": "number", "minimum": 0.0, "maximum": 1.0,
                          "description": "Fraction of the portfolio in stocks; higher raises both expected return and volatility."},
                "n_paths": {"type": "integer", "minimum": 100, "maximum": 10000,
                          "description": "Number of Monte Carlo paths (1000 recommended)."},
                "seed": {"type": "integer", "description": "Random seed for reproducibility."}
            },
            "required": ["years", "annual_contribution", "stock_allocation"]
        }
    }
}

Step E.3: Drive the agent loop.

Give the tool-capable model a plain-English goal plus the schema via Ollama’s /api/chat endpoint with a tools array; it responds with a tool call — a function name and a JSON object of arguments it selected. Your code executes run_retirement_sim with those arguments, then sends the resulting chart to llava (and the exact stats dict back in the prompt) for interpretation.

goal = ("I am 25 and want to know whether contributing $500 a month with a fairly "
        "aggressive portfolio gives me a good chance of retiring at 65 with over "
        "$1 million. Choose appropriate parameters, run the simulation, and "
        "interpret the results for me.")

payload = {
    "model": "llama3.1",
    "messages": [
        {"role": "system", "content": "You are a retirement planning assistant. Use the "
         "simulation tool, choosing parameters that faithfully reflect the user's situation."},
        {"role": "user", "content": goal},
    ],
    "tools": [RETIREMENT_TOOL_SCHEMA],
    "stream": False,
}
# TODO: POST to /api/chat, read message["tool_calls"][0]["function"],
#       invoke run_retirement_sim(**arguments), and save the goal, the tool call,
#       the tool result, and the interpretation to tool_call_transcript.txt.

Step E.4: Critique the agent (exercise).

The agent has now done two things a human analyst would do: chosen the experiment’s inputs and narrated its output. Audit both against simulation_stats.txt and the tool’s returned stats dict:

Parameter choices. The user said “$500 a month” and “retire at 65” starting at 25. Did the model’s annual_contribution equal $500 × 12? Did years equal 40? Is the chosen stock_allocation a defensible reading of “fairly aggressive,” and did the model justify it anywhere? Record each as correct, approximately correct, or wrong, quoting the tool call verbatim from tool_call_transcript.txt.
Interpretation. Compare the model’s narrative to the tool’s exact prob_million and to the ground truth in simulation_stats.txt. Does its qualitative language (“roughly three-quarters,” “more likely than not”) match the actual number? Quote the exact sentence and the exact statistic side by side, exactly as you did in Step D.
Compounding risk. In 2–3 sentences: when the same agent both picks the parameters and interprets the results, how can an early mistranslation (a wrong contribution, an unjustified allocation) compound into a confident but misleading recommendation — and which single check from steps 1–2 would you automate as a guardrail?

Checkpoint: You have completed Step E when:

Your run_retirement_sim tool runs from a single call and returns the stats dict plus a chart path

The model selected parameters via a tool call (not prose), and your transcript captures the goal, the call, the tool result, and the interpretation

Your critique covers at least one judgment of the agent’s parameter choices and one of its interpretation, each backed by a verbatim excerpt and a ground-truth number

Direction 2 Deliverables

Fold these into your Lab 2 submission ZIP and readme:

montecarlo.py — complete simulation, visualization, and multimodal analysis code
config.json — your baseline configuration file
retirement_simulation.png — the two-panel chart from your baseline run
simulation_stats.txt — the statistics summary from your baseline run
model_responses.txt — both turns of the AI conversation from your baseline run
sensitivity_results.txt or equivalent — the three-scenario comparison table
tool_call_transcript.txt — the Step E tool-calling transcript: the user goal, the model’s tool call (name and arguments), the tool’s returned statistics, and the model’s interpretation
Step E critique (in the readme) — the agent’s parameter choices and its interpretation each audited against simulation_stats.txt and the tool’s stats dict, with verbatim excerpts
A section in your readme writeup covering: (1) sensitivity analysis with all three scenarios and four statistics each, (2) critical analysis with three AI/human comparison items including at least one AI error with verbatim excerpt, (3) the prompt engineering change you tested and whether it helped, (4) your guardrail statement for deployment
pair_log.txt — driver/navigator swap log with timestamps and roles

What proficient work looks like (Direction 2)

The simulation is configurable via a JSON config file; the visualization includes the median, 10th and 90th percentile bands, and the final-balance histogram, with a text summary of key statistics saved alongside the image, and edge cases are handled.
The multimodal integration sends a valid base64-encoded PNG in the images array to /api/generate, parses the response from response.json()["response"], runs a structured Turn 1 (role, four numbered sections, audience) plus a Turn 2 that presses a specific quantitative claim, and saves both turns.
The comparative analysis names three specific AI-versus-human differences, with at least one showing a wrong or imprecise number backed by a verbatim AI excerpt alongside the true value from simulation_stats.txt, tests one prompt-engineering change and reports whether it helped, and (if the tool-calling extension is attempted) audits both the agent’s parameter choices and its interpretation.
The writeup tabulates the four statistics for all three sensitivity scenarios, states which parameter change moved the median more, judges AI interpretation quality with a verbatim excerpt, and delivers a 2–3 sentence plain-language user-facing guardrail.

Direction 2 Reflection Prompts

What does the spread of simulation paths tell you that a single projected number (like “you will have $800,000 at retirement”) does not? Point to a specific visual feature of your chart that would disappear if you replaced the simulation with a single-path projection.
The AI reported a specific probability from the histogram. How would you verify whether it was right? What tools would you need, and what does this verification challenge tell you about using AI for quantitative analysis of charts?
In the core lab you audited citations against text; here you audited a number against a chart. What was harder to verify — a text citation or a number read from an image — and why?
In Step C, the pessimistic and optimistic scenarios produced dramatically different outcomes despite both using “reasonable” parameters. What does this imply about how a financial planning tool should present parameter uncertainty to a non-expert user?
If collaboration beyond your pair occurred, identify it. Do you certify that this submission represents your pair’s original work? Please identify any and all portions of your submission that were not originally written by you.
Approximately how many hours did this direction take?

Submission

In your submission, please include answers to any questions asked on the assignment page, as well as the questions listed below, in your README file. If you wrote code as part of this assignment, please describe your design, approach, and implementation in a separate document prepared using a word processor or typesetting program such as LaTeX. This document should include specific instructions on how to build and run your code, and a description of each code module or function that you created suitable for re-use by a colleague. In your README, please include answers to the following questions:

Describe what you did, how you did it, what challenges you encountered, and how you solved them.
Please answer any questions found throughout the narrative of this assignment.
If collaboration with a buddy was permitted, did you work with a buddy on this assignment? If so, who? If not, do you certify that this submission represents your own original work?
Please identify any and all portions of your submission that were not originally written by you (for example, code originally written by your buddy, or anything taken or adapted from a non-classroom resource). It is always OK to use your textbook and instructor notes; however, you are certifying that any portions not designated as coming from an outside person or source are your own original work.
Approximately how many hours it took you to finish this assignment (I will not judge you for this at all...I am simply using it to gauge if the assignments are too easy or hard)?
Your overall impression of the assignment. Did you love it, hate it, or were you neutral? One word answers are fine, but if you have any suggestions for the future let me know.
Using the grading specifications on this page, discuss briefly the grade you would give yourself and why. Discuss each item in the grading specification.

Any other concerns that you have. For instance, if you have a bug that you were unable to solve but you made progress, write that here. The more you articulate the problem the more partial credit you will receive (it is fine to leave this blank).

Assignment Rubric

Description Pre-Emerging (< 50%) Beginning (50%) Progressing (85%) Proficient (100%)

Pipeline Implementation (30%) The pipeline fails to index or query due to major issues, or the program fails to run The pipeline runs but fails on test questions due to one or more minor issues The pipeline indexes and answers correctly with citations, but a component such as abstention or configuration is fragile or incomplete The pipeline indexes, retrieves, answers with cited bracketed source numbers, and abstains with the designated phrase when no chunk is relevant; a screenshot or log shows all three behaviors (answer-with-citation, abstention, and the bare-model hallucination contrast); configuration is externalized and exceptions are handled with located messages and tracebacks

Chunking Strategy and Justification (25%) A single arbitrary chunking is used without discussion A chunking choice is stated but not compared against an alternative Two chunking strategies are compared on a small question set with results reported At least two chunking strategies are compared on a defined question set; recall@k is reported for k in {1,3,5} for each strategy in a table; the shipped choice is defended with a specific numeric comparison (e.g., "strategy A achieves recall@3 of 0.80 vs. 0.60 for strategy B on our question set")

Evaluation and Citation Audit (25%) No evaluation is provided Informal trials are described without a protocol or metric A question set with recall at k and an answer accuracy measure is evaluated, with limited citation auditing A question set of at least ten questions is evaluated with recall@k and answer accuracy; every citation in a sample of at least ten answers is audited by hand for faithfulness; a faithfulness rate (e.g., "9/10 citations correctly supported the claim") is reported; any failures are shown verbatim and classified using the hallucination taxonomy from class

Code Quality and Documentation (10%) Code commenting and structure are absent, or code structure departs significantly from best practice Code commenting and structure is limited in ways that reduce the readability of the program Code documentation is present that re-states the explicit code definitions Every non-trivial function has a docstring; all network, embedding, and database operations are wrapped in exception handlers that print a located message (e.g., [lab2:query_corpus]) followed by a traceback; model name, chunk size, overlap, top-k, and abstention threshold are read from a JSON config file rather than hardcoded

Writeup, Reflection, and Submission (10%) An incomplete submission is provided The program is submitted, but not according to the directions in one or more ways The program is submitted according to the directions with a minor omission, with at least superficial responses to the reflection prompts The program is submitted according to the directions, including a readme writeup, a pair programming log with at least two timestamped role swaps, a corpus datasheet covering sources, time range, representation gaps, and known limitations, and reflection answers that each cite a specific experimental result from the lab rather than restating the prompt

Please refer to the Style Guide for code quality examples and guidelines.

Description	Pre-Emerging (< 50%)	Beginning (50%)	Progressing (85%)	Proficient (100%)
Pipeline Implementation (30%)	The pipeline fails to index or query due to major issues, or the program fails to run	The pipeline runs but fails on test questions due to one or more minor issues	The pipeline indexes and answers correctly with citations, but a component such as abstention or configuration is fragile or incomplete	The pipeline indexes, retrieves, answers with cited bracketed source numbers, and abstains with the designated phrase when no chunk is relevant; a screenshot or log shows all three behaviors (answer-with-citation, abstention, and the bare-model hallucination contrast); configuration is externalized and exceptions are handled with located messages and tracebacks
Chunking Strategy and Justification (25%)	A single arbitrary chunking is used without discussion	A chunking choice is stated but not compared against an alternative	Two chunking strategies are compared on a small question set with results reported	At least two chunking strategies are compared on a defined question set; recall@k is reported for k in {1,3,5} for each strategy in a table; the shipped choice is defended with a specific numeric comparison (e.g., "strategy A achieves recall@3 of 0.80 vs. 0.60 for strategy B on our question set")
Evaluation and Citation Audit (25%)	No evaluation is provided	Informal trials are described without a protocol or metric	A question set with recall at k and an answer accuracy measure is evaluated, with limited citation auditing	A question set of at least ten questions is evaluated with recall@k and answer accuracy; every citation in a sample of at least ten answers is audited by hand for faithfulness; a faithfulness rate (e.g., "9/10 citations correctly supported the claim") is reported; any failures are shown verbatim and classified using the hallucination taxonomy from class
Code Quality and Documentation (10%)	Code commenting and structure are absent, or code structure departs significantly from best practice	Code commenting and structure is limited in ways that reduce the readability of the program	Code documentation is present that re-states the explicit code definitions	Every non-trivial function has a docstring; all network, embedding, and database operations are wrapped in exception handlers that print a located message (e.g., [lab2:query_corpus]) followed by a traceback; model name, chunk size, overlap, top-k, and abstention threshold are read from a JSON config file rather than hardcoded
Writeup, Reflection, and Submission (10%)	An incomplete submission is provided	The program is submitted, but not according to the directions in one or more ways	The program is submitted according to the directions with a minor omission, with at least superficial responses to the reflection prompts	The program is submitted according to the directions, including a readme writeup, a pair programming log with at least two timestamped role swaps, a corpus datasheet covering sources, time range, representation gaps, and known limitations, and reflection answers that each cite a specific experimental result from the lab rather than restating the prompt