View on GitHub

CS357

Foundations of Artificial Intelligence

CS357: Foundations of Artificial Intelligence - The Local Agent Stack

Purpose

To wire the containers you can already run into one system, which is the prep the Local Agent Lab’s Compose-stack and hardening directions assume.

About This Tutorial

Supplemental: required prep only for the Local Agent Lab Directions 2-3 (the Compose-stack and container-hardening directions). This is self-paced reference material rather than a class session. Do the installs and image pulls before you start the direction that needs them: Docker Desktop plus roughly 6 GB of images. Build the 3-container minimal stack first (Ollama, llmproxy, Open WebUI); the full 20-service tour below is reference. Bring what will not start to the open studio in How I AI, Part III.

The Docker from Zero: Containers for Agent Builders tutorial gave you one container, and one container is a demo; a stack of containers that talk to each other is infrastructure. This tutorial deploys the course’s local AI ecosystem (model servers, a unifying gateway, tool servers, web frontends, and autonomous agents) and teaches the wiring discipline that makes two dozen services coexist: tiered roles, a port plan, per-service identity directories, and host.docker.internal as the connective tissue. The path today: the tier model → the inference foundation → the gateway → frontends and tools → agents → wiring and verification.

Key Concepts

Term Plain-English Definition Where You’ll Meet It
Container A self-contained, isolated software environment, like a shipping container that carries everything a program needs to run, so it works identically on any machine. Each service (Ollama, Open WebUI, etc.) runs in its own container
Tier A layer in a system where every component in that layer has the same job. Grouping by tier makes it easy to reason about which piece does what. The Inference tier runs models; the Gateway tier routes requests to them
Gateway A single entry point that routes incoming requests to the right backend service. Like a hotel switchboard: you call one number, and it connects you to housekeeping or room service. llmproxy on port 4000 routes to Ollama, LocalAI, or cloud models
Port A numbered “door” on a computer through which one specific service listens for connections. Two services cannot share the same port number on the same machine. Ollama listens on port 11434; Open WebUI on port 3000
host.docker.internal A special hostname that, from inside a Docker container, refers back to the host machine where Docker is running. Essential for containers that need to call services running on the host. The gateway uses this to reach Ollama running natively on the host
Identity Directory A folder on the host machine that is mounted into a container as its persistent storage. When the container is deleted and recreated, its state survives because it lived on the host. $HOME/agents/hermes/home persists the Hermes agent’s memory

Part I: Architecture Before Containers

In this part, you will learn the five-tier mental model that organizes every service in our stack, so that instead of memorizing twenty container names, you can place any new service by asking “what job does it do?” This mental model is the same one professional engineers use to design and debug complex distributed systems.

1. The Tier Model

Why this matters: When you first see a list of twenty container names, it looks overwhelming, like being handed a car’s parts list before you have seen a car. The tier model is the map that organizes all those parts. Instead of memorizing twenty names, you memorize five jobs, and every container slots into one. This is the same principle engineers use to design any complex system: layer it so each layer has one responsibility, and components in different layers communicate through clean interfaces. Think of it like a restaurant: the kitchen (inference) prepares food, the server (gateway) carries it to tables, the dining room (frontend) is where guests sit, and the delivery app (agent tier) places orders automatically.

Memorizing two dozen container names is hopeless; memorizing five tiers is easy, and every container in our stack slots into one:

Tier Job Our Containers Example / In Our Course
Inference Run AI models and expose them through a standard API that other software can call ollama, lmstudio, local-ai, longcat-video Ollama on port 11434 answers questions like “summarize this document”
Gateway Provide one stable endpoint that hides which backend model is actually running, so swapping models requires only one config change llmproxy LiteLLM on port 4000 routes your request to whichever model you configured
Tools Give agents capabilities beyond text generation: web search, databases, external APIs mcpproxy, searxng, surrealdb SearXNG lets agents search the web privately; SurrealDB stores results persistently
Frontends Provide human-facing interfaces (chat UIs, notebooks, voice, slides) that talk to the gateway open-webui, open-terminal, open-design, open-notebook, voicebox, presenton, calibre-web Open WebUI at port 3000 gives you a ChatGPT-style chat interface connected to your local models
Agents Run autonomous or task-bounded workers that can take multi-step actions without human input at each step hermes, freebuff, agent0, openhands-server/openhands, nanoclaw, nanoclaw-dind, zeroclaw, openclaw-gateway, n8n n8n at port 5678 runs scheduled workflows; Hermes handles tool-calling tasks

Two structural principles govern everything. First, all inference flows through the gateway: every frontend and agent sends OpenAI-compatible requests to llmproxy, which routes to Ollama, LocalAI, or a cloud free tier per one YAML file, so swapping a model never touches more than that file. Second, every service gets its own identity directory under $HOME/agents/<service>/, bind-mounted in, so any container can be destroyed and recreated without losing state, and no two services can contaminate each other.

2. The Port Plan

Why this matters: Port conflicts are the most common reason a new service silently fails to start: it tries to claim a port that is already taken, crashes quietly, and you spend an hour debugging what looks like a network problem. The discipline here is simple: write down every port assignment before you run anything. It takes two minutes and prevents hours of frustration. Think of ports like parking spaces in a lot: each car (service) needs its own assigned spot, and two cars cannot share one spot.

Containers collide on ports before they collide on anything else, so the plan comes first. The well-known defaults anchor the table; every other service is assigned a host port in its run.sh, and the table (kept in your stack repository) is the single source of truth:

Service Host Port Notes Example / In Our Course
ollama 11434 Native service or container; the workhorse inference engine curl http://localhost:11434/api/tags lists your downloaded models
lmstudio 1234 LM Studio’s local server default; adds a friendly GUI for model management Useful when you want a visual model browser instead of the command line
local-ai 8080 OpenAI-compatible GGUF inference; adds Whisper transcription and image generation Use when you need voice-to-text or image generation alongside language models
llmproxy 4000 The gateway; every frontend and agent points here instead of directly at models All your apps use http://localhost:4000/v1 as their “OpenAI” base URL
open-webui 3000 Browser-based chat frontend resembling ChatGPT Point your browser to http://localhost:3000 for a full chat UI
searxng 8081 Private metasearch engine; default image port 8080 is remapped to avoid collision with local-ai Agents call this to search the web without sending queries to Google
n8n 5678 Visual workflow automation, build pipelines with a drag-and-drop canvas “Every morning at 7am, summarize new emails and save to workspace”
surrealdb 8000 Multi-model database for persistent agent storage Agents write facts here and retrieve them across sessions
calibre-web 8083 E-book library management with a web interface Index your PDF collection and let agents search it
agent0 8082 Autonomous agent with a full web UI for monitoring its actions Watch Agent Zero plan and execute multi-step tasks in real time
remaining services assigned One row each in your stack’s port table, no exceptions Before adding any new service, add its row first

Notice the searxng row: its image default (8080) collides with local-ai, so we remap on the host side with -p 8081:8080, which is the entire point of the -p HOST:CONTAINER Docker flag. The rule: before any docker run, the service gets a row. Port conflicts then become impossible by construction rather than debugged at midnight.

Place the Pieces

Why this matters: Architecture decisions made on paper are far cheaper to revise than architecture decisions discovered in production at midnight. This model practices the reasoning you will use every time you add a new service: where does it belong, what does it need, and how does it connect to what is already running?

Questions to Work Through

  1. A teammate proposes having open-webui talk directly to ollama “to skip a hop.” What does the stack lose? Name two concrete things the gateway tier buys, using the one-YAML-file argument.

Hint: Imagine you have five frontends all pointed directly at Ollama. Now you want to swap Ollama for a different model. How many config files do you have to update? Now imagine they all point at the gateway. How many files change?

  1. nanoclaw-dind runs Docker inside Docker so its agent can spawn sandboxed sub-containers. Which tier does it belong to, and what governance property is the inner Docker layer providing? Connect to the irreversible-actions taxonomy from class.

    Hint: If an agent can execute arbitrary code, what is the worst thing it could accidentally do to the host machine? What does wrapping that code execution inside an inner Docker container prevent?

  2. Assign host ports to three unassigned services from the table, checking your choices against every existing row. State the collision you avoided.

Hint: List all reserved ports first: 11434, 1234, 8080, 4000, 3000, 8081, 5678, 8000, 8083, 8082. Your three new services must pick ports not in that list. Common safe ranges: 8084-8099, 7000-7999, 9000-9099.

Part II: Standing Up the Core

In this part, you will bring up the stack tier by tier: inference first, then gateway, then frontends and tools, then agents. Each tier must be working before the next one is added. This incremental approach is the professional practice: it isolates failures and makes debugging tractable.

3. Inference: Ollama First

Ollama is the inference foundation, the service that downloads model files and answers requests. Everything else can be added incrementally once it answers:


# Native install (preferred: keeps model I/O off the Docker path)
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:3b && ollama pull llama3 && ollama pull hermes3:8b
curl http://localhost:11434/api/tags        # verify: lists your models

lmstudio and local-ai are alternates with the same OpenAI-compatible surface (LM Studio adds a friendly GUI for model management on port 1234; LocalAI adds whisper transcription and image generation behind 8080); the gateway makes the choice invisible to everything downstream, so start with Ollama and add the others when a need appears.

4. The Gateway: llmproxy

The gateway (a LiteLLM-style router, a program that receives requests in the standard OpenAI format and forwards them to whichever local or cloud model you have configured) is one Compose service plus one routing file. Pay attention to the api_base field: this is where the host.docker.internal hostname (the special address that lets a container reach the host machine) becomes critical.

Build it as a numbered checklist; complete and verify each step before starting the next:

  1. Create the routing file at llmproxy/litellm_config.yaml. This maps each public model name to a backend:
   # llmproxy/litellm_config.yaml
   model_list:
     - model_name: llama3
       litellm_params:
         model: ollama/llama3
         api_base: http://host.docker.internal:11434
     - model_name: qwen2.5-3b
       litellm_params:
         model: ollama/qwen2.5:3b
         api_base: http://host.docker.internal:11434
  1. Create the Compose file at llmproxy/docker-compose.yml, mounting the routing file read-only:
   # llmproxy/docker-compose.yml
   services:
     llmproxy:
       image: ghcr.io/berriai/litellm:main-latest
       ports: ["4000:4000"]
       volumes: ["./litellm_config.yaml:/app/config.yaml:ro"]
       command: ["--config", "/app/config.yaml", "--port", "4000"]
       extra_hosts: ["host.docker.internal:host-gateway"]
       restart: unless-stopped
  1. Start the service from the llmproxy/ directory:
   docker compose up -d

Checkpoint: what should docker compose ps show now?

Expected answer Exactly one service, `llmproxy`, with status `Up` (or `running`) and the ports column showing `0.0.0.0:4000->4000/tcp`. If the status is `Restarting`, run `docker logs llmproxy`; the usual culprit is a typo in the mounted config path or invalid YAML in the routing file.
  1. Verify from the host that the gateway answers and knows its models:
   curl -s http://localhost:4000/models -H "Authorization: Bearer sk-litellm-local"

Checkpoint: what should this curl return?

Expected answer A JSON object whose `data` array lists the two configured model names, `llama3` and `qwen2.5-3b`. An empty list means the routing file did not mount into the container; `connection refused` means the container is not running; go back to the step 3 checkpoint.

Read the config’s api_base carefully: the gateway is itself a container, so it reaches host-resident Ollama via host.docker.internal, and the extra_hosts line is what makes that name resolve on Linux. This one file is where every future model, local or cloud, gets added.

5. Frontends and Tools

Next, build one visually. Once the stack is up, go on to the Visual Agent with Langflow activity and build the same kind of agent on a canvas, same concepts, no Python authorship. Part IV of that deck is the hands-on build.

With the gateway answering, frontends attach by URL. Open WebUI:

docker run -d --name open-webui -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v "$HOME/agents/openwebui/data:/app/backend/data" \
  ghcr.io/open-webui/open-webui:main

# Then in its Admin Settings -> Connections -> OpenAI:

#   URL http://host.docker.internal:4000/v1, key sk-litellm-local

Isolation and trust boundaries, meaning which tier may talk to which and what a container actually isolates, are worked through in the optional activity Containerization and Safety.

6. The Agent Tier

Agents are where the stack earns its name; they are the autonomous workers that use the inference, gateway, frontend, and tool tiers as their instruments. Each agent has a distinct personality and capability level, but they all deploy the same way: a port row, an identity directory, and a gateway URL in their config. The code below shows the pattern for hermes, a tool-calling agent; the others follow the same template.

docker pull nousresearch/hermes-agent:latest
mkdir -p "$HOME/agents/hermes/home"
docker run --rm -it --name hermes \
  --add-host=host.docker.internal:host-gateway \
  -v "$HOME/agents/hermes/home:/home/hermes/.hermes" \
  -v "$HOME/agents/workspace:/workspace" \
  nousresearch/hermes-agent:latest

The others slot in by personality: agent0 (Agent Zero) is the fully autonomous, self-improving end of the spectrum with a web UI; openhands-server plus openhands provide a software-engineering agent with its own runtime sandbox; freebuff is our task-bounded harness (give it a task and a workspace, it executes and exits); the nanoclaw/nanoclaw-dind/zeroclaw family and openclaw-gateway are lightweight Claude-style workers behind a routing gateway; and n8n is the workflow scheduler that strings any of them into timed pipelines (its visual editor at port 5678 makes it the natural home for “every morning, summarize new items”). Deploy each the same way, without exception: a port row, an identity directory, --add-host, gateway URL in its config. The uniformity is the lesson.

7. Verification: The Wiring Matrix

A stack is not up until its connections are proven, pairwise, from the right vantage points. The matrix below tests each critical link: host-to-service, container-to-host, and end-to-end through the full chain. Run these commands in order; a failure at any cell tells you exactly which link broke and where to look first.

Build the matrix:


# From the HOST: is each service alive?
curl -s http://localhost:11434/api/tags | head -3        # ollama
curl -s http://localhost:4000/models -H "Authorization: Bearer sk-litellm-local" | head -3
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3000   # open-webui

# From INSIDE a container: does the host alias resolve and answer?
docker exec -it open-webui curl -s http://host.docker.internal:4000/models \
  -H "Authorization: Bearer sk-litellm-local" | head -3

# End to end: one chat completion through the whole chain
curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-litellm-local" -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5-3b","messages":[{"role":"user","content":"Say STACK OK."}]}'

When a cell of the matrix fails, the diagnostic ladder from the Docker from Zero: Containers for Agent Builders tutorial applies: host-side first, container-side second, then the --add-host flag, the port row, and the logs, in that order.

Part III: Practice

In this part, you will build and verify a working minimal stack, extend it with a tool and an agent, and deliberately break and fix a common networking issue, so that when something goes wrong in your project work, you have already seen and diagnosed it.

8. Exercises

  1. Minimal viable stack. Deploy Ollama, llmproxy, and Open WebUI; complete a chat in the browser that round-trips through all three; submit your port table and the end-to-end curl output.
  1. Add a tool. Deploy searxng on its remapped port and confirm from inside another container that http://host.docker.internal:8081 answers a search query. Explain in two sentences why the remap was necessary.
  1. Add an agent. Deploy hermes (or agent0 if you prefer a web UI) with its identity directory, give it one small task in the shared workspace, destroy the container, recreate it, and demonstrate that its identity persisted.
  1. Break and fix. Remove the --add-host flag from one Linux deployment (or simulate by using localhost in a config), capture the exact failure symptom, restore it, and write the three-line postmortem.
  1. Automate one flow. In n8n, build a workflow with at least two nodes that calls the gateway on a schedule and writes the result into the workspace. Screenshot the canvas and the output file.

Reflection Prompt

Personal: Before this tutorial, what was your mental model of how an AI service like ChatGPT works at an infrastructure level? How has the tier diagram changed that mental model?

Technical: In your notebook: this stack gives you, personally, capabilities that required a funded lab five years ago, running on hardware you own, with no data leaving your desk. Which of the course’s governance concerns get easier under local-first architecture, and which one, honestly, gets harder when there is no provider watching?

Societal: The ability to run capable AI agents locally without any data leaving your machine is currently accessible only to people with the technical knowledge and hardware to do it. What are the equity implications of a world where “private AI” is a skill-gated privilege? Who benefits and who does not?

Where This Goes Next

Now that the stack is running, the Design First: Plan Before You Build activity follows the studio: before wiring more services together, we learn to plan a multi-agent system on paper. The stack knowledge from today feeds directly into the Local Agent Lab Directions 2-3 (the Compose-stack and container-hardening directions).

9. Further Reading