avc-phone-ai/CLAUDE.md

AVC Phone Agent — Project Specification

Claude Code authoritative reference. All architecture, security, and build decisions live here. Repo: git.activeblue.net/tocmo0nlord/avc-phone-ai Last updated: 2026-06-25 | Active Blue LLC

---

Project Overview

Name: AVC Phone Agent Owner: Active Blue LLC Client: Advanced Vision Care (AVC) — multi-location ophthalmology/optometry practice (FL + TX) Agent name: AVA (Advanced Vision Assistant) Purpose: Automated AI phone agent that answers patient calls, books tentative appointments into Odoo CRM with call recordings and transcripts attached, and self-improves via Claude-powered transcript monitoring and a fine-tuning feedback loop.

---

Existing Codebase — What to Keep, What to Change

The previous build at /home/tocmo0nlord/avc-phone/ is a working foundation. Do not rewrite what works. Apply only the changes documented in this section.

Files and their status

File	Status	Action
bot.py	Keep as-is	Whisper STT retained (real-time). Deepgram evaluated and rejected — see Change 1
server.py	Keep as-is	Twilio Auth Token retained. API Key swap evaluated and rejected — see Change 2
practice.py	Keep as-is	No changes
extract.py	Keep as-is	No changes
odoo_client.py	Keep as-is	Already uses API key auth correctly

What is already solved — do not touch

EndCallProcessor in bot.py — AVC-side call termination is fully implemented. Watches LLM text stream for closing keywords ("goodbye"), waits for TTS to finish via BotStoppedSpeakingFrame, pauses HANGUP_DELAY_SECS (default 4s) so the caller isn't clipped, then pushes EndTaskFrame upstream. TwilioFrameSerializer with auto_hang_up drops the carrier leg. Verified working in the Phase 1 gate (4/4 clean hang-ups).

It also deterministically guarantees the callback number is confirmed on booking calls: the 8B reads the number back only ~half the time, so if a closing is reached on a booking call (booking keyword seen) without the agent having spoken the number (phone_marker not seen in its replies), the hang-up is suppressed and a scripted confirmation line (phone_confirm_line, the caller-ID spelled out) is injected as a TTSSpeakFrame first. The agent's own readback satisfies the gate, so there's no double-ask in the common case; info-only calls (no booking keyword) are never asked for a number.

Mulaw 8kHz ↔ 16kHz conversion — handled internally by TwilioFrameSerializer. PIPELINE_SAMPLE_RATE = 16000, WIRE_SAMPLE_RATE = 8000 are already set correctly. No custom audio module needed.

VAD tuned for telephony — confidence=0.5, min_volume=0.15, start_secs=0.1 — kept sensitive so a quick/quiet "yes" isn't missed (a caller had to repeat it after the phone confirmation). This is safe because HalfDuplexGate gates out the agent's echo while it speaks, so sensitive VAD only listens hard during the caller's own turn and doesn't cause echo false-triggers. Addresses the repeat-yourself / missed-short-answer problem.

Capacity gating — MAX_CONCURRENT_CALLS=2 with atomic slot reservation in server.py prevents GPU thrashing. Keep it.

AudioHeartbeat — diagnostic processor that distinguishes VAD failure from transport stall. Keep it.

Call recording (AudioBufferProcessor) — every call is saved to recordings/<ts>_<callSID>.wav as stereo (caller = left, agent = right) for review/debugging. It sits at the end of the pipeline, so the caller channel is what the system received (post-HalfDuplexGate) — it does NOT capture caller audio that arrived while the agent was speaking (gated). RECORD_CALLS=false to disable. recordings/ is gitignored.

SilenceWatchdog in bot.py — if the caller goes silent after the agent finishes, it re-prompts ("are you still there?") after SILENCE_REPROMPT_SECS (7s), and after MAX_REPROMPTS closes gracefully. Backstop against dead air; silence_secs must stay > HANGUP_DELAY_SECS.

HalfDuplexGate in bot.py — fixes echo-induced mid-call silence. In this pipecat build interruptions are VAD-driven and always on (PipelineParams.allow_interruptions does NOT exist — it's silently ignored). On a phone line the agent's own TTS echoes back, the VAD reads it as the caller speaking (it produces NO transcript), and the broadcast interruption cancels the agent mid-reply → the caller hears silence. This gate sits BEFORE the VAD and withholds inbound audio while the bot is speaking (+ECHO_TAIL_SECS, default 0.5s) so echo never reaches the VAD. Trade-off: half-duplex — the caller can't barge in mid-utterance (fine for short replies). HALF_DUPLEX=false restores barge-in. Keep it on for telephony.

Post-call extraction (extract.py) — single JSON-mode completion after call ends. Correctly uses format: json, uses verified Twilio caller-ID instead of trusting model output, falls back to JSONL if Odoo is unreachable. Keep it.

Odoo integration (odoo_client.py) — already uses ODOO_API_KEY for XML-RPC auth, not password. Correct pattern. No changes.

SpokenKokoroTTSService in bot.py — number normalization for speech. Kokoro reads raw digit strings as cardinals with symbols spoken aloud ("983-4969" → "nine hundred eighty-three dash forty-nine sixty-nine"). This subclass normalizes the text in run_tts (which receives the full sentence) so US phone numbers and 4–5 digit runs (street numbers, zips) are spoken one digit at a time — country code dropped, no "dash"/parens; dates and times left natural ("Monday the fifth", "three thirty"). It also respells the all-caps agent name to AGENT_NAME_SPOKEN (Kokoro reads "AVA" as "A-V-A"; set to "Eva" so it says "EE-vuh"). Deterministic, so it's robust to whatever the model emits. Keep it. tts_normalize() holds the rules.

Note: don't rely on the model to read raw digits — it mangles them (it emitted "197-three five seven three…" once). The caller-ID is injected into the prompt already spelled out so AVA just repeats clean words; tts_normalize is the backstop for any other numbers.

---

Change 1 — Real-time STT stays on Whisper (bot.py)

Decision (2026-06-25): keep Whisper. Deepgram Nova-2 was evaluated and rejected.

Deepgram Nova-2 was trialed to cut STT latency (Whisper buffers ~1-3s before the LLM sees input). The swap was applied and then reverted — the project stays on local faster-whisper. No external STT dependency, no per-minute STT cost, and no audio leaving the box (HIPAA posture). Latency is instead managed via VAD tuning and the medium model on the RTX 5080.

Current bot.py STT (in place — do not change):

from pipecat.services.whisper.stt import WhisperSTTService

WHISPER_MODEL = os.environ.get("WHISPER_MODEL", "medium")   # tiny|base|small|medium
WHISPER_DEVICE = os.environ.get("WHISPER_DEVICE", "cuda")   # cuda for the 5080
WHISPER_COMPUTE = os.environ.get("WHISPER_COMPUTE", "float16")
WHISPER_HOTWORDS = os.environ.get("WHISPER_HOTWORDS", "...")  # domain vocab bias

# HintedWhisperSTTService wraps WhisperSTTService to inject faster-whisper `hotwords`
# (office cities + optometry terms) per call. Instantiated in run_agent():
stt = HintedWhisperSTTService(
    settings=WhisperSTTService.Settings(model=WHISPER_MODEL),
    device=WHISPER_DEVICE,
    compute_type=WHISPER_COMPUTE,
    hotwords=WHISPER_HOTWORDS,
)

Note: Whisper large-v3 also serves post-call transcription in Phase 3 (recording/transcriber.py). If real-time latency proves unacceptable in the Phase 1 gate, revisit a streaming STT then — but do not reintroduce the dependency speculatively.

---

Change 2 — Twilio webhook auth stays on the Auth Token (server.py)

Decision (2026-06-25): keep TWILIO_AUTH_TOKEN. The API Key swap was evaluated and rejected.

A Standard API Key (scoped, revocable) was trialed in place of the account Auth Token, but it cannot do what this server needs: Twilio signs inbound webhooks (X-Twilio-Signature) with the account Auth Token — an API Key Secret cannot validate that signature, so TWILIO_VALIDATE=true would reject every legitimate POST /voice (403). The TwilioFrameSerializer auto-hang-up also expects the account/Auth-Token credential pair. The swap was reverted.

Credential model (in place):

Twilio Account SID          (not secret on its own)
└── Auth Token              (TWILIO_AUTH_TOKEN — validates webhooks + REST/auto-hang-up)

Treat the Auth Token as a password: keep it only in .env (never committed), rotate on any suspected leak / team departure / quarterly. If finer-grained scoping is ever required, the correct design is a hybrid — Auth Token for X-Twilio-Signature validation, an API Key (SK SID + Secret) only for outbound REST — not a wholesale swap.

Current server.py (in place — do not change):

TWILIO_ACCOUNT_SID = os.environ.get("TWILIO_ACCOUNT_SID")
TWILIO_AUTH_TOKEN = os.environ.get("TWILIO_AUTH_TOKEN")

# _twilio_signature_ok(): HMAC-SHA1 keyed by the Auth Token (what Twilio signs with)
digest = hmac.new(TWILIO_AUTH_TOKEN.encode(), payload.encode("utf-8"), hashlib.sha1).digest()

# Validation gate + warning
if TWILIO_VALIDATE and TWILIO_AUTH_TOKEN:
    ...
elif not TWILIO_AUTH_TOKEN:
    logger.warning("/voice signature validation DISABLED (no TWILIO_AUTH_TOKEN set)")

# Serializer auto-hang-up uses the account SID + Auth Token pair
serializer = TwilioFrameSerializer(
    stream_sid=stream_sid,
    call_sid=call_sid,
    account_sid=TWILIO_ACCOUNT_SID,
    auth_token=TWILIO_AUTH_TOKEN,
)

Auth Token rotation procedure:

1. Generate a new primary Auth Token in the Twilio console (use the secondary-token flow)
2. Update TWILIO_AUTH_TOKEN in .env
3. Restart the service — no rebuild needed
4. Verify one test call succeeds (signature validation + auto-hang-up both rely on it)
5. Retire the old token in the Twilio console

Rotate on: any suspected leak, any team member departure, quarterly as routine.

---

Change 3 — .env

No swap. .env keeps TWILIO_AUTH_TOKEN and the Whisper STT vars; there is no TWILIO_API_KEY_* or DEEPGRAM_* (those were trialed and removed with Changes 1/2).

Full .env reference:

# Twilio — Auth Token validates webhooks + drives auto-hang-up. Never committed.
TWILIO_ACCOUNT_SID=AC...
TWILIO_AUTH_TOKEN=
TWILIO_PHONE_NUMBER=+1...
TWILIO_VALIDATE=true

# STT: Whisper (faster-whisper, real-time in-call; large-v3 also used post-call in Phase 3)
WHISPER_MODEL=medium
WHISPER_DEVICE=cuda
WHISPER_COMPUTE=float16

# LLM: Ollama
OLLAMA_URL=http://127.0.0.1:11434/v1
OLLAMA_MODEL=activeblue-avc:latest
LLM_PROVIDER=ollama
LLM_TEMPERATURE=0.3
LLM_MAX_TOKENS=160

# Anthropic (optional LLM swap + monitoring + synthetic data)
ANTHROPIC_API_KEY=
ANTHROPIC_MODEL=claude-sonnet-4-6

# TTS: Kokoro
KOKORO_VOICE=af_heart
KOKORO_MODEL_DIR=/home/tocmo0nlord/pipecat-run/models

# Odoo
ODOO_URL=https://avc.activeblue.net
ODOO_DB=avc
ODOO_USER=
ODOO_API_KEY=
ODOO_TARGET=crm
ODOO_STAGE_ID=
ODOO_TEAM_ID=
ODOO_USER_ID=

# Server
PUBLIC_HOST=avc-phone.activeblue.net
PORT=8200
BIND_HOST=127.0.0.1
MAX_CONCURRENT_CALLS=2
STREAM_TOKEN=

# Call behaviour
AGENT_NAME=AVA
AGENT_NAME_SPOKEN=Eva      # how the name is pronounced in TTS (logs/Odoo keep AGENT_NAME)
HANGUP_DELAY_SECS=4.0      # grace pause after the goodbye before dropping the carrier leg
ENABLE_TOOLS=
VAD_CONFIDENCE=0.5
VAD_MIN_VOLUME=0.3
VAD_START_SECS=0.2
VAD_STOP_SECS=0.5

# Monitoring (Phase 4)
MONITORING_ENABLED=true
MONITORING_SCHEDULE=0 2 * * *

# A/B model routing (Phase 5 only)
AB_SPLIT_PERCENT=0
AB_MODEL_B=

---

Call Workflow

AVA runs a directed script (system prompt in bot.py) — warm but direct, one short turn at a time, leading the call rather than waiting on the caller. Fixed order:

1. Reason first — find out what they're calling about (visit reason, or just a question → answer it).
2. Location — ask city/area, confirm the matching office (don't offer others — see office rule).
3. Caller info — full name (ask last name if only a first is given), then address the caller by name from there on; insurance (log only); preferred day/time in their words.
4. Confirm phone (no "yes" needed) — near the end, STATE the caller-ID back and invite a correction only ("I have your number as ; if that's not the best number, just let me know."), then flow on. No yes/no question, no waiting — depending on catching a "yes" right after a long utterance kept failing (echo/gate timing; verified via call recording — the caller's reply was received but VAD never registered it). Caller speaks only to correct it. Still backed by the deterministic EndCallProcessor safety net (also a "let me know if wrong" statement).
5. Wrap up — recap the booking as a REQUEST by name ("I've noted your request to come in…"), make clear staff will call to confirm, then ask "Is there anything else I can help you with?"

Never claims a booking: AVA must never say an appointment is "booked / scheduled / set / confirmed" — everything is a request staff confirm on callback. Insurance: never say "we accept/take" a plan (or invent one) — just note what the caller said; staff verify.

Keep momentum (prevents mid-call silence): until the booking is complete, every turn ends with the next question. A bare acknowledgment with no question (e.g. just "staff will verify your coverage") deadlocks the call — both sides wait, the pipeline goes idle (no GPU/LLM activity), and the caller hears silence. So acknowledgment + next question go in the same turn.

Closing is gated: the word "Goodbye" ends the call (triggers EndCallProcessor → hang-up), so it is never said in the same turn as confirming details and never before the anything-else question — only after the caller says they need nothing more.

Reliability: the script is prompt-driven on the local 8B (order followed well, not perfectly; it can re-ask a last name). The phone-confirmation step is the exception — it's now guaranteed by the deterministic EndCallProcessor safety net.

Call Data Capture

What AVA collects on a booking call and how it's logged. Driven by the system prompt (bot.py); persisted by the post-call extractor (extract.py → practice.py → Odoo lead). Replies are kept to one short sentence.

The six captured fields

Field	In-call behavior	Logged as
Full name	Asks for last name if only a first is given	patient_name / lead contact_name
Phone	Confirmed near the end (not led with); STATES the caller-ID back (injected pre-spelled, digit-by-digit) and invites a correction only — no "yes" required; uses a different number only if the caller gives one	callback_number (+ phone_confirmed)
Office / city	Asks city/area; when the caller names a place that matches an office, confirms that office and moves on — never offers/compares other offices or asks them to choose; names the nearest only if nothing matches	folded into reason prefix
Reason	Captured from the conversation	reason
Insurance	Log only, never suggest or guess — asks open-endedly (no plan names read out), captures only what the caller says, never fills in/completes/guesses the plan (asks to repeat if unclear), never says "we accept/take" a plan, never promises/confirms/denies coverage or treatment even for a listed plan; staff verify on callback	insurance (note: "log only — staff to verify")
Preferred day & time	Capture & defer — taken in the caller's own words; AVA does not compute or correct the date	preferred_time + best-effort resolved YYYY-MM-DD

Dates — capture & defer (do NOT compute in-call)

AVA takes the day/time in the caller's own words ("next Monday", "the fifth") and tells them staff will confirm the exact date on the callback. It must NOT work out, state, or correct the calendar date, and must never argue about what today's date is.

Why: an earlier build injected a 45-day calendar and had AVA validate/correct dates in-call. A real call derailed badly — AVA argued about the date, parroted the canned example, and (with the model's weak adherence) also hallucinated availability. Tested directly, the local 8B model computes appointment dates wrong ~5/5 times, so stating a computed date is a liability. The calendar injection and in-call validation were removed (commit on 2026-06-25).

The post-call extractor still gets today's date and records a best-effort resolved_date for staff convenience — it is staff-verified on callback, not authoritative, and never spoken to the caller. A deterministic (non-LLM) date resolver is the right hardening if accuracy is needed; tracked as a later item. Also: never claim appointment availability or that a slot is open — everything is a request staff confirm.

---

Model Configuration

Current production model: activeblue-avc:latest

Property	Value	Notes
Base	llama3.1:8b-instruct-q4_K_M	Llama 3.1 8B, Q4_K_M quantization
ID	366a6cc15bb7	Rebuilt clean 2026-06-23
Size	4.9GB	Down from 8.7GB Q8_0
VRAM usage	~4.5GB	Leaves 11.5GB headroom on RTX 5080
Context	8192 tokens	Raised from 4096 (2026-06-25) so long calls don't overflow mid-call — see note below
Temperature	0.3	Low — maximizes JSON schema compliance
Top-p	0.9	Standard
Adapter	None	44-pair LoRA adapter discarded

Modelfile (rebuild reference)

FROM llama3.1:8b-instruct-q4_K_M

PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER num_ctx 8192
PARAMETER temperature 0.3
PARAMETER top_p 0.9

TEMPLATE "{{- range .Messages }}<|start_header_id|>{{ .Role }}<|end_header_id|>
{{ .Content }}<|eot_id|>
{{- end }}<|start_header_id|>assistant<|end_header_id|>
"

Why num_ctx 8192 (was 4096) — fixes mid-call silence

Symptom: on longer calls AVA would go silent / stop replying partway through. Cause: the system prompt + a growing multi-turn transcript exceeded the 4096-token window mid-call, so Ollama truncated and re-evaluated the whole context every turn (cache miss) → multi-second stalls = dead air. The capture changes made it worse by briefly injecting a 45-day calendar (~600 tok/turn) — that injection was removed; raising num_ctx to 8192 gives long calls real headroom (RTX 5080 has the VRAM). Rebuild keeps the previous model as activeblue-avc:pre-ctx8k for rollback. Keep the live system prompt lean for the same reason.

Latency note — model is pinned warm

Per-turn latency is LLM-side, not STT: Whisper runs ~0.1s (VAD-stop → transcript), while transcript → first TTS is ~0.26s median. The tail (P95 ~3s) came from cold model reloads — Ollama unloads after its keep-alive window, so the first reply of a call after an idle gap paid a ~3s load. Fix: server.py has a lifespan handler that warms + pins the model with keep_alive=-1 on startup (ollama ps shows UNTIL = Forever). Residual ~3s spikes on some later turns are 8B generation variance. Switching Whisper size would NOT help — it's not the bottleneck (STT model medium is for accuracy, not latency).

VRAM budget — shared Whisper model (fixes OOM)

GPU is 16GB. Budget: pinned LLM ~6GB (num_ctx 8192) + one shared Whisper medium ~1.5GB + overhead ≈ 8GB, leaving headroom. Critical: Whisper is loaded once per process and reused across calls (_WHISPER_MODEL_CACHE in bot.py). Loading a new WhisperModel per call leaks VRAM — ctranslate2 doesn't release it when the call ends, so models accumulated and the GPU OOM'd after ~6–8 calls (CUDA failed with error out of memory, every call dropping right after answer). Symptom to watch: nvidia-smi shows the python process growing call-over-call. Don't reintroduce per-call model loads.

Why Q4_K_M not Q8_0

Q8_0 consumed ~8.5GB VRAM for weights alone. Under telephony load this caused inference latency spikes. Q4_K_M cuts weight VRAM to ~4.5GB with negligible quality difference at 8B scale.

Why no adapter

44-pair LoRA adapter was adding noise not signal. Minimum viable dataset is 200+ pairs per intent category. Rebuilt correctly in Phase 5 with 500+ pairs in JSON output format.

Ollama inventory (current)

activeblue-avc:latest          366a6cc15bb7    4.9GB    production
llama3.1:8b-instruct-q4_K_M    46e0c10c039e    4.9GB    base
nomic-embed-text:latest        0a109f422b47    274MB    embeddings

Phase 5 training note

Axolotl pulls from HuggingFace in safetensors format, not Ollama GGUF:

# Phase 5 only — do not run now
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct
# ~16GB on disk, separate from Ollama storage

---

Build Phases

Claude Code must not scaffold Phase N+1 until Phase N gate is marked complete.

Phase 1 — Reliable call loop

Goal: Every utterance gets a response. Zero silent failures. AVC hangs up — not the caller.

• Change 1: STT — Deepgram evaluated, reverted; staying on Whisper (base → medium)
• Change 2: Twilio auth — API Key evaluated, reverted; staying on Auth Token
• Change 3: .env — Auth Token + Whisper vars; OLLAMA_MODEL=activeblue-avc:latest
• EndCallProcessor AVC-side termination — confirmed in call logs (closing → hang-up); Twilio shows status completed
• AudioHeartbeat diagnostic logging — active ([audio-in] ticks ~every 5s)
• MAX_CONCURRENT_CALLS capacity gating — logic verified (scripts/score_calls.py aside): atomic reserve grants exactly 2, refuses the 3rd, frees on hangup, 10 simultaneous → 2 granted; /voice returns BUSY_MESSAGE + <Hangup/> at cap. End-to-end 3-live-phone test optional.

Gate — status:

1. ⏳ 10 consecutive calls, zero silent non-responses — zero genuine silent non-responses seen so far; no clean 10-in-a-row run after the latest fixes. RE-TEST.
2. ✅ Zero zombie pipeline instances — single process, slots release to 0/2 each call (ps/pgrep; bare process, not Docker).
3. ✅ AVC-side termination confirmed — logs (closing → hang-up) + Twilio call status completed.
4. ✅ JSON parse-failure rate visible — extractor logs every save/failure; 0% parse failures observed.
5. ⏳ Latency P95 < 3s — measured P95 ~3.18s (median 0.26s); cold-reload spikes removed by pinning the model warm. RE-MEASURE on a fresh batch.

Still needs live testing before Phase 1 is signed off: a clean 10-call consecutive run with normal (non-stress) input. Score it with python scripts/score_calls.py (reads the log; pairs with the stereo WAVs in recordings/). Latency P95 (LLM→TTS) is measuring ~0.4s on recent clean calls; capacity gating logic is verified.

Phase 1 — refinements since the revert

Beyond the three reverted changes, the following hardening is live (see git history):

• STT model — Whisper default raised base → medium for telephony accuracy; latency impact negligible (STT ≈ 0.1s; see latency note).
• LLM warmup/pin — server.py lifespan handler pins the model with keep_alive=-1 on startup so the first call turn isn't a cold reload (ollama ps → UNTIL = Forever).
• Context window — num_ctx 4096 → 8192 (fixes mid-call silence; see note above).
• Call workflow — directed script: reason → location → caller info (address by name) → verify phone (read back) → wrap-up "anything else?" before the gated "Goodbye". See Call Workflow.
• TTS — SpokenKokoroTTSService reads phone/street/zip digit-by-digit; agent name respelled via AGENT_NAME_SPOKEN=Eva; caller-ID injected pre-spelled so it isn't mangled.
• Dates — capture-and-defer (no in-call computation); post-call best-effort resolved_date.
• Insurance — log only; never suggest or guess a plan (don't read plan names from the list, never invent one not stated); capture only what the caller says.
• Reason capture — post-call extractor broadened to capture the eye problem/symptom as the reason (not just visit types); reason now shown in the log line and the Odoo lead title.
• Hang-up — HANGUP_DELAY_SECS=4 grace pause before dropping the carrier leg.
• Office selection — confirm the matching office; never offer/compare others.

Phase 2 — Accuracy (RAG + validation)

• Populate rag/data/*.jsonl with real AVC data (human task — see RAG section)
• ChromaDB RAG retriever wired into pipeline
• Response validator: JSON schema + factual cross-check + PHI leak scan
• Keyword blocklist (uncertainty phrases → handoff)
• Intent classifier routing
• Turn counter: max 3 failed turns before forced handoff + termination

Gate: 20 manual test calls, zero hallucinations on AVC-specific facts

Phase 3 — Booking

• Real-time calendar availability check (odoo/calendar.py)
• Whisper large-v3 post-call transcription (recording/transcriber.py)
• Recording + transcript attached to Odoo lead chatter
• Staff review flow confirmed in Odoo

Gate: Staff receives, reviews, and confirms a lead end-to-end

Phase 4 — Monitoring

• Transcript index (recordings/index.jsonl)
• Claude monitoring job
• Dashboard: toggle, alert queue, one-click apply, playback, quality tagging

Gate: First monitoring run produces actionable suggestions

Phase 5 — Fine-tuning

• Pull HuggingFace base (see model section)
• Synthetic data generation via Claude API in JSON output format
• Real call exporter using staff quality tags
• Axolotl QLoRA on RTX 5080
• Model registry + versioning + A/B routing

Gate: New model outperforms baseline over 50+ calls

---

Repository Structure

avc-phone-ai/
├── CLAUDE.md                          ← this file
├── README.md
├── .env                               ← never committed
├── .env.example
├── .gitignore                         ← includes .env, recordings/, *.gguf
│
├── bot.py                             ← Pipecat pipeline (Phase 1 changes here)
├── server.py                          ← Twilio webhook server (Phase 1 changes here)
├── practice.py                        ← AVC facts + Odoo persistence
├── extract.py                         ← post-call appointment extraction
├── odoo_client.py                     ← Odoo XML-RPC client
│
├── rag/                               ← Phase 2
│   ├── store.py
│   ├── loader.py
│   ├── retriever.py
│   └── data/
│       ├── avc_locations.jsonl
│       ├── avc_providers.jsonl
│       ├── avc_services.jsonl
│       ├── avc_hours.jsonl
│       ├── avc_insurance.jsonl
│       └── avc_faqs.jsonl
│
├── recording/                         ← Phase 3
│   ├── transcriber.py                 ← Whisper large-v3 post-call only
│   └── storage.py
│
├── monitoring/                        ← Phase 4
│   ├── monitor.py
│   ├── analyzer.py
│   ├── diff_engine.py
│   ├── scheduler.py
│   └── dashboard/
│       ├── app.py
│       └── static/
│
├── training/                          ← Phase 5 stub
│   └── README.md
│
├── tests/
│   ├── test_bot.py
│   ├── test_server.py
│   ├── test_odoo_client.py
│   ├── test_extract.py
│   └── fixtures/
│       └── sample_transcripts.jsonl
│
├── scripts/
│   ├── deploy.sh
│   └── smoke_test.sh
│
├── avc-phone.service                  ← existing systemd unit
└── traefik-avc-phone.yml              ← existing Traefik config

---

Infrastructure

Component	Host	Address	Notes
Pipecat pipeline	miaai	10.10.1.221	Python async, systemd
Ollama LLM	miaai	http://127.0.0.1:11434/v1	activeblue-avc:latest
ChromaDB (Phase 2)	miaai	http://10.10.1.221:8001	Docker volume
Twilio webhook	miaai	https://avc-phone.activeblue.net	Traefik + Let's Encrypt
Monitoring dashboard	miaai	https://avc-monitor.activeblue.net	internal only
Odoo CRM	—	https://avc.activeblue.net	XML-RPC, db: avc
Recordings	miaai	/home/tocmo0nlord/avc-phone/recordings/	local only
Gitea	—	https://git.activeblue.net/tocmo0nlord/avc-phone-ai	user: tocmo0nlord

---

RAG Store (Phase 2)

Stack: ChromaDB + nomic-embed-text:latest (already in Ollama) Collection: avc_knowledge Retrieval: Top-3 chunks per query on caller's current turn only

JSONL record format

{
  "id": "hours-kendall-weekday",
  "text": "The Kendall location is open Monday through Friday 8:00 AM to 5:00 PM.",
  "tags": ["hours", "kendall"],
  "last_updated": "2026-06-23"
}

Data files — populated before Phase 2, not before Phase 1

File	Content
avc_locations.jsonl	Address, phone, fax, parking per location
avc_providers.jsonl	Name, title, specialty, locations, languages
avc_services.jsonl	Exam types, procedures
avc_hours.jsonl	Hours per location, holiday closures, after-hours
avc_insurance.jsonl	Accepted plans per location
avc_faqs.jsonl	Approved Q&A pairs

Note: practice.py already contains real AVC location and insurance data scraped from advancedvisioncareflorida.com. Use it as the seed for the JSONL files rather than starting from scratch.

---

Claude Monitoring (Phase 4)

What it analyzes

• Facts stated by AVA contradicting RAG store
• System prompt violations
• Calls that should have been handoffs
• High failed turn counts — model or prompt signal
• RAG gaps (AVA said "I don't have that" — should it be added?)
• Phrasing that caused caller confusion

Output schema

{
  "call_sid": "CA...",
  "severity": "high",
  "issue_type": "factual_error",
  "description": "AVA stated Kendall closes at 6pm. RAG store says 5pm.",
  "suggested_action": "rag_update",
  "suggested_change": {
    "file": "rag/data/avc_hours.jsonl",
    "record_id": "hours-kendall-weekday",
    "field": "text",
    "old": "...open until 6pm...",
    "new": "...open until 5pm..."
  }
}

suggested_action: rag_update | prompt_change | blocklist_add | flag_for_review

Dashboard

FastAPI + HTML/JS at https://avc-monitor.activeblue.net (internal only).

Feature	Description
Enable/disable toggle	Pauses scheduler without redeployment
Alert queue	Suggestions sorted by severity
One-click apply	Applies change, commits via Gitea API to avc-phone-ai
Call playback	Audio + transcript side-by-side
Quality tagging	Staff tags calls from dashboard
Manual trigger	POST /monitor/run

---

Fine-Tuning Pipeline (Phase 5 — stub)

Not scaffolded until Phase 4 complete and monitoring has run minimum two weeks. See training/README.md — populated at Phase 5 start.

• Synthetic data: Claude API generates Q&A in JSON output format — schema not style
• Real calls: staff-tagged "good" + corrected bad calls
• Target: 500+ pairs per intent before first Axolotl run
• QLoRA via Axolotl on RTX 5080, base: HuggingFace meta-llama/Llama-3.1-8B-Instruct
• Versioned Ollama models: activeblue-avc:vN
• A/B routing: promote when new version wins on booking + hallucination rate over 50+ calls

---

HIPAA and Compliance

• AVA identifies as automated at call start — no exceptions
• No PHI in ChromaDB — practice information only
• Recordings on miaai only — no cloud storage
• Odoo API user: minimum permissions, not admin
• All endpoints HTTPS via Traefik
• .env never committed

---

Deploy Script (scripts/deploy.sh)

#!/bin/bash
set -e
cd /home/tocmo0nlord/avc-phone
git pull origin main
pip install -r requirements.txt --quiet
systemctl restart avc-phone
systemctl status avc-phone --no-pager
echo "[deploy] Done."

---

Development Conventions

• Python 3.13 (matches miaai miniconda environment)
• Async throughout — Pipecat is async-native
• loguru for all logging — already in use, keep consistent
• Structured log lines for all diagnostic events
• python-dotenv for local dev, env injection in prod
• Secrets never hardcoded
• Every module has if __name__ == "__main__": for isolated testing

---

Key Dependencies (current)

pipecat-ai==1.3.0           # installed at /opt/miniconda3
faster-whisper              # real-time STT (already installed in pipecat-run venv)
kokoro-tts                  # already installed
ollama                      # already installed
scipy / numpy               # already installed (pipecat deps)
chromadb                    # add for Phase 2
sentence-transformers        # add for Phase 2
anthropic                   # for monitoring + optional LLM swap
openai-whisper              # large-v3 for post-call transcription (Phase 3)
fastapi / uvicorn           # already installed
loguru                      # already installed
httpx                       # already installed

---

Open Items

• Confirm TWILIO_AUTH_TOKEN in .env is current (rotate if leaked/stale)
• Confirm ODOO_STAGE_ID, ODOO_TEAM_ID, ODOO_USER_ID from live avc db
• Confirm AVA voice — af_heart is current default, confirm with AVC before go-live
• Populate rag/data/*.jsonl before Phase 2 (seed from practice.py data)
• Define Odoo confirmed appointment flow: lead → opportunity → calendar event
• Staff training on monitoring dashboard quality tagging

---

Active Blue LLC | git.activeblue.net/tocmo0nlord/avc-phone-ai