Fix 4 failure modes from the 2026-07-05 live call (the 'Nina' call)
- callstate: corrections win (most recent caller statement replaces old
slots — 'no, my name is Carlos' was ignored); patient_name only from the
caller explicitly giving their own name; booking classified only on
explicit scheduling intent (small talk flipped a call to booking and
drove a re-ask loop); new will_call_back call type short-circuits to a
warm close instead of a message-taking flow.
- bot: per-call _today_line() injects the real current date (spoken form,
answer-only) — AVA guessed 'Tuesday' on a Sunday and argued; hours rule
now also bans stating open/closed days ('closed on Sundays' was invented).
- EndCallProcessor: phone-confirm gate reads CallState (booking/callback +
a captured name or reason) instead of the keyword heuristic when tracking
is on, and the injected confirmation now ends the call — the goodbye had
already played, and holding the line open confused the caller.
Verified: staged replay of the failed transcript on activeblue-avc:latest
(4 stages x 3 runs, all pass, incl. clean-booking control); ab_replay.py
--state 0 failures, latency med 0.52s. Deployed via systemd restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
43
CLAUDE.md
43
CLAUDE.md
@@ -40,13 +40,19 @@ Watches LLM text stream for closing keywords ("goodbye"), waits for TTS to finis
|
||||
clipped, then pushes `EndTaskFrame` upstream. `TwilioFrameSerializer` with `auto_hang_up`
|
||||
drops the carrier leg. Verified working in the Phase 1 gate (4/4 clean hang-ups).
|
||||
|
||||
It also **deterministically guarantees the callback number is confirmed** on booking calls:
|
||||
the 8B reads the number back only ~half the time, so if a closing is reached on a booking
|
||||
call (booking keyword seen) without the agent having spoken the number (`phone_marker` not
|
||||
seen in its replies), the hang-up is suppressed and a scripted confirmation line
|
||||
(`phone_confirm_line`, the caller-ID spelled out) is injected as a `TTSSpeakFrame` first.
|
||||
The agent's own readback satisfies the gate, so there's no double-ask in the common case;
|
||||
info-only calls (no booking keyword) are never asked for a number.
|
||||
It also **deterministically guarantees the callback number is confirmed** on real capture
|
||||
calls: the 8B reads the number back only ~half the time, so if a closing is reached without
|
||||
the agent having spoken the number (`phone_marker` not seen in its replies), a scripted
|
||||
confirmation line (`phone_confirm_line`, the caller-ID spelled out) is injected as a
|
||||
`TTSSpeakFrame` — **and the call then hangs up** (the goodbye already played; holding the
|
||||
line open after it confused a live caller on 2026-07-05: "did you say goodbye, or are we
|
||||
continuing?"). The injected wording tells the caller to call back if the number is wrong,
|
||||
since no reply is possible. The agent's own readback satisfies the gate, so there's no
|
||||
double-ask in the common case. Whether the gate applies comes from **CallState** when
|
||||
tracking is on (`state_provider` → call_type booking/callback AND a captured name or
|
||||
reason — a misheard word once flipped a chit-chat call to "booking" and forced a number
|
||||
confirmation nobody wanted); on the anthropic path (no tracking) it falls back to the
|
||||
booking-keyword heuristic. Info-only and will-call-back calls are never asked for a number.
|
||||
|
||||
**Mulaw 8kHz ↔ 16kHz conversion** — handled internally by `TwilioFrameSerializer`.
|
||||
`PIPELINE_SAMPLE_RATE = 16000`, `WIRE_SAMPLE_RATE = 8000` are already set correctly.
|
||||
@@ -118,7 +124,20 @@ before generation) it synchronously merges VAD-fragmented consecutive user messa
|
||||
insurance, preferred day/time`. It also carries call type (`callback` → "do NOT ask booking
|
||||
questions"). Verified via `scripts/ab_replay.py` (replays the real failed calls): llama3.1-8B
|
||||
raw = 3 failures, +CALL STATE = **0 failures**, chat latency 0.31s→0.55s med (system-message
|
||||
churn re-evals the prompt; acceptable, still ≪ the 3s gate). Env: `CALL_STATE_TRACKING`
|
||||
churn re-evals the prompt; acceptable, still ≪ the 3s gate).
|
||||
**Hardened after the 2026-07-05 "Nina" call** (STT misheard "Nah, I'll call you right back"
|
||||
as "Nina, …"; the extractor locked `patient_name: Nina`, never accepted "no, my name is
|
||||
Carlos", classified the call `booking` off small talk, and the checklist drove a booking
|
||||
re-ask loop): the extractor prompt now (1) treats the caller's MOST RECENT statement as
|
||||
authoritative — corrections replace old values; (2) takes `patient_name` ONLY from the
|
||||
caller explicitly giving their own name, never from assistant turns or stray words;
|
||||
(3) classifies `booking` ONLY on explicit scheduling intent (doubt → `unknown`); (4) adds
|
||||
call_type **`will_call_back`** ("I'll call you back" / "I have to go") whose state block
|
||||
short-circuits to a warm close — no name/number/booking questions, no message-taking.
|
||||
Groomer exposes `.state` (read by `EndCallProcessor`'s phone-confirm gate). All four cases
|
||||
verified against the real transcript on `activeblue-avc:latest` (3 runs each, incl. a clean
|
||||
booking control); `ab_replay.py --state` still 0 failures, latency med 0.52s.
|
||||
Env: `CALL_STATE_TRACKING`
|
||||
(default: on for ollama, off for anthropic — Claude tracks state fine on its own; extraction
|
||||
always runs on the local Ollama model). Qwen3-14B was A/B'd as an alternative and rejected
|
||||
for now: no better raw, ~3s/turn with state, needs `think:false` handling, ~11GB VRAM.
|
||||
@@ -390,6 +409,14 @@ the model's weak adherence) also hallucinated availability. Tested directly, the
|
||||
computes appointment dates wrong ~5/5 times, so stating a computed date is a liability. The
|
||||
calendar injection and in-call validation were removed (commit on 2026-06-25).
|
||||
|
||||
**One narrow exception (2026-07-05, `_today_line()` in `bot.py`):** the system prompt now
|
||||
gets ONE line with the real current day/date in spoken form, built fresh per call. A live
|
||||
call had AVA guess "Tuesday" on a Sunday and argue about it — with the true date on hand it
|
||||
can just answer "what day is it?". The line explicitly says it changes nothing else: no
|
||||
computing/stating/correcting appointment dates. This is NOT the 45-day calendar coming back
|
||||
(~30 tokens, answer-only). The hours rule was also tightened: never say which days an office
|
||||
is open or closed (AVA had invented "we're closed on Sundays").
|
||||
|
||||
The post-call extractor still gets today's date and records a **best-effort** `resolved_date`
|
||||
for staff convenience — it is staff-verified on callback, not authoritative, and never spoken to
|
||||
the caller. A deterministic (non-LLM) date resolver is the right hardening if accuracy is needed;
|
||||
|
||||
Reference in New Issue
Block a user