tocmo0nlord/avc-phone-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix 4 failure modes from the 2026-07-05 live call (the 'Nina' call)

Browse Source 
- callstate: corrections win (most recent caller statement replaces old
  slots — 'no, my name is Carlos' was ignored); patient_name only from the
  caller explicitly giving their own name; booking classified only on
  explicit scheduling intent (small talk flipped a call to booking and
  drove a re-ask loop); new will_call_back call type short-circuits to a
  warm close instead of a message-taking flow.
- bot: per-call _today_line() injects the real current date (spoken form,
  answer-only) — AVA guessed 'Tuesday' on a Sunday and argued; hours rule
  now also bans stating open/closed days ('closed on Sundays' was invented).
- EndCallProcessor: phone-confirm gate reads CallState (booking/callback +
  a captured name or reason) instead of the keyword heuristic when tracking
  is on, and the injected confirmation now ends the call — the goodbye had
  already played, and holding the line open confused the caller.

Verified: staged replay of the failed transcript on activeblue-avc:latest
(4 stages x 3 runs, all pass, incl. clean-booking control); ab_replay.py
--state 0 failures, latency med 0.52s. Deployed via systemd restart.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
tocmo0nlord
2026-07-05 05:44:12 +00:00
parent db986ecf76
commit e714262155
3 changed files with 137 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
CLAUDE.md
View File

@@ -40,13 +40,19 @@ Watches LLM text stream for closing keywords ("goodbye"), waits for TTS to finis
clipped, then pushes `EndTaskFrame` upstream. `TwilioFrameSerializer` with `auto_hang_up`
drops the carrier leg. Verified working in the Phase 1 gate (4/4 clean hang-ups).
It also **deterministically guarantees the callback number is confirmed** on booking calls:
the 8B reads the number back only ~half the time, so if a closing is reached on a booking
call (booking keyword seen) without the agent having spoken the number (`phone_marker` not
seen in its replies), the hang-up is suppressed and a scripted confirmation line
(`phone_confirm_line`, the caller-ID spelled out) is injected as a `TTSSpeakFrame` first.
The agent's own readback satisfies the gate, so there's no double-ask in the common case;
info-only calls (no booking keyword) are never asked for a number.
It also **deterministically guarantees the callback number is confirmed** on real capture
calls: the 8B reads the number back only ~half the time, so if a closing is reached without
the agent having spoken the number (`phone_marker` not seen in its replies), a scripted
confirmation line (`phone_confirm_line`, the caller-ID spelled out) is injected as a
`TTSSpeakFrame` — **and the call then hangs up** (the goodbye already played; holding the
line open after it confused a live caller on 2026-07-05: "did you say goodbye, or are we
continuing?"). The injected wording tells the caller to call back if the number is wrong,
since no reply is possible. The agent's own readback satisfies the gate, so there's no
double-ask in the common case. Whether the gate applies comes from **CallState** when
tracking is on (`state_provider` → call_type booking/callback AND a captured name or
reason — a misheard word once flipped a chit-chat call to "booking" and forced a number
confirmation nobody wanted); on the anthropic path (no tracking) it falls back to the
booking-keyword heuristic. Info-only and will-call-back calls are never asked for a number.
**Mulaw 8kHz ↔ 16kHz conversion** — handled internally by `TwilioFrameSerializer`.
`PIPELINE_SAMPLE_RATE = 16000`, `WIRE_SAMPLE_RATE = 8000` are already set correctly.
@@ -118,7 +124,20 @@ before generation) it synchronously merges VAD-fragmented consecutive user messa
insurance, preferred day/time`. It also carries call type (`callback` → "do NOT ask booking
questions"). Verified via `scripts/ab_replay.py` (replays the real failed calls): llama3.1-8B
raw = 3 failures, +CALL STATE = **0 failures**, chat latency 0.31s→0.55s med (system-message
churn re-evals the prompt; acceptable, still ≪ the 3s gate). Env: `CALL_STATE_TRACKING`
churn re-evals the prompt; acceptable, still ≪ the 3s gate).
**Hardened after the 2026-07-05 "Nina" call** (STT misheard "Nah, I'll call you right back"
as "Nina, …"; the extractor locked `patient_name: Nina`, never accepted "no, my name is
Carlos", classified the call `booking` off small talk, and the checklist drove a booking
re-ask loop): the extractor prompt now (1) treats the caller's MOST RECENT statement as
authoritative — corrections replace old values; (2) takes `patient_name` ONLY from the
caller explicitly giving their own name, never from assistant turns or stray words;
(3) classifies `booking` ONLY on explicit scheduling intent (doubt → `unknown`); (4) adds
call_type **`will_call_back`** ("I'll call you back" / "I have to go") whose state block
short-circuits to a warm close — no name/number/booking questions, no message-taking.
Groomer exposes `.state` (read by `EndCallProcessor`'s phone-confirm gate). All four cases
verified against the real transcript on `activeblue-avc:latest` (3 runs each, incl. a clean
booking control); `ab_replay.py --state` still 0 failures, latency med 0.52s.
Env: `CALL_STATE_TRACKING`
(default: on for ollama, off for anthropic — Claude tracks state fine on its own; extraction
always runs on the local Ollama model). Qwen3-14B was A/B'd as an alternative and rejected
for now: no better raw, ~3s/turn with state, needs `think:false` handling, ~11GB VRAM.
@@ -390,6 +409,14 @@ the model's weak adherence) also hallucinated availability. Tested directly, the
computes appointment dates wrong ~5/5 times, so stating a computed date is a liability. The
calendar injection and in-call validation were removed (commit on 2026-06-25).
**One narrow exception (2026-07-05, `_today_line()` in `bot.py`):** the system prompt now
gets ONE line with the real current day/date in spoken form, built fresh per call. A live
call had AVA guess "Tuesday" on a Sunday and argue about it — with the true date on hand it
can just answer "what day is it?". The line explicitly says it changes nothing else: no
computing/stating/correcting appointment dates. This is NOT the 45-day calendar coming back
(~30 tokens, answer-only). The hours rule was also tightened: never say which days an office
is open or closed (AVA had invented "we're closed on Sundays").
The post-call extractor still gets today's date and records a **best-effort** `resolved_date`
for staff convenience — it is staff-verified on callback, not authoritative, and never spoken to
the caller. A deterministic (non-LLM) date resolver is the right hardening if accuracy is needed;

80
bot.py
View File

@@ -11,6 +11,7 @@ the FastAPI/TwiML/WebSocket side and calls run_call() once per call.
"""
import asyncio
import datetime
import os
import re
@@ -262,7 +263,7 @@ SYSTEM_PROMPT = (
"is 'booked', 'scheduled', 'set', or 'confirmed' — not even in the recap. It is always a "
"REQUEST: say you've NOTED it and a staff member will call back to confirm the date and time.\n"
"- Hours are not published — say they vary by office and staff will confirm; never give "
"specific hours.\n"
"specific hours and never say which days an office is open or closed.\n"
"- You don't give medical advice and can't transfer calls. If the caller mentions an eye "
"problem, just note it as the reason and say a staff member or doctor will follow up.\n"
"- If you're not sure you heard something, simply ask them to repeat it.\n"
@@ -274,6 +275,28 @@ SYSTEM_PROMPT = (
"PRACTICE FACTS:\n" + practice_summary()
)
_DAY_ORDINALS = (
"first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth", "ninth",
"tenth", "eleventh", "twelfth", "thirteenth", "fourteenth", "fifteenth", "sixteenth",
"seventeenth", "eighteenth", "nineteenth", "twentieth", "twenty-first", "twenty-second",
"twenty-third", "twenty-fourth", "twenty-fifth", "twenty-sixth", "twenty-seventh",
"twenty-eighth", "twenty-ninth", "thirtieth", "thirty-first",
)
def _today_line() -> str:
"""One spoken-form line with the real current date, built fresh per call (the service
runs for days). Lets the agent answer 'what day is it?' truthfully — a live call had it
guess 'Tuesday' on a Sunday and argue — WITHOUT reintroducing in-call date math:
appointment dates stay capture-and-defer (see the DATES rule)."""
now = datetime.datetime.now()
return (
f"\n\nTODAY: right now it is {now:%A}, {now:%B} {_DAY_ORDINALS[now.day - 1]}. If the "
"caller asks the day or date, answer from this line only. This changes nothing else: "
"still take appointment days in the caller's own words, and never compute, state, or "
"correct an appointment's calendar date."
)
def _build_tools() -> ToolsSchema:
# Only the booking action is a tool. Practice facts already live in the system prompt,
@@ -309,17 +332,24 @@ class EndCallProcessor(FrameProcessor):
(TwilioFrameSerializer auto_hang_up drops the call).
Deterministic phone confirmation: the prompt asks the agent to read the callback number
back, but the 8B skips it ~half the time. So if a closing is reached and the agent never
spoke the number this call (`phone_marker` not seen in its replies), we suppress the
hang-up and inject a scripted confirmation turn first — guaranteeing it happens exactly
once (the agent's own readback satisfies the gate, so no double-ask in the common case)."""
back, but the 8B skips it ~half the time. So if a closing is reached on a call with a real
capture and the agent never spoke the number (`phone_marker` not seen in its replies), the
scripted confirmation line is injected after the goodbye and the call then hangs up —
since the goodbye has already been spoken, holding the line open afterwards just confuses
the caller (live call 2026-07-05: 'did you say goodbye, or are we continuing?'). The
agent's own readback satisfies the gate, so no double-ask in the common case.
Whether a capture is real comes from CallState via `state_provider` when tracking is on
(a misheard word once flipped a call to 'booking' and forced a number confirmation nobody
wanted); without tracking it falls back to the booking-keyword heuristic."""
_CLOSINGS = ("goodbye", "good-bye", "good bye", "adiós", "adios", "hasta luego")
# Only force phone confirmation when a booking was actually underway (not info-only calls).
# Fallback heuristic when no CallState is available (e.g. anthropic path).
_BOOKING_KWS = ("appointment", "schedule", "book", "insurance", "what day", "what time",
"come in", "preferred")
def __init__(self, phone_confirm_line: str | None = None, phone_marker: str | None = None):
def __init__(self, phone_confirm_line: str | None = None, phone_marker: str | None = None,
state_provider=None):
super().__init__()
self._buf = ""
self._should_end = False
@@ -330,12 +360,25 @@ class EndCallProcessor(FrameProcessor):
self._phone_confirmed = not phone_confirm_line
self._assistant_seen = ""
self._pending_phone_inject = False
self._state_provider = state_provider
@classmethod
def _is_closing(cls, text: str) -> bool:
t = (text or "").lower()
return any(c in t for c in cls._CLOSINGS)
def _capture_underway(self) -> bool:
"""Does this call actually need the phone confirmation? With CallState: only a
booking/callback call that captured a name or reason. Without: keyword heuristic."""
state = self._state_provider() if self._state_provider else None
if state:
ctype = (state.get("call_type") or "").strip().lower()
def _s(key):
v = state.get(key)
return v.strip() if isinstance(v, str) else ""
return ctype in ("booking", "callback") and bool(_s("patient_name") or _s("reason"))
return any(k in self._assistant_seen for k in self._BOOKING_KWS)
async def _hang_up_after_delay(self):
await asyncio.sleep(HANGUP_DELAY_SECS)
logger.info(f"{AGENT_NAME} ending task / hanging up")
@@ -353,20 +396,23 @@ class EndCallProcessor(FrameProcessor):
self._phone_confirmed = True # the agent read the number back itself
elif isinstance(frame, LLMFullResponseEndFrame):
if self._is_closing(self._buf):
booking = any(k in self._assistant_seen for k in self._BOOKING_KWS)
if self._phone_confirmed or not booking:
if self._phone_confirmed or not self._capture_underway():
self._should_end = True
logger.info(f"{AGENT_NAME} signalled closing -- will hang up "
f"{HANGUP_DELAY_SECS:.0f}s after she finishes speaking")
else:
# Booking call closing without the number confirmed — do it deterministically.
# Real capture closing without the number confirmed — speak it, then end.
self._pending_phone_inject = True
logger.info(f"{AGENT_NAME} reached closing w/o phone confirmation -- injecting it")
logger.info(f"{AGENT_NAME} reached closing w/o phone confirmation -- "
"injecting it, then hanging up")
self._buf = ""
elif isinstance(frame, BotStoppedSpeakingFrame):
if self._pending_phone_inject:
self._pending_phone_inject = False
self._phone_confirmed = True
# The goodbye already played — after the injected line finishes (next
# BotStoppedSpeakingFrame), hang up instead of leaving the line open.
self._should_end = True
await self.push_frame(TTSSpeakFrame(self._phone_confirm_line), FrameDirection.DOWNSTREAM)
elif self._should_end:
self._should_end = False
@@ -707,7 +753,7 @@ async def run_agent(transport, caller_number=None, call_sid=None, do_capture=Tru
"\n\nCALLER ID: no number is available — near the end, ask the caller for the best "
"phone number to reach them."
)
system_content = SYSTEM_PROMPT + caller_line
system_content = SYSTEM_PROMPT + _today_line() + caller_line
context_kwargs = {"messages": [{"role": "system", "content": system_content}]}
if ENABLE_TOOLS:
context_kwargs["tools"] = _build_tools()
@@ -756,13 +802,19 @@ async def run_agent(transport, caller_number=None, call_sid=None, do_capture=Tru
# having read the caller-ID back, EndCallProcessor speaks this scripted line first.
if caller_number:
_spoken = _spoken_phone(caller_number)
# Spoken AFTER the goodbye and the call then hangs up, so the correction path must
# not need a reply on this call — staff have the number for the callback anyway.
phone_confirm_line = (
f"Also, I have your number as {_spoken}; if that's not the best number, just let me know."
f"One last thing — I have your number as {_spoken} for the callback; if that's "
"not the best number, please give us a quick call back. Thank you, goodbye!"
)
phone_marker = _spoken.split(",")[0].strip() # e.g. "nine seven three"
else:
phone_confirm_line = phone_marker = None
endcall = EndCallProcessor(phone_confirm_line=phone_confirm_line, phone_marker=phone_marker)
endcall = EndCallProcessor(
phone_confirm_line=phone_confirm_line, phone_marker=phone_marker,
state_provider=(lambda: groomer.state) if groomer else None,
)
watchdog = SilenceWatchdog(
silence_secs=SILENCE_REPROMPT_SECS,

41
callstate.py
View File

@@ -33,17 +33,30 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
_STATE_INSTRUCTIONS = (
"You are tracking the state of a LIVE phone call between a caller and the receptionist "
"of an optometry practice. From the transcript, extract only what the CALLER has clearly "
"provided so far. Respond with ONLY a JSON object with these keys:\n"
' "call_type": "booking" (wants to schedule a visit), "callback" (wants something staff '
"must check off-phone: order/frames/lens/prescription status, billing, account lookup, "
'reach a person), "question" (just asking something), or "unknown"\n'
"provided so far. The assistant's lines are context only and MAY CONTAIN MISTAKES — never "
"treat something as true just because the assistant said it. Corrections win: if the "
"caller corrects a value ('no, my name is Carlos', 'actually Tuesday'), output the "
"caller's MOST RECENT statement and discard the old value entirely. "
"Respond with ONLY a JSON object with these keys:\n"
' "call_type": one of:\n'
' "booking" — ONLY if the caller explicitly asked to schedule/book a visit or come '
"in. Greetings, small talk, or asking what you can help with are NOT booking.\n"
' "callback" — wants something staff must check off-phone: order/frames/lens/'
"prescription status, billing, account lookup, reach a person.\n"
' "will_call_back" — the caller said THEY will call back later, have to go, or want '
"to end the call.\n"
' "question" — just asking something.\n'
' "unknown" — none of the above clearly applies yet. When in doubt, use "unknown", '
'not "booking".\n'
' "reason": string or null — WHAT the caller wants. For booking: the visit type or eye '
'problem (e.g. "annual exam", "eye pain"). For callback: what they want checked or done, '
'even if phrased as a question (e.g. "are my glasses ready", "status of an order", '
'"billing question"). Only \'an appointment\' with no visit reason is NOT a reason — '
"use null then.\n"
' "location": string or null — the office/city the caller wants\n'
' "patient_name": string or null — the caller\'s name as given (full or first-only)\n'
' "patient_name": string or null — ONLY a name the caller explicitly gave as their own '
"(\"my name is…\", \"this is…\", or answering a name question); NEVER a name guessed "
"from other words, a garbled transcription, or a name only the assistant used\n"
' "name_is_full": boolean — true only if it clearly has first AND last name\n'
' "insurance": string or null — the plan the caller named, exactly as said\n'
' "preferred_time": string or null — day/time in the caller\'s own words\n'
@@ -101,6 +114,18 @@ def build_state_block(state) -> str:
if not state:
return ""
ctype = (state.get("call_type") or "unknown").strip().lower()
if ctype == "will_call_back":
# Observed failure (2026-07-05 call): "I'll call you right back" got treated as a
# message-taking flow — the agent pushed for a callback number and invented a
# third-person caller. Cut straight to a warm close instead.
return (
"CALL STATE (auto-tracked from this conversation — trust it over your memory):\n"
"- The caller said THEY will call back. Do NOT ask for anything else — no name, "
"no phone number, no booking questions — and do not offer to take a message. "
"Warmly tell them they're welcome to call back anytime, and end your reply with "
"'Goodbye'."
)
got, needed = [], []
for key, label in _BOOKING_ORDER:
val = (state.get(key) or "").strip() if isinstance(state.get(key), str) else ""
@@ -197,6 +222,12 @@ class CallStateGroomer(FrameProcessor):
self._state = None
self._task = None
@property
def state(self):
"""Latest extracted slot state (dict or None). Read by EndCallProcessor's
phone-confirm gate so it only fires on real booking/callback captures."""
return self._state
def _extract_done(self, task):
self._task = None
if task.cancelled():