diff --git a/CLAUDE.md b/CLAUDE.md index 21a279e..99574da 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,13 +40,19 @@ Watches LLM text stream for closing keywords ("goodbye"), waits for TTS to finis clipped, then pushes `EndTaskFrame` upstream. `TwilioFrameSerializer` with `auto_hang_up` drops the carrier leg. Verified working in the Phase 1 gate (4/4 clean hang-ups). -It also **deterministically guarantees the callback number is confirmed** on booking calls: -the 8B reads the number back only ~half the time, so if a closing is reached on a booking -call (booking keyword seen) without the agent having spoken the number (`phone_marker` not -seen in its replies), the hang-up is suppressed and a scripted confirmation line -(`phone_confirm_line`, the caller-ID spelled out) is injected as a `TTSSpeakFrame` first. -The agent's own readback satisfies the gate, so there's no double-ask in the common case; -info-only calls (no booking keyword) are never asked for a number. +It also **deterministically guarantees the callback number is confirmed** on real capture +calls: the 8B reads the number back only ~half the time, so if a closing is reached without +the agent having spoken the number (`phone_marker` not seen in its replies), a scripted +confirmation line (`phone_confirm_line`, the caller-ID spelled out) is injected as a +`TTSSpeakFrame` — **and the call then hangs up** (the goodbye already played; holding the +line open after it confused a live caller on 2026-07-05: "did you say goodbye, or are we +continuing?"). The injected wording tells the caller to call back if the number is wrong, +since no reply is possible. The agent's own readback satisfies the gate, so there's no +double-ask in the common case. Whether the gate applies comes from **CallState** when +tracking is on (`state_provider` → call_type booking/callback AND a captured name or +reason — a misheard word once flipped a chit-chat call to "booking" and forced a number +confirmation nobody wanted); on the anthropic path (no tracking) it falls back to the +booking-keyword heuristic. Info-only and will-call-back calls are never asked for a number. **Mulaw 8kHz ↔ 16kHz conversion** — handled internally by `TwilioFrameSerializer`. `PIPELINE_SAMPLE_RATE = 16000`, `WIRE_SAMPLE_RATE = 8000` are already set correctly. @@ -118,7 +124,20 @@ before generation) it synchronously merges VAD-fragmented consecutive user messa insurance, preferred day/time`. It also carries call type (`callback` → "do NOT ask booking questions"). Verified via `scripts/ab_replay.py` (replays the real failed calls): llama3.1-8B raw = 3 failures, +CALL STATE = **0 failures**, chat latency 0.31s→0.55s med (system-message -churn re-evals the prompt; acceptable, still ≪ the 3s gate). Env: `CALL_STATE_TRACKING` +churn re-evals the prompt; acceptable, still ≪ the 3s gate). +**Hardened after the 2026-07-05 "Nina" call** (STT misheard "Nah, I'll call you right back" +as "Nina, …"; the extractor locked `patient_name: Nina`, never accepted "no, my name is +Carlos", classified the call `booking` off small talk, and the checklist drove a booking +re-ask loop): the extractor prompt now (1) treats the caller's MOST RECENT statement as +authoritative — corrections replace old values; (2) takes `patient_name` ONLY from the +caller explicitly giving their own name, never from assistant turns or stray words; +(3) classifies `booking` ONLY on explicit scheduling intent (doubt → `unknown`); (4) adds +call_type **`will_call_back`** ("I'll call you back" / "I have to go") whose state block +short-circuits to a warm close — no name/number/booking questions, no message-taking. +Groomer exposes `.state` (read by `EndCallProcessor`'s phone-confirm gate). All four cases +verified against the real transcript on `activeblue-avc:latest` (3 runs each, incl. a clean +booking control); `ab_replay.py --state` still 0 failures, latency med 0.52s. +Env: `CALL_STATE_TRACKING` (default: on for ollama, off for anthropic — Claude tracks state fine on its own; extraction always runs on the local Ollama model). Qwen3-14B was A/B'd as an alternative and rejected for now: no better raw, ~3s/turn with state, needs `think:false` handling, ~11GB VRAM. @@ -390,6 +409,14 @@ the model's weak adherence) also hallucinated availability. Tested directly, the computes appointment dates wrong ~5/5 times, so stating a computed date is a liability. The calendar injection and in-call validation were removed (commit on 2026-06-25). +**One narrow exception (2026-07-05, `_today_line()` in `bot.py`):** the system prompt now +gets ONE line with the real current day/date in spoken form, built fresh per call. A live +call had AVA guess "Tuesday" on a Sunday and argue about it — with the true date on hand it +can just answer "what day is it?". The line explicitly says it changes nothing else: no +computing/stating/correcting appointment dates. This is NOT the 45-day calendar coming back +(~30 tokens, answer-only). The hours rule was also tightened: never say which days an office +is open or closed (AVA had invented "we're closed on Sundays"). + The post-call extractor still gets today's date and records a **best-effort** `resolved_date` for staff convenience — it is staff-verified on callback, not authoritative, and never spoken to the caller. A deterministic (non-LLM) date resolver is the right hardening if accuracy is needed; diff --git a/bot.py b/bot.py index 5243b8f..c436b3c 100644 --- a/bot.py +++ b/bot.py @@ -11,6 +11,7 @@ the FastAPI/TwiML/WebSocket side and calls run_call() once per call. """ import asyncio +import datetime import os import re @@ -262,7 +263,7 @@ SYSTEM_PROMPT = ( "is 'booked', 'scheduled', 'set', or 'confirmed' — not even in the recap. It is always a " "REQUEST: say you've NOTED it and a staff member will call back to confirm the date and time.\n" "- Hours are not published — say they vary by office and staff will confirm; never give " - "specific hours.\n" + "specific hours and never say which days an office is open or closed.\n" "- You don't give medical advice and can't transfer calls. If the caller mentions an eye " "problem, just note it as the reason and say a staff member or doctor will follow up.\n" "- If you're not sure you heard something, simply ask them to repeat it.\n" @@ -274,6 +275,28 @@ SYSTEM_PROMPT = ( "PRACTICE FACTS:\n" + practice_summary() ) +_DAY_ORDINALS = ( + "first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth", "ninth", + "tenth", "eleventh", "twelfth", "thirteenth", "fourteenth", "fifteenth", "sixteenth", + "seventeenth", "eighteenth", "nineteenth", "twentieth", "twenty-first", "twenty-second", + "twenty-third", "twenty-fourth", "twenty-fifth", "twenty-sixth", "twenty-seventh", + "twenty-eighth", "twenty-ninth", "thirtieth", "thirty-first", +) + + +def _today_line() -> str: + """One spoken-form line with the real current date, built fresh per call (the service + runs for days). Lets the agent answer 'what day is it?' truthfully — a live call had it + guess 'Tuesday' on a Sunday and argue — WITHOUT reintroducing in-call date math: + appointment dates stay capture-and-defer (see the DATES rule).""" + now = datetime.datetime.now() + return ( + f"\n\nTODAY: right now it is {now:%A}, {now:%B} {_DAY_ORDINALS[now.day - 1]}. If the " + "caller asks the day or date, answer from this line only. This changes nothing else: " + "still take appointment days in the caller's own words, and never compute, state, or " + "correct an appointment's calendar date." + ) + def _build_tools() -> ToolsSchema: # Only the booking action is a tool. Practice facts already live in the system prompt, @@ -309,17 +332,24 @@ class EndCallProcessor(FrameProcessor): (TwilioFrameSerializer auto_hang_up drops the call). Deterministic phone confirmation: the prompt asks the agent to read the callback number - back, but the 8B skips it ~half the time. So if a closing is reached and the agent never - spoke the number this call (`phone_marker` not seen in its replies), we suppress the - hang-up and inject a scripted confirmation turn first — guaranteeing it happens exactly - once (the agent's own readback satisfies the gate, so no double-ask in the common case).""" + back, but the 8B skips it ~half the time. So if a closing is reached on a call with a real + capture and the agent never spoke the number (`phone_marker` not seen in its replies), the + scripted confirmation line is injected after the goodbye and the call then hangs up — + since the goodbye has already been spoken, holding the line open afterwards just confuses + the caller (live call 2026-07-05: 'did you say goodbye, or are we continuing?'). The + agent's own readback satisfies the gate, so no double-ask in the common case. + + Whether a capture is real comes from CallState via `state_provider` when tracking is on + (a misheard word once flipped a call to 'booking' and forced a number confirmation nobody + wanted); without tracking it falls back to the booking-keyword heuristic.""" _CLOSINGS = ("goodbye", "good-bye", "good bye", "adiós", "adios", "hasta luego") - # Only force phone confirmation when a booking was actually underway (not info-only calls). + # Fallback heuristic when no CallState is available (e.g. anthropic path). _BOOKING_KWS = ("appointment", "schedule", "book", "insurance", "what day", "what time", "come in", "preferred") - def __init__(self, phone_confirm_line: str | None = None, phone_marker: str | None = None): + def __init__(self, phone_confirm_line: str | None = None, phone_marker: str | None = None, + state_provider=None): super().__init__() self._buf = "" self._should_end = False @@ -330,12 +360,25 @@ class EndCallProcessor(FrameProcessor): self._phone_confirmed = not phone_confirm_line self._assistant_seen = "" self._pending_phone_inject = False + self._state_provider = state_provider @classmethod def _is_closing(cls, text: str) -> bool: t = (text or "").lower() return any(c in t for c in cls._CLOSINGS) + def _capture_underway(self) -> bool: + """Does this call actually need the phone confirmation? With CallState: only a + booking/callback call that captured a name or reason. Without: keyword heuristic.""" + state = self._state_provider() if self._state_provider else None + if state: + ctype = (state.get("call_type") or "").strip().lower() + def _s(key): + v = state.get(key) + return v.strip() if isinstance(v, str) else "" + return ctype in ("booking", "callback") and bool(_s("patient_name") or _s("reason")) + return any(k in self._assistant_seen for k in self._BOOKING_KWS) + async def _hang_up_after_delay(self): await asyncio.sleep(HANGUP_DELAY_SECS) logger.info(f"{AGENT_NAME} ending task / hanging up") @@ -353,20 +396,23 @@ class EndCallProcessor(FrameProcessor): self._phone_confirmed = True # the agent read the number back itself elif isinstance(frame, LLMFullResponseEndFrame): if self._is_closing(self._buf): - booking = any(k in self._assistant_seen for k in self._BOOKING_KWS) - if self._phone_confirmed or not booking: + if self._phone_confirmed or not self._capture_underway(): self._should_end = True logger.info(f"{AGENT_NAME} signalled closing -- will hang up " f"{HANGUP_DELAY_SECS:.0f}s after she finishes speaking") else: - # Booking call closing without the number confirmed — do it deterministically. + # Real capture closing without the number confirmed — speak it, then end. self._pending_phone_inject = True - logger.info(f"{AGENT_NAME} reached closing w/o phone confirmation -- injecting it") + logger.info(f"{AGENT_NAME} reached closing w/o phone confirmation -- " + "injecting it, then hanging up") self._buf = "" elif isinstance(frame, BotStoppedSpeakingFrame): if self._pending_phone_inject: self._pending_phone_inject = False self._phone_confirmed = True + # The goodbye already played — after the injected line finishes (next + # BotStoppedSpeakingFrame), hang up instead of leaving the line open. + self._should_end = True await self.push_frame(TTSSpeakFrame(self._phone_confirm_line), FrameDirection.DOWNSTREAM) elif self._should_end: self._should_end = False @@ -707,7 +753,7 @@ async def run_agent(transport, caller_number=None, call_sid=None, do_capture=Tru "\n\nCALLER ID: no number is available — near the end, ask the caller for the best " "phone number to reach them." ) - system_content = SYSTEM_PROMPT + caller_line + system_content = SYSTEM_PROMPT + _today_line() + caller_line context_kwargs = {"messages": [{"role": "system", "content": system_content}]} if ENABLE_TOOLS: context_kwargs["tools"] = _build_tools() @@ -756,13 +802,19 @@ async def run_agent(transport, caller_number=None, call_sid=None, do_capture=Tru # having read the caller-ID back, EndCallProcessor speaks this scripted line first. if caller_number: _spoken = _spoken_phone(caller_number) + # Spoken AFTER the goodbye and the call then hangs up, so the correction path must + # not need a reply on this call — staff have the number for the callback anyway. phone_confirm_line = ( - f"Also, I have your number as {_spoken}; if that's not the best number, just let me know." + f"One last thing — I have your number as {_spoken} for the callback; if that's " + "not the best number, please give us a quick call back. Thank you, goodbye!" ) phone_marker = _spoken.split(",")[0].strip() # e.g. "nine seven three" else: phone_confirm_line = phone_marker = None - endcall = EndCallProcessor(phone_confirm_line=phone_confirm_line, phone_marker=phone_marker) + endcall = EndCallProcessor( + phone_confirm_line=phone_confirm_line, phone_marker=phone_marker, + state_provider=(lambda: groomer.state) if groomer else None, + ) watchdog = SilenceWatchdog( silence_secs=SILENCE_REPROMPT_SECS, diff --git a/callstate.py b/callstate.py index 8d72bc8..47379d0 100644 --- a/callstate.py +++ b/callstate.py @@ -33,17 +33,30 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor _STATE_INSTRUCTIONS = ( "You are tracking the state of a LIVE phone call between a caller and the receptionist " "of an optometry practice. From the transcript, extract only what the CALLER has clearly " - "provided so far. Respond with ONLY a JSON object with these keys:\n" - ' "call_type": "booking" (wants to schedule a visit), "callback" (wants something staff ' - "must check off-phone: order/frames/lens/prescription status, billing, account lookup, " - 'reach a person), "question" (just asking something), or "unknown"\n' + "provided so far. The assistant's lines are context only and MAY CONTAIN MISTAKES — never " + "treat something as true just because the assistant said it. Corrections win: if the " + "caller corrects a value ('no, my name is Carlos', 'actually Tuesday'), output the " + "caller's MOST RECENT statement and discard the old value entirely. " + "Respond with ONLY a JSON object with these keys:\n" + ' "call_type": one of:\n' + ' "booking" — ONLY if the caller explicitly asked to schedule/book a visit or come ' + "in. Greetings, small talk, or asking what you can help with are NOT booking.\n" + ' "callback" — wants something staff must check off-phone: order/frames/lens/' + "prescription status, billing, account lookup, reach a person.\n" + ' "will_call_back" — the caller said THEY will call back later, have to go, or want ' + "to end the call.\n" + ' "question" — just asking something.\n' + ' "unknown" — none of the above clearly applies yet. When in doubt, use "unknown", ' + 'not "booking".\n' ' "reason": string or null — WHAT the caller wants. For booking: the visit type or eye ' 'problem (e.g. "annual exam", "eye pain"). For callback: what they want checked or done, ' 'even if phrased as a question (e.g. "are my glasses ready", "status of an order", ' '"billing question"). Only \'an appointment\' with no visit reason is NOT a reason — ' "use null then.\n" ' "location": string or null — the office/city the caller wants\n' - ' "patient_name": string or null — the caller\'s name as given (full or first-only)\n' + ' "patient_name": string or null — ONLY a name the caller explicitly gave as their own ' + "(\"my name is…\", \"this is…\", or answering a name question); NEVER a name guessed " + "from other words, a garbled transcription, or a name only the assistant used\n" ' "name_is_full": boolean — true only if it clearly has first AND last name\n' ' "insurance": string or null — the plan the caller named, exactly as said\n' ' "preferred_time": string or null — day/time in the caller\'s own words\n' @@ -101,6 +114,18 @@ def build_state_block(state) -> str: if not state: return "" ctype = (state.get("call_type") or "unknown").strip().lower() + + if ctype == "will_call_back": + # Observed failure (2026-07-05 call): "I'll call you right back" got treated as a + # message-taking flow — the agent pushed for a callback number and invented a + # third-person caller. Cut straight to a warm close instead. + return ( + "CALL STATE (auto-tracked from this conversation — trust it over your memory):\n" + "- The caller said THEY will call back. Do NOT ask for anything else — no name, " + "no phone number, no booking questions — and do not offer to take a message. " + "Warmly tell them they're welcome to call back anytime, and end your reply with " + "'Goodbye'." + ) got, needed = [], [] for key, label in _BOOKING_ORDER: val = (state.get(key) or "").strip() if isinstance(state.get(key), str) else "" @@ -197,6 +222,12 @@ class CallStateGroomer(FrameProcessor): self._state = None self._task = None + @property + def state(self): + """Latest extracted slot state (dict or None). Read by EndCallProcessor's + phone-confirm gate so it only fires on real booking/callback captures.""" + return self._state + def _extract_done(self, task): self._task = None if task.cancelled():