Log/surface the reason, pin LLM warm for latency, doc insurance rule
- Reason visibility: the reason WAS extracted ("disintegrated eyes") but only
lived in the Odoo description note. Add it to the post-call log line and to
the Odoo lead title so it's visible at a glance.
- Latency: split the timing — Whisper is ~0.1s, latency is LLM-side. The ~3s
tail was cold model reloads after Ollama's keep-alive expired. server.py now
warms + pins the model on startup (keep_alive=-1, ollama ps UNTIL=Forever),
removing cold first-turn stalls. Whisper size left alone (not the bottleneck).
- CLAUDE.md: insurance rule (never suggest/guess the plan), latency note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
11
CLAUDE.md
11
CLAUDE.md
@@ -277,7 +277,7 @@ Replies are kept to one short sentence.
|
|||||||
| Phone | Confirmed **near the end** (not led with); reads back the caller-ID — injected pre-spelled so it's said digit-by-digit — and if the caller declines, uses the number they give | `callback_number` (+ `phone_confirmed`) |
|
| Phone | Confirmed **near the end** (not led with); reads back the caller-ID — injected pre-spelled so it's said digit-by-digit — and if the caller declines, uses the number they give | `callback_number` (+ `phone_confirmed`) |
|
||||||
| Office / city | Asks city/area; when the caller names a place that matches an office, **confirms that office and moves on** — never offers/compares other offices or asks them to choose; names the nearest only if nothing matches | folded into `reason` prefix |
|
| Office / city | Asks city/area; when the caller names a place that matches an office, **confirms that office and moves on** — never offers/compares other offices or asks them to choose; names the nearest only if nothing matches | folded into `reason` prefix |
|
||||||
| Reason | Captured from the conversation | `reason` |
|
| Reason | Captured from the conversation | `reason` |
|
||||||
| Insurance | **Log only** — asks the plan, never promises/confirms/denies coverage or treatment (even a listed plan); staff verify on callback | `insurance` (note: "log only — staff to verify") |
|
| Insurance | **Log only, never suggest or guess** — asks open-endedly (no plan names read out), captures only what the caller says, never fills in/completes/guesses the plan (asks them to repeat if unclear), never promises/confirms/denies coverage or treatment even for a listed plan; staff verify on callback | `insurance` (note: "log only — staff to verify") |
|
||||||
| Preferred day & time | **Capture & defer** — taken in the caller's own words; AVA does not compute or correct the date | `preferred_time` + best-effort resolved `YYYY-MM-DD` |
|
| Preferred day & time | **Capture & defer** — taken in the caller's own words; AVA does not compute or correct the date | `preferred_time` + best-effort resolved `YYYY-MM-DD` |
|
||||||
|
|
||||||
### Dates — capture & defer (do NOT compute in-call)
|
### Dates — capture & defer (do NOT compute in-call)
|
||||||
@@ -343,6 +343,15 @@ stalls = dead air. The capture changes made it worse by briefly injecting a 45-d
|
|||||||
headroom (RTX 5080 has the VRAM). Rebuild keeps the previous model as `activeblue-avc:pre-ctx8k`
|
headroom (RTX 5080 has the VRAM). Rebuild keeps the previous model as `activeblue-avc:pre-ctx8k`
|
||||||
for rollback. Keep the live system prompt lean for the same reason.
|
for rollback. Keep the live system prompt lean for the same reason.
|
||||||
|
|
||||||
|
### Latency note — model is pinned warm
|
||||||
|
|
||||||
|
Per-turn latency is **LLM-side**, not STT: Whisper runs ~0.1s (VAD-stop → transcript), while
|
||||||
|
transcript → first TTS is ~0.26s median. The tail (P95 ~3s) came from **cold model reloads** —
|
||||||
|
Ollama unloads after its keep-alive window, so the first reply of a call after an idle gap paid
|
||||||
|
a ~3s load. Fix: `server.py` fires a startup warmup that pins the model with `keep_alive=-1`
|
||||||
|
(`ollama ps` shows UNTIL = Forever). Residual ~3s spikes on some later turns are 8B generation
|
||||||
|
variance. Switching Whisper size would NOT help — it's not the bottleneck.
|
||||||
|
|
||||||
### Why Q4_K_M not Q8_0
|
### Why Q4_K_M not Q8_0
|
||||||
|
|
||||||
Q8_0 consumed ~8.5GB VRAM for weights alone. Under telephony load this caused
|
Q8_0 consumed ~8.5GB VRAM for weights alone. Under telephony load this caused
|
||||||
|
|||||||
@@ -123,6 +123,7 @@ async def extract_and_record(messages, ollama_url, model, call_sid=None, caller_
|
|||||||
where = persist_appointment(record)
|
where = persist_appointment(record)
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Post-call appointment saved ({where}): {record['patient_name']} / "
|
f"Post-call appointment saved ({where}): {record['patient_name']} / "
|
||||||
f"{record['location']} / ins={record['insurance']} / when={record['preferred_time']}"
|
f"{record['location']} / reason={record['reason']} / ins={record['insurance']} / "
|
||||||
|
f"when={record['preferred_time']}"
|
||||||
)
|
)
|
||||||
return record
|
return record
|
||||||
|
|||||||
@@ -62,7 +62,7 @@ def create_appointment_request(patient_name, callback_number, reason, preferred_
|
|||||||
insurance=None, call_sid=None):
|
insurance=None, call_sid=None):
|
||||||
"""Create the request in Odoo. Returns (model, record_id) or raises OdooError."""
|
"""Create the request in Odoo. Returns (model, record_id) or raises OdooError."""
|
||||||
uid, models = _connect()
|
uid, models = _connect()
|
||||||
summary = f"📞 Phone appt request — {patient_name or 'caller'}"
|
summary = f"📞 Phone appt — {patient_name or 'caller'}" + (f": {reason}" if reason else "")
|
||||||
# description is an Odoo HTML field — build with <br/> so it renders in the UI.
|
# description is an Odoo HTML field — build with <br/> so it renders in the UI.
|
||||||
rows = [
|
rows = [
|
||||||
("Name", patient_name),
|
("Name", patient_name),
|
||||||
|
|||||||
21
server.py
21
server.py
@@ -64,6 +64,27 @@ BUSY_MESSAGE = os.environ.get(
|
|||||||
|
|
||||||
app = FastAPI()
|
app = FastAPI()
|
||||||
|
|
||||||
|
|
||||||
|
@app.on_event("startup")
|
||||||
|
async def _warm_llm():
|
||||||
|
"""Pin the LLM in VRAM (keep_alive=-1) so the first turn of a call isn't a cold model
|
||||||
|
reload. Cold reloads were adding ~3s of dead air to the first reply; latency is otherwise
|
||||||
|
LLM-side (Whisper STT is ~0.1s). Best-effort — a failure here never blocks startup."""
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
base = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434/v1").rstrip("/")
|
||||||
|
if base.endswith("/v1"):
|
||||||
|
base = base[:-3]
|
||||||
|
model = os.environ.get("OLLAMA_MODEL", "activeblue-avc:latest")
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=120) as c:
|
||||||
|
await c.post(f"{base}/api/generate",
|
||||||
|
json={"model": model, "prompt": "ok", "stream": False, "keep_alive": -1})
|
||||||
|
logger.info(f"Warmed + pinned Ollama model {model} (keep_alive=-1)")
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"LLM warmup failed (first call may be slow): {e!r}")
|
||||||
|
|
||||||
|
|
||||||
# Live count of active /ws pipelines (the real GPU consumers), guarded by a lock.
|
# Live count of active /ws pipelines (the real GPU consumers), guarded by a lock.
|
||||||
_active_calls = 0
|
_active_calls = 0
|
||||||
_active_lock = asyncio.Lock()
|
_active_lock = asyncio.Lock()
|
||||||
|
|||||||
Reference in New Issue
Block a user