chart/docs/async_refactor.md

Async refactor — findings, priorities, and how to keep it that way

Status: P0, P1 and the loop-lag probe are done (2026-08-11). P2 and P3 outstanding. To be implemented once the in-flight chart work has landed. Everything below is from reading the code on 2026-08-11 and measuring the running app; each finding names the path it was found on.

The goal is not "more async"

The target is nothing blocks the event loop, which is not the same thing as converting everything to async def. Getting this backwards would make the app worse, so state it plainly:

• FastAPI runs a sync def route in a threadpool. Blocking work inside one never touches the loop. That is protection, not a defect.
• Converting those routes to async def removes the protection: any blocking call inside then stalls the market stream and every WebSocket.
• So the rule is per-function. A handler that only awaits should be async def. A handler doing blocking or CPU work should stay def — and must not touch loop-owned objects (see P0).

What is already right

• Every outbound HTTP call is async: httpx.AsyncClient in notify/ntfy.py and market/yahoo.py; the Schwab StreamClient is built with asyncio=True.
• The market stream is an asyncio.Task owned by the app lifespan, and the WebSocket endpoint is a coroutine.
• ManualLineStore guards itself with a threading.RLock, which is the correct primitive precisely because both threadpool routes and the loop reach it. Do not "modernise" it to asyncio.Lock — that would protect only one of them.

---

P0 — Cross-thread access to asyncio.Queue (correctness) — DONE

Sync route handlers reach loop-owned objects from a worker thread:

create_line / create_price_alert / create_comment / patch_line / delete_line   (sync def, threadpool)
  -> runtime.rebuild_levels()
    -> broadcast_level_delta() -> broadcast()
      -> queue.put_nowait(event)          # asyncio.Queue, owned by the loop

asyncio.Queue is not thread-safe. It wakes a waiting consumer by setting a Future's result, and Futures must be resolved on the loop thread — from another thread that requires loop.call_soon_threadsafe. Writing directly can drop the wakeup or corrupt internal state.

Why nobody has noticed: the market stream broadcasts roughly once a second, so a dropped wakeup is papered over by the next event almost immediately. The visible symptom would be a drawing made in one browser not appearing in another until the next tick — easy to misread as network lag.

Fix. Give Runtime the loop it belongs to and post from the correct thread:

self._loop = asyncio.get_running_loop()          # captured in start()

def broadcast(self, event: dict) -> None:
    if threading.current_thread() is threading.main_thread() and self._loop.is_running():
        self._publish(event)                      # already on the loop
    else:
        self._loop.call_soon_threadsafe(self._publish, event)

Prefer this over making the routes async def: that would move level rebuilding (P1) onto the loop, trading a rare correctness bug for a guaranteed latency one.

Verify. A test that calls a mutating route through TestClient while a WebSocket subscriber waits, asserting the event arrives without another tick intervening. Today that passes by luck.

---

P1 — Level rebuilding is CPU-bound on the loop (latency) — DONE

Measured on the dev stack, Yahoo source:

	before	after
port accepting connections	121s	4s
worst loop lag	19,545ms	526ms
steady-state lag	—	0.2–0.6ms

Two changes did it. Seeding now accumulates into the store and derives price, ATR and the level set once at the end (settle_after_seed) instead of rebuilding per replayed bar. And request_rebuild coalesces rebuilds to at most one per 250ms with a guaranteed trailing pass, which matters because Yahoo's first poll emits a whole day of minutes in one burst — that burst, not the seed, was most of the remaining 20 seconds. Bar counts, all five daily MAs, prior-day levels and VWAP are unchanged; 125 python tests and 31 e2e tests pass.

Still open from the original list: incremental moving averages, and diffing levels by fingerprint rather than by re-serialising every point. Neither is needed while lag sits under a second.

Original analysis

rebuild_levels() recomputes all five daily moving averages and re-serialises their points to diff them, on every closed bar. Measured consequence: the seed replay runs the same path per bar and takes ~82 seconds, during which the port is closed. In steady state it is once a minute, which is survivable but is the largest single thing the loop does.

Threads do not help — it is genuine CPU under the GIL. The fix is algorithmic:

1. Incremental moving averages. indicators.sma is already a rolling sum; the waste is recomputing every window from scratch each rebuild rather than advancing the last one.
2. Bulk seeding. Load seeded bars into the store directly and rebuild levels once at the end, rather than replaying each bar through on_bar.
3. Diff without re-serialising. broadcast_level_delta() compares to_dict() output including hundreds of points per MA. Compare a cheap fingerprint (last point plus length) and serialise only what changed.

Do (2) first — it is contained, testable, and removes most of the 82 seconds.

Verify. A test asserting a seed of N bars completes under a threshold, and a loop-lag probe (below) staying under ~50ms while a bar closes.

---

P2 — Blocking disk write on the loop (small, real)

on_bar (coroutine)
  -> rebuild_clusters(evaluate_alerts=True) -> dispatch_alerts() -> disarm()
    -> manual_lines.update() -> save()      # write_text + atomic replace

A ~1KB write, usually sub-millisecond, but it lands on the loop at the exact moment an alert fires, and it is unbounded on a contended disk.

Fix. Either await asyncio.to_thread(self.manual_lines.update, ...) on that path, or mark the line disarmed in memory and flush outside the tick. The same applies to any future persistence work — see the cold-restart notes, which will add far more writing than this.

---

P3 — Seeding blocks startup (architectural)

Runtime.start() awaits both seeds before uvicorn binds, so the port refuses connections for the whole ~82 seconds and any open browser logs a wall of ERR_CONNECTION_REFUSED. Fixing P1(2) may reduce this enough on its own. If it does not, seed in a background task and serve immediately — but note that changes what /api/status's warm flags mean to every consumer, so it needs its own thought rather than being bolted on.

---

Explicitly not doing

• Converting sync routes to async def. They do in-memory work behind a threadpool hop, which is correct and cheap. Changing them adds risk for no gain, and would drag P1's CPU cost onto the loop.
• asyncio.Lock in ManualLineStore. It is reached from both the loop and threadpool threads; only a threading primitive covers both.
• Multiple uvicorn workers as a performance fix. See Procfile — a second process opens a second Schwab stream and duplicates every alert.

---

Keeping it this way

Findings decay unless something enforces them. Three layers, cheapest first.

1. Rules where agents actually read them

AGENTS.md is loaded automatically by both Claude Code (via the CLAUDE.md symlink) and OpenCode every session; nothing else in the repo is guaranteed to be read. Add a short Async rules section stating:

• Nothing blocking or CPU-heavy runs on the event loop.
• Sync def routes stay sync; they run in a threadpool by design.
• A threadpool thread must never touch asyncio objects directly — post through loop.call_soon_threadsafe.
• ManualLineStore's lock is a threading lock deliberately.

Keep it to a handful of lines. A long section is skimmed; a short one is read.

2. Comments at the point of danger

A rule in a document does not stop an edit; a comment on the line does. Already done for the worker count in Procfile. Add the same at:

• Runtime.broadcast — why it posts through the loop.
• ManualLineStore._lock — why it is a threading lock.
• Each mutating route — why it is def and not async def.

3. Make a regression visible

• Loop-lag probe. DONE — Runtime.loop_lag_watch samples 100ms scheduling drift and /api/status reports loop_lag_ms. It found P1 on its first run: 19,441ms worst at startup against 1.5ms in steady state. A stall then shows up as a number instead of as "the chart feels laggy". This is the single highest-value addition here, and it costs about ten lines.
• loop.set_debug(True) in dev, which logs any callback over 100ms with a traceback — it would have named P1 immediately.
• A seed-duration test, so P1 cannot silently regress once fixed.

4. Where the record lives

The risk register in plan.md §15 gets one row per finding, so a reader looking for known hazards finds them. This file holds the detail; the register holds the pointer.

---

Suggested order

1. P0 — correctness, small, self-contained.
2. Loop-lag probe — so P1's improvement is measurable rather than asserted.
3. P1(2) bulk seeding, then P1(1) and P1(3) if the probe still shows stalls.
4. P2 — trivial once P0 has established how work leaves the loop.
5. P3 — only if P1 leaves the startup window unacceptable.
6. Docs and comments alongside each change, not as a final sweep.