diff --git a/AGENTS.md b/AGENTS.md index 1e325ed..fe7356e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,8 +2,12 @@ ## Read current context first -Before planning work, read [`docs/IMPLEMENTATION_PLAN.md`](docs/IMPLEMENTATION_PLAN.md) -for verified decisions and the dated session log, then +Before planning work, read [`docs/plan.md`](docs/plan.md) for the decisions +and the reasoning behind them, and +[`docs/implementation.md`](docs/implementation.md) for the dated record of what +actually went wrong and how it was resolved — that one is the faster read when +debugging, because most entries describe something that looked like one bug and +turned out to be another. Then [`docs/NEXT_STEPS.md`](docs/NEXT_STEPS.md) for current recommendations and known deferred fixes. Mobile interaction work also has its own detailed plan in [`docs/mobile_enhance.md`](docs/mobile_enhance.md). diff --git a/README.md b/README.md index fcb94f6..7455a4e 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,11 @@ FastAPI backend + Vue 3 (from CDN, no build step) served at The app charts Yahoo's `ES=F` feed, builds CME-session-aware timeframes, daily moving averages, prior-day high/low/close and session VWAP, and alerts on confluence zones. -The full spec lives in -[`docs/IMPLEMENTATION_PLAN.md`](docs/IMPLEMENTATION_PLAN.md) — read it before writing -code; it records decisions and verified API facts that are expensive to rediscover. +The full spec lives in [`docs/plan.md`](docs/plan.md) — read it before writing +code; it records decisions and verified API facts that are expensive to +rediscover. What it cost to get there is in +[`docs/implementation.md`](docs/implementation.md): a dated log of problems and +their resolutions, kept as a learning record alongside git. ## Local development diff --git a/docs/NEXT_STEPS.md b/docs/NEXT_STEPS.md index efbaa2d..2c51a10 100644 --- a/docs/NEXT_STEPS.md +++ b/docs/NEXT_STEPS.md @@ -3,7 +3,8 @@ Last reviewed: 2026-08-11 05:58 CDT. This file is the short list of work worth considering next. Verified history, -measurements and completed work remain in `docs/IMPLEMENTATION_PLAN.md`. +measurements and completed work remain in `docs/implementation.md`, and the +decisions behind them in `docs/plan.md`. ## Fix next diff --git a/docs/async_refactor.md b/docs/async_refactor.md index b54d54b..18ff546 100644 --- a/docs/async_refactor.md +++ b/docs/async_refactor.md @@ -201,7 +201,7 @@ done for the worker count in `Procfile`. Add the same at: ### 4. Where the record lives -The risk register in `IMPLEMENTATION_PLAN.md` §15 gets one row per finding, so a +The risk register in `plan.md` §15 gets one row per finding, so a reader looking for known hazards finds them. This file holds the detail; the register holds the pointer. diff --git a/docs/implementation.md b/docs/implementation.md new file mode 100644 index 0000000..4b5eb13 --- /dev/null +++ b/docs/implementation.md @@ -0,0 +1,684 @@ +# Implementation log — what actually happened + +A dated record of problems encountered building this and how each was resolved. +Kept deliberately: git says what changed, this says *why it was hard*, what the +wrong theories were, and what the evidence turned out to be. Most entries +describe something that looked like one bug and was another — which is the part +worth reading before debugging something similar. + +The plan and the reasoning behind design decisions live in `docs/plan.md`. + +Newest entries are at the bottom. + +Dated record of problems hit and how they were resolved. Times are UTC; the +repo's commit timestamps are -0500. + +### 2026-08-10 — rebuild, and a chart that looked frozen + +**10:30 · The rebuild was genuinely required.** `schwab-py` had been added to +`requirements.txt`, but the running image was built at 2026-08-09 22:05, before +that line existed. The bind mount (`.:/app`) hides this: source edits appear +live, so the Schwab commits looked deployed while `pip freeze` in the container +showed no `schwab-py` at all. Anything imported rather than read from disk needs +`docker compose build`. Rebuilt to `schwab-py 1.5.1` and recreated the container. + +**10:30–10:31 · Startup takes ~82 seconds, and the port is closed the whole +time.** `Runtime.start()` replays every seeded bar through `on_bar`, and each +daily-bar update re-runs `rebuild_levels()` → `broadcast_level_delta()`, which +serialises and diffs five MA levels carrying ~730 points each. With a 730d/1h +seed plus an 8d/1m seed that is quadratic work before uvicorn binds. Measured: +10:30:24 "Waiting for application startup" → 10:31:46 "Application startup +complete". An open browser tab polling `/api/status` throughout logs a wall of +`ERR_CONNECTION_REFUSED`; that is the restart window, not a fault. +*Unfixed.* The fix is to bulk-load seeded bars and rebuild levels once at the +end, rather than once per bar. Related: M7 persistence would cut the seed itself. + +**Diagnosing a hang that is actually slowness:** `docker stats` reported ~0.1% +CPU while the process was in fact grinding, so it pointed the wrong way. What +worked was `faulthandler.dump_traceback_later(25, exit=True)`, which named the +exact frame (`indicators.py:sma` under `runtime.py:62`). `py-spy` is unusable +here — it needs `SYS_PTRACE`, which the container does not have. + +**Do not write scratch files into the repo while diagnosing.** A `_probe.py` +dropped in the project root is inside the bind mount, so `--reload` restarted +the lifespan and reset the 82-second clock — twice — which is what made +slow startup look like an infinite hang. Pipe throwaway scripts over stdin +(`docker exec -i … python -`) instead. Only `.py` changes trigger the reloader; +writing screenshots into `artifacts/` is safe. + +**10:35 · A blank chart canvas that was not a bug.** The Playwright container +has no usable locale, so Chromium reports `en-US@posix`; Lightweight Charts +formats its time axis through `Intl`, which throws `Invalid language tag` and +leaves the canvas empty. `docker-compose.yml` already sets +`LANG=en_US.UTF-8` for that service and it is *not* sufficient. Launch with +`chromium.launch({ args: ['--lang=en-US'] })` — with that, the page renders and +reports zero console errors. Worth stating plainly: this failure looks exactly +like a broken app, and it is not. + +**10:41–10:50 · The real bug — the chart sat ~10 hours behind a healthy feed.** +Symptom: header price live at 7785.00 while the last candle closed 7772.75, and +the series appeared to end at 00:20. Everything downstream checked out — +`/api/bars` newest 10:39 from `schwab`; `store.put` keeps bars strictly +ascending; the WebSocket snapshot delivered 1000 ascending bars ending 10:42 and +live `bar` events arrived every minute; the browser received all of it. +Interrogating `window.__chart` gave the answer: + +``` +seriesLen 1000 seriesLast 08-10T10:48 (7786.25) ← data complete +visible 08-07T20:41 → 08-10T00:35 ← viewport wrong +logical from 840 to 1005 +``` + +The series was complete; the *viewport* was 617 bars too far left — exactly +`bars_held.1d`. `setBars()` set a visible **logical** range from the candle +array length, then `syncVisibleLevels()` attached the daily MA series, whose 617 +daily points pre-date the 1m window; prepending them renumbered every logical +index and dragged the view off the live edge. Fixed in `static/chart.js` by +anchoring the viewport to a **time** range. Verified in a real browser: visible +range 08:02 → 10:49, last candle 7786.25 matching the header. See §9. + +**Method note.** Three checks in a row said "healthy" — the REST API, the +WebSocket, and the frontend source all looked correct in isolation, because each +of them *was* correct. Only querying the live page's own chart object separated +"the data is missing" from "the data is off-screen". Screenshots alone were +actively misleading here: the stale time axis was read as a session gap. + +### 2026-08-10 (later) — real-time ticks, and what to do about cold restarts + +**The chart now moves between minute closes.** `CHART_FUTURES` emits a bar only +once its minute is over, so the chart stepped once a minute and sat still in +between — read, reasonably, as a dead feed. `LEVEL_ONE_FUTURES` carries real +trades on the same socket (`delayed: False`, verified on this account back in +M6), and it was never subscribed. It is now, and it builds a forming bar for the +current minute which the authoritative `CHART_FUTURES` bar then supersedes. + +Three constraints shaped it, each of which would have caused a real bug: + +- **Tick bars must never reach the aggregator.** It accumulates with + `current.v += incoming.v`, so re-sending the same forming minute would add its + volume into every higher timeframe on every update. `Runtime.on_bar` returns + early for `not bar.closed`: store the bar, set the price, broadcast, stop. +- **Ticks are throttled** (`SCHWAB_TICK_SECONDS`, default 1.0). /ES trades many + times a second and each emission costs a store write plus a broadcast to every + open socket. Setting it negative drops the Level 1 subscription entirely and + returns the source to closed bars only. +- **A tick for a minute already closed is dropped**, or a late trade would + overwrite a settled exchange bar with a partial one. + +Alerts deliberately stay on closed bars. A level is judged on a settled bar, not +on a price that may not last the minute — and `on_bar` already gated on +`closed`, so this needed no change. Intra-bar alerting is a separate decision. + +Bid-only Level 1 updates are skipped rather than carried forward: a bid is not a +trade and must not extend a candle's high or low. Verified live — 15 forming +bars and 2 closed bars in 100 seconds, and in a browser the candle's high and low +visibly extend within the minute. + +**Cold restarts — the options, and a recommendation.** Every restart costs ~82 +seconds of refused connections, re-seeds from Yahoo, and starts with empty alert +cooldowns, so a deploy can re-alert whatever price is sitting on. + +1. *Make seeding non-quadratic.* Seeding replays every bar through `on_bar`, and + each daily-bar update rebuilds all five MA levels and diffs them. Bulk-load + the seeded bars and rebuild levels once at the end. Contained, testable, and + removes most of the 82 seconds. **Do this first** — it is the cheapest real + win and needs no new storage. +2. *Persist bars (M7, SQLite).* Restarts then seed only the gap. Removes the + Yahoo dependency from the startup path and shrinks the window further. This + is the durable answer, and the plan already scopes it. +3. *Persist alert cooldowns and armed state.* Independent of 1 and 2, and the + part that actually misbehaves rather than merely being slow: without it every + deploy re-alerts. Small table, big behavioural win. +4. *Serve before seeding finishes.* Start uvicorn immediately and seed in a + background task, so the port never refuses. The chart would open cold and + fill in, which is better than an unreachable page — but it changes what + "warm" means to every consumer of `/api/status`, so it wants its own thought. + +Recommended order: 1, then 3, then 2. 4 only if the window still bites after 1. + +**Stale bar events across a timeframe switch.** `Cannot update oldest data` +appeared in the console once ticks were live. Switching timeframe races: the +server answers `subscribe` with a fresh snapshot from one coroutine while +another is still draining bar events for the timeframe just left, so a 1m bar +can land after the 1h snapshot. Applied to the 1h series it is older than every +point in it, and Lightweight Charts throws rather than ignoring it — taking the +app down instead of dropping one bar. The race predates the tick feed; Level 1 +made bar events ~15x more frequent, which is what surfaced it. + +Guarded at both ends. `app.js` honours the `tf` the event already carries and +drops anything for a timeframe that is no longer selected. `chart.js` refuses a +bar older than the series' last point regardless of where it came from — a bar +behind the last one has nothing to contribute. Verified: 36 rapid timeframe +switches under a live tick feed produce zero errors, and calling +`candles.update()` directly with a stale bar still throws while the guarded +`updateBar()` does not. + +**Same-price trades were being dropped.** The candle still paused for 10–20 +seconds at a time after Level 1 went in. Instrumenting the raw stream settled +it: 87 messages in 90 seconds, only 33 carrying `LAST_PRICE`. Most of the rest +are pure bid/ask movement and correctly ignored — but a seventh of them look +like this: + +``` +['ASK_SIZE','ASK_TIME_MILLIS','BID_SIZE','BID_TIME_MILLIS', + 'LAST_SIZE','QUOTE_TIME_MILLIS','TOTAL_VOLUME','TRADE_TIME_MILLIS','key'] +``` + +Trade time, trade size, cumulative volume — and no `LAST_PRICE`, because Level 1 +sends only *changed* fields and the trade printed at the price of the one +before. Requiring `LAST_PRICE` threw those away along with their volume. +`parse_level_one` now treats size-plus-trade-time as a trade and returns a null +price for the caller to carry forward. Measured on the live feed: median gap +3.1s → 2.0s, worst 21.5s → 8.1s, and bar volume climbs within the minute instead +of standing still. + +Worth recording for the next person who reads a gap as a bug: the remaining +pauses are the market, not the pipe. In thin pre-open tape /ES genuinely goes +seconds without a price-changing trade, and then moves several ticks at once — +which is what a "gap up" after a quiet spell actually is. + +**The time axis reads local, the data stays UTC.** Lightweight Charts is +timezone-agnostic: it reads epoch seconds as UTC and labels them as UTC, which +is why the axis disagreed with the wall clock. Fixed with `tickMarkFormatter` +for the axis and `localization.timeFormatter` for the crosshair, both going +through the browser's own zone. + +Deliberately *not* fixed by shifting the bar timestamps, which is the other +common recipe. Every time in this codebase is epoch UTC by convention, and the +chart's own times feed trendline anchors, `indexAt`, hit testing and the values +posted back for manual lines — an offset applied to the data would put all of +them out by the offset, which is exactly the class of bug that once priced a +trendline 147 points away. + +One limit worth knowing: tick *placement* is still computed on UTC days, so the +day-change divider sits at 00:00 UTC rather than local midnight, labelled with +the local date. The labels are right; the divider is in the UTC place. + +### 2026-08-10 (afternoon) — update rate, the left scale, and volume + +**Schwab conflates Level 1 to one update per second.** Chasing "still slow in +market hours" ended at a hard ceiling rather than a bug. In regular hours the +gaps between updates are whole multiples of 1.005s — 2.01, 3.02, 4.03 — which +only happens if the source emits on a one-second cadence and some seconds carry +no trade. `SCHWAB_TICK_SECONDS` was the limiter at 1.0 and is now 0.25, where it +no longer binds. **One update per second is the source's ceiling.** Anything +faster would mean inventing prices between trades, which a chart must not do. + +Two real losses were found on the way and fixed: + +- Higher timeframes only moved once a minute, because tick bars are 1m and the + socket filters by subscriber timeframe. `Runtime.provisional_higher` now + combines the aggregator's committed state with the live minute — without + mutating it, since the aggregator accumulates volume and would double count. +- Trades carrying only a trade stamp and a moved `TOTAL_VOLUME` — no + `LAST_PRICE`, no `LAST_SIZE` — were skipped. 66 → 74 updates per 90s. + +**Daily context moved to the left price scale.** The right had prior-day levels, +session VWAP and five daily MAs competing with the live price and hand-drawn +intraday levels. The trap: a price scale takes its range from the series on it, +so moving levels across draws them against a different range and puts them at +the wrong height. A transparent candlestick mirror on the left scale feeds it +exactly the right scale's input; verified as a zero-pixel delta between the two. +Hand-drawn levels stay right, which is the space being cleared. `priceScaleId` +is fixed at series creation, so it is passed at construction and kept out of the +options reapplied afterwards. + +**Volume is finally drawn.** It travelled the entire pipeline — parsed from both +Schwab services, aggregated, stored, broadcast in every bar — and nothing +rendered it. Now an overlay histogram on its own hidden scale in the bottom +fifth. An overlay rather than a pane, and emphatically not the price scale: +volumes are five figures against four-figure prices, and sharing a scale would +flatten the candles to a line. + +**Sidebar vertical space.** Three cuts, all in §9.4's layer panel. The daily MA +periods sit on one line — `flex-wrap:nowrap` with tighter gaps and 12px boxes, +where 10px gaps and 22px indent had pushed 200 onto a line of its own. +Auto trendlines joins Manual lines as a parenthetical `(auto)` rather than +owning a row, which suits a control that is disabled until M8. The alert log +becomes a `
` like Confluence zones, closed by default with its count in +the summary — collapsed by default is the point, since leaving it open would +save nothing, and the count means activity is still visible while closed. + +**Sidebar vertical space.** The right column was taller than the viewport with +nothing selected. Five changes, no functionality removed: + +- The five daily MA periods fit one line (`flex-wrap:nowrap`, tighter gaps, 12px + boxes); 10px gaps and a 22px indent had pushed 200 onto a row of its own. +- Auto trendlines becomes a parenthetical `(auto)` on the Manual lines row + rather than owning one, which suits a control disabled until M8. +- Alert log becomes a `
` like Confluence zones, closed by default with + its count in the summary so activity still shows while shut. +- Tools becomes a `
` too, open by default, and each tool's panel is + bound to `armedTool` — only the armed tool shows its label, colour, width and + side controls. `armTool` already toggles and permits one armed tool at a time, + so the panels follow it exactly. +- Order is Layers, Tools, then the rest, with Layers collapsed by default. + +Measured with nothing armed: 1110px of content down to 900px, which is inside +the viewport rather than past it. `.sidebar-section:first-of-type` carries the +zeroed top margin so reordering cannot reintroduce a gap at the top. + +### 2026-08-10 (evening) — chart comments, and Drawings + +**Comments are drawings, not levels.** A comment is stored as a `ManualLine` +with `kind="comment"`, so it inherits persistence, the shared drawing-number +sequence, the sidebar list, filtering and deletion without a parallel set of +endpoints. The one rule that must never bend: `ManualLineStore.levels()` filters +comments out. A comment reaching the level list would join a confluence cluster +and push a phone notification about a piece of text. It is also created with +`armed=False`, and `PATCH /lines/{id}` returns `to_dict()` rather than +`to_level()` for one, so no caller is ever handed a level-shaped comment. + +`kind` is derived when absent — zero slope was always a typed level, anything +else a drawn trendline — so drawings saved before comments existed keep working. + +**Pinned or floating.** Pinned comments carry `anchor_t`/`anchor_p` and move with +the chart; floating ones carry `x`/`y` as fractions of the pane, hold their place +through any zoom, and can be dragged. Comments render as DOM rather than canvas: +they hold arbitrary text, collapse to a numbered dot, and a floating one has to +ignore the time scale entirely. A pinned comment scrolled out of view parks on +the edge it left, pointing back the way it went, so it is never simply lost. + +**"Lines & levels" becomes "Drawings"**, filtered by type and by text — the text +match covers the label, the kind and the `#number`, so `comment`, `cpi` and `7` +all narrow the list. Delete acts on what the filter shows, which is what makes +deleting by type or by string a single button. + +One CSS trap worth recording: `.trendline-row span { grid-column:2 }` captured +the comment row's icon span and dragged it into the text column. Scoped to +`span:not(.drawing-icon)`. + +**A comment lost its place when the timeframe changed.** Placed on a 30m bar, +then switched to 15m, it slid to the far left. `timeToCoordinate` answers only +for times that are data points on the current series, so a 30m bucket start +returned `null` on another timeframe — and `null` was being read as "off the +left edge". Anchors are now resolved to the bar that *contains* them, which is +timeframe-independent: an 09:30 note sits on the 09:30 bar at 15m and on the +09:00 bar at 1h. `setBars` also re-renders comments, since a timeframe switch +replaces the grid underneath every pinned one. + +Verified across 30m → 15m → 1h → 30m: the anchor stays 08:30 throughout, +resolving to the 08:30 bar on 15m and the 08:00 bar on 1h, never edge-parked, +and returning to its original x on the way back. Edge-parking still works where +it should — a comment scrolled 400 bars out parks right and comes back on +return to live. + +**The trendline Side control became inert.** Once snapping always lands on a +bar extreme, the side is inferred from *which* extreme — a high is resistance, a +low is support — so the dropdown could no longer affect anything. It now appears +only when "Snap to highs/lows" is off, which is the one case where there is no +extreme to infer from; otherwise the row reads "Side auto". Verified both ways: +snap on shows the note and no dropdown, snap off shows the dropdown. + +`created_at` (epoch seconds) is already stored on every drawing and returned by +`GET /api/drawings`, so filtering by age needs UI only, not a migration. + +**Trendline placement, third pass — and a regression I shipped.** Making a +pending anchor always win (previous entry) fixed the twitch case and broke the +opposite one: a genuine press-drag begun after an abandoned click was hijacked +by that stale anchor, so the line started far from the drag. That reached +production. The rule is now a single threshold — 12px of travel between press +and release makes it a drag, which is wide enough to survive a twitch on a +deliberate click and unambiguous for a real drag. A drag clears any half-placed +anchor rather than silently adopting it. + +**The crosshair was lying about the anchor.** Lightweight Charts defaults to +`CrosshairMode.Magnet`, which snaps the crosshair to the bar's *close*. Hovering +by a bar's low therefore drew the crosshair mid-bar, and a correctly-snapped +anchor looked wrong — measured: aiming 4px above a bar low placed the anchor at +the low (7773) and not the close (7773.25), while the crosshair sat at the +close. Arming a tool now switches the crosshair to `Normal`, and a snap dot +marks the exact point the anchor will use, coloured by the side it implies. + +Four gesture paths are verified in a browser: two clicks with a twitch on the +second, an abandoned click followed by a real drag, a plain press-drag, and +hovering. All start where they should and land on a bar extreme. + +Worth recording for diagnosis: a reported "line ended up high off the bar" +turned out to render exactly on its bar — zero pixels off at 1h, 30m and 15m — +because the anchor had snapped to the *drawn* timeframe extreme (the 09:00 1h +low, 7744.25) while being checked against 1m bars, where it matches neither +extreme. Always compare an anchor against the timeframe it was drawn on. + +**A zero price wrecked every timeframe's scale.** A LEVEL_ONE_FUTURES update +arrived with `LAST_PRICE: 0`. The parser rejected `None` but `0` is not `None`, +so a minute opened at zero — `o=0.0 h=7777.25 l=0.0` — and `provisional_higher` +carried that low into 5m, 15m, 30m, 1h and the daily bar, flattening the price +scale everywhere. Non-positive prices are now treated as absent, so the last +real price carries forward, and the tick still counts as a trade. + +**The exchange's own bars were being dropped.** `store.put` replaced a bar only +when it matched the *tail*. That held while one closed bar arrived per minute, +but ticks open the next minute before CHART_FUTURES delivers the previous one — +so the authoritative bar no longer matched the tail and was discarded, leaving +the tick approximation and its partial volume in place permanently. `put` now +searches back a bounded number of buckets, and refuses to let a provisional bar +overwrite a settled one. + +Both were introduced by the tick feature and both are covered by tests: a zero +price parses as a trade with no price, a late closed bar replaces its bucket and +keeps the exchange's volume, and a tick cannot overwrite a settled bar. + +**Snapping now measures distance on screen, not in time.** The rule was "take +the bar sharing the cursor's time, then its nearer extreme", which ignored how +far that extreme actually was. Pointing anywhere below a candle snapped to that +candle's low however distant, and the extreme genuinely under the cursor was +never considered — so zoomed out to ~360 bars at three pixels each, hitting the +intended bar took several attempts. `snapPoint` now scans six bars either side +and picks the extreme nearest in pixels. Proven by probe: with the cursor on one +bar's low but nudged two pixels so `coordinateToTime` resolves to its neighbour, +the snap takes the extreme under the cursor rather than the neighbour's. + +Worth recording because it was misdiagnosed twice: a report of "the snap dot +appears way above the bar" was, on the numbers, the dot landing correctly on the +bar's low while the cursor sat 151 points below it. The right price scale keeps +a `bottom: 0.1` margin and the volume overlay is drawn in it, so the lower fifth +of the pane is below every candle — an inviting place to point that contains no +price action at all. + +### e2e tests + +`bin/e2e` runs `tests/e2e/*.test.mjs` inside the playwright service against the +dev stack. Node's built-in test runner, no dependencies added to this repo: +Playwright is global in that container and `tests/e2e` is mounted at +`/repo/tests/e2e`. Every case in there is a bug that shipped — the viewport +parked ten hours back, hourly candles drawn as slivers, stale bar events +throwing, comments drifting on a timeframe switch, and three separate ways a +trendline anchor could disagree with its own preview. None of them could have +been caught by pytest, which is the argument for the suite existing. + +Tests clean up after themselves: `withChart` records the drawings that exist +before the body runs and deletes anything new afterwards, because the dev store +is shared with whoever is using the app. Select by title rather than class when +asserting on chart overlays, for the same reason. + +### The snapping rule, stated once + +**x picks the bar, y picks which extreme.** That is the whole rule. It is +written here because changing it reactively three times is what made trendlines +feel broken, not any inherent difficulty: + +1. An 8px proximity gate meant a cursor between the high and the low snapped to + neither, so the anchor kept a raw mid-bar price and the side silently fell + back to the dropdown. +2. Removing the gate fixed that. Then a nearest-in-2D search was tried, to make + a bar easier to hit when zoomed out — and broke sweeping along the bottom, + because whichever nearby bar had the lowest low won on total distance and the + dot skipped off the bar under the cursor. Reverted. +3. What actually made it feel wrong was never the rule: the crosshair was in + Lightweight Charts' default Magnet mode, snapping to the bar's *close*, so + the feedback pointed somewhere the anchor would never go. It is Normal + everywhere now, with the snap dot showing the real target. + +An e2e test sweeps the cursor along the bottom of a zoomed-out 1h chart and +requires every position to land on the low of the bar beneath it — 115 of 115. +That test is the rule, executable. + +**The snap leapt to the live edge — found by diagnostic mode.** Reported from +the user's own browser, which no headless run had reproduced: + +``` +cursor_x 1409.0 chart_w 1280.0 -> cursor_t None -> snapped to the last bar +cursor_x 1161.0 chart_w 1280.0 -> cursor_t None -> snapped to the last bar +cursor_x 1128.0 -> valid time, drift 0 bars +``` + +`coordinateToTime` answers `null` over the right-hand price axis, over the +whitespace past the last bar, and anywhere outside the chart — and `snapPoint` +read that as "the newest bar", so the dot jumped to the live edge from wherever +the cursor was. Two faults behind it: the tool's pointer listener is on `window` +and therefore fires over the sidebar (x=1409 on a 1280-wide chart), and a null +time meant a default rather than no answer. + +Now a pointer outside the plot hides the indicator entirely, and a null time +resolves to the bar nearest in *pixels* rather than the newest one. Covered by +an e2e test that hovers a bar, the axis, the sidebar, and back. + +The lesson is about method rather than geometry: four hypotheses were tested and +killed by measurement here — device pixel ratio, viewport size, resize +desynchronisation, and the chart scrolling under the gesture — while the actual +cause was visible in one line of the client's own numbers. When the browser is +on another machine, instrument it early instead of reproducing locally. + +### Overlays are positioned against the plot, not the element + +**The trendline snap was 66 pixels out, and so was everything else drawn over +the chart.** Lightweight Charts reports coordinates from the plot area's origin. +The chart *element* also contains the price scales, so once the left scale was +enabled for the daily labels, the plot started 66px into the element — and every +overlay positioned with `left:` against the element was displaced by exactly +that much, in both directions at once: + +- the cursor's element-x was read as a plot-x, resolving a bar ~66px to the + right of the pointer; +- the indicator was then drawn at that bar's plot-x interpreted as element-x, + landing ~66px left of where the bar is painted. + +Not near the cursor, not near the bar, and varying with zoom — 66px is a couple +of bars at 30m and a dozen at 1m, which is why it looked random rather than +offset. Every diagnostic number agreed with itself throughout, because +`dot_y`, `expected_y` and `bar_low_y` all derive from the same API and shared +the same wrong origin. Self-consistent instrumentation cannot see a systematic +error in its own frame of reference. + +All overlays now live in one container positioned over the plot canvas, so they +inherit plot coordinates untranslated: the snap dot and label, the comment +layer, the trendline anchor handles, the preview line, the tooltip, the price +tag and the context menu. `eventPoint` subtracts the same offset, so a pointer +position and a chart coordinate finally mean the same thing. The container is +repositioned on resize. + +This had been mis-diagnosed for hours: device pixel ratio, viewport size, resize +desynchronisation, the chart scrolling under the gesture, and the dead band +below the candles were each measured and ruled out. The measurement that found +it was comparing `canvas.width` to `element.clientWidth` — 0.894 — which is the +first thing that ever disagreed with itself. + +--- + +# Handoff: the snap indicator is ~10px left of where it belongs + +**Status: fixed.** Everything below is measured, not inferred. + +## The symptom + +With the Trendline tool armed, the snap dot sits about one bar to the left of +the cursor. The *height* is correct and the *bar it chooses* is correct — only +the horizontal drawing position is wrong. Reported from a real browser and +reproduced headlessly. + +## The measurement + +``` +bar spacing 6.96 px +dot centre - cursor -10.5 px (= 1.5 bars at that zoom) +chosen bar correct (label names the bar under the cursor) +``` + +And the cause, from `ConfluenceChart.syncOverlayLayer()`: + +``` +at load: containerLeft 56 true plot offset 66 <- 10px stale +after a re-sync: containerLeft 66 true plot offset 66 <- correct +``` + +## Why + +Lightweight Charts reports coordinates from the **plot area's** origin. The +chart *element* also contains the price scales, so with the left scale enabled +the plot begins 66px in. All overlays therefore live in a container +(`.chart-overlays`) positioned over the plot, so they can use chart coordinates +untranslated — see `create()` and `syncOverlayLayer()` in `static/chart.js`. + +`syncOverlayLayer()` runs once in a `requestAnimationFrame` during `create()`. +At that moment the left price scale has not finished sizing itself to its label +text, so the measured offset is 56. It settles at 66 once labels render, and +nothing re-measures. The container stays 10px left of the plot for the life of +the page, which drags every overlay with it: the snap dot and label, comments, +trendline anchor handles, the preview line, the tooltip and the price tag. + +Only `x` is affected. `y` never passes through this offset, which is why the +height has always looked right. + +## The fix to write + +Re-measure instead of measuring once. Options, cheapest first: + +1. `ResizeObserver` on the plot canvas — fires when the scale settles and on + every later change. Probably the right answer. +2. Call `syncOverlayLayer()` at the top of `renderComments()` and + `showSnapDot()`. Correct but does DOM reads on every mouse move. +3. Re-sync on `subscribeVisibleLogicalRangeChange` as well as on resize. Cheap, + but misses a scale that widens without the range changing. + +Beware: the left scale's width depends on its **label text**, so it changes when +the price range gains a digit or a longer level label appears. Whatever you +choose must survive that, not just the initial load. + +## How to verify + +```bash +./bin/e2e trendline # 7 cases, all currently pass — they do not catch this +``` + +The suite misses it because its assertions go through the same coordinate API +that carries the error. Add a test that measures in **page pixels**: place the +cursor exactly at a bar's centre and assert the dot's centre is within ~2px +horizontally. The reproduction is: + +```js +const r = el.getBoundingClientRect(); +const bx = chart.timeScale().timeToCoordinate(bar.t); +await page.mouse.move(r.x + c.plotOffsetX() + bx, r.y + c.candles.priceToCoordinate(bar.l) - 6); +// dot centre x should equal the cursor x; today it is ~10px left +``` + +Live numbers from the client are available without a console: open the chart +with `?diag=1`, then `docker compose logs api | grep SNAPDBG`. Note that +`SNAPDBG` will **not** show this bug — `dot_y`, `expected_y` and `bar_low_y` all +derive from the same API and share the same origin, so they agree with each +other while being wrong together. The error is only visible by comparing against +something outside that frame of reference: `canvas.getBoundingClientRect()` +against `element.getBoundingClientRect()`, or painted pixels. + +## Context worth having + +- `window.__chart` is a deliberate debug handle exposing the wrapper. +- The dev stack is at `http://localhost:8010`, and `http://api:8000` from inside + the playwright container. It runs on a remote machine; the user's browser does + not. Headless passes prove little about their screen — see "Where things run" + in AGENTS.md. +- Related history is above under "Overlays are positioned against the plot, not + the element", which fixed the 66px case this 10px residue survived. +- One e2e test, "clicking a comment collapses it", is flaky (roughly one run in + three) and unrelated. Worth fixing before trusting the suite. + +## Resolution + +`ConfluenceChart` now attaches a `ResizeObserver` to the plot canvas itself. +Unlike the outer chart element, that canvas changes width when Lightweight +Charts finishes sizing the left price scale or a longer price label appears. +The observer repositions the shared overlay layer and redraws its anchored DOM +content after each such change. + +The trendline suite now compares the snap dot and cursor in page pixels, then +forces a six-digit left-scale label and repeats the assertion. This catches the +stale coordinate frame that the chart API's self-consistent coordinates could +not. The comment-collapse flake was also fixed: a broad CSS rule had re-enabled +pointer events on the full-size handles SVG, which intermittently covered a +comment. Overlay surfaces now ignore pointers unless the actual control opts in. + +### 2026-08-11 02:55 CDT — test-only coverage, mobile plan and feed research + +Added regression coverage without changing production code. Pytest grew from +96 to 103 cases: Yahoo H1 seed bars must form the correct CME-session daily +OHLCV, manual alerts must remain disarmed through rebuild and restart, daily +anchors must follow Eastern DST in UTC, and two WebSocket clients must keep +their layer preferences isolated from one another and from global alert state. +The browser suite grew from 15 to 19 cases: partial or malformed persisted +preferences must still boot, volume must remain paired with candles on its own +scale across timeframe changes, and Backspace/Delete in a rename input must edit +text rather than delete the drawing. Full results: 103 backend and 19 browser +tests passing. + +One proposed test exposed a current defect and was deliberately not committed as +a failing test: `withinPlot()` compares against the outer chart element, which +includes the right price axis. Hovering that axis can therefore leave the snap +dot visible at the plot edge. The production fix is to compare plot-relative +coordinates against the measured plot canvas width and height, then add the +page-pixel price-axis case. Startup rebuild-count coverage is also deferred until +the bulk-seeding optimization exists; a duration assertion against today's slow +startup would encode the problem rather than protect a fix. + +Local and deploy test commands now live in README. Browser E2E stays local +because it creates and deletes drawings; production verification uses +`bin/wait-deploy`, `/api/health`, `/api/version`, and an authenticated +`/api/status` smoke check. + +The mobile audit is recorded in `docs/mobile_enhance.md`. Its recommended first +steps are one coordinate path for every gesture, touch-sized invisible hit +areas, persistent first-anchor feedback, and a sticky mobile tool rail before +more ambitious gesture changes. + +No durable unauthenticated source of free real-time CME `/ES` data was found. +Schwab remains the verified entitled live source and Yahoo the practical delayed +seed source. Tastytrade/dxLink is the best free-with-broker-account candidate to +test next; IBKR is the strongest low-cost fallback rather than a free one. CME, +TradingView and Barchart free pages are delayed displays, not licensed backend +APIs. For freshness UI, existing bar WebSocket messages can supply browser +receipt time without another call, but exact trade time and source heartbeat are +not retained yet. The best placement is the existing status strip below the +chart, with a compact mobile form such as `Updated 3s ago · Live`. + +### 2026-08-11 03:19 CDT — make recommendations discoverable + +Added `docs/NEXT_STEPS.md` as the concise home for deferred production fixes, +freshness UI semantics, mobile priorities and market-data alternatives. Linked +it, the implementation/session log and `docs/mobile_enhance.md` directly from +`AGENTS.md`, which every coding agent reads before working in this repository. +This keeps current advice visible without turning the historical implementation +plan into an undifferentiated backlog. + +### 2026-08-11 03:39 CDT — one price scale, left labels and editable-line snapping + +The left side was introduced to keep daily moving-average, VWAP and prior-day +labels away from the intraday labels on the right. That decluttering intent was +correct; implementing it as a second Lightweight Charts price scale was not. +Each scale autoscaled independently once its own overlays were attached, so the +same numeric price could occupy a different y-coordinate on each side. Daily +and intraday structure then looked directly comparable while being geometrically +unrelated. + +All price-bearing series and flat levels now share the candle series' right +scale. Long-term labels remain on the left as DOM overlays whose y positions are +computed through `candles.priceToCoordinate()`, so they preserve the intended +separation without creating another coordinate system. Context series suppress +their built-in right-side titles. A browser invariant verifies every overlay's +scale id, every flat level's host, and sub-pixel agreement between an MA price +and the candle coordinate for that same price. + +Editing a selected trendline had a separate defect: handle dragging used +element-relative x, snapped only to the nearest bar time, and retained a raw +cursor price. Initial placement used plot-relative coordinates and high/low +snapping, so moving an anchor could visibly jump off the bar and change the line +to an unusable slope. Placement, selection, handle dragging and context actions +now all use `eventPoint()` against the measured plot canvas. Handle movement +passes through the same `snapPoint()` high/low rule and displays the same snap +feedback as placement. This also closes the known right-price-axis containment +bug; a page-region test now proves the dot disappears over the actual axis and +returns over the plot. + +The 30-minute chart had only about one week of history because it was derived +solely from Yahoo's eight-day 1-minute seed. The 730-day hourly seed cannot +reconstruct half-hour candles. Yahoo's native `30m` interval was verified live +at `range=60d` (2,843 bars on 2026-08-11), so startup now seeds H1/730d, +M30/60d, then M1/8d. The finer minute aggregation replaces the recent overlap; +the native feed supplies older 30-minute bars. Regression tests pin both Yahoo's +native interval request and the startup seed sequence. Final verification: 105 +backend tests and 22 browser tests passing. + +Daily moving averages had also been rendered with `WithSteps` on every base +timeframe. Holding a daily value constant is correct when projecting it over +intraday candles, but on the daily chart it made the SMA itself look like a +staircase. MA line type is now timeframe-aware: stepped on intraday charts and +simple point-to-point lines on `1d`, with both modes pinned by browser coverage. diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/plan.md similarity index 59% rename from docs/IMPLEMENTATION_PLAN.md rename to docs/plan.md index c4dc5c6..f011119 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/plan.md @@ -11,6 +11,12 @@ alert when independently-derived levels from different timeframes converge. **Explicitly out of scope:** order execution. Nothing in this codebase places a trade. See [§14](#14-why-execution-is-out-of-scope) for why, and for the seam left behind. +> **Companion document:** `docs/implementation.md` records what actually +> happened — the problems hit while building this and how each was resolved. +> This file is the plan and the reasoning; that one is the experience. When they +> disagree, the log is what really occurred. + + --- ## 0. Start here @@ -1205,678 +1211,3 @@ enforces for data sources. | Viewport derived from `bars.length` | Chart looks frozen; feed is fine | Anchor the view by time, never by logical index — §9 | | Seed replays every bar through `on_bar` | ~82 s startup; port refuses connections | Known, unfixed — §16, 2026-08-10 | | Headless browser without a real locale | `Intl` throws; blank canvas mimics an app bug | Launch Chromium with `--lang=en-US` — §16 | - -## 16. Session log - -Dated record of problems hit and how they were resolved. Times are UTC; the -repo's commit timestamps are -0500. - -### 2026-08-10 — rebuild, and a chart that looked frozen - -**10:30 · The rebuild was genuinely required.** `schwab-py` had been added to -`requirements.txt`, but the running image was built at 2026-08-09 22:05, before -that line existed. The bind mount (`.:/app`) hides this: source edits appear -live, so the Schwab commits looked deployed while `pip freeze` in the container -showed no `schwab-py` at all. Anything imported rather than read from disk needs -`docker compose build`. Rebuilt to `schwab-py 1.5.1` and recreated the container. - -**10:30–10:31 · Startup takes ~82 seconds, and the port is closed the whole -time.** `Runtime.start()` replays every seeded bar through `on_bar`, and each -daily-bar update re-runs `rebuild_levels()` → `broadcast_level_delta()`, which -serialises and diffs five MA levels carrying ~730 points each. With a 730d/1h -seed plus an 8d/1m seed that is quadratic work before uvicorn binds. Measured: -10:30:24 "Waiting for application startup" → 10:31:46 "Application startup -complete". An open browser tab polling `/api/status` throughout logs a wall of -`ERR_CONNECTION_REFUSED`; that is the restart window, not a fault. -*Unfixed.* The fix is to bulk-load seeded bars and rebuild levels once at the -end, rather than once per bar. Related: M7 persistence would cut the seed itself. - -**Diagnosing a hang that is actually slowness:** `docker stats` reported ~0.1% -CPU while the process was in fact grinding, so it pointed the wrong way. What -worked was `faulthandler.dump_traceback_later(25, exit=True)`, which named the -exact frame (`indicators.py:sma` under `runtime.py:62`). `py-spy` is unusable -here — it needs `SYS_PTRACE`, which the container does not have. - -**Do not write scratch files into the repo while diagnosing.** A `_probe.py` -dropped in the project root is inside the bind mount, so `--reload` restarted -the lifespan and reset the 82-second clock — twice — which is what made -slow startup look like an infinite hang. Pipe throwaway scripts over stdin -(`docker exec -i … python -`) instead. Only `.py` changes trigger the reloader; -writing screenshots into `artifacts/` is safe. - -**10:35 · A blank chart canvas that was not a bug.** The Playwright container -has no usable locale, so Chromium reports `en-US@posix`; Lightweight Charts -formats its time axis through `Intl`, which throws `Invalid language tag` and -leaves the canvas empty. `docker-compose.yml` already sets -`LANG=en_US.UTF-8` for that service and it is *not* sufficient. Launch with -`chromium.launch({ args: ['--lang=en-US'] })` — with that, the page renders and -reports zero console errors. Worth stating plainly: this failure looks exactly -like a broken app, and it is not. - -**10:41–10:50 · The real bug — the chart sat ~10 hours behind a healthy feed.** -Symptom: header price live at 7785.00 while the last candle closed 7772.75, and -the series appeared to end at 00:20. Everything downstream checked out — -`/api/bars` newest 10:39 from `schwab`; `store.put` keeps bars strictly -ascending; the WebSocket snapshot delivered 1000 ascending bars ending 10:42 and -live `bar` events arrived every minute; the browser received all of it. -Interrogating `window.__chart` gave the answer: - -``` -seriesLen 1000 seriesLast 08-10T10:48 (7786.25) ← data complete -visible 08-07T20:41 → 08-10T00:35 ← viewport wrong -logical from 840 to 1005 -``` - -The series was complete; the *viewport* was 617 bars too far left — exactly -`bars_held.1d`. `setBars()` set a visible **logical** range from the candle -array length, then `syncVisibleLevels()` attached the daily MA series, whose 617 -daily points pre-date the 1m window; prepending them renumbered every logical -index and dragged the view off the live edge. Fixed in `static/chart.js` by -anchoring the viewport to a **time** range. Verified in a real browser: visible -range 08:02 → 10:49, last candle 7786.25 matching the header. See §9. - -**Method note.** Three checks in a row said "healthy" — the REST API, the -WebSocket, and the frontend source all looked correct in isolation, because each -of them *was* correct. Only querying the live page's own chart object separated -"the data is missing" from "the data is off-screen". Screenshots alone were -actively misleading here: the stale time axis was read as a session gap. - -### 2026-08-10 (later) — real-time ticks, and what to do about cold restarts - -**The chart now moves between minute closes.** `CHART_FUTURES` emits a bar only -once its minute is over, so the chart stepped once a minute and sat still in -between — read, reasonably, as a dead feed. `LEVEL_ONE_FUTURES` carries real -trades on the same socket (`delayed: False`, verified on this account back in -M6), and it was never subscribed. It is now, and it builds a forming bar for the -current minute which the authoritative `CHART_FUTURES` bar then supersedes. - -Three constraints shaped it, each of which would have caused a real bug: - -- **Tick bars must never reach the aggregator.** It accumulates with - `current.v += incoming.v`, so re-sending the same forming minute would add its - volume into every higher timeframe on every update. `Runtime.on_bar` returns - early for `not bar.closed`: store the bar, set the price, broadcast, stop. -- **Ticks are throttled** (`SCHWAB_TICK_SECONDS`, default 1.0). /ES trades many - times a second and each emission costs a store write plus a broadcast to every - open socket. Setting it negative drops the Level 1 subscription entirely and - returns the source to closed bars only. -- **A tick for a minute already closed is dropped**, or a late trade would - overwrite a settled exchange bar with a partial one. - -Alerts deliberately stay on closed bars. A level is judged on a settled bar, not -on a price that may not last the minute — and `on_bar` already gated on -`closed`, so this needed no change. Intra-bar alerting is a separate decision. - -Bid-only Level 1 updates are skipped rather than carried forward: a bid is not a -trade and must not extend a candle's high or low. Verified live — 15 forming -bars and 2 closed bars in 100 seconds, and in a browser the candle's high and low -visibly extend within the minute. - -**Cold restarts — the options, and a recommendation.** Every restart costs ~82 -seconds of refused connections, re-seeds from Yahoo, and starts with empty alert -cooldowns, so a deploy can re-alert whatever price is sitting on. - -1. *Make seeding non-quadratic.* Seeding replays every bar through `on_bar`, and - each daily-bar update rebuilds all five MA levels and diffs them. Bulk-load - the seeded bars and rebuild levels once at the end. Contained, testable, and - removes most of the 82 seconds. **Do this first** — it is the cheapest real - win and needs no new storage. -2. *Persist bars (M7, SQLite).* Restarts then seed only the gap. Removes the - Yahoo dependency from the startup path and shrinks the window further. This - is the durable answer, and the plan already scopes it. -3. *Persist alert cooldowns and armed state.* Independent of 1 and 2, and the - part that actually misbehaves rather than merely being slow: without it every - deploy re-alerts. Small table, big behavioural win. -4. *Serve before seeding finishes.* Start uvicorn immediately and seed in a - background task, so the port never refuses. The chart would open cold and - fill in, which is better than an unreachable page — but it changes what - "warm" means to every consumer of `/api/status`, so it wants its own thought. - -Recommended order: 1, then 3, then 2. 4 only if the window still bites after 1. - -**Stale bar events across a timeframe switch.** `Cannot update oldest data` -appeared in the console once ticks were live. Switching timeframe races: the -server answers `subscribe` with a fresh snapshot from one coroutine while -another is still draining bar events for the timeframe just left, so a 1m bar -can land after the 1h snapshot. Applied to the 1h series it is older than every -point in it, and Lightweight Charts throws rather than ignoring it — taking the -app down instead of dropping one bar. The race predates the tick feed; Level 1 -made bar events ~15x more frequent, which is what surfaced it. - -Guarded at both ends. `app.js` honours the `tf` the event already carries and -drops anything for a timeframe that is no longer selected. `chart.js` refuses a -bar older than the series' last point regardless of where it came from — a bar -behind the last one has nothing to contribute. Verified: 36 rapid timeframe -switches under a live tick feed produce zero errors, and calling -`candles.update()` directly with a stale bar still throws while the guarded -`updateBar()` does not. - -**Same-price trades were being dropped.** The candle still paused for 10–20 -seconds at a time after Level 1 went in. Instrumenting the raw stream settled -it: 87 messages in 90 seconds, only 33 carrying `LAST_PRICE`. Most of the rest -are pure bid/ask movement and correctly ignored — but a seventh of them look -like this: - -``` -['ASK_SIZE','ASK_TIME_MILLIS','BID_SIZE','BID_TIME_MILLIS', - 'LAST_SIZE','QUOTE_TIME_MILLIS','TOTAL_VOLUME','TRADE_TIME_MILLIS','key'] -``` - -Trade time, trade size, cumulative volume — and no `LAST_PRICE`, because Level 1 -sends only *changed* fields and the trade printed at the price of the one -before. Requiring `LAST_PRICE` threw those away along with their volume. -`parse_level_one` now treats size-plus-trade-time as a trade and returns a null -price for the caller to carry forward. Measured on the live feed: median gap -3.1s → 2.0s, worst 21.5s → 8.1s, and bar volume climbs within the minute instead -of standing still. - -Worth recording for the next person who reads a gap as a bug: the remaining -pauses are the market, not the pipe. In thin pre-open tape /ES genuinely goes -seconds without a price-changing trade, and then moves several ticks at once — -which is what a "gap up" after a quiet spell actually is. - -**The time axis reads local, the data stays UTC.** Lightweight Charts is -timezone-agnostic: it reads epoch seconds as UTC and labels them as UTC, which -is why the axis disagreed with the wall clock. Fixed with `tickMarkFormatter` -for the axis and `localization.timeFormatter` for the crosshair, both going -through the browser's own zone. - -Deliberately *not* fixed by shifting the bar timestamps, which is the other -common recipe. Every time in this codebase is epoch UTC by convention, and the -chart's own times feed trendline anchors, `indexAt`, hit testing and the values -posted back for manual lines — an offset applied to the data would put all of -them out by the offset, which is exactly the class of bug that once priced a -trendline 147 points away. - -One limit worth knowing: tick *placement* is still computed on UTC days, so the -day-change divider sits at 00:00 UTC rather than local midnight, labelled with -the local date. The labels are right; the divider is in the UTC place. - -### 2026-08-10 (afternoon) — update rate, the left scale, and volume - -**Schwab conflates Level 1 to one update per second.** Chasing "still slow in -market hours" ended at a hard ceiling rather than a bug. In regular hours the -gaps between updates are whole multiples of 1.005s — 2.01, 3.02, 4.03 — which -only happens if the source emits on a one-second cadence and some seconds carry -no trade. `SCHWAB_TICK_SECONDS` was the limiter at 1.0 and is now 0.25, where it -no longer binds. **One update per second is the source's ceiling.** Anything -faster would mean inventing prices between trades, which a chart must not do. - -Two real losses were found on the way and fixed: - -- Higher timeframes only moved once a minute, because tick bars are 1m and the - socket filters by subscriber timeframe. `Runtime.provisional_higher` now - combines the aggregator's committed state with the live minute — without - mutating it, since the aggregator accumulates volume and would double count. -- Trades carrying only a trade stamp and a moved `TOTAL_VOLUME` — no - `LAST_PRICE`, no `LAST_SIZE` — were skipped. 66 → 74 updates per 90s. - -**Daily context moved to the left price scale.** The right had prior-day levels, -session VWAP and five daily MAs competing with the live price and hand-drawn -intraday levels. The trap: a price scale takes its range from the series on it, -so moving levels across draws them against a different range and puts them at -the wrong height. A transparent candlestick mirror on the left scale feeds it -exactly the right scale's input; verified as a zero-pixel delta between the two. -Hand-drawn levels stay right, which is the space being cleared. `priceScaleId` -is fixed at series creation, so it is passed at construction and kept out of the -options reapplied afterwards. - -**Volume is finally drawn.** It travelled the entire pipeline — parsed from both -Schwab services, aggregated, stored, broadcast in every bar — and nothing -rendered it. Now an overlay histogram on its own hidden scale in the bottom -fifth. An overlay rather than a pane, and emphatically not the price scale: -volumes are five figures against four-figure prices, and sharing a scale would -flatten the candles to a line. - -**Sidebar vertical space.** Three cuts, all in §9.4's layer panel. The daily MA -periods sit on one line — `flex-wrap:nowrap` with tighter gaps and 12px boxes, -where 10px gaps and 22px indent had pushed 200 onto a line of its own. -Auto trendlines joins Manual lines as a parenthetical `(auto)` rather than -owning a row, which suits a control that is disabled until M8. The alert log -becomes a `
` like Confluence zones, closed by default with its count in -the summary — collapsed by default is the point, since leaving it open would -save nothing, and the count means activity is still visible while closed. - -**Sidebar vertical space.** The right column was taller than the viewport with -nothing selected. Five changes, no functionality removed: - -- The five daily MA periods fit one line (`flex-wrap:nowrap`, tighter gaps, 12px - boxes); 10px gaps and a 22px indent had pushed 200 onto a row of its own. -- Auto trendlines becomes a parenthetical `(auto)` on the Manual lines row - rather than owning one, which suits a control disabled until M8. -- Alert log becomes a `
` like Confluence zones, closed by default with - its count in the summary so activity still shows while shut. -- Tools becomes a `
` too, open by default, and each tool's panel is - bound to `armedTool` — only the armed tool shows its label, colour, width and - side controls. `armTool` already toggles and permits one armed tool at a time, - so the panels follow it exactly. -- Order is Layers, Tools, then the rest, with Layers collapsed by default. - -Measured with nothing armed: 1110px of content down to 900px, which is inside -the viewport rather than past it. `.sidebar-section:first-of-type` carries the -zeroed top margin so reordering cannot reintroduce a gap at the top. - -### 2026-08-10 (evening) — chart comments, and Drawings - -**Comments are drawings, not levels.** A comment is stored as a `ManualLine` -with `kind="comment"`, so it inherits persistence, the shared drawing-number -sequence, the sidebar list, filtering and deletion without a parallel set of -endpoints. The one rule that must never bend: `ManualLineStore.levels()` filters -comments out. A comment reaching the level list would join a confluence cluster -and push a phone notification about a piece of text. It is also created with -`armed=False`, and `PATCH /lines/{id}` returns `to_dict()` rather than -`to_level()` for one, so no caller is ever handed a level-shaped comment. - -`kind` is derived when absent — zero slope was always a typed level, anything -else a drawn trendline — so drawings saved before comments existed keep working. - -**Pinned or floating.** Pinned comments carry `anchor_t`/`anchor_p` and move with -the chart; floating ones carry `x`/`y` as fractions of the pane, hold their place -through any zoom, and can be dragged. Comments render as DOM rather than canvas: -they hold arbitrary text, collapse to a numbered dot, and a floating one has to -ignore the time scale entirely. A pinned comment scrolled out of view parks on -the edge it left, pointing back the way it went, so it is never simply lost. - -**"Lines & levels" becomes "Drawings"**, filtered by type and by text — the text -match covers the label, the kind and the `#number`, so `comment`, `cpi` and `7` -all narrow the list. Delete acts on what the filter shows, which is what makes -deleting by type or by string a single button. - -One CSS trap worth recording: `.trendline-row span { grid-column:2 }` captured -the comment row's icon span and dragged it into the text column. Scoped to -`span:not(.drawing-icon)`. - -**A comment lost its place when the timeframe changed.** Placed on a 30m bar, -then switched to 15m, it slid to the far left. `timeToCoordinate` answers only -for times that are data points on the current series, so a 30m bucket start -returned `null` on another timeframe — and `null` was being read as "off the -left edge". Anchors are now resolved to the bar that *contains* them, which is -timeframe-independent: an 09:30 note sits on the 09:30 bar at 15m and on the -09:00 bar at 1h. `setBars` also re-renders comments, since a timeframe switch -replaces the grid underneath every pinned one. - -Verified across 30m → 15m → 1h → 30m: the anchor stays 08:30 throughout, -resolving to the 08:30 bar on 15m and the 08:00 bar on 1h, never edge-parked, -and returning to its original x on the way back. Edge-parking still works where -it should — a comment scrolled 400 bars out parks right and comes back on -return to live. - -**The trendline Side control became inert.** Once snapping always lands on a -bar extreme, the side is inferred from *which* extreme — a high is resistance, a -low is support — so the dropdown could no longer affect anything. It now appears -only when "Snap to highs/lows" is off, which is the one case where there is no -extreme to infer from; otherwise the row reads "Side auto". Verified both ways: -snap on shows the note and no dropdown, snap off shows the dropdown. - -`created_at` (epoch seconds) is already stored on every drawing and returned by -`GET /api/drawings`, so filtering by age needs UI only, not a migration. - -**Trendline placement, third pass — and a regression I shipped.** Making a -pending anchor always win (previous entry) fixed the twitch case and broke the -opposite one: a genuine press-drag begun after an abandoned click was hijacked -by that stale anchor, so the line started far from the drag. That reached -production. The rule is now a single threshold — 12px of travel between press -and release makes it a drag, which is wide enough to survive a twitch on a -deliberate click and unambiguous for a real drag. A drag clears any half-placed -anchor rather than silently adopting it. - -**The crosshair was lying about the anchor.** Lightweight Charts defaults to -`CrosshairMode.Magnet`, which snaps the crosshair to the bar's *close*. Hovering -by a bar's low therefore drew the crosshair mid-bar, and a correctly-snapped -anchor looked wrong — measured: aiming 4px above a bar low placed the anchor at -the low (7773) and not the close (7773.25), while the crosshair sat at the -close. Arming a tool now switches the crosshair to `Normal`, and a snap dot -marks the exact point the anchor will use, coloured by the side it implies. - -Four gesture paths are verified in a browser: two clicks with a twitch on the -second, an abandoned click followed by a real drag, a plain press-drag, and -hovering. All start where they should and land on a bar extreme. - -Worth recording for diagnosis: a reported "line ended up high off the bar" -turned out to render exactly on its bar — zero pixels off at 1h, 30m and 15m — -because the anchor had snapped to the *drawn* timeframe extreme (the 09:00 1h -low, 7744.25) while being checked against 1m bars, where it matches neither -extreme. Always compare an anchor against the timeframe it was drawn on. - -**A zero price wrecked every timeframe's scale.** A LEVEL_ONE_FUTURES update -arrived with `LAST_PRICE: 0`. The parser rejected `None` but `0` is not `None`, -so a minute opened at zero — `o=0.0 h=7777.25 l=0.0` — and `provisional_higher` -carried that low into 5m, 15m, 30m, 1h and the daily bar, flattening the price -scale everywhere. Non-positive prices are now treated as absent, so the last -real price carries forward, and the tick still counts as a trade. - -**The exchange's own bars were being dropped.** `store.put` replaced a bar only -when it matched the *tail*. That held while one closed bar arrived per minute, -but ticks open the next minute before CHART_FUTURES delivers the previous one — -so the authoritative bar no longer matched the tail and was discarded, leaving -the tick approximation and its partial volume in place permanently. `put` now -searches back a bounded number of buckets, and refuses to let a provisional bar -overwrite a settled one. - -Both were introduced by the tick feature and both are covered by tests: a zero -price parses as a trade with no price, a late closed bar replaces its bucket and -keeps the exchange's volume, and a tick cannot overwrite a settled bar. - -**Snapping now measures distance on screen, not in time.** The rule was "take -the bar sharing the cursor's time, then its nearer extreme", which ignored how -far that extreme actually was. Pointing anywhere below a candle snapped to that -candle's low however distant, and the extreme genuinely under the cursor was -never considered — so zoomed out to ~360 bars at three pixels each, hitting the -intended bar took several attempts. `snapPoint` now scans six bars either side -and picks the extreme nearest in pixels. Proven by probe: with the cursor on one -bar's low but nudged two pixels so `coordinateToTime` resolves to its neighbour, -the snap takes the extreme under the cursor rather than the neighbour's. - -Worth recording because it was misdiagnosed twice: a report of "the snap dot -appears way above the bar" was, on the numbers, the dot landing correctly on the -bar's low while the cursor sat 151 points below it. The right price scale keeps -a `bottom: 0.1` margin and the volume overlay is drawn in it, so the lower fifth -of the pane is below every candle — an inviting place to point that contains no -price action at all. - -### e2e tests - -`bin/e2e` runs `tests/e2e/*.test.mjs` inside the playwright service against the -dev stack. Node's built-in test runner, no dependencies added to this repo: -Playwright is global in that container and `tests/e2e` is mounted at -`/repo/tests/e2e`. Every case in there is a bug that shipped — the viewport -parked ten hours back, hourly candles drawn as slivers, stale bar events -throwing, comments drifting on a timeframe switch, and three separate ways a -trendline anchor could disagree with its own preview. None of them could have -been caught by pytest, which is the argument for the suite existing. - -Tests clean up after themselves: `withChart` records the drawings that exist -before the body runs and deletes anything new afterwards, because the dev store -is shared with whoever is using the app. Select by title rather than class when -asserting on chart overlays, for the same reason. - -### The snapping rule, stated once - -**x picks the bar, y picks which extreme.** That is the whole rule. It is -written here because changing it reactively three times is what made trendlines -feel broken, not any inherent difficulty: - -1. An 8px proximity gate meant a cursor between the high and the low snapped to - neither, so the anchor kept a raw mid-bar price and the side silently fell - back to the dropdown. -2. Removing the gate fixed that. Then a nearest-in-2D search was tried, to make - a bar easier to hit when zoomed out — and broke sweeping along the bottom, - because whichever nearby bar had the lowest low won on total distance and the - dot skipped off the bar under the cursor. Reverted. -3. What actually made it feel wrong was never the rule: the crosshair was in - Lightweight Charts' default Magnet mode, snapping to the bar's *close*, so - the feedback pointed somewhere the anchor would never go. It is Normal - everywhere now, with the snap dot showing the real target. - -An e2e test sweeps the cursor along the bottom of a zoomed-out 1h chart and -requires every position to land on the low of the bar beneath it — 115 of 115. -That test is the rule, executable. - -**The snap leapt to the live edge — found by diagnostic mode.** Reported from -the user's own browser, which no headless run had reproduced: - -``` -cursor_x 1409.0 chart_w 1280.0 -> cursor_t None -> snapped to the last bar -cursor_x 1161.0 chart_w 1280.0 -> cursor_t None -> snapped to the last bar -cursor_x 1128.0 -> valid time, drift 0 bars -``` - -`coordinateToTime` answers `null` over the right-hand price axis, over the -whitespace past the last bar, and anywhere outside the chart — and `snapPoint` -read that as "the newest bar", so the dot jumped to the live edge from wherever -the cursor was. Two faults behind it: the tool's pointer listener is on `window` -and therefore fires over the sidebar (x=1409 on a 1280-wide chart), and a null -time meant a default rather than no answer. - -Now a pointer outside the plot hides the indicator entirely, and a null time -resolves to the bar nearest in *pixels* rather than the newest one. Covered by -an e2e test that hovers a bar, the axis, the sidebar, and back. - -The lesson is about method rather than geometry: four hypotheses were tested and -killed by measurement here — device pixel ratio, viewport size, resize -desynchronisation, and the chart scrolling under the gesture — while the actual -cause was visible in one line of the client's own numbers. When the browser is -on another machine, instrument it early instead of reproducing locally. - -### Overlays are positioned against the plot, not the element - -**The trendline snap was 66 pixels out, and so was everything else drawn over -the chart.** Lightweight Charts reports coordinates from the plot area's origin. -The chart *element* also contains the price scales, so once the left scale was -enabled for the daily labels, the plot started 66px into the element — and every -overlay positioned with `left:` against the element was displaced by exactly -that much, in both directions at once: - -- the cursor's element-x was read as a plot-x, resolving a bar ~66px to the - right of the pointer; -- the indicator was then drawn at that bar's plot-x interpreted as element-x, - landing ~66px left of where the bar is painted. - -Not near the cursor, not near the bar, and varying with zoom — 66px is a couple -of bars at 30m and a dozen at 1m, which is why it looked random rather than -offset. Every diagnostic number agreed with itself throughout, because -`dot_y`, `expected_y` and `bar_low_y` all derive from the same API and shared -the same wrong origin. Self-consistent instrumentation cannot see a systematic -error in its own frame of reference. - -All overlays now live in one container positioned over the plot canvas, so they -inherit plot coordinates untranslated: the snap dot and label, the comment -layer, the trendline anchor handles, the preview line, the tooltip, the price -tag and the context menu. `eventPoint` subtracts the same offset, so a pointer -position and a chart coordinate finally mean the same thing. The container is -repositioned on resize. - -This had been mis-diagnosed for hours: device pixel ratio, viewport size, resize -desynchronisation, the chart scrolling under the gesture, and the dead band -below the candles were each measured and ruled out. The measurement that found -it was comparing `canvas.width` to `element.clientWidth` — 0.894 — which is the -first thing that ever disagreed with itself. - ---- - -# Handoff: the snap indicator is ~10px left of where it belongs - -**Status: fixed.** Everything below is measured, not inferred. - -## The symptom - -With the Trendline tool armed, the snap dot sits about one bar to the left of -the cursor. The *height* is correct and the *bar it chooses* is correct — only -the horizontal drawing position is wrong. Reported from a real browser and -reproduced headlessly. - -## The measurement - -``` -bar spacing 6.96 px -dot centre - cursor -10.5 px (= 1.5 bars at that zoom) -chosen bar correct (label names the bar under the cursor) -``` - -And the cause, from `ConfluenceChart.syncOverlayLayer()`: - -``` -at load: containerLeft 56 true plot offset 66 <- 10px stale -after a re-sync: containerLeft 66 true plot offset 66 <- correct -``` - -## Why - -Lightweight Charts reports coordinates from the **plot area's** origin. The -chart *element* also contains the price scales, so with the left scale enabled -the plot begins 66px in. All overlays therefore live in a container -(`.chart-overlays`) positioned over the plot, so they can use chart coordinates -untranslated — see `create()` and `syncOverlayLayer()` in `static/chart.js`. - -`syncOverlayLayer()` runs once in a `requestAnimationFrame` during `create()`. -At that moment the left price scale has not finished sizing itself to its label -text, so the measured offset is 56. It settles at 66 once labels render, and -nothing re-measures. The container stays 10px left of the plot for the life of -the page, which drags every overlay with it: the snap dot and label, comments, -trendline anchor handles, the preview line, the tooltip and the price tag. - -Only `x` is affected. `y` never passes through this offset, which is why the -height has always looked right. - -## The fix to write - -Re-measure instead of measuring once. Options, cheapest first: - -1. `ResizeObserver` on the plot canvas — fires when the scale settles and on - every later change. Probably the right answer. -2. Call `syncOverlayLayer()` at the top of `renderComments()` and - `showSnapDot()`. Correct but does DOM reads on every mouse move. -3. Re-sync on `subscribeVisibleLogicalRangeChange` as well as on resize. Cheap, - but misses a scale that widens without the range changing. - -Beware: the left scale's width depends on its **label text**, so it changes when -the price range gains a digit or a longer level label appears. Whatever you -choose must survive that, not just the initial load. - -## How to verify - -```bash -./bin/e2e trendline # 7 cases, all currently pass — they do not catch this -``` - -The suite misses it because its assertions go through the same coordinate API -that carries the error. Add a test that measures in **page pixels**: place the -cursor exactly at a bar's centre and assert the dot's centre is within ~2px -horizontally. The reproduction is: - -```js -const r = el.getBoundingClientRect(); -const bx = chart.timeScale().timeToCoordinate(bar.t); -await page.mouse.move(r.x + c.plotOffsetX() + bx, r.y + c.candles.priceToCoordinate(bar.l) - 6); -// dot centre x should equal the cursor x; today it is ~10px left -``` - -Live numbers from the client are available without a console: open the chart -with `?diag=1`, then `docker compose logs api | grep SNAPDBG`. Note that -`SNAPDBG` will **not** show this bug — `dot_y`, `expected_y` and `bar_low_y` all -derive from the same API and share the same origin, so they agree with each -other while being wrong together. The error is only visible by comparing against -something outside that frame of reference: `canvas.getBoundingClientRect()` -against `element.getBoundingClientRect()`, or painted pixels. - -## Context worth having - -- `window.__chart` is a deliberate debug handle exposing the wrapper. -- The dev stack is at `http://localhost:8010`, and `http://api:8000` from inside - the playwright container. It runs on a remote machine; the user's browser does - not. Headless passes prove little about their screen — see "Where things run" - in AGENTS.md. -- Related history is above under "Overlays are positioned against the plot, not - the element", which fixed the 66px case this 10px residue survived. -- One e2e test, "clicking a comment collapses it", is flaky (roughly one run in - three) and unrelated. Worth fixing before trusting the suite. - -## Resolution - -`ConfluenceChart` now attaches a `ResizeObserver` to the plot canvas itself. -Unlike the outer chart element, that canvas changes width when Lightweight -Charts finishes sizing the left price scale or a longer price label appears. -The observer repositions the shared overlay layer and redraws its anchored DOM -content after each such change. - -The trendline suite now compares the snap dot and cursor in page pixels, then -forces a six-digit left-scale label and repeats the assertion. This catches the -stale coordinate frame that the chart API's self-consistent coordinates could -not. The comment-collapse flake was also fixed: a broad CSS rule had re-enabled -pointer events on the full-size handles SVG, which intermittently covered a -comment. Overlay surfaces now ignore pointers unless the actual control opts in. - -### 2026-08-11 02:55 CDT — test-only coverage, mobile plan and feed research - -Added regression coverage without changing production code. Pytest grew from -96 to 103 cases: Yahoo H1 seed bars must form the correct CME-session daily -OHLCV, manual alerts must remain disarmed through rebuild and restart, daily -anchors must follow Eastern DST in UTC, and two WebSocket clients must keep -their layer preferences isolated from one another and from global alert state. -The browser suite grew from 15 to 19 cases: partial or malformed persisted -preferences must still boot, volume must remain paired with candles on its own -scale across timeframe changes, and Backspace/Delete in a rename input must edit -text rather than delete the drawing. Full results: 103 backend and 19 browser -tests passing. - -One proposed test exposed a current defect and was deliberately not committed as -a failing test: `withinPlot()` compares against the outer chart element, which -includes the right price axis. Hovering that axis can therefore leave the snap -dot visible at the plot edge. The production fix is to compare plot-relative -coordinates against the measured plot canvas width and height, then add the -page-pixel price-axis case. Startup rebuild-count coverage is also deferred until -the bulk-seeding optimization exists; a duration assertion against today's slow -startup would encode the problem rather than protect a fix. - -Local and deploy test commands now live in README. Browser E2E stays local -because it creates and deletes drawings; production verification uses -`bin/wait-deploy`, `/api/health`, `/api/version`, and an authenticated -`/api/status` smoke check. - -The mobile audit is recorded in `docs/mobile_enhance.md`. Its recommended first -steps are one coordinate path for every gesture, touch-sized invisible hit -areas, persistent first-anchor feedback, and a sticky mobile tool rail before -more ambitious gesture changes. - -No durable unauthenticated source of free real-time CME `/ES` data was found. -Schwab remains the verified entitled live source and Yahoo the practical delayed -seed source. Tastytrade/dxLink is the best free-with-broker-account candidate to -test next; IBKR is the strongest low-cost fallback rather than a free one. CME, -TradingView and Barchart free pages are delayed displays, not licensed backend -APIs. For freshness UI, existing bar WebSocket messages can supply browser -receipt time without another call, but exact trade time and source heartbeat are -not retained yet. The best placement is the existing status strip below the -chart, with a compact mobile form such as `Updated 3s ago · Live`. - -### 2026-08-11 03:19 CDT — make recommendations discoverable - -Added `docs/NEXT_STEPS.md` as the concise home for deferred production fixes, -freshness UI semantics, mobile priorities and market-data alternatives. Linked -it, the implementation/session log and `docs/mobile_enhance.md` directly from -`AGENTS.md`, which every coding agent reads before working in this repository. -This keeps current advice visible without turning the historical implementation -plan into an undifferentiated backlog. - -### 2026-08-11 03:39 CDT — one price scale, left labels and editable-line snapping - -The left side was introduced to keep daily moving-average, VWAP and prior-day -labels away from the intraday labels on the right. That decluttering intent was -correct; implementing it as a second Lightweight Charts price scale was not. -Each scale autoscaled independently once its own overlays were attached, so the -same numeric price could occupy a different y-coordinate on each side. Daily -and intraday structure then looked directly comparable while being geometrically -unrelated. - -All price-bearing series and flat levels now share the candle series' right -scale. Long-term labels remain on the left as DOM overlays whose y positions are -computed through `candles.priceToCoordinate()`, so they preserve the intended -separation without creating another coordinate system. Context series suppress -their built-in right-side titles. A browser invariant verifies every overlay's -scale id, every flat level's host, and sub-pixel agreement between an MA price -and the candle coordinate for that same price. - -Editing a selected trendline had a separate defect: handle dragging used -element-relative x, snapped only to the nearest bar time, and retained a raw -cursor price. Initial placement used plot-relative coordinates and high/low -snapping, so moving an anchor could visibly jump off the bar and change the line -to an unusable slope. Placement, selection, handle dragging and context actions -now all use `eventPoint()` against the measured plot canvas. Handle movement -passes through the same `snapPoint()` high/low rule and displays the same snap -feedback as placement. This also closes the known right-price-axis containment -bug; a page-region test now proves the dot disappears over the actual axis and -returns over the plot. - -The 30-minute chart had only about one week of history because it was derived -solely from Yahoo's eight-day 1-minute seed. The 730-day hourly seed cannot -reconstruct half-hour candles. Yahoo's native `30m` interval was verified live -at `range=60d` (2,843 bars on 2026-08-11), so startup now seeds H1/730d, -M30/60d, then M1/8d. The finer minute aggregation replaces the recent overlap; -the native feed supplies older 30-minute bars. Regression tests pin both Yahoo's -native interval request and the startup seed sequence. Final verification: 105 -backend tests and 22 browser tests passing. - -Daily moving averages had also been rendered with `WithSteps` on every base -timeframe. Holding a daily value constant is correct when projecting it over -intraday candles, but on the daily chart it made the SMA itself look like a -staircase. MA line type is now timeframe-aware: stepped on intraday charts and -simple point-to-point lines on `1d`, with both modes pinned by browser coverage.