chris/chart
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
chart/docs/IMPLEMENTATION_PLAN.md
Chris Amow 0b3244b725 Make trendline placement match what the cursor shows 
Two faults, one of them a regression that reached production.

Making a pending anchor always win fixed a twitch on the second click stealing
the start point, and broke the opposite case: a genuine press-drag begun after
an abandoned click was hijacked by that stale anchor, so the line started far
from where the drag did. A single threshold decides now — 12px of travel between
press and release makes a gesture a drag, wide enough to survive a twitch on a
deliberate click and unambiguous for a real one. A drag abandons any half-placed
anchor instead of adopting it.

The crosshair was also lying. Lightweight Charts defaults to CrosshairMode.Magnet,
which snaps the crosshair to the bar's close, so hovering beside a bar's low drew
it mid-bar and a correctly-placed anchor looked wrong. Measured: aiming 4px above
a bar low anchors at the low, 7773, not the close, 7773.25 — the placement was
right and only the feedback was wrong. Arming a tool switches the crosshair to
Normal, and a snap dot now marks the exact point the anchor will use, coloured by
the side that extreme implies.

Verified in a browser across all four paths: two clicks with a twitch on the
second, an abandoned click then a real drag, a plain press-drag, and hovering.
Each starts where it should and lands on a bar extreme, and the stale anchor is
no longer adopted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 16:29:00 -05:00

1528 lines
75 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# /ES Multi-Timeframe Confluence Chart — Implementation Plan
**Audience:** the implementing agent. This document is the spec; it is written to be
executed top-to-bottom without re-deriving decisions.
**One-line goal:** stream `/ES` 1-minute bars from Schwab, aggregate them into every
larger timeframe locally, derive trendlines and moving averages on each timeframe,
project them all onto one chart in a shared `(time, price)` coordinate system, and
alert when independently-derived levels from different timeframes converge.
**Explicitly out of scope:** order execution. Nothing in this codebase places a trade.
See [§14](#14-why-execution-is-out-of-scope) for why, and for the seam left behind.
---
## 0. Start here
**Read §1, §2.1, §6, and §13 before writing anything.** The rest can be read as you
reach each milestone.
**Work on a branch — do not push to `main`.** `main` is wired to a Forgejo webhook that
triggers a Coolify production deploy at <https://chart.amow.com>. Pushing to main ships
whatever you wrote. Branch: `feat/chart-engine`.
**Build order is M0 → M1 → M2 → M3 → M3.5 → M4 → M5.** Stop after M5 and get feedback;
M6+ are separately scoped. No API keys are required for any of M0–M5.
**Existing repo state:** a placeholder FastAPI + Vue 3 (CDN, no build step) app.
`main.py` serves `static/index.html` and two toy `/api` endpoints. The serving and
deploy wiring is correct and should not be redesigned — extend it. The toy `/api/hello`
endpoint and its frontend button can be deleted.
### Dependencies to add
`requirements.txt` currently contains only `fastapi` and `uvicorn[standard]`. Add:
```
httpx # Yahoo fetches; async, already a FastAPI-adjacent standard
pydantic-settings # config.py
```
`requirements-dev.txt`:
```
pytest
pytest-asyncio
```
Do **not** add `schwab-py` until M6 — it is unused before then. Do **not** add
`yfinance`; the Yahoo chart endpoint is a plain HTTP GET and the extra dependency buys
nothing (verified — see §2.1).
### Test fixture already provided
`tests/fixtures/yahoo_es_1h.json` is a **real, trimmed** Yahoo response for
`ES=F&interval=1h` (40 bars). Use it to unit-test the parser offline. It deliberately
**contains a `null` in the OHLC arrays**, which is exactly the case §2.1 warns about —
if your parser doesn't filter those, that fixture will catch it.
### Conventions
- All times are **epoch seconds, UTC**, everywhere. No naive datetimes.
- Nothing outside `market/` may know which data source is in use.
- Nothing outside `market/schwab.py` may import broker-specific code.
- Analysis code takes lists of bars and returns values — no I/O, no clocks. This is
what makes the replay harness work.
---
## 1. Decisions already made — do not relitigate
| Decision | Choice | Why |
|---|---|---|
| Backend | FastAPI (already scaffolded) | Repo already runs it; native WebSocket support |
| Frontend | Vue 3 from CDN, **no build step** | Matches existing `static/` setup; keeps deploy trivial |
| Charting | TradingView Lightweight Charts **v5.2.0**, standalone build | Apache-2.0, canvas, built for incremental realtime updates |
| Data source | **Pluggable `MarketDataSource`.** Yahoo first, Schwab later | Yahoo needs no API key *and* has the history Schwab lacks — see §2.1 |
| Persistence | **In-memory first**, behind a `BarStore` interface | User confirmed deferring persistence is fine for v1 |
| Eventual persistence | **SQLite**, not Postgres | Single file, zero Coolify stack expansion. Revisit only if multi-process |
| Deployment | **Local first**, VPS later | Keep all config in env vars so the VPS move is config, not rewrite |
| Alerts | ntfy/Pushover phone push (+ free in-browser sound) | User selected phone push |
| Auth | Hardcoded shared secret from env | User confirmed; only matters once VPS-exposed |
| Trendlines | **Manual (hand-drawn) first.** Auto-detection deferred to M8 | Hand-drawn lines are correct by definition, so they validate the confluence engine with zero tuning risk — and later become the ground truth the auto-detector is tuned against |
| Moving averages | **Daily set (10/20/50/100/200 SMA) is the priority**, shown on 1m / 30m / 1d | The 200DMA is a level people actually trade against |
### Timeframe roles
> **4h was removed from the product on 2026-08-10.** It was never enabled, and
> dropping it deleted the fiddliest logic in `session.py` — the wall-clock ET 4h
> anchor and its DST edge cases — for a timeframe nobody was using. Later
> sections of this document still use 4h in examples; read those as
> illustrative, not as a spec to build. Nothing below a day is session-anchored
> any more, so `bucket_start` now special-cases only `1d`.
```
1m base chart + alert evaluation weight 1 ← a switchable base timeframe
2m display only
5m manual lines weight 1
15m manual lines weight 2
30m base chart + manual lines weight 3 ← a switchable base timeframe
1h manual lines (+ optional MAs) weight 4
1d base chart + THE DAILY MAs weight 16 ← a switchable base timeframe
```
Base timeframe controls the candles only. **Every level stays visible on every base
timeframe** — the 200DMA on a 1-minute chart is the point, not a side effect.
---
## 2.1 Data sources — build against Yahoo, swap in Schwab later
**Do not block on Schwab API keys.** The two sources are complementary, and the
Yahoo one is strictly easier to develop against:
| | Yahoo `ES=F` | Schwab `/ES` |
|---|---|---|
| Auth | none — plain HTTP GET | OAuth, keys, 7-day token refresh |
| Realtime | polled, possibly ~10 min delayed | true push websocket, realtime |
| 1m history | **8 days** (per-request cap) | ❌ none |
| 1h history | **~730 days** (17,387 bars, verified back to 2024-03) | ❌ none |
| 1d history | **~10 years** (2,517 bars, verified back to 2016) | ❌ none |
| Contract | continuous front-month, roll gaps | true contract |
Endpoint, verified working with no key and no `yfinance` dependency:
```
https://query1.finance.yahoo.com/v8/finance/chart/ES=F?interval=1h&range=730d
```
Requires a browser `User-Agent` header. Returns `chart.result[0]` with `timestamp[]`
and `indicators.quote[0].{open,high,low,close,volume}` as parallel arrays.
**Those arrays contain `null` holes — filter them before constructing `Bar`s.**
Therefore define one protocol and two implementations:
```python
class MarketDataSource(Protocol):
name: str
def supports_history(self) -> bool: ...
async def history(self, symbol, tf, start, end) -> list[Bar]: ...
def supports_stream(self) -> bool: ...
async def stream(self, symbol) -> AsyncIterator[Bar]: ... # yields 1m bars
```
- `YahooSource` — real `history()`. `stream()` is a **polling loop** (every 15–30 s,
`interval=1m&range=1d`, emit only bars newer than the last emitted) that presents the
same async-iterator interface as a real push stream.
- `SchwabSource` — `supports_history() -> False`. `stream()` is the true
`CHART_FUTURES` websocket.
- `ReplaySource` — reads a JSONL tape. Used by tests and offline development.
Nothing downstream of these may know which source it is using. Selection is one env
var. **In production both run at once:** Yahoo seeds history at startup, Schwab
provides the live tail.
### Do not use Yahoo's daily bars
Yahoo anchors `ES=F` daily bars to **midnight ET**, but the CME futures session runs
**18:00 → 17:00 ET** (§6). Mixing the two definitions yields daily candles that
disagree with every other chart you'll compare against.
**Build daily bars yourself by aggregating Yahoo's 1h bars** through the same
`aggregator.py` used for live data — one session definition everywhere. Yahoo's 1h bars
are anchored to the top of the ET hour, so they compose into session-anchored 1d
buckets cleanly. The ~730-day 1h window yields ~500 sessions: enough for a daily 200SMA
(~200 sessions) with room to spare.
Yahoo's native 1d bars may be used *only* for multi-year context, clearly labelled.
## 2.2 Schwab entitlements — **answered empirically 2026-08-10**
All verified against a live production app holding both Market Data Production and
Accounts and Trading Production.
| Question | Answer |
|---|---|
| Futures market data entitled? | **Yes** — but only via `get_quotes()` (plural) |
| Symbol format | **`/ES`**, which auto-resolves to the active contract `/ESU26` |
| `CHART_FUTURES` streaming | **Works** — one true-OHLCV minute bar per symbol per minute |
| `LEVEL_ONE_FUTURES` | **Works**, and reports `delayed: false` |
| Futures price history | **Still none.** Yahoo remains the only source of the past |
Three traps found the hard way, all of which cost a round trip:
- **`get_quote()` (singular) silently returns the wrong instrument.** It puts the
symbol in the URL *path*, where the leading slash is normalised away, so `/ES`
comes back as `ES` — Eversource Energy, an equity, at $72. HTTP 200 with a
populated body. `get_quotes()` passes symbols as a query parameter and returns
the future correctly. **Never treat a 200 as proof; check `assetMainType`.**
- **Streaming needs the Accounts and Trading product.** `StreamClient.login()`
reads `/trader/v1/userPreference` for its socket URL, and that path is not in
Market Data Production. A market-data-only app cannot stream at all.
- **Authorisation codes expire in about thirty seconds**, and an unwritable token
path spends one before revealing itself. `scripts/check_schwab.py` preflights
the key, the secret and the token path for exactly this reason.
Because `/ES` resolves to the active contract on Schwab's side, contract roll
handling — an open problem in §10 — needs no code here.
### The remaining unknowns for Schwab
1. **Futures market-data entitlement.** It is not publicly documented whether
`CHART_FUTURES` requires futures trading approval or a CME non-professional market
data agreement on the Schwab account. Verify empirically in M6.
2. **Symbol format.** `schwab-py` docs show both `/ES` (continuous front-month) and
`/ESZ25` (specific contract). Determine which the stream actually accepts, and
whether the continuous form auto-rolls. Record the answer in the README.
3. **Bar cadence and lateness.** Confirm `CHART_FUTURES` emits one message per symbol
per minute, whether it re-sends a bar (correction), and how bars behave across the
17:00–18:00 ET settlement break.
4. **Volume semantics.** Confirm `VOLUME` is per-minute, not cumulative-for-session.
### Confirmed facts (already verified — do not re-research)
**Realtime futures data: fully available.** Do not let §10 below suggest otherwise —
these are different axes and conflating them will send you down the wrong path.
| Service | Available? | Use here |
|---|---|---|
| `CHART_FUTURES` (1-min OHLCV, `/ES`) | ✅ streaming | The base feed. Everything derives from it |
| `LEVEL_ONE_FUTURES` (live quotes) | ✅ streaming | Current price/bid/ask for the status bar and alert evaluation |
| `LEVEL_ONE_FUTURES_OPTIONS` | ✅ streaming | Live pricing of a proposed /ES put spread in M7 |
| REST `get_price_history` for futures | ❌ **not available** | — see §10 |
| `CHART_HISTORY_FUTURES` service | ❌ does not exist in `schwab-py` | — |
| Futures / futures-options **order entry** | ❌ not offered | see §14 |
- `ChartFuturesFields`: `SYMBOL=0, CHART_TIME_MILLIS=1, OPEN_PRICE=2, HIGH_PRICE=3,
LOW_PRICE=4, CLOSE_PRICE=5, VOLUME=6`.
- **The gap is historical only, not realtime.** schwab-py's docs, verbatim: *"Schwab
provides price history for equities and ETFs. It does not provide price history for
options, futures, or any other instruments."* A Schwab account does not change this —
it is an API-surface limitation, not an entitlement one. Live `/ES` streams fine;
there is simply no way to ask for *yesterday's* `/ES` bars. This is the single
biggest constraint in the project; see [§10](#10-the-history-problem).
- Lightweight Charts 5.2.0 standalone build exposes a `window.LightweightCharts`
global containing `createChart`, `CandlestickSeries`, `LineSeries`,
`createSeriesMarkers`, `LineStyle`. CDN:
`https://unpkg.com/lightweight-charts@5.2.0/dist/lightweight-charts.standalone.production.js`
(~196 KB).
- **v5 changed the series API.** Use `chart.addSeries(LightweightCharts.CandlestickSeries, opts)`.
The v4 `chart.addCandlestickSeries(opts)` form **does not exist in v5** — most
tutorials online are v4 and will not work.
---
## 3. Architecture
```
YahooSource SchwabSource ReplaySource
(history + poll) (live websocket) (JSONL tape)
└──────────────────────┼──────────────────────┘
│ MarketDataSource protocol
▼ 1-minute OHLCV
┌─────────────────┐
│ StreamService │ single asyncio task, ONE connection
│ (+ recorder) │──────► raw JSONL tape (for replay)
└────────┬────────┘
│ Bar(1m)
▼
┌─────────────────┐
│ Aggregator │ session-aware bucketing
└────────┬────────┘
│ Bar(tf) closed / updated
┌──────────────┼──────────────┐
▼ ▼ ▼
┌──────────┐ ┌────────────┐ ┌──────────┐
│ BarStore │ │ Pivots → │ │ Moving │
│ (memory) │ │ Trendlines │ │ Averages │
└──────────┘ └─────┬──────┘ └────┬─────┘
│ │
└──────┬───────┘
▼ Level[] (unified type)
┌──────────────────┐
│ ConfluenceEngine │ cluster + score
└────────┬─────────┘
▼ Cluster[]
┌──────────────────┐
│ AlertEngine │ state machine + cooldown
└────────┬─────────┘
│
┌──────────────┴───────────────┐
▼ ▼
FastAPI WebSocket ntfy push
│
▼
Vue 3 + Lightweight Charts
```
### Critical process constraint
The Schwab stream is **one connection, one process, not thread-safe**. Therefore:
- Run the streamer as a single `asyncio` task owned by FastAPI's `lifespan`.
- **`uvicorn --workers 1` always.** More than one worker means more than one Schwab
connection, which will fight over the session.
- `--reload` in dev will tear down and re-establish the stream on every file save.
That is acceptable locally but expect reconnect churn.
- If the VPS deploy later needs multiple web workers, the streamer must be split into
its own process with a message bus. Do not design for that now, but keep
`StreamService` free of any FastAPI imports so the split stays cheap.
---
## 4. Module layout
```
main.py FastAPI app: lifespan, route mounting (exists, extend)
app/
config.py Settings from env (pydantic-settings)
auth.py Shared-secret gate (no-op when unset)
market/
base.py MarketDataSource protocol, Bar emission contract
yahoo.py YahooSource: history() + polled stream() ← build first
schwab.py SchwabSource: easy_client, CHART_FUTURES websocket
replay.py ReplaySource: JSONL tape
stream.py StreamService: owns a source, reconnect, emit Bar
recorder.py Record raw messages to JSONL
bars/
models.py Bar, Timeframe
session.py CME session calendar + bucket boundary math
aggregator.py 1m -> all timeframes
store.py BarStore protocol + InMemoryBarStore
analysis/
indicators.py ATR, SMA, EMA (pure functions over bar lists)
manual_lines.py Hand-drawn lines: CRUD + JSON persistence ← M5
pivots.py Fractal swing detection ← M8, deferred
trendlines.py Candidate generation, scoring, dedup ← M8, deferred
moving_averages.py MTF MA levels
levels.py Level type + registry, rebuild orchestration
confluence.py Clustering + scoring
alerts.py State machine, cooldown, dispatch
notify/
ntfy.py Phone push
api/
routes.py REST
ws.py WebSocket hub
static/
index.html Vue 3 + LWC script tags (exists, replace)
app.js Vue app root (exists, replace)
chart.js Lightweight Charts wrapper (new)
style.css (exists, extend)
tests/
...
```
---
## 5. Data model
Use dataclasses (or pydantic where it crosses the API boundary). All times are
**epoch seconds, UTC**. Never store naive local datetimes.
```python
class Timeframe(str, Enum):
M1="1m"; M2="2m"; M5="5m"; M15="15m"; M30="30m"; H1="1h"; D1="1d"
@property
def seconds(self) -> int: ... # D1 is session-defined, not 86400 — see §6
@dataclass
class Bar:
tf: Timeframe
t: int # epoch seconds, bucket OPEN time
o: float; h: float; l: float; c: float
v: int
closed: bool # False while still forming
symbol: str # e.g. "/ESZ25" — carried so contract rolls stay visible
```
`Level` is the unified abstraction that makes the whole design work. Trendlines and
moving averages both reduce to "a price at time t, with a weight and a side".
```python
class LevelKind(str, Enum):
MANUAL="manual" # hand-drawn — ships first (M5)
MA="ma" # moving average — ships first (M3)
TRENDLINE="trendline" # auto-detected — deferred to M8
HORIZONTAL="horizontal"
class Side(str, Enum):
SUPPORT="support"; RESISTANCE="resistance"
@dataclass
class Level:
id: str # stable hash — the UI diffs on this, so keep it stable
kind: LevelKind
tf: Timeframe
side: Side
weight: float # timeframe weight x quality multiplier
score: float # raw quality score before weighting
label: str # "4h resistance", "1h EMA21"
# Geometry. Trendline: price(t) = slope*(t - anchor_t) + anchor_p
anchor_t: int
anchor_p: float
slope: float # price units per SECOND. 0.0 for horizontal/MA-at-instant
# For MAs: the stepped polyline actually drawn
points: list[tuple[int, float]] | None
touches: int
first_t: int
last_t: int
provisional: bool # derived from a still-forming bar
hidden: bool # layer-panel visibility — see §9.4 for its effect on scoring
def price_at(self, t: int) -> float:
return self.anchor_p + self.slope * (t - self.anchor_t)
```
The weight table referenced throughout — define it once, in `config.py`:
```python
TIMEFRAME_WEIGHT = {
Timeframe.M1: 1, Timeframe.M2: 1, Timeframe.M5: 1, Timeframe.M15: 2,
Timeframe.M30: 3, Timeframe.H1: 4, Timeframe.D1: 16,
}
MA_WEIGHT_FACTOR = 0.75 # §7.4 — MAs weigh slightly less than drawn structure
```
```python
@dataclass
class Cluster:
id: str
side: Side
low: float; high: float; center: float
score: float # sum of member weights
members: list[Level]
distance: float # signed points from current price
```
---
## 6. Session and aggregation rules — read carefully
This is where a naive implementation silently produces wrong lines. CME ES is not a
9:30–16:00 instrument.
**Session definition (America/New_York, DST-aware via `zoneinfo`):**
- Trading week opens **Sunday 18:00 ET**.
- Daily settlement break **17:00–18:00 ET, Monday–Thursday**. No bars expected.
- Week closes **Friday 17:00 ET**.
- A **futures "day"** runs 18:00 ET → 17:00 ET the following calendar day, and is
conventionally labelled with the *following* calendar date. Sunday 18:00 bars belong
to Monday's daily bar.
**Bucketing rules:**
- `1m, 2m, 5m, 15m, 30m, 1h` — bucket on wall-clock UTC boundaries. These divide the
hour evenly, so session anchoring is unnecessary and UTC keeps it simple.
- `1d` — one bar per futures session as defined above. **This is the only
session-anchored timeframe.** 4h used to be the other one and was the reason this
section warned about DST; with 4h gone, that whole class of edge case went with it.
Implement this as `session.py::bucket_start(t: int, tf: Timeframe) -> int` and unit
test it hard, including both DST transitions and the Sunday open. **This function is
the highest-risk piece of pure logic in the project.** Write its tests first.
**Aggregator behaviour:**
- Maintain one forming bar per timeframe. On each incoming 1m bar:
- if `bucket_start(bar.t, tf)` differs from the forming bar's `t`, close the forming
bar (emit `closed=True`), then open a new one;
- otherwise fold in: `h=max`, `l=min`, `c=close`, `v+=`, emit `closed=False`.
- **Gaps do not close bars by time — they close by the arrival of a later bar.** Never
use a wall-clock timer to close a bucket; the market halts and holidays will fire it
incorrectly. Exception: emit a `closed=True` for the previous bucket when a bar
arrives that skips buckets entirely.
- Aggregation must be **deterministic and replayable**: feeding the same 1m sequence
twice must produce byte-identical output. No `datetime.now()` inside the aggregator.
---
## 7. Analysis engines
### 7.1 Indicators (`indicators.py`)
Pure functions, list-in/list-out, no state: `sma(values, period)`, `ema(values, period)`,
`atr(bars, period=14)`. ATR is the universal scale unit — every tolerance in this
project is expressed in ATR multiples, never in fixed points, so the same config works
whether ES is at 4,000 or 8,000.
### 7.2 Pivot detection (`pivots.py`)
Fractal method, not regression. Regression fits the *middle* of price action; humans
draw lines across *extremes*, and extremes are what other traders react to.
```
● pivot high (w bars lower on both sides)
/ \
/ \
● / \
/ \ / \
───●───●──────────────
pivot low
```
- `pivot_high(bars, w)`: index `i` qualifies if `high[i] >= max(high[i-w : i+w+1])`
and `i` is the leftmost such index in ties.
- Default `w = 3` for lower TFs, `w = 2` for `4h`/`1d` (fewer bars available).
- **Confirmation lag is `w` bars — this is intentional.** A pivot is only known `w`
bars after it forms. Do not "detect" pivots on the forming bar; that repaints, and a
repainting line is worse than no line.
- Significance filter: keep a pivot only if its prominence (distance to the
surrounding swing in the opposite direction) `>= 0.5 * ATR(14)` on that timeframe.
Drops noise pivots without hardcoding point values.
### 7.3 Automatic trendlines (`trendlines.py`) — **deferred to M8**
> Not part of the initial build. **Manual trendlines (§7.3a) ship first.** This section
> is retained because it is the eventual target and because §7.3a is deliberately
> designed to produce the same `Level` objects, so adopting this later changes nothing
> downstream. Skip to §7.3a on a first pass.
```
for each timeframe, for each side (highs → resistance, lows → support):
P = last N qualifying pivots (N = 25; ~300 candidate pairs, trivial)
for each pair (a, b) in P where a.t < b.t:
line = through (a.t, a.p) and (b.t, b.p)
evaluate(line) -> score or reject
dedup, keep top K = 4 per side per timeframe
```
**Evaluation of a candidate line:**
Let `tol = 0.25 * ATR(14)` on that timeframe.
- **Violation** — a bar *closes* beyond the line by more than `tol`
(above for resistance, below for support). Wicks do not count as violations; wicks
through a level are normal and often the point.
- **Touch** — a bar's extreme comes within `tol` of the line without violating it.
- **Reject** the line if `violations > 1` between the two anchors, or if any violation
occurred after the later anchor (the line is broken and no longer active).
**Score:**
```
score = 3.0 * touches
+ 1.5 * log1p(span_in_bars)
+ 2.0 * recency_decay(last_touch) # exp(-age_bars / halflife), halflife=50
- 4.0 * violations
```
Normalize within each timeframe/side group so weights stay comparable across
timeframes regardless of how many candidates a given timeframe happened to produce:
```python
best = max(l.score for l in group) # after dedup, before truncation to K
quality = clamp(l.score / best, 0.0, 1.0) if best > 0 else 0.0
level.weight = quality * TIMEFRAME_WEIGHT[tf]
```
So the strongest 4h line contributes the full 8.0, a mediocre one proportionally less,
and a 5m line can never outweigh a 4h line no matter how many touches it has.
**The four coefficients above (3.0 / 1.5 / 2.0 / 4.0) are starting values, not
derived constants.** They cannot be got right on paper — expect to tune them by
eye against replayed tapes in M4. Put them in `config.py`, not inline, and treat "the
lines land where a human would draw them" as the acceptance criterion.
**Dedup:** two lines are duplicates if, evaluated at *now*, their prices are within
`tol` **and** their slopes differ by less than 20%. Keep the higher score. Without
this you get a fan of ten nearly-identical lines from the same swing.
**Recompute policy:** only on a **bar close** for that timeframe, never on every tick.
A 4h line recomputes 6× per day. This is what keeps the whole thing cheap.
### 7.3a Manual trendlines (`manual_lines.py`) — ships in M5
A hand-drawn line is just a `Level` with `kind=MANUAL`. It flows into confluence,
projection, and alerts through exactly the same path as everything else — that is what
makes deferring the automatic engine cheap rather than a detour.
**Why this ordering is better than it looks:** hand-drawn lines are *correct by
definition* (you drew them). So they validate the confluence engine without the
automatic detector's tuning risk, and they later become the ground truth that M8's
scoring coefficients get tuned against.
**Geometry.** Anchors are stored in **absolute epoch seconds and price** — never bar
indices. This is why a line drawn on the 4h chart renders correctly on the 1m chart
with no conversion: both are the same `(time, price)` plane. The existing
`Level.price_at(t)` already handles it.
**Timeframe attribution.** Tag the line with the timeframe that was *displayed when it
was drawn*. A line drawn on the 4h chart is a 4h line and carries weight 8. This is the
single most important field — without it every manual line would score identically.
**Weight.** `quality = 1.0` always. The user drew it; it is not a candidate to be
scored. `weight = TIMEFRAME_WEIGHT[tf]`.
**Drawing interaction** (all APIs verified present in LWC 5.2.0):
| Action | Implementation |
|---|---|
| Enter draw mode | Toolbar button; changes cursor |
| Place endpoints | `chart.subscribeClick(handler)` → two clicks |
| Pixel → price | `series.coordinateToPrice(param.point.y)` |
| Pixel → time | `chart.timeScale().coordinateToTime(param.point.x)` |
| Render | `LineSeries` with 2 points, extended right (same as §9 auto lines) |
| Select | Click within ~6px of a line — hit-test in price space via `price_at(t)` |
| Delete | `Delete`/`Backspace` on selection, plus a button |
| Edit | **Delete and redraw.** Endpoint dragging is real work — do not build it in M5 |
**Snapping.** When placing an endpoint, snap to the nearest bar high/low within ~8px.
Cheap to implement and it is the difference between a usable drawing tool and a
frustrating one. Make it toggleable; snap to high for resistance, low for support,
inferred from drag direction or nearest extreme.
**Persistence — required in M5, not deferred.** A user who redraws their lines after
every restart abandons the tool. This does *not* require the bar store or a database:
write to a **JSON file** (`data/manual_lines.json`), loaded at startup. Bar persistence
can stay deferred to M7; these two are unrelated decisions.
```json
{"id":"ml_01H...","tf":"4h","side":"resistance","anchor_t":1754600000,
"anchor_p":6412.5,"slope":-0.0000031,"created_at":1754700000,"note":"","hidden":false}
```
CRUD via `POST /api/lines`, `DELETE /api/lines/{id}`, `PATCH /api/lines/{id}`. On any
change the server recomputes levels and broadcasts `{"type":"levels",...}` — the
drawing client must render optimistically and then reconcile, not wait on the round
trip.
### 7.4 Multi-timeframe moving averages (`moving_averages.py`)
**Primary requirement: the daily MA set — SMA 10, 20, 50, 100, 200 — computed on daily
bars and displayed on the 1d, 30m, and 1m charts, with the base timeframe switchable.**
This is the headline use of the multi-timeframe projection: the 200DMA is a level
people trade against, and on a 1-minute chart it should appear as a near-horizontal
line that steps once per session. Do not compute a "200-period MA of 1-minute bars" for
the 1m chart — that is a completely different and far less useful line. **The MA's
period is always tied to the timeframe it was computed on, never to the chart being
displayed.**
```
MA_SETS = {
"1d": [("sma", 10), ("sma", 20), ("sma", 50), ("sma", 100), ("sma", 200)],
# optional, off by default:
"1h": [("ema", 9), ("ema", 21)],
}
BASE_TIMEFRAMES = ["1m", "30m", "1d"] # the switcher; others remain available
```
Config-driven, so adding a set is a config change, not code. Other timeframes' MAs stay
supported by the same machinery but ship disabled — the daily set is what matters.
**History check:** a 200DMA needs 200 sessions. Yahoo's 730-day 1h window yielded
17,387 bars ≈ 750 sessions, so all five daily MAs are warm from startup with roughly
3× margin. Verified, not assumed.
**Expect small disagreements with thinkorswim/TradingView on the daily MAs.** We build
daily bars on the CME session (18:00→17:00 ET, §6); other platforms sometimes anchor
differently or use settlement prices. A one- or two-point difference in the 200DMA is
this, not a bug. Keep the daily anchor configurable so it can be matched if it matters.
The key rendering decision: an MA from a higher timeframe drawn on a 1-minute chart is
a **step function**, held constant between higher-timeframe closes.
```
1h EMA21 rendered on a 1m chart:
┌────────
┌──┘ ← steps at each 1h close, NOT a smooth interpolation
┌──┘
```
Interpolating between higher-TF closes would draw a line that was never true at the
time it appears to have been true. Emit `points` as a stepped polyline and render with
LWC's `lineType: LightweightCharts.LineType.WithSteps`.
- The value from the **forming** higher-TF bar is emitted with `provisional=True` and
rendered dashed. It *will* move until that bar closes — that is honest, not a bug.
- An MA's **current value is a price level**, so it enters the confluence engine on
equal footing with trendlines, at `0.75 × timeframe_weight` (MAs are slightly less
reactive than drawn structure, but a 4h 200SMA is still a wall).
- **Warm-up:** an MA needs `period` closed bars on its timeframe. A daily 200SMA needs
200 sessions. Until warm, emit nothing — never emit a partially-warmed MA. Yahoo
seeding (§10) makes every MA up to a daily 200SMA warm from startup; without it,
the higher-timeframe MAs stay cold for months.
### 7.5 Confluence (`confluence.py`)
This is the actual product. Everything above exists to feed it.
```
1. Collect every active Level, evaluate price_at(now).
2. Split by side relative to current price (levels above → resistance, below → support).
A level's stored `side` is its structural nature; its *effective* side is
positional. Use positional — a broken resistance acting as support is the
interesting case, not an error.
3. Sort by price. Single-linkage cluster: extend the current cluster while the gap to
the next level is <= clusterTol.
clusterTol = 0.4 * ATR14(15m)
4. Cluster score = sum of member weights.
5. Emit clusters with >= 2 members OR score >= 8 (a lone 1d line matters by itself).
```
```
5m resistance 6404.25 weight 1
15m resistance 6405.00 weight 2
1h resistance 6403.75 weight 4
4h resistance 6404.50 weight 8
↓
RESISTANCE CLUSTER 6403.75 – 6405.00
CONFLUENCE 15
```
Recompute on every closed 1m bar, and on any level rebuild.
### 7.6 Alerts (`alerts.py`)
The failure mode to design against is notification fatigue. A per-line alert makes the
system useless within a day.
Per-cluster state machine, keyed on `(side, round(center / clusterTol))` so a cluster
keeps its identity as members drift:
```
ARMED ──price within alertTol of cluster──► FIRED ──► COOLDOWN ──┐
▲ │
└────── price moves > 2*alertTol away AND cooldown elapsed ◄────┘
```
- `alertTol = 0.5 * ATR14(15m)`
- `cooldown = 15 minutes`
- Minimum score threshold to fire: **configurable, starting value 6 — but this
certainly needs recalibrating in M4.** With the daily MA set as the primary levels,
each daily MA carries `0.75 × 16 = 12`, so *any two* of them near each other scores
24 and a threshold of 6 would fire constantly. Either raise the threshold well above
24, or damp the weight when several MAs from the same timeframe cluster (they are not
independent evidence the way a 4h line and a 1h line are). Decide this against a
replayed tape, not on paper. **Target: single-digit alerts per session.**
- Re-arming requires *both* the price separation and the cooldown. Time alone lets a
price oscillating on a level fire forever.
- Fire on **closed 1m bars only**, not intra-bar ticks.
Dispatch to: WebSocket (UI banner + sound) and ntfy. Payload:
```
BEARISH ZONE /ES 6404.25
Resistance confluence 15 @ 6403.75–6405.00
5m, 15m, 1h, 4h
```
---
## 8. API contract
Fix these shapes now; the frontend and backend are built against them.
### REST
| Route | Returns |
|---|---|
| `GET /api/status` | `{stream: "connected"\|"disconnected"\|"replay", symbol, last_bar_t, bars_held: {tf: n}, warm: {tf: bool}}` |
| `GET /api/bars?tf=5m&limit=500` | `{tf, bars: [{t,o,h,l,c,v,closed}]}` — oldest first |
| `GET /api/levels?tf=all` | `{levels: [Level]}` |
| `GET /api/confluence` | `{price, clusters: [Cluster]}` |
| `GET /api/health` | existing |
### WebSocket `/ws`
Client → server on connect:
```json
{"type": "subscribe", "tf": "5m"}
```
Server → client:
```json
{"type":"snapshot","tf":"5m","bars":[...],"levels":[...],"clusters":[...],"price":6404.25}
{"type":"bar","tf":"5m","bar":{"t":1754700000,"o":6403.5,"h":6405.0,"l":6403.0,"c":6404.25,"v":812,"closed":false}}
{"type":"levels","levels":[...]}
{"type":"clusters","price":6404.25,"clusters":[...]}
{"type":"alert","cluster":{...},"message":"..."}
{"type":"status","stream":"disconnected"}
```
Rules:
- Send `bar` on **every** update of the forming bar (that is the live chart) but batch
`levels` — they only change on higher-TF closes.
- Always send a full `snapshot` on connect and after any reconnect. The client must
never try to reconcile a gap.
- Levels are sent for **all** timeframes regardless of the displayed timeframe. That is
the entire point: a 4h line drawn through a 1m chart.
---
## 9. Frontend
`static/index.html` loads Vue 3 and Lightweight Charts as script tags — no bundler,
matching the existing app.
```html
<script src="https://unpkg.com/vue@3/dist/vue.global.prod.js"></script>
<script src="https://unpkg.com/lightweight-charts@5.2.0/dist/lightweight-charts.standalone.production.js"></script>
```
### Vue + Lightweight Charts integration — the one real gotcha
**Never put the chart or series objects in `ref()` or `reactive()`.** Vue's deep
reactive proxy will wrap the library's internal objects, which breaks identity checks
inside the library and destroys performance on every update. Use `shallowRef`, or
better, a plain module-scoped variable / `markRaw`.
```js
const chartApi = shallowRef(null); // ✅
// const chart = ref(null); // ❌ will appear to work, then misbehave
```
### Logical indices address the whole chart, not your bar array
**Never derive a viewport from `bars.length`.** A logical index addresses the
chart's *shared* time scale — the union of the time points of every series on
it — not the candle array. Any series whose points pre-date the candle window
prepends to that scale and shifts every logical index by its count.
```js
// ❌ off by however many points the other series contribute
timeScale().setVisibleLogicalRange({ from: bars.length - 160, to: bars.length + 5 });
// ✅ an instant cannot be renumbered by a later series
timeScale().setVisibleRange({ from: bars.at(-160).t, to: bars.at(-1).t + step * 5 });
```
This is not theoretical — see the 2026-08-10 entry in §16. The daily MAs carry
one point per daily bar (617 of them, back ~2 years). They are attached by
`syncVisibleLevels()` *immediately after* `setBars()`, so a logical range that
was correct when set silently slid 617 bars — about ten hours — into the past
one tick later. The chart looked frozen while the socket was perfectly healthy.
Structure:
- `chart.js` — a plain, framework-free wrapper class owning the LWC instance:
`create(el)`, `setBars()`, `updateBar()`, `syncLevels(levels)`, `destroy()`.
- `app.js` — Vue app owning state (timeframe, connection status, clusters, alert log)
and the WebSocket. Calls into the wrapper imperatively in `onMounted` / watchers.
- Chart lifecycle in `onMounted`; `chart.remove()` in `onUnmounted`; a `ResizeObserver`
driving `chart.applyOptions({width, height})`.
### v5 API usage
```js
const chart = LightweightCharts.createChart(el, {...});
const candles = chart.addSeries(LightweightCharts.CandlestickSeries, {...});
const line = chart.addSeries(LightweightCharts.LineSeries, {
lineType: LightweightCharts.LineType.WithSteps, // for MTF moving averages
});
candles.setData(bars); // once
candles.update(bar); // every tick — never re-call setData
```
### Rendering levels
`syncLevels()` must **diff by `Level.id`**, not clear-and-rebuild. Rebuilding every
series on each update causes visible flicker and leaks series objects.
- Trendline → a `LineSeries` with two points: `(anchor_t, anchor_p)` and
`(now + rightExtension, price_at(now + rightExtension))`. Extend ~20% of the visible
range into the future so the line is usable ahead of price.
- MA → a `LineSeries` fed the stepped `points` array.
- Colour **by timeframe** (one hue per TF, consistent everywhere including the
confluence panel). Line width scales with timeframe weight. `provisional` levels
render dashed via `lineStyle: LightweightCharts.LineStyle.Dashed`.
### 9.4 Layer panel (checkboxes)
Ships with M3. Visibility control is load-bearing here, not decoration — five daily MAs
plus manual lines plus any optional intraday MA sets stack up fast.
```
CHART [ 1m ] [ 30m ] [ 1d ] ← base timeframe switcher
LAYERS
─────────────────────────────────
☑ Daily MAs ██
☑ 10 ☑ 20 ☑ 50 ☑ 100 ☑ 200
─────────────────────────────────
☐ 1h MAs ▓▓
☐ EMA9 ☐ EMA21
─────────────────────────────────
☑ Manual lines ░░ (M5)
☐ Auto trendlines (M8)
─────────────────────────────────
☐ Hidden levels still count toward confluence
```
- **The base timeframe switcher changes only the candles.** Every level stays on screen
— a 200DMA is equally valid on a 1m chart. That is the entire premise of the product.
- **The group checkbox is a master toggle** — unchecking "Daily MAs" hides all five at
once; individual periods nest under it.
- The colour swatch beside each timeframe is that timeframe's hue, used identically on
the chart and in the confluence panel. One hue per timeframe, everywhere.
- **Persist to `localStorage`.** These settings are pure UI preference and must survive
reloads; they do not belong on the server.
**Visibility vs. scoring — decide this explicitly.** Hiding a level defaults to
*also* removing it from confluence scoring, because "I don't want to see this" almost
always means "I don't care about this." The last checkbox decouples the two for anyone
who wants a clean chart with full scoring.
That default has an architectural consequence: **confluence is computed server-side, so
the client's enabled set must reach the server.** Send it over the WebSocket on change:
```json
{"type":"prefs",
"base_tf":"1m",
"enabled":{"ma":{"1d":[10,20,50,100,200]},"manual":true,"auto":false},
"hidden_levels_score":false}
```
Store it per connection. Single-user app — no need for anything more elaborate.
### Panels
- **Drawing toolbar** (M5) — trendline tool, snap toggle, delete selection.
- **Confluence panel** — clusters sorted by `|distance|`, each showing member
timeframes, the zone range, and the score. This is the primary readout; give it more
visual weight than the price itself.
- **Status bar** — stream state, contract symbol, last bar age, which timeframes are
warm. When the stream drops, the chart must *say so*, not quietly show stale candles.
- **Alert log** — recent fires, most recent first.
- **Timeframe selector** — switches the candle series only. Levels stay.
Keep the existing dark/light CSS-variable scheme in `style.css`.
---
## 10. The history problem
**Mostly solved by the Yahoo source (§2.1) — but read the caveats.**
Schwab alone would leave the system knowing nothing before the moment it connects.
Since the highest-weighted timeframes need the most history, Schwab-only cold start
would mean waiting months for the parts of the product that matter most:
| Timeframe | Usable after, Schwab-only | With Yahoo seeding |
|---|---|---|
| 5m, 15m | ~2–4 hours | immediate |
| 30m, 1h | ~1–2 sessions | immediate |
| 1d | months | immediate (~500 sessions) |
| 1d 200SMA | ~10 months | immediate |
**Startup sequence:**
1. Seed from Yahoo: `interval=1h&range=730d`, plus `interval=1m&range=8d` for the
fine detail near the present.
2. Aggregate through `aggregator.py` into every timeframe, using our own session rules.
3. Attach the live source (Yahoo poll, or Schwab once keys exist) and continue forward.
4. Persist everything (M7) so subsequent restarts need less seeding.
**Caveats that must be honoured in code:**
- **Tag every bar with its `source`** (`"yahoo"` / `"schwab"` / `"replay"`), and expose
the seam in the UI. Silently blending delayed continuous data with live per-contract
data produces trendlines nobody else can see.
- **Roll gaps.** Yahoo `ES=F` is a continuous front-month series; quarterly contract
rolls leave price discontinuities that will read as a trendline break or generate a
bogus pivot. For M0–M5 this is acceptable. If it proves noisy, the fix is either
panama-adjusting the seeded series or dropping pivots within ±1 bar of a known roll
date (third Friday of Mar/Jun/Sep/Dec). Do not build roll adjustment pre-emptively.
- **The 1m request cap is 8 days.** Longer 1m ranges must be fetched in ≤8-day windows
and stitched. In practice you don't need deep 1m history — seed 1h and let the
aggregator do the rest.
- **Never mix Yahoo daily bars in** — see §2.1.
If per-contract accuracy ever matters more than convenience, **Databento** sells proper
CME history with real roll handling. Not needed now.
---
## 11. Configuration
All via env, read in `config.py`. Ship a `.env.example`; `.env` is already gitignored.
```
# --- data sources ---
LIVE_SOURCE=yahoo # yahoo | schwab | replay
SEED_SOURCE=yahoo # yahoo | none
YAHOO_SYMBOL=ES=F
YAHOO_POLL_SECONDS=20
SEED_1H_RANGE=730d
SEED_1M_RANGE=8d # 8d is Yahoo's hard per-request cap
# --- schwab (only needed once LIVE_SOURCE=schwab) ---
SCHWAB_API_KEY=
SCHWAB_APP_SECRET=
SCHWAB_CALLBACK_URL=https://127.0.0.1:8182
SCHWAB_TOKEN_PATH=./.schwab_token.json
SCHWAB_ACCOUNT_ID=
SCHWAB_SYMBOL=/ES
# --- timeframes & indicators ---
TIMEFRAMES=1m,2m,5m,15m,30m,1h,1d
BASE_TIMEFRAMES=1m,30m,1d # the chart switcher
MAX_BARS_PER_TF=5000 # in-memory ring buffer bound
# Daily MA set is the primary requirement; others ship disabled. See §7.4
MA_SETS__1D=sma10,sma20,sma50,sma100,sma200
MA_SETS__4H=
MA_SETS__1H=
DAILY_ANCHOR_ET=18:00 # CME session open; change to match another platform
MANUAL_LINES_PATH=./data/manual_lines.json
CONFLUENCE_MIN_SCORE=6 # MUST be recalibrated in M4 — see §7.6
ALERT_COOLDOWN_SECONDS=900
NTFY_TOPIC=
NTFY_SERVER=https://ntfy.sh
CHART_AUTH_TOKEN= # empty = auth disabled (local dev)
REPLAY_FILE= # set to replay a tape instead of connecting
```
**Token persistence:** `schwab-py`'s refresh token expires every **7 days** and
re-auth is an interactive browser flow. Locally, keep `.schwab_token.json` out of the
repo (add to `.gitignore`). On the VPS later, it **must** live on a Coolify persistent
volume or every rebuild logs you out. Same for the eventual SQLite file.
---
## 12. Testing
The market is closed most of the time you will be working. Build for that.
**Record/replay is a milestone-1 deliverable, not a nicety.** `recorder.py` writes
every raw stream message to JSONL with its arrival timestamp; replay feeds them back
through the identical code path, either at wall-clock speed or as fast as possible.
Everything downstream of `StreamService` is then testable, deterministically, offline.
Required tests:
- `session.py` — bucket boundaries. Both DST transitions, Sunday 18:00 open, the
17:00–18:00 break, and Friday close. **Write these first.**
- `aggregator.py` — 1m→all TFs on synthetic bars; gap handling; idempotent replay.
- `moving_averages.py` — a known daily series produces known 10/20/50/100/200 values;
assert nothing is emitted before warm-up; assert the stepped projection onto 1m holds
its value for a whole session and changes exactly at the session boundary.
- `manual_lines.py` — round-trip JSON persistence; a line drawn on 4h evaluates to the
same price on the 1m chart at the same instant.
- *(M8)* `pivots.py` — known fixtures; assert no repainting (a pivot, once emitted,
never changes when more bars arrive).
- *(M8)* `trendlines.py` — hand-built fixtures where the correct line is obvious;
assert violation rejection and dedup.
- `confluence.py` — synthetic levels producing a known cluster and score.
- `alerts.py` — assert no re-fire within cooldown, and that oscillation around a level
produces exactly one alert.
Add `pytest` to a `requirements-dev.txt`.
---
## 13. Milestones
Ordered so the user's stated priority — **live realtime charts first** — lands
earliest, and so nothing later is blocked on market hours.
**No API keys are required until M6.** M0–M5 run entirely on Yahoo.
### M0 — Yahoo source (no keys, no blockers)
`market/base.py` protocol + `market/yahoo.py`: `history()` over the verified chart
endpoint, and `stream()` as a polling loop presenting the same async-iterator
interface. Null-filtering, 8-day 1m windowing, bar `source` tagging. Recorder writes a
tape; `ReplaySource` reads it back.
**Done when:** a script prints seeded 1h bars back to 2024 and then live-ish 1m bars,
and a recorded tape replays identically.
### M1 — Live chart end to end ⭐ primary deliverable
`StreamService` → in-memory 1m store → WebSocket → Vue 3 + LWC candlestick chart
updating live. Status bar showing source + bar age. No analysis yet.
**Done when:** the browser shows a live-updating /ES 1-minute candle chart, and the
same chart can be reproduced offline from a tape.
### M2 — Aggregation + timeframe switching
`session.py` + `aggregator.py` with full test suite. Timeframe selector drives the
candle series. `GET /api/bars`.
**Done when:** switching to 15m shows correctly bucketed bars, and replaying a tape
twice yields identical output.
### M3 — Multi-timeframe moving averages ⭐ start here for analysis
MAs are the right first analysis layer: **fully deterministic, no parameters to tune,
no judgment calls**, and Yahoo seeding makes them warm from startup. They validate the
entire overlay concept — stepped rendering, level diffing by `id`, TF colour scheme —
without any of the ambiguity trendlines carry.
**Done when:** the 10/20/50/100/200 DMAs render on the 1m, 30m and 1d charts, stepping
once per session on the intraday views, dashed while the current session is unfinished.
### M3.5 — Layer panel
Checkbox tree to show/hide levels by timeframe and by individual MA (§9.4). Ships with
M3 because 6 timeframes × 4 MAs = 24 lines is unreadable without it.
**Done when:** unchecking a group removes its every level from the chart and the state
survives a reload.
### M4 — Confluence + alerts (on moving averages alone) ⭐ first genuinely useful build
`confluence.py`, `alerts.py`, confluence panel, ntfy push, in-browser sound — scored
over MA levels only. Multi-timeframe MA confluence is a real signal in its own right;
this is a complete, useful product with zero hand-drawn input and zero tuning.
**Done when:** a replayed tape produces a sane number of alerts (single digits per
session), each corresponding to a real multi-timeframe convergence.
### M5 — Manual trendlines
Two-click drawing, snapping, persistence, feeding the same confluence engine (§7.3a).
Hand-drawn lines are authoritative — full weight, no quality discount.
**Done when:** a line drawn on the 4h chart appears correctly projected on the 1m chart
and raises the confluence score of a cluster it lands in.
### M6 — Schwab live source (needs API keys)
Implement `market/schwab.py` against the existing `MarketDataSource` protocol and
answer every question in [§2.2](#22-verify-these-when-adding-the-schwab-source-m6-not-before).
Yahoo continues to handle seeding; Schwab takes over the live tail.
**Done when:** flipping `LIVE_SOURCE=yahoo` → `schwab` changes nothing visible except
lower latency and true per-contract prices. **If this milestone requires touching any
file outside `market/`, the abstraction in M0 was wrong — fix it there, not here.**
### M7 — Persistence
`SqliteBarStore` behind the existing `BarStore` protocol. Backfill-on-start from disk,
falling back to Yahoo seeding only for what's missing.
**Done when:** restarting the process loses no history.
### M8 — Automatic trendline detection (optional)
`pivots.py`, `trendlines.py` per §7.3. Deliberately last among the analysis work:
by this point the hand-drawn lines from M5 are **ground truth**, so the scoring
coefficients can be tuned to agree with lines you actually drew, rather than guessed at
in the abstract. Auto lines render in a distinct style and are individually
dismissable; they never silently replace a manual line.
**Done when:** on a replayed tape, auto-detected lines land where the manual ones were
drawn, and the layer panel can hide them independently.
### M9 — Bias panel
BULLISH / BEARISH toggle recording the user's directional call against the current
confluence state, persisted, with a journal view. **Records only — trades nothing.**
Because `LEVEL_ONE_FUTURES_OPTIONS` streams, this milestone can go further than the
original sketch: given a bias, resolve the 1-DTE strikes, subscribe to the two legs,
and display the **live spread price** — so the readout becomes actionable enough to
hand-execute in thinkorswim:
```
BEARISH /ES — resistance confluence 15 @ 6403.75–6405.00
Proposed: 1-DTE 6405/6415 put spread ~2.35 x 10 (live)
```
Strike/expiration symbol resolution for futures options is fiddly; treat it as its own
sub-task and verify the symbol format against `LEVEL_ONE_FUTURES_OPTIONS` empirically,
the same way M6 verifies `/ES`.
### M10 — VPS deploy (when wanted)
Shared-secret auth on, `workers=1`, persistent volume for token + DB, Coolify domain
port suffix preserved per README.
---
## 14. Why execution is out of scope
Note the asymmetry: Schwab **does** stream futures-options *quotes*
(`LEVEL_ONE_FUTURES_OPTIONS`), so we can price a spread live — we just cannot transmit
it. Read anything below as being about order entry only.
Schwab's Trader API exposes no futures or futures-options **order entry**. thinkScript
`AddOrder()` places *simulated* orders for backtesting only. Automating clicks in the
thinkorswim UI is the wrong reliability model for near-expiration leveraged
instruments — window focus, stale quotes, partial fills, and dialogs all fail silently,
and an execution path must be able to tell the program what the broker actually did.
The interim workflow is therefore: this app produces a decision, the human executes it
in thinkorswim.
Keep the seam clean. Broker-specific code lives only in `market/schwab.py`; nothing in
`analysis/`, `bars/`, or `api/` may import it. When an execution adapter is added —
Schwab, if they ever ship futures-options orders, or IBKR — it consumes `Cluster` and
the M9 bias signal and nothing else. This is the same discipline the M6 acceptance test
enforces for data sources.
---
## 15. Risk register
| Risk | Impact | Mitigation |
|---|---|---|
| Futures entitlement missing on the Schwab account | Delays M6 only | No longer blocks — M0–M5 run on Yahoo |
| No futures history from Schwab | Higher TFs cold for months | Solved: Yahoo seeding, §10 |
| Yahoo endpoint is unofficial — may rate-limit or change shape | Dev source breaks | Isolated in `market/yahoo.py`; cache seeds to disk (M7) so it's fetched rarely; back off on 429 |
| Yahoo daily bars anchored midnight ET, not session | Daily candles disagree with every other chart | Never use them — build 1d from 1h, §2.1 |
| Contract roll gaps fake a trendline break | Bad signals on seeded data | Store `symbol` + `source` per bar; surface rolls in UI; §10 |
| Source abstraction leaks Schwab/Yahoo specifics upward | M6 turns into a rewrite | M6 acceptance test: no file outside `market/` may change |
| `session.py` bucket math wrong | Silently wrong lines everywhere | Tests written first; both DST transitions |
| Repainting pivots | Lines that "were always there" | `w`-bar confirmation lag, enforced by test |
| Alert fatigue | Product becomes unusable | Cluster-level alerts, cooldown + separation re-arm |
| Multiple uvicorn workers | Duplicate Schwab connections | `workers=1`; streamer in `lifespan` |
| Schwab token expiry (7 days) | Stream dies | Surface prominently in status bar; document re-auth |
| Vue reactivity wrapping chart objects | Perf collapse, odd bugs | `shallowRef`/`markRaw` — §9 |
| LWC v4 tutorials copied | Code silently wrong for v5 | `addSeries(SeriesType, ...)` only |
| Viewport derived from `bars.length` | Chart looks frozen; feed is fine | Anchor the view by time, never by logical index — §9 |
| Seed replays every bar through `on_bar` | ~82 s startup; port refuses connections | Known, unfixed — §16, 2026-08-10 |
| Headless browser without a real locale | `Intl` throws; blank canvas mimics an app bug | Launch Chromium with `--lang=en-US` — §16 |
## 16. Session log
Dated record of problems hit and how they were resolved. Times are UTC; the
repo's commit timestamps are -0500.
### 2026-08-10 — rebuild, and a chart that looked frozen
**10:30 · The rebuild was genuinely required.** `schwab-py` had been added to
`requirements.txt`, but the running image was built at 2026-08-09 22:05, before
that line existed. The bind mount (`.:/app`) hides this: source edits appear
live, so the Schwab commits looked deployed while `pip freeze` in the container
showed no `schwab-py` at all. Anything imported rather than read from disk needs
`docker compose build`. Rebuilt to `schwab-py 1.5.1` and recreated the container.
**10:30–10:31 · Startup takes ~82 seconds, and the port is closed the whole
time.** `Runtime.start()` replays every seeded bar through `on_bar`, and each
daily-bar update re-runs `rebuild_levels()` → `broadcast_level_delta()`, which
serialises and diffs five MA levels carrying ~730 points each. With a 730d/1h
seed plus an 8d/1m seed that is quadratic work before uvicorn binds. Measured:
10:30:24 "Waiting for application startup" → 10:31:46 "Application startup
complete". An open browser tab polling `/api/status` throughout logs a wall of
`ERR_CONNECTION_REFUSED`; that is the restart window, not a fault.
*Unfixed.* The fix is to bulk-load seeded bars and rebuild levels once at the
end, rather than once per bar. Related: M7 persistence would cut the seed itself.
**Diagnosing a hang that is actually slowness:** `docker stats` reported ~0.1%
CPU while the process was in fact grinding, so it pointed the wrong way. What
worked was `faulthandler.dump_traceback_later(25, exit=True)`, which named the
exact frame (`indicators.py:sma` under `runtime.py:62`). `py-spy` is unusable
here — it needs `SYS_PTRACE`, which the container does not have.
**Do not write scratch files into the repo while diagnosing.** A `_probe.py`
dropped in the project root is inside the bind mount, so `--reload` restarted
the lifespan and reset the 82-second clock — twice — which is what made
slow startup look like an infinite hang. Pipe throwaway scripts over stdin
(`docker exec -i … python -`) instead. Only `.py` changes trigger the reloader;
writing screenshots into `artifacts/` is safe.
**10:35 · A blank chart canvas that was not a bug.** The Playwright container
has no usable locale, so Chromium reports `en-US@posix`; Lightweight Charts
formats its time axis through `Intl`, which throws `Invalid language tag` and
leaves the canvas empty. `docker-compose.yml` already sets
`LANG=en_US.UTF-8` for that service and it is *not* sufficient. Launch with
`chromium.launch({ args: ['--lang=en-US'] })` — with that, the page renders and
reports zero console errors. Worth stating plainly: this failure looks exactly
like a broken app, and it is not.
**10:41–10:50 · The real bug — the chart sat ~10 hours behind a healthy feed.**
Symptom: header price live at 7785.00 while the last candle closed 7772.75, and
the series appeared to end at 00:20. Everything downstream checked out —
`/api/bars` newest 10:39 from `schwab`; `store.put` keeps bars strictly
ascending; the WebSocket snapshot delivered 1000 ascending bars ending 10:42 and
live `bar` events arrived every minute; the browser received all of it.
Interrogating `window.__chart` gave the answer:
```
seriesLen 1000 seriesLast 08-10T10:48 (7786.25) ← data complete
visible 08-07T20:41 → 08-10T00:35 ← viewport wrong
logical from 840 to 1005
```
The series was complete; the *viewport* was 617 bars too far left — exactly
`bars_held.1d`. `setBars()` set a visible **logical** range from the candle
array length, then `syncVisibleLevels()` attached the daily MA series, whose 617
daily points pre-date the 1m window; prepending them renumbered every logical
index and dragged the view off the live edge. Fixed in `static/chart.js` by
anchoring the viewport to a **time** range. Verified in a real browser: visible
range 08:02 → 10:49, last candle 7786.25 matching the header. See §9.
**Method note.** Three checks in a row said "healthy" — the REST API, the
WebSocket, and the frontend source all looked correct in isolation, because each
of them *was* correct. Only querying the live page's own chart object separated
"the data is missing" from "the data is off-screen". Screenshots alone were
actively misleading here: the stale time axis was read as a session gap.
### 2026-08-10 (later) — real-time ticks, and what to do about cold restarts
**The chart now moves between minute closes.** `CHART_FUTURES` emits a bar only
once its minute is over, so the chart stepped once a minute and sat still in
between — read, reasonably, as a dead feed. `LEVEL_ONE_FUTURES` carries real
trades on the same socket (`delayed: False`, verified on this account back in
M6), and it was never subscribed. It is now, and it builds a forming bar for the
current minute which the authoritative `CHART_FUTURES` bar then supersedes.
Three constraints shaped it, each of which would have caused a real bug:
- **Tick bars must never reach the aggregator.** It accumulates with
`current.v += incoming.v`, so re-sending the same forming minute would add its
volume into every higher timeframe on every update. `Runtime.on_bar` returns
early for `not bar.closed`: store the bar, set the price, broadcast, stop.
- **Ticks are throttled** (`SCHWAB_TICK_SECONDS`, default 1.0). /ES trades many
times a second and each emission costs a store write plus a broadcast to every
open socket. Setting it negative drops the Level 1 subscription entirely and
returns the source to closed bars only.
- **A tick for a minute already closed is dropped**, or a late trade would
overwrite a settled exchange bar with a partial one.
Alerts deliberately stay on closed bars. A level is judged on a settled bar, not
on a price that may not last the minute — and `on_bar` already gated on
`closed`, so this needed no change. Intra-bar alerting is a separate decision.
Bid-only Level 1 updates are skipped rather than carried forward: a bid is not a
trade and must not extend a candle's high or low. Verified live — 15 forming
bars and 2 closed bars in 100 seconds, and in a browser the candle's high and low
visibly extend within the minute.
**Cold restarts — the options, and a recommendation.** Every restart costs ~82
seconds of refused connections, re-seeds from Yahoo, and starts with empty alert
cooldowns, so a deploy can re-alert whatever price is sitting on.
1. *Make seeding non-quadratic.* Seeding replays every bar through `on_bar`, and
each daily-bar update rebuilds all five MA levels and diffs them. Bulk-load
the seeded bars and rebuild levels once at the end. Contained, testable, and
removes most of the 82 seconds. **Do this first** — it is the cheapest real
win and needs no new storage.
2. *Persist bars (M7, SQLite).* Restarts then seed only the gap. Removes the
Yahoo dependency from the startup path and shrinks the window further. This
is the durable answer, and the plan already scopes it.
3. *Persist alert cooldowns and armed state.* Independent of 1 and 2, and the
part that actually misbehaves rather than merely being slow: without it every
deploy re-alerts. Small table, big behavioural win.
4. *Serve before seeding finishes.* Start uvicorn immediately and seed in a
background task, so the port never refuses. The chart would open cold and
fill in, which is better than an unreachable page — but it changes what
"warm" means to every consumer of `/api/status`, so it wants its own thought.
Recommended order: 1, then 3, then 2. 4 only if the window still bites after 1.
**Stale bar events across a timeframe switch.** `Cannot update oldest data`
appeared in the console once ticks were live. Switching timeframe races: the
server answers `subscribe` with a fresh snapshot from one coroutine while
another is still draining bar events for the timeframe just left, so a 1m bar
can land after the 1h snapshot. Applied to the 1h series it is older than every
point in it, and Lightweight Charts throws rather than ignoring it — taking the
app down instead of dropping one bar. The race predates the tick feed; Level 1
made bar events ~15x more frequent, which is what surfaced it.
Guarded at both ends. `app.js` honours the `tf` the event already carries and
drops anything for a timeframe that is no longer selected. `chart.js` refuses a
bar older than the series' last point regardless of where it came from — a bar
behind the last one has nothing to contribute. Verified: 36 rapid timeframe
switches under a live tick feed produce zero errors, and calling
`candles.update()` directly with a stale bar still throws while the guarded
`updateBar()` does not.
**Same-price trades were being dropped.** The candle still paused for 10–20
seconds at a time after Level 1 went in. Instrumenting the raw stream settled
it: 87 messages in 90 seconds, only 33 carrying `LAST_PRICE`. Most of the rest
are pure bid/ask movement and correctly ignored — but a seventh of them look
like this:
```
['ASK_SIZE','ASK_TIME_MILLIS','BID_SIZE','BID_TIME_MILLIS',
'LAST_SIZE','QUOTE_TIME_MILLIS','TOTAL_VOLUME','TRADE_TIME_MILLIS','key']
```
Trade time, trade size, cumulative volume — and no `LAST_PRICE`, because Level 1
sends only *changed* fields and the trade printed at the price of the one
before. Requiring `LAST_PRICE` threw those away along with their volume.
`parse_level_one` now treats size-plus-trade-time as a trade and returns a null
price for the caller to carry forward. Measured on the live feed: median gap
3.1s → 2.0s, worst 21.5s → 8.1s, and bar volume climbs within the minute instead
of standing still.
Worth recording for the next person who reads a gap as a bug: the remaining
pauses are the market, not the pipe. In thin pre-open tape /ES genuinely goes
seconds without a price-changing trade, and then moves several ticks at once —
which is what a "gap up" after a quiet spell actually is.
**The time axis reads local, the data stays UTC.** Lightweight Charts is
timezone-agnostic: it reads epoch seconds as UTC and labels them as UTC, which
is why the axis disagreed with the wall clock. Fixed with `tickMarkFormatter`
for the axis and `localization.timeFormatter` for the crosshair, both going
through the browser's own zone.
Deliberately *not* fixed by shifting the bar timestamps, which is the other
common recipe. Every time in this codebase is epoch UTC by convention, and the
chart's own times feed trendline anchors, `indexAt`, hit testing and the values
posted back for manual lines — an offset applied to the data would put all of
them out by the offset, which is exactly the class of bug that once priced a
trendline 147 points away.
One limit worth knowing: tick *placement* is still computed on UTC days, so the
day-change divider sits at 00:00 UTC rather than local midnight, labelled with
the local date. The labels are right; the divider is in the UTC place.
### 2026-08-10 (afternoon) — update rate, the left scale, and volume
**Schwab conflates Level 1 to one update per second.** Chasing "still slow in
market hours" ended at a hard ceiling rather than a bug. In regular hours the
gaps between updates are whole multiples of 1.005s — 2.01, 3.02, 4.03 — which
only happens if the source emits on a one-second cadence and some seconds carry
no trade. `SCHWAB_TICK_SECONDS` was the limiter at 1.0 and is now 0.25, where it
no longer binds. **One update per second is the source's ceiling.** Anything
faster would mean inventing prices between trades, which a chart must not do.
Two real losses were found on the way and fixed:
- Higher timeframes only moved once a minute, because tick bars are 1m and the
socket filters by subscriber timeframe. `Runtime.provisional_higher` now
combines the aggregator's committed state with the live minute — without
mutating it, since the aggregator accumulates volume and would double count.
- Trades carrying only a trade stamp and a moved `TOTAL_VOLUME` — no
`LAST_PRICE`, no `LAST_SIZE` — were skipped. 66 → 74 updates per 90s.
**Daily context moved to the left price scale.** The right had prior-day levels,
session VWAP and five daily MAs competing with the live price and hand-drawn
intraday levels. The trap: a price scale takes its range from the series on it,
so moving levels across draws them against a different range and puts them at
the wrong height. A transparent candlestick mirror on the left scale feeds it
exactly the right scale's input; verified as a zero-pixel delta between the two.
Hand-drawn levels stay right, which is the space being cleared. `priceScaleId`
is fixed at series creation, so it is passed at construction and kept out of the
options reapplied afterwards.
**Volume is finally drawn.** It travelled the entire pipeline — parsed from both
Schwab services, aggregated, stored, broadcast in every bar — and nothing
rendered it. Now an overlay histogram on its own hidden scale in the bottom
fifth. An overlay rather than a pane, and emphatically not the price scale:
volumes are five figures against four-figure prices, and sharing a scale would
flatten the candles to a line.
**Sidebar vertical space.** Three cuts, all in §9.4's layer panel. The daily MA
periods sit on one line — `flex-wrap:nowrap` with tighter gaps and 12px boxes,
where 10px gaps and 22px indent had pushed 200 onto a line of its own.
Auto trendlines joins Manual lines as a parenthetical `(auto)` rather than
owning a row, which suits a control that is disabled until M8. The alert log
becomes a `<details>` like Confluence zones, closed by default with its count in
the summary — collapsed by default is the point, since leaving it open would
save nothing, and the count means activity is still visible while closed.
**Sidebar vertical space.** The right column was taller than the viewport with
nothing selected. Five changes, no functionality removed:
- The five daily MA periods fit one line (`flex-wrap:nowrap`, tighter gaps, 12px
boxes); 10px gaps and a 22px indent had pushed 200 onto a row of its own.
- Auto trendlines becomes a parenthetical `(auto)` on the Manual lines row
rather than owning one, which suits a control disabled until M8.
- Alert log becomes a `<details>` like Confluence zones, closed by default with
its count in the summary so activity still shows while shut.
- Tools becomes a `<details>` too, open by default, and each tool's panel is
bound to `armedTool` — only the armed tool shows its label, colour, width and
side controls. `armTool` already toggles and permits one armed tool at a time,
so the panels follow it exactly.
- Order is Layers, Tools, then the rest, with Layers collapsed by default.
Measured with nothing armed: 1110px of content down to 900px, which is inside
the viewport rather than past it. `.sidebar-section:first-of-type` carries the
zeroed top margin so reordering cannot reintroduce a gap at the top.
### 2026-08-10 (evening) — chart comments, and Drawings
**Comments are drawings, not levels.** A comment is stored as a `ManualLine`
with `kind="comment"`, so it inherits persistence, the shared drawing-number
sequence, the sidebar list, filtering and deletion without a parallel set of
endpoints. The one rule that must never bend: `ManualLineStore.levels()` filters
comments out. A comment reaching the level list would join a confluence cluster
and push a phone notification about a piece of text. It is also created with
`armed=False`, and `PATCH /lines/{id}` returns `to_dict()` rather than
`to_level()` for one, so no caller is ever handed a level-shaped comment.
`kind` is derived when absent — zero slope was always a typed level, anything
else a drawn trendline — so drawings saved before comments existed keep working.
**Pinned or floating.** Pinned comments carry `anchor_t`/`anchor_p` and move with
the chart; floating ones carry `x`/`y` as fractions of the pane, hold their place
through any zoom, and can be dragged. Comments render as DOM rather than canvas:
they hold arbitrary text, collapse to a numbered dot, and a floating one has to
ignore the time scale entirely. A pinned comment scrolled out of view parks on
the edge it left, pointing back the way it went, so it is never simply lost.
**"Lines & levels" becomes "Drawings"**, filtered by type and by text — the text
match covers the label, the kind and the `#number`, so `comment`, `cpi` and `7`
all narrow the list. Delete acts on what the filter shows, which is what makes
deleting by type or by string a single button.
One CSS trap worth recording: `.trendline-row span { grid-column:2 }` captured
the comment row's icon span and dragged it into the text column. Scoped to
`span:not(.drawing-icon)`.
**A comment lost its place when the timeframe changed.** Placed on a 30m bar,
then switched to 15m, it slid to the far left. `timeToCoordinate` answers only
for times that are data points on the current series, so a 30m bucket start
returned `null` on another timeframe — and `null` was being read as "off the
left edge". Anchors are now resolved to the bar that *contains* them, which is
timeframe-independent: an 09:30 note sits on the 09:30 bar at 15m and on the
09:00 bar at 1h. `setBars` also re-renders comments, since a timeframe switch
replaces the grid underneath every pinned one.
Verified across 30m → 15m → 1h → 30m: the anchor stays 08:30 throughout,
resolving to the 08:30 bar on 15m and the 08:00 bar on 1h, never edge-parked,
and returning to its original x on the way back. Edge-parking still works where
it should — a comment scrolled 400 bars out parks right and comes back on
return to live.
**The trendline Side control became inert.** Once snapping always lands on a
bar extreme, the side is inferred from *which* extreme — a high is resistance, a
low is support — so the dropdown could no longer affect anything. It now appears
only when "Snap to highs/lows" is off, which is the one case where there is no
extreme to infer from; otherwise the row reads "Side auto". Verified both ways:
snap on shows the note and no dropdown, snap off shows the dropdown.
`created_at` (epoch seconds) is already stored on every drawing and returned by
`GET /api/drawings`, so filtering by age needs UI only, not a migration.
**Trendline placement, third pass — and a regression I shipped.** Making a
pending anchor always win (previous entry) fixed the twitch case and broke the
opposite one: a genuine press-drag begun after an abandoned click was hijacked
by that stale anchor, so the line started far from the drag. That reached
production. The rule is now a single threshold — 12px of travel between press
and release makes it a drag, which is wide enough to survive a twitch on a
deliberate click and unambiguous for a real drag. A drag clears any half-placed
anchor rather than silently adopting it.
**The crosshair was lying about the anchor.** Lightweight Charts defaults to
`CrosshairMode.Magnet`, which snaps the crosshair to the bar's *close*. Hovering
by a bar's low therefore drew the crosshair mid-bar, and a correctly-snapped
anchor looked wrong — measured: aiming 4px above a bar low placed the anchor at
the low (7773) and not the close (7773.25), while the crosshair sat at the
close. Arming a tool now switches the crosshair to `Normal`, and a snap dot
marks the exact point the anchor will use, coloured by the side it implies.
Four gesture paths are verified in a browser: two clicks with a twitch on the
second, an abandoned click followed by a real drag, a plain press-drag, and
hovering. All start where they should and land on a bar extreme.
Worth recording for diagnosis: a reported "line ended up high off the bar"
turned out to render exactly on its bar — zero pixels off at 1h, 30m and 15m —
because the anchor had snapped to the *drawn* timeframe extreme (the 09:00 1h
low, 7744.25) while being checked against 1m bars, where it matches neither
extreme. Always compare an anchor against the timeframe it was drawn on.
Reference in a new issue View git blame Copy permalink