From 8e50d5cbc2ecead10aca18b2992e67bb03a93422 Mon Sep 17 00:00:00 2001 From: Chris Amow Date: Sun, 9 Aug 2026 20:31:40 -0500 Subject: [PATCH] Add implementation plan for /ES multi-timeframe confluence chart Planning-only commit: no application code yet. The plan specifies a realtime /ES chart that derives moving averages and trendlines across multiple timeframes, projects them onto one chart in a shared (time, price) plane, and alerts when levels from different timeframes converge. Key findings that shaped it, all verified against source rather than assumed: - Schwab streams realtime futures fine (CHART_FUTURES, LEVEL_ONE_FUTURES) but provides no futures price *history* at all. An account does not change this; it is an API-surface limit. - Yahoo's chart endpoint needs no key and has exactly what Schwab lacks: ~730d of hourly ES=F (~750 sessions), enough to warm a 200DMA from startup. So it serves as both the no-keys dev source and the history seeder, behind one MarketDataSource protocol. - Yahoo anchors daily bars to midnight ET while the CME session runs 18:00-17:00 ET, so daily bars are built from hourly using our own session rules instead. - Lightweight Charts v5 replaced addCandlestickSeries() with addSeries(CandlestickSeries, ...); most tutorials online are v4. Build order defers judgment-heavy work: moving averages first (fully deterministic), then confluence scoring, then hand-drawn trendlines. Automatic trendline detection comes last, tuned against the hand-drawn lines as ground truth. Includes a real trimmed Yahoo response as a test fixture; it contains a null in the OHLC arrays, which is the parsing case that needs handling. Co-Authored-By: Claude Opus 5 --- README.md | 5 + docs/IMPLEMENTATION_PLAN.md | 1145 +++++++++++++++++++++++++++++++ tests/fixtures/yahoo_es_1h.json | 368 ++++++++++ 3 files changed, 1518 insertions(+) create mode 100644 docs/IMPLEMENTATION_PLAN.md create mode 100644 tests/fixtures/yahoo_es_1h.json diff --git a/README.md b/README.md index 09c7e1c..80cbb5d 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,11 @@ FastAPI backend + Vue 3 (from CDN, no build step) served at Currently a placeholder: the frontend calls `/api/hello` and prints the JSON. +**Where this is going:** a realtime `/ES` chart that derives trendlines and moving +averages across many timeframes and alerts when they converge. The full spec lives in +[`docs/IMPLEMENTATION_PLAN.md`](docs/IMPLEMENTATION_PLAN.md) — read it before writing +code; it records decisions and verified API facts that are expensive to rediscover. + ## Local development ```bash diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md new file mode 100644 index 0000000..d5e493a --- /dev/null +++ b/docs/IMPLEMENTATION_PLAN.md @@ -0,0 +1,1145 @@ +# /ES Multi-Timeframe Confluence Chart — Implementation Plan + +**Audience:** the implementing agent. This document is the spec; it is written to be +executed top-to-bottom without re-deriving decisions. + +**One-line goal:** stream `/ES` 1-minute bars from Schwab, aggregate them into every +larger timeframe locally, derive trendlines and moving averages on each timeframe, +project them all onto one chart in a shared `(time, price)` coordinate system, and +alert when independently-derived levels from different timeframes converge. + +**Explicitly out of scope:** order execution. Nothing in this codebase places a trade. +See [§14](#14-why-execution-is-out-of-scope) for why, and for the seam left behind. + +--- + +## 0. Start here + +**Read §1, §2.1, §6, and §13 before writing anything.** The rest can be read as you +reach each milestone. + +**Work on a branch — do not push to `main`.** `main` is wired to a Forgejo webhook that +triggers a Coolify production deploy at . Pushing to main ships +whatever you wrote. Branch: `feat/chart-engine`. + +**Build order is M0 → M1 → M2 → M3 → M3.5 → M4 → M5.** Stop after M5 and get feedback; +M6+ are separately scoped. No API keys are required for any of M0–M5. + +**Existing repo state:** a placeholder FastAPI + Vue 3 (CDN, no build step) app. +`main.py` serves `static/index.html` and two toy `/api` endpoints. The serving and +deploy wiring is correct and should not be redesigned — extend it. The toy `/api/hello` +endpoint and its frontend button can be deleted. + +### Dependencies to add + +`requirements.txt` currently contains only `fastapi` and `uvicorn[standard]`. Add: + +``` +httpx # Yahoo fetches; async, already a FastAPI-adjacent standard +pydantic-settings # config.py +``` + +`requirements-dev.txt`: + +``` +pytest +pytest-asyncio +``` + +Do **not** add `schwab-py` until M6 — it is unused before then. Do **not** add +`yfinance`; the Yahoo chart endpoint is a plain HTTP GET and the extra dependency buys +nothing (verified — see §2.1). + +### Test fixture already provided + +`tests/fixtures/yahoo_es_1h.json` is a **real, trimmed** Yahoo response for +`ES=F&interval=1h` (40 bars). Use it to unit-test the parser offline. It deliberately +**contains a `null` in the OHLC arrays**, which is exactly the case §2.1 warns about — +if your parser doesn't filter those, that fixture will catch it. + +### Conventions + +- All times are **epoch seconds, UTC**, everywhere. No naive datetimes. +- Nothing outside `market/` may know which data source is in use. +- Nothing outside `market/schwab.py` may import broker-specific code. +- Analysis code takes lists of bars and returns values — no I/O, no clocks. This is + what makes the replay harness work. + +--- + +## 1. Decisions already made — do not relitigate + +| Decision | Choice | Why | +|---|---|---| +| Backend | FastAPI (already scaffolded) | Repo already runs it; native WebSocket support | +| Frontend | Vue 3 from CDN, **no build step** | Matches existing `static/` setup; keeps deploy trivial | +| Charting | TradingView Lightweight Charts **v5.2.0**, standalone build | Apache-2.0, canvas, built for incremental realtime updates | +| Data source | **Pluggable `MarketDataSource`.** Yahoo first, Schwab later | Yahoo needs no API key *and* has the history Schwab lacks — see §2.1 | +| Persistence | **In-memory first**, behind a `BarStore` interface | User confirmed deferring persistence is fine for v1 | +| Eventual persistence | **SQLite**, not Postgres | Single file, zero Coolify stack expansion. Revisit only if multi-process | +| Deployment | **Local first**, VPS later | Keep all config in env vars so the VPS move is config, not rewrite | +| Alerts | ntfy/Pushover phone push (+ free in-browser sound) | User selected phone push | +| Auth | Hardcoded shared secret from env | User confirmed; only matters once VPS-exposed | + +| Trendlines | **Manual (hand-drawn) first.** Auto-detection deferred to M8 | Hand-drawn lines are correct by definition, so they validate the confluence engine with zero tuning risk — and later become the ground truth the auto-detector is tuned against | +| Moving averages | **Daily set (10/20/50/100/200 SMA) is the priority**, shown on 1m / 30m / 1d | The 200DMA is a level people actually trade against | + +### Timeframe roles + +``` +1m base chart + alert evaluation weight 1 ← a switchable base timeframe +2m display only +5m manual lines weight 1 +15m manual lines weight 2 +30m base chart + manual lines weight 3 ← a switchable base timeframe +1h manual lines (+ optional MAs) weight 4 +4h manual lines (+ optional MAs) weight 8 +1d base chart + THE DAILY MAs weight 16 ← a switchable base timeframe +``` + +Base timeframe controls the candles only. **Every level stays visible on every base +timeframe** — the 200DMA on a 1-minute chart is the point, not a side effect. + +--- + +## 2.1 Data sources — build against Yahoo, swap in Schwab later + +**Do not block on Schwab API keys.** The two sources are complementary, and the +Yahoo one is strictly easier to develop against: + +| | Yahoo `ES=F` | Schwab `/ES` | +|---|---|---| +| Auth | none — plain HTTP GET | OAuth, keys, 7-day token refresh | +| Realtime | polled, possibly ~10 min delayed | true push websocket, realtime | +| 1m history | **8 days** (per-request cap) | ❌ none | +| 1h history | **~730 days** (17,387 bars, verified back to 2024-03) | ❌ none | +| 1d history | **~10 years** (2,517 bars, verified back to 2016) | ❌ none | +| Contract | continuous front-month, roll gaps | true contract | + +Endpoint, verified working with no key and no `yfinance` dependency: + +``` +https://query1.finance.yahoo.com/v8/finance/chart/ES=F?interval=1h&range=730d +``` + +Requires a browser `User-Agent` header. Returns `chart.result[0]` with `timestamp[]` +and `indicators.quote[0].{open,high,low,close,volume}` as parallel arrays. +**Those arrays contain `null` holes — filter them before constructing `Bar`s.** + +Therefore define one protocol and two implementations: + +```python +class MarketDataSource(Protocol): + name: str + def supports_history(self) -> bool: ... + async def history(self, symbol, tf, start, end) -> list[Bar]: ... + def supports_stream(self) -> bool: ... + async def stream(self, symbol) -> AsyncIterator[Bar]: ... # yields 1m bars +``` + +- `YahooSource` — real `history()`. `stream()` is a **polling loop** (every 15–30 s, + `interval=1m&range=1d`, emit only bars newer than the last emitted) that presents the + same async-iterator interface as a real push stream. +- `SchwabSource` — `supports_history() -> False`. `stream()` is the true + `CHART_FUTURES` websocket. +- `ReplaySource` — reads a JSONL tape. Used by tests and offline development. + +Nothing downstream of these may know which source it is using. Selection is one env +var. **In production both run at once:** Yahoo seeds history at startup, Schwab +provides the live tail. + +### Do not use Yahoo's daily bars + +Yahoo anchors `ES=F` daily bars to **midnight ET**, but the CME futures session runs +**18:00 → 17:00 ET** (§6). Mixing the two definitions yields daily candles that +disagree with every other chart you'll compare against. + +**Build daily bars yourself by aggregating Yahoo's 1h bars** through the same +`aggregator.py` used for live data — one session definition everywhere. Yahoo's 1h bars +are anchored to the top of the ET hour, so they compose into session-anchored 4h and 1d +buckets cleanly. The ~730-day 1h window yields ~500 sessions: enough for a daily 200SMA +(~200 sessions) with room to spare. + +Yahoo's native 1d bars may be used *only* for multi-year context, clearly labelled. + +## 2.2 Verify these when adding the Schwab source (M6, not before) + +These are load-bearing unknowns for Schwab specifically. They no longer block the +project — M0–M5 run entirely on Yahoo. If any fails, stop and report. + +1. **Futures market-data entitlement.** It is not publicly documented whether + `CHART_FUTURES` requires futures trading approval or a CME non-professional market + data agreement on the Schwab account. Verify empirically in M6. +2. **Symbol format.** `schwab-py` docs show both `/ES` (continuous front-month) and + `/ESZ25` (specific contract). Determine which the stream actually accepts, and + whether the continuous form auto-rolls. Record the answer in the README. +3. **Bar cadence and lateness.** Confirm `CHART_FUTURES` emits one message per symbol + per minute, whether it re-sends a bar (correction), and how bars behave across the + 17:00–18:00 ET settlement break. +4. **Volume semantics.** Confirm `VOLUME` is per-minute, not cumulative-for-session. + +### Confirmed facts (already verified — do not re-research) + +**Realtime futures data: fully available.** Do not let §10 below suggest otherwise — +these are different axes and conflating them will send you down the wrong path. + +| Service | Available? | Use here | +|---|---|---| +| `CHART_FUTURES` (1-min OHLCV, `/ES`) | ✅ streaming | The base feed. Everything derives from it | +| `LEVEL_ONE_FUTURES` (live quotes) | ✅ streaming | Current price/bid/ask for the status bar and alert evaluation | +| `LEVEL_ONE_FUTURES_OPTIONS` | ✅ streaming | Live pricing of a proposed /ES put spread in M7 | +| REST `get_price_history` for futures | ❌ **not available** | — see §10 | +| `CHART_HISTORY_FUTURES` service | ❌ does not exist in `schwab-py` | — | +| Futures / futures-options **order entry** | ❌ not offered | see §14 | + +- `ChartFuturesFields`: `SYMBOL=0, CHART_TIME_MILLIS=1, OPEN_PRICE=2, HIGH_PRICE=3, + LOW_PRICE=4, CLOSE_PRICE=5, VOLUME=6`. +- **The gap is historical only, not realtime.** schwab-py's docs, verbatim: *"Schwab + provides price history for equities and ETFs. It does not provide price history for + options, futures, or any other instruments."* A Schwab account does not change this — + it is an API-surface limitation, not an entitlement one. Live `/ES` streams fine; + there is simply no way to ask for *yesterday's* `/ES` bars. This is the single + biggest constraint in the project; see [§10](#10-the-history-problem). +- Lightweight Charts 5.2.0 standalone build exposes a `window.LightweightCharts` + global containing `createChart`, `CandlestickSeries`, `LineSeries`, + `createSeriesMarkers`, `LineStyle`. CDN: + `https://unpkg.com/lightweight-charts@5.2.0/dist/lightweight-charts.standalone.production.js` + (~196 KB). +- **v5 changed the series API.** Use `chart.addSeries(LightweightCharts.CandlestickSeries, opts)`. + The v4 `chart.addCandlestickSeries(opts)` form **does not exist in v5** — most + tutorials online are v4 and will not work. + +--- + +## 3. Architecture + +``` + YahooSource SchwabSource ReplaySource + (history + poll) (live websocket) (JSONL tape) + └──────────────────────┼──────────────────────┘ + │ MarketDataSource protocol + ▼ 1-minute OHLCV + ┌─────────────────┐ + │ StreamService │ single asyncio task, ONE connection + │ (+ recorder) │──────► raw JSONL tape (for replay) + └────────┬────────┘ + │ Bar(1m) + ▼ + ┌─────────────────┐ + │ Aggregator │ session-aware bucketing + └────────┬────────┘ + │ Bar(tf) closed / updated + ┌──────────────┼──────────────┐ + ▼ ▼ ▼ + ┌──────────┐ ┌────────────┐ ┌──────────┐ + │ BarStore │ │ Pivots → │ │ Moving │ + │ (memory) │ │ Trendlines │ │ Averages │ + └──────────┘ └─────┬──────┘ └────┬─────┘ + │ │ + └──────┬───────┘ + ▼ Level[] (unified type) + ┌──────────────────┐ + │ ConfluenceEngine │ cluster + score + └────────┬─────────┘ + ▼ Cluster[] + ┌──────────────────┐ + │ AlertEngine │ state machine + cooldown + └────────┬─────────┘ + │ + ┌──────────────┴───────────────┐ + ▼ ▼ + FastAPI WebSocket ntfy push + │ + ▼ + Vue 3 + Lightweight Charts +``` + +### Critical process constraint + +The Schwab stream is **one connection, one process, not thread-safe**. Therefore: + +- Run the streamer as a single `asyncio` task owned by FastAPI's `lifespan`. +- **`uvicorn --workers 1` always.** More than one worker means more than one Schwab + connection, which will fight over the session. +- `--reload` in dev will tear down and re-establish the stream on every file save. + That is acceptable locally but expect reconnect churn. +- If the VPS deploy later needs multiple web workers, the streamer must be split into + its own process with a message bus. Do not design for that now, but keep + `StreamService` free of any FastAPI imports so the split stays cheap. + +--- + +## 4. Module layout + +``` +main.py FastAPI app: lifespan, route mounting (exists, extend) +app/ + config.py Settings from env (pydantic-settings) + auth.py Shared-secret gate (no-op when unset) + market/ + base.py MarketDataSource protocol, Bar emission contract + yahoo.py YahooSource: history() + polled stream() ← build first + schwab.py SchwabSource: easy_client, CHART_FUTURES websocket + replay.py ReplaySource: JSONL tape + stream.py StreamService: owns a source, reconnect, emit Bar + recorder.py Record raw messages to JSONL + bars/ + models.py Bar, Timeframe + session.py CME session calendar + bucket boundary math + aggregator.py 1m -> all timeframes + store.py BarStore protocol + InMemoryBarStore + analysis/ + indicators.py ATR, SMA, EMA (pure functions over bar lists) + manual_lines.py Hand-drawn lines: CRUD + JSON persistence ← M5 + pivots.py Fractal swing detection ← M8, deferred + trendlines.py Candidate generation, scoring, dedup ← M8, deferred + moving_averages.py MTF MA levels + levels.py Level type + registry, rebuild orchestration + confluence.py Clustering + scoring + alerts.py State machine, cooldown, dispatch + notify/ + ntfy.py Phone push + api/ + routes.py REST + ws.py WebSocket hub +static/ + index.html Vue 3 + LWC script tags (exists, replace) + app.js Vue app root (exists, replace) + chart.js Lightweight Charts wrapper (new) + style.css (exists, extend) +tests/ + ... +``` + +--- + +## 5. Data model + +Use dataclasses (or pydantic where it crosses the API boundary). All times are +**epoch seconds, UTC**. Never store naive local datetimes. + +```python +class Timeframe(str, Enum): + M1="1m"; M2="2m"; M5="5m"; M15="15m"; M30="30m"; H1="1h"; H4="4h"; D1="1d" + + @property + def seconds(self) -> int: ... # D1 is session-defined, not 86400 — see §6 + +@dataclass +class Bar: + tf: Timeframe + t: int # epoch seconds, bucket OPEN time + o: float; h: float; l: float; c: float + v: int + closed: bool # False while still forming + symbol: str # e.g. "/ESZ25" — carried so contract rolls stay visible +``` + +`Level` is the unified abstraction that makes the whole design work. Trendlines and +moving averages both reduce to "a price at time t, with a weight and a side". + +```python +class LevelKind(str, Enum): + MANUAL="manual" # hand-drawn — ships first (M5) + MA="ma" # moving average — ships first (M3) + TRENDLINE="trendline" # auto-detected — deferred to M8 + HORIZONTAL="horizontal" + +class Side(str, Enum): + SUPPORT="support"; RESISTANCE="resistance" + +@dataclass +class Level: + id: str # stable hash — the UI diffs on this, so keep it stable + kind: LevelKind + tf: Timeframe + side: Side + weight: float # timeframe weight x quality multiplier + score: float # raw quality score before weighting + label: str # "4h resistance", "1h EMA21" + + # Geometry. Trendline: price(t) = slope*(t - anchor_t) + anchor_p + anchor_t: int + anchor_p: float + slope: float # price units per SECOND. 0.0 for horizontal/MA-at-instant + + # For MAs: the stepped polyline actually drawn + points: list[tuple[int, float]] | None + + touches: int + first_t: int + last_t: int + provisional: bool # derived from a still-forming bar + hidden: bool # layer-panel visibility — see §9.4 for its effect on scoring + + def price_at(self, t: int) -> float: + return self.anchor_p + self.slope * (t - self.anchor_t) +``` + +The weight table referenced throughout — define it once, in `config.py`: + +```python +TIMEFRAME_WEIGHT = { + Timeframe.M1: 1, Timeframe.M2: 1, Timeframe.M5: 1, Timeframe.M15: 2, + Timeframe.M30: 3, Timeframe.H1: 4, Timeframe.H4: 8, Timeframe.D1: 16, +} +MA_WEIGHT_FACTOR = 0.75 # §7.4 — MAs weigh slightly less than drawn structure +``` + +```python +@dataclass +class Cluster: + id: str + side: Side + low: float; high: float; center: float + score: float # sum of member weights + members: list[Level] + distance: float # signed points from current price +``` + +--- + +## 6. Session and aggregation rules — read carefully + +This is where a naive implementation silently produces wrong lines. CME ES is not a +9:30–16:00 instrument. + +**Session definition (America/New_York, DST-aware via `zoneinfo`):** + +- Trading week opens **Sunday 18:00 ET**. +- Daily settlement break **17:00–18:00 ET, Monday–Thursday**. No bars expected. +- Week closes **Friday 17:00 ET**. +- A **futures "day"** runs 18:00 ET → 17:00 ET the following calendar day, and is + conventionally labelled with the *following* calendar date. Sunday 18:00 bars belong + to Monday's daily bar. + +**Bucketing rules:** + +- `1m, 2m, 5m, 15m, 30m, 1h` — bucket on wall-clock UTC boundaries. These divide the + hour evenly, so session anchoring is unnecessary and UTC keeps it simple. +- `4h` — **anchor to the 18:00 ET session open**, not UTC midnight. Buckets are + 18:00, 22:00, 02:00, 06:00, 10:00, 14:00 ET. UTC-anchored 4h buckets straddle the + session boundary and produce meaningless bars. +- `1d` — one bar per futures session as defined above. + +Implement this as `session.py::bucket_start(t: int, tf: Timeframe) -> int` and unit +test it hard, including both DST transitions and the Sunday open. **This function is +the highest-risk piece of pure logic in the project.** Write its tests first. + +**Aggregator behaviour:** + +- Maintain one forming bar per timeframe. On each incoming 1m bar: + - if `bucket_start(bar.t, tf)` differs from the forming bar's `t`, close the forming + bar (emit `closed=True`), then open a new one; + - otherwise fold in: `h=max`, `l=min`, `c=close`, `v+=`, emit `closed=False`. +- **Gaps do not close bars by time — they close by the arrival of a later bar.** Never + use a wall-clock timer to close a bucket; the market halts and holidays will fire it + incorrectly. Exception: emit a `closed=True` for the previous bucket when a bar + arrives that skips buckets entirely. +- Aggregation must be **deterministic and replayable**: feeding the same 1m sequence + twice must produce byte-identical output. No `datetime.now()` inside the aggregator. + +--- + +## 7. Analysis engines + +### 7.1 Indicators (`indicators.py`) + +Pure functions, list-in/list-out, no state: `sma(values, period)`, `ema(values, period)`, +`atr(bars, period=14)`. ATR is the universal scale unit — every tolerance in this +project is expressed in ATR multiples, never in fixed points, so the same config works +whether ES is at 4,000 or 8,000. + +### 7.2 Pivot detection (`pivots.py`) + +Fractal method, not regression. Regression fits the *middle* of price action; humans +draw lines across *extremes*, and extremes are what other traders react to. + +``` + ● pivot high (w bars lower on both sides) + / \ + / \ + ● / \ + / \ / \ +───●───●────────────── + pivot low +``` + +- `pivot_high(bars, w)`: index `i` qualifies if `high[i] >= max(high[i-w : i+w+1])` + and `i` is the leftmost such index in ties. +- Default `w = 3` for lower TFs, `w = 2` for `4h`/`1d` (fewer bars available). +- **Confirmation lag is `w` bars — this is intentional.** A pivot is only known `w` + bars after it forms. Do not "detect" pivots on the forming bar; that repaints, and a + repainting line is worse than no line. +- Significance filter: keep a pivot only if its prominence (distance to the + surrounding swing in the opposite direction) `>= 0.5 * ATR(14)` on that timeframe. + Drops noise pivots without hardcoding point values. + +### 7.3 Automatic trendlines (`trendlines.py`) — **deferred to M8** + +> Not part of the initial build. **Manual trendlines (§7.3a) ship first.** This section +> is retained because it is the eventual target and because §7.3a is deliberately +> designed to produce the same `Level` objects, so adopting this later changes nothing +> downstream. Skip to §7.3a on a first pass. + +``` +for each timeframe, for each side (highs → resistance, lows → support): + P = last N qualifying pivots (N = 25; ~300 candidate pairs, trivial) + for each pair (a, b) in P where a.t < b.t: + line = through (a.t, a.p) and (b.t, b.p) + evaluate(line) -> score or reject + dedup, keep top K = 4 per side per timeframe +``` + +**Evaluation of a candidate line:** + +Let `tol = 0.25 * ATR(14)` on that timeframe. + +- **Violation** — a bar *closes* beyond the line by more than `tol` + (above for resistance, below for support). Wicks do not count as violations; wicks + through a level are normal and often the point. +- **Touch** — a bar's extreme comes within `tol` of the line without violating it. +- **Reject** the line if `violations > 1` between the two anchors, or if any violation + occurred after the later anchor (the line is broken and no longer active). + +**Score:** + +``` +score = 3.0 * touches + + 1.5 * log1p(span_in_bars) + + 2.0 * recency_decay(last_touch) # exp(-age_bars / halflife), halflife=50 + - 4.0 * violations +``` + +Normalize within each timeframe/side group so weights stay comparable across +timeframes regardless of how many candidates a given timeframe happened to produce: + +```python +best = max(l.score for l in group) # after dedup, before truncation to K +quality = clamp(l.score / best, 0.0, 1.0) if best > 0 else 0.0 +level.weight = quality * TIMEFRAME_WEIGHT[tf] +``` + +So the strongest 4h line contributes the full 8.0, a mediocre one proportionally less, +and a 5m line can never outweigh a 4h line no matter how many touches it has. + +**The four coefficients above (3.0 / 1.5 / 2.0 / 4.0) are starting values, not +derived constants.** They cannot be got right on paper — expect to tune them by +eye against replayed tapes in M4. Put them in `config.py`, not inline, and treat "the +lines land where a human would draw them" as the acceptance criterion. + +**Dedup:** two lines are duplicates if, evaluated at *now*, their prices are within +`tol` **and** their slopes differ by less than 20%. Keep the higher score. Without +this you get a fan of ten nearly-identical lines from the same swing. + +**Recompute policy:** only on a **bar close** for that timeframe, never on every tick. +A 4h line recomputes 6× per day. This is what keeps the whole thing cheap. + +### 7.3a Manual trendlines (`manual_lines.py`) — ships in M5 + +A hand-drawn line is just a `Level` with `kind=MANUAL`. It flows into confluence, +projection, and alerts through exactly the same path as everything else — that is what +makes deferring the automatic engine cheap rather than a detour. + +**Why this ordering is better than it looks:** hand-drawn lines are *correct by +definition* (you drew them). So they validate the confluence engine without the +automatic detector's tuning risk, and they later become the ground truth that M8's +scoring coefficients get tuned against. + +**Geometry.** Anchors are stored in **absolute epoch seconds and price** — never bar +indices. This is why a line drawn on the 4h chart renders correctly on the 1m chart +with no conversion: both are the same `(time, price)` plane. The existing +`Level.price_at(t)` already handles it. + +**Timeframe attribution.** Tag the line with the timeframe that was *displayed when it +was drawn*. A line drawn on the 4h chart is a 4h line and carries weight 8. This is the +single most important field — without it every manual line would score identically. + +**Weight.** `quality = 1.0` always. The user drew it; it is not a candidate to be +scored. `weight = TIMEFRAME_WEIGHT[tf]`. + +**Drawing interaction** (all APIs verified present in LWC 5.2.0): + +| Action | Implementation | +|---|---| +| Enter draw mode | Toolbar button; changes cursor | +| Place endpoints | `chart.subscribeClick(handler)` → two clicks | +| Pixel → price | `series.coordinateToPrice(param.point.y)` | +| Pixel → time | `chart.timeScale().coordinateToTime(param.point.x)` | +| Render | `LineSeries` with 2 points, extended right (same as §9 auto lines) | +| Select | Click within ~6px of a line — hit-test in price space via `price_at(t)` | +| Delete | `Delete`/`Backspace` on selection, plus a button | +| Edit | **Delete and redraw.** Endpoint dragging is real work — do not build it in M5 | + +**Snapping.** When placing an endpoint, snap to the nearest bar high/low within ~8px. +Cheap to implement and it is the difference between a usable drawing tool and a +frustrating one. Make it toggleable; snap to high for resistance, low for support, +inferred from drag direction or nearest extreme. + +**Persistence — required in M5, not deferred.** A user who redraws their lines after +every restart abandons the tool. This does *not* require the bar store or a database: +write to a **JSON file** (`data/manual_lines.json`), loaded at startup. Bar persistence +can stay deferred to M7; these two are unrelated decisions. + +```json +{"id":"ml_01H...","tf":"4h","side":"resistance","anchor_t":1754600000, + "anchor_p":6412.5,"slope":-0.0000031,"created_at":1754700000,"note":"","hidden":false} +``` + +CRUD via `POST /api/lines`, `DELETE /api/lines/{id}`, `PATCH /api/lines/{id}`. On any +change the server recomputes levels and broadcasts `{"type":"levels",...}` — the +drawing client must render optimistically and then reconcile, not wait on the round +trip. + +### 7.4 Multi-timeframe moving averages (`moving_averages.py`) + +**Primary requirement: the daily MA set — SMA 10, 20, 50, 100, 200 — computed on daily +bars and displayed on the 1d, 30m, and 1m charts, with the base timeframe switchable.** + +This is the headline use of the multi-timeframe projection: the 200DMA is a level +people trade against, and on a 1-minute chart it should appear as a near-horizontal +line that steps once per session. Do not compute a "200-period MA of 1-minute bars" for +the 1m chart — that is a completely different and far less useful line. **The MA's +period is always tied to the timeframe it was computed on, never to the chart being +displayed.** + +``` +MA_SETS = { + "1d": [("sma", 10), ("sma", 20), ("sma", 50), ("sma", 100), ("sma", 200)], + # optional, off by default: + "4h": [("ema", 9), ("ema", 21)], + "1h": [("ema", 9), ("ema", 21)], +} +BASE_TIMEFRAMES = ["1m", "30m", "1d"] # the switcher; others remain available +``` + +Config-driven, so adding a set is a config change, not code. Other timeframes' MAs stay +supported by the same machinery but ship disabled — the daily set is what matters. + +**History check:** a 200DMA needs 200 sessions. Yahoo's 730-day 1h window yielded +17,387 bars ≈ 750 sessions, so all five daily MAs are warm from startup with roughly +3× margin. Verified, not assumed. + +**Expect small disagreements with thinkorswim/TradingView on the daily MAs.** We build +daily bars on the CME session (18:00→17:00 ET, §6); other platforms sometimes anchor +differently or use settlement prices. A one- or two-point difference in the 200DMA is +this, not a bug. Keep the daily anchor configurable so it can be matched if it matters. + +The key rendering decision: an MA from a higher timeframe drawn on a 1-minute chart is +a **step function**, held constant between higher-timeframe closes. + +``` +1h EMA21 rendered on a 1m chart: + + ┌──────── + ┌──┘ ← steps at each 1h close, NOT a smooth interpolation +┌──┘ +``` + +Interpolating between higher-TF closes would draw a line that was never true at the +time it appears to have been true. Emit `points` as a stepped polyline and render with +LWC's `lineType: LightweightCharts.LineType.WithSteps`. + +- The value from the **forming** higher-TF bar is emitted with `provisional=True` and + rendered dashed. It *will* move until that bar closes — that is honest, not a bug. +- An MA's **current value is a price level**, so it enters the confluence engine on + equal footing with trendlines, at `0.75 × timeframe_weight` (MAs are slightly less + reactive than drawn structure, but a 4h 200SMA is still a wall). +- **Warm-up:** an MA needs `period` closed bars on its timeframe. A daily 200SMA needs + 200 sessions. Until warm, emit nothing — never emit a partially-warmed MA. Yahoo + seeding (§10) makes every MA up to a daily 200SMA warm from startup; without it, + the higher-timeframe MAs stay cold for months. + +### 7.5 Confluence (`confluence.py`) + +This is the actual product. Everything above exists to feed it. + +``` +1. Collect every active Level, evaluate price_at(now). +2. Split by side relative to current price (levels above → resistance, below → support). + A level's stored `side` is its structural nature; its *effective* side is + positional. Use positional — a broken resistance acting as support is the + interesting case, not an error. +3. Sort by price. Single-linkage cluster: extend the current cluster while the gap to + the next level is <= clusterTol. + clusterTol = 0.4 * ATR14(15m) +4. Cluster score = sum of member weights. +5. Emit clusters with >= 2 members OR score >= 8 (a lone 1d line matters by itself). +``` + +``` +5m resistance 6404.25 weight 1 +15m resistance 6405.00 weight 2 +1h resistance 6403.75 weight 4 +4h resistance 6404.50 weight 8 + ↓ + RESISTANCE CLUSTER 6403.75 – 6405.00 + CONFLUENCE 15 +``` + +Recompute on every closed 1m bar, and on any level rebuild. + +### 7.6 Alerts (`alerts.py`) + +The failure mode to design against is notification fatigue. A per-line alert makes the +system useless within a day. + +Per-cluster state machine, keyed on `(side, round(center / clusterTol))` so a cluster +keeps its identity as members drift: + +``` +ARMED ──price within alertTol of cluster──► FIRED ──► COOLDOWN ──┐ + ▲ │ + └────── price moves > 2*alertTol away AND cooldown elapsed ◄────┘ +``` + +- `alertTol = 0.5 * ATR14(15m)` +- `cooldown = 15 minutes` +- Minimum score threshold to fire: **configurable, starting value 6 — but this + certainly needs recalibrating in M4.** With the daily MA set as the primary levels, + each daily MA carries `0.75 × 16 = 12`, so *any two* of them near each other scores + 24 and a threshold of 6 would fire constantly. Either raise the threshold well above + 24, or damp the weight when several MAs from the same timeframe cluster (they are not + independent evidence the way a 4h line and a 1h line are). Decide this against a + replayed tape, not on paper. **Target: single-digit alerts per session.** +- Re-arming requires *both* the price separation and the cooldown. Time alone lets a + price oscillating on a level fire forever. +- Fire on **closed 1m bars only**, not intra-bar ticks. + +Dispatch to: WebSocket (UI banner + sound) and ntfy. Payload: + +``` +BEARISH ZONE /ES 6404.25 +Resistance confluence 15 @ 6403.75–6405.00 +5m, 15m, 1h, 4h +``` + +--- + +## 8. API contract + +Fix these shapes now; the frontend and backend are built against them. + +### REST + +| Route | Returns | +|---|---| +| `GET /api/status` | `{stream: "connected"\|"disconnected"\|"replay", symbol, last_bar_t, bars_held: {tf: n}, warm: {tf: bool}}` | +| `GET /api/bars?tf=5m&limit=500` | `{tf, bars: [{t,o,h,l,c,v,closed}]}` — oldest first | +| `GET /api/levels?tf=all` | `{levels: [Level]}` | +| `GET /api/confluence` | `{price, clusters: [Cluster]}` | +| `GET /api/health` | existing | + +### WebSocket `/ws` + +Client → server on connect: +```json +{"type": "subscribe", "tf": "5m"} +``` + +Server → client: +```json +{"type":"snapshot","tf":"5m","bars":[...],"levels":[...],"clusters":[...],"price":6404.25} +{"type":"bar","tf":"5m","bar":{"t":1754700000,"o":6403.5,"h":6405.0,"l":6403.0,"c":6404.25,"v":812,"closed":false}} +{"type":"levels","levels":[...]} +{"type":"clusters","price":6404.25,"clusters":[...]} +{"type":"alert","cluster":{...},"message":"..."} +{"type":"status","stream":"disconnected"} +``` + +Rules: +- Send `bar` on **every** update of the forming bar (that is the live chart) but batch + `levels` — they only change on higher-TF closes. +- Always send a full `snapshot` on connect and after any reconnect. The client must + never try to reconcile a gap. +- Levels are sent for **all** timeframes regardless of the displayed timeframe. That is + the entire point: a 4h line drawn through a 1m chart. + +--- + +## 9. Frontend + +`static/index.html` loads Vue 3 and Lightweight Charts as script tags — no bundler, +matching the existing app. + +```html + + +``` + +### Vue + Lightweight Charts integration — the one real gotcha + +**Never put the chart or series objects in `ref()` or `reactive()`.** Vue's deep +reactive proxy will wrap the library's internal objects, which breaks identity checks +inside the library and destroys performance on every update. Use `shallowRef`, or +better, a plain module-scoped variable / `markRaw`. + +```js +const chartApi = shallowRef(null); // ✅ +// const chart = ref(null); // ❌ will appear to work, then misbehave +``` + +Structure: +- `chart.js` — a plain, framework-free wrapper class owning the LWC instance: + `create(el)`, `setBars()`, `updateBar()`, `syncLevels(levels)`, `destroy()`. +- `app.js` — Vue app owning state (timeframe, connection status, clusters, alert log) + and the WebSocket. Calls into the wrapper imperatively in `onMounted` / watchers. +- Chart lifecycle in `onMounted`; `chart.remove()` in `onUnmounted`; a `ResizeObserver` + driving `chart.applyOptions({width, height})`. + +### v5 API usage + +```js +const chart = LightweightCharts.createChart(el, {...}); +const candles = chart.addSeries(LightweightCharts.CandlestickSeries, {...}); +const line = chart.addSeries(LightweightCharts.LineSeries, { + lineType: LightweightCharts.LineType.WithSteps, // for MTF moving averages +}); +candles.setData(bars); // once +candles.update(bar); // every tick — never re-call setData +``` + +### Rendering levels + +`syncLevels()` must **diff by `Level.id`**, not clear-and-rebuild. Rebuilding every +series on each update causes visible flicker and leaks series objects. + +- Trendline → a `LineSeries` with two points: `(anchor_t, anchor_p)` and + `(now + rightExtension, price_at(now + rightExtension))`. Extend ~20% of the visible + range into the future so the line is usable ahead of price. +- MA → a `LineSeries` fed the stepped `points` array. +- Colour **by timeframe** (one hue per TF, consistent everywhere including the + confluence panel). Line width scales with timeframe weight. `provisional` levels + render dashed via `lineStyle: LightweightCharts.LineStyle.Dashed`. + +### 9.4 Layer panel (checkboxes) + +Ships with M3. Visibility control is load-bearing here, not decoration — five daily MAs +plus manual lines plus any optional intraday MA sets stack up fast. + +``` +CHART [ 1m ] [ 30m ] [ 1d ] ← base timeframe switcher + +LAYERS +───────────────────────────────── +☑ Daily MAs ██ + ☑ 10 ☑ 20 ☑ 50 ☑ 100 ☑ 200 +───────────────────────────────── +☐ 4h MAs ██ + ☐ EMA9 ☐ EMA21 +☐ 1h MAs ▓▓ + ☐ EMA9 ☐ EMA21 +───────────────────────────────── +☑ Manual lines ░░ (M5) +☐ Auto trendlines (M8) +───────────────────────────────── +☐ Hidden levels still count toward confluence +``` + +- **The base timeframe switcher changes only the candles.** Every level stays on screen + — a 200DMA is equally valid on a 1m chart. That is the entire premise of the product. +- **The group checkbox is a master toggle** — unchecking "Daily MAs" hides all five at + once; individual periods nest under it. +- The colour swatch beside each timeframe is that timeframe's hue, used identically on + the chart and in the confluence panel. One hue per timeframe, everywhere. +- **Persist to `localStorage`.** These settings are pure UI preference and must survive + reloads; they do not belong on the server. + +**Visibility vs. scoring — decide this explicitly.** Hiding a level defaults to +*also* removing it from confluence scoring, because "I don't want to see this" almost +always means "I don't care about this." The last checkbox decouples the two for anyone +who wants a clean chart with full scoring. + +That default has an architectural consequence: **confluence is computed server-side, so +the client's enabled set must reach the server.** Send it over the WebSocket on change: + +```json +{"type":"prefs", + "base_tf":"1m", + "enabled":{"ma":{"1d":[10,20,50,100,200]},"manual":true,"auto":false}, + "hidden_levels_score":false} +``` + +Store it per connection. Single-user app — no need for anything more elaborate. + +### Panels + +- **Drawing toolbar** (M5) — trendline tool, snap toggle, delete selection. +- **Confluence panel** — clusters sorted by `|distance|`, each showing member + timeframes, the zone range, and the score. This is the primary readout; give it more + visual weight than the price itself. +- **Status bar** — stream state, contract symbol, last bar age, which timeframes are + warm. When the stream drops, the chart must *say so*, not quietly show stale candles. +- **Alert log** — recent fires, most recent first. +- **Timeframe selector** — switches the candle series only. Levels stay. + +Keep the existing dark/light CSS-variable scheme in `style.css`. + +--- + +## 10. The history problem + +**Mostly solved by the Yahoo source (§2.1) — but read the caveats.** + +Schwab alone would leave the system knowing nothing before the moment it connects. +Since the highest-weighted timeframes need the most history, Schwab-only cold start +would mean waiting months for the parts of the product that matter most: + +| Timeframe | Usable after, Schwab-only | With Yahoo seeding | +|---|---|---| +| 5m, 15m | ~2–4 hours | immediate | +| 30m, 1h | ~1–2 sessions | immediate | +| 4h | ~1–2 weeks | immediate (from 1h) | +| 1d | months | immediate (~500 sessions) | +| 1d 200SMA | ~10 months | immediate | + +**Startup sequence:** + +1. Seed from Yahoo: `interval=1h&range=730d`, plus `interval=1m&range=8d` for the + fine detail near the present. +2. Aggregate through `aggregator.py` into every timeframe, using our own session rules. +3. Attach the live source (Yahoo poll, or Schwab once keys exist) and continue forward. +4. Persist everything (M7) so subsequent restarts need less seeding. + +**Caveats that must be honoured in code:** + +- **Tag every bar with its `source`** (`"yahoo"` / `"schwab"` / `"replay"`), and expose + the seam in the UI. Silently blending delayed continuous data with live per-contract + data produces trendlines nobody else can see. +- **Roll gaps.** Yahoo `ES=F` is a continuous front-month series; quarterly contract + rolls leave price discontinuities that will read as a trendline break or generate a + bogus pivot. For M0–M5 this is acceptable. If it proves noisy, the fix is either + panama-adjusting the seeded series or dropping pivots within ±1 bar of a known roll + date (third Friday of Mar/Jun/Sep/Dec). Do not build roll adjustment pre-emptively. +- **The 1m request cap is 8 days.** Longer 1m ranges must be fetched in ≤8-day windows + and stitched. In practice you don't need deep 1m history — seed 1h and let the + aggregator do the rest. +- **Never mix Yahoo daily bars in** — see §2.1. + +If per-contract accuracy ever matters more than convenience, **Databento** sells proper +CME history with real roll handling. Not needed now. + +--- + +## 11. Configuration + +All via env, read in `config.py`. Ship a `.env.example`; `.env` is already gitignored. + +``` +# --- data sources --- +LIVE_SOURCE=yahoo # yahoo | schwab | replay +SEED_SOURCE=yahoo # yahoo | none +YAHOO_SYMBOL=ES=F +YAHOO_POLL_SECONDS=20 +SEED_1H_RANGE=730d +SEED_1M_RANGE=8d # 8d is Yahoo's hard per-request cap + +# --- schwab (only needed once LIVE_SOURCE=schwab) --- +SCHWAB_API_KEY= +SCHWAB_APP_SECRET= +SCHWAB_CALLBACK_URL=https://127.0.0.1:8182 +SCHWAB_TOKEN_PATH=./.schwab_token.json +SCHWAB_ACCOUNT_ID= +SCHWAB_SYMBOL=/ES + +# --- timeframes & indicators --- +TIMEFRAMES=1m,2m,5m,15m,30m,1h,4h,1d +BASE_TIMEFRAMES=1m,30m,1d # the chart switcher +MAX_BARS_PER_TF=5000 # in-memory ring buffer bound + +# Daily MA set is the primary requirement; others ship disabled. See §7.4 +MA_SETS__1D=sma10,sma20,sma50,sma100,sma200 +MA_SETS__4H= +MA_SETS__1H= +DAILY_ANCHOR_ET=18:00 # CME session open; change to match another platform + +MANUAL_LINES_PATH=./data/manual_lines.json + +CONFLUENCE_MIN_SCORE=6 # MUST be recalibrated in M4 — see §7.6 +ALERT_COOLDOWN_SECONDS=900 + +NTFY_TOPIC= +NTFY_SERVER=https://ntfy.sh + +CHART_AUTH_TOKEN= # empty = auth disabled (local dev) +REPLAY_FILE= # set to replay a tape instead of connecting +``` + +**Token persistence:** `schwab-py`'s refresh token expires every **7 days** and +re-auth is an interactive browser flow. Locally, keep `.schwab_token.json` out of the +repo (add to `.gitignore`). On the VPS later, it **must** live on a Coolify persistent +volume or every rebuild logs you out. Same for the eventual SQLite file. + +--- + +## 12. Testing + +The market is closed most of the time you will be working. Build for that. + +**Record/replay is a milestone-1 deliverable, not a nicety.** `recorder.py` writes +every raw stream message to JSONL with its arrival timestamp; replay feeds them back +through the identical code path, either at wall-clock speed or as fast as possible. +Everything downstream of `StreamService` is then testable, deterministically, offline. + +Required tests: + +- `session.py` — bucket boundaries. Both DST transitions, Sunday 18:00 open, the + 17:00–18:00 break, Friday close, and the 4h session anchor. **Write these first.** +- `aggregator.py` — 1m→all TFs on synthetic bars; gap handling; idempotent replay. +- `moving_averages.py` — a known daily series produces known 10/20/50/100/200 values; + assert nothing is emitted before warm-up; assert the stepped projection onto 1m holds + its value for a whole session and changes exactly at the session boundary. +- `manual_lines.py` — round-trip JSON persistence; a line drawn on 4h evaluates to the + same price on the 1m chart at the same instant. +- *(M8)* `pivots.py` — known fixtures; assert no repainting (a pivot, once emitted, + never changes when more bars arrive). +- *(M8)* `trendlines.py` — hand-built fixtures where the correct line is obvious; + assert violation rejection and dedup. +- `confluence.py` — synthetic levels producing a known cluster and score. +- `alerts.py` — assert no re-fire within cooldown, and that oscillation around a level + produces exactly one alert. + +Add `pytest` to a `requirements-dev.txt`. + +--- + +## 13. Milestones + +Ordered so the user's stated priority — **live realtime charts first** — lands +earliest, and so nothing later is blocked on market hours. + +**No API keys are required until M6.** M0–M5 run entirely on Yahoo. + +### M0 — Yahoo source (no keys, no blockers) +`market/base.py` protocol + `market/yahoo.py`: `history()` over the verified chart +endpoint, and `stream()` as a polling loop presenting the same async-iterator +interface. Null-filtering, 8-day 1m windowing, bar `source` tagging. Recorder writes a +tape; `ReplaySource` reads it back. +**Done when:** a script prints seeded 1h bars back to 2024 and then live-ish 1m bars, +and a recorded tape replays identically. + +### M1 — Live chart end to end ⭐ primary deliverable +`StreamService` → in-memory 1m store → WebSocket → Vue 3 + LWC candlestick chart +updating live. Status bar showing source + bar age. No analysis yet. +**Done when:** the browser shows a live-updating /ES 1-minute candle chart, and the +same chart can be reproduced offline from a tape. + +### M2 — Aggregation + timeframe switching +`session.py` + `aggregator.py` with full test suite. Timeframe selector drives the +candle series. `GET /api/bars`. +**Done when:** switching to 15m shows correctly bucketed bars, and replaying a tape +twice yields identical output. + +### M3 — Multi-timeframe moving averages ⭐ start here for analysis +MAs are the right first analysis layer: **fully deterministic, no parameters to tune, +no judgment calls**, and Yahoo seeding makes them warm from startup. They validate the +entire overlay concept — stepped rendering, level diffing by `id`, TF colour scheme — +without any of the ambiguity trendlines carry. +**Done when:** the 10/20/50/100/200 DMAs render on the 1m, 30m and 1d charts, stepping +once per session on the intraday views, dashed while the current session is unfinished. + +### M3.5 — Layer panel +Checkbox tree to show/hide levels by timeframe and by individual MA (§9.4). Ships with +M3 because 6 timeframes × 4 MAs = 24 lines is unreadable without it. +**Done when:** unchecking "4h" removes every 4h level from the chart and the state +survives a reload. + +### M4 — Confluence + alerts (on moving averages alone) ⭐ first genuinely useful build +`confluence.py`, `alerts.py`, confluence panel, ntfy push, in-browser sound — scored +over MA levels only. Multi-timeframe MA confluence is a real signal in its own right; +this is a complete, useful product with zero hand-drawn input and zero tuning. +**Done when:** a replayed tape produces a sane number of alerts (single digits per +session), each corresponding to a real multi-timeframe convergence. + +### M5 — Manual trendlines +Two-click drawing, snapping, persistence, feeding the same confluence engine (§7.3a). +Hand-drawn lines are authoritative — full weight, no quality discount. +**Done when:** a line drawn on the 4h chart appears correctly projected on the 1m chart +and raises the confluence score of a cluster it lands in. + +### M6 — Schwab live source (needs API keys) +Implement `market/schwab.py` against the existing `MarketDataSource` protocol and +answer every question in [§2.2](#22-verify-these-when-adding-the-schwab-source-m6-not-before). +Yahoo continues to handle seeding; Schwab takes over the live tail. +**Done when:** flipping `LIVE_SOURCE=yahoo` → `schwab` changes nothing visible except +lower latency and true per-contract prices. **If this milestone requires touching any +file outside `market/`, the abstraction in M0 was wrong — fix it there, not here.** + +### M7 — Persistence +`SqliteBarStore` behind the existing `BarStore` protocol. Backfill-on-start from disk, +falling back to Yahoo seeding only for what's missing. +**Done when:** restarting the process loses no history. + +### M8 — Automatic trendline detection (optional) +`pivots.py`, `trendlines.py` per §7.3. Deliberately last among the analysis work: +by this point the hand-drawn lines from M5 are **ground truth**, so the scoring +coefficients can be tuned to agree with lines you actually drew, rather than guessed at +in the abstract. Auto lines render in a distinct style and are individually +dismissable; they never silently replace a manual line. +**Done when:** on a replayed tape, auto-detected lines land where the manual ones were +drawn, and the layer panel can hide them independently. + +### M9 — Bias panel +BULLISH / BEARISH toggle recording the user's directional call against the current +confluence state, persisted, with a journal view. **Records only — trades nothing.** + +Because `LEVEL_ONE_FUTURES_OPTIONS` streams, this milestone can go further than the +original sketch: given a bias, resolve the 1-DTE strikes, subscribe to the two legs, +and display the **live spread price** — so the readout becomes actionable enough to +hand-execute in thinkorswim: + +``` +BEARISH /ES — resistance confluence 15 @ 6403.75–6405.00 +Proposed: 1-DTE 6405/6415 put spread ~2.35 x 10 (live) +``` + +Strike/expiration symbol resolution for futures options is fiddly; treat it as its own +sub-task and verify the symbol format against `LEVEL_ONE_FUTURES_OPTIONS` empirically, +the same way M6 verifies `/ES`. + +### M10 — VPS deploy (when wanted) +Shared-secret auth on, `workers=1`, persistent volume for token + DB, Coolify domain +port suffix preserved per README. + +--- + +## 14. Why execution is out of scope + +Note the asymmetry: Schwab **does** stream futures-options *quotes* +(`LEVEL_ONE_FUTURES_OPTIONS`), so we can price a spread live — we just cannot transmit +it. Read anything below as being about order entry only. + +Schwab's Trader API exposes no futures or futures-options **order entry**. thinkScript +`AddOrder()` places *simulated* orders for backtesting only. Automating clicks in the +thinkorswim UI is the wrong reliability model for near-expiration leveraged +instruments — window focus, stale quotes, partial fills, and dialogs all fail silently, +and an execution path must be able to tell the program what the broker actually did. + +The interim workflow is therefore: this app produces a decision, the human executes it +in thinkorswim. + +Keep the seam clean. Broker-specific code lives only in `market/schwab.py`; nothing in +`analysis/`, `bars/`, or `api/` may import it. When an execution adapter is added — +Schwab, if they ever ship futures-options orders, or IBKR — it consumes `Cluster` and +the M9 bias signal and nothing else. This is the same discipline the M6 acceptance test +enforces for data sources. + +--- + +## 15. Risk register + +| Risk | Impact | Mitigation | +|---|---|---| +| Futures entitlement missing on the Schwab account | Delays M6 only | No longer blocks — M0–M5 run on Yahoo | +| No futures history from Schwab | Higher TFs cold for months | Solved: Yahoo seeding, §10 | +| Yahoo endpoint is unofficial — may rate-limit or change shape | Dev source breaks | Isolated in `market/yahoo.py`; cache seeds to disk (M7) so it's fetched rarely; back off on 429 | +| Yahoo daily bars anchored midnight ET, not session | Daily candles disagree with every other chart | Never use them — build 1d from 1h, §2.1 | +| Contract roll gaps fake a trendline break | Bad signals on seeded data | Store `symbol` + `source` per bar; surface rolls in UI; §10 | +| Source abstraction leaks Schwab/Yahoo specifics upward | M6 turns into a rewrite | M6 acceptance test: no file outside `market/` may change | +| `session.py` bucket math wrong | Silently wrong lines everywhere | Tests written first; both DST transitions | +| Repainting pivots | Lines that "were always there" | `w`-bar confirmation lag, enforced by test | +| Alert fatigue | Product becomes unusable | Cluster-level alerts, cooldown + separation re-arm | +| Multiple uvicorn workers | Duplicate Schwab connections | `workers=1`; streamer in `lifespan` | +| Schwab token expiry (7 days) | Stream dies | Surface prominently in status bar; document re-auth | +| Vue reactivity wrapping chart objects | Perf collapse, odd bugs | `shallowRef`/`markRaw` — §9 | +| LWC v4 tutorials copied | Code silently wrong for v5 | `addSeries(SeriesType, ...)` only | diff --git a/tests/fixtures/yahoo_es_1h.json b/tests/fixtures/yahoo_es_1h.json new file mode 100644 index 0000000..c799e8e --- /dev/null +++ b/tests/fixtures/yahoo_es_1h.json @@ -0,0 +1,368 @@ +{ + "chart": { + "result": [ + { + "meta": { + "currency": "USD", + "symbol": "ES=F", + "exchangeName": "CME", + "fullExchangeName": "CME", + "instrumentType": "FUTURE", + "firstTradeDate": 969249600, + "regularMarketTime": 1786324854, + "hasPrePostMarketData": false, + "gmtoffset": -14400, + "timezone": "EDT", + "exchangeTimezoneName": "America/New_York", + "regularMarketPrice": 7778.0, + "fiftyTwoWeekHigh": 7820.25, + "fiftyTwoWeekLow": 6353.25, + "regularMarketDayHigh": 7779.25, + "regularMarketDayLow": 7763.0, + "regularMarketVolume": 25260, + "shortName": "E-Mini S&P 500 Sep 26", + "chartPreviousClose": 7765.5, + "previousClose": 7779.75, + "scale": 3, + "priceHint": 2, + "currentTradingPeriod": { + "pre": { + "timezone": "EDT", + "end": 1786248000, + "start": 1786248000, + "gmtoffset": -14400 + }, + "regular": { + "timezone": "EDT", + "end": 1786334340, + "start": 1786248000, + "gmtoffset": -14400 + }, + "post": { + "timezone": "EDT", + "end": 1786334340, + "start": 1786334340, + "gmtoffset": -14400 + } + }, + "tradingPeriods": [ + [ + { + "timezone": "EDT", + "end": 1785902340, + "start": 1785816000, + "gmtoffset": -14400 + } + ], + [ + { + "timezone": "EDT", + "end": 1785988740, + "start": 1785902400, + "gmtoffset": -14400 + } + ], + [ + { + "timezone": "EDT", + "end": 1786075140, + "start": 1785988800, + "gmtoffset": -14400 + } + ], + [ + { + "timezone": "EDT", + "end": 1786161540, + "start": 1786075200, + "gmtoffset": -14400 + } + ], + [ + { + "timezone": "EDT", + "end": 1786334340, + "start": 1786248000, + "gmtoffset": -14400 + } + ] + ], + "dataGranularity": "1h", + "range": "5d", + "validRanges": [ + "1d", + "5d", + "1mo", + "3mo", + "6mo", + "1y", + "2y", + "5y", + "10y", + "ytd", + "max" + ] + }, + "timestamp": [ + 1785816000, + 1785819600, + 1785823200, + 1785826800, + 1785830400, + 1785834000, + 1785837600, + 1785841200, + 1785844800, + 1785848400, + 1785852000, + 1785855600, + 1785859200, + 1785862800, + 1785866400, + 1785870000, + 1785873600, + 1785877200, + 1785880800, + 1785884400, + 1785888000, + 1785891600, + 1785895200, + 1785898800, + 1785902400, + 1785906000, + 1785909600, + 1785913200, + 1785916800, + 1785920400, + 1785924000, + 1785927600, + 1785931200, + 1785934800, + 1785938400, + 1785942000, + 1785945600, + 1785949200, + 1785952800, + 1785956400 + ], + "indicators": { + "quote": [ + { + "open": [ + 7644.0, + 7645.25, + 7647.75, + 7644.5, + 7643.5, + 7637.0, + 7637.75, + 7643.25, + 7654.5, + 7651.0, + 7685.25, + 7715.5, + 7736.5, + 7755.25, + 7768.25, + 7773.0, + 7763.75, + null, + 7772.0, + 7780.75, + 7777.25, + 7784.25, + 7784.5, + 7786.75, + 7792.75, + 7791.25, + 7798.5, + 7798.25, + 7792.75, + 7784.25, + 7788.5, + 7797.5, + 7798.0, + 7799.5, + 7818.75, + 7786.5, + 7757.75, + 7769.75, + 7767.0, + 7765.5 + ], + "close": [ + 7645.5, + 7647.5, + 7644.5, + 7643.5, + 7637.25, + 7637.5, + 7643.0, + 7654.5, + 7651.0, + 7685.5, + 7715.25, + 7736.5, + 7755.25, + 7768.25, + 7773.0, + 7764.75, + 7774.5, + null, + 7780.75, + 7777.5, + 7784.25, + 7784.25, + 7786.75, + 7792.0, + 7791.25, + 7798.75, + 7798.0, + 7793.0, + 7784.25, + 7788.5, + 7797.5, + 7798.0, + 7799.5, + 7819.25, + 7786.25, + 7758.0, + 7769.5, + 7767.0, + 7765.5, + 7748.25 + ], + "low": [ + 7641.75, + 7642.75, + 7642.75, + 7641.5, + 7635.25, + 7631.75, + 7635.5, + 7641.5, + 7651.0, + 7649.25, + 7682.5, + 7714.0, + 7733.25, + 7754.25, + 7763.5, + 7761.25, + 7761.25, + null, + 7771.0, + 7775.75, + 7776.0, + 7777.75, + 7783.0, + 7786.5, + 7788.25, + 7790.5, + 7791.0, + 7788.0, + 7780.25, + 7782.5, + 7787.5, + 7792.5, + 7795.5, + 7795.0, + 7776.0, + 7754.5, + 7750.5, + 7761.75, + 7763.25, + 7745.75 + ], + "volume": [ + 0, + 5511, + 7319, + 14850, + 18165, + 14744, + 19853, + 29305, + 21498, + 205567, + 243043, + 187171, + 123068, + 152916, + 128635, + 293219, + 80121, + null, + 0, + 6231, + 8543, + 8701, + 8414, + 7996, + 4935, + 6946, + 9271, + 15543, + 21633, + 12652, + 12910, + 14971, + 28403, + 203055, + 265756, + 189580, + 148224, + 81542, + 72103, + 207321 + ], + "high": [ + 7646.5, + 7649.5, + 7650.0, + 7648.0, + 7645.75, + 7638.5, + 7647.75, + 7657.75, + 7656.25, + 7686.5, + 7716.5, + 7741.25, + 7755.75, + 7783.75, + 7777.25, + 7786.0, + 7781.25, + null, + 7783.25, + 7782.75, + 7785.5, + 7786.0, + 7788.75, + 7792.5, + 7792.75, + 7800.0, + 7799.5, + 7799.5, + 7796.75, + 7792.0, + 7798.0, + 7799.75, + 7805.25, + 7820.25, + 7818.75, + 7791.5, + 7773.75, + 7773.0, + 7774.5, + 7771.5 + ] + } + ] + } + } + ], + "error": null + } +} \ No newline at end of file