1240 lines
59 KiB
Markdown
1240 lines
59 KiB
Markdown
# /ES Multi-Timeframe Confluence Chart — Implementation Plan
|
||
|
||
**Audience:** whoever is changing this next. It was written as a spec to execute
|
||
top-to-bottom; it is now a reference for a running system. Read the section that
|
||
covers what you are touching, not the whole thing.
|
||
|
||
**This is a living document.** When a decision here stops being true, change it
|
||
here — a plan that disagrees with the code is worse than no plan, because it is
|
||
believed. What went wrong on the way belongs in `docs/implementation.md`.
|
||
|
||
**One-line goal:** stream `/ES` 1-minute bars from Schwab, aggregate them into every
|
||
larger timeframe locally, derive trendlines and moving averages on each timeframe,
|
||
project them all onto one chart in a shared `(time, price)` coordinate system, and
|
||
alert when independently-derived levels from different timeframes converge.
|
||
|
||
**Explicitly out of scope:** order execution. Nothing in this codebase places a trade.
|
||
See [§14](#14-why-execution-is-out-of-scope) for why, and for the seam left behind.
|
||
|
||
> **Companion document:** `docs/implementation.md` records what actually
|
||
> happened — the problems hit while building this and how each was resolved.
|
||
> This file is the plan and the reasoning; that one is the experience. When they
|
||
> disagree, the log is what really occurred.
|
||
|
||
|
||
---
|
||
|
||
## 0. Start here
|
||
|
||
> **This document is now mostly history.** M0–M10 are built and deployed. The
|
||
> build order, branch instructions and "existing repo state" that used to open
|
||
> this file described a greenfield app and were actively misleading by August
|
||
> 2026, so they are gone. What remains below is the reasoning behind decisions
|
||
> already made — read it to understand *why* something works the way it does,
|
||
> not to find out what to build.
|
||
>
|
||
> **For current work, start with `AGENTS.md`**, which every agent loads
|
||
> automatically. It points at the live planning documents:
|
||
> `docs/NEXT_STEPS.md` for the short list, `docs/async_refactor.md`,
|
||
> `docs/multi_user.md`, `docs/feature_undo.md`, `docs/mobile_enhance.md`,
|
||
> `docs/vite_build.md`, `docs/plan_light_dark_themes.md` and
|
||
> `docs/plan_dma_alerts.md` for designs not yet built.
|
||
>
|
||
> **`main` deploys to production.** A push triggers a Forgejo webhook and
|
||
> Coolify rebuild of <https://chart.amow.com>. That is the intended workflow now,
|
||
> not an accident to avoid — but it means every push is a deploy, and a deploy
|
||
> restarts the market stream.
|
||
>
|
||
> §16 onward is a dated log of problems and their resolutions. It is the most
|
||
> useful part of this file for anyone debugging: most entries record something
|
||
> that looked like one bug and was another.
|
||
|
||
### Dependencies to add
|
||
|
||
`requirements.txt` currently contains only `fastapi` and `uvicorn[standard]`. Add:
|
||
|
||
```
|
||
httpx # Yahoo fetches; async, already a FastAPI-adjacent standard
|
||
pydantic-settings # config.py
|
||
```
|
||
|
||
`requirements-dev.txt`:
|
||
|
||
```
|
||
pytest
|
||
pytest-asyncio
|
||
```
|
||
|
||
Do **not** add `schwab-py` until M6 — it is unused before then. Do **not** add
|
||
`yfinance`; the Yahoo chart endpoint is a plain HTTP GET and the extra dependency buys
|
||
nothing (verified — see §2.1).
|
||
|
||
### Test fixture already provided
|
||
|
||
`tests/fixtures/yahoo_es_1h.json` is a **real, trimmed** Yahoo response for
|
||
`ES=F&interval=1h` (40 bars). Use it to unit-test the parser offline. It deliberately
|
||
**contains a `null` in the OHLC arrays**, which is exactly the case §2.1 warns about —
|
||
if your parser doesn't filter those, that fixture will catch it.
|
||
|
||
### Conventions
|
||
|
||
- All times are **epoch seconds, UTC**, everywhere. No naive datetimes.
|
||
- Nothing outside `market/` may know which data source is in use.
|
||
- Nothing outside `market/schwab.py` may import broker-specific code.
|
||
- Analysis code takes lists of bars and returns values — no I/O, no clocks. This is
|
||
what makes the replay harness work.
|
||
|
||
---
|
||
|
||
## 1. Decisions already made — do not relitigate
|
||
|
||
| Decision | Choice | Why |
|
||
|---|---|---|
|
||
| Backend | FastAPI (already scaffolded) | Repo already runs it; native WebSocket support |
|
||
| Frontend | Vue 3 from CDN, **no build step** | Matches existing `static/` setup; keeps deploy trivial. Destination is Vite — see `docs/vite_build.md`. Do not treat this row as a reason to reject that move. Stay cheap — `AGENTS.md` § Stay cheap. No per-pixel overlay work, no extra work on the event loop each tick. |
|
||
| Charting | TradingView Lightweight Charts **v5.2.0**, standalone build | Apache-2.0, canvas, built for incremental realtime updates |
|
||
| Data source | **Pluggable `MarketDataSource`.** Yahoo first, Schwab later | Yahoo needs no API key *and* has the history Schwab lacks — see §2.1 |
|
||
| Persistence | **In-memory first**, behind a `BarStore` interface | User confirmed deferring persistence is fine for v1 |
|
||
| Eventual persistence | **SQLite**, not Postgres | Single file, zero Coolify stack expansion. Revisit only if multi-process |
|
||
| Deployment | **Local first**, VPS later | Keep all config in env vars so the VPS move is config, not rewrite |
|
||
| Alerts | ntfy/Pushover phone push (+ free in-browser sound) | User selected phone push |
|
||
| Auth | Hardcoded shared secret from env | User confirmed; only matters once VPS-exposed |
|
||
|
||
| Trendlines | **Manual (hand-drawn) first.** Auto-detection deferred to M8 | Hand-drawn lines are correct by definition, so they validate the confluence engine with zero tuning risk — and later become the ground truth the auto-detector is tuned against |
|
||
| Moving averages | **Daily set (10/20/50/100/200 SMA) is the priority**, shown on 1m / 30m / 1d | The 200DMA is a level people actually trade against |
|
||
|
||
### Timeframe roles
|
||
|
||
> **4h was removed from the product on 2026-08-10.** It was never enabled, and
|
||
> dropping it deleted the fiddliest logic in `session.py` — the wall-clock ET 4h
|
||
> anchor and its DST edge cases — for a timeframe nobody was using. Later
|
||
> sections of this document still use 4h in examples; read those as
|
||
> illustrative, not as a spec to build. Nothing below a day is session-anchored
|
||
> any more, so `bucket_start` now special-cases only `1d`.
|
||
|
||
```
|
||
1m base chart + alert evaluation weight 1 ← a switchable base timeframe
|
||
2m display only
|
||
5m manual lines weight 1
|
||
15m manual lines weight 2
|
||
30m base chart + manual lines weight 3 ← a switchable base timeframe
|
||
1h manual lines (+ optional MAs) weight 4
|
||
1d base chart + THE DAILY MAs weight 16 ← a switchable base timeframe
|
||
```
|
||
|
||
Base timeframe controls the candles only. **Every level stays visible on every base
|
||
timeframe** — the 200DMA on a 1-minute chart is the point, not a side effect.
|
||
|
||
---
|
||
|
||
## 2.1 Data sources — build against Yahoo, swap in Schwab later
|
||
|
||
**Do not block on Schwab API keys.** The two sources are complementary, and the
|
||
Yahoo one is strictly easier to develop against:
|
||
|
||
| | Yahoo `ES=F` | Schwab `/ES` |
|
||
|---|---|---|
|
||
| Auth | none — plain HTTP GET | OAuth, keys, 7-day token refresh |
|
||
| Realtime | polled, possibly ~10 min delayed | true push websocket, realtime |
|
||
| 1m history | **8 days** (per-request cap) | ❌ none |
|
||
| 1h history | **~730 days** (17,387 bars, verified back to 2024-03) | ❌ none |
|
||
| 1d history | **~10 years** (2,517 bars, verified back to 2016) | ❌ none |
|
||
| Contract | continuous front-month, roll gaps | true contract |
|
||
|
||
Endpoint, verified working with no key and no `yfinance` dependency:
|
||
|
||
```
|
||
https://query1.finance.yahoo.com/v8/finance/chart/ES=F?interval=1h&range=730d
|
||
```
|
||
|
||
Requires a browser `User-Agent` header. Returns `chart.result[0]` with `timestamp[]`
|
||
and `indicators.quote[0].{open,high,low,close,volume}` as parallel arrays.
|
||
**Those arrays contain `null` holes — filter them before constructing `Bar`s.**
|
||
|
||
Therefore define one protocol and two implementations:
|
||
|
||
```python
|
||
class MarketDataSource(Protocol):
|
||
name: str
|
||
def supports_history(self) -> bool: ...
|
||
async def history(self, symbol, tf, start, end) -> list[Bar]: ...
|
||
def supports_stream(self) -> bool: ...
|
||
async def stream(self, symbol) -> AsyncIterator[Bar]: ... # yields 1m bars
|
||
```
|
||
|
||
- `YahooSource` — real `history()`. `stream()` is a **polling loop** (every 15–30 s,
|
||
`interval=1m&range=1d`, emit only bars newer than the last emitted) that presents the
|
||
same async-iterator interface as a real push stream.
|
||
- `SchwabSource` — `supports_history() -> False`. `stream()` is the true
|
||
`CHART_FUTURES` websocket.
|
||
- `ReplaySource` — reads a JSONL tape. Used by tests and offline development.
|
||
|
||
Nothing downstream of these may know which source it is using. Selection is one env
|
||
var. **In production both run at once:** Yahoo seeds history at startup, Schwab
|
||
provides the live tail.
|
||
|
||
### Do not use Yahoo's daily bars
|
||
|
||
Yahoo anchors `ES=F` daily bars to **midnight ET**, but the CME futures session runs
|
||
**18:00 → 17:00 ET** (§6). Mixing the two definitions yields daily candles that
|
||
disagree with every other chart you'll compare against.
|
||
|
||
**Build daily bars yourself by aggregating Yahoo's 1h bars** through the same
|
||
`aggregator.py` used for live data — one session definition everywhere. Yahoo's 1h bars
|
||
are anchored to the top of the ET hour, so they compose into session-anchored 1d
|
||
buckets cleanly. The ~730-day 1h window yields ~500 sessions: enough for a daily 200SMA
|
||
(~200 sessions) with room to spare.
|
||
|
||
Yahoo's native 1d bars may be used *only* for multi-year context, clearly labelled.
|
||
|
||
## 2.2 Schwab entitlements — **answered empirically 2026-08-10**
|
||
|
||
All verified against a live production app holding both Market Data Production and
|
||
Accounts and Trading Production.
|
||
|
||
| Question | Answer |
|
||
|---|---|
|
||
| Futures market data entitled? | **Yes** — but only via `get_quotes()` (plural) |
|
||
| Symbol format | **`/ES`**, which auto-resolves to the active contract `/ESU26` |
|
||
| `CHART_FUTURES` streaming | **Works** — one true-OHLCV minute bar per symbol per minute |
|
||
| `LEVEL_ONE_FUTURES` | **Works**, and reports `delayed: false` |
|
||
| Futures price history | **Still none.** Yahoo remains the only source of the past |
|
||
|
||
Three traps found the hard way, all of which cost a round trip:
|
||
|
||
- **`get_quote()` (singular) silently returns the wrong instrument.** It puts the
|
||
symbol in the URL *path*, where the leading slash is normalised away, so `/ES`
|
||
comes back as `ES` — Eversource Energy, an equity, at $72. HTTP 200 with a
|
||
populated body. `get_quotes()` passes symbols as a query parameter and returns
|
||
the future correctly. **Never treat a 200 as proof; check `assetMainType`.**
|
||
- **Streaming needs the Accounts and Trading product.** `StreamClient.login()`
|
||
reads `/trader/v1/userPreference` for its socket URL, and that path is not in
|
||
Market Data Production. A market-data-only app cannot stream at all.
|
||
- **Authorisation codes expire in about thirty seconds**, and an unwritable token
|
||
path spends one before revealing itself. `scripts/check_schwab.py` preflights
|
||
the key, the secret and the token path for exactly this reason.
|
||
|
||
Because `/ES` resolves to the active contract on Schwab's side, contract roll
|
||
handling — an open problem in §10 — needs no code here.
|
||
|
||
### The remaining unknowns for Schwab
|
||
|
||
1. **Futures market-data entitlement.** It is not publicly documented whether
|
||
`CHART_FUTURES` requires futures trading approval or a CME non-professional market
|
||
data agreement on the Schwab account. Verify empirically in M6.
|
||
2. **Symbol format.** `schwab-py` docs show both `/ES` (continuous front-month) and
|
||
`/ESZ25` (specific contract). Determine which the stream actually accepts, and
|
||
whether the continuous form auto-rolls. Record the answer in the README.
|
||
3. **Bar cadence and lateness.** Confirm `CHART_FUTURES` emits one message per symbol
|
||
per minute, whether it re-sends a bar (correction), and how bars behave across the
|
||
17:00–18:00 ET settlement break.
|
||
4. **Volume semantics.** Confirm `VOLUME` is per-minute, not cumulative-for-session.
|
||
|
||
### Confirmed facts (already verified — do not re-research)
|
||
|
||
**Realtime futures data: fully available.** Do not let §10 below suggest otherwise —
|
||
these are different axes and conflating them will send you down the wrong path.
|
||
|
||
| Service | Available? | Use here |
|
||
|---|---|---|
|
||
| `CHART_FUTURES` (1-min OHLCV, `/ES`) | ✅ streaming | The base feed. Everything derives from it |
|
||
| `LEVEL_ONE_FUTURES` (live quotes) | ✅ streaming | Current price/bid/ask for the status bar and alert evaluation |
|
||
| `LEVEL_ONE_FUTURES_OPTIONS` | ✅ streaming | Live pricing of a proposed /ES put spread in M7 |
|
||
| REST `get_price_history` for futures | ❌ **not available** | — see §10 |
|
||
| `CHART_HISTORY_FUTURES` service | ❌ does not exist in `schwab-py` | — |
|
||
| Futures / futures-options **order entry** | ❌ not offered | see §14 |
|
||
|
||
- `ChartFuturesFields`: `SYMBOL=0, CHART_TIME_MILLIS=1, OPEN_PRICE=2, HIGH_PRICE=3,
|
||
LOW_PRICE=4, CLOSE_PRICE=5, VOLUME=6`.
|
||
- **The gap is historical only, not realtime.** schwab-py's docs, verbatim: *"Schwab
|
||
provides price history for equities and ETFs. It does not provide price history for
|
||
options, futures, or any other instruments."* A Schwab account does not change this —
|
||
it is an API-surface limitation, not an entitlement one. Live `/ES` streams fine;
|
||
there is simply no way to ask for *yesterday's* `/ES` bars. This is the single
|
||
biggest constraint in the project; see [§10](#10-the-history-problem).
|
||
- Lightweight Charts 5.2.0 standalone build exposes a `window.LightweightCharts`
|
||
global containing `createChart`, `CandlestickSeries`, `LineSeries`,
|
||
`createSeriesMarkers`, `LineStyle`. CDN:
|
||
`https://unpkg.com/lightweight-charts@5.2.0/dist/lightweight-charts.standalone.production.js`
|
||
(~196 KB).
|
||
- **v5 changed the series API.** Use `chart.addSeries(LightweightCharts.CandlestickSeries, opts)`.
|
||
The v4 `chart.addCandlestickSeries(opts)` form **does not exist in v5** — most
|
||
tutorials online are v4 and will not work.
|
||
|
||
---
|
||
|
||
## 3. Architecture
|
||
|
||
```
|
||
YahooSource SchwabSource ReplaySource
|
||
(history + poll) (live websocket) (JSONL tape)
|
||
└──────────────────────┼──────────────────────┘
|
||
│ MarketDataSource protocol
|
||
▼ 1-minute OHLCV
|
||
┌─────────────────┐
|
||
│ StreamService │ single asyncio task, ONE connection
|
||
│ (+ recorder) │──────► raw JSONL tape (for replay)
|
||
└────────┬────────┘
|
||
│ Bar(1m)
|
||
▼
|
||
┌─────────────────┐
|
||
│ Aggregator │ session-aware bucketing
|
||
└────────┬────────┘
|
||
│ Bar(tf) closed / updated
|
||
┌──────────────┼──────────────┐
|
||
▼ ▼ ▼
|
||
┌──────────┐ ┌────────────┐ ┌──────────┐
|
||
│ BarStore │ │ Pivots → │ │ Moving │
|
||
│ (memory) │ │ Trendlines │ │ Averages │
|
||
└──────────┘ └─────┬──────┘ └────┬─────┘
|
||
│ │
|
||
└──────┬───────┘
|
||
▼ Level[] (unified type)
|
||
┌──────────────────┐
|
||
│ ConfluenceEngine │ cluster + score
|
||
└────────┬─────────┘
|
||
▼ Cluster[]
|
||
┌──────────────────┐
|
||
│ AlertEngine │ state machine + cooldown
|
||
└────────┬─────────┘
|
||
│
|
||
┌──────────────┴───────────────┐
|
||
▼ ▼
|
||
FastAPI WebSocket ntfy push
|
||
│
|
||
▼
|
||
Vue 3 + Lightweight Charts
|
||
```
|
||
|
||
### Critical process constraint
|
||
|
||
The Schwab stream is **one connection, one process, not thread-safe**. Therefore:
|
||
|
||
- Run the streamer as a single `asyncio` task owned by FastAPI's `lifespan`.
|
||
- **`uvicorn --workers 1` always.** More than one worker means more than one Schwab
|
||
connection, which will fight over the session.
|
||
- `--reload` in dev will tear down and re-establish the stream on every file save.
|
||
That is acceptable locally but expect reconnect churn.
|
||
- If the VPS deploy later needs multiple web workers, the streamer must be split into
|
||
its own process with a message bus. Do not design for that now, but keep
|
||
`StreamService` free of any FastAPI imports so the split stays cheap.
|
||
|
||
---
|
||
|
||
## 4. Module layout
|
||
|
||
```
|
||
main.py FastAPI app: lifespan, route mounting (exists, extend)
|
||
app/
|
||
config.py Settings from env (pydantic-settings)
|
||
auth.py Shared-secret gate (no-op when unset)
|
||
market/
|
||
base.py MarketDataSource protocol, Bar emission contract
|
||
yahoo.py YahooSource: history() + polled stream() ← build first
|
||
schwab.py SchwabSource: easy_client, CHART_FUTURES websocket
|
||
replay.py ReplaySource: JSONL tape
|
||
stream.py StreamService: owns a source, reconnect, emit Bar
|
||
recorder.py Record raw messages to JSONL
|
||
bars/
|
||
models.py Bar, Timeframe
|
||
session.py CME session calendar + bucket boundary math
|
||
aggregator.py 1m -> all timeframes
|
||
store.py BarStore protocol + InMemoryBarStore
|
||
analysis/
|
||
indicators.py ATR, SMA, EMA (pure functions over bar lists)
|
||
manual_lines.py Hand-drawn lines: CRUD + JSON persistence ← M5
|
||
pivots.py Fractal swing detection ← M8, deferred
|
||
trendlines.py Candidate generation, scoring, dedup ← M8, deferred
|
||
moving_averages.py MTF MA levels
|
||
levels.py Level type + registry, rebuild orchestration
|
||
confluence.py Clustering + scoring
|
||
alerts.py State machine, cooldown, dispatch
|
||
notify/
|
||
ntfy.py Phone push
|
||
api/
|
||
routes.py REST
|
||
ws.py WebSocket hub
|
||
static/
|
||
index.html Vue 3 + LWC script tags (exists, replace)
|
||
app.js Vue app root (exists, replace)
|
||
chart.js Lightweight Charts wrapper (new)
|
||
style.css (exists, extend)
|
||
tests/
|
||
...
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Data model
|
||
|
||
Use dataclasses (or pydantic where it crosses the API boundary). All times are
|
||
**epoch seconds, UTC**. Never store naive local datetimes.
|
||
|
||
```python
|
||
class Timeframe(str, Enum):
|
||
M1="1m"; M2="2m"; M5="5m"; M15="15m"; M30="30m"; H1="1h"; D1="1d"
|
||
|
||
@property
|
||
def seconds(self) -> int: ... # D1 is session-defined, not 86400 — see §6
|
||
|
||
@dataclass
|
||
class Bar:
|
||
tf: Timeframe
|
||
t: int # epoch seconds, bucket OPEN time
|
||
o: float; h: float; l: float; c: float
|
||
v: int
|
||
closed: bool # False while still forming
|
||
symbol: str # e.g. "/ESZ25" — carried so contract rolls stay visible
|
||
```
|
||
|
||
`Level` is the unified abstraction that makes the whole design work. Trendlines and
|
||
moving averages both reduce to "a price at time t, with a weight and a side".
|
||
|
||
```python
|
||
class LevelKind(str, Enum):
|
||
MANUAL="manual" # hand-drawn — ships first (M5)
|
||
MA="ma" # moving average — ships first (M3)
|
||
TRENDLINE="trendline" # auto-detected — deferred to M8
|
||
HORIZONTAL="horizontal"
|
||
|
||
class Side(str, Enum):
|
||
SUPPORT="support"; RESISTANCE="resistance"
|
||
|
||
@dataclass
|
||
class Level:
|
||
id: str # stable hash — the UI diffs on this, so keep it stable
|
||
kind: LevelKind
|
||
tf: Timeframe
|
||
side: Side
|
||
weight: float # timeframe weight x quality multiplier
|
||
score: float # raw quality score before weighting
|
||
label: str # "4h resistance", "1h EMA21"
|
||
|
||
# Geometry. Trendline: price(t) = slope*(t - anchor_t) + anchor_p
|
||
anchor_t: int
|
||
anchor_p: float
|
||
slope: float # price units per SECOND. 0.0 for horizontal/MA-at-instant
|
||
|
||
# For MAs: the stepped polyline actually drawn
|
||
points: list[tuple[int, float]] | None
|
||
|
||
touches: int
|
||
first_t: int
|
||
last_t: int
|
||
provisional: bool # derived from a still-forming bar
|
||
hidden: bool # layer-panel visibility — see §9.4 for its effect on scoring
|
||
|
||
def price_at(self, t: int) -> float:
|
||
return self.anchor_p + self.slope * (t - self.anchor_t)
|
||
```
|
||
|
||
The weight table referenced throughout — define it once, in `config.py`:
|
||
|
||
```python
|
||
TIMEFRAME_WEIGHT = {
|
||
Timeframe.M1: 1, Timeframe.M2: 1, Timeframe.M5: 1, Timeframe.M15: 2,
|
||
Timeframe.M30: 3, Timeframe.H1: 4, Timeframe.D1: 16,
|
||
}
|
||
MA_WEIGHT_FACTOR = 0.75 # §7.4 — MAs weigh slightly less than drawn structure
|
||
```
|
||
|
||
```python
|
||
@dataclass
|
||
class Cluster:
|
||
id: str
|
||
side: Side
|
||
low: float; high: float; center: float
|
||
score: float # sum of member weights
|
||
members: list[Level]
|
||
distance: float # signed points from current price
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Session and aggregation rules — read carefully
|
||
|
||
This is where a naive implementation silently produces wrong lines. CME ES is not a
|
||
9:30–16:00 instrument.
|
||
|
||
**Session definition (America/New_York, DST-aware via `zoneinfo`):**
|
||
|
||
- Trading week opens **Sunday 18:00 ET**.
|
||
- Daily settlement break **17:00–18:00 ET, Monday–Thursday**. No bars expected.
|
||
- Week closes **Friday 17:00 ET**.
|
||
- A **futures "day"** runs 18:00 ET → 17:00 ET the following calendar day, and is
|
||
conventionally labelled with the *following* calendar date. Sunday 18:00 bars belong
|
||
to Monday's daily bar.
|
||
|
||
**Bucketing rules:**
|
||
|
||
- `1m, 2m, 5m, 15m, 30m, 1h` — bucket on wall-clock UTC boundaries. These divide the
|
||
hour evenly, so session anchoring is unnecessary and UTC keeps it simple.
|
||
- `1d` — one bar per futures session as defined above. **This is the only
|
||
session-anchored timeframe.** 4h used to be the other one and was the reason this
|
||
section warned about DST; with 4h gone, that whole class of edge case went with it.
|
||
|
||
Implement this as `session.py::bucket_start(t: int, tf: Timeframe) -> int` and unit
|
||
test it hard, including both DST transitions and the Sunday open. **This function is
|
||
the highest-risk piece of pure logic in the project.** Write its tests first.
|
||
|
||
**Aggregator behaviour:**
|
||
|
||
- Maintain one forming bar per timeframe. On each incoming 1m bar:
|
||
- if `bucket_start(bar.t, tf)` differs from the forming bar's `t`, close the forming
|
||
bar (emit `closed=True`), then open a new one;
|
||
- otherwise fold in: `h=max`, `l=min`, `c=close`, `v+=`, emit `closed=False`.
|
||
- **Gaps do not close bars by time — they close by the arrival of a later bar.** Never
|
||
use a wall-clock timer to close a bucket; the market halts and holidays will fire it
|
||
incorrectly. Exception: emit a `closed=True` for the previous bucket when a bar
|
||
arrives that skips buckets entirely.
|
||
- Aggregation must be **deterministic and replayable**: feeding the same 1m sequence
|
||
twice must produce byte-identical output. No `datetime.now()` inside the aggregator.
|
||
|
||
---
|
||
|
||
## 7. Analysis engines
|
||
|
||
### 7.1 Indicators (`indicators.py`)
|
||
|
||
Pure functions, list-in/list-out, no state: `sma(values, period)`, `ema(values, period)`,
|
||
`atr(bars, period=14)`. ATR is the universal scale unit — every tolerance in this
|
||
project is expressed in ATR multiples, never in fixed points, so the same config works
|
||
whether ES is at 4,000 or 8,000.
|
||
|
||
### 7.2 Pivot detection (`pivots.py`)
|
||
|
||
Fractal method, not regression. Regression fits the *middle* of price action; humans
|
||
draw lines across *extremes*, and extremes are what other traders react to.
|
||
|
||
```
|
||
● pivot high (w bars lower on both sides)
|
||
/ \
|
||
/ \
|
||
● / \
|
||
/ \ / \
|
||
───●───●──────────────
|
||
pivot low
|
||
```
|
||
|
||
- `pivot_high(bars, w)`: index `i` qualifies if `high[i] >= max(high[i-w : i+w+1])`
|
||
and `i` is the leftmost such index in ties.
|
||
- Default `w = 3` for lower TFs, `w = 2` for `4h`/`1d` (fewer bars available).
|
||
- **Confirmation lag is `w` bars — this is intentional.** A pivot is only known `w`
|
||
bars after it forms. Do not "detect" pivots on the forming bar; that repaints, and a
|
||
repainting line is worse than no line.
|
||
- Significance filter: keep a pivot only if its prominence (distance to the
|
||
surrounding swing in the opposite direction) `>= 0.5 * ATR(14)` on that timeframe.
|
||
Drops noise pivots without hardcoding point values.
|
||
|
||
### 7.3 Automatic trendlines (`trendlines.py`) — **deferred to M8**
|
||
|
||
> Not part of the initial build. **Manual trendlines (§7.3a) ship first.** This section
|
||
> is retained because it is the eventual target and because §7.3a is deliberately
|
||
> designed to produce the same `Level` objects, so adopting this later changes nothing
|
||
> downstream. Skip to §7.3a on a first pass.
|
||
|
||
```
|
||
for each timeframe, for each side (highs → resistance, lows → support):
|
||
P = last N qualifying pivots (N = 25; ~300 candidate pairs, trivial)
|
||
for each pair (a, b) in P where a.t < b.t:
|
||
line = through (a.t, a.p) and (b.t, b.p)
|
||
evaluate(line) -> score or reject
|
||
dedup, keep top K = 4 per side per timeframe
|
||
```
|
||
|
||
**Evaluation of a candidate line:**
|
||
|
||
Let `tol = 0.25 * ATR(14)` on that timeframe.
|
||
|
||
- **Violation** — a bar *closes* beyond the line by more than `tol`
|
||
(above for resistance, below for support). Wicks do not count as violations; wicks
|
||
through a level are normal and often the point.
|
||
- **Touch** — a bar's extreme comes within `tol` of the line without violating it.
|
||
- **Reject** the line if `violations > 1` between the two anchors, or if any violation
|
||
occurred after the later anchor (the line is broken and no longer active).
|
||
|
||
**Score:**
|
||
|
||
```
|
||
score = 3.0 * touches
|
||
+ 1.5 * log1p(span_in_bars)
|
||
+ 2.0 * recency_decay(last_touch) # exp(-age_bars / halflife), halflife=50
|
||
- 4.0 * violations
|
||
```
|
||
|
||
Normalize within each timeframe/side group so weights stay comparable across
|
||
timeframes regardless of how many candidates a given timeframe happened to produce:
|
||
|
||
```python
|
||
best = max(l.score for l in group) # after dedup, before truncation to K
|
||
quality = clamp(l.score / best, 0.0, 1.0) if best > 0 else 0.0
|
||
level.weight = quality * TIMEFRAME_WEIGHT[tf]
|
||
```
|
||
|
||
So the strongest 4h line contributes the full 8.0, a mediocre one proportionally less,
|
||
and a 5m line can never outweigh a 4h line no matter how many touches it has.
|
||
|
||
**The four coefficients above (3.0 / 1.5 / 2.0 / 4.0) are starting values, not
|
||
derived constants.** They cannot be got right on paper — expect to tune them by
|
||
eye against replayed tapes in M4. Put them in `config.py`, not inline, and treat "the
|
||
lines land where a human would draw them" as the acceptance criterion.
|
||
|
||
**Dedup:** two lines are duplicates if, evaluated at *now*, their prices are within
|
||
`tol` **and** their slopes differ by less than 20%. Keep the higher score. Without
|
||
this you get a fan of ten nearly-identical lines from the same swing.
|
||
|
||
**Recompute policy:** only on a **bar close** for that timeframe, never on every tick.
|
||
A 4h line recomputes 6× per day. This is what keeps the whole thing cheap.
|
||
|
||
### 7.3a Manual trendlines (`manual_lines.py`) — ships in M5
|
||
|
||
A hand-drawn line is just a `Level` with `kind=MANUAL`. It flows into confluence,
|
||
projection, and alerts through exactly the same path as everything else — that is what
|
||
makes deferring the automatic engine cheap rather than a detour.
|
||
|
||
**Why this ordering is better than it looks:** hand-drawn lines are *correct by
|
||
definition* (you drew them). So they validate the confluence engine without the
|
||
automatic detector's tuning risk, and they later become the ground truth that M8's
|
||
scoring coefficients get tuned against.
|
||
|
||
**Geometry.** Anchors are persisted as **absolute epoch seconds and price** — never
|
||
bar indices — so changing the algorithm does not rewrite drawings. A sloped line is
|
||
evaluated in the logical bar space of its attributed timeframe, however: charts
|
||
compress a weekend to one slot, so wall-clock `price_at(t)` would advance through 49
|
||
hours in which no bars exist. The stored `slope` recovers the second endpoint price;
|
||
the source-timeframe bar sequence determines interpolation and projection. The
|
||
browser samples that canonical geometry at displayed candle timestamps and at
|
||
the existing future-whitespace timestamps. The server uses the same source series
|
||
for clusters and alerts. A source history that no longer reaches an anchor leaves
|
||
the drawing visible but unresolved and unable to cluster or alert rather than
|
||
silently extrapolating from the edge of a shorter window.
|
||
`TRENDLINE_SOURCE_GEOMETRY=false` restores the previous displayed/1m-grid behavior
|
||
without changing persisted data.
|
||
|
||
**Timeframe attribution.** Tag the line with the timeframe that was *displayed when it
|
||
was drawn*. A line drawn on the 4h chart is a 4h line and carries weight 8. This is the
|
||
single most important field — without it every manual line would score identically.
|
||
|
||
**Weight.** `quality = 1.0` always. The user drew it; it is not a candidate to be
|
||
scored. `weight = TIMEFRAME_WEIGHT[tf]`.
|
||
|
||
**Drawing interaction** (all APIs verified present in LWC 5.2.0):
|
||
|
||
| Action | Implementation |
|
||
|---|---|
|
||
| Enter draw mode | Toolbar button; changes cursor |
|
||
| Place endpoints | `chart.subscribeClick(handler)` → two clicks |
|
||
| Pixel → price | `series.coordinateToPrice(param.point.y)` |
|
||
| Pixel → time | `chart.timeScale().coordinateToTime(param.point.x)` |
|
||
| Render | One `LineSeries`, sampled onto displayed candles and existing future-whitespace slots; SVG only bridges off-grid endpoints and provides interaction overlays |
|
||
| Higher-TF style | When viewed below its attributed timeframe, a manual trendline is dashed and rendered at twice its stored width; native/lower-TF views use the stored width and solid style |
|
||
| Select | Click within ~6px of the canonical price at that displayed bar |
|
||
| Delete | `Delete`/`Backspace` on selection, plus a button |
|
||
| Edit | Endpoint handles, whole-line drag, keyboard nudge, cutoff and duplicate; time shifts use source bars |
|
||
|
||
**Snapping.** When placing an endpoint, snap to the nearest bar high/low within ~8px.
|
||
Cheap to implement and it is the difference between a usable drawing tool and a
|
||
frustrating one. Make it toggleable; snap to high for resistance, low for support,
|
||
inferred from drag direction or nearest extreme.
|
||
|
||
**Persistence — required in M5, not deferred.** A user who redraws their lines after
|
||
every restart abandons the tool. This does *not* require the bar store or a database:
|
||
write to a **JSON file** (`data/manual_lines.json`), loaded at startup. Bar persistence
|
||
can stay deferred to M7; these two are unrelated decisions.
|
||
|
||
```json
|
||
{"id":"ml_01H...","tf":"4h","side":"resistance","anchor_t":1754600000,
|
||
"anchor_p":6412.5,"slope":-0.0000031,"created_at":1754700000,"note":"","hidden":false}
|
||
```
|
||
|
||
CRUD via `POST /api/lines`, `DELETE /api/lines/{id}`, `PATCH /api/lines/{id}`. On any
|
||
change the server recomputes levels and broadcasts `{"type":"levels",...}` — the
|
||
drawing client must render optimistically and then reconcile, not wait on the round
|
||
trip.
|
||
|
||
### 7.4 Multi-timeframe moving averages (`moving_averages.py`)
|
||
|
||
**Primary requirement: the daily MA set — SMA 10, 20, 50, 100, 200 — computed on daily
|
||
bars and displayed on the 1d, 30m, and 1m charts, with the base timeframe switchable.**
|
||
|
||
This is the headline use of the multi-timeframe projection: the 200DMA is a level
|
||
people trade against, and on a 1-minute chart it should appear as a near-horizontal
|
||
line that steps once per session. Do not compute a "200-period MA of 1-minute bars" for
|
||
the 1m chart — that is a completely different and far less useful line. **The MA's
|
||
period is always tied to the timeframe it was computed on, never to the chart being
|
||
displayed.**
|
||
|
||
```
|
||
MA_SETS = {
|
||
"1d": [("sma", 10), ("sma", 20), ("sma", 50), ("sma", 100), ("sma", 200)],
|
||
# optional, off by default:
|
||
"1h": [("ema", 9), ("ema", 21)],
|
||
}
|
||
BASE_TIMEFRAMES = ["1m", "30m", "1d"] # the switcher; others remain available
|
||
```
|
||
|
||
Config-driven, so adding a set is a config change, not code. Other timeframes' MAs stay
|
||
supported by the same machinery but ship disabled — the daily set is what matters.
|
||
|
||
**History check:** a 200DMA needs 200 sessions. Yahoo's 730-day 1h window yielded
|
||
17,387 bars ≈ 750 sessions, so all five daily MAs are warm from startup with roughly
|
||
3× margin. Verified, not assumed.
|
||
|
||
**Expect small disagreements with thinkorswim/TradingView on the daily MAs.** We build
|
||
daily bars on the CME session (18:00→17:00 ET, §6); other platforms sometimes anchor
|
||
differently or use settlement prices. A one- or two-point difference in the 200DMA is
|
||
this, not a bug. Keep the daily anchor configurable so it can be matched if it matters.
|
||
|
||
The key rendering decision: an MA from a higher timeframe drawn on a 1-minute chart is
|
||
a **step function**, held constant between higher-timeframe closes.
|
||
|
||
```
|
||
1h EMA21 rendered on a 1m chart:
|
||
|
||
┌────────
|
||
┌──┘ ← steps at each 1h close, NOT a smooth interpolation
|
||
┌──┘
|
||
```
|
||
|
||
Interpolating between higher-TF closes would draw a line that was never true at the
|
||
time it appears to have been true. Emit `points` as a stepped polyline and render with
|
||
LWC's `lineType: LightweightCharts.LineType.WithSteps`.
|
||
|
||
- The value from the **forming** higher-TF bar is emitted with `provisional=True` and
|
||
rendered dashed. It *will* move until that bar closes — that is honest, not a bug.
|
||
- An MA's **current value is a price level**, so it enters the confluence engine on
|
||
equal footing with trendlines, at `0.75 × timeframe_weight` (MAs are slightly less
|
||
reactive than drawn structure, but a 4h 200SMA is still a wall).
|
||
- **Warm-up:** an MA needs `period` closed bars on its timeframe. A daily 200SMA needs
|
||
200 sessions. Until warm, emit nothing — never emit a partially-warmed MA. Yahoo
|
||
seeding (§10) makes every MA up to a daily 200SMA warm from startup; without it,
|
||
the higher-timeframe MAs stay cold for months.
|
||
|
||
### 7.5 Confluence (`confluence.py`)
|
||
|
||
This is the actual product. Everything above exists to feed it.
|
||
|
||
```
|
||
1. Collect every active Level, evaluate price_at(now).
|
||
2. Split by side relative to current price (levels above → resistance, below → support).
|
||
A level's stored `side` is its structural nature; its *effective* side is
|
||
positional. Use positional — a broken resistance acting as support is the
|
||
interesting case, not an error.
|
||
3. Sort by price. Single-linkage cluster: extend the current cluster while the gap to
|
||
the next level is <= clusterTol.
|
||
clusterTol = 0.4 * ATR14(15m)
|
||
4. Cluster score = sum of member weights.
|
||
5. Emit clusters with >= 2 members OR score >= 8 (a lone 1d line matters by itself).
|
||
```
|
||
|
||
```
|
||
5m resistance 6404.25 weight 1
|
||
15m resistance 6405.00 weight 2
|
||
1h resistance 6403.75 weight 4
|
||
4h resistance 6404.50 weight 8
|
||
↓
|
||
RESISTANCE CLUSTER 6403.75 – 6405.00
|
||
CONFLUENCE 15
|
||
```
|
||
|
||
Recompute on every closed 1m bar, and on any level rebuild.
|
||
|
||
### 7.6 Alerts (`alerts.py`)
|
||
|
||
The failure mode to design against is notification fatigue. A per-line alert makes the
|
||
system useless within a day.
|
||
|
||
Per-cluster state machine, keyed on `(side, round(center / clusterTol))` so a cluster
|
||
keeps its identity as members drift:
|
||
|
||
```
|
||
ARMED ──price within alertTol of cluster──► FIRED ──► COOLDOWN ──┐
|
||
▲ │
|
||
└────── price moves > 2*alertTol away AND cooldown elapsed ◄────┘
|
||
```
|
||
|
||
- `alertTol = 0.5 * ATR14(15m)` by default. A hand-placed price level may set
|
||
`alert_early_points`; resistance then qualifies that many points below the
|
||
level and support that many points above it. This changes notification timing,
|
||
not the level's chart geometry.
|
||
- `cooldown = 15 minutes`
|
||
- Minimum score threshold to fire: **configurable, starting value 6 — but this
|
||
certainly needs recalibrating in M4.** With the daily MA set as the primary levels,
|
||
each daily MA carries `0.75 × 16 = 12`, so *any two* of them near each other scores
|
||
24 and a threshold of 6 would fire constantly. Either raise the threshold well above
|
||
24, or damp the weight when several MAs from the same timeframe cluster (they are not
|
||
independent evidence the way a 4h line and a 1h line are). Decide this against a
|
||
replayed tape, not on paper. **Target: single-digit alerts per session.**
|
||
- Re-arming requires *both* the price separation and the cooldown. Time alone lets a
|
||
price oscillating on a level fire forever.
|
||
- Fire on **closed 1m bars only**, not intra-bar ticks.
|
||
|
||
Dispatch to: WebSocket (UI banner + sound) and ntfy. Payload:
|
||
|
||
```
|
||
BEARISH ZONE /ES 6404.25
|
||
Resistance confluence 15 @ 6403.75–6405.00
|
||
5m, 15m, 1h, 4h
|
||
```
|
||
|
||
---
|
||
|
||
## 8. API contract
|
||
|
||
Fix these shapes now; the frontend and backend are built against them.
|
||
|
||
### REST
|
||
|
||
| Route | Returns |
|
||
|---|---|
|
||
| `GET /api/status` | `{stream: "connected"\|"disconnected"\|"replay", symbol, last_bar_t, bars_held: {tf: n}, warm: {tf: bool}}` |
|
||
| `GET /api/bars?tf=5m&limit=500` | `{tf, bars: [{t,o,h,l,c,v,closed}]}` — oldest first |
|
||
| `GET /api/levels?tf=all` | `{levels: [Level]}` |
|
||
| `GET /api/confluence` | `{price, clusters: [Cluster]}` |
|
||
| `GET /api/health` | existing |
|
||
|
||
### WebSocket `/ws`
|
||
|
||
Client → server on connect:
|
||
```json
|
||
{"type": "subscribe", "tf": "5m"}
|
||
```
|
||
|
||
Server → client:
|
||
```json
|
||
{"type":"snapshot","tf":"5m","bars":[...],"levels":[...],"clusters":[...],"price":6404.25}
|
||
{"type":"bar","tf":"5m","bar":{"t":1754700000,"o":6403.5,"h":6405.0,"l":6403.0,"c":6404.25,"v":812,"closed":false}}
|
||
{"type":"levels","levels":[...]}
|
||
{"type":"clusters","price":6404.25,"clusters":[...]}
|
||
{"type":"alert","cluster":{...},"message":"..."}
|
||
{"type":"status","stream":"disconnected"}
|
||
```
|
||
|
||
Rules:
|
||
- Send `bar` on **every** update of the forming bar (that is the live chart) but batch
|
||
`levels` — they only change on higher-TF closes.
|
||
- Always send a full `snapshot` on connect and after any reconnect. The client must
|
||
never try to reconcile a gap.
|
||
- Levels are sent for **all** timeframes regardless of the displayed timeframe. That is
|
||
the entire point: a 4h line drawn through a 1m chart.
|
||
|
||
---
|
||
|
||
## 9. Frontend
|
||
|
||
`static/index.html` loads Vue 3 and Lightweight Charts as script tags — no bundler,
|
||
matching the existing app.
|
||
|
||
```html
|
||
<script src="https://unpkg.com/vue@3/dist/vue.global.prod.js"></script>
|
||
<script src="https://unpkg.com/lightweight-charts@5.2.0/dist/lightweight-charts.standalone.production.js"></script>
|
||
```
|
||
|
||
### Vue + Lightweight Charts integration — the one real gotcha
|
||
|
||
**Never put the chart or series objects in `ref()` or `reactive()`.** Vue's deep
|
||
reactive proxy will wrap the library's internal objects, which breaks identity checks
|
||
inside the library and destroys performance on every update. Use `shallowRef`, or
|
||
better, a plain module-scoped variable / `markRaw`.
|
||
|
||
```js
|
||
const chartApi = shallowRef(null); // ✅
|
||
// const chart = ref(null); // ❌ will appear to work, then misbehave
|
||
```
|
||
|
||
### Logical indices address the whole chart, not your bar array
|
||
|
||
**Never derive a viewport from `bars.length`.** A logical index addresses the
|
||
chart's *shared* time scale — the union of the time points of every series on
|
||
it — not the candle array. Any series whose points pre-date the candle window
|
||
prepends to that scale and shifts every logical index by its count.
|
||
|
||
```js
|
||
// ❌ off by however many points the other series contribute
|
||
timeScale().setVisibleLogicalRange({ from: bars.length - 160, to: bars.length + 5 });
|
||
// ✅ an instant cannot be renumbered by a later series
|
||
timeScale().setVisibleRange({ from: bars.at(-160).t, to: bars.at(-1).t + step * 5 });
|
||
```
|
||
|
||
This is not theoretical — see the 2026-08-10 entry in §16. The daily MAs carry
|
||
one point per daily bar (617 of them, back ~2 years). They are attached by
|
||
`syncVisibleLevels()` *immediately after* `setBars()`, so a logical range that
|
||
was correct when set silently slid 617 bars — about ten hours — into the past
|
||
one tick later. The chart looked frozen while the socket was perfectly healthy.
|
||
|
||
Structure:
|
||
- `chart.js` — a plain, framework-free wrapper class owning the LWC instance:
|
||
`create(el)`, `setBars()`, `updateBar()`, `syncLevels(levels)`, `destroy()`.
|
||
- `app.js` — Vue app owning state (timeframe, connection status, clusters, alert log)
|
||
and the WebSocket. Calls into the wrapper imperatively in `onMounted` / watchers.
|
||
- Chart lifecycle in `onMounted`; `chart.remove()` in `onUnmounted`; a `ResizeObserver`
|
||
driving `chart.applyOptions({width, height})`.
|
||
|
||
### v5 API usage
|
||
|
||
```js
|
||
const chart = LightweightCharts.createChart(el, {...});
|
||
const candles = chart.addSeries(LightweightCharts.CandlestickSeries, {...});
|
||
const line = chart.addSeries(LightweightCharts.LineSeries, {
|
||
lineType: LightweightCharts.LineType.WithSteps, // for MTF moving averages
|
||
});
|
||
candles.setData(bars); // once
|
||
candles.update(bar); // every tick — never re-call setData
|
||
```
|
||
|
||
### Rendering levels
|
||
|
||
`syncLevels()` must **diff by `Level.id`**, not clear-and-rebuild. Rebuilding every
|
||
series on each update causes visible flicker and leaks series objects.
|
||
|
||
- Trendline → a `LineSeries` with two points: `(anchor_t, anchor_p)` and
|
||
`(now + rightExtension, price_at(now + rightExtension))`. Extend ~20% of the visible
|
||
range into the future so the line is usable ahead of price.
|
||
- MA → a `LineSeries` fed the stepped `points` array.
|
||
- Colour **by timeframe** (one hue per TF, consistent everywhere including the
|
||
confluence panel). Line width scales with timeframe weight. `provisional` levels
|
||
render dashed via `lineStyle: LightweightCharts.LineStyle.Dashed`.
|
||
|
||
### 9.4 Layer panel (checkboxes)
|
||
|
||
Ships with M3. Visibility control is load-bearing here, not decoration — five daily MAs
|
||
plus manual lines plus any optional intraday MA sets stack up fast.
|
||
|
||
```
|
||
CHART [ 1m ] [ 30m ] [ 1d ] ← base timeframe switcher
|
||
|
||
LAYERS
|
||
─────────────────────────────────
|
||
☑ Daily MAs ██
|
||
☑ 10 ☑ 20 ☑ 50 ☑ 100 ☑ 200
|
||
─────────────────────────────────
|
||
☐ 1h MAs ▓▓
|
||
☐ EMA9 ☐ EMA21
|
||
─────────────────────────────────
|
||
☑ Manual lines ░░ (M5)
|
||
☐ Auto trendlines (M8)
|
||
─────────────────────────────────
|
||
☐ Hidden levels still count toward confluence
|
||
```
|
||
|
||
- **The base timeframe switcher changes only the candles.** Every level stays on screen
|
||
— a 200DMA is equally valid on a 1m chart. That is the entire premise of the product.
|
||
- **The group checkbox is a master toggle** — unchecking "Daily MAs" hides all five at
|
||
once; individual periods nest under it.
|
||
- The colour swatch beside each timeframe is that timeframe's hue, used identically on
|
||
the chart and in the confluence panel. One hue per timeframe, everywhere.
|
||
- **Persist to `localStorage`.** These settings are pure UI preference and must survive
|
||
reloads; they do not belong on the server.
|
||
|
||
**Visibility vs. scoring — decide this explicitly.** Hiding a level defaults to
|
||
*also* removing it from confluence scoring, because "I don't want to see this" almost
|
||
always means "I don't care about this." The last checkbox decouples the two for anyone
|
||
who wants a clean chart with full scoring.
|
||
|
||
That default has an architectural consequence: **confluence is computed server-side, so
|
||
the client's enabled set must reach the server.** Send it over the WebSocket on change:
|
||
|
||
```json
|
||
{"type":"prefs",
|
||
"base_tf":"1m",
|
||
"enabled":{"ma":{"1d":[10,20,50,100,200]},"manual":true,"auto":false},
|
||
"hidden_levels_score":false}
|
||
```
|
||
|
||
Store it per connection. Single-user app — no need for anything more elaborate.
|
||
|
||
### Panels
|
||
|
||
- **Drawing toolbar** (M5) — trendline tool, snap toggle, delete selection.
|
||
- **Confluence panel** — clusters sorted by `|distance|`, each showing member
|
||
timeframes, the zone range, and the score. This is the primary readout; give it more
|
||
visual weight than the price itself.
|
||
- **Status bar** — stream state, contract symbol, last bar age, which timeframes are
|
||
warm. When the stream drops, the chart must *say so*, not quietly show stale candles.
|
||
- **Alert log** — recent fires, most recent first.
|
||
- **Timeframe selector** — switches the candle series only. Levels stay.
|
||
|
||
Keep the existing dark/light CSS-variable scheme in `style.css`.
|
||
|
||
---
|
||
|
||
## 10. The history problem
|
||
|
||
**Mostly solved by the Yahoo source (§2.1) — but read the caveats.**
|
||
|
||
Schwab alone would leave the system knowing nothing before the moment it connects.
|
||
Since the highest-weighted timeframes need the most history, Schwab-only cold start
|
||
would mean waiting months for the parts of the product that matter most:
|
||
|
||
| Timeframe | Usable after, Schwab-only | With Yahoo seeding |
|
||
|---|---|---|
|
||
| 5m, 15m | ~2–4 hours | immediate |
|
||
| 30m, 1h | ~1–2 sessions | immediate |
|
||
| 1d | months | immediate (~500 sessions) |
|
||
| 1d 200SMA | ~10 months | immediate |
|
||
|
||
**Startup sequence:**
|
||
|
||
1. Seed from Yahoo: `interval=1h&range=730d`, plus `interval=1m&range=8d` for the
|
||
fine detail near the present.
|
||
2. Aggregate through `aggregator.py` into every timeframe, using our own session rules.
|
||
3. Attach the live source (Yahoo poll, or Schwab once keys exist) and continue forward.
|
||
4. Persist everything (M7) so subsequent restarts need less seeding.
|
||
|
||
**Caveats that must be honoured in code:**
|
||
|
||
- **Tag every bar with its `source`** (`"yahoo"` / `"schwab"` / `"replay"`), and expose
|
||
the seam in the UI. Silently blending delayed continuous data with live per-contract
|
||
data produces trendlines nobody else can see.
|
||
- **Roll gaps.** Yahoo `ES=F` is a continuous front-month series; quarterly contract
|
||
rolls leave price discontinuities that will read as a trendline break or generate a
|
||
bogus pivot. For M0–M5 this is acceptable. If it proves noisy, the fix is either
|
||
panama-adjusting the seeded series or dropping pivots within ±1 bar of a known roll
|
||
date (third Friday of Mar/Jun/Sep/Dec). Do not build roll adjustment pre-emptively.
|
||
- **The 1m request cap is 8 days.** Longer 1m ranges must be fetched in ≤8-day windows
|
||
and stitched. In practice you don't need deep 1m history — seed 1h and let the
|
||
aggregator do the rest.
|
||
- **Never mix Yahoo daily bars in** — see §2.1.
|
||
|
||
If per-contract accuracy ever matters more than convenience, **Databento** sells proper
|
||
CME history with real roll handling. Not needed now.
|
||
|
||
---
|
||
|
||
## 11. Configuration
|
||
|
||
All via env, read in `config.py`. Ship a `.env.example`; `.env` is already gitignored.
|
||
|
||
```
|
||
# --- data sources ---
|
||
LIVE_SOURCE=yahoo # yahoo | schwab | replay
|
||
SEED_SOURCE=yahoo # yahoo | none
|
||
YAHOO_SYMBOL=ES=F
|
||
YAHOO_POLL_SECONDS=20
|
||
SEED_1H_RANGE=730d
|
||
SEED_1M_RANGE=8d # 8d is Yahoo's hard per-request cap
|
||
|
||
# --- schwab (only needed once LIVE_SOURCE=schwab) ---
|
||
SCHWAB_API_KEY=
|
||
SCHWAB_APP_SECRET=
|
||
SCHWAB_CALLBACK_URL=https://127.0.0.1:8182
|
||
SCHWAB_TOKEN_PATH=./.schwab_token.json
|
||
SCHWAB_ACCOUNT_ID=
|
||
SCHWAB_SYMBOL=/ES
|
||
|
||
# --- timeframes & indicators ---
|
||
TIMEFRAMES=1m,2m,5m,15m,30m,1h,1d
|
||
BASE_TIMEFRAMES=1m,30m,1d # the chart switcher
|
||
MAX_BARS_PER_TF=5000 # in-memory ring buffer bound
|
||
TRENDLINE_SOURCE_GEOMETRY=true # false = rollback to displayed/1m-grid pricing
|
||
|
||
# Daily MA set is the primary requirement; others ship disabled. See §7.4
|
||
MA_SETS__1D=sma10,sma20,sma50,sma100,sma200
|
||
MA_SETS__4H=
|
||
MA_SETS__1H=
|
||
DAILY_ANCHOR_ET=18:00 # CME session open; change to match another platform
|
||
|
||
MANUAL_LINES_PATH=./data/manual_lines.json
|
||
|
||
CONFLUENCE_MIN_SCORE=6 # MUST be recalibrated in M4 — see §7.6
|
||
ALERT_COOLDOWN_SECONDS=900
|
||
|
||
NTFY_TOPIC=
|
||
NTFY_SERVER=https://ntfy.sh
|
||
|
||
CHART_AUTH_TOKEN= # empty = auth disabled (local dev)
|
||
REPLAY_FILE= # set to replay a tape instead of connecting
|
||
```
|
||
|
||
**Token persistence:** `schwab-py`'s refresh token expires every **7 days** and
|
||
re-auth is an interactive browser flow. Locally, keep `.schwab_token.json` out of the
|
||
repo (add to `.gitignore`). On the VPS later, it **must** live on a Coolify persistent
|
||
volume or every rebuild logs you out. Same for the eventual SQLite file.
|
||
|
||
---
|
||
|
||
## 12. Testing
|
||
|
||
The market is closed most of the time you will be working. Build for that.
|
||
|
||
**Record/replay is a milestone-1 deliverable, not a nicety.** `recorder.py` writes
|
||
every raw stream message to JSONL with its arrival timestamp; replay feeds them back
|
||
through the identical code path, either at wall-clock speed or as fast as possible.
|
||
Everything downstream of `StreamService` is then testable, deterministically, offline.
|
||
|
||
Required tests:
|
||
|
||
- `session.py` — bucket boundaries. Both DST transitions, Sunday 18:00 open, the
|
||
17:00–18:00 break, and Friday close. **Write these first.**
|
||
- `aggregator.py` — 1m→all TFs on synthetic bars; gap handling; idempotent replay.
|
||
- `moving_averages.py` — a known daily series produces known 10/20/50/100/200 values;
|
||
assert nothing is emitted before warm-up; assert the stepped projection onto 1m holds
|
||
its value for a whole session and changes exactly at the session boundary.
|
||
- `manual_lines.py` — round-trip JSON persistence; a line drawn on 4h evaluates to the
|
||
same price on the 1m chart at the same instant.
|
||
- *(M8)* `pivots.py` — known fixtures; assert no repainting (a pivot, once emitted,
|
||
never changes when more bars arrive).
|
||
- *(M8)* `trendlines.py` — hand-built fixtures where the correct line is obvious;
|
||
assert violation rejection and dedup.
|
||
- `confluence.py` — synthetic levels producing a known cluster and score.
|
||
- `alerts.py` — assert no re-fire within cooldown, and that oscillation around a level
|
||
produces exactly one alert.
|
||
|
||
Add `pytest` to a `requirements-dev.txt`.
|
||
|
||
---
|
||
|
||
## 13. Milestones
|
||
|
||
> **All of M0–M10 are built and deployed.** This section is kept as a record of
|
||
> what each subsystem was required to do, not as a queue. The "Done when"
|
||
> criteria still earn their place: they describe correct behaviour, and several
|
||
> have since become tests. Treat them as the specification of a working
|
||
> subsystem — and if one no longer matches reality, the code changed and this
|
||
> did not, which is a bug in this document.
|
||
|
||
Ordered so the user's stated priority — **live realtime charts first** — lands
|
||
earliest, and so nothing later is blocked on market hours.
|
||
|
||
**No API keys are required until M6.** M0–M5 run entirely on Yahoo.
|
||
|
||
### M0 — Yahoo source (no keys, no blockers)
|
||
`market/base.py` protocol + `market/yahoo.py`: `history()` over the verified chart
|
||
endpoint, and `stream()` as a polling loop presenting the same async-iterator
|
||
interface. Null-filtering, 8-day 1m windowing, bar `source` tagging. Recorder writes a
|
||
tape; `ReplaySource` reads it back.
|
||
**Done when:** a script prints seeded 1h bars back to 2024 and then live-ish 1m bars,
|
||
and a recorded tape replays identically.
|
||
|
||
### M1 — Live chart end to end ⭐ primary deliverable
|
||
`StreamService` → in-memory 1m store → WebSocket → Vue 3 + LWC candlestick chart
|
||
updating live. Status bar showing source + bar age. No analysis yet.
|
||
**Done when:** the browser shows a live-updating /ES 1-minute candle chart, and the
|
||
same chart can be reproduced offline from a tape.
|
||
|
||
### M2 — Aggregation + timeframe switching
|
||
`session.py` + `aggregator.py` with full test suite. Timeframe selector drives the
|
||
candle series. `GET /api/bars`.
|
||
**Done when:** switching to 15m shows correctly bucketed bars, and replaying a tape
|
||
twice yields identical output.
|
||
|
||
### M3 — Multi-timeframe moving averages ⭐ start here for analysis
|
||
MAs are the right first analysis layer: **fully deterministic, no parameters to tune,
|
||
no judgment calls**, and Yahoo seeding makes them warm from startup. They validate the
|
||
entire overlay concept — stepped rendering, level diffing by `id`, TF colour scheme —
|
||
without any of the ambiguity trendlines carry.
|
||
**Done when:** the 10/20/50/100/200 DMAs render on the 1m, 30m and 1d charts, stepping
|
||
once per session on the intraday views, dashed while the current session is unfinished.
|
||
|
||
### M3.5 — Layer panel
|
||
Checkbox tree to show/hide levels by timeframe and by individual MA (§9.4). Ships with
|
||
M3 because 6 timeframes × 4 MAs = 24 lines is unreadable without it.
|
||
**Done when:** unchecking a group removes its every level from the chart and the state
|
||
survives a reload.
|
||
|
||
### M4 — Confluence + alerts (on moving averages alone) ⭐ first genuinely useful build
|
||
`confluence.py`, `alerts.py`, confluence panel, ntfy push, in-browser sound — scored
|
||
over MA levels only. Multi-timeframe MA confluence is a real signal in its own right;
|
||
this is a complete, useful product with zero hand-drawn input and zero tuning.
|
||
**Done when:** a replayed tape produces a sane number of alerts (single digits per
|
||
session), each corresponding to a real multi-timeframe convergence.
|
||
|
||
### M5 — Manual trendlines
|
||
Two-click drawing, snapping, persistence, feeding the same confluence engine (§7.3a).
|
||
Hand-drawn lines are authoritative — full weight, no quality discount.
|
||
**Done when:** a line drawn on the 4h chart appears correctly projected on the 1m chart
|
||
and raises the confluence score of a cluster it lands in.
|
||
|
||
### M6 — Schwab live source (needs API keys)
|
||
Implement `market/schwab.py` against the existing `MarketDataSource` protocol and
|
||
answer every question in [§2.2](#22-verify-these-when-adding-the-schwab-source-m6-not-before).
|
||
Yahoo continues to handle seeding; Schwab takes over the live tail.
|
||
**Done when:** flipping `LIVE_SOURCE=yahoo` → `schwab` changes nothing visible except
|
||
lower latency and true per-contract prices. **If this milestone requires touching any
|
||
file outside `market/`, the abstraction in M0 was wrong — fix it there, not here.**
|
||
|
||
### M7 — Persistence
|
||
`SqliteBarStore` behind the existing `BarStore` protocol. Backfill-on-start from disk,
|
||
falling back to Yahoo seeding only for what's missing.
|
||
**Done when:** restarting the process loses no history.
|
||
|
||
### M8 — Automatic trendline detection (optional)
|
||
`pivots.py`, `trendlines.py` per §7.3. Deliberately last among the analysis work:
|
||
by this point the hand-drawn lines from M5 are **ground truth**, so the scoring
|
||
coefficients can be tuned to agree with lines you actually drew, rather than guessed at
|
||
in the abstract. Auto lines render in a distinct style and are individually
|
||
dismissable; they never silently replace a manual line.
|
||
**Done when:** on a replayed tape, auto-detected lines land where the manual ones were
|
||
drawn, and the layer panel can hide them independently.
|
||
|
||
### M9 — Bias panel
|
||
BULLISH / BEARISH toggle recording the user's directional call against the current
|
||
confluence state, persisted, with a journal view. **Records only — trades nothing.**
|
||
|
||
Because `LEVEL_ONE_FUTURES_OPTIONS` streams, this milestone can go further than the
|
||
original sketch: given a bias, resolve the 1-DTE strikes, subscribe to the two legs,
|
||
and display the **live spread price** — so the readout becomes actionable enough to
|
||
hand-execute in thinkorswim:
|
||
|
||
```
|
||
BEARISH /ES — resistance confluence 15 @ 6403.75–6405.00
|
||
Proposed: 1-DTE 6405/6415 put spread ~2.35 x 10 (live)
|
||
```
|
||
|
||
Strike/expiration symbol resolution for futures options is fiddly; treat it as its own
|
||
sub-task and verify the symbol format against `LEVEL_ONE_FUTURES_OPTIONS` empirically,
|
||
the same way M6 verifies `/ES`.
|
||
|
||
### M10 — VPS deploy (when wanted)
|
||
Shared-secret auth on, `workers=1`, persistent volume for token + DB, Coolify domain
|
||
port suffix preserved per README.
|
||
|
||
---
|
||
|
||
## 14. Why execution is out of scope
|
||
|
||
Note the asymmetry: Schwab **does** stream futures-options *quotes*
|
||
(`LEVEL_ONE_FUTURES_OPTIONS`), so we can price a spread live — we just cannot transmit
|
||
it. Read anything below as being about order entry only.
|
||
|
||
Schwab's Trader API exposes no futures or futures-options **order entry**. thinkScript
|
||
`AddOrder()` places *simulated* orders for backtesting only. Automating clicks in the
|
||
thinkorswim UI is the wrong reliability model for near-expiration leveraged
|
||
instruments — window focus, stale quotes, partial fills, and dialogs all fail silently,
|
||
and an execution path must be able to tell the program what the broker actually did.
|
||
|
||
The interim workflow is therefore: this app produces a decision, the human executes it
|
||
in thinkorswim.
|
||
|
||
Keep the seam clean. Broker-specific code lives only in `market/schwab.py`; nothing in
|
||
`analysis/`, `bars/`, or `api/` may import it. When an execution adapter is added —
|
||
Schwab, if they ever ship futures-options orders, or IBKR — it consumes `Cluster` and
|
||
the M9 bias signal and nothing else. This is the same discipline the M6 acceptance test
|
||
enforces for data sources.
|
||
|
||
---
|
||
|
||
## 15. Risk register
|
||
|
||
| Risk | Impact | Mitigation |
|
||
|---|---|---|
|
||
| Futures entitlement missing on the Schwab account | Delays M6 only | No longer blocks — M0–M5 run on Yahoo |
|
||
| No futures history from Schwab | Higher TFs cold for months | Solved: Yahoo seeding, §10 |
|
||
| Yahoo endpoint is unofficial — may rate-limit or change shape | Dev source breaks | Isolated in `market/yahoo.py`; cache seeds to disk (M7) so it's fetched rarely; back off on 429 |
|
||
| Yahoo daily bars anchored midnight ET, not session | Daily candles disagree with every other chart | Never use them — build 1d from 1h, §2.1 |
|
||
| Contract roll gaps fake a trendline break | Bad signals on seeded data | Store `symbol` + `source` per bar; surface rolls in UI; §10 |
|
||
| Source abstraction leaks Schwab/Yahoo specifics upward | M6 turns into a rewrite | M6 acceptance test: no file outside `market/` may change |
|
||
| `session.py` bucket math wrong | Silently wrong lines everywhere | Tests written first; both DST transitions |
|
||
| Repainting pivots | Lines that "were always there" | `w`-bar confirmation lag, enforced by test |
|
||
| Alert fatigue | Product becomes unusable | Cluster-level alerts, cooldown + separation re-arm |
|
||
| Multiple uvicorn workers | Duplicate Schwab connections | `workers=1`; streamer in `lifespan`; warned in `Procfile` |
|
||
| JWT signing key derived from the password | A leaked cookie brute-forces the password offline; blocks multi-user outright | Server-side random secret — `docs/multi_user.md` |
|
||
| Threadpool routes touching `asyncio.Queue` | Dropped socket wakeups, rare corruption | Post via `call_soon_threadsafe` — `docs/async_refactor.md` P0 |
|
||
| Level rebuilds run CPU-bound on the event loop | 82s startup; a stall every closed bar | Bulk seed, incremental MAs — `docs/async_refactor.md` P1 |
|
||
| Alert disarm writes to disk on the loop | Stream stalls when an alert fires | Offload the write — `docs/async_refactor.md` P2 |
|
||
| Schwab token expiry (7 days) | Stream dies | Keepalive REST ping writes a new refresh token; header reconnect when the grant is dead |
|
||
| Vue reactivity wrapping chart objects | Perf collapse, odd bugs | `shallowRef`/`markRaw` — §9 |
|
||
| LWC v4 tutorials copied | Code silently wrong for v5 | `addSeries(SeriesType, ...)` only |
|
||
| Viewport derived from `bars.length` | Chart looks frozen; feed is fine | Anchor the view by time, never by logical index — §9 |
|
||
| Seed replays every bar through `on_bar` | ~82 s startup; port refuses connections | Known, unfixed — §16, 2026-08-10 |
|
||
| Headless browser without a real locale | `Intl` throws; blank canvas mimics an app bug | Launch Chromium with `--lang=en-US` — §16 |
|