chart/README.md

# chart

FastAPI backend + Vue 3 (from CDN, no build step) served at
<https://chart.amow.com>.

The app charts Yahoo's `ES=F` feed, builds CME-session-aware timeframes, daily moving
averages, prior-day high/low/close and session VWAP, and alerts on confluence zones.
The full spec lives in [`docs/plan.md`](docs/plan.md) — read it before writing
code; it records decisions and verified API facts that are expensive to
rediscover. What it cost to get there is in
[`docs/implementation.md`](docs/implementation.md): a dated log of problems and
their resolutions, kept as a learning record alongside git.

Both are living documents. The app is past the point of being built and into
being maintained, so a change that makes either one wrong is not finished —
correcting the plan, or logging what was hard, is part of the work rather than
housekeeping after it.

## Local development

```bash
docker compose up --build
```

Then open <http://localhost:8010>. Override the host port with, for example,
`PORT=8020 docker compose up`. The port binds to all host interfaces, so another
machine can connect at `http://HOST_IP:8010`. The source tree is bind-mounted and uvicorn
runs with `--reload`, so edits to `main.py` or `static/` take effect without a
rebuild. Rebuild only when `requirements.txt` changes.

The Compose stack also includes Playwright for browser screenshots. It reaches
the app over the internal Compose network and writes ignored artifacts locally:

```bash
docker compose exec playwright playwright screenshot \
  --lang en-US --wait-for-timeout 5000 http://api:8000 /artifacts/chart.png
```

Without Docker:

```bash
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload
```

Copy settings from `.env.example` as needed. To recalibrate the alert threshold against
Yahoo's current eight-day minute tape:

```bash
python3 -m scripts.calibrate_alerts
```

It sweeps threshold and cooldown in a single replay pass and prints alerts per session.

The 2026-08-09 calibration ran when daily moving averages were the only levels, and
`28` produced no alerts at all — there was nothing for a daily MA to cluster *with*.
Adding prior-day H/L/C and VWAP changed that completely: the same threshold went to 247
alerts over six sessions, 185 of them in one day.

Two fixes brought it back, in this order:

- **Cluster identity** was `sha1(side + round(center / tolerance))`, and `tolerance`
  derives from ATR — so it moved every bar. The same zone was continually issued a new
  id, never matched the cooldown table, and the cooldown was silently defeated. 247 → 54.
- **Alert suppression** keyed on cluster identity, so a level drifting in or out of a
  group counted as a new zone. It now suppresses by *proximity*: two zones within one
  ATR are the same zone. 54 → 40.
- **Suppression also matched on side**, and side is positional — a level sitting at
  price flips between support and resistance every time price ticks across it. Each
  flip read as a new zone. Found by putting a real line at the live price and getting
  four pushes in two minutes. 40 → 26.

Only then does the cooldown do anything useful. At threshold `28` the sweep reads:

| cooldown | total | max/session |
|---|---|---|
| 900s | 26 | 19 |
| 3600s | 18 | 12 |
| 7200s | 12 | 7 |
| **14400s (selected)** | **10** | **5** |

Note the threshold itself is a blunt control: scores are sums of 12s (moving averages,
VWAP) and 16s (prior-day levels), so `20`, `24` and `28` behave identically and `32`
falls to zero. Cooldown is the finer knob. Revisit both as more varied tapes are
recorded — six sessions is not much, and one of them dominates the totals.

## Testing

Run the complete backend suite in the same container environment as the app:

```bash
docker exec chart-api-1 sh -c "cd /app && python -m pytest -q"
```

Pass a path or pytest selector for a focused run:

```bash
docker exec chart-api-1 sh -c \
  "cd /app && python -m pytest -q tests/test_aggregator.py::test_closed_yahoo_hours_form_the_right_cme_daily_bar_across_1800_et"
```

Run every browser test against the local Compose stack, or filter by filename:

```bash
./bin/e2e
./bin/e2e trendline
./bin/e2e preferences
```

The browser suite creates drawings and removes them afterward. Run it against
the local stack, not production: production has the persistent drawing store,
live alerts, and an authenticated feed. The helper deliberately runs tests one
at a time because the local drawing store is shared with anyone using the dev
chart.

Without Docker, install the development requirements before running pytest:

```bash
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
python -m pytest -q
```

Recommended pre-deploy and deploy verification:

```bash
docker exec chart-api-1 sh -c "cd /app && python -m pytest -q"
./bin/e2e
git push && bin/wait-deploy
curl -fsS https://chart.amow.com/api/health
curl -fsS https://chart.amow.com/api/version
```

For an authenticated production status smoke test, use the chart token without
printing it:

```bash
curl -fsS -H "X-Chart-Token: $TOKEN" https://chart.amow.com/api/status | jq .
```

## Layout

| Path | Purpose |
|---|---|
| `main.py` | FastAPI lifespan and app wiring; JSON under `/api`, SPA at `/` |
| `app/` | Market sources, aggregation, analysis, alerts, and API |
| `static/` | `index.html`, `app.js`, `style.css` — Vue 3 loaded from unpkg |
| `requirements.txt` | Python deps |
| `Procfile` | Start command; **nixpacks needs this** or the deploy has nothing to run |
| `bin/wait-deploy` | Blocks until the live site serves your latest commit |
| `Dockerfile.dev`, `docker-compose.yml` | Local dev only — production does not use them |

## Deployment

Push to `main` → Forgejo webhook → Coolify rebuilds with nixpacks → live.
Rebuild time depends on whether Docker's build cache is warm: measured at
**~20s warm** (you pushed recently) and **~90s cold** (the cache goes stale
after an idle hour or so, which is the usual case). Nothing changes on the site
until the new container swaps in at the very end, so the old version keeps
serving for the whole build.

To know when your commit is actually live, rather than guessing:

```bash
git push && bin/wait-deploy
```

It polls `/api/version` (which returns the `SOURCE_COMMIT` Coolify bakes into
the container) until it matches your local `HEAD`, then exits. Runs from any
machine — no Coolify token, no SSH tunnel. Check by hand any time with:

```bash
curl -s https://chart.amow.com/api/version
```

One more trap worth naming: if your change only touches an `/api` endpoint, the
HTML is byte-identical and a browser refresh looks like nothing happened even
after a successful deploy. Verify the endpoint, not the page.

Two things to know before changing the deploy config:

- The Coolify domain is registered as `https://chart.amow.com:8000`. The port
  suffix is how Coolify decides the Traefik target port; drop it and every
  request 502s while the container looks perfectly healthy. It does not appear
  in the public URL.
- Ignore a 502 in the first minute after a deploy — that's the rolling container
  swap, and it clears on its own.

Manual redeploy:

```bash
curl -X POST -H "Authorization: Bearer $(cat ~/.coolify-token)" \
  "http://127.0.0.1:8000/api/v1/deploy?uuid=dgvch0xqv8uvjfor7dl8bwl9&force=true"
```


## Putting the live feed on production

Production runs the default `LIVE_SOURCE=yahoo` and therefore shows "yahoo" with
a ~10 minute delay. `LIVE_SOURCE=schwab` and the credentials live in `.env`,
which is gitignored — so the switch has never crossed the deploy boundary. It
cannot: nothing in the repo carries it.

Four things are needed, and all of them are set on the VPS rather than here.

**1. Environment, in Coolify:**

```
LIVE_SOURCE=schwab
SCHWAB_API_KEY=...          # same values as the local .env
SCHWAB_APP_SECRET=...
```

`SCHWAB_CALLBACK_URL` already defaults to `https://chart.amow.com/api/qt`, which
is what the Schwab app is registered with. Do not change it — a registered
callback edit can send the app back through approval.

**2. The OAuth token, at `data/.schwab_token.json` on the persistent volume.**
No new login is required: the local token was minted against the production
callback, and Schwab tokens are bound to the app rather than the machine. Copy
the file's contents into that path on the volume. If the path is not persistent,
the next deploy erases it and production falls silently back to Yahoo.

**3. Stop the dev stream first.** One refresh token means one Schwab streaming
connection. Dev and production both streaming will fight for it — see the risk
register on duplicate connections. Set `LIVE_SOURCE=yahoo` in the local `.env`,
or stop the local stack, before production goes live.

**4. Plan for expiry.** The Schwab refresh token lasts about seven days. When it
lapses, production drops back to Yahoo and needs a fresh token by the same
route: run the flow locally, paste the callback URL from `/api/qt` into the
waiting prompt, then copy the new token onto the volume.

Verify from anywhere:

```bash
curl -s -H "X-Chart-Token: $TOKEN" https://chart.amow.com/api/status
# want: "source":"schwab","delay_minutes":0
```

## Browser password and API token

`CHART_AUTH_TOKEN` guards everything under `/api` plus the `/ws` stream. Leave
it blank and the app is wide open, which is what you want locally — nothing
prompts. Scripts can supply it as the `X-Chart-Token` header or a `?token=`
query parameter.

Set `CHART_PASSWORD` to a human-friendly passphrase for browser access. The
browser posts it once to `/api/login`; the server returns a signed, HttpOnly,
30-day HS256 JWT cookie used by both API requests and the WebSocket. The opaque
API token is never returned to or stored by the browser. If `CHART_PASSWORD` is
temporarily absent, the login accepts `CHART_AUTH_TOKEN` as a migration fallback.
Existing browsers that stored a token under the old flow exchange it once for a
session and remove it from `localStorage`. `POST /api/logout` clears the session.

In production both secrets live in **Coolify's environment variables**, not in
this repo and not in `.env` — that file is gitignored and never exists in the
built container. Coolify re-injects its env vars into every container it
builds, so the token survives redeploys and reboots.

`/api/health` and `/api/version` deliberately stay open — `bin/wait-deploy`
polls the latter from whatever machine you pushed from, and neither reveals
anything about the market data or the configuration.

## Setting an alert on a price

Type the price into **Price alert** in the sidebar. It appears as a horizontal line
across the chart with a price-axis label, and alerts when price reaches it.

A price alert is stored as a manual line with `slope = 0`, so it inherits the whole
manual-line pipeline — JSON persistence, renaming, recolouring, deletion, clustering
with nearby levels — rather than being a parallel system. Two consequences worth
knowing:

- It **alerts regardless of confluence score**, like any hand-placed level. Weight
  exists to rank levels you did not ask for; you asked for this one.
- It has no drag handles and no "end line here" menu, because a price line has no
  endpoints to grab. Manage it from the sidebar list.

If it lands near other levels it clusters with them and the score adds up, so a typed
level sitting on the prior-day close reads as one zone rather than two alerts.

## Market data: Yahoo for the past, Schwab for the present

Both sources run together. This is the intended configuration, not a fallback:

- **Schwab** streams real-time `/ES` minute bars over `CHART_FUTURES`
  (`delayed: false`), but serves **no futures history at all** — everything it
  knows starts when you connect.
- **Yahoo** has roughly 730 days of hourly data, 60 days of native 30-minute
  bars, and eight days of minute data. Hourly history makes a 200-day moving
  average warm at startup; the native 30-minute seed avoids limiting that chart
  to the minute endpoint's eight-day window. Yahoo lags ~10 minutes.

Switch the live feed with `LIVE_SOURCE=schwab`; seeding stays on Yahoo whatever
you set, because Schwab has nothing to seed from. The symbols differ — Yahoo says
`ES=F`, Schwab says `/ES` — and `Settings.live_symbol` picks the right one. Every
bar carries a `source` tag so the seam stays visible.

**Expect a gap of up to ten minutes at the right-hand edge after a restart.**
Yahoo's history reaches to *now − 10 min* while the stream starts at *now*, so
the most recent bars are briefly missing. It backfills itself as Yahoo catches
up. Live price and alerts are unaffected — those come from the stream. Persisting
bars would remove it entirely.

`/ES` resolves to the active contract (`/ESU26` today) on Schwab's side, so
contract rolls need no handling.

### Rate limits

The developer portal shows **Order Limit: 120**, which caps orders per minute —
this app places none. Schwab separately rate-limits REST calls, commonly cited at
120 per minute.

Neither constrains us, because **streaming is not REST**. The WebSocket is a
single connection and bars arrive by push, so steady-state Schwab REST usage is
essentially zero. This is a concrete advantage of the stream over the polling
fallback: polling `get_quotes()` once a second would have sat at roughly half the
limit permanently, forever.

The exception is reconnects. Each one calls `get_user_preferences()` to fetch the
socket URL, and `StreamService` retries every five seconds, so a sustained outage
generates about twelve REST calls a minute. Comfortably under, but not nothing —
worth remembering before shortening that backoff.

Yahoo is a different service and none of these limits apply to it.

### Authenticating

```bash
python3 -m scripts.check_schwab                      # prints the login URL
python3 -m scripts.check_schwab --redirect-url '…'   # exchanges the code
python3 -m scripts.check_stream 60 /ES               # confirms bars arrive
```

Two steps, neither interactive, so the browser can be on a different machine —
you copy a URL out and paste one back. **The authorisation code expires in about
thirty seconds**, so have the second command ready before you approve. The
callback page 404s until this branch is deployed; that is cosmetic, the code is
in the address bar regardless.

The token refreshes itself for seven days, then needs the flow again. It is
per-machine: `.schwab_token.json` locally, and under `data/` in production, which
is the Coolify volume. Locally `data/` is owned by root because Docker created it
through the bind mount, which is why the local path differs.

## Alerts and ntfy

Alerts are evaluated **server-side**, once per closed 1m bar, by a single engine
living in `Runtime`. They do not depend on a browser being connected — that is the
whole point of the phone push — and opening two tabs does not double-notify.

`NTFY_TOPIC` must be set or nothing sends; `send_ntfy` returns immediately on a blank
topic. Set it in **Coolify's environment variables** for production, not in this repo.

**Cooldown state survives a restart.** The fired-zone table is written to
`ALERT_STATE_PATH` (`./data/alert_state.json`), which in production is the same
persistent volume as the trendlines. Before that, every deploy started with empty
cooldowns and the next closed bar re-alerted whatever zone price was sitting on —
with a four-hour cooldown, each push produced a burst of notifications for zones
that had already had their say.

Two consequences worth knowing. The file has to be on the volume, or the problem
comes straight back on the next deploy. And a corrupt or unreadable state file is
deliberately non-fatal: it logs and starts empty, costing one burst of duplicate
alerts rather than refusing to start the stream.

**Still prefer a blank topic locally.** Persistence removes the restart bursts, but
an in-memory engine is only half the story — a dev instance watching the same
symbol will happily push real alerts to your phone. Leaving `NTFY_TOPIC` blank keeps
the in-browser sound and banner while suppressing the push; set a `-dev` topic only
while testing the push path itself.

Note that ntfy topics are public by default: anyone who knows the name can both read
your alerts and publish to it. Treat the topic name as a secret.

## Saved trendlines

Manual trendlines are written to `MANUAL_LINES_PATH` (`./data/manual_lines.json`).
In production `/app/data` is a **Coolify persistent volume** — without it the
container filesystem is ephemeral and every deploy would silently wipe every
line you've drawn.