# Task: Unified Market Data Service + Redis Message Log

**Priority:** HIGH  
**Assigned:** Glitch (after trader-dashboard-controls completes)  
**Status:** Queued  
**Date:** 2026-03-01  
**Depends on:** trader-dashboard-controls (in progress)

## Problem

Two services independently fetch the same OHLCV data from Binance.us:
1. **Dashboard scanner** (`lib/server/scanner.ts`) — 29 coins × 1h × every 30s
2. **TA Service** (`ta_service.py`) — 29 coins × 3 timeframes (5m/1h/4h) × own cycle

This doubles API calls, wastes bandwidth, and increases rate limit risk.

## Solution

### New Service: `coinex-market-data`
A single long-running service that:
1. Fetches ALL OHLCV data from `api.binance.us` (29 coins × 3 timeframes)
2. Caches everything in Redis with TTLs
3. Publishes updates to Redis pub/sub channels on each refresh
4. Exposes a health endpoint

**No other service should hit Binance directly after this.**

### Redis Schema

```
# Cached OHLCV candle data
market:ohlcv:{symbol}:{timeframe}  → JSON array of candles (TTL: varies by timeframe)
  - 5m:  TTL 120s (refresh every 60s)
  - 1h:  TTL 300s (refresh every 120s)  
  - 4h:  TTL 600s (refresh every 300s)

# Latest price/ticker for quick reads
market:price:{symbol}  → JSON { price, change24h, volume, timestamp } (TTL: 60s)

# Pub/sub channels for real-time consumers
channel: market:update:{timeframe}  → published after each timeframe refresh cycle
channel: market:update:all          → published after complete scan cycle

# Message log (see below)
market:log  → Redis Stream (XADD) of all messages passing through Redis
```

### Refresh Cycle
```
Every 30s:  Fetch 5m candles for all 29 coins → cache → publish market:update:5m
Every 120s: Fetch 1h candles for all 29 coins → cache → publish market:update:1h  
Every 300s: Fetch 4h candles for all 29 coins → cache → publish market:update:4h
```

### Consumer Changes

**Dashboard scanner (`scanner.ts`):**
- Remove ALL direct Binance API calls
- Read from `market:ohlcv:{symbol}:1h` Redis keys instead
- Subscribe to `market:update:1h` channel to trigger recalculation
- Calculate RSI/VWAP/BB from cached candles (same logic, different data source)

**TA Service (`ta_service.py`):**
- Remove ALL direct Binance API calls
- Read from `market:ohlcv:{symbol}:{timeframe}` Redis keys
- Subscribe to `market:update:{timeframe}` channels to trigger indicator calculation
- Keep all indicator logic (EMA ribbons, TTM Squeeze, Stoch RSI) unchanged

### Redis Message Log

Use **Redis Streams** (`XADD`/`XRANGE`/`XLEN`) to log ALL messages passing through Redis:

```
# Every publish, cache write, and signal gets logged
Stream key: redis:message_log

Fields per entry:
  - type: "publish" | "cache_write" | "cache_read" | "signal" | "trade" | "error"
  - channel: the pub/sub channel or key involved
  - source: "market-data" | "ta-service" | "dashboard" | "trader-bot"
  - summary: human-readable one-liner
  - payload_size: bytes
  - timestamp: ISO string
```

The market data service wraps Redis operations to auto-log:
- Every `PUBLISH` → logged with channel + payload size
- Every `SETEX` (cache write) → logged with key + TTL
- Every signal published by TA service → logged
- Every trade action by trader bot → logged

**Dashboard API:**
- `GET /api/redis/log?limit=100&type=signal&source=ta-service` — filterable log viewer
- Displayed in the Status page under a new "Redis Activity" section

Stream is capped at 10,000 entries (`XADD ... MAXLEN ~ 10000`) to prevent unbounded growth.

## Architecture

```
                    ┌─────────────────────┐
                    │   Binance.us API    │
                    └─────────┬───────────┘
                              │ (ONLY connection)
                    ┌─────────▼───────────┐
                    │  Market Data Service │  ← NEW (Python, systemd)
                    │  Port 8895 /health   │
                    └─────────┬───────────┘
                              │ SETEX + PUBLISH + XADD
                    ┌─────────▼───────────┐
                    │       Redis          │
                    │  OHLCV cache         │
                    │  Pub/sub channels    │
                    │  Message stream log  │
                    └──┬──────┬────────┬──┘
                       │      │        │
              ┌────────▼┐ ┌───▼─────┐ ┌▼──────────┐
              │Dashboard │ │TA Svc   │ │Trader Bot │
              │Scanner   │ │(signals)│ │(trades)   │
              │(scores)  │ │         │ │           │
              └──────────┘ └─────────┘ └───────────┘
```

## Tech Stack
- **Python** (asyncio + aiohttp + redis-py)
- **FastAPI** health endpoint on port 8895
- **Systemd** user service: `coinex-market-data.service`
- **Redis Streams** for message logging

## Files
- New service: `~/.openclaw/workspace/projects/coinex-market-data/`
- Dashboard scanner: `~/.openclaw/workspace/projects/coinex-dashboard/lib/server/scanner.ts`
- TA service: `~/.openclaw/workspace-glitch/projects/coinex-ta-service/ta_service.py`
- Redis: 127.0.0.1:6379

## Git Repository
- **Create private Gitea repo FIRST**: `https://git.letsgetnashty.com/case/coinex-market-data`
- Git credentials: `case:Gh0st%21nTh3Mach1n3` (URL-encoded in remote URL)
- Git config: user `Case`, email `case-lgn@protonmail.com`
- Push initial commit before starting development
- Commit frequently throughout build

## Constraints
- `api.binance.us` ONLY (not api.binance.com — 451 geo-block)
- Respect Binance rate limits (1200 req/min)
- All existing indicator/scoring logic untouched
- Dashboard must work identically from user perspective
- Include unit tests
- Context7 mandatory for library docs

## Definition of Done
- [ ] Market data service running, fetching all 29 coins × 3 timeframes
- [ ] All OHLCV data cached in Redis with appropriate TTLs
- [ ] Dashboard scanner reads from Redis (zero Binance calls)
- [ ] TA service reads from Redis (zero Binance calls)
- [ ] Redis message log capturing all pub/sub + cache activity
- [ ] `/api/redis/log` endpoint on dashboard with filtering
- [ ] Redis Activity section on Status page
- [ ] Health endpoint at :8895/health
- [ ] Systemd service created and running
- [ ] Gitea repo created and pushed
- [ ] Unit tests passing
- [ ] All 3 services confirmed working together end-to-end