workspace/projects/kip/PROJECT.md

Kip — Voice Assistant

Codename: Kip Purpose: Alexa replacement for D J's girlfriend Architecture: Steam Deck (thin client) ↔ Proxmox LXC (brains)

---

Overview

Kip is a privacy-first voice assistant. The Steam Deck acts as a dumb terminal (mic, speaker, screen). All intelligence runs on an LXC container on Proxmox.

Hardware

Steam Deck (Client)

• Always-on, propped up in kitchen/living room on charging dock
• Runs: wake word detection, audio capture, audio playback, display UI
• Connects to LXC over local WiFi

LXC Container (Server)

• OS: Ubuntu 22.04 or 24.04
• RAM: 4GB recommended (Whisper needs ~1.5GB, Piper ~200MB, OpenClaw ~500MB)
• Disk: 10GB (models + data)
• CPU: 2-4 cores (Whisper STT is CPU-bound)
• Network: Static IP on LAN, accessible from Steam Deck

Software Stack

LXC Container

Component	Purpose	Tool
STT	Speech-to-text	Faster Whisper (base.en model)
TTS	Text-to-speech	Piper (en_US voice)
Agent	Intelligence	OpenClaw with Kip agent
API	Communication	FastAPI HTTP server
Data	Grocery list, timers	JSON files + SQLite

Steam Deck

Component	Purpose	Tool
Wake word	"Hey Kip" detection	OpenWakeWord
Audio capture	Record after wake	PyAudio / sounddevice
Audio playback	Play TTS responses	PyAudio / sounddevice
UI	Display info	Web browser (fullscreen PWA) or PyQt
Client	Talk to LXC	Python HTTP client

---

LXC Setup Checklist

D J creates the LXC with:

• Ubuntu 22.04 or 24.04 template
• 4GB RAM, 2-4 CPU cores, 10GB disk
• Static IP on LAN (e.g., 192.168.86.XX)
• SSH access enabled (key-based)
• Audio passthrough NOT needed (LXC doesn't play audio — Deck does)
• Internet access (for OpenClaw, model downloads)
• Hostname: kip (optional, nice to have)

Once created, give Case:

• IP address
• SSH credentials or key

---

API Design (LXC ↔ Steam Deck)

POST /listen

Steam Deck sends audio, gets back text response + TTS audio.

Request:
  Content-Type: multipart/form-data
  Body: audio file (WAV, 16kHz mono)

Response:
  {
    "text": "user said this",
    "response": "Kip says this",
    "audio": "<base64 WAV of TTS response>",
    "ui_update": {
      "type": "grocery_list",
      "data": ["eggs", "milk", "bread"]
    }
  }

GET /status

Health check + current state (timers, lists, etc.)

GET /grocery

Returns current grocery list (for phone web view)

POST /grocery

Add/remove items (for phone web view)

GET /ui

Returns current display state for the Deck's screen.

---

Kip Agent (OpenClaw)

Kip gets its own OpenClaw agent with:

SOUL.md (Personality)

• Name: Kip
• Friendly, concise, warm
• Optimized for voice — short responses, no markdown
• Knows the household (D J, girlfriend, 4 cats)
• Designed for non-technical user

Capabilities

• Weather: "Hey Kip, what's the weather?" → Nashville forecast
• Timers: "Hey Kip, set a timer for 15 minutes" → countdown with alarm
• Grocery list: "Hey Kip, add eggs to the list" → persistent list
• Grocery check: "Hey Kip, what's on the grocery list?" → reads it back
• General Q&A: "Hey Kip, how long do I bake chicken at 400?" → answer
• Time/Date: "Hey Kip, what time is it?"

Future Capabilities

• Calendar integration
• Music control (Spotify)
• Smart home (if they get devices)
• Recipe lookup
• Kroger API for prices/ordering

---

Phone Web View

Simple responsive web page served by the LXC:

• Shows grocery list
• Can add/remove items by tapping
• Accessible at http://kip.local:8080 or http://192.168.86.XX:8080
• Girlfriend can open it on her iPhone in the store

---

Build Phases

Phase 1: Voice Loop (MVP)

• LXC setup + dependencies installed
• Faster Whisper running (base.en model)
• Piper TTS running (pick a good voice)
• FastAPI server handling /listen endpoint
• Steam Deck: wake word + record + send + play response
• Test: "Hey Kip, hello" → Kip responds with voice
• Goal: End-to-end voice working

Phase 2: Grocery List + Weather

• Grocery list CRUD (voice + API)
• Weather skill (Nashville)
• Timer system with alarm sounds
• Phone web view for grocery list
• Steam Deck display: clock + weather + active timers
• Goal: Actually useful in the kitchen

Phase 3: OpenClaw Integration

• Kip agent running on OpenClaw
• General Q&A via Claude/Qwen
• Smarter conversations (context, follow-ups)
• Cost optimization: simple commands (timer, list) handled locally, only complex Q&A hits Claude
• Goal: Smart assistant, not just a voice command box

Phase 4: Polish

• Custom wake word model trained on "Hey Kip"
• Better TTS voice selection
• Deck UI polish (nice weather widget, timer display, list view)
• Ambient mode (clock/weather when idle)
• Multiple room support (add Pi later)
• Goal: Girlfriend actually wants to use it daily

---

Cost

Item	Cost
Steam Deck	Already owned
LXC container	Free (Proxmox)
OpenWakeWord	Free (open source)
Faster Whisper	Free (open source)
Piper TTS	Free (open source)
OpenClaw	Already running
Claude API for Q&A	Covered by existing subscription
Total	$0

---

File Structure

projects/kip/
├── PROJECT.md          # This file
├── server/             # LXC-side code
│   ├── main.py         # FastAPI server
│   ├── stt.py          # Whisper STT wrapper
│   ├── tts.py          # Piper TTS wrapper
│   ├── skills/         # Timer, grocery, weather handlers
│   ├── data/           # Grocery lists, state
│   └── requirements.txt
├── client/             # Steam Deck code
│   ├── kip_client.py   # Main client app
│   ├── wake_word.py    # OpenWakeWord listener
│   ├── audio.py        # Record/playback
│   ├── ui/             # Display UI
│   └── requirements.txt
└── agent/              # Kip's OpenClaw agent config
    ├── SOUL.md
    └── config.yaml

---

Notes

• All voice processing (STT/TTS) on LXC, not Deck — keeps client thin
• Wake word is the ONLY thing that runs on Deck locally
• Grocery list syncs to a web page for phone access
• Simple commands (timer, list) should be handled WITHOUT hitting Claude to save tokens
• Only complex Q&A ("how long to bake chicken?") routes through OpenClaw/Claude
• Qwen on Ollama (192.168.86.137) as fallback for simple Q&A