# 2026-03-05 — Massive Build Night

## What Happened
Late night (22:40 → 01:14 EST) massive build session with Pat. Built/overhauled 4 new hub pages, major upgrades to 3 existing ones, full audit of all 28 pages.

## New Hub Pages Built
- **27-model-lab** — Unified model hub replacing 23+24. Chat playground (text+image), eval breakdowns with explanations, model discovery scanner, web model search, ollama pull. All proxied through port 8097.
- **28-health-monitor** — Live service health for all 9 services. Latency sparklines, uptime %, state change event log. Auto-refresh 15s.

## Major Upgrades
- **25-research-lab** — Added: abort button, pause/resume, config hot-reload, paper export+library, auto-RAG feeding, research iteration, 3D visualization (Three.js), Research Autopilot (survey existing work → propose → launch), cost controls (free/paid model toggles), agent dropdown
- **26-computer-use-v2** — Prompt-optional sessions (quick-start buttons), sidebar session management (stop/archive/model swap), comprehensive help modal, all model lists proxied through 8097
- **04-system-dashboard** — Added non-Ollama models section (Jess's llama.cpp)

## Backend Changes
- **models.py** — Added llama.cpp routing (port 18080), `list_models_detailed()`, `llamacpp_generate()`
- **engine.py** — Added `abort_session()`, `pause_session()`, `resume_session()`, `iterate_session()`, `export_paper()`, `feed_to_rag()`, `list_papers()`, config hot-reload between stages
- **server.py** — New endpoints: `/api/chat` (proxy), `/api/models/pull`, `/api/models/search/local`, `/api/models/search/web`, `/api/autopilot/survey`, `/api/autopilot/status`, `/api/autopilot/launch`, `/api/sessions/abort`, `/api/sessions/iterate`, `/api/sessions/export`, `/api/sessions/config`, `/api/papers`, `/api/papers/download/{id}`
- **computer-use/server.py** — Added `/api/sessions/stop` and `/api/sessions/delete`

## Ollama Cleanup
- Removed 3 broken qwen3.5:35b* models from Ollama (~82GB freed)
- All 35b model routing goes to llama.cpp port 18080

## Verified Working
- Research Lab → llama.cpp: 2296 tokens in 58s on ideation stage
- Full 7-stage pipeline running end-to-end
- Paper auto-export (rs-c812484c has a real paper on RAG optimization)
- Autopilot survey: 4096 tokens, surveyed 4 sessions + 25 KB docs, proposed 3 research projects
- Chat proxy: works from mobile via port 8097
- All 9 services running, all ports responding

## Architecture Decisions
- Port 8097 (Research Lab) is now the universal API proxy for mobile — handles model listing, chat, model search/pull
- Async polling pattern for long-running ops (autopilot survey, model pull) instead of blocking requests
- Touch events added to 3D viz for mobile
- All 28 hub pages now have back-to-hub navigation

## Service Map (verified running)
| Service | Port | SystemD |
|---------|------|---------|
| Hub | 8090 | max-web-gallery |
| Stats API | 8091 | dgx-stats-api |
| Commands | 8092 | (see service) |
| Eval | 8093 | dgx-agent-eval |
| Status | 8094 | dgx-agent-status |
| Computer Use | 8095 | dgx-computer-use |
| Comms | 8096 | dgx-comms |
| Research Lab | 8097 | dgx-research-lab |
| Ollama | 11434 | system |
| llama.cpp | 18080 | jess-watchdog |
| Gateway | 18789 | openclaw-gateway |

## Session 2 (~01:14-01:34 EST) — Chat + Eval + Computer Use Fixes

### Model Lab Chat — Multi-Turn Conversation
- Chat was stateless (every message sent alone, no context window)
- Fixed: now sends full `messages[]` array with conversation history
- Both Ollama `/api/chat` and llama.cpp `/v1/chat/completions` support multi-turn
- Tested: 4b correctly remembers "My name is Pat" across turns, 35b too
- Vision images: only sent to vision-capable models; warning shown for non-vision models

### Eval Tests — Full Transparency
- Pat wanted to see the actual questions being tested (not just category names)
- Added all 16 test questions to Eval Details tab with exact prompts + pass criteria
- Categories: Reasoning (4), Code (3), Instructions (3), Knowledge (3), JSON (2), Speed (1)

### Computer Use V2 — Critical Bugs Fixed
- **Session ID mismatch**: Server created `ui-{ts}` but agent reported to `task-{ts}` → 0 steps shown. Fixed: both use `task-*`
- **Blank page instant quit**: No URL → `about:blank` → vision says "done". Fixed: starts at Google.com
- **Planner too weak**: Would quit on step 1. Added: min 3 actions, must include actual data, clear nav rules
- **Planner tokens**: 400→600 to prevent truncated JSON
- Commits: `f26ea44`

### Server.py `/api/chat` Rewrite
- Was single-turn (`/api/generate`). Now multi-turn: Ollama→`/api/chat`, llama.cpp→`/v1/chat/completions`
- Imported `_get_external_config` + `OLLAMA_URL` from models.py

## Session 3 (~01:34-01:42 EST) — Eval System Overhaul + Vision Fix

### All Qwen Models Now Vision-Enabled
- Removed vision gating in Model Lab — all Qwen models accept images
- Changed detection: `m.name.includes('qwen')` now matches (was only vl/llava/vision)
- Images always sent when attached, no warning dialog

### Eval System Rebuilt — File-Based Test Groups
- **evaluator.py** fully rewritten: tests are JSON-serializable with check type registry
- Check types: `contains`, `contains_any`, `exact`, `regex`, `line_count`, `valid_json`, `json_contains`, `code_has`
- Test groups saved to `~/.openclaw/computer-use-data/eval/test-groups/*.json`
- `get_tests(group_name)` resolves check functions at runtime from JSON definitions
- Default 16 tests preserved as built-in, always available

### New Eval API Endpoints (port 8093)
- `GET /api/tests` — returns full test definitions with prompts + check defs
- `GET /api/groups` — list all test groups with metadata
- `GET /api/groups/<name>` — get specific group
- `GET /api/groups/<name>/export` — export as JSON
- `POST /api/groups/import` — import JSON test group
- `POST /api/groups/generate` — AI generates tests from topic description (uses 4b)
- `POST /api/groups/delete/<name>` — delete a group
- `POST /api/evaluate` — now accepts `group` param
- `GET /api/status` — track running evals/generations

### Test Generation Working
- Generated "advanced-math-and-calculus" group: 10 tests, 5 categories
- Categories: Differentiation, Integration, Sequences/Series, Linear Algebra, Probability
- Uses qwen3.5:4b to generate questions + check definitions
- Auto-saves to test-groups directory

### Model Lab UI Updates
- Test group selector dropdown in Eval tab
- "Generate New" button → modal with topic input + model selector + async polling
- "Import JSON" button → textarea modal for pasting test groups
- "Export" button → downloads current group as JSON file
- Per-model eval results now show actual responses (click to expand)
- Eval trigger passes current test group to API

### Git Commits
- `9a405c0` — Full eval overhaul + vision fix
- `f26ea44` — Computer Use V2 fixes
- Backup snapshot pushed

### Still Running
- 4b eval against math group was kicked off, results pending

## Session 4 (~09:00-12:20 EST) — Computer Use V2 Fix + Video Understanding + GPU Management

### Computer Use V2 — Root Cause Diagnosis & Fix
**Root cause of failures (from log analysis of 10 recent sessions):**
1. **#1 killer: Google CAPTCHA** — Automated Playwright hitting Google → reCAPTCHA → agent stuck looping 15 times
2. **Selector timeouts** — Planner outputs CSS selectors that don't exist (Page.fill: Timeout 5000ms)
3. **Poor error recovery** — Errors in history but planner repeats same mistake
4. **Premature done** — Agent says "done" on step 1 with no data
5. **Missing params** — `goto` with no URL, `click` with no target

**Fixes applied (via sub-agent):**
- Default search engine → DuckDuckGo (no CAPTCHA)
- Fallback to Bing if DuckDuckGo fails
- Error recovery: "LAST ACTION FAILED" warning to planner
- Min 3 steps enforced before allowing "done"
- Param validation before executing actions
- Type fallback: if CSS selector fails, fall back to keyboard typing

**New features:**
- POST /api/sessions/continue — resume from where a session stopped
- POST /api/sessions/research — auto-research why a task failed
- GET /api/sessions/history — browse all past sessions
- GPU management panel (GET/POST /api/gpu/*)
- Continue + Research Fix buttons on session cards

### Video Understanding System (port 8099)
Built comprehensive video analysis pipeline:
- **YouTube**: yt-dlp → Whisper → ffmpeg frames → qwen3.5:4b vision → merge → summary
- **Local files**: Upload → ffprobe metadata → ffmpeg audio extract → same pipeline
- **Live streams**: iPhone camera (via OpenClaw nodes), RTSP, HTTP, screen capture
- **Real-time mode**: Frame every 10s + audio clip every 30s + running summary

Files: tools/youtube-understand/ (analyzer.py, server.py, cli.py)
Hub: 29-youtube-understand/ (later renamed to "Video Understanding")
Service: dgx-youtube-understand.service on port 8099

### Model Routing
- Added `_model_generate()` method to both VideoAnalyzer and LiveStreamAnalyzer
- Routes to Ollama (port 11434) for normal models
- Routes to llama.cpp (port 18080) for "jess:35b" selection
- Auto-falls back to Ollama 4B for vision tasks with Jess (llama.cpp doesn't support images)

### Full Model Dropdowns
Vision and Summary dropdowns now show ALL models in organized groups:
- ⚡ Fast: 0.8B, 2B, 4B, LLaVA
- 🎯 Quality: Qwen2.5-VL 7B, Qwen3-VL 8B, Qwen3 8/14B, DeepSeek-R1 14B
- 🧠 Best: Qwen3 32B, 30B MoE, Coder 30B, Flash 30B, GPT-OSS 20/120B, MiniMax M2.5
- 🔧 External: Jess (35B via llama.cpp)

### Ecosystem Expansion
Research Lab ecosystem now tracks 11 services (was 6):
- Added: Jess (18080), Gateway (18789), Video Understanding (8099), Hub (8090), Stats API (8091)
- Health check paths configured for each

### Mobile Access Fix
- Fixed hardcoded `localhost:8099` in Video Understanding → uses `window.location.hostname`
- Verified ALL hub pages use dynamic host detection (no remaining localhost refs)
- All ports reachable via Tailscale IP from phone

### Known Issues (to fix next session)
- Stats API (8091) keeps crashing (WorkingDirectory issue, pre-existing)
- Video Understanding real-time updates need improvement (like Computer Use step display)
- Health Monitor shows transient offline states
- Computer Use V2 needs testing with the DuckDuckGo fix
- Need self-healing: auto-restart crashed services, health watchdog

### Backup
- Git committed: 7079c36
- Backup sync pushed to All_Openclaw_agents repo