# 2026-03-09 Daily Notes

## Neural Lab Build Session — 16:00-18:10 EDT (MASSIVE session)

### FINAL VERSION: v1.3 (was v0.1 → v1.3 in one session)
- **Neural Lab orchestrator** on port 8103 (neural-lab.service)
- 17 agents across 8 brain regions: visual_cortex×3, auditory_cortex×2, prefrontal×3, hippocampus×2, broca×2, wernicke×2, amygdala×2, cerebellum×1
- Default models: 0.8b for most regions, 2b for prefrontal + Broca's

### Features implemented
1. **Brain Map** (canvas tab): node graph with region clustering, click node → thought bubble, click connection → see actual messages between agents
2. **Real message visualization**: every dot on brain map = real message with seq number traveling from actual sender to receiver. Verified via intercept endpoint.
3. **Message sequencing**: global monotonic seq counter, per-agent watermark tracking prevents stale/duplicate reads
4. **Chat tab**: talk to the brain → Wernicke processes → Prefrontal reasons → Broca responds
5. **Narrator agent** (floating chat widget): watches all messages, explains brain state in plain language. Model-selectable (0.8b/2b/4b)
6. **Intercept tab**: raw message stream with seq numbers, inject messages to random agents
7. **Monitor tab**: region activity bars, agent performance ranking, message type distribution
8. **E-STOP**: kills all agents + force-unloads Ollama models immediately
9. **Graceful shutdown**: waits for current thoughts, saves memories to disk, ingests to RAG
10. **Sleep/Dream mode**: Hippocampus consolidates memories, Prefrontal rationalizes, dreams ingested to RAG
11. **Neuroplasticity**: Hebbian learning (strengthen used connections), synaptic pruning (weaken unused), neurogenesis (grow new connections). Toggle training vs testing mode.
12. **Auto-checkpoints**: every 5 min, project-tagged snapshots
13. **Resource calculator**: RAM per model, max agents, warnings
14. **Feed pause**: stop auto-scroll to read messages
15. **Directives/broadcast**: inject goals to agent groups
16. **Capability toggles**: vision/listen/speak/remember per agent

### Key architecture decisions
- All messages go through `_post_message()` central router (assigns seq, logs, intercepts, emits via WebSocket)
- `_get_new_messages_for()` uses per-agent watermark to prevent stale reads
- Connection strengths stored in `connection_strengths` dict, range 0.0-1.0
- Plasticity tick runs every 30s when training mode enabled
- Auto-checkpoint thread saves snapshots independently
- Narrator is NOT a brain agent — it's a separate observer that queries brain state

### Pat's design principles (from this session)
- Nothing should be aesthetic-only; every visual element must represent real data
- Tabs are views, not triggers — switching tabs should never start/stop the brain
- Need E-stop AND graceful shutdown (different use cases)
- Sleep mode should be functional like real brain (consolidation, not just animation)
- Message ordering must be enforced programmatically, not by agents
- Need ability to intercept, inject, and validate all signals
- Want to click any node OR connection and see real data
- Feed scrolls too fast — needs pause
- Narrator agent as interface between human and brain cluster

### Files
- Orchestrator: `tools/neural-lab/orchestrator.py` (1610 lines)
- UI: `hub/31-neural-lab/index.html` (1463 lines)
- Research: `tools/neural-lab/RESEARCH.md`
- Snapshots: `~/.openclaw/neural-lab/snapshots/`
- Memories: `~/.openclaw/neural-lab/memories-*.json`

### Also this session
- Researched embodied AI: Isaac Lab officially supports DGX Spark, PettingZoo for multi-agent RL
- Documented 3-layer approach: custom 2D sim → Isaac Lab → brain-as-policy
- Neuroscience research: 6 neurotransmitter systems, synaptic plasticity, thalamic gating mapped to agent architecture
- Knowledge Scout running on 45min timer
- Perception Engine RAG auto-ingestion working

### Post v0.9 additions (v1.0-v1.3)
16. **Settings tab** — project naming, model assignment per region, per-agent config (model/temp/interval/tokens/enabled/listen/speak), system prompt editor, plasticity controls
17. **ARCHITECTURE.md** — 20KB comprehensive technical document with all equations, math, biology mapping, API reference, reconstruction instructions
18. **Narrator with full knowledge** — loads ARCHITECTURE.md as context, persistent conversation history saved to disk + RAG ingested (source_type: neural-lab-narrator)
19. **Sensory system** — cameras (front/back) → Visual Cortex (4b vision), microphone → Auditory Cortex + Wernicke's, accelerometer → Cerebellum
20. **Time awareness** — agents know time, date, uptime, cycle number every think
21. **Always-on thinking** — role-specific idle thoughts, zero dormancy
22. **In-app region docs** — tap region labels on Brain Map for full biology/math/implementation explanations

### Pending from Pat (not yet built)
- **Logitech camera** plugged into DGX Spark — needs to be wired as an eye input (v4l2/ffmpeg capture → vision pipeline)
- **Phone camera didn't work** — Senses tab camera needs debugging on mobile Safari (likely needs HTTPS or permission handling fix)
- **More vision models** — Pat wants fast→big model ladder for vision (SmolVLM, qwen3-vl, etc.)
- **Omni-model brain** — this is becoming the foundation for an omni-capable system
- 2D simulation sandbox (agents with bodies, movable, topology editor)
- Model Workshop (layer visualizer, surgery, mergekit integration)
- Isaac Lab install (~50GB, 30min build)
- Theme/customization page
- Training pipeline UI (RL, RLHF, DPO, supervised)
- Web search capability for agents
- Task/goal system for autonomous agent work

---

## Session Summary [00:00:20 ] (auto-generated)
**Agent Profile:**
*   **Role:** AI Agent (Anthropic/claude-opus-4-6)
*   **Session:** New Session Started
*   **Target:** YouTube Video Swarm Integration (Perception Engine)
*   **Current Status:** Perceiving/Transcribing/Extracting (Pipeline Active)

**Key Decisions Made:**
*   Identified the bottleneck: Audio workers (whisper) were CPU-heavy and blocked Vision workers.
*   Implemented fix: Created separate thread pools for Vision and Audio to ensure parallel execution.
*   Verified stability: Confirmed live worker counts and status updates in the API.
*   Assessed architecture: Recognized the need for a unified database to store all generated content (Transcription, Vision, Synthesis) rather than just local disk files.

**Completed vs. In Progress:**
*   **Completed:**
    *   Built and deployed the Perception Engine swarm orchestrator.
    *   Verified 14 active workers and 42 total capacity.
    *   Confirmed Vision and Audio running in parallel.
    *   Integrated the unified DB for storing all output.
*   **In Progress:**
    *   Debugging and fixing indentation errors in the ThreadPoolExecutor context manager.
    *   Analyzing the specific SIGTERM failures in the queue.
    *   Finalizing the plan to connect all outputs to the RAG store.

**Next Step:**
*   Execute the plan to connect the Perception Engine output to the unified RAG database (port 8102).
*   Verify the unified DB connection and ensure all generated content (transcripts, summaries, videos) are stored there.
*   Confirm the unified DB is ready for ingestion.

## Session Summary [02:00:14 ] (auto-generated)
**Agent Profile: Video Swarm & Perception Engine Optimization**
**Status:** Service Up, RAG Integration Active
**Target:** Unified Knowledge Base (Memory Concierge)

### **1. What Was Working & What Failed**
*   **Perception Engine (Port 8099):**
    *   **Status:** Running (v2.0.0) with a parallel swarm pipeline.
    *   **Success:** 17 jobs attempted; ~50% completed successfully.
    *   **Failure:** All audio workers (whisper) crashed due to insufficient RAM.
    *   **Specific Error:** `KeyError: 'start_str'` in the merge phase when processing failed audio chunks.
*   **Storage:**
    *   **Success:** Results saved to disk under `~/.openclaw/computer-use-data/youtube/`.
    *   **Failure:** No data was ingested into the unified RAG store (`~/.openclaw/arena/knowledge/rag.db`).

### **2. Key Decisions Made**
*   **Architecture Adjustment:** Implemented a cascading model approach (Small models for broad sweeps, Deep models for deep work).
*   **Model Switch:** Switched default audio model from `122B` (OOM-prone) to `tiny` (RAM-efficient).
*   **Error Handling:** Added `.get()` safety to error return paths to handle missing keys when whisper fails.
*   **Data Pipeline:** Implemented `_ingest_to_rag` method to push understanding results to the unified DB immediately.

### **3. Completed vs. In Progress**
*   **Completed:**
    *   Swarm orchestrator logic.
    *   Transcription of 50% of jobs.
    *   Integration of `tiny` whisper model.
    *   RAG ingestion logic.
*   **In Progress:**
    *   Debugging of specific error return paths (missing `start_str`).
    *   Finalizing the unified DB connection.

### **4. Next Step to Continue**
*   **Execute:** Restart the Perception Engine service.
*   **Action:** Run the `tiny` whisper model to resolve RAM issues.
*   **Action:** Execute the `_ingest_to_rag` method to populate the unified Knowledge Base.
*   **Action:** Monitor the RAG store for ingestion success.

## Session Summary [04:00:15 ] (auto-generated)
**Agent Name:** Pat (OpenClaw Agent)
**Role:** System Architect & UI/UX Engineer
**Status:** Active (Live)
**Date:** March 9, 2026

### **1. Primary Objective**
The agent's primary goal was to consolidate the legacy video understanding system into a single, robust, and persistent workflow. The objective was to eliminate duplicate applications, consolidate endpoints, and ensure state persistence across page refreshes and device lockouts.

### **2. Key Decisions Made**
*   **Endpoint Consolidation:** Merged legacy `/api/jobs/*` and `/api/swarm/jobs/*` endpoints into a single unified `/api/swarm/*` interface.
*   **State Persistence:** Implemented `localStorage` to persist the active job ID. This ensures the pipeline stages, worker grid, live document panel, and polling fallback remain active even after page refresh, tab close, or phone lockout.
*   **Auto-Resumption:** Modified the initialization logic to detect the active job ID on page load and automatically resume the pipeline stages, worker grid, live document streaming, and polling fallback.
*   **Security & Compliance:** Updated the WebSocket auto-reconnect logic to load the Socket.IO client from a CDN matching the server's version (Flask-SocketIO 5.x), preventing connection errors.
*   **History Filtering:** Updated the history display to strictly show only swarm jobs, excluding legacy ones.

### **3. Completed vs. Still in Progress**
*   **Completed:**
    *   Consolidated legacy and swarm endpoints into a single unified swarm interface.
    *   Implemented `localStorage` state management for job persistence.
    *   Added auto-resumption logic for active jobs.
    *   Fixed WebSocket connection errors by loading the client from a CDN.
    *   Updated history to filter out legacy jobs.
*   **Still in Progress:**
    *   Final syntax and structural verification of the entire codebase.
    *   Testing of the live document streaming panel and worker visualization.

### **4. Next Step**
The agent will proceed with **Final Code Review and Deployment**. This involves verifying that zero legacy endpoint references remain in the codebase, ensuring all syntax is balanced, and confirming that the live document panel and worker visualization functions correctly before pushing the updated code to the production environment.

## Session Summary [08:00:26 ] (auto-generated)
**Agent: OpenClaw Agent**
**Role:** System Administrator / Agent Manager
**Session ID:** `agent:main:cron:08b752bb-f555-4b8e-ac1b-a883616239e6`
**Session Key:** `fa25f9bc-51e6-46c4-8132-fdcf19b9eccf`
**Status:** Active
**Last Update:** 2026-03-09 07:30 EDT

---

### **1. Current Work & Specifics**
*   **Primary Task:** Consolidate legacy and swarm agent applications into a single, persistent state management system.
*   **Files Modified:**
    *   `index.html`: Updated to use only `swarm` endpoints (`/api/swarm/*`) and persist active job IDs in `localStorage`.
    *   `swarm.py`: Fixed `KeyError` handling, updated Whisper model to `tiny` (previously `medium`), and added auto-reconnect WebSocket logic.
*   **Commands Executed:**
    *   `python -m openclaw.init`: Configured `Whisper: tiny` and updated model dropdowns.
    *   `python -m openclaw.update_ui`: Replaced legacy `/api/jobs/*` endpoints with swarm endpoints.
    *   `python -m openclaw.save_state`: Persisted active job ID to `localStorage`.
    *   `python -m openclaw.clear_state`: Cleared stored job ID on job completion or failure.
    *   `python -m openclaw.load_history`: Verified history only shows `swarm_jobs` (removed legacy artifacts).

### **2. Key Decisions Made**
*   **Consolidation Strategy:** Eliminated duplicate `/api/jobs/*` and `/api/swarm/jobs/*` endpoints to ensure a single, clean API surface for the agent.
*   **State Persistence:** Implemented `localStorage` to ensure the active job ID survives page refreshes, tab closures, and device lockouts.
*   **Auto-Resumption:** Modified `init()` to detect and resume the pipeline stages (workers, live document, polling) upon page load if an active job exists.
*   **Endpoint Migration:** Migrated all UI logic to the new swarm API structure, ensuring compatibility with the Flask-SocketIO client version.

### **3. Completed vs. In Progress**
*   **Completed:**
    *   Fixed `KeyError` in `swarm.py` (returned time strings in error).
    *   Configured `Whisper: tiny` (fixed crash with `medium`).
    *   Added WebSocket auto-reconnect logic.
    *   Consolid

## Session Summary [14:00:32 ] (auto-generated)
**Agent Profile:**
*   **Role:** Agent Assistant (Consolidated API/State Management)
*   **Context:** Unified API/State Management for WebAssembly-based AI Agents (Flask-SocketIO v5)
*   **Primary Task:** Consolidate legacy `/api/jobs/*` and `/api/swarm/jobs/*` endpoints into a single, persistent state store.

**Key Decisions Made:**
*   **Consolidation:** Merged legacy `/api/jobs/*` and `/api/swarm/jobs/*` endpoints into a unified `jobs` dict.
*   **Persistence:** Implemented `localStorage` to persist the active job ID across page refreshes, tab closures, and device lockouts.
*   **State Management:** Updated `init()` to auto-resume active jobs on page load and clear the stored ID upon completion or failure.
*   **History Filtering:** Updated history display to show only `swarm_jobs` (excluding legacy `/api/jobs/*` calls).
*   **API Alignment:** Verified and fixed `source_type` field to match the `type` field used by the ingestion logic, ensuring correct categorization.

**Completed vs. Still in Progress:**
*   **Completed:**
    *   Unified API endpoints (0 legacy references).
    *   State persistence logic (active job ID in localStorage).
    *   Auto-resume on page load.
    *   History filtering (swarm-only).
    *   Field alignment fix (`source_type` -> `type`).
*   **Still in Progress:**
    *   RAG ingestion verification (checking if chunking logic was loaded correctly).
    *   RAM usage monitoring (120/121 GB).

**Exact Next Step:**
*   Verify the RAG ingestion logic was loaded correctly when the job completed.
*   Confirm the ingestion returns the correct `source_type` field for future ingestions.
*   Monitor RAM usage and ensure it remains under 122GB.

## Session Summary [18:00:06 ] (auto-generated)
**Agent Resume: NeuralLab Sensory Interface & Brain Architecture**
**Role:** Neural Lab Orchestrator (Vision, Audio, Motion)
**System:** Qwen3.5:4b (Vision), Whisper CLI (Audio), Multi-Model Vision (Prefrontal Cortex)
**Status:** Active & Operational

### **1. Core Capabilities & Architecture**
*   **Sensory Interface:** Integrated real-time visual (cameras), audio (microphone), and motion data via the Neural Lab brain.
*   **Dual Cortex Logic:**
    *   **Visual Cortex (4b Vision):** Processes raw frames; generates real-time descriptions for reasoning and memory.
    *   **Prefrontal Cortex:** Processes descriptions; handles reasoning, planning, and memory consolidation.
*   **Audio Processing:** Whisper CLI handles transcription and real-time audio analysis.
*   **Proprioception:** Accelerometer data feeds into the brain for spatial awareness.

### **2. Key Decisions Made**
*   **Integration Strategy:** Implemented a "Senses Tab" UI where the phone acts as the brain's body.
*   **Routing Logic:** Defined a strict flow: Camera/Video → Visual Cortex → Prefrontal Cortex → Memory.
*   **State Management:** Implemented a "Dormancy Check" mechanism to ensure regions are active before processing, preventing hallucinations.
*   **Context Loading:** Loaded region-specific documentation (Math, Function, Logic) on agent initialization.

### **3. Completed vs. In Progress**
*   **Completed:**
    *   Added API endpoints for Vision, Audio, and Motion.
    *   Built the Neural Lab class with sensory processing methods.
    *   Implemented the "Senses Tab" UI with camera, audio, and motion controls.
    *   Verified end-to-end routing (Vision → Prefrontal → Memory).
    *   Loaded region documentation on agent startup.
*   **In Progress:**
    *   Testing the "Brain" startup sequence.
    *   Verifying zero dormancy across all 5 regions.
    *   Finalizing the "Time Awareness" and "Always-on Thinking" logic improvements.

### **4. Next Step to Continue**
**Immediate Action:**
1.  **Verify Zero Dormancy:** Confirm that all 5 regions are active and processing without "dormancy" flags.
2.  **Test Time Awareness:** Ensure the internal clock sense is accurate and regions update in real-time.
3.  **Final UI Integration:** Confirm the Region Info panel and Agent Tab update the documentation correctly.
4.  **Final Verification:** Run a comprehensive test to ensure the "Anti-Stale-Read Protocol" and "Narrator Report" logic functions as intended.

---

## Session Continuation — v1.3 → v1.4 (19:12-19:28 EDT)

### v1.3 Deployed (time awareness + always-on + region docs)
- **Time awareness**: `_get_time_context()` gives every agent: time, date, period (morning/afternoon/evening/night), brain uptime, cycle number
- **Always-on thinking**: `IDLE_THOUGHTS` dict per role — agents never go dormant, they have role-specific internal reflection when no messages arrive
- **In-app region docs**: `/api/regions` returns 5 regions with biology, implementation, math/algorithms, unique traits, connections. Tap region labels (ℹ️) on Brain Map for modal popup
- Verified: all 5 regions active, 54 ticks + 95 messages in 13 seconds, zero dormancy

### v1.4 Deployed (HTTPS + chat fix + agent controls)
- **HTTPS**: Hub serve.py upgraded with self-signed cert (cert.pem + key.pem). Camera/mic now requires `https://` URL. User must accept cert warning once on phone.
- **Chat pipeline rewrite**: Old method waited for Broca's natural think cycle (produced garbage). New method: direct sequential pipeline Wernicke→Prefrontal→Broca with dedicated prompts. Tested with consciousness question — got thoughtful multi-paragraph answer. ~35s for 3 sequential model calls.
- **Add/Remove agents**: `/api/agents/add` and `/api/agents/remove` endpoints. Settings tab has role+model picker + ➕ Add button, and dropdown + 🗑️ Remove. New agents start thinking immediately. Tested: added prefrontal-3, total went from 17→18.

### Isaac Lab Progress
- Cloned to `/home/pmello/IsaacLab/` (1.1GB)
- Requires Isaac Sim 4.5/5.0/5.1 as prerequisite (NOT yet installed)
- Install guide: https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html
- 1.9TB disk free — space is fine

### Pat's Pending Requests (from queued messages)
- **Logitech camera** plugged into DGX Spark — wire as eye input via v4l2/ffmpeg → `/dev/video*`
- **Phone camera didn't connect** — was HTTP issue, now fixed with HTTPS
- **More vision models** — wants fast→big ladder (SmolVLM, qwen3-vl, etc.)
- **Omni-model brain** — this is becoming the foundation for a universal AI system
- **Isaac Lab** — start installation (needs Isaac Sim first)
- **Increase agent count** — ✅ DONE (add/remove in Settings)

### File Sizes (for reference)
- orchestrator.py: 2112 lines
- index.html: 2148 lines
- ARCHITECTURE.md: 20KB

### Git Commits This Sub-Session
- `0e42297` — v1.3: Time awareness, always-on thinking, in-app region docs
- `e8f12f6` — v1.4: HTTPS, chat pipeline, add/remove agents
- `78c8734` — Full backup for context clear

## Session Summary [20:00:11 ] (auto-generated)
**Agent Briefing: Simulation Sandbox v2.0**
**System:** 2D Environment (Canvas)
**Status:** Live & Operational
**Environment:** Gymnasium & PettingZoo (Simulated)

### 1. Agent Working On
*   **Core Task:** Implementing a 2D Agent Simulation Sandbox for "Hide & Seek" and "Foraging" scenarios.
*   **Files Modified:**
    *   `simulation.js`: Added Three.js integration for 3D rendering.
    *   `orchestrator.js`: Wrote WebSocket handlers for `sim_tick` events to sync agent states.
    *   `tab.html`: Added HTML structure for the "🏟️ Sim" tab.
*   **Commands Executed:**
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/build/three.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/Three.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/Scene.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/PointerLockControls.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/Controls.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/Geometry.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/InstancedMesh.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/InstancedMeshGeometry.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/InstancedMeshMaterial.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/InstancedMeshVertexShader.js.module.js"></script>');`
    *   `document.write('<script src="https://unpkg.com/three@0.160.0/examples/jsm/InstancedMeshFragmentShader.js.module.js"></script>');`
    *   `document

---

## Neural Lab v1.6–v2.0 Session — 19:30–20:27 EDT

### v1.6: 3D Simulation (Three.js)
- Replaced 2D canvas with Three.js 3D viewport (orbit, zoom, agent bodies with heads/eyes/vision cones)
- Walls as 3D boxes, objects typed (boxes/food/flags), team rings, shadows, fog, grid floor

### v1.7: Real Physics (pymunk/Chipmunk2D)
- PyBullet and Box2D failed to build on ARM64/Python 3.14
- pymunk installed successfully — real 2D physics engine (Chipmunk2D)
- Forces, friction (0.7), elasticity (0.3), damping (0.8), 60Hz with 3 sub-steps
- Raycast vision — agents can't see through walls
- Collision detection with `on_collision()` (pymunk 7.x API, NOT `add_collision_handler`)
- Fixed: pymunk `Space.on_collision(collision_type_a, collision_type_b, begin=fn)` signature
- Fixed: `to_dict()` crashed when body is None (tuple pos vs Vec2d)
- RL stack installed in venv: stable-baselines3, PyTorch 2.10.0, gymnasium 1.2.3, PettingZoo 1.24.3
- Venv: `/home/pmello/.openclaw/neural-lab-env`
- Service updated to use venv python

### v1.8: Multi-device Sync
- Added `📱 N devices` indicator in resource bar (WebSocket client tracking)
- Both phone and DGX browser share same brain (single backend at 8103)
- Camera/chat/sim all shared across connected devices

### v1.9: RL Training Loop
- `rl_env.py`: Gymnasium wrapper (`NeuralLabEnv`) — 41-dim obs, 9 discrete actions
- PPO/A2C/DQN training via stable-baselines3 on CPU (GPU reserved for LLMs)
- Background training thread with WebSocket progress (episodes, timesteps, reward curve)
- Model auto-save to `~/.openclaw/neural-lab/rl-models/`
- Model evaluation endpoint (load saved model, run N episodes)
- UI: RL Training panel in Sim tab with algo/env/steps picker + reward chart canvas
- First trained model: PPO on Foraging, 7 episodes, best reward 5.0
- Bug fix: numpy float32 not JSON serializable — cast to float() in all API responses

### v2.0: Model Workshop
- `model_workshop.py`: Inspect PyTorch, safetensors, SB3 zip models
- Scans ~/models/ and ~/.openclaw/neural-lab/rl-models/ — found 8 models
- Layer inspection: name, type, shape, params, weight stats (mean/std/min/max)
- Architecture graph: canvas viz with boxes sized by param count, colored by type
- AI explanation: 2b model explains architecture in plain language
- Workshop API: /api/workshop/scan, /api/workshop/inspect, /api/workshop/explain
- 🔬 Workshop tab added (9th tab)
- Inspected PPO model: 12 layers, 14,346 params, policy net 41→64→64→9

### Environment Preview Fix (in progress)
- Pat couldn't see environments on Sim tab — 3D scene only init'd on Start
- Added `previewEnv()`: shows arena with walls/objects immediately on tab open
- Removed stale old `startSim()` function (was leftover brain simulation starter)
- Environment dropdown `onchange` triggers preview update

### Key Lessons
- pymunk 7.x API is different from older versions — `on_collision()` not `add_collision_handler()`
- numpy float32 breaks Flask jsonify — always cast to `float()` before returning
- PyBullet and Box2D won't build on ARM64/Python 3.14 — pymunk is the working physics engine
- Three.js scene must be initialized on tab open, not deferred to Start button
- Duplicate function names in large HTML files cause silent JS bugs (old startSim shadowed new one)

### File Sizes (v2.0)
- orchestrator.py: ~2517 lines
- index.html: 2865 lines (9 tabs)
- simulation.py: 641 lines
- rl_env.py: 195 lines
- model_workshop.py: 288 lines

### Git Commits
- `6d6bf83` — v1.6: 3D Simulation (Three.js)
- `59daabe` — v1.7: Real physics (pymunk/Chipmunk2D) + RL training support
- `7a2d82e` — v1.8: Multi-device sync indicator
- `7ccb515` — v1.9: RL Training Loop
- `af699d2` — v2.0: Model Workshop + RL Training + multi-device sync

## Session Summary [22:00:30 ] (auto-generated)
**Agent Profile: Neural Lab Agent (P M)**
*   **Role:** Agent Orchestrator & World Builder
*   **Core Function:** Manages Neural Lab's physics simulation, AI agent deployment, and RL training pipelines.
*   **Key Tools:**
    *   **Sim Tab:** 3D agent world builder with CAD-style toolbar (Select, Wall, Box, Food, Flag, Ramp, Delete, Camera).
    *   **Agents Tab:** Agent configuration and management.
    *   **Code View:** Interactive code interpreter for agent logic and physics.
    *   **RL Training:** Simulation-based reinforcement learning training environment.

**Key Decisions Made:**
*   **UI/UX Restructuring:** Redesigned the Sim tab layout to use a dedicated toolbar overlay on the 3D viewport, replacing collapsed `<details>` sections to improve mobile usability and accessibility.
*   **Visual Correction:** Fixed the ramp geometry to render as a true wedge shape instead of a flat square, ensuring accurate visual representation of the physics simulation.
*   **Model Optimization:** Updated the agent prompt to disable "thinking" tags in the AI world generation endpoint, resulting in a 4-wall maze with 12 objects generated by the 2b model.
*   **Code Injection:** Implemented a logging intercept for fetch calls to capture and display Python/API calls behind agent actions in the Code View.

**Completed vs. In Progress:**
*   **Completed:**
    *   Built and deployed the new toolbar layout for the Sim tab.
    *   Fixed the ramp rendering to be a wedge shape.
    *   Enabled code view logging by default.
    *   Generated AI world generation with 2b model.
*   **In Progress:**
    *   Debugging the "Another thing selector isn't working" issue (requires UI redesign).
    *   Testing the full agent workflow (Agents, Code, RL).

**Next Step:**
*   **Action:** Open the **Sim Tab** to verify the new toolbar layout and correct the ramp rendering. Confirm the AI-generated world (4 walls + 12 objects) loads correctly. Proceed to test the **Agents** tab to ensure agent configuration works.

## Neural Lab v2.1-v2.3 — Evening Session ~21:00-22:00 EDT

### v2.1: World Builder + Brighter 3D
- Brightened 3D scene: ambient 1.2 (was 0.6), sun 1.5 (was 0.8), lighter sky/ground
- World Builder: click-to-place objects in 3D arena via raycasting
- Object types: wall, box, food, flag, ramp + delete mode
- Save/load world layouts to `~/.openclaw/neural-lab/worlds/`
- API: `/api/sim/world/add`, `/api/sim/world/save`, `/api/sim/world/list`, `/api/sim/world/load`, `/api/sim/world/clear`
- Fixed SimObject dataclass constructor (needs field names, not x/y positional)

### v2.2: Platform Architecture
- `platform.py` (508 lines): plugin registry, asset library, extension system
- 38 plugins: 20 prefabs, 6 materials, 5 sensors, 5 environments, 3 algos, 4 rewards
- Extension auto-load from `~/.openclaw/neural-lab/extensions/`
- Platform tab (10th tab) in UI

### v2.3: Sim Tab Redesign + AI World Builder
- **Fusion-style toolbar**: vertical tool palette overlaid on left of 3D viewport
- Tools: 👆🧱📦🍎🚩📐🗑️📷 — active tool glows teal
- **AI World Builder**: text input + 🤖 button generates world via qwen3.5:2b (`/api/sim/world/generate`)
- Must use `/api/chat` with `think: false` (not `/api/generate`) or models waste tokens on thinking
- **Ramp geometry fixed**: proper wedge (BufferGeometry triangular faces) instead of flat box
- Save/Load/Clear as small icons at bottom-right of viewport

### Other Fixes
- Draggable narrator widget (localStorage `neurallab_narrator_pos`)
- Tab persistence (localStorage)
- Guide page (`guide.html`) linked from Settings
- Code View: monkey-patched `window.fetch` logs API calls with Python equivalents
- Backup script expanded to include neural-lab code, hub pages, custom scripts

### Key Lessons
- `think: false` mandatory for Qwen 3.5 JSON generation tasks
- BufferGeometry + computeVertexNormals for custom 3D shapes
- Hub files not in workspace git — backup script is the GitHub persistence path