← Back to Neural Lab

# 🧬 Neural Lab Guide

A local AI research platform for DGX Spark — brain simulation, RL training, model surgery, and world building.

## 🏗️ Architecture Overview

### How It Runs

Neural Lab is a Python Flask + Socket.IO server running as a systemd user service on the DGX Spark. The UI is a single-page HTML/JS app served by the Clawboard hub.

    ┌──────────────────────────────────────────────────────────┐
    │  Your Phone / Browser                                     │
    │  http://100.109.173.109:8090/31-neural-lab/               │
    │  (HTML + JS + Three.js + Socket.IO client)                │
    └────────────┬─────────────────────────────────┬────────────┘
                 │ REST API                        │ WebSocket
                 ▼                                 ▼
    ┌──────────────────────────────────────────────────────────┐
    │  Neural Lab Backend (port 8103)                           │
    │  orchestrator.py — Flask + Flask-SocketIO                 │
    │  ├── SwarmOrchestrator (brain agents, message bus)        │
    │  ├── SimulationEngine (pymunk physics)                    │
    │  ├── NeuralLabPlatform (registry, assets, extensions)     │
    │  ├── RL Training (stable-baselines3 + PyTorch)            │
    │  └── Model Workshop (safetensors inspection)              │
    │  Service: neural-lab.service                              │
    │  Python: /home/pmello/.openclaw/neural-lab-env/           │
    └────────────┬─────────────────────────────────┬────────────┘
                 │ HTTP (Ollama API)               │ File I/O
                 ▼                                 ▼
    ┌─────────────────────────┐    ┌────────────────────────────┐
    │  Ollama (port 11434)     │    │  Local Files                │
    │  qwen3.5:0.8b (agents)  │    │  ~/.openclaw/neural-lab/    │
    │  qwen3.5:2b (prefrontal)│    │  ├── snapshots/             │
    │  qwen3.5:4b (vision)    │    │  ├── memories-*.json        │
    │                          │    │  ├── rl-models/             │
    │  llama.cpp (port 18080)  │    │  ├── worlds/                │
    │  122B-A10B (narrator)    │    │  ├── extensions/            │
    └─────────────────────────┘    │  └── logs/                  │
                                    └────────────────────────────┘

### Connection Flow

1. Hub server (port 8090) serves the static HTML page
2. Page loads and connects Socket.IO to port 8103 (the backend)
3. UI polls `/api/state` every 1-2 seconds for agent/brain state
4. Real-time events (messages, training progress) come via WebSocket
5. All brain agents call Ollama (0.8b/2b) for thinking
6. Camera/mic from your phone → backend API → Visual/Auditory Cortex agents
7. Multiple devices connect to the same backend (shared brain)

## 🚀 Quick Start

### 1. Start the Brain

Click 🧠 Start Brain. This launches 17 agents across 5 brain regions:

Region | Agents | Model | Role
--- | --- | --- | ---
🧠 Prefrontal Cortex | 3 | 2b | Planning, decisions, executive control
👁️ Visual Cortex | 3 | 0.8b | Process camera input, scene understanding
👂 Auditory Cortex | 3 | 0.8b | Process audio/speech (via Whisper)
🧩 Hippocampus | 3 | 0.8b | Memory formation, pattern recognition
💬 Broca's Area | 3 | 2b | Speech production, responses

### 2. Chat with the Brain

Use the 💬 Chat tab or the 🧠 Narrator widget. Your messages flow:

    You → Wernicke's (comprehension) → Prefrontal (thinking) → Broca's (response) → You

The Narrator is separate — it observes the brain and explains what's happening.

### 3. Run a Simulation

1. Go to 🏟️ Sim tab
2. Select an environment (Hide & Seek, Foraging, Maze, Open Field)
3. Click ▶ Start
4. Watch agents move in the 3D arena
5. Enable Brain-controlled to let the LLM drive agent decisions

### 4. Train with RL

1. Expand 🧪 RL Training at the bottom of the Sim tab
2. Pick algorithm (PPO, A2C, DQN), environment, and timesteps
3. Click 🚀 Train
4. Watch the reward curve update in real-time
5. Trained models auto-save to `~/.openclaw/neural-lab/rl-models/`

### 5. Inspect Models

1. Go to 🔬 Workshop tab — models auto-scan
2. Click any model to see layers, shapes, weight statistics
3. Click 🤖 AI Explain for a plain-language architecture summary
4. Click 📋 Duplicate for Editing to make a safe copy

### 6. Build Worlds

1. In 🏟️ Sim tab, expand 🏗️ World Builder
2. Select a tool (Wall, Box, Food, Flag, Ramp)
3. Click on the 3D arena to place objects
4. Save your world for later, or load environment templates from the 🧩 Platform tab

## 🧩 Platform & Extensions

Everything in Neural Lab is a plugin. The 🧩 Platform tab shows all registered components:

- 20 Object Prefabs — walls, boxes, food, traps, portals, water, mud, shelters
- 6 Physics Materials — concrete, ice, rubber, wood, metal, sand
- 5 Environment Templates — arena, maze, CTF, obstacle course, foraging
- 5 Sensors — vision cone, proximity, audio, compass, energy
- 3 Training Algorithms — PPO, A2C, DQN
- 4 Reward Functions — foraging, exploration, survival, team

### Writing Extensions

Drop a `.py` file into `~/.openclaw/neural-lab/extensions/`:

    # my_extension.py
    def register(registry):
        from platform import PluginEntry, PluginType
        registry.register(PluginEntry(
            id='env-my-world',
            name='My Custom World',
            type=PluginType.ENVIRONMENT,
            description='A custom environment',
            tags=['custom'],
        ))

## 💻 CLI Tools

    # Model Workshop
    workshop scan              # List all 15+ models
    workshop inspect gpt2      # See every layer & weight stats
    workshop layers gpt2 attn  # Filter to attention layers
    workshop explain whisper    # AI explains the architecture
    workshop duplicate gpt2 my-exp  # Safe copy for editing
    workshop python            # Interactive Python + safetensors

## 📡 API Reference

Endpoint | Method | Description
--- | --- | ---
`/api/state` | GET | Full brain state (agents, messages, connections)
`/api/start` | POST | Start brain with mode & agents config
`/api/ask` | POST | Send a message to the brain
`/api/narrate` | POST | Get narrator summary
`/api/sim/start` | POST | Start simulation {environment, num_agents}
`/api/sim/state` | GET | Current sim state (agents, physics, objects)
`/api/sim/world/add` | POST | Add object {type, x, y}
`/api/sim/world/save` | POST | Save world layout {name}
`/api/rl/start` | POST | Start RL training {algorithm, environment, total_timesteps}
`/api/rl/status` | GET | Training progress (episodes, rewards, timesteps)
`/api/workshop/scan` | GET | Scan for model files
`/api/workshop/inspect` | POST | Inspect a model {path}
`/api/platform` | GET | Full platform state (registry, assets, extensions)
`/api/platform/templates/<id>/apply` | POST | Apply environment template

## 🔧 Service Management

    # Check status
    systemctl --user status neural-lab.service

    # Restart after code changes
    systemctl --user restart neural-lab.service

    # View logs
    journalctl --user -u neural-lab.service -f

    # The service uses a Python venv with all dependencies:
    # /home/pmello/.openclaw/neural-lab-env/
    #   pymunk, torch, stable-baselines3, gymnasium, pettingzoo,
    #   safetensors, flask, flask-socketio, requests

## 📁 File Structure

    ~/.openclaw/workspace-main/tools/neural-lab/
    ├── orchestrator.py    # Main server (~2700 lines)
    ├── simulation.py      # Physics engine (pymunk)
    ├── rl_env.py          # Gymnasium RL environment wrapper
    ├── model_workshop.py  # Model inspection/surgery
    ├── platform.py        # Plugin registry, assets, extensions
    ├── ARCHITECTURE.md    # Full technical specification
    └── RESEARCH.md        # Research directions

    ~/.openclaw/hub/31-neural-lab/
    ├── index.html         # Main UI (~3300 lines, 10 tabs)
    ├── guide.html         # This guide
    └── lib/three.min.js   # Three.js (local copy)

    ~/.openclaw/neural-lab/
    ├── snapshots/         # Brain state snapshots
    ├── memories-*.json    # Agent memories per session
    ├── rl-models/         # Trained RL models (.zip)
    ├── worlds/            # Saved world layouts
    ├── extensions/        # Custom plugins (.py)
    └── logs/              # Session logs

    ~/models/workshop/     # Model library (safetensors)
    ├── bert-tiny-4m/      # 4M params — fast experiments
    ├── gpt2-124m/         # 124M — classic transformer
    ├── whisper-tiny-39m/  # Audio transcription model
    ├── clip-vit-base-150m/# Vision+text multimodal
    ├── tinyllama-1.1b/    # Llama architecture
    └── ... (15 models total)

## ⚡ Hardware Requirements

Component | Requirement | DGX Spark
--- | --- | ---
RAM | 16GB minimum | 128GB ✓
GPU | Optional (CPU training works) | GB10 ✓
Ollama | 0.8b model minimum | 0.8b, 2b, 4b ✓
Python | 3.12+ | 3.14 ✓
Storage | ~10GB for models | 1.9TB free ✓

Neural Lab v2.2 — Built on DGX Spark with ⚡ by the ARC platform