# Migration Plan: AWS EC2 → Hetzner Cloud

## Decision

Moving Craig from AWS EC2 (t3.large, 2 vCPU, 8 GB, $61/mo) to Hetzner Cloud
(CCX43, 16 dedicated vCPU, 64 GB RAM, 360 GB NVMe, ~$108/mo). 8x CPU, 8x RAM,
dedicated cores (no burst throttling), Ashburn VA datacenter (same region as
current us-east-1).

**Why:** Craig keeps hitting RAM limits. 8 GB is not enough for OpenClaw gateway
+ building Electron apps, running test suites, or compiling Rust projects. AWS
pricing for equivalent dedicated-CPU machines is 5-6x more expensive than
Hetzner Cloud for the same specs.

**Why not bare metal (Hetzner AX line):** No `hcloud` CLI support. We want CLI
provisioning for scriptability and easy resizing.

**Why CCX43 and not CCX33:** Going mega. 64 GB means RAM is never a concern
again. 16 dedicated vCPUs handle parallel builds and multiple subagents. Can
always downgrade to CCX33 ($54/mo) later if it's overkill.

## Non-goals

- No infrastructure-as-code (Terraform, etc). Craig is one box with SSH.
  Keep it simple.
- No container orchestration. Same bare-Ubuntu + systemd setup as today.
- No multi-region. Single Ashburn box.

## What needs to transfer

| Asset | Location on EC2 | Transfer method |
|-------|----------------|-----------------|
| CraigClaw workspace | `~/Code/merit-systems/CraigClaw` | Fresh `git clone` |
| OpenClaw fork | `~/Code/openclaw/openclaw` | Fresh `git clone` |
| Craig's memories | `memory/*.md` | In git, comes with clone |
| `secrets.json` | `~/.openclaw/secrets.json` | `scp` via local machine |
| GitHub App key | `~/.config/craig/github-app-key.pem` | `scp` via local machine |
| x402 wallet | `~/.x402scan-mcp/wallet.json` | `scp` via local machine ($252 USDC) |
| Email dedup state | `~/.craig-notified-emails` | `scp` — prevents re-notifying old emails |
| Proactive task gate | `~/.craig-last-proactive` | `scp` — or let it reset (harmless) |
| Installed skill packs | `~/.agents/skills/` | Re-install via `deploy/skills.txt` |
| OpenClaw config | `~/.openclaw/openclaw.json` | Regenerated by deploy |
| Credential scripts | `~/.config/craig/*` | Deployed from repo |
| Systemd services | `~/.config/systemd/user/` | Created by onboard + deploy |
| Cloned work repos | `~/Code/merit-systems/{poncho,enrichx402,...}` | Craig re-clones as needed |
| Swapfile | `/swapfile` (2 GB) | Not needed — 64 GB RAM |
| Installed packages | Node 22, pnpm, gh, PyJWT | `setup.sh` handles |

**Key insight:** Almost everything is in git or regenerated by the deploy
pipeline. Only 3 secret files + 1 state file strictly need manual transfer.

## Cutover strategy

**Both Craigs will be online simultaneously during migration.** This will cause
temporary confusion in Discord (two bots responding). This is acceptable — we
want to verify new Craig is fully functional before killing old Craig.

Sequence:
1. Provision new box, deploy, verify Craig works
2. Both Craigs alive for a short window (minutes to hours)
3. Stop old Craig's `openclaw-gateway` service (silences him, keeps box alive)
4. Verify new Craig is sole responder
5. Leave old EC2 running (stopped service) for a day as rollback
6. Terminate old EC2

## Steps

### Phase 0: Audit EC2 state before migration

SSH into the EC2 and verify everything important is either in git or accounted
for in the transfer plan. **Do not proceed until this audit passes.**

```bash
ssh -i ~/.ssh/craigclaw-key.pem ubuntu@[REDACTED_PHONE]
```

**0a. Verify memories are committed and pushed:**
```bash
cd ~/Code/merit-systems/CraigClaw
git status
# If uncommitted memory files exist, commit and push them NOW:
# git add MEMORY.md memory/ USER.md
# git commit -m "chore: sync memory pre-migration"
# git push origin main
```

**0b. Verify secret files exist and are non-empty:**
```bash
ls -la ~/.openclaw/secrets.json          # OpenClaw secrets (API keys, Discord tokens)
ls -la ~/.config/craig/github-app-key.pem # GitHub App private key
ls -la ~/.x402scan-mcp/wallet.json        # x402 wallet ($252 USDC — DO NOT LOSE)
```

**0c. Capture EC2-only state files:**
```bash
ls -la ~/.craig-notified-emails   # Email dedup (prevents re-notifying all old emails)
ls -la ~/.craig-last-proactive    # Proactive task timestamp
```
If `~/.craig-notified-emails` doesn't exist, that's fine — Craig will just
skip the first email check cycle. If it does exist, transfer it to avoid a
flood of duplicate email notifications on new Craig's first heartbeat.

**0d. Check for uncommitted work in other repos:**
```bash
# Craig may have cloned repos and left uncommitted work
for dir in ~/Code/merit-systems/*/; do
  echo "=== $dir ==="
  cd "$dir" && git status --short 2>/dev/null
done
```
If any repo has uncommitted changes, either commit+push them or flag to Sam.
These repos will NOT be transferred — Craig re-clones as needed.

**0e. Check for unexpected state files:**
```bash
# Look for anything in home directory that might be important
ls -la ~/.*craig* ~/.*openclaw* ~/.*x402* ~/.*agents* 2>/dev/null
ls -la ~/.config/craig/
```
Review output. If anything unexpected shows up that's not in the transfer
table above, add it to the plan before proceeding.

**0f. Snapshot current wallet balance:**
```bash
# Record exact USDC balance for post-migration verification
mcporter call x402 get_wallet_info 2>/dev/null || echo "mcporter not available, check wallet.json directly"
```

---

### Phase 1: Provision Hetzner

```bash
# 1a. Create SSH key
ssh-keygen -t ed25519 -f ~/.ssh/hetzner-craig -C "craig@hetzner"
hcloud ssh-key create --name craig --public-key-from-file ~/.ssh/hetzner-craig.pub

# 1b. Create server
hcloud server create \
  --name craig \
  --type ccx43 \
  --image ubuntu-24.04 \
  --ssh-key craig \
  --location ash

# 1c. Note the IP address from output
# hcloud server ip craig
```

### Phase 2: Bootstrap the server

```bash
HETZNER_IP=$(hcloud server ip craig)

# 2a. Create ubuntu user (setup.sh assumes /home/ubuntu paths)
ssh -i ~/.ssh/hetzner-craig root@$HETZNER_IP bash <<'USERSETUP'
adduser --disabled-password --gecos "" ubuntu
usermod -aG sudo ubuntu
echo "ubuntu ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ubuntu
mkdir -p /home/ubuntu/.ssh
cp /root/.ssh/authorized_keys /home/ubuntu/.ssh/
chown -R ubuntu:ubuntu /home/ubuntu/.ssh
chmod 700 /home/ubuntu/.ssh
chmod 600 /home/ubuntu/.ssh/authorized_keys
USERSETUP

# 2b. Run setup.sh as ubuntu
ssh -i ~/.ssh/hetzner-craig ubuntu@$HETZNER_IP 'bash -s' \
  < deploy/setup.sh
```

### Phase 3: Transfer secrets and state (via local machine)

```bash
EC2_IP=[REDACTED_PHONE]
HETZNER_IP=$(hcloud server ip craig)

# 3a. Pull secrets + state off EC2 to local temp
scp -i ~/.ssh/craigclaw-key.pem ubuntu@$EC2_IP:~/.openclaw/secrets.json /tmp/craig-migrate-secrets.json
scp -i ~/.ssh/craigclaw-key.pem ubuntu@$EC2_IP:~/.config/craig/github-app-key.pem /tmp/craig-migrate-app-key.pem
scp -i ~/.ssh/craigclaw-key.pem ubuntu@$EC2_IP:~/.x402scan-mcp/wallet.json /tmp/craig-migrate-wallet.json
scp -i ~/.ssh/craigclaw-key.pem ubuntu@$EC2_IP:~/.craig-notified-emails /tmp/craig-migrate-notified-emails 2>/dev/null || echo "No email dedup file (OK)"

# 3b. Create target directories on Hetzner
ssh -i ~/.ssh/hetzner-craig ubuntu@$HETZNER_IP \
  'mkdir -p ~/.openclaw ~/.config/craig ~/.x402scan-mcp'

# 3c. Push secrets + state to Hetzner
scp -i ~/.ssh/hetzner-craig /tmp/craig-migrate-secrets.json ubuntu@$HETZNER_IP:~/.openclaw/secrets.json
scp -i ~/.ssh/hetzner-craig /tmp/craig-migrate-app-key.pem ubuntu@$HETZNER_IP:~/.config/craig/github-app-key.pem
scp -i ~/.ssh/hetzner-craig /tmp/craig-migrate-wallet.json ubuntu@$HETZNER_IP:~/.x402scan-mcp/wallet.json
[ -f /tmp/craig-migrate-notified-emails ] && \
  scp -i ~/.ssh/hetzner-craig /tmp/craig-migrate-notified-emails ubuntu@$HETZNER_IP:~/.craig-notified-emails

# 3d. Clean up local copies immediately
rm -f /tmp/craig-migrate-secrets.json /tmp/craig-migrate-app-key.pem \
  /tmp/craig-migrate-wallet.json /tmp/craig-migrate-notified-emails
```

### Phase 4: Install & start OpenClaw on Hetzner

```bash
ssh -i ~/.ssh/hetzner-craig ubuntu@$HETZNER_IP
```

Then on the box:

```bash
# 4a. Install PyJWT (needed for GitHub token generation)
pip3 install pyjwt[crypto]

# 4b. Set API key and onboard OpenClaw
export ANTHROPIC_API_KEY='<key>'
cd ~/Code/openclaw/openclaw

# Checkout pinned version
PIN=$(cat ~/Code/merit-systems/CraigClaw/deploy/openclaw-pin | tr -d '[:space:]')
git fetch origin
git checkout $PIN
pnpm install --frozen-lockfile
pnpm build
sudo npm install -g .

# Onboard
openclaw onboard --non-interactive --accept-risk \
  --mode local \
  --auth-choice apiKey \
  --anthropic-api-key "$ANTHROPIC_API_KEY" \
  --gateway-port 18789 \
  --gateway-bind loopback \
  --install-daemon \
  --daemon-runtime node

# 4c. Render config
jq -s '.[0] * .[1]' \
  ~/Code/merit-systems/CraigClaw/deploy/openclaw.json \
  ~/.openclaw/secrets.json \
  > ~/.openclaw/openclaw.json

# 4d. Install credential scripts + systemd override
cp ~/Code/merit-systems/CraigClaw/deploy/craig/*.sh ~/.config/craig/
chmod +x ~/.config/craig/*.sh
mkdir -p ~/.config/systemd/user/openclaw-gateway.service.d
cp ~/Code/merit-systems/CraigClaw/deploy/craig/github.conf \
  ~/.config/systemd/user/openclaw-gateway.service.d/
systemctl --user daemon-reload

# 4e. Install skill packs
grep -v '^#' ~/Code/merit-systems/CraigClaw/deploy/skills.txt | while read skill; do
  [ -n "$skill" ] && openclaw skills add "$skill"
done

# 4f. Apply patches (if patch scripts exist in deploy/)
# ls deploy/patch-*.sh && for p in deploy/patch-*.sh; do bash "$p"; done

# 4g. Enable linger (start on boot without SSH)
loginctl enable-linger ubuntu

# 4h. Start Craig
systemctl --user restart openclaw-gateway

# 4i. Verify
systemctl --user status openclaw-gateway
journalctl --user -u openclaw-gateway -f
```

### Phase 5: Verify new Craig on Discord

- Poke Craig in Discord, confirm he responds
- **Both Craigs will be responding at this point** — that's expected
- Test a build task (clone a repo, npm install, etc) to verify the bigger
  machine is working
- Check `free -h` to confirm 64 GB available

### Phase 6: Silence old Craig

```bash
# Stop old Craig's service (keeps EC2 alive for rollback)
ssh -i ~/.ssh/craigclaw-key.pem ubuntu@[REDACTED_PHONE] \
  'systemctl --user stop openclaw-gateway'
```

- Verify only new Craig responds in Discord
- Leave old EC2 running (stopped service) for 24 hours as safety net

### Phase 7: Update GitHub Actions

Update these secrets in `Merit-Systems/CraigClaw` repo settings:
- `EC2_HOST` → new Hetzner IP (from `hcloud server ip craig`)
- `EC2_SSH_KEY` → contents of `~/.ssh/hetzner-craig` (private key)

Push a trivial change to `main` and verify the deploy action succeeds.

### Phase 8: Update documentation

- Update `.claude/infrastructure.md` with new Hetzner details
- Update SSH command references
- Remove AWS-specific resize procedures
- Add `hcloud` commands for resizing:
  ```bash
  # Resize (requires server stop)
  hcloud server shutdown craig
  hcloud server change-type craig --server-type ccx53  # upgrade to 128 GB
  hcloud server poweron craig
  ```

### Phase 9: Tear down AWS (after 24h+ on Hetzner)

```bash
# Terminate EC2 instance (EBS volume auto-deletes if set to delete-on-terminate)
aws ec2 terminate-instances --instance-ids i-0e2fb16c6e0fb83d0 --region us-east-1

# Release Elastic IP
aws ec2 release-address --allocation-id eipalloc-0155d2c49a8667e13 --region us-east-1

# Verify nothing is still billing
# Check AWS Cost Explorer in a few days
```

## Rollback

If new Craig has issues:
```bash
# Restart old Craig
ssh -i ~/.ssh/craigclaw-key.pem ubuntu@[REDACTED_PHONE] \
  'systemctl --user start openclaw-gateway'

# Revert GitHub Actions secrets to old values
```

Old EC2 is untouched until Phase 9. Full rollback takes 30 seconds.

## Follow-up: Move state files into the repo

After migration, move EC2-only state files into the CraigClaw repo so they're
version-controlled and survive future migrations automatically:

| File | Current location | Proposed repo location | Why |
|------|-----------------|----------------------|-----|
| `~/.craig-notified-emails` | Home dir on EC2 | `state/notified-emails` | Prevents duplicate email notifications after migration |
| `~/.craig-last-proactive` | Home dir on EC2 | `state/last-proactive` | Proactive task gate timestamp |

Craig's heartbeat scripts (`HEARTBEAT.md`) would need to reference the new
paths. The memory sync heartbeat already commits and pushes — state files
could ride the same commit cycle.

This is a small follow-up PR, not a migration blocker.

## Cost comparison

| | Current (AWS) | New (Hetzner) |
|---|---|---|
| Compute | t3.large $61/mo | CCX43 $108/mo |
| Storage | 100 GB EBS $8/mo | 360 GB included |
| Total | **$69/mo** | **$108/mo** |
| Specs | 2 vCPU (burst), 8 GB | 16 vCPU (dedicated), 64 GB |
| $/GB RAM | $8.63 | $1.69 |

We pay $39/mo more for 8x CPU and 8x RAM with no burst throttling. An
equivalent AWS machine (m7i.4xlarge, 16 vCPU, 64 GB) would cost $588/mo.