I Said I Wasn't Rebuilding My AI Agent. A Month Later I Did Anyway.

My personal AI agent ran on a macOS VM for a year. Then its memory system silently broke, and I ended up rebuilding the whole thing on Cloudflare Workers. Here's what actually works, tested, and what doesn't yet.

Agu is my personal AI agent. He runs on Telegram and a small website, and he actually does things for me — registers my visitor parking, keeps track of things I tell him, browses the web, sends emails, takes screenshots. He’s been running for over a year, most recently on a small virtual computer I kept always-on at home.

A month ago I wrote about how Cloudflare had open-sourced something architecturally identical to what I’d built for Agu at home. I ended that post by saying I wasn’t going to rebuild him. What’s running works, I said. I’m not breaking something that registers my visitor parking reliably just to move it to a cleaner setup.

I rebuilt it anyway. Here’s what happened, what’s actually live and tested, and what still isn’t.

Why gbrain existed in the first place

Before gbrain, Agu’s memory was a MEMORY.md file and a folder of daily log files, read back with roughly a two-day lookback. That’s not memory, that’s a diary nobody re-reads. Ask about something from three weeks ago and the honest answer was “I don’t know” — not because the information wasn’t valuable, but because nothing was actually searching it.

gbrain — a fork of the open-source garrytan/gbrain — was the fix. It’s a small vector database: every fact gets converted into an embedding (a numeric representation of its meaning) and stored, so a new question can be matched against everything ever recorded by meaning, not just exact keywords. The fork’s one real change was swapping the embedding model from OpenAI’s paid API to Cloudflare Workers AI’s own embedding model, which meant free embeddings and zero extra API keys to manage. It ran as a local database file on the VM, and for months it worked exactly as intended — Agu could recall a decision from four months earlier as easily as one from yesterday.

What broke first

One day it stopped starting. Twenty pages of memory — a year of context about my projects, my family, decisions we’d made — locked behind a database engine that crashed on every single attempt to open it. The one public report of this exact bug blamed a specific version of macOS. I didn’t buy it, so I tested that theory myself rather than take the internet’s word for it: I tried opening the same database on a completely different kind of computer, running a completely different operating system. It crashed there too, identically. Whatever was actually broken, it had nothing to do with macOS.

That’s what actually triggered the rebuild — not a desire for a cleaner architecture, but a real memory system that was down, with no fix in sight and a public diagnosis I’d just proven wrong.

Technical note. The database was a fork of a project called PGLite — Postgres, compiled to WebAssembly, embedded directly in the app rather than run as a separate server. It was aborting during its WASM runtime initialization on every version I tried, on both Apple Silicon (macOS) and x86_64 (Linux). Since the crash was environment-independent, the fix wasn’t going to be “use a different OS.”

The actual recovery was simpler than fixing the database: a PGLite data directory is just an ordinary Postgres data directory underneath. I didn’t need the database engine to boot at all — I needed to read its raw table files, which are plain structured files on disk.

find base/5 -type f -size +50k | while read f; do
  strings -n 50 "$f" | grep -i "keyword"
done

strings — a decades-old Unix tool that just pulls readable text out of any binary file — got the actual row content straight out, no database engine required. All twenty pages, recovered and re-embedded into the new system, in about fifteen minutes once I stopped trying to fix the database and started reading its files directly.

The new architecture

Agu now runs as a Cloudflare Worker instead of a VM. No always-on Mac, no LaunchAgents, no VNC password to remember.

 Telegram  ─┐
             ├──▶  Agu's brain (one per conversation)
 Web UI    ─┘        │  decides which model handles this message
                      │
        ┌─────────────┼──────────────────┐
        ▼             ▼                  ▼
    Memory ◀───── AI models         Real browser
  (Vectorize)    (Claude / GPT /   (a small Linux box
       ▲          Gemini / Kimi)    at home, real logins)
       │
   Claude Code ── reads and writes the
   (on my Mac)    exact same memory, no
                  conversation needed

Agu’s “brain” — one small, isolated instance per conversation, so my chat history never bleeds into anyone else’s — decides per message which model handles it: a free, fast model for simple questions, a smarter one when the request needs memory, a browser, or an outside action. Memory lives in Cloudflare’s vector database now, not a local file. Anything that needs an actual browser — logging into a site, filling a form, taking a screenshot — goes to a small Linux box on my home server running a persistent copy of Chrome. Same box, same login sessions, whether the request came in five minutes ago or five days ago.

That Linux box wasn’t the first thing I tried. Cloudflare has its own “computer use” capability — a package that lets a Worker actually execute shell commands — and I reached for it first, since it would have meant one platform for everything. It needs a compatibility flag that Cloudflare currently blocks outside local development, so it works fine on your own machine and not at all once you actually deploy. That ruled it out for anything that has to run for real. The Linux box at home is the answer to the same problem — a browser with real, persistent logins — solved outside Cloudflare’s sandbox instead of inside it.

Memory is the one piece two completely separate AI systems both touch directly. Claude Code — the coding assistant I used to build all of this — reads and writes the same memory store Agu does, over a plain authenticated API, with no conversation in between. Something I tell Claude Code while writing code shows up when I ask Agu about it on Telegram an hour later, and vice versa.

Technical note. The “brain” is a Cloudflare Durable Object — a small, stateful compute instance that Cloudflare automatically creates per conversation and keeps close to wherever the conversation is happening. Memory is Cloudflare Vectorize, a vector database with the same free tier and per-account isolation as everything else running on Cloudflare Workers.

What’s actually tested, not just built

I’m being specific here on purpose, because “I built X” and “I verified X works” are different claims, and it’s easy to blur them when you’ve spent a long day shipping.

Confirmed working, with a real test behind each one:

  • Shared memory. I wrote a fact to memory from my own machine, and Agu recalled it correctly in a completely separate conversation seconds later. That’s the actual sync mechanism working, not a description of one.
  • Visitor parking. Real registration against the actual condo portal — logged in headlessly, filled the form, submitted, downloaded the permit PDF, delivered it to Telegram. Not a mock.
  • Vision. Sent a real photo through; the model correctly identified it. Runs on a Workers AI model that bills to a separate, cheaper bucket than the main AI Gateway credits — worth checking which bucket a model bills to before assuming “AI” means one price.
  • Voice. Synthesized a real spoken sentence, sent it through Whisper, got back a perfect transcript with word-level timestamps.
  • Email. Two real emails, delivered.
  • Screenshots on demand. Asked Agu to screenshot a live site; it arrived on Telegram in seconds.
  • Live takeover for logins and captchas. This is the one I’m most pleased with. When Agu hits a login wall or a captcha it can’t solve, instead of failing silently it opens a live, streamed view of the actual blocked browser session — I can watch it, click into it, type into it, from my phone — and hand control back once I’m through. I tested it against a real captcha, not a staged one: browsing Google search directly tripped Google’s own bot detection, and the handoff worked exactly as designed.
  • The links are single-use. The first click on a handoff link consumes it. A second click on the same link — even from a device that’s still logged in — gets refused outright. That mattered more than it sounds: those links get delivered over Telegram, which means they sit in chat history indefinitely.

Built, not yet verified end-to-end: Agu can now receive photos and voice notes over Telegram in principle — the underlying model calls are tested — but Telegram itself is still routed to the old VM while I run both systems in parallel, so I haven’t watched a real inbound Telegram photo go through the new path yet. I’d rather say that plainly than round it up to “done.”

A bug worth knowing about if you’re building something similar

The cheapest model in the routing chain — the one with no tool access — was, for a while, telling me it had “checked memory and found nothing” when a question stumped it. It hadn’t checked anything. It had no ability to.

The cause: its system prompt still listed tools it didn’t actually have wired up. A model that’s told it has a capability, and then can’t use it, doesn’t say “I don’t know” — it invents having tried. Once I gave that specific model an honest prompt (you have no tools, say so if you don’t know), the fabrication stopped immediately.

The lesson generalizes past this one bug: never describe a capability to a model that isn’t actually attached to that call. It will act as if the description is true.

What this doesn’t do yet

No inbox reading. No outbound phone calls. No video generation. All requested, none built. I’d rather list them here than let the rest of this post imply otherwise.

Telegram still points at the old VM. Both systems are running side by side right now, which is not a permanent state — it’s what “test before you cut over” actually looks like in practice.

What I’d tell someone starting this today

If your agent’s memory is a single file on a single machine, you have a single point of failure you haven’t met yet. Mine introduced itself with a WASM crash and an incorrect public diagnosis. Yours will pick its own moment.

The fix wasn’t “use a database with better uptime.” It was: don’t trust a database’s own tooling to be the only way to get your data back out of it. Everything durable eventually needs to be recoverable without its normal interface.

Built with: Cloudflare Workers, Durable Objects, Vectorize, AI Gateway, Workers AI, Proxmox.