Agent memory that’s safe to use across multiple clients.

The takeaway

Most AI agents don’t know what happened on an account last week, but answer anyway. I solved that problem for my own agents with a memory layer built from a simple idea and open-source software: a folder of plain Markdown notes that my agents and I both write into, and a store underneath it that turns those notes into knowledge an agent searches automatically at the start of every session. I wired those pieces into one system and designed the rules for what goes into it, how it stays current, and how agents use it.

At a glance
Role	Sole builder and operator. Decided the architecture, set the memory rules, directed the agents that built the sync and query wrappers, and found and fixed the failures.
Context	Personal system, used on my own projects and on client delivery work.
Outcome	Running daily across about a dozen project workspaces, with no cross-project memory mix-ups observed.

The situation

I run agents on real client accounts. Their work is only as good as what they know, and most of what they need to know isn’t written down in the code or the project files. It’s institutional knowledge: who decided what, which date is the real one, what was tried last month and didn’t work, what a client won’t accept.

I started from Andrej Karpathy’s “LLM Wiki” idea: a folder of Markdown files the AI refers to for context, which grows into a wiki that both the human and the agent contribute to. A big part of that method is “gardening,” regularly tending the wiki so the notes stay current and the links between them still work.

I knew gardening would fall off my priority list within a couple of weeks if I had to do it by hand, so I needed a way to keep the notes current without a person doing the upkeep. The rest of the system is built around that.

The calls that mattered

Procedure goes in skills; state goes in the memory vault. Early in my AI journey, I kept a personal prompt library. Each prompt carried all the context and step-by-step detail the AI needed for its task, so they were long, and slow to write and maintain. At one point I started having the AI write its own prompts. I’d give it a brain dump of what I wanted and what I knew, ask it to turn that into a coherent prompt and fill any gaps even if it had to make things up, then correct the judgment calls and inventions I disagreed with before pasting the result into a fresh chat to run.

Then models got much better at using context from outside the prompt, and Anthropic shipped the first version of Skills: a stored, named set of instructions that an agent finds and follows on its own. Skills replaced my prompt library, because the procedure now lived in one place I could edit once and adapt for each use. But a skill only tells an agent how to do something. It can’t tell it what’s true about a project today, and that’s what the memory vault is for.

PMB became the system’s operational memory. It stores knowledge as individual facts and lessons, so agents can retrieve relevant context without repeatedly searching entire documents. I originally designed the Markdown wiki to be the master copy, with PMB as a searchable layer I could rebuild from it. In practice, I rarely browsed the wiki, and the system evolved to ingest meeting transcripts, emails, and other sources alongside my notes. Markdown is now one of several ways I contribute knowledge, while PMB is where that knowledge is organized and used by the AI.

Each project gets its own memory, and agents ask for it by name. Every project has its own database store. An agent working on one project can’t pull another project’s data by accident, because retrieval is scoped before the question is asked. Agents are told to name the project they want, and never to go looking in another project’s memory when an answer comes back empty.

What failed while the system kept working

The Markdown sync was the first part to fail. I had set up a nightly job to read the notes folder and load anything new into PMB. For four weeks, PMB itself kept working: agents could read existing memories and write new lessons directly, but Markdown files were not reaching it. Because that was only one of several ways knowledge entered the system, PMB stayed useful enough to hide the failure.

I eventually noticed that the memories imported from Markdown weren’t growing. When I started troubleshooting, I found a path error. My notes sync through iCloud, and Finder shows the vault as a folder inside iCloud Drive, but that’s only how Finder displays it. On disk, Obsidian keeps its iCloud-synced files in a separate hidden folder of its own, so the location I’d pointed the sync job at was an alias that didn’t actually exist.

The sync job had a line that said, in effect, if this folder isn’t there, skip it and move on. That can make sense for an individual project folder: there may simply be nothing to import. But the wrong path was the top-level folder, so every project beneath it appeared to be missing and the job skipped all of them without a word. It finished, reported zero errors and exited clean. The sync could not tell “there was no new Markdown” apart from “the place I was told to look doesn’t exist,” while PMB’s other activity made both states look plausible.

Fixing the path was the easy part. Most of what I changed was about the silence:

• The job checks the vault’s top-level folder before anything else. If that path is wrong, everything after it is wrong too, so the job stops there.
• If a project folder is missing, the job logs it and the run ends as a failure, but only after it has worked through every other project. One bad path shouldn’t cost a whole night’s import.
• Each project reports how many files it loaded and how many memories it now holds, so there’s a number to glance at.
• If every project loads nothing, the job writes a warning. That can be legitimate if I didn’t change any notes that day, but it is also what a complete Markdown sync failure looks like, so the job flags it either way.

I found a related bug while reviewing this write-up. My agents do each task in a fresh, throwaway folder. If the memory tool wasn’t told which project the task belonged to, it guessed from that folder. The write succeeded, but each lesson landed in a brand new and otherwise empty database store linked to a folder that would be gone as soon as the agent finished its task. The real project stores still looked healthy, so I had no obvious reason to suspect anything. PMB wasn’t broken; it was putting the memories exactly where I had told it to.

I added a small router that maps each task to a project and pins that choice before the agent can read or write memory. If the router can’t place a task, it sends the memory to one shared catch-all store that I know to check. I also ran a second script to move the orphaned lessons into the right projects.

The two bugs had the same shape: everything ran, nothing complained, and some of the knowledge still went missing or landed in the wrong place. I found both by looking at the result instead of trusting the status message. Agents can maintain their own memory, but I still need to check that the right thing is stored in the right place so that another agent can retrieve it later.

Where it landed

Today, PMB is the operational memory for about a dozen project workspaces. Markdown notes, meeting transcripts, emails, and agent-written lessons all feed into it, where they are processed into individual facts tied to a specific project. New information can supersede an older fact without erasing its history, so the system keeps both the current answer and how it changed. Obsidian remains a durable, human-readable way for me to contribute knowledge, but it is no longer the master copy of everything the agents know as in Karpathy’s original concept.

Before an agent reads or writes memory, a router identifies which project the task belongs to. In Claude Code, hooks query that project’s memory at the start of a session and put the results in front of the agent automatically; other tools can reach the same memory through MCP. The practical result is continuity without crossover: agents can pick up project context and lessons from previous sessions without me restating them, while each client’s memory stays isolated. The system runs daily, and I haven’t observed an agent mixing up two projects. I still verify what it retrieves, because a successful run is not the same thing as a correct result.

What I’d do differently

I would build every layer to be replaceable. The best tool for a job depends on the job and on when you ask. My answer today isn’t the one I’d have given a year ago, and it won’t be the one I give next year. That’s exciting to work in, and it’s also an ongoing tax: keeping up with the constant slew of new platforms, methods and ideologies takes hours every week, and hours spent reading about a better approach are hours not spent delivering anything.

I use “runtime hooks” to connect the system to Claude Code directly in its binary. It connects to other runtimes over MCP, but the binary hooks make it more reliable for enforcing previous lessons with Claude Code. Vendor lock-in is something that I’ve tried to avoid, and if I moved to a different runtime provider tomorrow, those automatic hooks wouldn’t come with me and accuracy would decrease. Solving for system lock-in at this level is one thing I would change.

Another thing I would do differently (and have incorporated in other more recent systems) is to check that it works, not just that it ran. Both failures in this case study were the same mistake: I took “nothing errored” to mean “it worked.” A handful of questions I already know the answers to, asked on a schedule, would have caught both.

Under the hood

What follows is technical detail for readers who want it.

My relationship to the code. I read JavaScript, SQL, Liquid and shell well enough to follow and question them logically, and I write a little HTML and CSS. I don’t write application code. The memory engine and the note-taking app are software other people wrote and maintain. The sync wrapper, the query wrapper, the workspace map and the isolation rules are mine, built in agent sessions I directed and reviewed.

The three pieces. The vault is Obsidian over plain Markdown, synced through iCloud. It’s split into a raw intake tree, an authored wiki organized as wiki/client-projects/[slug], a drafts folder for agents, weekly project status updates, and a journal I write by hand, which is excluded from every import so I can think out loud without the AI treating it as fact. The memory layer is PMB, an open-source, local-first memory tool for AI agents: Python, SQLite for the event store, LanceDB for the meaning-based search with keyword search alongside it, and an MCP server agents call to read and write. PMB isn’t tied to one AI tool. It speaks MCP and ships connectors for Codex, Cursor, Gemini CLI, Zed, VS Code and others. I run it in Claude Code, where hooks put whatever the search returns in front of the model before it starts work and record what it did afterward. That automatic part is specific to Claude Code; other tools get the same memory through the MCP tools, but only when they ask for it.

Why the sync wrapper exists at all. PMB’s import command only ever adds. I tested it against a scratch workspace: re-importing a folder of three files after changing one of them produced three new entries rather than one update, taking the total from three to six. There’s no deduplication by filename, hash or content. So the wrapper keeps its own record of each file’s content hash, stages only the files whose hash changed, and archives the old entry for a changed file before re-importing it. Archiving is reversible; nothing is hard-deleted. PMB also stores only a file’s name, not its path in the vault, so the wrapper folds the path into the staged filename to keep two notes with the same name from colliding.

Searching by meaning. One step of the import turns each note into a list of numbers, chosen so that two notes about the same subject come out close together when you compare them. That’s what’s meant to let an agent find the right note by asking a question in its own words instead of guessing the exact phrase I happened to type.

Provenance. Because the store keeps every version of a note as its own archived entry, an agent can see what a value is now, what it was before, and when it changed. That’s what provenance means here, and it’s why the wrapper never deletes an old entry, only archives it.

Lessons. When an agent solves a problem, a hook tells it to write the solution up as a lesson, linked to the ideas involved, so a later session can look it up.

Related: the Toolport piece covers the gateway most of my other tool connections run through.