Clients need good documentation. I built a pipeline that was too powerful.

When I work with direct-to-consumer companies on their Shopify storefronts, my clients themselves are frequently content admins; but they can’t always tell how a given feature was built, and the documentation handed over at launch starts going stale the week after. I envisioned a system that would answer a user’s natural-language questions on demand, generated from reading three separate sources of information: our curated documentation, the platform’s documentation, and the actual codebase. I envisioned the user getting a ready-made tutorial in response, complete with images showing the step-by-step walkthrough. I created a team of three AI agents to scope and build a proof-of-concept. Working with the agents, we made infrastructure calls based on the server capacity we had available.

I reviewed all AI output and in fact I found defects with what the AI team put together. What follows is a more detailed accounting of using AI to solve problems that these defects introduced. Nonetheless, I ultimately concluded that the whole approach was more machinery than the problem deserved. I ended development and met the need instead with plain markdown-based help pages hosted on GitHub Pages, and written and kept current by AI whenever a new feature ships.

At a glance

Role	Systems design, agent direction, infrastructure decisions, and the call to stop
Context	Internal proof of concept at my own studio: an AI-powered documentation portal giving client marketing and content editors on-demand guidance
Outcome	Development ended. The need was met instead with static help pages, written and updated by AI on each feature release and served as ordinary search results.

The situation

Every Shopify build ends with a documentation handoff, in Word documents, PDFs and walkthrough videos, and it starts going stale the week after launch. Meanwhile the client has a live store, a theme customized for them specifically, and a question. The goal was to answer those questions in detail, on demand, without slowing down our executional team.

What makes them hard is that the correct way to make a given edit might be set by Shopify, by the base theme framework the store was built on, or by a custom implementation our own team built. Take this example I tried to test with: why does the size chart not show up on our bundle products? Answering it well requires Shopify’s own documentation (how it stores extra information about a product), our handoff docs (what we named things and why), and the client’s actual store code (the rule that excludes bundles). A tool holding one of those three gives a confident, wrong answer.

It stayed that way because the cost was easy to ignore: a few minutes here and there of a senior person re-explaining something that was already in a document. So I went looking for something the client could ask directly, instead of asking us.

The first candidate was NotebookLM, Google’s early experimental tool for holding a set of documents in context at a time when little else did. It was built intended to be used individually, so sharing it, particularly outside your own company, was very difficult. We looked at commercial platforms that might be NotebookLM-but-shareable, and somewhere in that search the problem changed shape. The more research I did, the clearer it became that the AI part was not fundamentally difficult to solve. The hard part was the interface and the roles, where the development team supplies the updates and the content authors get the answers, and doing all of it on a minimal budget.

Two versions of the answer were on the table, and the gap between them is where the project lived.

• A static docs site, written and updated by AI whenever a feature ships. This is the easy part, and it exists today. The pages are still fixed content, served as ordinary search results via GitHub Pages.
• Generating the answer at the moment of the question: a step-by-step with screenshots, or a video walkthrough, assembled on demand from the three sources.

Even the second version is not hard technically. What killed it was economic. The generation end of the pipeline, and the image generation in particular, needs frontier models, and frontier models are expensive per answer. The whole point was to cost less than a commercial tool like Cassidy, which meant running the entire thing for under $20 a month.

The calls that mattered

One app for every client, not a build per client. Each client got their own notebook inside a single deployment, so a fix or a new feature reached all of them at once. Building one per client would have been fine for the first and unworkable across a whole account list. The trade-off is that clients then have to be kept apart inside one shared system, instead of by running separate systems, and that separation had to hold in every part of it.

Reading the client’s own store code, not only documentation. Without it the system could describe how Shopify works generically but could not explain what this particular store does, which was the entire question. The cost is that the system’s knowledge of a store has to be refreshed every time that store’s code changes.

Renting a database instead of stretching the server I already pay for. The budget rule said run it on my existing server. That server’s database was too old to hold the AI search index, the embeddings that let the system find the right passage, so the first plan kept the app’s data at home and sent the index to a second database somewhere else. Two databases on two machines, with the app responsible for keeping them in step. Before building that, I compared four services (Pinecone, Qdrant, Supabase and Neon), and the boring option won: one modern Postgres database on Neon that holds the data and the index together. It costs a modest monthly fee, and a whole class of “the two stores disagree” bugs can no longer happen.

Agents build, I verify. I created three agents for this project, each responsible for one part of the system. The first built the plumbing: the database, the pipeline that pulls in the documents and the store code, and the deployment. The second handled retrieval, finding the right passages across the three sources and assembling them into an answer. The third tested what the other two produced. Giving each agent a whole area rather than a list of tasks meant it always had the full context for its part, and its work was easy to review.

What went wrong, and what I caught

The biggest reason the app wasn’t useful was that embeddings are expensive. But there were technical reasons for that cost.

I had planned to keep costs near zero by hosting the embeddings on the web server I already pay for, with an OpenAI key for the model calls. The server turned out to be too old to run a modern Postgres, which is what the index needs, so the embeddings had to go to a rented database instead. That is where the monthly bill started. Then the first deployment ran, and immediately reported that it couldn’t reach that database, with an empty error message. The empty message was the clue. The connection was failing before any password was checked, so the credentials were fine and the problem sat somewhere between the two machines. I couldn’t test the server directly from where I was, so I had the agent build a diagnostic before a fix: report the exact error code, then act on it. The shared host was blocking the standard database port, and switching to a driver that travels over ordinary HTTPS got through.

The second problem is the one I’d point to if I could only tell you about one. Reading back the documentation the agents had written as they worked, I noticed that a rule saved into the database early on named one AI model, while the decision we locked in three pieces of work later named a different one. The two produce data of the same shape, so the database looked fine and every test passed. On the first real run it would have rejected every record the system tried to save.

Nothing in the testing was wrong. The tests just never touched the disagreement, because each agent had done exactly what its own ticket asked, and each ticket was correct on its own. That kind of defect only shows up when one person reads the whole system instead of one ticket at a time. The agents build; the verification is mine.

Where it landed

Shipped and verified

• A managed database provisioned and set up for this kind of index, with three rounds of structural changes applied and checked
• The application live on its own web address, serving from a private code repository
• The deploy loop proven end to end by observation rather than configuration: push a change, pull it on the server, restart, reload the page, see the new output
• What the build taught written back to a five-page internal wiki covering infrastructure, schema, retrieval gotchas and open issues

Never proven

• Whether the answers would have been good across all three kinds of source
• Whether they would have deflected client questions
• Any adoption or time-saved figure

Those three are not open questions any more. Ending the project means they stay unanswered, and that is the honest cost of the decision.

Why I ended it

The system worked the way it was designed to, and it was still the wrong thing to build.

Keeping this kind of index current costs real computing time, and that cost is permanent. It does not amortize the way a build does. More to the point, the thing was becoming a product in its own right. Its remit was a simple way for a client to get instructions. It had grown a pipeline for taking material in, a layer for finding the right parts of it, a library of images, and a way of keeping clients apart inside one system, with the maintenance burden that implies.

And the capability was commoditizing underneath us. Holding a set of documents in context and answering from them was the novel part when we started; by the time I stopped, it was turning into something you buy. That makes it much harder to justify more time or money.

So I ended development. The client need was met with static help documentation on a simple published site. AI still writes and updates those pages whenever a new feature ships, and the authoring was always the part that saved the team time. What changed is the delivery: prewritten pages served as ordinary search results, rather than answers generated on demand.

What I would do differently

I optimized for the wrong cost. The budget rule saved me a modest monthly hosting fee. In exchange it cost the whole exercise of working out where to keep the data, a two-database design that got thrown away, and the connection failure that came after it. Paying for a managed database from day one would have been cheaper by any count. The rule was a real decision at the time, so I’ve left it in the write-up, but I’d weigh it differently now.

I should have tested the shape of the solution before building the infrastructure. The question that ended this project, whether generating answers on demand is worth its maintenance cost for this particular need, could have been asked in week one with a hand-built mock-up and no infrastructure at all. Instead I spent the first third of the project on hosting and storage, because that is the part that feels like progress.

Decisions should be written down before agents start, not discovered afterward. The model mismatch happened because the choice appeared in one ticket, was revisited days later in another, and nothing reconciled them. A one-page decision record that every schema change had to cite would have made the conflict impossible rather than merely findable.

---

Under the hood

Everything the piece has to say is above. What follows is the technical detail, for readers who want it.

Constraints

Five constraints did most of the design work.

1. Minimal incremental budget. The app had to run on infrastructure I already pay for: a shared cPanel Linux box with a jailed shell, Node available only through cPanel’s app manager, with PostgreSQL 10.23 that the host confirmed cannot be upgraded; and a Claude Max subscription.
2. One app, many clients. A notebook per client, but a single deployment, so a feature or fix reaches every client at once.
3. Shareable to non-technical clients. Public links, no account creation, no Google Workspace dependency.
4. Answers, not results. Synthesis across sources was the product. A ranked list of passages would have been a worse version of something the client already had.
5. I do not write production code. My contribution is specification, decision and verification. Anything that required me to sit in a file for three days was, for this project, not a viable design.

Constraint 1 turned out to be the expensive one. Postgres 10 sits below pgvector’s minimum supported version, so the single most obvious architecture, documents and embeddings in one local database, was unavailable before the first line was written.

The stack

A single multi-tenant Node application that ingests three kinds of source per client, embeds them into one vector-capable database, and synthesizes cited answers with matching visuals.

Node 20 on cPanel/Passenger; Neon Postgres 18 + pgvector 0.8.1; OpenAI text-embedding-3-small (1536-dim); Claude for synthesis and vision captions; GitHub and Shopify Help Center ingestion. The replacement documentation is static markdown on GitHub Pages. Scale of work: 45 tickets across 9 epics, with 3 purpose-built AI agents owning the build.

The decisions in full

Decision	What I chose	Why
Where embeddings live	Neon, a managed Postgres 18 database	My server could only serve Postgres 10, which was acceptable for the app data but unusable for embeddings. The first plan split them across two databases on different servers. I evaluated Pinecone and Qdrant against Supabase and Neon, and consolidating won: one datastore, one query language, no cross-store joins to keep consistent.
Embedding model	OpenAI text-embedding-3-small, 1536 dimensions	Matches the VECTOR(1536) schema, and at these ingest volumes the quality gap to the large model did not justify the cost. The API key is scoped to embeddings only.
Synthesis model	Claude, also generating vision captions for indexed images	Long-context synthesis across three heterogeneous sources was the actual product surface, and at the time Opus was the only model offering 1M context.
Database driver	Neon’s serverless driver over HTTPS/443, rather than the standard pg driver over TCP 5432	Forced by the host, not chosen up front. See The two catches in full, below.
Code as a source	Ingest the client’s theme repo from GitHub	Without it the system can describe how Shopify works generically, but cannot explain what this theme does specifically.
Visuals	A library of pre-captured images of the Shopify admin, drawn on and annotated as each answer needs	Where is the button is the most common question shape. An answer without a picture of the interface is a partial answer.
Who builds it	Three scoped AI agents: a Docs App Engineer (plumbing, ingestion, crawlers, routes), a RAG/Retrieval Engineer (retrieval, synthesis, visual selection), and a QA agent (browser capture, app QA)	Scoping by epic rather than by task kept each agent’s context coherent and made review tractable. The existing agent roster was Shopify-delivery-shaped and had no general build role; I created the two that were missing.

Open question on the Visuals row: why a pre-captured library rather than capture-on-demand? If the reason was latency, that belongs in the table. Sam to fill.

How it works

Three ingestion paths converge on one embedding store; one question path reads from it and assembles an answer with a picture attached.

Reading it: the design called for everything on the left to be re-indexed on change, so a theme commit would update what the system knew about that client’s code. The ingestion pipeline was never finished, so that stayed a design rather than a behaviour. Everything from the question rightward was meant to happen live.

Tenancy ran the length of it by design, with chunks, images and code fragments scoped per client so that one deployment could serve many notebooks. The schema was built for that. Retrieval never ran in production, so the boundary was never tested under real conditions.

Getting it running

The system never reached client hands, so this is about the delivery loop rather than adoption. Getting a foundation to provably live took more infrastructure work than application work. Two representative obstacles:

• The subdomain. The plan was to serve the app at a path under my studio’s existing domain. The path collided with the WordPress and openresty proxy already sitting there, which 301-redirected it away. Resolved by giving the app a dedicated subdomain instead of fighting the proxy, a five-minute fix once correctly diagnosed.
• The deploy loop. The box pulls from a private repo with a read-only deploy key, and commits come from the development side. I did not consider it done when it was configured. I considered it done when a push visibly changed the live application once: push, git pull, touch the restart file, re-hit the endpoint, see the new output.

The two catches in full

The connection failure. The app deployed cleanly but its health check reported the database unreachable, with an empty error detail, which meant the connection was failing before authentication, so credentials and environment variables were not the cause. The leading hypothesis was that the shared host firewalls outbound TCP 5432. I could not run a box-side socket test from where I was, so I specified a diagnostic first: surface the driver’s error code in the health endpoint, then, if confirmed, move the data layer to Neon’s serverless driver, which travels over 443 and is not blocked on shared hosts. That became a standing rule for this host.

The embedding-model constraint. Reviewing the generated documentation, I found that the schema’s embedding_model check constraint pinned text-embedding-3-large, written into an early migration ticket, while the locked decision three tickets later was text-embedding-3-small. Both are 1536 dimensions, so the column type was fine and nothing failed in testing. It would have rejected every row on the first production ingest.

The proof of concept did its job. It proved the opposite of what I expected.

---