QA, automated. In record time and with incredible accuracy. Autonomously.

The takeaway

A client finds something wrong with a site we are building, draws a box around it in their browser, and types a sentence. What happens next used to be my job: read it, work out what it actually means, decide who should do it, chase them, check the result, and write back to the client to close the request. I replaced the time-intensive UAT triage process with a set of agents that pick each ticket up, work it, check each other, and hand back a single merged preview once a day. My job became reviewing that preview and making the calls the agents escalated. I was a servant-leader helping to clear blockers and keep momentum just like any other project I’ve served on as a project manager; it’s just here, the team was AI agents.

This UAT response team has now run two accounts and counting end to end. On a direct-to-consumer pet brand it absorbed a launch crunch: 134 tickets in twelve working days. On a health supplements brand it ran a month of sustained testing, resulting in 114 tickets. Across both, intake, triage, the fix, the verification and the drafted reply happened without me dispatching any of it.

The two accounts were both ostensibly doing the same work, using identical skills and prompts, but the accounts and testing workflows were very different. One project fanned out a team of eight different testers to file bugs, while the other was a single contact who logged every bug, and indeed many things that were not bugs but rather questions. The loop absorbed both, and the gap between them taught me more about what I had actually built than the first account did on its own.

At a glance

Role	I designed the loop, wrote the agent briefs and the standing rules, reviewed and accepted the output, and made the calls the agents escalated. The agents did the initial triage, the executional fix work, and the first draft of most client replies.
Context	Contract delivery work through a digital agency, on Shopify Plus migration projects.
Outcome	134 tickets closed in twelve active days on one account and 114 on the second; my review surface reduced to one merged preview per day and simply reviewing the client asks and answer, already ready for review.

User acceptance testing is where web projects go to die slowly. The client is finally looking at the real thing, so everything they notice arrives at once, in their words rather than our terminology, and through whatever channel is nearest to hand. On the first account that meant a bug-reporting tool wired into the staging site, plus a client-facing project thread, plus the agency testers filing against the same build.

Volume alone would be manageable, but each ticket needs a different kind of attention: some are simple theme settings in the admin UI, some are data problems, and some are not defects at all but rather new requests submitted as a bug. Sorting falls to whoever has the most context, which is one or two people, and they become the constraint.

134 tickets arrived over twelve active days from eight reporters, two-thirds from the client side and the rest from the agency testers and the development partner. One person reading, sorting, assigning and replying to all of that is a full-time job, and one that produces no code at that.

What I built to absorb that is less a tool than a small team of AI agents in an automated loop. In this workflow, a client files a ticket; an agent picks it up and works it in an isolated copy of the site. When it calls the ticket done, another agent checks the work against the request and sends it back if it doesn’t match. A project-manager agent watches the board and restarts anything that goes quiet. At the end of each day, everything that clears the team’s review is merged into one single preview for me. That preview, along with the text replies the agents have drafted, is what I use for validation before approving outbound notification to the client. The calls below are the ones that make this loop work.

The calls that mattered

Regressions are managed in the infrastructure, not in the instructions. Taking myself out of the review path meant the agents were now working unsupervised for long stretches, and a boundary that only exists as a sentence in a prompt is no boundary at all. Each agent works in its own git worktree against an unpublished staging environment, and none of them can write to the live site or to the top-level branches; everything converges on one second-level branch. What that means for the agents is that the wrong action is unavailable rather than forbidden, because an instruction not to touch production is only a request, while a credential that cannot reach production is a fact.

The team needs to review its own work, but the reviewer is a different agent from the builder. While I have a working knowledge of reading some basic coding languages, the reality is that using my time to review line-by-line diffs of the agents’ work is not a good use of that time. I wanted my UAT process to include a review step that catches most issues before they ever reach me. And that is why I had to have an agent team, not a single agent. The agent that writes a fix does not get to decide the fix is good; a separate agent checks the work against the ticket and bounces it back for rework on its own authority, so the first version of anything never reaches me. This is the part most people skip, and skipping it is why so many agent pipelines produce confident garbage; an agent that writes and grades its own work will always pass itself. Splitting those roles is what makes a once-a-day review from me sufficient rather than negligent.

A supervisor on a schedule, not more robust agents. Agents run as long as they need to in order to complete their tasks, but sometimes when left alone they pause work and stall out. Agents fail for boring reasons, such as session quotas, credential timeouts, or a session that simply stopped typing mid-thought, so the simple answer is periodic reconciliation rather than trying to make every agent individually resilient. A project-manager agent runs on a recurring schedule and reconciles ticket state against agent state, restarting any sessions that might have stalled in the process. It is a watchdog that also serves to keep my PM agent consistently involved, updating notes and tickets and assigning new work.

One merged preview a day instead of one preview per fix. All that reliability bought me a new problem: a day’s worth of fixes, each sitting in its own preview, is the same review load I was trying to get rid of. The team iterates a fix in their local dev environment, and when it comes time for me to review, all the work from the day is merged into a single preview branch, which is both faster and surfaces potential regressions stemming from the merge.

Agents draft client replies. They never send them. The code is in front of me once a day, but even if a ticket is fixed, if the client hasn’t heard about it, then it’s still an open ticket as far as they are concerned. Drafting is encouraged, because in a triage loop most of the value is in the reply. But sending is mine, and the same rule covers anything destructive: present the plan to me first.

What went wrong, and what I caught

The agents could check whether a fix matched a ticket, but there were still key limitations, and my judgement when working with them still mattered.

The automated reviews had three recurring issues: Spot-checking, but not checking every change and therefore overlooking something; Build errors downstream from agent reviews; and inability to judge decisions that depended on taste or prior client conversation.

The clearest example was a set of five design tickets comparing a half-populated theme with the design file. The agents were capable of completing the work, but I rejected all five because the comparison was happening too early. Content needed to be populated first, and one ticket would have undone a change the client had specifically requested the week before. The system had not failed to execute the tickets. It had no basis for knowing that the tickets should not be executed yet.

That became the useful division of labor. The agents caught fixes that did not match the request; I caught requests that were incomplete, premature or contradicted earlier decisions. My daily review was not another pass over their work. It was the layer that supplied the context they did not have.

The second account exposed a different bottleneck

The first account had eight people testing a launch build and mostly reporting things that looked wrong. The second had just a single client stakeholder using the same ticketing tool to ask questions while learning the content management system. 80 of its 114 tickets included a question. The loop was no longer just fixing defects; it was also carrying a large share of the client communication.

The agents handled their part: they investigated the question and drafted an answer. But every answer still needed my approval before it went to the client. Those drafts appeared alongside routine updates from the agents, with no clear distinction between “work in progress” and “reply ready for Sam.” The bottleneck wasn’t answering the questions but rather surfacing the finished answers when they needed me.

I had already built a workaround without recognizing it as one. I built a convention for titling tickets that helped me to quickly identify the issues. This naming convention helped me scan the stream of ticket updates more quickly and identify comments that included drafts for my review.

I didn’t prioritize fixing this. I intended to fix it because it felt like it was slowing the whole process down, but when I dug into the data, I found that tickets with questions closed at a median of 44 hours, compared with 49 hours for tickets with fixes. Even with my review as a bottleneck, issues were moving through the process quickly.

Where it landed

What is true. The loop closed 134 tickets in twelve active days on the first account and it has handled 114 on the second account over about a month. Across both, intake, triage, the fix, the verification and the first draft of the client reply happened without me dispatching any of it, and my review surface reduced to one merged preview a day.

What the numbers say about speed. On the first account, half of all tickets reached their final state within about two days of being filed, three-quarters within five days, and the slowest tenth took twelve days or more. That is good for user acceptance testing, but it is not the same claim as same-day turnaround. A daily review cadence and a twelve-day tail are both true, because that tail is mostly tickets that were waiting on a decision, our team’s or the client’s, rather than on an agent.

No data on my own queue. I know how many tickets closed and how quickly, but not how long a drafted reply sat waiting for me. That wait happens between an agent finishing and a human noticing, and nothing records it.

The existing platform stack integrated cleanly. The development teams were already comfortable using Marker.io as a UAT platform, and that platform offers both an MCP connection and a native sync to Jira. We used both; agents grabbed issue details directly from Marker and updated the corresponding work item in Jira. That way, we maintain a full audit trail for internal and external audiences, the same way that we would if humans were executing the work.

What I would do differently

Instrument the handoff from the start. The one number that would prove the reviewer is doing its job is how many fixes the QA agent bounced back before anything reached me, and nobody recorded it, so it cannot be reconstructed.

Decide what counts as a non-defect before acceptance testing opens. A large share of the second account tickets are not bugs but rather requests: add an option, increase a range, allow a new placement. They arrive through a bug-reporting tool, which means they arrive pre-labeled as defects, and the loop will cheerfully work them. That is scope entering a project through a side door, and my system made it faster rather than catching it.

Give the drafts review queue its own place. The orchestration layer is a conversation, and a conversation is the wrong shape for a work queue. Everything the agents produced went into the same stream in the same format, including the handful of items that needed a decision from me. A better naming convention is what I reached for first, and more notifications would be the next wrong answer. What I actually need is a short list of things waiting on a human, kept apart from the agent conversation and concise enough to keep it at Inbox Zero.

Under the hood

Everything that decides the story is above; the rest is mechanics.

The roster

On a given account the UAT review loop is comprised of a team that consists of a project manager, a theme engineer, a data and migration engineer, a production-to-staging parity auditor, a QA and verification agent, a tracking and privacy engineer, and a client communications drafter. Tickets that need a human are assigned to me, and I work with the agents asynchronously in issue comments; when an agent hits an access problem, it raises that with me directly.

The standing rules

These apply to every agent regardless of what its credentials would technically permit. Never write to a live production store. Never take a destructive action without presenting a full plan first. Never send a message to a client. The rules are backed up with fine-grained access control.

What the handoff contains

The QA agent attaches before-and-after screenshots of anything visible on the front end, so my job is to reproduce a working state rather than establish that one exists. The project-manager agent opens each handoff with the original request and a summary of what the agents said to each other while building it, which during acceptance testing is often the first time that day I have looked at the underlying question.

How the numbers were obtained

Ticket counts, date windows, reporter counts and durations were queried directly from the bug-reporting tool API while this was being written. Durations measure time from ticket creation to last modification, since a ticket that is listed as resolved can still be commented on or reopened.