Human-in-the-loop design — a stakeholder review on running code

I gave the stakeholder a working application to review in their browser. They returned 48 notes organized by screen.

Those notes made the next version possible, but they did not define it. One note described the visible problem accurately. Another referred to wording that was not where the reviewer thought it was. A third combined two requests that required separate decisions.

I compared each note with the running product, identified the underlying concern, and decided what belonged in the next version. Claude Code then implemented those decisions. I reviewed the result to confirm that the code preserved their meaning.

AI made another version cheaper to produce. That speed increased the need to agree on what the next version should be before producing it.

Running software made the feedback specific

A static mockup asks reviewers to imagine how the product will behave. A running application lets them experience that behavior directly.

The stakeholder could scroll the tables, use the controls, inspect the hierarchy, and read the final labels in context. Their feedback could therefore refer to the product they had used rather than behavior they had inferred.

That made the notes easier to locate. It did not make them self-explanatory.

A reviewer may notice a real problem but misidentify its cause. They may refer to the wrong label or combine several possible solutions in one request. The running product exposes these differences because the note can be compared with the interface that prompted it.

The artifact gives everyone the same object to discuss. Reaching the same interpretation still requires communication.

metin-oasis-v6.fly.dev · build under reviewOpen

This is the running application the stakeholder reviewed and the 48 notes describe.

Every response passed through two human decisions

Turning feedback into a new release required five translations:

The next running version then gives the stakeholder another concrete artifact to review.

Meaning can change at every transition. The note may name the wrong cause. My interpretation may miss the reviewer’s concern. The agent may follow the words while violating the intent. The resulting code may work correctly while communicating the wrong thing.

For that reason, no output was treated as self-validating. The stakeholder’s note was checked against the product. My interpretation was checked in the next build. The agent’s code was checked against the approved decision.

Claude Code accelerated implementation. It did not replace the decisions before and after it.

Each note had to answer three questions

The 48 notes covered six product surfaces. Many requested concrete changes: rename a label, distinguish positive from negative performance, or keep columns visible during horizontal scrolling.

Specific wording made a note easier to find in the interface and repository. It did not guarantee that the requested solution was correct.

I evaluated each note in order:

What did the reviewer notice?This identifies the friction behind the note. The underlying problem may differ from the solution the reviewer proposed.
Does the note describe the product accurately?I compared the wording with the running interface. If the referenced element or text was not present, the note could not be implemented literally.
What belongs in this version?Once the concern was clear, I could accept the request, reshape it, split it into separate decisions, or hold it for later.

The order mattered. A note that failed the second question could not proceed as written. A note that reached the third could still contain more work than the current version needed.

Where the notes landed

Home17 notes
Jobs13 notes
AiR Sessions5 notes
Overall Body1 note
Ask OASES6 notes · held this round
Top Navigation6 notes · held this round

The review ledger records 36 notes as addressed and 12 as held.

“Addressed” means the concern received a documented response in the resulting build. It does not mean that every sentence became one literal code change. The following examples show why that distinction matters.

Three notes required three kinds of judgment

The value of the review was not the number of notes processed. It was the reasoning that connected each note to the right product change.

The note described the problem accurately

The stakeholder saw four performance cards with nearly identical amber styling. A result 20% above its target looked too similar to one 7% below it.

In this case, the note matched the visible problem and the intended outcome: positive and negative performance needed to look different.

Because the problem affected a repeated pattern, I solved it at the system level rather than restyling four cards independently. The next version introduced four semantic states. Each state combines a coloured edge, matching value text, and written language.

The decision was already clear, so Claude Code could implement it directly and consistently.

Four performance cards in a row, all with plain white backgrounds and identical amber value text, so a card 20 percent over target looks the same as one 7 percent short of it.

The same four cards, each now carrying a coloured left edge with matching value text — orange, red, teal and amber — so over-target and short-of-target read differently at a glance.

The note named the wrong text

Another note asked to rename “Executive Intervention Queue” in a specific part of the screen. That phrase did not appear there.

The visible interface instead contained “Intervention Que,” which was misspelled, and the heading lacked enough visual weight.

A literal response could have found “Executive Intervention Queue” elsewhere in the code, changed the wrong occurrence, and marked the request complete. The code would have matched the sentence without fixing the screen the stakeholder reviewed.

I therefore treated the note as evidence of confusion rather than as an exact instruction. The next version corrected “Que” to “Queue” and strengthened the heading with an existing design-system treatment.

The request was not implemented as written, but the concern it revealed was addressed.

Anchor bar reading Key Performance Indicators and Intervention Que, with Queue misspelled.

The same anchor bar, now reading Key Performance Indicators and Intervention Queue.

One note contained two decisions

A third note requested filters and collapsible intervention rows as though they formed one change.

They addressed different needs. The existing Filter link did not provide enough control, so better filtering belonged in the next version. Collapsible rows would add new behavior, controls, and state, so that request required a separate justification.

I split the note. The next version introduced a control group for filtering, exporting, and sorting. The collapsible-row behavior was held.

Claude Code could have built both requests quickly. That did not make both requests valuable. Separating them improved the controls without adding unsupported complexity.

Intervention section heading with a single Filter link floated to its right.

Intervention Queue heading with a Filter button beneath it and a right-aligned toolbar of three controls for filtering, export and sorting.

Claude Code shortened implementation, not interpretation

After deciding how to respond, I brought the feedback into Claude Code and pointed it at the repository used to produce the reviewed build.

This reduced the time between an approved decision and working software. It also allowed changes based on the design system to be applied consistently across the product.

The agent still could not determine:

whether a note named the correct element
whether several requests should remain together
whether a literal edit would solve the observed problem
whether new behavior justified its added complexity
whether the result communicated the intended meaning

Those questions required knowledge of the conversation, priorities, and product intent. They therefore remained human decisions.

AI can implement a misunderstanding as quickly as it can implement a sound decision. The review process prevents implementation speed from outrunning agreement.

The response returned as another running product

The approved changes affected four files across four of the six reviewed surfaces. I published the result at a separate URL rather than replacing the original build.

Keeping both versions available served two purposes. The stakeholder could compare the result directly with the product they had reviewed, and the next conversation could begin with working software rather than a written account of the changes.

metin-oasis-v7.fly.dev · next versionOpen

The resulting version remains available beside the build it answered.

The ledger’s 36 addressed notes describe this documented response. They do not imply that 36 requests were implemented word for word. Some were implemented directly, some were reinterpreted, and some were divided into separate decisions.

Communication is part of the quality system

AI lowers the cost of producing another version. As a result, implementation can finish before the people involved have fully interpreted what the previous version revealed.

Faster production does not remove that interpretive work. It makes skipping it easier.

Stakeholders still need to describe their experience. Designers still need to distinguish symptoms from causes and proposed solutions from underlying intent. Product and engineering partners still need to weigh value against cost and scope. Someone must then compare the generated result with the decision the group believed it had made.

Running software improves this process by giving the conversation a concrete object. AI improves it by turning a clarified decision into something testable sooner.

Neither creates shared understanding on its own. The faster implementation becomes, the more deliberately the review process must preserve meaning.

I own the collaboration between stakeholders and agents