Building WriterzRoom, Part 6: The Quality Gate Is a Decision About What Can Ship

Imagine a generated financial explainer whose table adds up correctly but whose closing paragraph promises a guaranteed return. In this hypothetical, publishing the draft unchecked would turn a fixable writing problem into a customer-facing claim. A quality gate exists to catch that failure before the Publisher receives the asset.

The harder question is what happens when the table cannot be parsed, a citation looks valid but has not been checked, or an editor never ran. Each situation needs a different outcome. Treating them all as a green check would hide the very information a verification layer should preserve.

Part 6 of building WriterzRoom focuses on that trust layer: verification, the mandatory Quality Gate, repeatable evaluations, and the content passport that accompanies an asset. The argument running through it is that a useful verification system makes explicit decisions about publication and preserves the limits of what it checked.

Writers need actionable feedback. Editors need a defensible review record. Engineers need behavior they can test and reproduce. These stakeholders share a pipeline, but they do not share the same definition of success.

Why Verification Sits Before the Publisher

By this stage of the series, the relevant question has shifted from producing content to deciding whether that content can leave the system. We do not need to reconstruct the earlier pipeline to understand the boundary. Whatever route a generation takes, WriterzRoom requires it to converge on the Quality Gate before reaching the Publisher.

That placement makes the gate a publication control. An earlier agent can recommend changes, and a prompt can describe rules, but neither guarantees that the final artifact obeys them. The gate checks the artifact and the run record at the point where failure still has a contained consequence.

WriterzRoom raises ValueError("Quality gate failed: ...") when a blocking check fails. The exception creates a clear stop condition for downstream code. You should not have to infer publication eligibility from a review paragraph that sounds generally positive.

The checks have different scopes. SEO readability blocks only when the SEO agent actually ran. Vertical-specific requirements cover citation minimums, mandatory disclaimers, forbidden claims, and certain conditional claims. Critical or high findings in the enterprise quality report block publication, as does a high-severity novelty failure involving a near-duplicate of the user's prior work.

Other signals remain advisory. Source overlap and claim-level evidence are recorded. Optional, tier-gated model judging and semantic entailment checks also produce recorded findings rather than independent publication permission. A domain verifier follows the enforcement setting declared for its vertical.

This separation prevents an advisory observation from accidentally becoming a product-wide prohibition. It also prevents a serious violation from disappearing into an average quality score. A draft can be readable, original, and well structured while still containing a claim that the applicable policy forbids.

Deterministic contract failures get another response: a directed Writer retry, followed by a recorded result. Examples include the wrong post count or a mismatch between platform and content type. That bounded retry gives repair a defined place in the workflow without allowing an endless cycle of rewriting and rechecking.

The gate resembles an airport departure check in one limited sense: different checks answer different questions before something leaves. A valid ticket does not establish that every other requirement has been met. Likewise, correct arithmetic cannot stand in for compliant language or completed editorial review.

For the Writer, a Finding Must Explain the Repair

A writer experiences verification through the feedback it returns. "Quality failed" offers little help. A useful finding identifies the affected passage, the rule it triggered, and the kind of repair available. The writer can then change the draft without guessing which part of an opaque score caused the rejection.

WriterzRoom's claim checks address a gap that prompts alone leave open. Vertical configuration can tell the model to avoid certain language, but a generated draft can still contain it. The implementation in core/policy.py turns some of those instructions into executable checks.

Forbidden claims use a case-insensitive phrase match over the body, excluding the bibliography. That scope is deliberate: a phrase appearing in a source title should not automatically be treated as an assertion made by the article. Context still matters within the body, especially when a disclaimer explicitly denies a prohibited promise.

The negation guard handles that situation. In a hypothetical legal draft, "We cannot guarantee any outcome" contains language associated with a guarantee while rejecting the guarantee itself. Blocking that sentence would punish the writer for including a legitimate disclaimer. The guard accepts a limited false-negative risk to avoid that specific false-positive behavior.

You can see the stakeholder trade-off here. The writer wants legitimate language to survive. The editor wants prohibited promises caught. The engineer needs a rule whose behavior can be described and tested. A phrase detector can support those goals, but it cannot resolve every possible meaning in unrestricted prose.

Conditional claims require a different response. Some conditions ask for a nearby citation or a declared replacement term, which makes them mechanically checkable. Others require a judgment about whether the prose actually describes qualifications, circumstances, or supporting context. WriterzRoom records those unresolved conditions as conditional_claim_needs_review.

For the writer, that status should mean a specific review task remains. It should never mean the system silently accepted the claim. A draft may proceed under an advisory policy, but the unresolved condition still belongs in its record.

This also changes how feedback should appear in a writing interface. A blocking finding needs a clear publication consequence. An advisory finding needs a clear review consequence. Combining both into an undifferentiated list encourages writers to fix the easiest items while overlooking the one that actually prevents delivery.

For the Editor, Assurance Must Describe the Actual Run

Editors need to know what happened to this asset. A tier name describes an intended service level; it does not establish that every promised component successfully ran. Failover, missing research, or a degraded editor step can separate the nominal workflow from the completed workflow.

WriterzRoom distinguishes declared assurance from achieved assurance. The declared levels are basic for Quick, reviewed for Standard, and verified for Premium. Achieved assurance comes from the execution log through core/assurance.py, so the label reflects completed components.

At the basic level, the structural, contract, and compliance gate has run. Reviewed assurance additionally requires the editor to have run and the gate to have passed. Verified assurance additionally requires the researcher to have run and produced sources. These are operational definitions, so the interface should explain them rather than expecting readers to interpret the labels intuitively.

The next level, attested, requires the artifact's own domain objects to be parsed, decided by executable verifiers, and passed. No tier declares that level in advance. An individual generation earns it through the checks actually completed.

A useful editorial dashboard would keep the declared and achieved values visible together. If a Premium run achieves only reviewed assurance because research produced no sources, meets_tier_contract: false records the shortfall. The system also logs an ASSURANCE SHORTFALL. The downgrade becomes an inspectable fact instead of a hidden implementation detail.

Citation audit status needs similarly careful interpretation. passed means every marker was checked against retrieved sources. repaired means invalid markers were stripped. suppressed_by_template_contract means the template has no attribution surface, while null means the audit did not run.

Removing an invalid citation marker does not establish the truth of the sentence it accompanied. An editor may still need to remove, qualify, or substantiate that claim. Repairing attribution syntax and establishing factual support are different tasks with different consequences for publication.

The same restraint applies to deterministic verification. A financial table can reconcile without its inputs being accurate. A legal citation can be internally consistent without establishing that the cited matter exists or remains current. Editors need these limits beside the results, because a positive technical verdict can otherwise sound broader than the check itself.

WriterzRoom keeps those boundaries in VERIFIER_LIMITATIONS within core/product_manifest.py and serves them to API consumers. That gives an integrating application enough information to explain a result accurately, provided its interface preserves the limitation.

For the Engineer, "Could Not Decide" Must Survive the Pipeline

Engineers face a deceptively simple modeling problem: should a verification result be a Boolean? A Boolean works when every input can be decided. Document verification often cannot make that promise. The system may encounter an unfamiliar table layout, ambiguous dates, or prose that never states the relationship a check needs.

WriterzRoom therefore uses four verdicts. VERIFIED means the verifier parsed the relevant objects and found them consistent. VIOLATION means it parsed them and found an inconsistency. UNVERIFIABLE means it could not decide. NOT_APPLICABLE records that the check did not apply and contributes no verified object or coverage.

Preserving those states is a practical requirement for storage, API responses, and user interfaces. If your frontend turns UNVERIFIABLE into an empty cell, the reader loses the coverage gap. If it turns the result into a tick, the interface invents assurance that the verifier never supplied.

The document verifiers follow a narrow design rule: reconcile two statements about the same quantity or object. A stated total can be compared with its rows. A price per square foot can be compared with price and area. A downtime budget can be compared with an availability target. Internal redundancy provides something executable to test.

That rule also sets the limit. The real estate verifier cannot establish that the property's stated area is accurate. The availability verifier cannot establish that the service met its commitment. An ISBN check digit can be validated, while DOI syntax can only be assessed for form because a DOI has no checksum.

Some domains do not offer a suitable relationship to reconcile. WriterzRoom deliberately gives entertainment no domain verifier and records why. Building a checker merely to populate every category would create a reassuring interface without a dependable decision behind it.

Registration alone does not activate a verifier. A vertical must name it and choose enforcement:

verification:
  enforcement: "blocking"
  verifiers:
    - "fintech.tables"
    - "common.quantities"

This configuration is part of publication policy. A checker calibrated for one document shape should not silently run across unrelated content. An artifact contract can require at least advisory enforcement, but applicability still needs an explicit definition.

Structured regulatory proposals use a separate registry from document text. Their checks live in core/verification/regulatory/proposals.py, and findings bind a rule version to the exact proposal-input hash. Keeping those evaluations separate prevents coverage of structured fields from being presented as coverage of prose.

A less obvious engineering issue appears before any verdict: text layout can change whether the parser sees a relationship. WriterzRoom's corpus includes hard-wrapped documents. A phrase split across lines can evade a pattern that succeeds on every single-line unit test.

The fix is small. Flattening individual newlines to spaces lets the pattern read the phrase while preserving offsets through a one-for-one character substitution. More aggressive normalization would require an offset map, or the finding might highlight the wrong passage in the original draft.

Practical Applications: Test the Gate, Then Preserve Its Decisions

If you're building a similar system, begin with the publication boundary and work backward. Define which failures stop delivery, which findings require review, and which signals merely describe the artifact. Then make those distinctions part of the result schema before designing a summary score.

Evaluations, often shortened to evals, provide repeatable tests for those behaviors. Across a writing pipeline, they can assess correctness, tone, structure, and grounding. The verification harness described here has a narrower job: test whether deterministic checkers detect defined defects without wrongly blocking clean documents.

WriterzRoom runs that harness through:

PYTHONPATH=. python -m langgraph_app.evals --gate

The harness takes a clean corpus document, changes it in one known way, and checks that the verifier reports the specific expected check ID. No model participates in this test. The result is reproducible enough to run in continuous integration on each pull request.

For a hypothetical financial fixture, you could change a stated total while leaving its rows intact. The assertion should require the total-reconciliation finding. Accepting any failure would let an unrelated parser error make the test appear successful, even though the intended defect went undetected.

Mutation behavior itself needs tests. If the mutation cannot find a suitable target, it returns None, and the case is skipped. If it claims to mutate a document but changes nothing, it raises. Otherwise, the harness would penalize a verifier for failing to detect a defect that was never introduced.

Skipped cases also need coverage accounting. A registered mutation can remain in the repository while applying to no current fixture. WriterzRoom distinguishes "no mutation registered" from "mutations registered but never applied" and fails the evaluation gate when a verifier has no measured coverage.

Apply the same rule to your own review process: an existing test function is no proof that the relevant behavior was exercised. A corpus document of the right shape must reach the checker, receive the intended mutation, and produce the expected decision.

Adding a WriterzRoom verifier therefore requires more than a module. The implementation needs registration and imports, entries in EXPECTED_VERIFIER_IDS and VERIFIER_LIMITATIONS, activation in the vertical configuration, a seeded mutation, and a suitable corpus document. The evaluation gate checks for missing test coverage.

Mutations reuse the verifier's parsers to keep the meaning of an authority, range, or other object consistent. You should still review fixtures independently. Shared parsing can align the test and implementation while leaving a shared blind spot undetected.

The resulting baseline measures performance on defined seeded defects in a limited corpus. It does not establish general real-world accuracy. Clean-document checks are equally important: a system that detects seeded mistakes but wrongly blocks acceptable drafts can make the writer's workflow unusable.

For tone and broader editorial quality, use a separate evaluation design with explicit criteria and human-reviewed examples. An automated first reviewer can prioritize concerns, but subjective judgments need calibration. Keep those scores separate from arithmetic findings and policy violations so their different meanings remain visible.

The final practical step is preserving the run context. core/provenance.py records identifiers for the request, tier, template, style profile, vertical, platform, resolved models, and source set. Recording resolved models matters because role selection happens at call time and can change through failover.

Configuration identity also needs care. YAML content can change without an enforced version bump, so an identifier alone cannot fully describe the rules used. A reproducibility design should retain the effective configuration through a snapshot or content hash. That is a design requirement to address, rather than a capability to assume from a version label.

The source-set identifier uses a hash over sorted source identities, making it stable across retrieval order; sources without URLs use a locator. Such an identifier helps distinguish evidence sets, although replaying a generation can also require the retrieved content and effective configuration.

WriterzRoom's content passport, implemented in core/passport.py, is a versioned document intended to leave through export, audit, or customer compliance review. Its schema uses additive-only changes. For an integrating application, that document is the place to carry the asset's review context beyond the generation service, while preserving the distinction between completed checks and unresolved findings.

Choose One Failure That Must Stop Publication

The next decision for your team is which single failure class must block publication in the first version of the gate. Choose something consequential, detectable, and explainable, such as contradictory financial totals or a prohibited outcome promise. Give it a clean fixture, a seeded defect, and a result the writer can act on.

Starting there connects every stakeholder's needs. The writer gets a repair path, the editor gets a clear release condition, and the engineer gets a behavior that can be tested repeatedly. Additional checks can grow around that boundary without changing what a pass means.

The enduring value of verification is the record it leaves of what was decided. A publishable asset should carry both its completed checks and its remaining uncertainty. That gives the next person who handles it a sounder basis for judgment than fluent prose alone.