When the test agrees with the bug

Verification is only as strong as its ability to disagree with the system it tests. This seems like a given when taken into conscious mind, but it is easy to forget. It’s easy to forget because agreement looks so much like evidence. We tend to take this path for granted and sometimes we don’t even realize we’re taking it. The implementation returns what the fixture predicts.

But what happens when the fixture returns what the implementation predicts. The mock accepts the expected input. The success path keeps moving. Soon, one assertion becomes ten, ten become fifty, and somewhere along the way the agreement gets promoted into evidence.

The trouble is that seeing that venerable all green has a way of making people polite and stroking their inner sense of security. Nobody wants to cross-examine fifty passing tests. The suite has earned its green. It has done its job. That is the test’s job. That is what matters, ain’t it? Yeah right…

People tend to forget that pressure testing requires difference. If every part of the verification inherited the same assumptions that made the implementation coherent, agreement may prove only that one model has learned how to repeat itself and provide the illusion of evidence while proving its own truth in its own court.

Like during an instance, one production path carried fifty green integration tests. Those tests exercised record creation, evidence linking, event delivery, status changes, and approval behavior. They did not merely check that a function had been called. They appeared to verify the feature all the way through.

The eye opening thing was that the feature had never worked in production. Thankfully, it was caught by something that I had been implementing as a safeguard in my work predicitng the exact outcome.

Call me insane.

Two representations, one untested boundary

The failure began with an ordinary boundary. On one side, storage held a raw representation but the other had the application receiving a normalized representation produced by the data layer. Both described the same setting, but they were not interchangeable.

What happened was the production crossed that boundary. The data layer translated the stored value and returned the application-facing form but the tests did not. Their fixture supplied the raw storage-facing form directly to the application. That happened to be the form the broken implementation expected.

But when you looked inside the test world, everything seemed aligned, the fixture matched the implementation, the implementation found the setting and every later action became reachable. Even the records were created, evidence was being linked, the programmed events were emitted and the statuses changed. The assertions were not individually dishonest; they all followed correctly from the world the fixture had supplied.

The production supplied a different world: the one produced by the real boundary. The application received the normalized form, the broken access failed, and the flow stopped before any of the behavior downstream could begin. Basically, fifty tests had verified what happened after a gate that production could not cross.

The test rehearsed a world the software never intended.

This is where test count becomes misleading. Every new downstream assertion made the suite more persuasive without making its starting assumption more independent. The suite was accumulating confidence on top of a premise it was structurally unable to challenge. Otherwise, the implementation is asserting itself twice.

Agreement is not independence

A test does not become an independent source of evidence because it lives in another file, uses another object, or runs in another process. Independence depends on where its expectations came from and what boundary it is capable of making visible. The practise of separating the test from the implementation is not a safeguard. It is a starting point.

Consider a compatibility check meant to prove that a migration preserved every method in an interface. The test compares the methods on the migrated object with the methods exposed by the composed public surface. When if it passes, the sets are assumed identical.

But the public surface holds the exact same object reference. If a method disappears from the migrated object, it disappears from both sides of the comparison at once. The assertion still passes. It can prove that the public name points at the intended object; it cannot prove that the object preserved its methods.

That distinction is not pedantry, in fact, it is the difference between wiring evidence and preservation evidence. The test may be useful, but its authority has a ceiling and many a times that ceiling is broken without the test ever turning red.

The same failure appears when a tolerant mock ignores an input that the real implementation honors, when an offline fixture repeats an assumed provider response shape, when a hand-built descriptor stands in for a different production registry, or when a test exits before reaching the assertion named in its title, the local mechanisms differ. The shared condition is that verification and implementation agree without encountering a constraint capable of separating them. Heavy words, I know, but the idea is quite simple in practice and retrospect.

Mock Truth

A dangerous result is not simply a case of a bad mock. Dont get me wrong, mocks are useful, mocks are cool af. Mock fixtures are necessary. They provide controlled environments which make focused verification possible. The danger arrives when a bounded test model becomes persuasive enough to inherit the authority of system evidence, then continues to hold that authority after the contract it represents has moved. This is what I call Mock Truth.

Mock Truth is an outcome, its like a state of evidence that is not true anymore as it has shifted/moved away from where it originally started. It is not an accusation about intent. It often begins with a reasonable model. A fixture captures the boundary as it is understood today. A mock is intended to remove an expensive test dependency. Downstream tests build on that representation because using the shared fixture is consistent and efficient.

Then one side changes, or, was misunderstood from the beginning. The real adapter normalizes a value but the provider returns a different field shape. The production registry uses another implementation and an ambient file changes the conditions around the test runner. Nothing forces the verification model to meet the changed boundary, so it remains internally consistent.

That consistency is what gives Mock Truth its force. A visibly broken test attracts suspicion whereas, a coherent model with hundreds of green dependents attracts trust. When production later fails somewhere downstream, the green suite directs investigation away from the boundary where the two accounts first separated. This inturn costs countless hours of debugging and the cost of a production failure.

Mock Truth can become and cause drifts without ever becoming a test failure. The test are useful but we are dealing with a situation where the test is not failing but the system is. This is a dangerous situation. What is the core issue here? Is it a wrong test design leading to mock truth occurring or not accounting for system drift over time?

Production is another model

The answer is not to declare production infallible. A mock may encode the intended contract more faithfully than a broken production adapter. That is its purpose altogether. A live response may expose current behavior while violating a documented promise. A snapshot can preserve a deliberate compatibility boundary that a refactor accidentally breaks.

A test never reaches an unmediated reality. Specifications, schemas, fixtures, live responses, integration environments, and production observations are all models at the end of the day. They carry histories and blind spots of their own.

The point of disagreement is therefore not to crown whichever model looks more real. It is to stop silent agreement from being treated as a verdict. When two differently sourced models collide, the conflict becomes visible. Contract ownership, current intent, and provenance can then decide which account needs to be repaired.

Confidence comes from differently sourced models colliding, not from declaring production truth. This surfaces an area of opportunity to improve the test suite. We are exposed to the risk of Mock Truth when a test passes without ever reaching the boundary where the two accounts first separated.

The Disagreement Test

The first question is not whether the suite is green. It is not even whether a test looks thorough. The first question is: what fact in this verification can disagree with the implementation?

If the expected value was copied from the implementation itself, both sides may change together. If the fixture represents the consumer’s preferred shape rather than the boundary’s contract, the test may protect the mistake. If a mock removes the exact constraint whose behavior is being claimed, the test may verify a mechanism while borrowing authority for an outcome it never observed.

The evidence itself needs testing. That test is the Disagreement Test.

Disagreement reveals divergence. It does not decide the case. The test may be wrong. Even the production may be wrong. The specification may be stale. The contract may never have had an owner. The method earns its value by making those conditions impossible to hide behind undisturbed agreement. We need to surface those disagreements to bring them all to the table.

Production model

1. 01The promised contract
2. 02The real boundary transforms it
3. 03Production behavior answers back

unchecked contract drift

Verification model

1. 01The same promised contract
2. 02A cooperative representation replaces the boundary
3. 03Green assertions accumulate authority

INTERVENTION · THE DISAGREEMENT TEST

Force the models back into contact.

1. 01Name the claim and its evidence ceiling.
2. 02Trace provenance and shared ancestry.
3. 03Locate the boundary their agreement avoided.
4. 04Force a collision and resolve what the disagreement means.
5. 05Record scoped confidence and preserve a future collision point.

There are five phases to this design

1. Name the claim and its evidence ceiling.

   • Name what the test claims to prove, be it wiring, local mechanism, boundary translation, contract compliance, composition, or a user-visible outcome.
   • State what this class of test can prove and what it cannot. Do this before the number of assertions begins negotiating for more authority.

2. Trace provenance and shared ancestry.

   • Trace where the implementation input, fixture, mock, expected value, and invariant came from. Determine if it was a specification, schema, implementation detail, production observation, or another test.
   • Find the shared ancestry. If the implementation and its expectation inherited the same assumption, treat them as one source regardless of how many files repeat it.

3. Locate the boundary their agreement avoided.

   • Identify the mapping, serialization, adapter, storage layer, transport, lifecycle transition, registry, or environment that the verification replaced with a cooperative representation.
   • Introduce a differently sourced constraint: the real boundary, an explicit current contract, an invariant, a recorded live shape, or an alternate implementation path.
   • Predict the red condition. Name the plausible drift that must make the verification fail. If nothing can turn it red, reduce the claim or redesign the evidence.

4. Force the models to collide.

   • Bring the two accounts into contact at the boundary where they are supposed to agree. Do not declare either one authoritative in advance.
   • Resolve the disagreement by provenance and ownership. Repair the model that violates a current explicit contract. If behavior changed intentionally, revise the contract before teaching both sides the new answer. If ownership is ambiguous, withhold the confidence that depended on it.

5. Preserve honestly scoped confidence.

   • Record what the evidence now proves. Wiring remains wiring. Boundary translation remains boundary translation. An integrated outcome requires an integrated path.
   • Keep a future collision point and name what must trigger its review. Models will evolve. The protection is not permanent agreement; it is preventing them from drifting indefinitely without contact.

This isn’t a demand for every unit test to boot production. It is a demand that the authority granted to evidence match the boundary it actually encounters.

The useful mock and the honest claim

Cooperative mocks remain useful for local control flow. A same-reference comparison can prove composition wiring. A fixture can make error paths deterministic. A fake transport can test normalization without spending money or depending on a network.

The failure begins when those instruments inherit claims they are structurally incapable of disproving.

A local mechanism test may deserve a fast, cooperative world. It should say that it proves the mechanism. A boundary contract deserves at least one check, sourced from the other side of that boundary. A live integration deserves a smoke test through the real transport. A production-shaped workflow deserves evidence that the packaged artifact, registry, environment, and lifecycle participate in the path being trusted.

Different risks need different collision points. The convention is proportional, not maximal: find the claim carrying release confidence, then make sure something inside its evidence can answer no.

Every green light has a jurisdiction

The most dangerous suite is not the smallest one. It is the persuasive one whose authority has grown beyond the jurisdiction of its models. It is the one that can say “green” without ever disagreeing with the implementation.

Green means that the observed account remained coherent under the constraints the test supplied. That can be valuable. It can also be much less than the surrounding workflow needs it to mean.

Systems engineering requires knowing that difference.

Pressure testing a system means introducing a constraint capable of exposing what its familiar supports concealed. The same obligation applies to the evidence used to trust that system. Salt substitution Referenced concept Salt Substitution Remove a convenience or abstraction to expose what the underlying system can no longer carry without it. tests the supports while the Disagreements test the proof.

Every test should claim only the authority its model has earned. It is our job to make sure that we know that difference and our testing systems dont start agreeing with the bug.