the suite was green and the numbers were wrong

A client runs on six services that don’t talk to each other. I built the thing that makes them talk, then spent most of the project finding out how much of my own test suite had been agreeing with me instead of testing anything.

What it is. Six business services behind one typed interface. I’m not going to tell you which six, because that’s somebody’s operating setup and it isn’t mine to publish, but the shape is the point: each one is a Python module with the same interface, so an ordinary business question turns into code. What’s outstanding, what’s overdue, what’s in flight. All 583 tests run with no network and no credentials. That last sentence is the one I’d put on a badge, and it’s also the one that caused the most trouble.

Six services, six dialects. Each one has its own authentication, its own pagination, its own error format and its own undocumented opinions about all three. So most of the library is the boring part: retries with backoff, a rate limiter per service, streaming downloads, and one exception type that carries the status code and whether a retry is worth attempting. The interesting part was finding out where I’d been confidently wrong.

The suite was green. That was the problem. The tests inject a mock transport, which is what makes them fast, hermetic, and completely incapable of telling me whether the API agreed with me. A mocked test proves the code matches the mock. Nothing else. The mocks never once told me I was wrong. I’ve come to think of a green suite as a colleague who has learned to agree with me. I knew that in the abstract, the way you know most things.

Then the first integration with real data behind it arrived, and every single create, read, update and delete was broken. Entity routes are lowercase, and I had used the capitalized name — which is the response envelope key, not the route. The mocks used the capitalized name too, so they were delighted.

The one I like most: a count method that returned zero. Always. On everything. It survived the entire suite, because the provider reports a total as count on some responses and totalCount on others and I was reading one of them, and every mock encoded my assumption, so every mock was satisfied. It read zero for weeks and nobody noticed, because nobody had asked the number for anything yet. The only reason it survived that long is that it was wrong in a quiet room. A function that is confidently wrong is worse than one that throws.

Then pagination. Four identical paginated reads returned four different totals, with zero duplicates between them. Not fewer results — different results, silently skipping records, from the same query, four times in a row. Four identical questions, four different answers. If a person did that you’d stop asking them things, and you’d be right to. I rewrote the consumer to report what’s stable instead of a total that looks precise and isn’t. A number you can’t trust is worse than no number at all.

And the small ones. A webhook field was always empty because a naive snake-to-camel conversion produces callbackUrl while the provider spells it callbackURL. Every field ending in an acronym is now suspect to me on sight. A write endpoint returned no body whatsoever: the write succeeded, the object handed back was empty, and the helper reading a header from it got nothing, so it presented as a failure. It wasn’t one. If you’ve ever sent an important email and heard nothing back, you know roughly how that feels. That cost an hour and earned a paragraph of documentation.

Something like 17 defects came out of the two integrations I tested hardest, and almost none of them were reachable from a mock. That’s the whole argument for driving every connector against the real service even when the suite is green, and it isn’t a subtle argument.

Two things I wrote down because I got them wrong. I concluded a capability was impossible after inspecting one endpoint’s payload. I concluded a value set was complete because the documentation listed an enum. The real payloads disagreed both times. "A documented enum isn’t a complete enum" is now a sentence I keep where I can see it.

The tooling fought back as well. On macOS a launchd agent can’t read ~/Documents — TCC blocks it — and the error you get is a Python traceback about the site module and "Operation not permitted" on the virtualenv. It reads as a corrupted environment. It isn’t one. The tell is the error kind: EPERM, not EACCES. Separately, a compiler shipped with the Command Line Tools refused the SDK shipped beside it, on the grounds that the compiler didn’t support it. Which is why the Swift build loop now tries the SDKs on disk newest-first rather than trusting any of them.

And my favorite: a UI defect that looked like a color problem was a container problem. The content was being drawn inside a menu, where text stays dim. Two rounds of color fixes achieved nothing, because the colors were fine. The renderer I was using to check the work couldn’t show me the bug, because it drew the view and not the container around it.

The half nobody asks for. A library is half a deliverable. The rest is making the numbers visible: one self-contained HTML page with a panel per business question, and every panel that can’t answer its question says so on the page rather than rendering a placeholder. A blank is better than a plausible wrong number. I would rather the dashboard admit ignorance than guess out loud where someone might believe it. Confidence is cheap to produce and expensive to be wrong with, and this is somebody’s actual business.

There’s a machine-readable snapshot written beside the page, so anything that isn’t a browser can read the same numbers without re-running a 50-second sweep across every live service. It rebuilds every 15 minutes. There’s a menu bar app in Swift that reads the snapshot, built with the Command Line Tools alone — no Xcode, no signing identity — which is its own small argument with the toolchain.

The installers refuse to install. If the configuration is one known to fail silently, the script stops and says why, and the reasons are asserted in tests. A job that looks healthy and does nothing is the worst outcome available, and it’s the one you get by default. It is also, with the benefit of hindsight, a fair description of me before about ten in the morning.

Where it stands. Six connectors, all verified against the live services. Five have automated verifiers. The webhook path was checked end to end over a public tunnel, with forged and unsigned deliveries both rejected, because a webhook that accepts anything is worse than no webhook.

What I’d tell the next version of me. Write the mocks — they’re what makes the suite fast enough to run constantly. Then run everything against the real thing anyway, because the mocks are the one part of the suite that can’t be wrong on your behalf. And when a number looks precise, check it’s stable before you put it in front of somebody who’ll make a decision with it. Somebody always makes a decision with it. That’s the part that keeps me up.