What actually breaks when two systems talk to each other
By Michael James, Principal & Platform Lead at GO Build Labs
An integration that works on launch day proves very little. Two systems exchanging data is not a feature you build once. It is a relationship you maintain, and it fails in ways that are specifically designed to be hard to notice.
Here is what actually goes wrong, in rough order of how often we have had to fix it.
1. The other side changes without telling you
A vendor deprecates an endpoint. A field that was always a string starts sometimes being null. An enum gains a value nobody documented. None of these produce an outage. They produce a small percentage of records that quietly fail to sync, and by the time somebody notices, the gap is three weeks wide.
The defense is boring and effective: validate the shape of what arrives rather than assuming it, and treat anything unexpected as an error worth alerting on rather than a row to skip.
2. Auth expires at the worst time
OAuth tokens expire. Certificates expire. Service accounts get disabled during an offboarding because nobody knew what it was for. This one has a signature: the integration works perfectly for months and then stops completely, usually on a weekend.
Refresh tokens automatically, and put an expiry date for anything that cannot be refreshed on a calendar that a person actually looks at.
3. The queue backs up
Everything is fine at fifty records an hour. Then a customer uploads a spreadsheet with nine thousand rows, or the other system is slow for an afternoon, and work piles up faster than it drains. If there is no back pressure, this is where you get duplicate records, half-applied updates, and a very bad Monday.
Retries need to back off rather than hammer. Anything that fails repeatedly needs somewhere to go, a dead-letter queue, where a human can look at it later. Losing a message silently is worse than failing loudly.
4. The same thing happens twice
Networks time out after the work was done but before the confirmation arrived. The sender retries. Now there are two orders. Any operation that can be retried, which is all of them, needs to be safe to run twice, keyed on something stable from the source system.
This is the failure that costs real money, because unlike the others it produces plausible-looking wrong data rather than an obvious gap.
The alarm most people do not build
Almost everyone alerts on errors. Far fewer alert on silence, and silence is what most of these failures actually look like.
If orders normally flow every few minutes during business hours, then two hours of nothing on a Tuesday afternoon is an incident, even though no error was raised anywhere. An alert on "expected activity has not happened" catches the expired token, the deprecated endpoint and the stuck queue, all three, and it is perhaps two hours of work to build.
Official APIs, and what to do when there is not one
Build on a documented API where one exists. Screen-scraping and browser automation work right up until the vendor changes a button, and they change buttons without announcing it because from their side that is not a breaking change.
Sometimes there genuinely is not an API. Plenty of the systems small businesses depend on only offer file drops. That is workable, and we build on it regularly, but treat it as a contract you're inferring rather than one you have been given. Validate every file, alert when one does not arrive, and never assume the column order.
What good looks like
An integration you can trust has four properties, and none of them are visible in a demo:
- It validates what arrives instead of assuming it.
- It is safe to run the same operation twice.
- Failures go somewhere a person will find them.
- It alerts when nothing has happened, not only when something went wrong.
We build integrations as part of connecting the systems you already run, and the monitoring above is included rather than an upgrade. Not because it is generous, but because we are the ones who get called when it breaks.