What an integration actually costs

Ask what an integration costs and you will usually be told how long it takes to make the request, which is the one part of the work that is genuinely trivial. Reading the documentation, authenticating and getting a successful response is an afternoon on most modern APIs. The estimate that follows from that afternoon is wrong by somewhere between four and twenty times, and the gap is entirely made up of questions that only arise once the thing is in production.

Those questions are not exotic.

• What happens when the other side does not answer.
• What happens when it answers slowly.
• What happens when it answers twice.
• What happens when it says yes and then loses the record.
• And who finds out.

Each has an answer that is a product decision rather than a technical one, and an integration quoted without them is a quote for the afternoon.

What happens when it is down

The other system will be unavailable, and the only question is what your system does during that window. There are three honest options and they have very different costs. You can fail the user’s action, which is cheap to build and sometimes correct — if the integration is a credit check, proceeding without it may not be allowed. You can queue the work and complete it later, which is the usual right answer and the expensive one. Or you can proceed without it and reconcile afterwards, which is cheapest to build and most expensive to get wrong.

Queueing is expensive because a queue is a system. It needs durable storage so a restart does not lose it, a worker that drains it, a retry policy with a backoff, a limit on attempts, somewhere for the items that exceed the limit to go, and a human process for looking at that place. Teams who say they will "just add a retry" have usually built the first two and none of the last four, which means the queue silently accumulates failures nobody reads.

That last place — the dead letter queue, whatever it is called — is the single most commonly omitted component in integration work and the one that determines whether a failure is noticed in minutes or in a quarterly reconciliation. It needs to be somewhere a person looks at, with an alert when it is non-empty, because a queue of failures that nobody is told about is functionally identical to having dropped the data.

Slow is harder than down

A system that is down fails fast and is easy to detect. A system that takes forty seconds to respond is far more damaging, because it holds your resources while it does nothing.

1. Your connection pool fills with requests waiting on it.
2. Requests unrelated to the integration start queueing behind them.
3. A slowdown in a non-critical third party becomes an outage of your own product.

This is the failure mode that takes a whole site down, and the integration that caused it is usually something peripheral like an address lookup.

The defences are a timeout, which must be set explicitly because most client libraries default to none or to something absurd, and a circuit breaker, which stops calling a service that is clearly failing and gives it room to recover. The timeout number is a product decision: it is the longest a user should wait for this feature, which is usually two or three seconds, and is almost never the thirty that a default provides.

Isolate the resource

Beyond timeouts, the structural fix is to stop the integration sharing a resource pool with everything else, so that exhausting it cannot starve unrelated work. Giving the third-party calls their own small pool means the worst case is that the feature stops working rather than that the product does, and it converts a site-wide outage into a degraded feature. This is a few lines of configuration and it is the cheapest resilience available in this area.

Everything will be delivered twice

Any integration with retries will eventually process the same message twice, and any webhook provider will eventually deliver the same event twice, and both of those are correct behaviour rather than bugs. A network timeout does not tell the caller whether the work was done, so the only safe thing it can do is try again. The consequence is that your handler must be able to receive the same thing twice without doing it twice.

The mechanism is a unique key per operation, stored before the work is attempted and checked on arrival, so a repeat is recognised and discarded. This is simple to describe and routinely omitted, and the symptom is the one that generates the most unpleasant class of support ticket: a customer charged twice, an order shipped twice, a notification sent twice. The second of those is recoverable and the first damages trust in a way that takes months to repair.

There is a subtlety about where the key comes from. If you generate it, you can only deduplicate your own retries; if the other side provides an event ID, you can deduplicate theirs too, which is what you need for webhooks. Most providers do supply one and most integrations ignore it, which means the deduplication covers the less common of the two sources of duplicates.

The two systems will disagree

Over any length of time, your record of what happened and theirs will diverge. A request timed out after the work was done, a webhook was missed during a deployment, somebody edited a record directly in their admin panel. This is not avoidable by being careful; it is a property of two systems holding related state with a network between them, and the only question is whether you find out.

So a real integration includes a reconciliation job: something that periodically compares the two sides on a key figure — counts, totals, a list of IDs — and reports the differences. Daily is usually right. It is perhaps a day of work, it is almost always left out of the quote, and it is the only thing standing between a silent divergence and a finance team discovering a mismatch three months later with no way to tell when it started.

The reconciliation also pays for itself in a way that is hard to predict: it is how you discover that your understanding of their API was wrong. Twice we have found, through a daily count mismatch, that a status we treated as final was not final on their side, which is a semantic misunderstanding no amount of reading the documentation had caught.

Their sandbox is not their production

Development against a sandbox establishes that your code is correct and almost nothing about whether the integration will work. Sandboxes typically have no rate limits, no latency, a tiny data set and a tolerance for malformed input that production does not share. The first contact with the real thing therefore produces a cluster of surprises, and the most common is a rate limit that the documentation understates or does not mention.

Rate limits are worth a paragraph because the mistake is subtle. People check their average request rate against the published limit and conclude they are comfortably inside it, when limits are enforced on short windows and traffic arrives in bursts. The figure that matters is the peak over whatever window the limit uses — per second, per minute — and a service averaging ten requests a second can easily exceed a hundred-per-second limit during a batch job.

So budget a week for the first week in production, and plan the first release as a small percentage of traffic rather than all of it. That week is where the real cost of the integration becomes visible, and discovering the rate limit at ten percent of volume is a configuration change while discovering it at full volume is an incident.

The contract changes without notice

An integration is a promise about a data shape made by somebody who does not work for you, and that promise gets amended. A field that was always present becomes optional, an enumeration gains a value, a number arrives as a string, a date changes timezone convention. None of these is announced in a way that reaches your team, and all of them are indistinguishable from your own bug until somebody reads the payload.

The defence is to validate what arrives at the boundary and to fail loudly and specifically when it does not match. A validation error that says "field status received value PENDING_REVIEW which is not in the known set" turns a three-hour debugging session into a thirty-second diagnosis, and it attributes the change to the right party, which matters when the fix requires their cooperation.

There is a judgement call inside that, which is how strictly to validate. Rejecting an unknown enumeration value protects your data and stops the integration; accepting it and carrying it through protects availability and risks a silent wrong decision downstream. The answer differs per field and is worth deciding deliberately — strict on anything that drives money or permissions, tolerant on anything decorative.

Credentials expire on their schedule

Every integration holds a secret that has a lifetime, and the lifetime is set by the other side. API keys get rotated under their policy, OAuth refresh tokens expire after an idle period, certificates have an expiry date somebody wrote down once, and signing keys change when the provider has an incident. Each of those produces an outage that looks exactly like the provider being down, on a date nobody put in a calendar.

The cheap mitigation is to monitor the expiry rather than the request. Anything with a known end date — a certificate, a key with a stated rotation period, a token with an issued-at claim — can be checked on a schedule and warned about a fortnight ahead, which converts a Sunday outage into a Tuesday task. This takes an afternoon to build and is the single highest-return piece of integration monitoring there is.

The related trap is a credential that works in one direction only. A webhook signed with a shared secret keeps verifying long after your outbound key has been rotated, so half the integration is healthy and half is not, and the dashboard shows green because the half being monitored is the half that works. Both directions need their own check, and the inbound one needs a synthetic event rather than a wait.

Rate limits are a design constraint

A rate limit is not an error condition to be retried, it is a budget that the design has to fit inside. If the provider allows sixty calls a minute and your nightly job needs four thousand, no amount of retry logic fixes that — the job needs to batch, cache, or run over a longer window. Discovering this during the first full-volume run is the normal sequence of events and it is avoidable by reading the limit before writing the loop.

And the limit is usually shared

The subtlety is that the limit usually applies to your whole account, not to one process, so a report somebody runs by hand can exhaust the budget that the sync job was relying on. That makes it a shared resource with no coordination, and the symptom is an intermittent failure that correlates with nothing in your own logs. A single client with a queue in front of it is the fix, and it has to be the only path out.

When you do hit the limit, the retry has to respect what the provider tells you. Most return a header saying when to come back; honouring it is both faster and politer than an exponential backoff that guesses. Ignoring it is how an account gets a longer penalty than the original limit, which is a failure mode where your own retry logic is the thing extending the outage.

Log enough to settle an argument

Sooner or later a dispute arrives in which you believe you sent something and the provider believes you did not, and that dispute is resolved by whoever has the record. The record that settles it contains the request as sent, the response as received, the exact timestamp, and whatever correlation identifier the provider issued. Without the identifier you are describing an event; with it you are citing one, and their support team can look it up.

That logging has to be built with redaction from the start, because the payloads contain exactly the data you are not allowed to keep — card numbers, national identifiers, personal details. A field-level allow-list is the only approach that stays correct as the payload changes; a deny-list fails silently the first time the provider adds a field, and it fails by writing the new sensitive field to disk.

• The request as sent, through a field allow-list.
• The response status, body and timing.
• Their correlation identifier, kept searchable.
• Your own idempotency key for the same operation.
• A retention period somebody has actually agreed.

Someone has to be told

The last item, and the one most often missing entirely, is who finds out when the integration is failing. An integration that fails loudly to a log file nobody reads has the same business outcome as one that fails silently. What is needed is an alert on the thing that matters — the dead letter queue being non-empty, the reconciliation reporting a difference, the error rate crossing a threshold — routed to a person who can act.

And the alert has to be accompanied by a decision about who that person is, which is an organisational question that engineering cannot settle alone. In several of the integrations we have reviewed, the failures were being detected correctly and emailed to a shared mailbox that was created for the project and read by nobody, which is a particularly frustrating way to lose data because every technical component worked exactly as designed.

1. An explicit timeout, chosen as a product decision.
2. A durable queue with an attempt limit and dead letters.
3. A deduplication key, using their event ID where offered.
4. A separate resource pool, so their slowness is contained.
5. A daily reconciliation on a figure both sides hold.
6. An alert, and a named person who receives it.

The useful consequence of asking these questions early is that some of them have a cheap answer for a given feature. If the integration is a non-critical enrichment — looking up a company logo, say — then failing silently and trying again tomorrow is entirely acceptable, and knowing that lets you build the simple version deliberately rather than by omission. The expensive mistake is not building the simple version; it is building the simple version while believing you built the robust one.