Third-party integration horror stories and how to avoid them

Every software project reaches the same moment. The build is going well, everyone is pleased with themselves, and then someone says “it just needs to talk to our accounting system”. That is the point where third-party integration enters the picture, and it is where a surprising number of otherwise healthy projects quietly come apart.
At Element Software we have been called in to rescue enough of these to notice a pattern. The failures are rarely exotic. They are the same handful of mistakes repeated across different platforms and industries, usually made by people who did nothing obviously wrong at the time. Here are four of them (details changed, because nobody needs naming) and what we do differently.
Horror story one: the API that changed without warning
A distribution business had orders flowing automatically from their online store into their inventory system. It ran without complaint for two years. Then the store platform deprecated an endpoint, announced it on a developer blog nobody at the business had ever read, and switched it off.
Orders stopped syncing. Nothing broke loudly. The integration just returned an error into the void, and because nobody had built any alerting, the void kept it. Nine days passed before a customer rang to ask where their delivery was. By then there were four hundred unprocessed orders and a very long weekend ahead.
The integration didn’t fail. It failed silently, which is so much worse. If something breaks at 2am you want to know at 2am, not when a customer tells you a week or two latter with weeks worth of fixes on top of the integration fix.
Ravinder, Head of Software Development, Element Software
Horror story two: the sync that ate itself
A services company connected their CRM to their email marketing platform. Both systems were set up to push contact updates to the other, which sounded sensible in the meeting. In practice each update triggered a corresponding update in the other system, which triggered another one back.
The loop ran for a weekend. On Monday there were tens of thousands of duplicate contact records, a marketing list that could no longer be trusted, and a sales team who had lost confidence in their own database. Cleaning it up took longer than building the integration had.
The mistake was not technical. It was that nobody had decided which system owned the customer record.
Horror story three: the integration only one person understood
A contractor built a genuinely clever piece of middleware connecting a job management system to an accounting package. It worked. It also lived entirely in that contractor’s head, authenticated through their personal account, and was documented nowhere.
The contractor moved on. Some months later their account was deactivated during a routine tidy-up, the access token died with it, and the business discovered they had no idea how any of it worked. No documentation, no credentials, no repository they could find. They ended up paying to have the whole thing rebuilt from scratch.
Horror story four: it worked perfectly in testing
An operations platform was tested against a sample data set of a few hundred records. It was fast, it was clean, everyone signed off. Then it went live against the real database of nearly half a million rows.
The vendor API had a rate limit that the test volume never came close to touching. The overnight sync started, hit the ceiling within twenty minutes, and began timing out. It never finished. It never finished the next night either, or the one after that, and each failed run left the data in a slightly different half-migrated state.
Testing an integration on sample data tells you the logic works. It tells you nothing about whether it will survive Monday morning. Those are two different questions.
Ravinder, Head of Software Development, Element Software
How to avoid third-party integration disasters
Now that you know the possible horror stories, you’ll want to know how to avoid them. Luckily someone already has the experience, unfortunately, some of it was learned from experience. But you get the benefit of our experience.
Map the data before anyone writes code
Decide which system is the “source of truth” for each type of record. Customers, invoices, jobs, stock levels: pick one owner for each. Write it down. Most sync horror stories trace back to two systems both believing they were in charge.
Assume the API will change
Third-party APIs are not your APIs. They get versioned, deprecated and rate limited on somebody else’s schedule. Build against a versioned endpoint where one exists, subscribe to the vendor’s developer change notifications, and treat their roadmap as something you actively monitor rather than something you hope stays still.
Design for failure, not the happy path
A good third-party integration assumes things will go wrong and plans for it. That means retries with sensible backoff, queuing so a failed call is not a lost call, logging you can actually read and alerting that reaches a human. If your integration cannot tell you it is broken, you do not have an integration, you have a liability.
Half the work in a solid integration is the error handling. Clients sometimes wonder why custom software takes as long as it does. It’s because we are building the part that (hopefully) you never see or notice.
Own your credentials and your documentation
API keys and service accounts belong to the business, not to an individual and not to a contractor. Use dedicated service accounts. Store credentials somewhere the business controls. Keep a plain-language document describing what connects to what, what it does and who to call when it stops. It takes an hour and it has rescued more projects than any clever code ever has.
Test with real volumes
Run the integration against production-scale data before you go anywhere near a launch date. Rate limits, timeouts and memory ceilings only show up under load, and finding them in testing is an inconvenience while finding them in production is an incident.
When a third-party integration is the wrong answer
Sometimes it is. If two systems only need to share data once a day, a scheduled export can be more robust and far cheaper than a live connection. If the platform you are integrating with has a poor API, no versioning and no support, the honest advice is often to replace the platform rather than build a bridge to it.
Often times, it’s a rickety old bridge like in Indiana Jones. Not naming the accounting software I’m talking about (for legal reasons). But, you know who you are!
We would rather tell a client that up front than build something we know will need rescuing in eighteen months.
Getting third-party integration right
The systems in most businesses were never designed to work together. Making them do it well is unglamorous work that depends far more on planning, ownership and error handling than it does on clever code. Done properly it disappears into the background and simply works, which is exactly the point.
If you are planning a third-party integration, or you are living with one that has become a source of dread, we are happy to take a look.