Problem
Integration pipelines often fail unpredictably, and partial failures can create duplicates, data corruption, or difficult reconciliation work.
Context
Operational systems integrating with third-party platforms where data quality varies, failures are inevitable, and reversing incorrect transactions is expensive.
Solution
Integration pipelines should assume that failure will occur. The goal is not to prevent failure entirely, but to ensure failures are contained, visible, and recoverable.
A reliable pipeline separates processing into clearly defined stages, where each stage has a single responsibility and can stop processing safely if validation fails.
A typical structure looks like this:
- Export detected
- Export retrieved
- Parsed
- Validated
- Matched
- Queued for posting
- Posted to destination system
Each stage exists to prevent incorrect data from progressing further downstream.
1. Export detected
The system identifies that new data is available but does not process it immediately.
This stage should only confirm availability. Avoid performing transformations or validation here. Separating detection from processing allows retries without duplicating work.
2. Export retrieved
The data is downloaded or pulled into a controlled environment.
At this stage the goal is to preserve the original payload exactly as received. Storing the raw input allows later investigation if discrepancies appear.
3. Parsed
The file or payload is read and converted into a structured format.
Parsing failures should stop processing immediately. Attempting to continue with partially parsed data introduces unpredictable behaviour later in the pipeline.
4. Validated
Validation occurs before any attempt is made to post data downstream.
Typical validation includes:
- required fields present
- numeric totals matching expectations
- date ranges valid
- duplicate detection
- business rule checks
Validation failures should move items into a review state rather than attempting automatic correction.
5. Matched
Entities are resolved against internal records.
Examples include:
- vendor matching
- personnel mapping
- account or cost centre resolution
- approver identification
Matching should not modify source data. Instead, it should produce a mapping layer that can be reviewed independently.
6. Queued for posting
Only fully validated and matched records are allowed to enter the posting queue.
This separation ensures that posting failures do not require re-running earlier stages.
7. Posted to destination system
The final stage performs the external API or database operation.
Success and failure responses must be recorded explicitly. Silent failures create the most difficult reconciliation problems.
Trade-offs
Breaking a pipeline into multiple stages increases implementation complexity and storage requirements. It may also introduce additional tables or status tracking.
However, a single-stage pipeline tends to fail in ways that are difficult to diagnose. When parsing, validation, and posting occur together, it becomes unclear which assumption failed.
The staged approach trades simplicity for observability and recovery.
Checks
After implementing or modifying a pipeline:
- confirm that failures stop progression to later stages
- verify that retries do not create duplicate records
- ensure raw input data can be inspected after processing
- confirm that operational users can see where items are stuck
- test failure scenarios intentionally, not only success paths
Pipelines that are only tested with valid data tend to fail unpredictably in production.
Notes
Most integration failures are not technical errors but data assumptions that prove incorrect over time. Systems evolve independently, and validation rules that were once sufficient become incomplete.
Designing for safe failure means accepting that incorrect data will arrive and ensuring the system responds predictably when it does.
A pipeline that fails early and visibly is usually more reliable than one that attempts to recover automatically.