A webhook can arrive twice without the sender having created two business operations. To handle that safely, separate receiving the message from carrying out its business effect. Give the event and the operation useful identities, record their progress, and decide how an interrupted attempt will be investigated before anyone replays it.
That distinction matters when an event creates an order, changes stock or asks another system to perform work. A receiver can recognise a duplicate message yet still be uncertain whether an earlier downstream action completed. The integration needs a decision for that uncertainty, not simply another attempt.
Decide which business effect must happen once
Start with the operation a member of staff would recognise. “Receive an order event” describes transport. “Create one fulfilment request for this accepted order” describes the outcome that must be protected. Write that outcome down before choosing a deduplication mechanism.
Twilio's event-delivery documentation describes retries and duplicate events for its Event Streams service. Its behaviour illustrates why a receiving application needs an explicit duplicate policy. The retry timing and acknowledgement rules of that service should not be assumed for a different provider.
Delivery, processing and business effects are different
Use separate questions in the requirements: Have we received this event? Have we finished interpreting it? Has the intended operation happened? A positive answer to the first question does not establish the third. Equally, a missing completion record does not prove the operation failed.
A useful business-effect identity might include the source order and the type of action. Two different events could concern that same action, while one event could legitimately initiate several distinct actions. The engineering design must express those relationships rather than treating every payload as an independent instruction.
Record the event before choosing the next action
A durable processing record gives the team somewhere to look after a worker stops or a connection fails. Its useful contents include the source, event identity, relevant operation identity, processing state and the evidence needed to find the downstream result. Avoid filling it with unnecessary customer details when a controlled reference will do.
Distinguish an event waiting for work from one being processed, completed or awaiting investigation. Those labels need agreed meanings. “Completed” should describe the promised operation, not just the fact that the receiver returned a successful HTTP response.
Check the scope of every identifier. Two connected providers or customer accounts may use values that look alike. The record should make the relevant source and business context unambiguous. Also ask whether the provider distinguishes an event identity from a delivery-attempt identity: recognising a new attempt does not necessarily mean there is a new operation to perform.
If a new endpoint is needed, agree its contract when designing the receiving API, including identity and error responses.
Handle two workers receiving the same event
Ask the implementation team how competing workers claim work. A design that checks for a record and creates it later can leave a gap between those decisions. Request evidence for concurrent arrivals and for a worker stopping after claiming an event. The appropriate coordination mechanism depends on the storage and execution model.
Deduplication also does not replace permission checks. OWASP's authorisation guidance supports checking authority at the appropriate boundary. A recognised event is not, by itself, permission to perform every action described in its contents.
Plan retries without hiding failed work
Group failures by the decision they require. A temporary connection fault might allow a later attempt. A missing required identifier might need correction. An unknown downstream result needs reconciliation. Giving all three situations the same retry policy makes the operation difficult to understand and support.
Record who controls the retry: the original provider, the receiving queue or a person using an operations tool. Otherwise several layers can repeat work independently. A provider retry and a manual replay should still encounter the same business-effect rules.
Agree how long identities and outcome evidence remain useful. Consider the provider's delivery behaviour, delayed messages and the business's own replay process. Choosing a retention period without those inputs can make an old event appear new after its protection record has disappeared.
The support view should make unresolved work visible. It should explain the last known state, the next permitted action and the responsible owner. A growing collection of failed events is not resolved merely because automatic retries eventually stop.
Review how operators will find evidence without exposing more data than they need. References to an order, task and processing attempt can be more useful than a large unfiltered payload. Agree who may inspect sensitive fields and which records can be retained or removed under the organisation's actual policy.
Use an event-and-effect replay matrix
The following cases are hypothetical requirements examples. They describe decisions to agree, not tests already performed or incidents from a customer project.
| Observed situation | Effect evidence | Decision to specify |
|---|---|---|
| The same event arrives again | The original operation is confirmed | Follow the provider's acknowledgement contract without repeating the effect |
| A worker stops after calling another system | The downstream outcome is unknown | Investigate that operation before issuing it again |
| An older update follows a newer one | A later business version is already applied | Use the agreed ordering or version rule |
| A new event concerns an existing operation | The business action is already complete | Apply the operation rule as well as event deduplication |
Imagine a fictional order event that requests a CRM follow-up task. The worker creates the task but loses its connection before recording completion. Marking the event failed and creating another task could duplicate the work. Marking it completed without looking could conceal a genuinely missing task.
The requirement is to find reliable evidence for that particular operation. If the CRM supports an appropriate operation reference or idempotent request contract, use its documented behaviour. If it does not, agree a controlled investigation route. Do not invent certainty from a timeout or promise that every external service offers the same recovery feature.
Test duplicate, delayed and out-of-order deliveries
Acceptance cases should follow observable business outcomes. Sending a duplicate event and seeing a successful response is insufficient if two tasks were created. Check the effect, processing record and any unresolved exception together.
- Deliver the same event while its first attempt is still active.
- Interrupt work before the effect, after the effect and before the completion record.
- Deliver a later update before an earlier update for the same resource.
- Replay an event through the operations route used by support staff.
- Try an event whose business reference is missing or no longer valid.
For each case, name the expected outcome and the evidence that will show it. Include who can authorise a manual replay and what that person must inspect. Use an approved test environment and fictional or appropriately controlled data rather than experimenting with live orders.
Include duplicate delivery and uncertain downstream outcomes in the agreed software testing coverage.
Agree ownership before connecting the systems
Bring the receiver owner, downstream system owner and operations team into the same review. Ask them to explain an ordinary delivery, a confirmed duplicate and an uncertain outcome. Differences in their answers identify requirements that remain unresolved.
A practical handover includes event and operation identities, acknowledgement expectations, retry ownership, replay authority, evidence locations and an escalation route. Keep those decisions with the integration documentation so a later provider change does not leave support relying on memory.
This is general guidance for custom business integrations. It is not documentation for a launched ZakCodeX customer webhook product. The useful next step is to scope the actual systems and effects that need protection.
Discuss the events and business effects that need protection in your API integration work with ZakCodeX.
