Concepts

Conduit moves events through three stages: a source accepts them, a pipeline transforms and batches them, and a sink writes them to a destination. This page explains that flow, the shape of an event, and what happens when delivery fails.

Data flow

Every event follows the same path. The pipeline stage applies its configured transforms in order, filters out events that a transform drops, and buffers the survivors until batch_size is reached or the flush interval elapses.

  Source            Pipeline                         Sink
  ------            --------                         ----
 +--------+       +------------------------+       +-----------+
 |  HTTP  | ----> |  transform -> filter   | ----> |  Postgres |
 |  Kafka |       |          -> batch      |       |    S3     |
 |  SQS   |       +-----------+------------+       |  Webhook  |
 +--------+                   |                    +-----------+
                              | write fails
                              v
                       +--------------+
                       | Dead-letter  |
                       |    queue     |
                       +--------------+

Event schema

Once accepted, every event is stored in a single canonical shape. The API assigns event_id and received_at; the rest comes from your request.

Field	Type	Description
`event_id`	string	Server-assigned unique identifier, for example `evt_5b1c9d`.
`pipeline_id`	string	Pipeline that processed the event.
`event_type`	string	Dot-namespaced event name, for example `order.created`.
`payload`	object	The JSON body you submitted, after transforms.
`received_at`	string	RFC 3339 timestamp of acceptance.

json

{
  "event_id": "evt_5b1c9d",
  "pipeline_id": "pl_8f3a",
  "event_type": "order.created",
  "payload": { "orderId": "ord_1042", "amount": 4999, "currency": "usd" },
  "received_at": "2026-06-21T14:09:33Z"
}

Important

Conduit deduplicates on idempotency_key, not on payload contents. Two requests with the same key within 24 hours resolve to one stored event; the second call returns the original event_id. Generate one key per logical event and reuse it on every retry.

Error handling

A sink can reject a write because it is unreachable, slow, or returns an error. Conduit treats these as transient and retries the batch according to the retry block.

Retry: a failed batch is re-attempted up to retry.max_attempts times (default 5), waiting between attempts per retry.backoff (default exponential: 1s, 2s, 4s, 8s, 16s).
Dead-letter queue: an event that exhausts all attempts is moved to the dead-letter queue rather than dropped. It is held for 7 days, where you can inspect and replay it.

Inspect the dead-letter queue for a pipeline and replay a held event:

bash

conduit dlq list --pipeline pl_8f3a
conduit dlq replay --pipeline pl_8f3a --event evt_5b1c9d

Warning

The encoded payload is capped at 256 KB. A larger event is rejected at ingest with 413 Payload Too Large and never enters the pipeline, so it cannot be recovered from the dead-letter queue. Split oversized records before sending.

Tip

When an event seems missing, tail the pipeline with conduit events tail --pipeline pl_8f3a to watch ingest in real time, then check conduit dlq list for anything that failed delivery.