Configuration Reference

A pipeline is declared in a pipeline.yaml file and applied with conduit pipelines create --file pipeline.yaml. This page documents every field, grouped by the top-level source, pipeline, sink, and retry blocks.

source

Where events enter the pipeline. The source block is required and accepts one inbound connector.

Field	Type	Default	Description
`source.type`	string	—	Required. Connector kind: `http`, `kafka`, or `sqs`.
`source.uri`	string	—	Required. Connection URI for the source, for example `https://ingest.conduit.dev/webhook`.

pipeline

How events are named, transformed, and batched before they reach the sink.

Field	Type	Default	Description
`pipeline.name`	string	—	Required. Human-readable label shown in the dashboard and API.
`pipeline.transforms`	list	`[]`	Ordered list of transform names applied to each event, in sequence.
`pipeline.batch_size`	int	`100`	Events buffered before a flush to the sink. Range 1–1000.

sink

Where processed events are written. The sink block is required and accepts one outbound connector.

Field	Type	Default	Description
`sink.type`	string	—	Required. Destination kind: `postgres`, `s3`, or `webhook`.
`sink.endpoint`	string	—	Required. Destination URL, for example `postgres://db.internal:5432/events`.

retry

How the pipeline handles a sink that rejects or times out a write. The retry block is optional; the defaults below apply when it is omitted.

Field	Type	Default	Description
`retry.max_attempts`	int	`5`	Number of delivery attempts before an event is sent to the dead-letter queue.
`retry.backoff`	string	`exponential`	Wait strategy between attempts: `exponential` or `fixed`.

Complete example

A full pipeline.yaml using every block. It reads HTTP webhooks, redacts and enriches each event, batches in groups of 100, writes to Postgres, and retries failed writes up to five times with exponential backoff.

yaml

# pipeline.yaml — orders ingestion
source:
  type: http
  uri: https://ingest.conduit.dev/webhook

pipeline:
  name: orders-stream
  transforms:
    - redact_pii
    - enrich_geo
  batch_size: 100

sink:
  type: postgres
  endpoint: postgres://db.internal:5432/events

retry:
  max_attempts: 5
  backoff: exponential

Tip

Validate a config without creating the pipeline using conduit pipelines validate --file pipeline.yaml. It reports unknown fields and type mismatches before you apply.

Next steps

Apply the file with the create command from the Quickstart.
See how retry interacts with the dead-letter queue in Concepts.