The Outbox Pattern and Change Data Capture: Solving the Dual-Write Problem with Debezium
Saving an order and announcing it to the rest of your system sounds like one job. In reality, it is two jobs on two systems, and that gap is where data quietly goes missing.
The dual-write problem in plain words
Imagine an order service. A customer clicks “buy”, so the service writes a row to its database. Then, right after that, it publishes an OrderCreated event so that shipping, billing, and search can react.
The database and the message broker are separate systems. Consequently, they cannot share a single transaction without heavy coordination. As the transactional outbox pattern page explains, sending a message in the middle of a transaction is unreliable unless you adopt two-phase commit.
Both orderings fail, just differently
You might think that choosing the right order of operations solves it. Unfortunately, it only changes the symptom. Table 1 shows what happens when the process crashes at the risky moment.
| Approach | Crash happens… | Result |
|---|---|---|
| Write to the database, then publish | after the commit, before the publish | Order exists, event is lost |
| Publish, then write to the database | after the publish, before the commit | Event exists for an order that does not |
| Outbox, then relay | after the publish, before the relay marks it done | Event may be sent twice, never lost |
Table 1. What a crash at the risky moment does to each approach.
Retrying does not rescue you either. After a crash, the process no longer remembers that it still owed the world an event. Therefore, the intent has to live somewhere durable, and the database is the obvious place.
The outbox pattern: one transaction, two rows
The idea is refreshingly simple. Instead of publishing directly, the service writes the event into an outbox table. It does so in the same transaction as the business change. Either both rows commit, or neither does.
A separate component, the message relay, then reads the outbox and forwards events to the broker. In Chris Richardson’s terms, the pattern has three participants: the database, the message outbox, and the message relay.
A tiny example you can run
The snippet below creates an orders table and an outbox table, then inserts both rows atomically. The outbox columns match the defaults that the Debezium Outbox Event Router expects. I ran it on PostgreSQL 16, and it works on any PostgreSQL 13 or newer database, because gen_random_uuid() is built in from that version.
psql -d demo2 <<'SQL'
CREATE TABLE orders (
id bigserial PRIMARY KEY,
customer_id int NOT NULL,
total numeric(10,2) NOT NULL
);
CREATE TABLE outbox (
id uuid PRIMARY KEY,
aggregatetype varchar(255) NOT NULL,
aggregateid varchar(255) NOT NULL,
type varchar(255) NOT NULL,
payload jsonb
);
BEGIN;
WITH o AS (
INSERT INTO orders (customer_id, total)
VALUES (123, 69.97)
RETURNING id, customer_id, total
)
INSERT INTO outbox
SELECT gen_random_uuid(), 'order', o.id::text, 'OrderCreated',
jsonb_build_object('id', o.id,
'customerId', o.customer_id,
'total', o.total)
FROM o;
COMMIT;
SELECT aggregatetype, aggregateid, type, payload FROM outbox;
SQL
Replace demo2 with any database you have created. The final query should return one OrderCreated row whose aggregateid matches the new order.
Where Debezium and change data capture come in
Now the relay needs a way to find new outbox rows. According to the same pattern page, there are two common options: a polling publisher, or transaction log tailing. Change data capture (CDC) is the second one.
PostgreSQL stores every committed change in its write-ahead log. Moreover, logical decoding can turn that log into a readable stream of row changes. Debezium taps into this stream, so your application never has to poll anything.
Polling versus log tailing
Both relays work, but they trade different things. For instance, a polling publisher is easy to build yet adds delay and query load. A log-based relay reacts faster, although it adds infrastructure.
| Aspect | Polling publisher | Log tailing with Debezium |
|---|---|---|
| How it finds events | Queries the outbox table on a timer | Reads the database log |
| Added delay | About half the poll interval on average | Follows the log, no timer |
| Extra database queries | Yes, repeated | No polling queries |
| Moving parts | Your own scheduler | Kafka Connect and a connector |
Table 2. Two ways to build the message relay.
What the Outbox Event Router does
Debezium alone would emit raw table changes. That is not what consumers want. The Outbox Event Router, a single message transformation, reshapes each outbox row into a clean business event.
By default, the router sends the event to a topic named outbox.event. followed by the aggregatetype value. It uses aggregateid as the message key, which keeps events for one order in order within a Kafka partition. In addition, it places the event id in a header, so consumers can drop duplicates.
The documented configuration is short. Add these lines to a Debezium connector, and set table.include.list so that it captures only the outbox table:
transforms=outbox transforms.outbox.type=io.debezium.transforms.outbox.EventRouter transforms.outbox.table.expand.json.payload=true value.converter=org.apache.kafka.connect.json.JsonConverter
These lines come straight from the Debezium documentation. The expand.json.payload option turns the escaped JSON string into real JSON, which is far easier to read. For a complete setup that you can run end to end, Debezium maintains an outbox demo in its examples repository.
What the pattern really costs
Nothing is free, so let us measure. First, the safety gain. I ran a small failure-injection simulation with 100,000 orders and a 1% crash chance at the risky moment.
As Figure 2 shows, both direct approaches lose consistency at roughly the injected rate. The outbox never loses an event. Instead, it trades loss for duplicates, and duplicates are the easier problem. Debezium itself provides at-least-once delivery, so consumers should be ready for repeats anyway. Fortunately, the event id makes deduplication straightforward.
The write overhead is small
Second, the price. The outbox adds one more insert to every transaction. To see how much, I used pgbench on PostgreSQL 16 with one client, comparing an order-only insert against an order plus outbox insert.
The median rose from 0.350 ms to 0.433 ms per transaction, which is about 0.08 ms. That is a real cost, but a modest one next to a network call to a broker. Still, measure it on your own hardware before you trust any single number.
Mind the replication slot
There is one operational trap. A PostgreSQL replication slot keeps the log that Debezium has not yet confirmed. According to the Debezium PostgreSQL connector docs, slots retain all the log Debezium needs, even during outages, so you must monitor them to avoid filling the disk. Therefore, alert on slot lag, and drop the slot if you delete a connector for good.
Finally, the outbox table itself should not grow forever. The Debezium router expects only inserts, and it automatically filters out delete operations. As a result, you can clean old rows without producing extra events.
Outbox checklist
- Write the business row and the outbox row in the same transaction.
- Capture only the outbox table with the connector.
- Use a stable event
idand deduplicate on the consumer side. - Use
aggregateidas the key so related events stay ordered. - Monitor the replication slot and clean old outbox rows.
What we have learned
The dual-write problem appears whenever one service must update its database and notify others, because no single transaction spans both systems. Publishing first or committing first only changes how it fails, as our simulation showed with about 1,000 lost or phantom events per 100,000 orders. The outbox pattern fixes this by writing the event into the same transaction as the business change, so the intent is never lost. Debezium then reads the database log and the Outbox Event Router turns each row into a clean event, without polling. In return, you accept at-least-once delivery, so consumers must deduplicate, and you must watch the replication slot. In our test, the extra insert added only about 0.08 ms per transaction, a small price for reliable events.
