Retain events and recover deliveries

Copy Markdown View Source

Start with the resources, repository, migrations, and secret resolver in get started. This guide extends that application. Apply the migration before deploying the resource declarations. The application owns policies, schedules, and access to retained payloads.

Store events independently of deliveries

Add this resource and register it in MyApp.Domain. It records an event even when no endpoint is subscribed or enabled. It needs its own retention policy.

defmodule MyApp.OutboundEvent do
  use Ash.Resource,
    domain: MyApp.Domain,
    data_layer: AshPostgres.DataLayer,
    extensions: [AshHooks.OutboundEvent]

  postgres do
    table "outbound_events"
    repo MyApp.Repo
  end

  actions do
    defaults [:read]
  end

  attributes do
    timestamps()
  end
end

Add this section to MyApp.OutboundDelivery to record replay requests:

outbound_delivery do
  resend_tracking :on
end

Add these statements to a new Ecto.Migration's change/0:

create table(:outbound_events, primary_key: false) do
  add :id, :uuid, primary_key: true
  add :event_uuid, :text, null: false
  add :event_type, :text, null: false
  add :version, :integer
  add :payload, :binary, null: false
  timestamps(type: :utc_datetime_usec)
end

create unique_index(:outbound_events, [:event_uuid])

alter table(:outbound_deliveries) do
  add :resend_baseline, :integer, null: false, default: 0
  add :resend_count, :integer, null: false, default: 0
  add :last_resend_by, :text
  add :last_resend_at, :utc_datetime_usec
end

Add events MyApp.OutboundEvent to the outbound :order_paid declaration. The catalog guide shows a complete replacement webhooks section that includes it.

Preview and replay a retained event

After dispatching an event, select it by its original ID. This example uses event and endpoint from the dispatch example. Authorize the operator before calling these system helpers. requested_by is a durable audit identifier, not a credential.

{:ok, %{records: [logged]}} =
  AshHooks.OutboundEvent.list(MyApp.OutboundEvent, event_uuid: event.id)

replay_options = [
  event_uuid: logged.event_uuid,
  deliveries: MyApp.OutboundDelivery,
  endpoints: MyApp.WebhookEndpoint,
  declaration: {MyApp.Order, :order_paid},
  requested_by: "operator-recovery",
  limit: 100,
  per_second: 25
]

{:ok, %{count: 1}} =
  AshHooks.OutboundEvent.replay(
    MyApp.OutboundEvent,
    endpoint,
    Keyword.put(replay_options, :dry_run, true)
  )

{:ok, replay_counts} =
  AshHooks.OutboundEvent.replay(MyApp.OutboundEvent, endpoint, replay_options)

A new endpoint gets a new delivery preserving the event ID and bytes. An existing terminal delivery is rearmed; a pending or retrying delivery is skipped. Check created, re_sent, and skipped rather than treating a successful call as proof every event was delivered. An unknown event returns :event_not_found; missing or mismatched declarations fail before replay.

For a queued application, pass the same enqueue: {MyApp.WebhookDeliveryWorker, :enqueue} callback used by dispatch. Without it, rows remain pending for the direct runner. Anonymous queue callbacks require a stable enqueue_key for scheduled recovery. Schedule AshHooks.reconcile_pending/3 for each declaration, queue route, and tenant. It recovers persistence/enqueue gaps and expired leases.

Filter bulk replay with event_type: "order_paid" and explicit from: / to: bounds instead of event_uuid:. Each call selects a bounded batch; replay has no continuation cursor. Advance the time range or select IDs to progress past skipped rows. Receivers must deduplicate the preserved webhook-id because network delivery remains at least once.

Own retention and tenant boundaries

Schedule event pruning independently of delivery and attempt retention:

cutoff = DateTime.add(DateTime.utc_now(), -30 * 86_400, :second)
{:ok, deleted_events} = AshHooks.OutboundEvent.prune(MyApp.OutboundEvent, older_than: cutoff)

A log write failure emits [:ash_hooks, :event, :record_failed] and does not undo delivery creation. Alert on that event and repair storage before relying on the log for recovery. List and prune return storage errors to the caller. Protect the injected :record and :prune actions with application policies.

For a multitenant application, declare matching attribute tenancy on the emitter, endpoints, subscriptions, deliveries, attempts, and event log. Add the tenant column to each migration and tenant to every unique index. Pass the same tenant: to dispatch, list, replay, pruning, workers, and reconciliation. Missing tenants fail before data access; never run a global recovery query for tenants.