AshHooks.Delivery (AshHooks v3.3.0)

Copy Markdown View Source

Drives one durable outbound delivery through signing, HTTP, and result persistence. It uses your resource modules and configured HTTP adapter. AshHooks.Worker adds an Oban trigger; this driver also works without Oban.

The delivery row owns retry policy: the driver refuses to re-send :succeeded/:dead_letter rows, waits on next_attempt_at, counts attempts against the ceiling, and honors Retry-After (bounded). Oban is the durable trigger; {:snooze, s} re-drives later without exhausting the job (snooze extends max_attempts; only this module's ceiling decides dead-letter).

Every attempt runs in an owned monitored process under one finite deadline. The budget includes endpoint reads, secret resolution, signing, destination validation, DNS/connect/HTTP work, response handling, and the result write. The durable send lease is derived from that deadline plus a bounded finalization allowance, and every result write is fenced by the row's full primary key, source, attempt token, sending state, and live lease.

Lease construction and comparisons use the configured application clock (:now) or DateTime.utc_now/0. Deploy executors with synchronized clocks; clock skew can shorten or extend admission windows, so this module does not claim a database-clock distributed timing guarantee.

Response classification:

  • 2xx → :succeeded (status + allowlisted content-type summary, with no body bytes by default; a per-call snippet_capture: true config persists a bounded, redacted body marked [captured])
  • 408/429/5xx → retryable, Retry-After when present (integer seconds or HTTP-date; clamped [1, retry_after_cap_seconds]), else backoff
  • 410 → :disable_pending, then conditional endpoint disable and :dead_letter
  • other 4xx → :dead_letter (client errors do not self-heal)
  • 3xx → :dead_letter (redirect_refused_<status>, such as redirect_refused_302; redirects are never followed)
  • transport error / byte-secret-resolution failure / unavailable custody → retryable backoff
  • KeyKindMismatch, UnknownKey after the one-time owner refresh, or :kind_not_allowed → :dead_letter (permanent key configuration failures)
  • a transport resolver that raises, throws, exits, or returns an invalid value refuses the attempt and retries under transport_resolution; nil keeps the HTTP adapter
  • unresolvable DNS at the send-time destination check → retryable backoff under nxdomain (every family answered authoritatively negative) or resolve_timeout (resolver outage, congestion, mixed evidence). These failures are transport errors, not a verdict on the destination; a permanently dead domain dead-letters at the row ceiling carrying its label. The HTTP adapter's umbrella for the same failures is unresolved_host
  • send-time SSRF refusal / disabled-or-gone endpoint → :dead_letter
  • a configured :destination_policy decides ahead of the packaged SSRF gate (AshHooks.DestinationPolicy): :deny → :dead_letter (destination_denied), :allow → the send proceeds without the packaged gate, :default → the gate runs unchanged; a contained callback failure retries under destination_policy_failed

Backoff: let step = min(base * 2^min(attempts, 16), max_backoff). Add a random value from zero through step - 1, then apply the maximum again. The delay is at least one second.

The 410 auto-disable is a durable two-step system write. A live attempt first stores :disable_pending with the endpoint configuration snapshot; recovery then conditionally disables that same configuration and finalizes the row. A changed endpoint is never disabled by an older 410. One 410 disables the matching endpoint as a system action (authorize?: false). The write has no application actor. Applications that keep an audit history must attribute this system change themselves. Without application notifications, delivery stops silently from the tenant's perspective. Attach [:ash_hooks, :delivery, :disable] to your operator or notification surface; it carries the tenant and endpoint IDs. The endpoint's status or mapped switch also exposes the durable disabled state.

Summary

Functions

Lists delivery rows by filter with keyset pagination (newest first), providing the delivery-log view behind endpoint pages and resend UIs.

Retention hook: deletes TERMINAL delivery rows (:succeeded, :dead_letter) older than older_than, by the resource's inserted_at (add Ash timestamps() to the resource and its migration). Non-terminal rows are never deleted. Returns {:ok, deleted_count}, or {:error, error} when the resource lacks inserted_at, following the same error contract as AshHooks.Ingress.prune/2.

The built-in response-body redaction for opt-in capture.

Re-arms matching terminal delivery rows through resend/2.

Re-arms one TERMINAL delivery row (:succeeded / :dead_letter) for a resend: the row keeps its id and event_uuid (a Standard Webhooks receiver deduplicates the repeat POST by webhook-id), its lifetime attempts counter keeps climbing (attempt-history rows stay unique), and a fresh retry ladder starts at the current count (resend_baseline).

Drives one delivery from the worker's JSON-safe args. Current jobs carry delivery_pk, delivery_resource, endpoint_resource, endpoint_id, event_uuid, dispatch_source, dispatch_route, and an optional tenant. String keys are Oban's JSON round-trip shape; atom keys are tolerated. Legacy jobs carrying only endpoint_id and event_uuid remain readable. The tenant arg is the enqueue seam's serialized row tenant (the worker applies tenant_from_attribute to the stored attribute); run/2 re-parses it through the resource's tenancy pipeline by passing it as the tenant on every data call; a worker restart or replica handoff recovers full tenant context from args alone.

The default response summary: a fixed grammar over the status and one ALLOWLISTED content-type token, never body bytes or a body-derived digest (a hash is correlation material that explains nothing).

Sends one signed synthetic TEST event of a chosen type to ONE endpoint, OUTSIDE subscriptions; no subscription row is consulted or needed, and the delivery lands in your ledger like any other (the delivery log is the record).

Functions

list(deliv_mod, opts \\ [])

@spec list(module(), keyword()) ::
  {:ok, %{records: [struct()], next_cursor: binary() | nil}} | {:error, term()}

Lists delivery rows by filter with keyset pagination (newest first), providing the delivery-log view behind endpoint pages and resend UIs.

Options:

  • :endpoint_id: one endpoint's deliveries.
  • :status: a list of statuses (e.g. [:failed_retryable, :dead_letter]).
  • :event_type: one event type's deliveries.
  • :test: true/false on test_tracking :on resources: filter the delivery log to (or exclude) test sends.
  • :from / :to: DateTime bounds on inserted_at (inclusive).
  • :limit: page size (default 50, max 500; anything else is {:error, :invalid_limit}).
  • :after: the cursor from a previous page's :next_cursor (a malformed cursor is {:error, :invalid_cursor}).
  • :tenant: required on multitenant resources (the named {:error, :tenant_required} pre-flight otherwise).

The resource needs inserted_at (add timestamps()) and an :id attribute (the keyset tiebreak); otherwise the call fails with the named error before any data access.

Returns {:ok, %{records: [row], next_cursor: binary | nil}}.

prune(deliv_mod, opts)

@spec prune(module(), keyword()) :: {:ok, non_neg_integer()} | {:error, term()}

Retention hook: deletes TERMINAL delivery rows (:succeeded, :dead_letter) older than older_than, by the resource's inserted_at (add Ash timestamps() to the resource and its migration). Non-terminal rows are never deleted. Returns {:ok, deleted_count}, or {:error, error} when the resource lacks inserted_at, following the same error contract as AshHooks.Ingress.prune/2.

:tenant scopes the sweep (required on multitenant resources; a tenant-less global sweep over them is the named {:error, :tenant_required} before any data access). :batch_size defaults to 500 and accepts integers from 1 through 5,000; invalid values return {:error, :invalid_batch_size}.

With :attempts (an AshHooks.OutboundAttempt resource), each batch deletes deliveries and their history in one transaction. A history-deletion error is returned and rolls back the batch, retaining both the delivery and its history. Both resources must support transactions; otherwise this returns {:error, :transactions_not_supported} before deleting anything. For SQLite, enable write_transactions?/0 on the repo.

redact(body)

@spec redact(term()) :: String.t() | nil

The built-in response-body redaction for opt-in capture.

Normalizes Unicode, decodes encoded text, removes recognized secret patterns and control characters, and limits the result to 2,048 bytes. Invalid UTF-8 returns [binary]. Applications must add rules for sensitive values the built-in patterns do not recognize.

replay(deliv_mod, opts \\ [])

@spec replay(module(), keyword()) ::
  {:ok, %{resent: non_neg_integer(), skipped: non_neg_integer()}}
  | {:ok, %{count: non_neg_integer()}}
  | {:error, term()}

Re-arms matching terminal delivery rows through resend/2.

Filters: :endpoint_id, :event_type, required :status list, and :from / :to DateTime bounds on inserted_at. The resource needs that timestamp attribute.

Options: required :endpoints and :requested_by, plus :enqueue, :tenant, :now, :dry_run, :limit (default 100, maximum 1,000), and :per_second (default 50). Dry run counts the selected batch after applying :limit, returning {:ok, %{count: n}} without changing rows.

A mutating call sleeps between second-sized batches to limit re-arming. Gone or disabled endpoints are skipped: {:ok, %{resent: n, skipped: n}}. Repeated calls can select skipped rows again; this API returns no pagination cursor.

resend(row, opts \\ [])

@spec resend(struct(), keyword()) ::
  {:ok,
   %{
     optional(:enqueue) => {:error, term()},
     status: :resent,
     delivery: struct()
   }}
  | {:error, term()}

Re-arms one TERMINAL delivery row (:succeeded / :dead_letter) for a resend: the row keeps its id and event_uuid (a Standard Webhooks receiver deduplicates the repeat POST by webhook-id), its lifetime attempts counter keeps climbing (attempt-history rows stay unique), and a fresh retry ladder starts at the current count (resend_baseline).

Options:

  • :endpoints (required): the endpoint resource, for the live check.
  • :requested_by (required, nonempty binary ≤ 255): recorded on the row (last_resend_by/last_resend_at, resend_count bumps).
  • :enqueue: the same enqueue seam dispatch uses ({module, function} or a 2-arity fn); nil re-arms the row and leaves driving to your worker/recovery.
  • :tenant: the tenancy pre-flight applies.
  • :now: the application clock for last_resend_at.

Only live endpoints resend: gone or disabled endpoints return {:error, :endpoint_not_live} before any write; a non-terminal row is {:error, :not_resendable}; a resource without resend_tracking :on fails with a named error (the columns are its migration). Every resend emits [:ash_hooks, :delivery, :resend] carrying the ids and requested_by. Returns {:ok, %{status: :resent, delivery: row}}.

run(args, config)

@spec run(map(), keyword()) :: :ok | {:snooze, pos_integer()} | {:error, term()}

Drives one delivery from the worker's JSON-safe args. Current jobs carry delivery_pk, delivery_resource, endpoint_resource, endpoint_id, event_uuid, dispatch_source, dispatch_route, and an optional tenant. String keys are Oban's JSON round-trip shape; atom keys are tolerated. Legacy jobs carrying only endpoint_id and event_uuid remain readable. The tenant arg is the enqueue seam's serialized row tenant (the worker applies tenant_from_attribute to the stored attribute); run/2 re-parses it through the resource's tenancy pipeline by passing it as the tenant on every data call; a worker restart or replica handoff recovers full tenant context from args alone.

Returns :ok (terminal or attempted-to-terminal), {:snooze, seconds} (retry later), or {:error, term} for a broken trigger (row missing → :ok; the durable row is the record; a missing row is a completed or reaped delivery, not a failure). A tenant-less trigger against multitenant resources is {:error, :tenant_required} before any data access. A trigger whose resource, source, route, or complete primary key does not match the fetched row fails explicitly.

An enclosing transaction on the delivery resource returns {:error, :inside_transaction} before starting a process or accessing the database. Enqueue a job for execution after the host transaction commits.

Required config keys are :deliveries and :endpoints. Byte slots need :secret_resolver here or on the endpoint resource; custody slots read the endpoint's custodian and kind markers. :attempt_timeout defaults to 25,000 ms and :finalization_allowance to 5,000 ms. Retry defaults are 10 attempts, 2-second base backoff, 3,600-second maximum backoff, and an 86,400-second Retry-After cap. Omitted or nil retry options use these defaults. :http, :http_opts, :tenant_aware_secrets, :snippet_capture, :snippet_redactor, :transport, :destination_policy ({module, function} or a 2-arity fn, as described in AshHooks.DestinationPolicy), and :now customize delivery. :transport accepts {module, function} or a two-argument function receiving the endpoint and tenant. Return an AshHooks.Transport module or nil for the HTTP adapter.

Optional :retryable_statuses replaces the retry set with a list of HTTP status integers/ranges. nil preserves the default; [] retries no ordinary HTTP failures. 2xx success, redirect refusal, and 410 disablement remain fixed. Invalid sets return {:error, :invalid_retryable_statuses} before any claim.

:delivery_headers is a map/list, {module, function}, or three-argument function (delivery, endpoint, tenant) -> headers. Package signature, content-type, and framing headers are reserved case-insensitively. Invalid headers or callback failure retry without sending.

:certificate_pin is a pin or (endpoint, tenant) resolver ({module, function} or function). It overrides http_opts[:certificate_pin], including a nil result. Pins require AshHooks.Http.Bounded and HTTPS; resolver/config failures and mismatches are non-retryable. See AshHooks.Http.Bounded for pin formats. No configuration is cached in process-local state.

Set :max_attempts here to configure the delivery row's ceiling. The worker macro calls this option :delivery_max_attempts; its own :max_attempts configures Oban's job attempts. Other retry option names are the same.

summarize(status, headers)

@spec summarize(integer() | nil, keyword() | list() | nil) :: String.t()

The default response summary: a fixed grammar over the status and one ALLOWLISTED content-type token, never body bytes or a body-derived digest (a hash is correlation material that explains nothing).

# the type was allowlisted
"200 json token=application/json"
# anything else collapses to other
"200 text token=other"

The status is an integer, the kind comes from a fixed vocabulary (json | html | text | xml | binary | other), and the token is either an exact allowlist member or the literal other; a hostile Content-Type header cannot smuggle material into the ledger through this string.

test_send(endpoint, opts \\ [])

@spec test_send(struct(), keyword()) ::
  {:ok, %{status: atom(), delivery: struct()}} | {:error, term()}

Sends one signed synthetic TEST event of a chosen type to ONE endpoint, OUTSIDE subscriptions; no subscription row is consulted or needed, and the delivery lands in your ledger like any other (the delivery log is the record).

The event gets a fresh msg_ id (every test send is a new event, with no dedup suppression), the chosen type verbatim, and the :signing_mode option, defaulting to :standard. The default payload marks itself:

{"test": true, "type": "invoice.paid", "timestamp": "2026-11-14T12:00:00Z"}

A caller-supplied :payload (exact bytes) is used VERBATIM; wrap your own test marker into it; the package never rewrites caller bytes.

No subscription signing mode is inferred. :signing_mode defaults to :standard.

Returns {:error, :inside_transaction} before starting a process or accessing the database if the delivery resource has an enclosing transaction.

Options: :type (required, atom or nonempty binary), :payload, :signing_mode, :deliveries (the delivery resource, required), :secret_resolver (the byte-slot driver contract, optional for custody endpoints), :max_attempts (default 1, so a test send reports one attempt's outcome; a higher value permits later retries through an explicit run/2 call, while test_send/2 itself invokes the runner once), plus the driver passthroughs (:http, :http_opts, :destination_policy, :ssrf_check, :tenant_aware_secrets, :attempts, :tenant, :now). The endpoint must be live ({:error, :endpoint_not_live} otherwise); the tenancy pre-flight applies across the delivery and endpoint resources.

Emits [:ash_hooks, :delivery, :test] (%{endpoint_id, type}) beside the normal attempt/result lifecycle, then drives the row synchronously: {:ok, %{status: final_status, delivery: row}} (the row is the record) or {:error, reason} for pre-send failures.