AshHooks.Ingress (AshHooks v3.3.0)

Copy Markdown View Source

The inbound pipeline driver: verify → ingest → claim → handle → mark, over a ledger resource carrying the AshHooks.InboundDelivery extension.

AshHooks.Plug is the HTTP entry point. ingest/4 is the lower-level sync entry point for custom transports and pipelines, called with the exact raw request body. Capture it pre-parser via AshHooks.BodyReader, or read it before any parser consumes it; parsed params cannot recover those bytes. Request context carries the signature, request headers, public request method/URI, scope, connection, and tenant. It does not accept or forward a request-controlled clock; replay-window verification uses the runtime system clock. The machine's crash-safety contract:

  • the decoded payload persists before handling; payload_digest binds it to the signed raw bytes;
  • a verified body that cannot decode to a map or list is stored by :record_malformed once its event identity resolves, with placeholder payload %{}, status :failed_permanent, and error class "malformed_payload". A provider identity callback that rejects the placeholder refuses the request before :record_malformed runs (with ingress_tracking :on, ingress attempts to record the refusal as a rejection row). A newly created malformed row is never claimed or handled; its attempts and fencing token stay 0. A duplicate still drives the surviving row's stored payload;
  • the unique index preserves one row per provider/event/scope identity. With the injected writable UUID key, the surviving key distinguishes :created from :duplicate. Custom generated keys use a pre-read; concurrent first arrivals may both be reported as :created, while the stored row and claim remain deduplicated;
  • a claim is a WHERE-gated update (status == :received, a retryable failure, or an expired lease) that bumps the monotonic fencing token, allowing redelivery to recover a stranded row;
  • marks are gated on the caller's token and an unexpired lease; a stale owner (superseded or expired) is rejected.

Returns {:ok, :created | :duplicate, delivery}; the delivery's status carries the outcome (:processed, :failed_retryable, :failed_permanent, or retained :superseded). Returned retry and permanent failures are recorded. Verification and configuration failures do not run the handler. With ingress_tracking :on, ingress may attempt a rejection-row write before returning the error. Paused sources, unavailable custody keys, and unrecognized callback failures do not create rejection rows or increase the failed-verification count.

The individual machine steps (ingest_delivery/2, claim_delivery/3, mark_processed/4, mark_failed/6, renew/4, reap/2) are public for composition and monitoring; a consumer-driven async pipeline (your own 202-then-claim flow) and the reaper drive the same steps. Each accepts the tenancy options; the arities from before 1.2.0 (one fewer argument) keep working for single-tenant ledgers.

Internal ledger operations use authorize?: false; callers must restrict access to these machine functions. Signature verification gates the sync ingest path. Your domain policies govern application read access to decoded provider payloads and signed-body digests. See the README's retention and security section. The returned row can remain :received or :claimed when this call did not obtain ownership. Inspect its status before choosing an HTTP acknowledgment. Handler and claimed-receipt parser exceptions, exits, and throws are recorded as :failed_retryable with error class "handler_crashed", using the current claim's fencing token. Both synchronous ingestion and reaping use this path. Any handler result outside {:ok, value} or a retry/permanent error triple, and any unexpected parser result, records "invalid_handler_result" as a retryable failure. Each recorded failure emits the ingress failure event; a stale owner reloads without recording or emitting a failure.

Summary

Types

The request context. :signature is the signature header value for providers whose signature_source/0 is :header (the default); :headers carries the request headers, preferably as the ordered {name, value} list Plug provides in conn.req_headers, or as a map; :method and :request_uri must reproduce the public request values covered by the provider signature; :scope carries values for the ledger's declared scope_identity slots; :connection is the per-connection provider's secret source argument; and :tenant is required on multitenant ledgers. Ingress constructs the provider verification context from these known keys and never forwards an input :now_ms clock override. Optional :host_context is passed only to verification, without persistence or inspection. The provider remains responsible for binding any upstream proof to this body, tenant and source.

Functions

Applies one reviewed, quiesced provider-identity adoption partition.

Claims a delivery: WHERE-gated on status == :received, a re-driveable failure, or an EXPIRED lease. Bumps the fencing token atomically and takes a fresh lease. A live foreign lease (or a terminal row) returns {:error, :lease_held}. :tenant scopes the claim (required on multitenant ledgers).

Counts the inbound story over a window (:from / :to on inserted_at, inclusive; :source optional; :tenant applies)

Drives one inbound delivery through the full sync pipeline. The tenant rides the ctx (ctx[:tenant]) and threads every ledger call; the dedup identity, the claim fence, and every mark are per-tenant.

Persists the decoded payload and signed-body digest before handling via the no-touch unique upsert. Returns {:ok, created?, row} or {:error, reason}. For a writable :id primary key, compares the surviving ID with a fresh client-generated ID. Other primary keys use a preliminary identity read; concurrent inserts or rows hidden by a read filter can be reported as created even when the database upsert deduplicates them. The env carries the resolved tenant (env[:tenant]). env[:decode_result] is :ok for decoded payloads (the default when omitted), or {:error, :malformed_payload} to record a terminal malformed row via :record_malformed. Both actions preserve the surviving row on conflict and share duplicate classification and telemetry.

Lists inbound deliveries by filter with keyset pagination (newest first), providing the inbound delivery log.

Marks a claimed delivery failed, using the same token/lease gate as mark_processed/4. permanent? selects :failed_permanent over :failed_retryable.

Marks a claimed delivery processed, gated on the caller's token under an unexpired lease. :tenant scopes the mark (required on multitenant ledgers).

Builds the legacy provider-identity adoption audit without writing rows.

Retention hook: deletes one bounded batch of TERMINAL ledger rows (:processed, :failed_permanent, :superseded) older than older_than, by the resource's inserted_at (add Ash timestamps() to the resource and its migration). Non-terminal rows are never deleted; retryable and lease-held deliveries keep their dedup identity and re-drive path.

Re-drives deliveries whose claims died with an expired lease (crash between claim and mark). Returns the number re-driven; unexpired leases and terminal rows are left alone. :tenant scopes the sweep; a tenant-less global reap over a multitenant ledger is the named {:error, :tenant_required} (the unscoped read would otherwise fail); per-tenant sweeps compose with AshHooks.reap_all/2.

Retention field-redaction hook: replaces the stored payload of a CLAIMED delivery with redactor.(payload) under the caller's token and an unexpired lease (the mark_processed/3 fence). The row keeps its dedup identity; only the payload changes. payload_digest is deliberately NOT updated (it binds the ORIGINAL signed bytes for the audit trail).

Extends the caller's lease, gated on the caller's token under an unexpired lease.

Resolves one inbound declaration, casting and constraining scope before a source lookup. A source row is read in a tenant phase; an absent row returns UnknownSource. Declarations without source_record perform no source read. Options: :tenant, :scope. The returned struct can be passed as :resolved_source to ingest, as the HTTP entry point does.

Types

ctx()

@type ctx() :: %{
  optional(:signature) => String.t(),
  optional(:headers) =>
    [{String.t(), String.t()}] | %{optional(String.t()) => String.t()},
  optional(:method) => String.t() | nil,
  optional(:request_uri) => String.t() | nil,
  optional(:connection) => term(),
  optional(:host_context) => term(),
  optional(:scope) => %{optional(atom() | String.t()) => term()},
  optional(:tenant) => term()
}

The request context. :signature is the signature header value for providers whose signature_source/0 is :header (the default); :headers carries the request headers, preferably as the ordered {name, value} list Plug provides in conn.req_headers, or as a map; :method and :request_uri must reproduce the public request values covered by the provider signature; :scope carries values for the ledger's declared scope_identity slots; :connection is the per-connection provider's secret source argument; and :tenant is required on multitenant ledgers. Ingress constructs the provider verification context from these known keys and never forwards an input :now_ms clock override. Optional :host_context is passed only to verification, without persistence or inspection. The provider remains responsible for binding any upstream proof to this body, tenant and source.

Functions

adopt_legacy_identity(resource, name, opts \\ [])

@spec adopt_legacy_identity(module(), atom(), keyword()) ::
  {:ok, AshHooks.InboundDelivery.LegacyAdoption.plan()} | {:error, term()}

Applies one reviewed, quiesced provider-identity adoption partition.

claim_delivery(resource, delivery_id, opts \\ [])

@spec claim_delivery(module(), term(), keyword()) ::
  {:ok, non_neg_integer(), struct()} | {:error, :lease_held | term()}

Claims a delivery: WHERE-gated on status == :received, a re-driveable failure, or an EXPIRED lease. Bumps the fencing token atomically and takes a fresh lease. A live foreign lease (or a terminal row) returns {:error, :lease_held}. :tenant scopes the claim (required on multitenant ledgers).

counts(resource, opts \\ [])

@spec counts(module(), keyword()) ::
  {:ok,
   %{
     received: non_neg_integer(),
     processed: non_neg_integer(),
     duplicates: non_neg_integer(),
     failed_verification: non_neg_integer(),
     handler_failures: non_neg_integer()
   }}
  | {:error, term()}

Counts the inbound story over a window (:from / :to on inserted_at, inclusive; :source optional; :tenant applies):

{:ok,
 %{received: 120, processed: 100, duplicates: 18, failed_verification: 2, handler_failures: 3}} =
  AshHooks.Ingress.counts(MyApp.Ledger, from: an_hour_ago)

received counts the window's accepted rows (rejection rows are excluded because a refused request was never received); processed and handler_failures (:failed_retryable + :failed_permanent) come off the statuses. duplicates and failed_verification require ingress_tracking :on on the ledger (the duplicates counter column and the payload-less :failed_verification rejection rows); without them both report 0. Paused sources, unavailable custody keys, and unrecognized callback failures are excluded from rejection rows and their count. Window counts run as data-layer count and sum aggregates, without loading ledger rows or payloads into application memory. A storage read failure returns {:error, reason}; zero counts mean the read succeeded and found no matching observations.

ingest(resource, name, raw_body, ctx)

@spec ingest(module(), atom(), binary() | nil, ctx()) ::
  {:ok, :created | :duplicate, struct()} | {:error, term()}

Drives one inbound delivery through the full sync pipeline. The tenant rides the ctx (ctx[:tenant]) and threads every ledger call; the dedup identity, the claim fence, and every mark are per-tenant.

ingest_delivery(resource, env)

@spec ingest_delivery(module(), map()) ::
  {:ok, boolean(), struct()} | {:error, term()}

Persists the decoded payload and signed-body digest before handling via the no-touch unique upsert. Returns {:ok, created?, row} or {:error, reason}. For a writable :id primary key, compares the surviving ID with a fresh client-generated ID. Other primary keys use a preliminary identity read; concurrent inserts or rows hidden by a read filter can be reported as created even when the database upsert deduplicates them. The env carries the resolved tenant (env[:tenant]). env[:decode_result] is :ok for decoded payloads (the default when omitted), or {:error, :malformed_payload} to record a terminal malformed row via :record_malformed. Both actions preserve the surviving row on conflict and share duplicate classification and telemetry.

list(resource, opts \\ [])

@spec list(module(), keyword()) ::
  {:ok, %{records: [struct()], next_cursor: binary() | nil}} | {:error, term()}

Lists inbound deliveries by filter with keyset pagination (newest first), providing the inbound delivery log.

Options: :source (the inbound declaration's name, e.g. :orders), :status (a list of AshHooks.InboundDelivery.statuses/0), :event_type, :from / :to (DateTime bounds on inserted_at), :limit (default 50, max 500), :after (the previous page's cursor), and :tenant. The resource needs timestamps() and an :id. Returns {:ok, %{records:, next_cursor:}}.

mark_failed(resource, delivery_id, token, error_class, permanent?, opts \\ [])

@spec mark_failed(
  module(),
  term(),
  non_neg_integer(),
  String.t(),
  boolean(),
  keyword()
) ::
  :ok | {:error, :stale_token | term()}

Marks a claimed delivery failed, using the same token/lease gate as mark_processed/4. permanent? selects :failed_permanent over :failed_retryable.

mark_processed(resource, delivery_id, token, opts \\ [])

@spec mark_processed(module(), term(), non_neg_integer(), keyword()) ::
  :ok | {:error, :stale_token | term()}

Marks a claimed delivery processed, gated on the caller's token under an unexpired lease. :tenant scopes the mark (required on multitenant ledgers).

plan_legacy_identity_adoption(resource, name, opts \\ [])

@spec plan_legacy_identity_adoption(module(), atom(), keyword()) ::
  {:ok, AshHooks.InboundDelivery.LegacyAdoption.plan()} | {:error, term()}

Builds the legacy provider-identity adoption audit without writing rows.

prune(resource, opts)

@spec prune(module(), keyword()) :: {:ok, non_neg_integer()} | {:error, term()}

Retention hook: deletes one bounded batch of TERMINAL ledger rows (:processed, :failed_permanent, :superseded) older than older_than, by the resource's inserted_at (add Ash timestamps() to the resource and its migration). Non-terminal rows are never deleted; retryable and lease-held deliveries keep their dedup identity and re-drive path.

Deleting a terminal row re-opens its dedup identity: a replayed delivery of the same webhook processes again (inbound), and on the outbound side, a re-emission of the same event re-dispatches and double-sends. Set the TTL beyond any replay/re-emission horizon. :batch_size defaults to 100 and is capped at 1,000. Only primary-key columns are selected before deletion. Returns {:ok, deleted_count} for this batch; call again until it returns a count below the requested size.

reap(resource, opts \\ [])

@spec reap(module(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, :tenant_required | :tenancy_mismatch}

Re-drives deliveries whose claims died with an expired lease (crash between claim and mark). Returns the number re-driven; unexpired leases and terminal rows are left alone. :tenant scopes the sweep; a tenant-less global reap over a multitenant ledger is the named {:error, :tenant_required} (the unscoped read would otherwise fail); per-tenant sweeps compose with AshHooks.reap_all/2.

Rows that cannot be re-driven (inbound declaration removed, provider unresolved) are skipped without stopping the sweep. Handler and parser exceptions, exits, and throws are recorded as retryable failures and count as re-driven when that write succeeds. One failing row never stops the remaining rows.

redact_payload(resource, delivery_id, token, redactor, opts \\ [])

@spec redact_payload(module(), term(), non_neg_integer(), term(), keyword()) ::
  :ok
  | {:error, :stale_token | :redactor_crash | :invalid_redactor_result | term()}

Retention field-redaction hook: replaces the stored payload of a CLAIMED delivery with redactor.(payload) under the caller's token and an unexpired lease (the mark_processed/3 fence). The row keeps its dedup identity; only the payload changes. payload_digest is deliberately NOT updated (it binds the ORIGINAL signed bytes for the audit trail).

The redactor ({m, f} | 1-arity fun) receives the stored payload (map or list, including event batches) and returns the replacement (same type) or nil to leave it unchanged. A crashing or invalid redactor returns an error and leaves the payload UNCHANGED (preserving the audit record so the caller may retry). Redact, then mark promptly: a lease expiry between redact and mark re-drives the row with the redacted payload.

renew(resource, delivery_id, token, opts \\ [])

@spec renew(module(), term(), non_neg_integer(), keyword()) ::
  :ok | {:error, :stale_token | term()}

Extends the caller's lease, gated on the caller's token under an unexpired lease.

resolve_source(resource, name, opts \\ [])

@spec resolve_source(module(), atom(), keyword()) ::
  {:ok, AshHooks.Inbound.ResolvedSource.t()} | {:error, term()}

Resolves one inbound declaration, casting and constraining scope before a source lookup. A source row is read in a tenant phase; an absent row returns UnknownSource. Declarations without source_record perform no source read. Options: :tenant, :scope. The returned struct can be passed as :resolved_source to ingest, as the HTTP entry point does.