Multi-tenant adoption checklist

Copy Markdown View Source

ash_hooks supports Ash attribute multitenancy: pass an explicit tenant to dispatch, ingest, delivery runs, sweeps, and reconciliation. Tenant validation checks the resource set touched by each operation, including optional history and event logs. This checklist walks one app (two tenants or two thousand) through the same steps. AshHooks.Tenancy documents the supported tenancy strategies and validation rules.

The contract

Declare compatible attribute tenancy on every resource an operation touches: subscriptions, endpoints, outbound deliveries, enabled attempt and event logs, inbound ledgers, inbound sources, and custody registries. Entry points supply every resource they read or write to AshHooks.Tenancy.resolve/2. Event-log replay also forwards the tenant when it rearms an existing terminal delivery.

Add a non-nullable tenant attribute (for example :org_id) and declare multitenancy with strategy :attribute and attribute :org_id on each resource. The Livebook executes complete multitenant resource declarations and calls.

Two declarations are rejected outright with {:error, :tenancy_mismatch}:

  • global?: true: it looks consistent while silently allowing tenant-less reads, which would disable every guarantee below.
  • non-:attribute strategies: attribute is the only strategy that works across every data layer the package supports.

The ordered steps

If you are also upgrading from a version >= 1.0.0 and < 2.0.0 to 3.3.0, complete the intervening schema and declaration/route migrations in UPGRADING before enabling the new runtime.

0. Drain Oban jobs created before 1.2.0 (skip if you have no queued deliveries). The moment your resources declare multitenancy, jobs enqueued by the old version, whose args carry no tenant, start failing closed with {:error, :tenant_required} when they run. That is the correct, safe behavior, but it is noisy: drain or complete the old queue BEFORE deploying the version whose resources declare tenancy.

1. Add the tenant columns, then backfill them. The following four tables are examples. Apply the same column, backfill, constraint, and isolation checks to every package resource your application uses, including optional attempt logs, event logs, inbound sources, and custody registries. First the columns (in the adopter's migration):

ALTER TABLE webhook_endpoints ADD COLUMN org_id TEXT;
ALTER TABLE webhook_subscriptions ADD COLUMN org_id TEXT;
ALTER TABLE inbound_deliveries ADD COLUMN org_id TEXT;
ALTER TABLE outbound_deliveries ADD COLUMN org_id TEXT;

Then backfill: every legacy row must get its tenant value. This example assumes the application's authoritative account_tenants(account_id, org_id) mapping contains exactly one non-NULL tenant for every legacy account. Endpoints and inbound deliveries already carry account_id; subscriptions and outbound deliveries carry endpoint_id, referencing webhook_endpoints.id. Populate and validate that mapping from the application's ownership records first. Derive each child row's tenant from its endpoint, rather than independently assigning a tenant that could disagree with that endpoint:

UPDATE webhook_endpoints
SET org_id = (
  SELECT account_tenants.org_id
  FROM account_tenants
  WHERE account_tenants.account_id = webhook_endpoints.account_id
);

UPDATE inbound_deliveries
SET org_id = (
  SELECT account_tenants.org_id
  FROM account_tenants
  WHERE account_tenants.account_id = inbound_deliveries.account_id
);

UPDATE webhook_subscriptions
SET org_id = (
  SELECT webhook_endpoints.org_id
  FROM webhook_endpoints
  WHERE webhook_endpoints.id = webhook_subscriptions.endpoint_id
);

UPDATE outbound_deliveries
SET org_id = (
  SELECT webhook_endpoints.org_id
  FROM webhook_endpoints
  WHERE webhook_endpoints.id = outbound_deliveries.endpoint_id
);

Verify zero NULL tenants remain. All four result rows must be 0; a missing account mapping or endpoint reference leaves NULL and prevents proceeding:

SELECT COUNT(*) FROM webhook_endpoints WHERE org_id IS NULL
UNION ALL
SELECT COUNT(*) FROM webhook_subscriptions WHERE org_id IS NULL
UNION ALL
SELECT COUNT(*) FROM inbound_deliveries WHERE org_id IS NULL
UNION ALL
SELECT COUNT(*) FROM outbound_deliveries WHERE org_id IS NULL;

A NULL-tenant row is a quiet trap: the unique identities become tenant-scoped in step 2, and NULL never equals a tenant value, so a new delivery creates a separate row instead of matching the old one and may repeat its network effect. NULL-tenant rows are also invisible to every tenant-scoped sweep. Backfill before proceeding; the count above is the gate.

2. Regenerate migrations so the unique identities include the tenant. The unique identities (unique_delivery on endpoint+event, unique_ingest on provider+external id + scope) become tenant-prefixed unique indexes; run your mix ash.codegen / migration generator after adding the declarations. Dedup is per-tenant by construction from here: the same provider event id in two tenants is two rows, both processed.

3. Deploy, and thread the tenant through your call sites.

Pass tenant: tenant to AshHooks.dispatch/4 and include tenant: tenant in the request context passed to AshHooks.Ingress.ingest/4. The generated worker serializes the tenant from the row into job args and restores it on execution.

4. Adopt the per-tenant operations surface.

  • Sweeps are per-tenant: reap/2 and both prune/2s take :tenant; AshHooks.reap_all/2 and AshHooks.prune_all/2 sweep a list of tenants sequentially (one tenant's failure never stops the others) and return each tenant's result plus a total: {:ok, %{results: %{"org_a" => {:ok, 3}, ...}, total: 3}}.
  • Reconciliation of stale pending rows, enqueue failures, due retries, expired sends, and pending endpoint disables: AshHooks.reconcile_pending(Order, :order_paid, tenant: org.id, enqueue: {MyWorker, :enqueue}). The delivery resource needs timestamps() (inserted_at drives the staleness cutoff; the default is five minutes). Use the same declaration and durable queue route. Concurrent reconcilers use separate expiring enqueue claims; the enqueue callback receives the event reconstructed from the row (id, type, and payload) with an empty metadata map, so write custom callbacks accordingly.
  • Tenant-aware secrets (all optional): the worker macro's tenant_aware_secrets: true (resolver becomes f(ref, tenant)), the inbound secret fn tenant -> {:ok, secret} end, and the provider webhook_signing_secret(connection, tenant) override for per-tenant signing-secret resolution.

Canonical custody tenants and row security

For a stored row, call AshHooks.Tenancy.from_row/1 before reusing its tenant in entry-point options. It applies the resource's tenant_from_attribute inverse; reading the raw attribute bypasses that inverse and may apply a nonidentity parser twice. Custody then calls AshHooks.Custody.canonical_tenant/3 with the owner resource, following its create-path tenant conversion and constrained cast. UUID and UUIDv7 values normalize case, integers use decimal text, strings stay exact UTF-8, and ci_strings normalize case. Non-multitenant owners use nil. Invalid values return :invalid_tenant; unsupported types return :unsupported_tenant_type. tenant_type_supported?/2 checks type coverage.

Other attribute types require a custody_tenant_key {module, function, args} option on the endpoint or inbound source. Its cast-value callback must return equal UTF-8 binaries whenever the database treats two values as equal. The registry stores this canonical custody identity separately from its database tenant and uses it after owner deletion. See the key guide.

Source records use AshHooks.Ingress.resolve_source/3 with both tenant and scope. It validates matching source/ledger tenancy and casts all scope fields before a source read. Reuse its result through the same request's :resolved_source ingest context. Schedule endpoint/source expiry, registry sweeps, registry failure recovery, and alias adoption per tenant, as well as delivery retention.

Package database work uses bounded tenant transactions, and reads inside each phase use the write repository on AshPostgres and transactional AshSqlite. PostgreSQL hosts bind transaction-local row-security settings in the repository's on_transaction_begin/1. Secure package tables and both Oban tables, and bind an enclosing host transaction to the same tenant. Use an insert-only Oban instance on the runtime ledger repository and a separate engine-role instance for queues and maintenance, sharing the prefix and PostgreSQL notifier. The row-security guide provides exact role, grant, policy, and supervision examples; SQLite tenant filtering does not exercise PostgreSQL row security.

The isolation guarantees (what you may rely on)

  • A dispatch for tenant A cannot create a delivery row for tenant B's endpoint; a subscription pointing at another tenant's endpoint_id resolves to not-found and is skipped; the 410-disable cannot be aimed cross-tenant.
  • The inbound dedup identity is per-tenant; the claim fence, marks, renewal, and redaction are per-tenant.
  • A tenant-less call against a multitenant set returns {:error, :tenant_required} before any data access.
  • Multitenant jobs store the row's tenant in args["tenant"]. The tenant must survive Oban's JSON encoding unchanged: strings (UUIDs and slugs) and integers do; an atom arrives as a string. Pair a custom parse_attribute with tenant_from_attribute so the stored attribute and job tenant round-trip. Use args->>'tenant', the tenant's text form, in the runtime role's job policies. The runtime role can select and insert only its tenant's jobs; it cannot update jobs or access oban_peers. The separate engine role needs policies over all rows of oban_jobs and oban_peers, with row security enabled and forced on both tables. A tenant-only policy hides queued jobs from Oban's unbound engine.
  • Recovery is fenced to the row's declaration, route, tenant, and enqueue token. A crash after queue admission can still invoke a custom enqueue callback again; make that callback idempotent. The generated Oban worker requires a persisted matching runnable job before reporting successful admission.

Single-tenant apps

Leave multitenancy undeclared and omit :tenant. The same entry points work with a single-tenant ledger; the schema upgrade introduced in 2.0.1 still applies.