Improve v0.16 migration instructions

UPGRADING/v0_16.md

@@ -523,30 +523,32 @@ Most failures come from trying to create an object whose parent does not exist y
 
 `apply` runs operations in plan order and stops on the first error. When a `create` fails halfway through, every prior `create` in the same run has already been committed to the database. Re-running the same plan now fails with `primaryKeyViolation` (the objects exist) or `invalidForeignKey` (a parent that did not get created the first time is still missing).
 
-The cleanest recovery is to wipe the registry and apply again. While the server is still in recovery mode, list and delete the migrated objects:
+> **Do not bulk-delete `Account` objects to recover.** The migration plan creates each account with its original v0.15 account id (the `restore-<id>` mechanism), so a migrated `Account` points at the existing v0.15 mailbox data in the data store. Deleting that `Account` schedules account destruction, which unlinks and erases all mail, calendars, and contacts stored under that id. On the community edition this runs immediately, with no retention window. Never run `delete Account` against a data store that already contains v0.15 mail.
+
+Recovery does not require deleting accounts. An account that a partial `apply` already created is correct and is reused as-is on the next run; the only objects that need clearing are the registry-only ones that carry no mailbox data and whose re-creation would otherwise raise `primaryKeyViolation`. While the server is still in recovery mode:
 
 ```bash
-$ stalwart-cli query Domain --json | jq -r '.[].id' \
-    | stalwart-cli delete Domain --stdin
-$ stalwart-cli query Account --json | jq -r '.[].id' \
-    | stalwart-cli delete Account --stdin
-$ stalwart-cli query Tenant --json | jq -r '.[].id' \
-    | stalwart-cli delete Tenant --stdin
 $ stalwart-cli query DkimSignature --json | jq -r '.[].id' \
     | stalwart-cli delete DkimSignature --stdin
 $ stalwart-cli query Certificate --json | jq -r '.[].id' \
     | stalwart-cli delete Certificate --stdin
+$ stalwart-cli query Domain --json | jq -r '.[].id' \
+    | stalwart-cli delete Domain --stdin
+$ stalwart-cli query Tenant --json | jq -r '.[].id' \
+    | stalwart-cli delete Tenant --stdin
 ```
 
-Then fix the underlying cause in `export.json` (most often a domain that fails the v0.16 hostname check, an account whose local-part contains `@`, or a stale `/opt/stalwart` path embedded by the migration script) and rerun:
+`Domain` and `Tenant` hold only directory metadata and are safe to delete and recreate; `Account` is deliberately omitted. Then fix the underlying cause in `export.json` (most often a domain that fails the v0.16 hostname check, an account whose local-part contains `@`, or a stale `/opt/stalwart` path embedded by the migration script), remove from `export.json` the `create` operation for `Account` (and any other object that already committed before the failure, so re-applying it does not raise `primaryKeyViolation`), and rerun:
 
 ```bash
 $ stalwart-cli apply --file export.json
 ```
 
-If the failure was caused by data that the migration script itself produced incorrectly, also rerun the script with the latest version from `main` before applying. Fixes during the v0.16.0 / v0.16.1 window addressed several edge cases (group names containing `@`, ACME base64 padding, single-URL Redis stores, paths embedded in custom storage backends).
+If you must start over with the accounts as well, do not delete them: point the new deployment at an empty data store (or restore the v0.15 data-store backup) before re-running `apply`, so that destroying and recreating accounts cannot reach live mail.
 
-For deployments where individual objects are easier to identify than to wipe wholesale, use `stalwart-cli query <type>` to list ids and `stalwart-cli delete <type> --ids <id>` to remove a specific one.
+If the failure was caused by data that the migration script itself produced incorrectly, also rerun the script with the latest version from `main` before applying. Fixes during the v0.16.0 / v0.16.1 window addressed several edge cases (group names containing `@`, ACME base64 padding, single-URL Redis stores, paths embedded in custom storage backends, and `%{file:...}%` / `%{env:...}%` macros in DKIM private keys and certificates, which are now expanded by the script instead of being passed through verbatim and aborting the `apply`).
+
+For deployments where individual objects are easier to identify than to wipe wholesale, use `stalwart-cli query <type>` to list ids and `stalwart-cli delete <type> --ids <id>` to remove a specific one. The same warning applies: deleting an `Account` destroys the mail stored under it. Only `Domain`, `Tenant`, `DkimSignature`, and `Certificate` are safe to delete and recreate during recovery.
 
 ### Bootstrapping a real administrator from the CLI
 

resources/scripts/migrate_v016.py

@@ -31,6 +31,7 @@ from __future__ import annotations
 import argparse
 import base64
 import json
+import os
 import re
 import sys
 import urllib.parse
@@ -637,6 +638,57 @@ def secret_text(value: str | None) -> dict[str, Any]:
         return {"@type": "None"}
     return {"@type": "Text", "secret": value}
 
+_MACRO_RE = re.compile(r"%\{(cfg|env|file):([^}]*)\}%")
+
+def resolve_macros(
+    value: str | None,
+    settings: dict[str, str],
+    _seen: frozenset[str] = frozenset(),
+) -> tuple[str | None, list[str]]:
+    if value is None or "%{" not in value:
+        return value, []
+    errors: list[str] = []
+
+    def repl(m: "re.Match[str]") -> str:
+        kind = m.group(1)
+        arg = m.group(2).strip()
+        if kind == "cfg":
+            if arg in _seen:
+                errors.append(f"circular %{{cfg:{arg}}}% reference")
+                return ""
+            raw = settings.get(arg)
+            if raw is None:
+                errors.append(f"unknown setting referenced by %{{cfg:{arg}}}%")
+                return ""
+            nested, nested_errors = resolve_macros(
+                raw, settings, _seen | {arg}
+            )
+            errors.extend(nested_errors)
+            return nested or ""
+        if kind == "env":
+            env = os.environ.get(arg)
+            if env is None:
+                errors.append(
+                    f"environment variable {arg!r} (from %{{env:{arg}}}%) "
+                    f"is not set"
+                )
+                return ""
+            return env
+        try:
+            with open(arg, "r", encoding="utf-8") as fh:
+                return fh.read()
+        except OSError as exc:
+            errors.append(f"cannot read file {arg!r} (from %{{file:...}}%): {exc}")
+            return ""
+
+    prev = value
+    for _ in range(8):
+        cur = _MACRO_RE.sub(repl, prev)
+        if cur == prev:
+            break
+        prev = cur
+    return prev, errors
+
 _REDIS_PROTOCOL_MAP = {
     "resp2": "resp2",
     "resp3": "resp3",
@@ -1158,11 +1210,30 @@ class Converter:
             canon = sub.get("canonicalization", "relaxed/relaxed").strip().lower()
             if not canon:
                 canon = "relaxed/relaxed"
+            private_key, key_errors = resolve_macros(
+                sub.get("private-key"), self.settings
+            )
+            if private_key is not None:
+                private_key = private_key.strip()
+            if key_errors:
+                print(
+                    f"warning: skipping DKIM signature {sid!r}: could not "
+                    f"resolve private-key: {'; '.join(key_errors)}",
+                    file=sys.stderr,
+                )
+                continue
+            if not private_key or "%{" in private_key:
+                print(
+                    f"warning: skipping DKIM signature {sid!r}: private-key is "
+                    f"empty or still contains an unresolved macro",
+                    file=sys.stderr,
+                )
+                continue
             body: dict[str, Any] = {
                 "@type": tag,
                 "canonicalization": canon,
                 "domainId": "#" + dom_cid,
-                "privateKey": secret_text(sub.get("private-key")),
+                "privateKey": secret_text(private_key),
                 "selector": selector,
             }
             t_cid = self.domain_cid_to_tenant_cid.get(dom_cid)
@@ -1687,8 +1758,21 @@ class Converter:
         for sid, sub in sorted(
             build_sub_trees(self.settings, "certificate", "cert").items()
         ):
-            cert = sub.get("cert", "").strip()
+            cert, cert_errors = resolve_macros(
-            key = sub.get("private-key", "").strip()
+                sub.get("cert", ""), self.settings
+            )
+            key, key_errors = resolve_macros(
+                sub.get("private-key", ""), self.settings
+            )
+            cert = (cert or "").strip()
+            key = (key or "").strip()
+            if cert_errors or key_errors:
+                print(
+                    f"warning: skipping certificate.{sid}: could not resolve "
+                    f"value: {'; '.join(cert_errors + key_errors)}",
+                    file=sys.stderr,
+                )
+                continue
             if not cert or not key:
                 print(
                     f"warning: skipping certificate.{sid}: "