filip/Stalwart
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Improve v0.16 migration instructions

Browse Source
This commit is contained in:
Maurus Decimus
2026-05-19 18:13:08 +02:00
parent 628d5520f6
commit 79907f018d
2 changed files with 99 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
UPGRADING/v0_16.md
View File

@@ -523,30 +523,32 @@ Most failures come from trying to create an object whose parent does not exist y
`apply` runs operations in plan order and stops on the first error. When a `create` fails halfway through, every prior `create` in the same run has already been committed to the database. Re-running the same plan now fails with `primaryKeyViolation` (the objects exist) or `invalidForeignKey` (a parent that did not get created the first time is still missing).
The cleanest recovery is to wipe the registry and apply again. While the server is still in recovery mode, list and delete the migrated objects:
> **Do not bulk-delete `Account` objects to recover.** The migration plan creates each account with its original v0.15 account id (the `restore-<id>` mechanism), so a migrated `Account` points at the existing v0.15 mailbox data in the data store. Deleting that `Account` schedules account destruction, which unlinks and erases all mail, calendars, and contacts stored under that id. On the community edition this runs immediately, with no retention window. Never run `delete Account` against a data store that already contains v0.15 mail.
Recovery does not require deleting accounts. An account that a partial `apply` already created is correct and is reused as-is on the next run; the only objects that need clearing are the registry-only ones that carry no mailbox data and whose re-creation would otherwise raise `primaryKeyViolation`. While the server is still in recovery mode:
```bash
$ stalwart-cli query Domain --json | jq -r '.[].id' \
| stalwart-cli delete Domain --stdin
$ stalwart-cli query Account --json | jq -r '.[].id' \
| stalwart-cli delete Account --stdin
$ stalwart-cli query Tenant --json | jq -r '.[].id' \
| stalwart-cli delete Tenant --stdin
$ stalwart-cli query DkimSignature --json | jq -r '.[].id' \
| stalwart-cli delete DkimSignature --stdin
$ stalwart-cli query Certificate --json | jq -r '.[].id' \
| stalwart-cli delete Certificate --stdin
$ stalwart-cli query Domain --json | jq -r '.[].id' \
| stalwart-cli delete Domain --stdin
$ stalwart-cli query Tenant --json | jq -r '.[].id' \
| stalwart-cli delete Tenant --stdin
```
Then fix the underlying cause in `export.json` (most often a domain that fails the v0.16 hostname check, an account whose local-part contains `@`, or a stale `/opt/stalwart` path embedded by the migration script) and rerun:
`Domain` and `Tenant` hold only directory metadata and are safe to delete and recreate; `Account` is deliberately omitted. Then fix the underlying cause in `export.json` (most often a domain that fails the v0.16 hostname check, an account whose local-part contains `@`, or a stale `/opt/stalwart` path embedded by the migration script), remove from `export.json` the `create` operation for `Account` (and any other object that already committed before the failure, so re-applying it does not raise `primaryKeyViolation`), and rerun:
```bash
$ stalwart-cli apply --file export.json
```
If the failure was caused by data that the migration script itself produced incorrectly, also rerun the script with the latest version from `main` before applying. Fixes during the v0.16.0 / v0.16.1 window addressed several edge cases (group names containing `@`, ACME base64 padding, single-URL Redis stores, paths embedded in custom storage backends).
If you must start over with the accounts as well, do not delete them: point the new deployment at an empty data store (or restore the v0.15 data-store backup) before re-running `apply`, so that destroying and recreating accounts cannot reach live mail.
For deployments where individual objects are easier to identify than to wipe wholesale, use `stalwart-cli query <type>` to list ids and `stalwart-cli delete <type> --ids <id>` to remove a specific one.
If the failure was caused by data that the migration script itself produced incorrectly, also rerun the script with the latest version from `main` before applying. Fixes during the v0.16.0 / v0.16.1 window addressed several edge cases (group names containing `@`, ACME base64 padding, single-URL Redis stores, paths embedded in custom storage backends, and `%{file:...}%` / `%{env:...}%` macros in DKIM private keys and certificates, which are now expanded by the script instead of being passed through verbatim and aborting the `apply`).
For deployments where individual objects are easier to identify than to wipe wholesale, use `stalwart-cli query <type>` to list ids and `stalwart-cli delete <type> --ids <id>` to remove a specific one. The same warning applies: deleting an `Account` destroys the mail stored under it. Only `Domain`, `Tenant`, `DkimSignature`, and `Certificate` are safe to delete and recreate during recovery.
### Bootstrapping a real administrator from the CLI

90
resources/scripts/migrate_v016.py
View File

@@ -31,6 +31,7 @@ from __future__ import annotations
import argparse
import base64
import json
import os
import re
import sys
import urllib.parse
@@ -637,6 +638,57 @@ def secret_text(value: str | None) -> dict[str, Any]:
return {"@type": "None"}
return {"@type": "Text", "secret": value}
_MACRO_RE = re.compile(r"%\{(cfg|env|file):([^}]*)\}%")
def resolve_macros(
value: str | None,
settings: dict[str, str],
_seen: frozenset[str] = frozenset(),
) -> tuple[str | None, list[str]]:
if value is None or "%{" not in value:
return value, []
errors: list[str] = []
def repl(m: "re.Match[str]") -> str:
kind = m.group(1)
arg = m.group(2).strip()
if kind == "cfg":
if arg in _seen:
errors.append(f"circular %{{cfg:{arg}}}% reference")
return ""
raw = settings.get(arg)
if raw is None:
errors.append(f"unknown setting referenced by %{{cfg:{arg}}}%")
return ""
nested, nested_errors = resolve_macros(
raw, settings, _seen | {arg}
)
errors.extend(nested_errors)
return nested or ""
if kind == "env":
env = os.environ.get(arg)
if env is None:
errors.append(
f"environment variable {arg!r} (from %{{env:{arg}}}%) "
f"is not set"
)
return ""
return env
try:
with open(arg, "r", encoding="utf-8") as fh:
return fh.read()
except OSError as exc:
errors.append(f"cannot read file {arg!r} (from %{{file:...}}%): {exc}")
return ""
prev = value
for _ in range(8):
cur = _MACRO_RE.sub(repl, prev)
if cur == prev:
break
prev = cur
return prev, errors
_REDIS_PROTOCOL_MAP = {
"resp2": "resp2",
"resp3": "resp3",
@@ -1158,11 +1210,30 @@ class Converter:
canon = sub.get("canonicalization", "relaxed/relaxed").strip().lower()
if not canon:
canon = "relaxed/relaxed"
private_key, key_errors = resolve_macros(
sub.get("private-key"), self.settings
)
if private_key is not None:
private_key = private_key.strip()
if key_errors:
print(
f"warning: skipping DKIM signature {sid!r}: could not "
f"resolve private-key: {'; '.join(key_errors)}",
file=sys.stderr,
)
continue
if not private_key or "%{" in private_key:
print(
f"warning: skipping DKIM signature {sid!r}: private-key is "
f"empty or still contains an unresolved macro",
file=sys.stderr,
)
continue
body: dict[str, Any] = {
"@type": tag,
"canonicalization": canon,
"domainId": "#" + dom_cid,
"privateKey": secret_text(sub.get("private-key")),
"privateKey": secret_text(private_key),
"selector": selector,
}
t_cid = self.domain_cid_to_tenant_cid.get(dom_cid)
@@ -1687,8 +1758,21 @@ class Converter:
for sid, sub in sorted(
build_sub_trees(self.settings, "certificate", "cert").items()
):
cert = sub.get("cert", "").strip()
key = sub.get("private-key", "").strip()
cert, cert_errors = resolve_macros(
sub.get("cert", ""), self.settings
)
key, key_errors = resolve_macros(
sub.get("private-key", ""), self.settings
)
cert = (cert or "").strip()
key = (key or "").strip()
if cert_errors or key_errors:
print(
f"warning: skipping certificate.{sid}: could not resolve "
f"value: {'; '.join(cert_errors + key_errors)}",
file=sys.stderr,
)
continue
if not cert or not key:
print(
f"warning: skipping certificate.{sid}: "