Private message history sync: how it works and how to operate it

Related: requirements · plan · implementation and evidence · storage findings · sender-assistance implementation.

What the feature does

Private message history sync helps approved devices read older messages using encrypted access supplied by a device that can already read them. Your own devices can help with sent and received messages. A sender’s devices can also help you recover messages that sender originally sent to you. It runs automatically during authenticated visits, without a sync banner or a separate sharing prompt. This guide includes the sender-assisted extension; merge and production deployment are separate steps.

For example, Alice's laptop can read her old conversations, but her newly approved phone cannot. Alice visits the site on the laptop, then later opens Messages on the phone. The laptop leaves encrypted recovery material for the phone, allowing the phone to read and verify the original messages. The two devices do not need to be online together.

If Alice has lost every old device, Bob may still be able to help: after Alice’s phone is approved, Bob visits the site on a device that can read his messages to Alice. That visit leaves encrypted access for Alice’s phone. Bob can help restore his messages to Alice, not Alice’s messages to him. Bob receives no recovery report or read receipt. Any currently approved Bob key with verified access can contribute; it need not be the key that originally signed the message.

How recovery happens

1. A key is approved for the account. The application treats currently approved keys sharing the normalized username as that account's keys. All those keys are authorized for its history, including keys approved before this feature existed. Original senders may supply access to their outgoing messages to those approved recipient keys; no additional sharing prompt is required.

2. A device with access visits the site. It can contribute from an ordinary authenticated page; opening Messages on that device is unnecessary. The browser checks a bounded portion of history and verifies messages it can already decrypt.

3. The browser prepares encrypted access. Each original message has a session key that unlocks its encrypted contents. The contributing browser signs a bundle of these keys and encrypts it specifically for an approved target device. The bundle identifies the destination account, source and target keys, and the exact original messages. Sender-assisted bundles additionally bind the source account and sender-assistance purpose.

4. The server retains the encrypted bundle. This is private recovery data, separate from conversations. It creates no chat messages and does not mark messages read. Original encrypted messages remain unchanged.

5. The target reads the originals. Its browser decrypts the bundle with its own private key, checks the donor and message bindings, then decrypts each original message and verifies the original sender's signature before displaying trusted text. A donor's signature alone is insufficient.

Transfers stay available after successful recovery, so reloading the target does not require another donor visit. Multiple devices can supply different parts of history, and a recovered device can help later approved devices. The updated browser continues useful work in bounded turns during one visible visit; interrupted work resumes on a later visit. Recovery still depends on eligible donors and is not guaranteed to finish in one visit. Existing per-message unavailable states and reading retries remain available.

What is protected, and what remains limited

• Private identity keys stay in their browsers. Message plaintext and unwrapped session keys are not sent to the server or added to persistent recovery caches.

• The server stores encrypted messages, encrypted recovery bundles and synchronization metadata. It can see account, key and message relationships; this does not hide that metadata from the operator.

• Only currently approved keys participate. Each contribution is either within one account or from the original sender’s account to the original recipient’s account. Authorization is checked for each message during upload, retrieval and confirmation. Removing approval prevents future eligible transfer use, but cannot retract secrets a device already received.

• Recovery requires an eligible device with suitable access or a retained eligible transfer. If neither exists, approval alone cannot restore history. Original messages must also still exist on the server.

• Sender assistance restores incoming messages only. It does not grant unrelated conversation access or allow a recipient to give history back to the sender. Sender-visible work and upload responses do not depend on recipient confirmations; the sender receives no recipient recovery counts or read receipts. Ordinary replies can still reveal that someone read a message.

• Messages/unread data and recovery data use separate private databases. A sync-store outage can interrupt historical recovery while ordinary sending and direct reading continue. Both stores need coordinated private backups; neither belongs in public downloads or offline snapshots.

The remaining sections document the API, exact limits, deployment, backup and rollback for technical readers and operators.

Storage and deployment

Deploy PHP and browser assets together. Messages and unread state remain in PRIVATE_MESSAGE_DATABASE_PATH (default state/private/messages.sqlite3). Sync uses its own lazy connection to PRIVATE_MESSAGE_HISTORY_SYNC_DATABASE_PATH (default state/private/message_history_sync.sqlite3). Configure both paths outside the document root, canonical repository, static releases, offline snapshots, and public backups. Identical paths, symlinks and hardlinks to the message database are rejected. New directories use mode 0700 and the sync file 0600; preserve private permissions for backups and SQLite sidecars too.

The first authenticated sync request creates history_sync_transfers, history_sync_items, history_sync_coverage, history_sync_scan, and history_sync_sender_scan. The sender-assisted upgrade adds only the last table, with independent per-source sender checkpoints; existing transfer rows, account checkpoints and the original-message schema are preserved. No original-message migration or approval-event replay is required: existing approved keys are discovered too. The sync connection uses a one-second busy timeout and FULL synchronous durability, retaining SQLite's default rollback journal on newly created files. Operators should preserve this policy; separate files isolate writer locks, not disk load or original-message reads. There are no attached databases, cross-store foreign keys, or cross-store transactions.

The activity/bounded-recovery upgrade adds private history_sync_activity, history_sync_admission, history_sync_verification, history_sync_schedule and history_sync_destinations state, plus arrival/donor revision tokens and mutation triggers. The message store gains history_messages_revision tokens without changing original envelopes. These are additive lazy upgrades; retained transfers, receipts and old checkpoints remain usable. Read-model schema 39 adds the indexed approved-key traversal; schema 40 adds membership revision tokens/triggers. Normal application startup rebuilds older derived models before use. An in-place rebuild regenerates the membership epoch as well as profiles; the public read model remains disposable.

Admission retains at most 64 events per account and discards expired events on an admitted request; rejected attempts do not rewrite accounting or extend lockout. Verification state retains only the current page, at most ten entries per account/key/mode. Activity is coalesced at five-second granularity. Destination progress replaces the prior revision for each donor/target pair. These are bounds on individual records/work, not a global disk quota: account/key/destination metadata has no automatic retirement policy. Retained encrypted transfers are not swept after recovery; existing replacement cleanup removes only bundles no longer referenced by item indexes or coverage. Do not delete recovery data to clear a rate limit.

Ordinary messaging initializes only its existing store. A missing/unwritable/unavailable sync path yields a generic retryable 503; Messages can still send and directly read messages. Never repair a sync error by deleting the retained transfers. A missing file can be recreated, but old coverage alone cannot recover the lost keys.

API contract

All endpoints require the existing authenticated, currently approved identity and return Cache-Control: no-store on successes and errors. Source authority derives from the authenticated account/key. An upload may name a destination account, but every original message must authorize that sender-to-recipient relationship and the target must currently belong to that destination. POST requires JSON, X-Requested-With: ForumPrivateMessages, and rejects cross-site/same-site fetches. Request bodies are limited to 100,000 bytes; unknown methods/actions return 405, unavailable authentication 401/403, invalid input 400, unavailable storage 503, and history admission throttling 429 with a positive integer Retry-After in seconds. Every history route, including legacy GET and foreground candidate retrieval, shares rolling limits of 16 requests per authenticated key and 64 per account per ten seconds. Sending, mailbox reads and unread status are outside that admission budget. Responses use {status:"ok", ...} or {status:"error", error:...}.

Account work offers up to ten originals per page. Discovery shares a ceiling of 160 message/target decisions and 64 loaded public-key records per request; non-idle responses stay within 4 MiB of encoded JSON. Larger valid work continues through bounded pages. A message includes its original envelope, digest, bounded sender-key subset and candidate availability. The recorded submitting identity is only the first verification hint: other currently approved sender keys remain eligible, and a failed signature does not imply failed decryption. Opaque verification progress accompanies existing checkpoints; timeout-rewritten legacy checkpoints do not discard unfinished subsets. Progress never creates a readable-access receipt. No send-time signer restriction or legacy upgrade-required response is introduced.

Retrieval returns up to ten candidates with source/target/account, source public key, server-derived source_account, transfer ID and ciphertext. A candidate's additive items list authorizes browser reuse only for those current message/digest bindings; each original is checked independently. Pagination advances through revoked-source rows without exposing them. total counts account originals, not recovered messages. Missing original checkpoint anchors restart discovery.

Sender work scans only outgoing originals, batches one recipient at a time, rotates that batch through approved recipient keys, then advances history. Its checkpoints are separate from account scans. Recipients without approved keys are skipped and revisited; missing anchors restart scanning. No recipient receipt is consulted and no recipient recovery total is returned. Legacy GET discovery deliberately revisits contributions even after recipient recovery. Combined POST discovery omits contributions already retained for this donor, target and original digest, independently of recipient recovery. The legacy GET and upload APIs remain available for explicit same-donor repair of an invalid contribution; other donors remain independent. A retained upload is evidence of contribution, never proof that its recipient can read it.

Uploads have at most ten distinct current message/digest bindings and 65,536 bytes of armored ciphertext. Exact upload retries deduplicate. Unconfirmed candidates from the same donor can be replaced; verified retained candidates stay reachable. Changed digests and missing retained blobs reopen work. Different donors remain independent. Acknowledgments always report the authenticated device’s own verified access, including while it is donating to another account; an empty transfer ID reports direct access. They are scheduling hints and do not mark messages seen. Acknowledgment never deletes ciphertext. Original IDs/digests are rechecked on reads and writes, tolerating capture skew between the two databases.

Version 1's signed, target-encrypted JSON is {version:1,account,source,target,entries:[{message_id,digest,keys:[{algorithm,data}]}]}. Same-account writes remain v1. Sender assistance uses {version:2,purpose:"sender-history",source_account,account,source,target,entries} with the same entry structure; account is the original recipient. A v1 bundle cannot be used as a cross-account transfer, and a v2 bundle cannot be rebound as same-account recovery. Fingerprints are full lowercase OpenPGP fingerprints; data is base64 session-key bytes. Browsers verify signer and target fingerprints, current account/key bindings and original envelope digest, then decrypt and verify the original sender signature before showing plaintext or reporting coverage. Bad candidates, unsigned originals and failed signatures cannot establish verified reading. Server indexes expose account/key/message relationships and encrypted bundles, never plaintext or unwrapped keys. Both bundled OpenPGP versions have real-key protocol coverage.

Browser behavior and limits

The existing unread response includes has_history, derived from any sent or received message for the authenticated account with a bounded existence query. The browser waits for that status: unused accounts make no dedicated synchronization calls. First send/receipt is picked up on the next normal unread refresh; visiting Messages and browser-local usage are not eligibility requirements. The gate does not block foreground reading/retry.

An eligible visit starts with combined POST work. Recently foreground-active same-account keys receive three priority selections, followed by one stable-order fairness selection. Approved sign-in and explicit foreground unread refresh record activity privately; background sync and receipts do not promote it. Recipient activity and coverage are never used to schedule another account's sender work.

Completed account/sender passes persist across visits. Relevant message/digest, approved membership, incoming transfer, donor contribution removal and store/rebuild epoch changes reopen them. Activity alone and unrelated profile counts do not reopen completed cryptography. Once settled, the ordinary visit uses one response of at most 8 KiB with no repeated original, candidate or receipt traffic. Large sets of still-unsupplied destinations can require finite background work even after the visible device has recovered.

Each background turn admits at most sixteen HTTP attempts including retries, ten distinct originals and three concurrent requests. New work stops at ten seconds; pending network work is canceled at fifteen seconds, without claiming to preempt ongoing cryptography. Useful unfinished work yields at least one second before continuing; 429 honors Retry-After. No-progress/completed passes stop their timers. Web Locks coalesce tabs where supported; server admission remains the fallback. Five-second individual HTTP timeouts and one network/503 retry remain, within the turn budget. A 405 on combined discovery selects legacy GET for that page.

Foreground readers wait for an active background turn and get priority over its continuation. Queued readers recheck newly learned throttle deadlines before retrieval. Candidate entries and verified original keys are bounded transient memory, never persistent browser storage; cancellation, hidden/navigation/identity boundaries and turn completion clear shared secret caches. Foreground candidate sharing also expires after one second of inactivity. Receipts coalesce in batches of ten over 50 ms; a failed signature or upload alone cannot create one. Early reads retry when the page learns the session viewer. Existing per-message retry and unread-failure recovery remain available; there is no global sync banner or polling after settlement.

Older clients keep their existing bounded-visit behavior and can consume paged verification subsets through existing work/checkpoint fields; only the updated coordinator supplies the new automatic continuation behavior. Deploy backend and browser assets together.

Devices need not be online together. Retained bundles permit reading after reload and allow a restored device to help later approved keys. Recovery depends on at least one eligible device retaining access; a status check does not prove every historical message is recoverable. Only successful original-message verification establishes readable content; uploads alone do not. Normal message ordering, drafts, sending, signature warnings, compact unavailable groups and unread/seen boundaries remain controlled by their existing components. Revocation prevents future eligible transfer use, but cannot retract already delivered secrets.

Coordinated private backup and restore

1. Resolve and record both configured absolute paths and the corresponding account/approval repository state. Preserve file ownership and private permissions. Take private backups outside release artifacts.

2. Quiesce message sends and sync writes and drain active requests. Use SQLite's backup API or an equivalent consistent snapshot for each file while writes remain stopped. A live bare cp can miss transactions or journal/WAL state. Keeping writes stopped across both captures gives a common logical backup point; the application does not provide cross-database atomic snapshots.

3. Verify each backup with PRAGMA integrity_check; retain the capture times/configuration. Back up the canonical identity/approval repository through its existing procedure. Resume writes after both private captures finish.

4. For restore, stop/drain writers again, restore the matching pair and compatible identity/approval state, and handle journals using SQLite's supported restore procedure. Preserve private permissions, then start the matching code/assets and run the smoke journey below.

5. With mismatched captures, stable message IDs and envelope digests reject orphaned or changed references. Missing transfer ciphertext reopens coverage; missing scan anchors restart discovery. Allow donor visits to reconcile and inspect remaining unavailable messages. Older sync captures may permanently lose the only retained transfer if no eligible donor can return; receipts cannot reconstruct it. Existing unread metadata has its own restore constraints—see unread rollout.

Rollback code and assets together while retaining both databases. Rolling back sender assistance to the same-account release preserves v1 recovery and ignores the additive sender checkpoint table; its backend excludes foreign-account donors and its browser rejects v2 bundles. History dependent only on sender assistance becomes unavailable until re-upgrade. Rolling back further, to pre-sync code, disables all transfer-based recovery. These are code limitations, not reasons to delete transfers. Re-upgrading reuses retained bundles/current membership and resumes scans. Do not copy sync tables into the message database.

Rolling back this activity/bounded-recovery release to its preceding sender-assisted code also retains both private databases. Older code ignores the additive scheduling/activity/verification tables and can still use retained transfers; on re-upgrade, current membership and mutation revisions govern resumed work. The local check exercised an actual pre-cycle store implementation from 4b14e622f, then current code, the old store again and current code again, preserving transfer ciphertext, receipts and checkpoints with a clean integrity check. This is a store-level rollback/re-upgrade check, complemented by real-key frozen legacy-coordinator tests; it is not a full previous-release deployment rehearsal.

Mixed versions fail conservatively: old clients retain v1 same-account recovery against the new backend but cannot consume v2 sender bundles; new clients against the old backend detect unsupported sender work and continue account work. Older unread responses without the new eligibility field permit legacy synchronization rather than permanently disabling recovery. Deploy PHP and assets together for the full feature. A rollback does not revoke secrets already delivered to browsers.

Idle synchronization deployment checks

After authorized deployment, use the production browser network panel on an ordinary forum page:

1. For an approved account with no sent or received messages, wait for unread status and focus/reload the page. Expect zero requests under /api/private_messages/history_sync/.

2. Send or receive the first message, then return/focus or reload normally. The next successful unread refresh enables synchronization without a separate setup step.

3. For settled history, observe one POST work per successful idle run and no transfer/acknowledgment. The existing unread call is separate. Large histories can report bounded scan continuation; record that separately from genuine idle.

4. Approve another device, let eligible donors visit ordinary pages, and verify older content through Messages. Repeat after an interrupted visit. Existing per-message retry remains available; failed unread status uses Retry on Messages. Missing access still requires an eligible donor or retained valid transfer.

5. Disable messaging and verify sync stops, then re-enable and confirm recovery resumes. Do not infer production acceptance from local test results.

Local evidence: activity and bounded-recovery acceptance and implementation summary. Earlier idle synchronization evidence and the user’s eventual-recovery report remain historical evidence, not production acceptance of this release.

Deployment smoke and evidence boundary

In a controlled environment, publish and approve a new same-account key through the normal signed approval path. Verify pending access is denied, an authenticated donor visit on a non-Messages page contributes, and the target later reads sent/received originals across at least three batches with the donor closed. Repeat for 31 incoming originals with all recipient donors absent and sender visits occurring before recipient reading. Exercise partial donors, multiple recipient accounts/keys, retry, reload, mobile/keyboard, restored-sender contributions and restored-recipient forwarding. Confirm originals are unchanged and uploads alone do not clear unavailable state or unread counts.

Temporarily make only the sync store unavailable: sync APIs should return a retryable error while ordinary sends/direct reads remain usable, without a global sync banner. Restore it and verify retained recovery. Rehearse paired and mismatched restore plus rollback, inspect no-store/auth/origin checks, and scan logs/public artifacts for private data. Run:

php tests/run.php PrivateMessageHistoryCryptoTest PrivateMessageHistorySyncTest PrivateMessageReaderTest PrivateMessageListTest PrivateMessageReleaseIsolationTest
node tests/browser/private_message_history_sync_browser.mjs
node tests/browser/private_message_history_sync_browser.mjs --sender
node tests/browser/private_message_history_sync_browser.mjs --direct
node tests/browser/private_message_history_sync_browser.mjs --scale
node tests/browser/private_message_list_browser.mjs

Automated local Chromium covers same-account and sender-assisted signed-approval journeys, partial donors, retained reload/forwarding, outage, paired restore, and existing messaging regressions. PHP tests cover concurrent writers, stale-key messages, aliases, revoked/foreign keys and mismatched restores. These are local checks, not production capacity or deployment evidence. Physical mobile keyboards and other browser engines remain manual follow-ups. No production deployment was performed.