fix: resolve sync-loop hang + auto-key-forwarding + UTD visibility #3

Merged

jmz merged 1 commit from kitty/sync-wedge-resilience-fixes into main

2026-05-02 11:30:15 +00:00

AGit

kitty commented

2026-05-02 11:24:24 +00:00

Contributor

Root causes (three independent bugs, all fixed here):

Sync loop wedges silently on dropped long-poll connections.
client.rs::start_sync calls client.sync_once(timeout=30s) in a
tight loop. matrix-rust-sdk uses HTTP long-poll for /sync. When the
upstream silently drops the TCP connection (NAT idle timeout, network
flapping), the future awaits bytes that never arrive — no Err raised,
the Err arm of the loop never fires, the loop hangs inside await
forever. Fix: wrap each iteration in tokio::time::timeout(60s, ...).
The 30s long-poll timeout the homeserver uses + 30s of slack means a
genuine slow response still completes, but a dead connection now
abandons after 60s and reissues — reqwest opens a fresh connection
automatically.
read_messages returns stale data when sync is mid-recovery.
room.messages(MessagesOptions::backward()) paginates from the
room's prev_batch token in the local store, which is only as fresh
as the most recent successful sync. Fix: do a defensive
sync_once(timeout=0) (no long-poll, just "give me whatever the
homeserver has now") before each read, wrapped in a 15s outer
timeout so a wedged HTTP layer can't block reads. If the read-time
sync fails for any reason, fall through and serve local store
(degraded but not blocked).
automatic-room-key-forwarding feature flag was off.
Without this matrix-sdk feature, matrix-sdk-crypto's GossipMachine
defaults room_key_requests_enabled AND
room_key_forwarding_enabled to false
(gossiping/machine.rs:96-100). This means: (a) we never ask peer
devices for missing Megolm keys when we encounter a UTD, and (b) we
never respond to peers asking us for keys. Forward-direction key
sharing on send still works (that's unconditional), so new messages
decrypt fine — but recovery from any sync gap is impossible. Fix:
enable the feature in Cargo.toml.
UTD events were silently filtered out of read_messages.
mcp/tools.rs::read_messages had .filter(|m| !m.decryption_failed)
— events that failed to decrypt got dropped from the response,
making the room look like it had fewer messages than it actually
did. Without this filter we surface them as
[encrypted message — unable to decrypt] placeholders, which is
honest to the caller and makes the gap obvious.

Net effect:
After this patch, the bridge: (a) self-heals from network blips
within 60s, (b) always serves current homeserver state on read,
(c) properly participates in Megolm key gossip with peers, (d) never
silently swallows events.

Caveat for operators:
Other devices in the room (peer bridges, mobile/desktop clients) need
to ALSO have automatic-room-key-forwarding enabled before our key
requests get responses. Element clients have it on by default;
custom-built bridges (matrix-bridge-mcp on other hosts) need the
Cargo.toml change applied + binary rebuilt.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Symptoms (observed 2026-05-01): After ~3-6 hours of idle MCP traffic, the bridge would stop receiving new messages from peers. Restarting the bridge process would briefly appear to fix it, but reads still capped at the timestamp where the loop originally wedged. Encrypted messages from peers during the wedge window stayed permanently encrypted on the local device. Root causes (three independent bugs, all fixed here): 1. **Sync loop wedges silently on dropped long-poll connections.** `client.rs::start_sync` calls `client.sync_once(timeout=30s)` in a tight loop. matrix-rust-sdk uses HTTP long-poll for /sync. When the upstream silently drops the TCP connection (NAT idle timeout, network flapping), the future awaits bytes that never arrive — no Err raised, the Err arm of the loop never fires, the loop hangs inside `await` forever. Fix: wrap each iteration in `tokio::time::timeout(60s, ...)`. The 30s long-poll timeout the homeserver uses + 30s of slack means a genuine slow response still completes, but a dead connection now abandons after 60s and reissues — reqwest opens a fresh connection automatically. 2. **`read_messages` returns stale data when sync is mid-recovery.** `room.messages(MessagesOptions::backward())` paginates from the room's `prev_batch` token in the local store, which is only as fresh as the most recent successful sync. Fix: do a defensive `sync_once(timeout=0)` (no long-poll, just "give me whatever the homeserver has now") before each read, wrapped in a 15s outer timeout so a wedged HTTP layer can't block reads. If the read-time sync fails for any reason, fall through and serve local store (degraded but not blocked). 3. **`automatic-room-key-forwarding` feature flag was off.** Without this matrix-sdk feature, matrix-sdk-crypto's GossipMachine defaults `room_key_requests_enabled` AND `room_key_forwarding_enabled` to `false` (gossiping/machine.rs:96-100). This means: (a) we never *ask* peer devices for missing Megolm keys when we encounter a UTD, and (b) we never *respond* to peers asking us for keys. Forward-direction key sharing on send still works (that's unconditional), so new messages decrypt fine — but recovery from any sync gap is impossible. Fix: enable the feature in Cargo.toml. 4. **UTD events were silently filtered out of `read_messages`.** `mcp/tools.rs::read_messages` had `.filter(|m| !m.decryption_failed)` — events that failed to decrypt got dropped from the response, making the room look like it had fewer messages than it actually did. Without this filter we surface them as `[encrypted message — unable to decrypt]` placeholders, which is honest to the caller and makes the gap obvious. Net effect: After this patch, the bridge: (a) self-heals from network blips within 60s, (b) always serves current homeserver state on read, (c) properly participates in Megolm key gossip with peers, (d) never silently swallows events. Caveat for operators: Other devices in the room (peer bridges, mobile/desktop clients) need to ALSO have `automatic-room-key-forwarding` enabled before our key requests get responses. Element clients have it on by default; custom-built bridges (matrix-bridge-mcp on other hosts) need the Cargo.toml change applied + binary rebuilt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

kitty added 1 commit

2026-05-02 11:24:24 +00:00

fix: resolve sync-loop hang + auto-key-forwarding + UTD visibility 5d3cf0856d

Symptoms (observed 2026-05-01):
  After ~3-6 hours of idle MCP traffic, the bridge would stop receiving
  new messages from peers. Restarting the bridge process would briefly
  appear to fix it, but reads still capped at the timestamp where the
  loop originally wedged. Encrypted messages from peers during the
  wedge window stayed permanently encrypted on the local device.

Root causes (three independent bugs, all fixed here):

1. **Sync loop wedges silently on dropped long-poll connections.**
   `client.rs::start_sync` calls `client.sync_once(timeout=30s)` in a
   tight loop. matrix-rust-sdk uses HTTP long-poll for /sync. When the
   upstream silently drops the TCP connection (NAT idle timeout, network
   flapping), the future awaits bytes that never arrive — no Err raised,
   the Err arm of the loop never fires, the loop hangs inside `await`
   forever. Fix: wrap each iteration in `tokio::time::timeout(60s, ...)`.
   The 30s long-poll timeout the homeserver uses + 30s of slack means a
   genuine slow response still completes, but a dead connection now
   abandons after 60s and reissues — reqwest opens a fresh connection
   automatically.

2. **`read_messages` returns stale data when sync is mid-recovery.**
   `room.messages(MessagesOptions::backward())` paginates from the
   room's `prev_batch` token in the local store, which is only as fresh
   as the most recent successful sync. Fix: do a defensive
   `sync_once(timeout=0)` (no long-poll, just "give me whatever the
   homeserver has now") before each read, wrapped in a 15s outer
   timeout so a wedged HTTP layer can't block reads. If the read-time
   sync fails for any reason, fall through and serve local store
   (degraded but not blocked).

3. **`automatic-room-key-forwarding` feature flag was off.**
   Without this matrix-sdk feature, matrix-sdk-crypto's GossipMachine
   defaults `room_key_requests_enabled` AND
   `room_key_forwarding_enabled` to `false`
   (gossiping/machine.rs:96-100). This means: (a) we never *ask* peer
   devices for missing Megolm keys when we encounter a UTD, and (b) we
   never *respond* to peers asking us for keys. Forward-direction key
   sharing on send still works (that's unconditional), so new messages
   decrypt fine — but recovery from any sync gap is impossible. Fix:
   enable the feature in Cargo.toml.

4. **UTD events were silently filtered out of `read_messages`.**
   `mcp/tools.rs::read_messages` had `.filter(|m| !m.decryption_failed)`
   — events that failed to decrypt got dropped from the response,
   making the room look like it had fewer messages than it actually
   did. Without this filter we surface them as
   `[encrypted message — unable to decrypt]` placeholders, which is
   honest to the caller and makes the gap obvious.

Net effect:
  After this patch, the bridge: (a) self-heals from network blips
  within 60s, (b) always serves current homeserver state on read,
  (c) properly participates in Megolm key gossip with peers, (d) never
  silently swallows events.

Caveat for operators:
  Other devices in the room (peer bridges, mobile/desktop clients) need
  to ALSO have `automatic-room-key-forwarding` enabled before our key
  requests get responses. Element clients have it on by default;
  custom-built bridges (matrix-bridge-mcp on other hosts) need the
  Cargo.toml change applied + binary rebuilt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>