Error accessing document because of different docworker

Riel_Notermans · May 25, 2025, 12:07pm

In a kubernetes-environment, we run a few pods, docworkers and the home. Initially, this runs fine.
But, when the docworker-pods restart or the deployment creates new pods, the documents become unaccessible:

“You can try reloading the document, or using recovery mode. Recovery mode opens the document to be fully accessible to owners, and inaccessible to others. It also disables formulas. [Doc belongs to a different DocWorker (grist-doc-worker-68bbb7dd69-fcxh5): kHAzQq6qoQF8KiGUKn9X3o]”

I cannot really figure out the expected behaviour here, since we should be able to reboot everything and still end up with a working setup. Is this a configuration issue we oversee?

It is not really clear what the process of assisnging or finding docworkers is. I can consistently recreate this scenario.

Riel_Notermans · May 25, 2025, 1:09pm

I found a solution, but also added an issue:

github.com/gristlabs/grist-core

Grist doesn't detect unavailable workers or reassign documents automatically

opened 01:09PM - 25 May 25 UTC

rielzzapps

### Describe the current behavior **Problem: "Doc belongs to a different Do…cWorker" error when accessing Grist documents.** **What we found** 1. Document was assigned to dead worker: grist-doc-worker-68bbb7dd69-fcxh5 2. Current worker was different: grist-doc-worker-767cd8c9c9-kx6bw 3. Multiple stale entries in Redis: - Worker registrations; worker-grist-doc-worker-* keys - Document assignments; - Workers set: workers containing dead worker IDs **Solution applied** 1. Manually cleaned Redis entries: - Removed stale workers from workers set - Deleted old worker-* registration keys - Cleared document assignment keys (doc-*) **Result** Document now gets assigned to the current live worker and opens successfully. ### Steps to reproduce **Root cause:** - Grist stores doc worker assignments in Redis with no automatic cleanup - When Kubernetes pods restart/scale, old worker IDs persist in Redis - Documents remain assigned to dead workers that no longer exist - Grist doesn't detect unavailable workers or reassign documents automatically ### Describe the expected behavior Let Grist find a new available worker and clean redis by itself? ### Where have you encountered this bug? - [ ] On [docs.getgrist.com](https://docs.getgrist.com) - [x] On a self-hosted instance ### Instance information (when self-hosting only) * Grist instance: * Version: grist.demo.mosa.cloud | * Installation mode: kubernetes * Architecture: multi-workers