Resynchronizing snapshots with S3 versions?

Florent_F · September 20, 2023, 3:06pm

In our production instance, we had to manually remove versions of documents in our S3 bucket.

Do you have a proper way to remove snapshots whose versionId does not exist anymore?

If not, I can actually share with you a script (which transforms the meta.json file), but I would like to know if there exist some cleaner ways by chance.

paul-grist · September 20, 2023, 3:47pm

In principle, if you just delete the meta.json file from the external store and from whatever worker is hosting the document (if any), it should get recreated. Granted, that’s not something I’ve done outside of tests, and it could also be a little slow if there are very many versions of the document that remain.

paul-grist · September 20, 2023, 3:57pm

There is an endpoint for removing snapshots with some options:

github.com

gristlabs/grist-core/blob/1b1970c07546ef8b1d16566710f8ac4d4372ac5f/app/server/lib/DocApi.ts#L939-L967


      
          this._app.post('/api/docs/:docId/snapshots/remove', isOwner, withDoc(async (activeDoc, req, res) => {
            const docSession = docSessionFromRequest(req);
            const snapshotIds = req.body.snapshotIds as string[];
            if (snapshotIds) {
              await activeDoc.removeSnapshots(docSession, snapshotIds);
              res.json({snapshotIds});
              return;
            }
            if (req.body.select === 'unlisted') {
              // Remove any snapshots not listed in inventory.  Ideally, there should be no
              // snapshots, and this undocumented feature is just for fixing up problems.
              const full = (await activeDoc.getSnapshots(docSession, true)).snapshots.map(s => s.snapshotId);
              const listed = new Set((await activeDoc.getSnapshots(docSession)).snapshots.map(s => s.snapshotId));
              const unlisted = full.filter(snapshotId => !listed.has(snapshotId));
              await activeDoc.removeSnapshots(docSession, unlisted);
              res.json({snapshotIds: unlisted});
              return;
            }
            if (req.body.select === 'past') {
              // Remove all but the latest snapshot.  Useful for sanitizing history if something

This file has been truncated. show original

None of the options are what you want. unlisted for sure is not (I think it is the reverse). past can be handy for getting rid of unwanted history. But it could be a place to put the kind of repair you want (essentially just to regenerate the meta.json file)

Florent_F · September 20, 2023, 4:06pm

Interesting! I’ll try using the endpoint. That would probably be cleaner that overriding the meta.json file.