Grist Sqlite New Environment Variables

Ask for Help
1

I noticed there are new environment variables regarding SQLite that can be initialized to lower potential data corruptions. Before initializing the variables I just wanted to ask a few questions regarding the potential impacts.

  1. Does changing to GRIST_SQLITE_MODE=wal automatically convert our existing documents within the work space?

  2. Will the existing documents that have corruptions prevent us from doing initializing the variables?

  3. Will the document ID change when doing this?

  4. Will we be able to migrate/repair the documents directly in the future if needed?

  5. And what exactly are the performance/stability benefits?

Thank you!

2

Hi @Tazwar_Belal ! Thanks for your questions.

  1. Setting wal mode will change how all existing documents are opened, as they are opened. There is no mass conversion. Documents that have not been opened are untouched.
  2. Existing corrupt documents may remain unopenable but won’t prevent opening other documents.
  3. Document IDs will not change.
  4. The documents remain SQLite documents. When they are open by Grist, you will see two parallel files, a -shm and -wal file. As you make changes, the -wal file will change and only occasionally get flushed to the main file. When Grist is shut down and disconnects from documents the -shm and -wal file will disappear as SQLite integrates all changes into the original. All SQLite-level tools for data recovery remain usable.
  5. In wal mode, doing what you were doing of opening a doc and then rebooting while it was potentially in the middle of doing writes to that doc for formulas, should now be a safe operation. So there’s a stability improvement. Performance is a bit better in some cases, may be slightly worse in others, that’s not a major motivating factor.

Also please remember to MAKE BACKUPS before making operational changes like this.

2 Likes