My colleagues and I have been using Omnibus successfully by increasing the EC2 instance size and optimizing our formulas for better efficiency. However, we’ve encountered performance slowdowns occasionally, which we’ve mitigated by rebooting the instance. While this temporarily resolves the issue, it seems to have caused problems with our document file.
Although the document functions correctly within Grist, we’re now unable to import it. Additionally, when attempting to dump the file into MySQL, we encounter numerous data integrity issues.
Could you advise on how to resolve these issues? Specifically, we’re curious why the document works seamlessly in Grist but cannot be imported without errors.
Thank you.
A Grist document is an SQLite database. There are some general tips for repairing SQLite databases here:
https://www.sqlite.org/recovery.html
If the document has data integrity issues, but works within Grist, then you’re lucky! It may be the problems lie in action history, which can become quite large but is rarely used by the Grist app itself unless you are specifically looking through the history.
Slowdowns that get mitigated by reboots are a bit surprising! I’d be interested in more details about everything that is happening to you.
So our document is quite big, approximately a 2.1 GB with the history. There are times where someone would try to create formulas and it would display “Still Working” for seemingly an endless period of time and that’s where an instance reboot would take place and this would restart the whole hosted server and then we would be able to make changes on the document again.
I have noticed some of our snapshot does have issues, however, when trying to download the document without the history and re-importing, that process does not work and that downloaded document without the history has integrity issues which I would think is the reason why the imports are no longer working.
Ultimately since this document consists of a lot of our data for our project, there is the worry that if accidently deleted, the .grist file won’t be retrievable.
Did you get a chance to try Recovering Data From A Corrupt SQLite Database? It sounds like you have low level corruption at the SQLite-level. In that case there’s nothing really to be done at the Grist level. If you’re willing to share your file with me by some means, I’d be happy to take a look. I understand it is probably confidential.
That would be great, can I send the file via an email or the support email?
We placed the file onto DB browser and ran vacuum to get rid of the integrity issues for the most part but still not importable so if you would be able to take a look and see what else might be an issue that would be great.
I’d be happy to take a look. I can’t make any promises as to the result. I assume this is a large file? If you go to https://support.getgrist.com, click on the chat bubble in bottom right, then click on “Ask”, you can attach a file there and I think it’ll be handled better than email. Just mention my name and this thread. If you have any trouble and you’re comfortable with email, you can send it to me directly at paul@getgrist.com
Just sent an email with the attachment!
1 Like
@Tazwar_Belal1 I uploaded a version of file after going through this sequence:
sqlite3 <old-filename> .dump | sqlite3 <new-filename>
The new version opens for me (eventually!). Let me know if you get a chance to test it and compare it with the expected content. I can confirm that your file is indeed corrupt at a low-level. We’ve started some work to change the default SQLite settings used by Grist based on your feedback.
Life saver! Thank you very much!
