I’m happy to present Pygrister GitHub - ricpol/pygrister a Python wrapper for the Grist API. This grew out of a contingent need for another project, after realising that the existing Python library is lagging rather far behind.
Pygrister covers all the documented Grist APIs, comes with documentation and a test suite, also useful to have some working examples.
At the moment it is still pretty raw, and I haven’t published it on PyPI yet. You may install it directly from Github, if you want to take it for a spin - careful, though: it should work, more or less, but it’s not ready for production.
Feedback is always appreciated!
7 Likes
Wow, glad I made it into the newsletter!
In the meantime, I have done quite a bit of work on Pygrister, uniforming and simplifying the apis, adding support for self-hosted Grist url patterns, and so on.
I have released version 0.3 just a few hours ago on Github, and I feel that overall the apis are more stable now. I think I’m ready to release the thing on PyPI in the next few days…
4 Likes
Wow,
Pygrister sounds like a much needed update for interacting with the Grist API. You needed appriciation for adding support for self-hosted URLs and the effort to make the APIs more stable. Looking forward to seeing it on PyPI soon.
All the best!
1 Like
Ta-da!, it’s on PyPI now… Pygrister · PyPI so this means that you may install it with python -m pip install pygrister…
What’s next? Probably supporting a few on the undocumented apis, and adding at least a basic type system for records… I should also look into the Requests support for keep-alive connections, and see if pygrister could benefit from requests sessions… we’ll see…
.
1 Like
Oh well, after much debate and self-doubt, I finally figured out a way to provide type support in Pygrister.
It’s committed to the repo, but not yet released on PyPI… so if you want to try it out, you’ll have to install directly from GitHub.
Writing the documentation is the part that took me the longest, and it says a lot about my doubts about all this…feedback is always appreciated!
1 Like
Hey @Riccardo_Polignieri.
Really nice documentation, really informative. I didn’t realise many of those inconsistency issues existed with the SQLite later / GUI / API.
Reading it through, it feels like you’ve put in a lot of work to bypass/fix many of the inconsistencies on Grist’s side.
Given you’ve just done a deep dive into all of this, are there any changes we could make to Grist’s API that would help simplify type handling and reduce some of this complexity? 
Ha! Now that’s a though question…
Broadly speaking, I think that Grist usually does the good, common-sense thing, given the wide array of possibilities supported, and that consistency is the hobgoblin etc. etc. Truth is, most of what I changed in Pygrister is also a matter of taste. I don’t know how much of this actually needs to be fixed (plus, if you start changing the APIs now, I’ll have to fix Pygrister too!)
That being said, in writing Pygrister I came across 3 kinds of “personal distaste”:
- a few APIs returns are just too complex/nested: those I simplify a little;
- a few APIs are, in fact, inconsistent;
- recently, I surveyed quite a few oddities about the type system in specific, but these are not really a problem with the API, and I choose not to get involved (Pygrister only provides a hook for custom converters… let the user deal with every specific case!).
As for the first two problems, if you just browse my code (start from line 300) you’ll be able to spot them very easily:
- every time a Pygrister function returns something like
return self.apicall(url, ...), this basically means that I agree with the underlying Grist API and return it unchanged - TBH, this is the most common case; - when I start writing things like
return res['whatever']or worse, this is where I think the original API needs some massaging.
Now, most of the time it’s just because Grist APIs like to return an object with a label attached that just says “object”. For instance, {'users': [<an array of users]} instead of just [<an array of users>]. Or, even worse to me, {'tables': [{'id': 'tablea'}, {'id': 'tableb'}, ...]} instead of just ['tablea', 'tableb'] when you are just returning a list of IDs. In such cases I want to simplify and cut to the chase but maybe Grist has its own good reasons too (maybe it’s even some sort of standard somewhere? My ignorance about rest api design is impossible to overestimate).
The format of records is a particular case where I got annoyed by the Grist APIs, as I documented. To me, a “record” must be a dictionary, and a “list of records”, just a list of dictionaries. This is also a very common way of doing things in the popular sqlite3 Python library. Working my way around the various Grist record formats resulted in a few nasty one-liners that really made my day!
A couple of times I just gave up: for the sake of simplicity, in Pygrister both GET workspaces/id/access and docs/id/access APIs leave the maxInheritedRole bit out from the returned result. This is because, at this point, I had a rather elegant and consistent schema in mind, about what kind of objects the various Pygrister functions shall return (it is detailed here), and these two Grist APIs just did not fit.
Then we have the occasional case of plain inconsistency in the Grist API, that I “fixed” in Pygrister. For instance, a couple of webhooks APIs return a rather odd success message {success: true} that I simply choose to ignore.
This is about the output values in the Grist APIs… It’s not that I didn’t have my eyebrows raised with the parameters too, at times: but here I choose a more conservative approach, staying always close the original API (mainly because if I started changing too many things, then I would have to write my own docstrings, instead of just pointing to the Grist api reference). (The notable exception is the already mentioned unified record format: add_records, for instance, accepts a “Pygrister record list”, then converts it to Grist’s own format.)
I remember being mildly annoyed by the widgetOptions field in column APIs, which required a separate treatment.
I am not a fan of the complex filter option that has to be url-encoded in the path
I plainly dislike the choice of doubling parameters in the docs/id/attachments API (Pygrister will just put them in the url).
Sometimes I wish I had a better Grist documentation: for instance, parameters’ defaults are missing, and this forced me to always include some of them (well, I didn’t take the time to test, tbh)… Or, I never quite figured out the noparse parameter in the records APIs…
Or, what’s the matter with the PATCH docs/id/tables API, having a completely different body schema than its POST version? (I even left an angry note in my tests, from a moment when I haven’t realized this yet. At some point I’ll have to revisit and write a better example/test, but I mean… grrrr…)
I am not speaking here of the overall api design… It has been noticed that some needed APIs are missing and so on… But it’s a broader scope than Pygrister’s concerns…
Then, there is the whole matter of the Sqlite/Grist GUI/Grist API type system to consider. I have decided that Pygrister should not be involved, as I said. However, while researching for a viable strategy, I made a test table for types and I found a few oddities. But this post is getting way too long. I’ll add more about this in a few days…
1 Like