Advanced Monitoring for Grist Core

Developers
1

Advanced Monitoring for Grist Core: API Limitations vs Direct File Access

Context

I’m working on implementing comprehensive monitoring for our self-hosted Grist Core instance for governance purposes. We need to track user actions, document history, and usage patterns across all documents in our organization.

Current Implementation

I’ve successfully created a monitoring solution that:

  • Uses the Grist REST API to access team sites and workspaces
  • Analyzes document history using the /sql endpoint to decode action history payloads
  • Stores monitoring data in DigDash
  • Works well for documents the API key owner has access to

The Problem

API Limitation: The Grist API only provides access to team sites accessible to the account associated with the API key. This means:

  • :white_check_mark: Can monitor: Team workspaces where the user is a member
  • :white_check_mark: Can monitor: The API key owner’s personal site
  • :x: Cannot monitor: Other users’ personal sites
  • :x: Cannot monitor: Team sites where the API user isn’t a member

For comprehensive governance monitoring, we need visibility into ALL documents across the instance.

Potential Solutions I’ve Identified

Solution 1: Admin API Key (Enterprise Only?)

Based on Grist’s admin controls documentation, there might be administrative API access, but this appears to be limited to Grist Enterprise, not Grist Core.

Solution 2: Direct File System Access

Since Grist Core stores documents as SQLite files in /persist/docs/, we could:

  • Read all .grist files directly from the file system
  • Parse SQLite databases to extract action history and metadata

Questions:

  • Has anyone implemented monitoring by directly accessing Grist’s SQLite filesin /persists/docs/ ?

Any insights, experiences, or alternative approaches would be greatly appreciated!

4 Likes
2

Hello @Arthur ! The Audit log feature seems close to the monitoring that you are looking for.

It is an Enterprise-only feature, because it’s typically needed by larger organizations. Since Grist is built primarily as an open-source product that invididuals and small orgs can self-host freely, we (Grist Labs) rely on sales to larger organizations to subsidize its development. We hope that Enterprise features would be useful enough to the larger orgs, that it would be worth it for them to pay for an Enterprise license rather than rebuild functionality. (And more Enterprise features are coming!)

I’d be interested in discussing how to enable your organization to accept an Enterprise version, both in terms of agreeing on a price, and on a license (e.g. it could be one of the open-source-like source-available licenses if that’s what’s important).

3

@dmitry-grist

My personal opinion:

Surely, enterprise support is necessary (for Grist’s current business model) i guess, to keep the project afloat and even thrive.

But let’s not distort reality for its sake.

Audit logs, notifications…these features aren’t limited to enterprise, or even large business, use-cases these days, Open Source project or not.

It really is a shame that so much paywalling is still in practice many in Open Source projects.

4

I’ve been thinking a lot about this (I have an interest: I am the founder and primary investor in Grist). I have a blog post where I try to get at the root of the problem: Sovereign Lumber.

And I’ve been thinking more over the last week, which I spent at the UN Open Source Week.

Perhaps some day public tax money can fully fund excellent software that would be better than the commercial kind. It seems neither realistic nor fair. Until then, it makes more sense for users (whether private or public) to pay something proportional to the value they are deriving: less than they might pay to the commercial software giants, but enough to support the initial development, innovation, and maintenance.

Of course, the paid features are useful for smaller organizations too. Smaller organizations are welcome to pay for the license (if our pricing model seems off, reach out and we’ll have a sales call).

There are licenses like Directus’s BUSL (which gives free usage permission to orgs that aren’t too large) or FCL (which provides open-source protections except for circumventing license key functionality). I wonder if Grist open-source community (which @Shrey is a vocal advocate for), would find something like this a more attractive way for Grist to be licensed than our current open-core approach.

Without a way to monetize (or “paywall” as some say), I see no way for the project to stay afloat, much less thrive.

2 Likes
5

Hello @dmitry-grist,

I’m taking the liberty to respond because this is a crucial topic for Grist’s future, an exceptional tool you’ve created that could benefit to local authorities in France. I’ll share my business perspective and explain concretely what I can or cannot unlock in terms of budget as a project manager, as well as my vision of what constitutes a healthy and viable economic model.

On the economic model

I strongly disagree with the criticisms mentioned. You are absolutely right to reserve certain features for the paid version to establish a sustainable economic model that allows you to generate sufficient revenue. This is not only legitimate but necessary for the project’s sustainability.

Our organization and budget constraints

Let me explain how we plan to structure our Grist deployment, as it illustrates well the challenges faced by organizations like ours:

We plan to divide our usage into three distinct workspaces:

  1. “Personal Space”: An environment where users freely experiment with the tool. Some will open it for 2 minutes out of curiosity, others will share it with 2-3 colleagues for occasional testing. It’s a space for discovery and learning.
  2. “Transition Space”: Thanks to monitoring (hence its crucial importance), I can identify files that are starting to become important in daily operations. At this stage, I help users professionalize their usage, secure their data, and anticipate maintenance needs.
  3. “Critical Space”: Applications that have become essential to our services’ functioning. If these files went down, it would have a critical impact on our activity. These files have mostly locked structures and need maintenance and evolution over time.

Budget reality

My budget is constrained, and my organization will never grant me €22 licenses per user for people who use the tool sporadically. This pricing would be perceived as disproportionate to the value generated.

However, it would be much easier to justify and negotiate a €22 rate per license for the critical space, where we would have:

  • Proven files used daily
  • A real need for support to avoid service interruptions
  • GristLabs man-days for file maintenance and evolution
  • ETL to our warehouse

Prospects

In any case, I hope to negotiate compensation for GristLabs in the coming months (subject to agreement with our finance department), whether in the form of a support contract or a partial enterprise contract as mentioned above.

This approach seems balanced to me: it recognizes the value of your work while taking into account the real budget constraints of public organizations.

Additional quick questions:

  • Why is the Grist Enterprise price the same for internal and external machines?
  • €22 remains high, have you considered splitting licenses by user type (simple user / Grist creator)?
  • What ETL services do you offer with Grist in the Enterprise version?

PS: On my side, I need monitoring anyway, because without it I’m unable to identify files that could become critical, so my critical space remains empty.

2 Likes
6

Hi @Arthur! Thank you for the detailed message. It’s great to hear compatible opinions on the user side! It is, of course, in our interest, to come up with pricing and licensing that make Grist both appealing to large organizations, and possible to get approval from finance departments :slight_smile:

There is some complexity in the details, since it’s pretty clear that our Enterprise pricing – though simple – is not feasible in your case. Would you be open to a conversation? You can schedule a time with me here: https://calendly.com/d/cwrz-6jx-yf5/30-minute-eu.