GRIST-Omnibus Integrate image (data) in Markdown, HTML from self hosted LAN

[Sylvain_Page]

Hello Grist community,

Again thanks for your efforts in (Grist, forum, communications, documentation, listening, …)

I’m having trouble integrating data images into templates (HTML, markdown, etc.). On the cloud, using the API for attachments poses no issue, as described here Photo Markdown - Grist, provided the document is public.

However, when attempting this locally on a self-hosted Grist as below :

• Docker omnibus installation and email/password authentication were successful.
• Since it’s self-hosted on a LAN, the URL follows the format http://192… with no HTTPS.
• I tried incorporating a render_html custom widget with the following code:

html

<h1>Test</h1>
<img src=http://192.168.1.25:9999/api/docs/3o1CbBsHfEKR/attachments/1/download>

Unfortunately, the rendered images appears broken.

• A media type column with a similar link works well, allowing image downloads.
• The link appears correct in the DOM inspector, but the console shows a request transformation to HTTPS, resulting in a 404 error.

Suspecting sanitization, I removed DOMPurify from the render_html widget, but encountered the same issue.

While not well-versed in network details, it seems there’s a normal secure request transformation dut to the use of external ressources to render html (render_html custom widget served from grist github).
Is it correct ?

If confirmed, do serving custom_plugins locally on the same machine would might resolve the issue, ensuring all requests originate from the same source and bypass HTTPS transformation ?

If yes , what would be the simplest way to serve custom_widgets from ?

Many thanks for your clarifications

[Sylvain_Page]

Here’s the most simple I solved this → IIS with W10.
Shortly gave a trial to Apache but it needed additionnal package (VCRruntime140 error, seems link to .dll missing etc …)

So I lanch a website through IIS serving on locahost:80, put all grist_pluging in it.
It works as expected (I know this seems so simple network experts …) with renderhtml & markdown without CORS issues.

[paul-grist]

Hi @Sylvain_Page, thanks for reporting what worked for you. I had trouble understanding your initial scenario but now I think I get it. Might be an argument for having grist-electron bundle more of the custom widgets locally, like you ended up doing manually. There is a bundling mechanism sitting there ready to go, but needs a developer to adopt the project and give it some love :-). So I’m glad you found a workaround.

[Sylvain_Page]

Hi @paul-grist,
I had the issue with both Grist-Electron and Docker, I am not sure it is linked to Grist-Electron itself but more to being behind a http protocol (no https)…

As far as I understood:

• it is not possible to run Grist-Omnibus with a https://192.168.XX.XX :8484 (neither it is for Grist-Electron). So I must use a http protocol.
• When using plugins, ressources coming from external source seems rejected by the browsers.
  → This why I tried to serve them on the same machine through a standard http protocol
  (though of course less secureed, yet balanced by the fact there is no external access).

[paul-grist]

@Sylvain_Page

• it is not possible to run Grist-Omnibus with a https://192.168.XX.XX :8484 (neither it is for Grist-Electron). So I must use a http protocol.
• When using plugins, ressources coming from external source seems rejected by the browsers.

While letsencrypt isn’t usable for local ip addresses, I think you could bring your own self-signed cert if you want to use https locally, with the HTTPS=manual setting.

The same bundling mechanism available in grist-electron is also available in grist-core (and therefore in grist-omnibus), but last time I looked at it, it just bundled the calendar widget, since that is tightly integrated with the app.

[Emanuele_Gissi]

I probably missed the capability of locally bundling the widgets.
Where can I find any instruction on how to do that?

[Sylvain_Page]

@Emanuele_Gissi ,

Not sure there is specific documentation, it is not really bundled.

The main idea is to serve the files → this ones : GitHub - gristlabs/grist-widget: A repository of custom widgets to embed in Grist documents locally
(yet your server must have internet access to use the javascript stuff modules stored on cdn…)

These are standard static Html webfiles.
So basically simple server should work. On W10 I used IIS (Internet Information Service) because it is included (just need to validate parameters) and with few clicks you can host a static web file whose root is on a folder you choose. And also because I have limited resources and want to keep it as simple as possible.

I guess you can make : python -m http.server provided you have a python distribution locally.

So to summarize :
1 - Install a local webserver (Apache, python,…),set-up ports (different from Grist,… )
2 - locate your root docs ( from the environnement datas of your server)
3 - Download Grist plugins from here in a zip file → GitHub - gristlabs/grist-widget: A repository of custom widgets to embed in Grist documents
4 - extract all the zip stuff in the root docs of your local server
5 - launch the server
6 - In Grist when you add the custom_widget, choose custom_url instead of choosing the widget type.
7 - fill the url served by your local server : http:/localhost/renderhtml (the grist_widget directory if you did not changed the name), you may have http://localhost:[port]/[some_directory]/renderhtml depending the way you extracted the .zip
8 - Of course you can add your own widget in its own directory

Hope it helps…

[paul-grist]

@Sylvain_Page’s solution is very clear.

For completeness, I’ll mention an entirely undocumented method for bundling that Grist uses internally for calendars. Grist has a plugins directory, and material in there will be automatically served by Grist, so no need for a separate webserver.

For custom widgets, the structure would be like mentioned here in this test:

gristlabs/grist-core/blob/9ce8ed3f25884bb482893e5a6c042908ac8773d8/test/nbrowser/CustomWidgets.ts#L755-L765

The my-widgets directory name is arbitrary. The manifest.yml file can just be something like:

name: My Widgets
components:
  widgets: widgets.json

And widgets.json is in the same format as https://gristlabs.github.io/grist-widget/manifest.json except that URLs can be relative to the current directory, so you can just put html/css/js material there. There’s an example in this comment.

The grist-widget repo can pack up widgets in the expected format with the command run by yarn run prepack but that is even less clear how it works, you’d basically have to read the code.

[jperon]

@paul-grist Would it be possible, building upon this feature, to build a custom version of grist-electron integrating custom widgets? I’d find it very interesting for pug_py.

[paul-grist]

Yes, it would be. Part of the motivation for the bundling work was to make sure grist-electron could run offline with selected custom widgets. We needed this since we implemented the Calendar widget as a custom widget rather than something built into the codebase.

There’s a lot of cool stuff really close to working here. At Grist Labs, we had to shelve this line of work in favor of other priorities (like forms) but if you were interested in poking at it I could give some pointers

[jperon]

Interested? Without any doubt!

[Emanuele_Gissi]

@paul-grist
I am trying to apply the solution you proposed for having a local stable repo of custom widgets with no external dependency.

I set up the GRIST_USER_ROOT=/persist/user/ environment variable.
And populated the user directory, with a plugins directory. Inside it I created the my-widgets directory.

Then I put the actionbutton widget dir in the plugins directory, and set up the manifest.yml:

name: my-widgets
components:
  widgets: widgets.json

and the widgets.json file:

[
  {
    "name": "Action button",
    "url": "./actionbutton/index.html",
    "widgetId": "@eg/widget-actionbutton2",
    "published": true,
    "accessLevel": "full",
    "renderAfterReady": true
  },
]

Here is the final tree:

/persist/user/plugins/my-widgets/
--> actionbutton
--> manifest.yml
--> widgets.json

But when I run, grist crashes and the following error is sent to the log:

apps-grist    | SyntaxError: Unexpected token ] in JSON at position 203
apps-grist    |     at JSON.parse (<anonymous>)
apps-grist    |     at DiskWidgetRepository.getWidgets (/grist/_build/app/server/lib/WidgetRepository.js:65:30)
apps-grist    |     at async CombinedWidgetRepository.getWidgets (/grist/_build/app/server/lib/WidgetRepository.js:108:32)
apps-grist    |     at async CombinedWidgetRepository.getWidgets (/grist/_build/app/server/lib/WidgetRepository.js:108:32)
apps-grist    |     at async CachedWidgetRepository.getWidgets (/grist/_build/app/server/lib/WidgetRepository.js:223:22)
apps-grist    |     at async FlexServer.finalizePlugins (/grist/_build/app/server/lib/FlexServer.js:1530:32)
apps-grist    |     at async main (/grist/_build/app/server/mergedServerMain.js:148:9)
apps-grist    |     at async main (/grist/_build/stubs/app/server/server.js:144:20)

I have been working for two days on this, but I cannot find the offending json file…
Any idea?

[paul-grist]

I think a problem may be the comma in },. A comma with no item following it is fine in JavaScript but not in JSON.

[Emanuele_Gissi]

Thank you Paul I’ll try that first thing tomorrow morning and let you know. Thanks again

[Emanuele_Gissi]

You were right @paul-grist, the comma was the problem (sorry for not thinking to that!).
Now the local widgets are visible:

Unfortunately there is another hurdle.

I prepare a test case on getgrist.com to be sure that the actionbutton works:

Then I move the test case locally, and update the link to the custom widgets to the local ones (Action button (my-widgets)).

It seems that the local Grist fails to load the local custom widget. There is no ACTION config panel, no buttons are shown:

I double checked local widget file permissions. They should be ok. And in fact the widget json is read.

There is no hint of error in the log either.

As another double check, I confirm that the same widget, but taken from the remote repository, works:

Where can I look for debugging?

[paul-grist]

@Emanuele_Gissi

Where can I look for debugging?

Your browser developer console should be able to help, it should hopefully show some page being fetched and failing because the URL is silly. What are you using to serve Grist? When serving plugins, Grist wants them to be served from a separate origin so that the browser will prevent them from having too much access to the main part of Grist. There are flags for configuring this. If you trust your plugins, the easiest setup is with GRIST_TRUST_PLUGINS=1. That will serve plugins along with the Grist app.

[Emanuele_Gissi]

GRIST_TRUST_PLUGINS=1 fixed the issue!
Thank you again

[Emanuele_Gissi]

Would you like me to write a short howto that you can add to the documentation?

What about setting up a wiki for contributing draft documentation pages for your evaluation and eventual inclusion into official documentation?

[paul-grist]

@Emanuele_Gissi

What about setting up a wiki for contributing draft documentation pages for your evaluation and eventual inclusion into official documentation?

Thanks for offering @Emanuele_Gissi! In the past I’ve found running a wiki surprisingly high maintenance. What do you think of making a post in Showcase - Grist Creators synthesizing what you got going? It isn’t a wiki since others won’t be able to edit what you wrote, but they can add comments. And the material would be there ready for anyone up for adding it to https://support.getgrist.com (which is backed by the grist-help repo) .