Bad gateway with grist-omnibus docker when accessing from LAN

Hi, I’ve been trial-ing Grist with a simple single-user docker and now I am trying a multi-user LAN setup with the grist-omnibus docker.

I’ve tried the suggested default docker run arguments as well as my own, and also tried all the tips from this thread (making sure the TEAM variable value is lowercase and removing Google/Microsoft connectors): Unable to open URL with grist-omnibus docker in TrueNAS

I don’t receive any errors in the container log other than some not ready: FetchError lines before getting 200 on dex. When I try to connect using the browser, I get Bad Gateway.

I suspect that it may be because I am trying to access the app over the LAN (i.e. http://192.168.1.147:9999 in browser), as the container is running on a local server without a desktop interface (but I have no way of testing this). I’ve tried setting the URL variable to both localhost as well as the LAN address of the server, and have also tried a variety of ports with no success.

LOGOUT_REDIRECT=http://localhost:9999/signed-out
TEAM=cool-beans
APP_HOST=localhost
GRIST_ORG_IN_PATH=false
WHOAMI_PORT=17102
HOSTNAME=a41f84a270f1
URL=http://localhost:9999
HOME=/root
APP_HOME_URL=http://localhost:9999
GRIST_FORCE_LOGIN=true
EXT_PORT=9999
GRIST_HIDE_UI_ELEMENTS=helpCenter,billing,templates,multiSite,multiAccounts
GRIST_SANDBOX_FLAVOR=gvisor
GRIST_FORWARD_AUTH_LOGOUT_PATH=_oauth/logout
DEX_PORT=9999
TERM=xterm
DEFAULT_PROVIDER=oidc
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
EMAIL=owner@example.com
GRIST_DEFAULT_EMAIL=owner@example.com
GRIST_SINGLE_ORG=cool-beans
PROVIDERS_OIDC_CLIENT_ID=q391r3FNaAtlXYky5mVt
PROVIDERS_OIDC_CLIENT_SECRET=SKZWle5pSASm4mY2VAU2
SECRET=qY8s7xzuFynvPzWOVoEx
GRIST_SESSION_SECRET=Ice4wRmKxbvv9vOtvnAP
PWD=/
TFA_PORT=17101
PASSWORD=topsecret
PROVIDERS_OIDC_ISSUER_URL=http://localhost:9999/dex
GRIST_FORWARD_AUTH_HEADER=X-Forwarded-User
GRIST_PORT=17100

2023/04/16 21:53:44 Starting up on port 17102
INFO[0000] Configuration loaded from flags.
No /custom/dex.yaml
Checking dex... at http://localhost:9999/dex/.well-known/openid-configuration
  not ready: FetchError: request to http://localhost:9999/dex/.well-known/openid-configuration failed, reason: connect ECONNREFUSED 127.0.0.1:9999
time="2023-04-16T21:53:45Z" level=info msg="Dex Version: v2.33.1-dirty, Go Version: go1.18.4, Go OS/ARCH: linux amd64"
time="2023-04-16T21:53:45Z" level=info msg="config using log level: debug"
time="2023-04-16T21:53:45Z" level=info msg="config issuer: http://localhost:9999/dex"
time="2023-04-16T21:53:45Z" level=info msg="config storage: sqlite3"
time="2023-04-16T21:53:45Z" level=info msg="config static client: Grist"
time="2023-04-16T21:53:45Z" level=info msg="config connector: google"
time="2023-04-16T21:53:45Z" level=info msg="config connector: microsoft"
time="2023-04-16T21:53:45Z" level=info msg="config connector: local passwords enabled"
time="2023-04-16T21:53:45Z" level=info msg="config skipping approval screen"
time="2023-04-16T21:53:45Z" level=info msg="config refresh tokens rotation enabled: true"
Checking dex... at http://localhost:9999/dex/.well-known/openid-configuration
  not ready: FetchError: request to http://localhost:9999/dex/.well-known/openid-configuration failed, reason: connect ECONNREFUSED 127.0.0.1:9999
Checking dex... at http://localhost:9999/dex/.well-known/openid-configuration
  not ready: FetchError: request to http://localhost:9999/dex/.well-known/openid-configuration failed, reason: connect ECONNREFUSED 127.0.0.1:9999
time="2023-04-16T21:53:45Z" level=info msg="keys expired, rotating"
Checking dex... at http://localhost:9999/dex/.well-known/openid-configuration
  not ready: FetchError: request to http://localhost:9999/dex/.well-known/openid-configuration failed, reason: connect ECONNREFUSED 127.0.0.1:9999
time="2023-04-16T21:53:45Z" level=info msg="keys rotated, next rotation: 2023-04-17 03:53:45.590367697 +0000 UTC"
time="2023-04-16T21:53:45Z" level=info msg="listening (http) on 0.0.0.0:9999"
Checking dex... at http://localhost:9999/dex/.well-known/openid-configuration
  got: 200
Happy with dex
Starting traefik-forward-auth
I think everything has started up now
Listening internally on 80, externally at http://localhost:9999

Hi, there. I am back. :rofl:

I ran into the same problem that @Ethan had.

My grist is good untill this morning. I watched the Custom Widgets Webinars this morning and these widgets look super useful. But I cannot find the list of Premade Custom Widgets as shown in the red box of the figure. I suspected the gristlabs/grist-omnibus image needs to be upgraded, so I repulled the image to the container. Then, I got the Bad Gateway. Here is the log:

2023-04-17 09:33:04.085741+00:00Calling traefik [
2023-04-17 09:33:04.085788+00:00'--providers.file.filename=/settings/traefik.yaml',
2023-04-17 09:33:04.085794+00:00'--entryPoints.web.address=:80'
2023-04-17 09:33:04.085798+00:00]
2023-04-17 09:33:04.086535+00:00/grist/sandbox/run.sh: line 6: ./sandbox/gvisor/update_engine_checkpoint.sh: No such file or directory
2023-04-17 09:33:04.087648+00:00LOGOUT_REDIRECT=http://my.public.ip.address:28484/signed-out
2023-04-17 09:33:04.087665+00:00TEAM=shanghai
2023-04-17 09:33:04.087669+00:00KUBERNETES_SERVICE_PORT=443
2023-04-17 09:33:04.087672+00:00KUBERNETES_PORT=tcp://172.17.0.1:443
2023-04-17 09:33:04.087674+00:00APP_HOST=my.public.ip.address
2023-04-17 09:33:04.087683+00:00GRIST_ORG_IN_PATH=false
2023-04-17 09:33:04.087685+00:00WHOAMI_PORT=17102
2023-04-17 09:33:04.087688+00:00HOSTNAME=grist-ix-chart-79f9f8cd84-64w29
2023-04-17 09:33:04.087690+00:00URL=http://my.public.ip.address:28484
2023-04-17 09:33:04.087693+00:00HOME=/root
2023-04-17 09:33:04.087698+00:00APP_HOME_URL=http://my.public.ip.address:28484
2023-04-17 09:33:04.087701+00:00GRIST_FORCE_LOGIN=true
2023-04-17 09:33:04.087704+00:00INSECURE_COOKIE=true
2023-04-17 09:33:04.087708+00:00EXT_PORT=28484
2023-04-17 09:33:04.087711+00:00GRIST_IX_CHART_SERVICE_HOST=172.17.183.167
2023-04-17 09:33:04.087713+00:00PGID=1000
2023-04-17 09:33:04.087718+00:00GRIST_HIDE_UI_ELEMENTS=helpCenter,billing,templates,multiSite,multiAccounts
2023-04-17 09:33:04.087721+00:00GRIST_SANDBOX_FLAVOR=gvisor
2023-04-17 09:33:04.087723+00:00GRIST_FORWARD_AUTH_LOGOUT_PATH=_oauth/logout
2023-04-17 09:33:04.087726+00:00DEX_PORT=9999
2023-04-17 09:33:04.087731+00:00GRIST_IX_CHART_PORT=tcp://172.17.183.167:80
2023-04-17 09:33:04.087733+00:00GRIST_IX_CHART_SERVICE_PORT_IX_GRIST_28484_0=80
2023-04-17 09:33:04.087736+00:00GRIST_IX_CHART_SERVICE_PORT=80
2023-04-17 09:33:04.087739+00:00DEFAULT_PROVIDER=oidc
2023-04-17 09:33:04.087741+00:00KUBERNETES_PORT_443_TCP_ADDR=172.17.0.1
2023-04-17 09:33:04.087746+00:00PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-04-17 09:33:04.087748+00:00EMAIL=example@email.com
2023-04-17 09:33:04.087751+00:00KUBERNETES_PORT_443_TCP_PORT=443
2023-04-17 09:33:04.087753+00:00GRIST_DEFAULT_EMAIL=example@email.com
2023-04-17 09:33:04.087756+00:00GRIST_SINGLE_ORG=shanghai
2023-04-17 09:33:04.087760+00:00PROVIDERS_OIDC_CLIENT_ID=...
2023-04-17 09:33:04.087763+00:00KUBERNETES_PORT_443_TCP_PROTO=tcp
2023-04-17 09:33:04.087766+00:00PROVIDERS_OIDC_CLIENT_SECRET=...
2023-04-17 09:33:04.087769+00:00GRIST_IX_CHART_PORT_80_TCP_ADDR=172.17.183.167
2023-04-17 09:33:04.087774+00:00PUID=1000
2023-04-17 09:33:04.087776+00:00GRIST_IX_CHART_PORT_80_TCP_PORT=80
2023-04-17 09:33:04.087780+00:00SECRET=...
2023-04-17 09:33:04.087783+00:00GRIST_IX_CHART_PORT_80_TCP_PROTO=tcp
2023-04-17 09:33:04.087786+00:00GRIST_SESSION_SECRET=...
2023-04-17 09:33:04.087788+00:00KUBERNETES_PORT_443_TCP=tcp://172.17.0.1:443
2023-04-17 09:33:04.087792+00:00KUBERNETES_SERVICE_PORT_HTTPS=443
2023-04-17 09:33:04.087795+00:00KUBERNETES_SERVICE_HOST=172.17.0.1
2023-04-17 09:33:04.087797+00:00PWD=/
2023-04-17 09:33:04.087800+00:00TFA_PORT=17101
2023-04-17 09:33:04.087802+00:00PROVIDERS_OIDC_ISSUER_URL=http://my.public.ip.address:28484/dex
2023-04-17 09:33:04.087807+00:00GRIST_IX_CHART_PORT_80_TCP=tcp://172.17.183.167:80
2023-04-17 09:33:04.087810+00:00GRIST_FORWARD_AUTH_HEADER=X-Forwarded-User
2023-04-17 09:33:04.087812+00:00GRIST_PORT=17100
2023-04-17 09:33:04.087815+00:002023-04-17T09:33:04.087815565Z
2023-04-17 09:33:04.092762+00:00Found EMAIL without a matching PASSWORD, skipping
2023-04-17 09:33:04.092807+00:00Using /custom/dex.yaml
2023-04-17 09:33:04.099243+00:002023/04/17 09:33:04 Starting up on port 17102
2023-04-17 09:33:04.099285+00:00Checking dex... at http://my.public.ip.address:28484/dex/.well-known/openid-configuration
2023-04-17 09:33:04.216843+00:00time="2023-04-17T09:33:04Z" level=info msg="Dex Version: v2.33.1-dirty, Go Version: go1.18.4, Go OS/ARCH: linux amd64"
2023-04-17 09:33:04.216894+00:00time="2023-04-17T09:33:04Z" level=info msg="config using log level: debug"
2023-04-17 09:33:04.216900+00:00time="2023-04-17T09:33:04Z" level=info msg="config issuer: http://my.public.ip.address:28484/dex"
2023-04-17 09:33:04.218826+00:00time="2023-04-17T09:33:04Z" level=info msg="config storage: sqlite3"
2023-04-17 09:33:04.218843+00:00time="2023-04-17T09:33:04Z" level=info msg="config static client: Grist"
2023-04-17 09:33:04.218850+00:00time="2023-04-17T09:33:04Z" level=info msg="config connector: local passwords enabled"
2023-04-17 09:33:04.218865+00:00time="2023-04-17T09:33:04Z" level=info msg="config skipping approval screen"
2023-04-17 09:33:04.218901+00:00time="2023-04-17T09:33:04Z" level=info msg="config refresh tokens rotation enabled: true"
2023-04-17 09:33:04.220325+00:00time="2023-04-17T09:33:04Z" level=info msg="listening (http) on 0.0.0.0:9999"
2023-04-17 09:33:04.425249+00:00time="2023-04-17T09:33:04Z" level=info msg="Configuration loaded from flags."
2023-04-17 09:33:09.129459+00:00got: 200
2023-04-17 09:33:09.129586+00:00Happy with dex
2023-04-17 09:33:09.129767+00:00Starting traefik-forward-auth
2023-04-17 09:33:10.136383+00:00I think everything has started up now
2023-04-17 09:33:10.136606+00:00Listening internally on 80, externally at http://my.public.ip.address:28484
2023-04-17 10:03:37.456904+00:00time="2023-04-17T10:03:37Z" level=error msg="Error while Peeking first byte: read tcp 172.16.1.109:80->116.246.2.247:50689: read: connection timed out"
2023-04-17 10:41:34.815657+00:00time="2023-04-17T10:41:34Z" level=error msg="Error while Peeking first byte: read tcp 172.16.1.109:80->116.246.2.247:52504: read: connection timed out"
2023-04-17 10:47:18.882713+00:00time="2023-04-17T10:47:18Z" level=error msg="Error while Peeking first byte: read tcp 172.16.1.109:80->116.246.2.247:52763: read: connection timed out"

Perhaps it is due to /grist/sandbox/run.sh: line 6: ./sandbox/gvisor/update_engine_checkpoint.sh: No such file or directory. I set an environment variable GRIST_SANDBOX_FLAVOR=unsandboxed, and the error disappeared. However, the Bad Gateway is not solved. Here is the log:

2023-04-17 14:01:16.487024+00:00Calling traefik [
2023-04-17 14:01:16.487070+00:00'--providers.file.filename=/settings/traefik.yaml',
2023-04-17 14:01:16.487075+00:00'--entryPoints.web.address=:80'
2023-04-17 14:01:16.487079+00:00]
2023-04-17 14:01:16.490909+00:00LOGOUT_REDIRECT=http://my.public.ip.address:28484/signed-out
2023-04-17 14:01:16.490946+00:00TEAM=shanghai
2023-04-17 14:01:16.490950+00:00KUBERNETES_PORT=tcp://172.17.0.1:443
2023-04-17 14:01:16.490953+00:00KUBERNETES_SERVICE_PORT=443
2023-04-17 14:01:16.490956+00:00APP_HOST=my.public.ip.address
2023-04-17 14:01:16.490958+00:00GRIST_ORG_IN_PATH=false
2023-04-17 14:01:16.490961+00:00WHOAMI_PORT=17102
2023-04-17 14:01:16.490967+00:00HOSTNAME=grist-ix-chart-64bd49956f-m9kkb
2023-04-17 14:01:16.490969+00:00URL=http://my.public.ip.address:28484
2023-04-17 14:01:16.490972+00:00HOME=/root
2023-04-17 14:01:16.490974+00:00APP_HOME_URL=http://my.public.ip.address:28484
2023-04-17 14:01:16.490977+00:00GRIST_FORCE_LOGIN=true
2023-04-17 14:01:16.490981+00:00INSECURE_COOKIE=true
2023-04-17 14:01:16.490984+00:00EXT_PORT=28484
2023-04-17 14:01:16.490987+00:00GRIST_IX_CHART_SERVICE_HOST=172.17.183.167
2023-04-17 14:01:16.490989+00:00PGID=1000
2023-04-17 14:01:16.490992+00:00GRIST_HIDE_UI_ELEMENTS=helpCenter,billing,templates,multiSite,multiAccounts
2023-04-17 14:01:16.490998+00:00GRIST_SANDBOX_FLAVOR=unsandboxed
2023-04-17 14:01:16.491001+00:00GRIST_FORWARD_AUTH_LOGOUT_PATH=_oauth/logout
2023-04-17 14:01:16.491003+00:00DEX_PORT=9999
2023-04-17 14:01:16.491006+00:00GRIST_IX_CHART_PORT=tcp://172.17.183.167:80
2023-04-17 14:01:16.491009+00:00GRIST_IX_CHART_SERVICE_PORT=80
2023-04-17 14:01:16.491015+00:00GRIST_IX_CHART_SERVICE_PORT_IX_GRIST_28484_0=80
2023-04-17 14:01:16.491018+00:00DEFAULT_PROVIDER=oidc
2023-04-17 14:01:16.491020+00:00KUBERNETES_PORT_443_TCP_ADDR=172.17.0.1
2023-04-17 14:01:16.491023+00:00PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-04-17 14:01:16.491028+00:00EMAIL=example@email.com
2023-04-17 14:01:16.491030+00:00KUBERNETES_PORT_443_TCP_PORT=443
2023-04-17 14:01:16.491033+00:00GRIST_DEFAULT_EMAIL=example@email.com
2023-04-17 14:01:16.491035+00:00GRIST_SINGLE_ORG=shanghai
2023-04-17 14:01:16.491038+00:00PROVIDERS_OIDC_CLIENT_ID=pgbRRXfK1MnjyzXIGpHM
2023-04-17 14:01:16.491043+00:00KUBERNETES_PORT_443_TCP_PROTO=tcp
2023-04-17 14:01:16.491045+00:00PROVIDERS_OIDC_CLIENT_SECRET=......
2023-04-17 14:01:16.491048+00:00GRIST_IX_CHART_PORT_80_TCP_ADDR=172.17.183.167
2023-04-17 14:01:16.491050+00:00PUID=1000
2023-04-17 14:01:16.491054+00:00GRIST_IX_CHART_PORT_80_TCP_PORT=80
2023-04-17 14:01:16.491058+00:00SECRET=.....
2023-04-17 14:01:16.491061+00:00GRIST_IX_CHART_PORT_80_TCP_PROTO=tcp
2023-04-17 14:01:16.491064+00:00GRIST_SESSION_SECRET=.....
2023-04-17 14:01:16.491066+00:00KUBERNETES_SERVICE_PORT_HTTPS=443
2023-04-17 14:01:16.491069+00:00KUBERNETES_PORT_443_TCP=tcp://172.17.0.1:443
2023-04-17 14:01:16.491073+00:00KUBERNETES_SERVICE_HOST=172.17.0.1
2023-04-17 14:01:16.491076+00:00PWD=/
2023-04-17 14:01:16.491079+00:00TFA_PORT=17101
2023-04-17 14:01:16.491082+00:00PROVIDERS_OIDC_ISSUER_URL=http://my.public.ip.address:28484/dex
2023-04-17 14:01:16.491086+00:00GRIST_IX_CHART_PORT_80_TCP=tcp://172.17.183.167:80
2023-04-17 14:01:16.491090+00:00GRIST_FORWARD_AUTH_HEADER=X-Forwarded-User
2023-04-17 14:01:16.491093+00:00GRIST_PORT=17100
2023-04-17 14:01:16.491096+00:002023-04-17T14:01:16.491096426Z
2023-04-17 14:01:16.496572+00:00Using /custom/dex.yaml
2023-04-17 14:01:16.496585+00:00Found EMAIL without a matching PASSWORD, skipping
2023-04-17 14:01:16.499574+00:00Checking dex... at http://my.public.ip.address:28484/dex/.well-known/openid-configuration
2023-04-17 14:01:16.506648+00:002023/04/17 14:01:16 Starting up on port 17102
2023-04-17 14:01:16.530944+00:00internal/modules/cjs/loader.js:934
2023-04-17 14:01:16.530983+00:00throw err;
2023-04-17 14:01:16.530988+00:00^
2023-04-17 14:01:16.531003+00:002023-04-17T14:01:16.531003909Z
2023-04-17 14:01:16.531008+00:00Error: Cannot find module '/_build/stubs/app/server/server.js'
2023-04-17 14:01:16.531013+00:00at Function.Module._resolveFilename (internal/modules/cjs/loader.js:931:15)
2023-04-17 14:01:16.531017+00:00at Function.Module._load (internal/modules/cjs/loader.js:774:27)
2023-04-17 14:01:16.531024+00:00at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
2023-04-17 14:01:16.531029+00:00at internal/main/run_main_module.js:17:47 {
2023-04-17 14:01:16.531034+00:00code: 'MODULE_NOT_FOUND',
2023-04-17 14:01:16.531038+00:00requireStack: []
2023-04-17 14:01:16.531041+00:00}
2023-04-17 14:01:16.671835+00:00time="2023-04-17T14:01:16Z" level=info msg="Dex Version: v2.33.1-dirty, Go Version: go1.18.4, Go OS/ARCH: linux amd64"
2023-04-17 14:01:16.671870+00:00time="2023-04-17T14:01:16Z" level=info msg="config using log level: debug"
2023-04-17 14:01:16.671876+00:00time="2023-04-17T14:01:16Z" level=info msg="config issuer: http://my.public.ip.address:28484/dex"
2023-04-17 14:01:16.673715+00:00time="2023-04-17T14:01:16Z" level=info msg="config storage: sqlite3"
2023-04-17 14:01:16.673741+00:00time="2023-04-17T14:01:16Z" level=info msg="config static client: Grist"
2023-04-17 14:01:16.673746+00:00time="2023-04-17T14:01:16Z" level=info msg="config connector: local passwords enabled"
2023-04-17 14:01:16.673750+00:00time="2023-04-17T14:01:16Z" level=info msg="config skipping approval screen"
2023-04-17 14:01:16.673768+00:00time="2023-04-17T14:01:16Z" level=info msg="config refresh tokens rotation enabled: true"
2023-04-17 14:01:16.676265+00:00time="2023-04-17T14:01:16Z" level=info msg="listening (http) on 0.0.0.0:9999"
2023-04-17 14:01:16.783354+00:00time="2023-04-17T14:01:16Z" level=info msg="Configuration loaded from flags."
2023-04-17 14:01:21.544404+00:00got: 200
2023-04-17 14:01:21.544591+00:00Happy with dex
2023-04-17 14:01:21.544755+00:00Starting traefik-forward-auth
2023-04-17 14:01:22.552675+00:00I think everything has started up now
2023-04-17 14:01:22.552926+00:00Listening internally on 80, externally at http://my.public.ip.address:28484

Then I have no idea how to solve the problem… :thinking:

By the way, I can replicate the behavior @Ethan see by setting the path -v /.../dex.yaml:/settings/dex.yaml (proposed in Unable to open URL with grist-omnibus docker in TrueNAS). If you use the setting -v /.../dex.yaml:/custom/dex.yaml, No /custom/dex.yaml will change to Using /custom/dex.yaml. I think the path -v /.../dex.yaml:/settings/dex.yaml is no longer effective in the future version of grist.

Hi Ethan, can you try again with the latest version of grist-omnibus? Looks like I had broken something, also reported here:

Thanks @Kenny for flagging the ./sandbox/gvisor/update_engine_checkpoint.sh error. I’ll look into the dex.yaml problem you mention.

Hi, @paul-grist, I have tried repulling the latest version of grist-omnibus, and I can open my grist now. However, there still exists a problem. My previous workspaces and documents in the site are gone. What I see is a brand new account. But my files in /persist/docs seems normal.

Thanks for reporting this @Kenny , in hindsight a “squashing” step I’d added in a misguided attempt to help another user introduced a bunch of problems I hadn’t noticed. I’ve removed it.

Thanks @paul-grist! Confirming everything is working now on latest image.

Thanks @paul-grist! Everything is working.