Bad gateway with grist-omnibus docker when accessing from LAN

Hi, there. I am back. :rofl:

I ran into the same problem that @Ethan had.

My grist is good untill this morning. I watched the Custom Widgets Webinars this morning and these widgets look super useful. But I cannot find the list of Premade Custom Widgets as shown in the red box of the figure. I suspected the gristlabs/grist-omnibus image needs to be upgraded, so I repulled the image to the container. Then, I got the Bad Gateway. Here is the log:

2023-04-17 09:33:04.085741+00:00Calling traefik [
2023-04-17 09:33:04.085788+00:00'--providers.file.filename=/settings/traefik.yaml',
2023-04-17 09:33:04.085794+00:00'--entryPoints.web.address=:80'
2023-04-17 09:33:04.085798+00:00]
2023-04-17 09:33:04.086535+00:00/grist/sandbox/run.sh: line 6: ./sandbox/gvisor/update_engine_checkpoint.sh: No such file or directory
2023-04-17 09:33:04.087648+00:00LOGOUT_REDIRECT=http://my.public.ip.address:28484/signed-out
2023-04-17 09:33:04.087665+00:00TEAM=shanghai
2023-04-17 09:33:04.087669+00:00KUBERNETES_SERVICE_PORT=443
2023-04-17 09:33:04.087672+00:00KUBERNETES_PORT=tcp://172.17.0.1:443
2023-04-17 09:33:04.087674+00:00APP_HOST=my.public.ip.address
2023-04-17 09:33:04.087683+00:00GRIST_ORG_IN_PATH=false
2023-04-17 09:33:04.087685+00:00WHOAMI_PORT=17102
2023-04-17 09:33:04.087688+00:00HOSTNAME=grist-ix-chart-79f9f8cd84-64w29
2023-04-17 09:33:04.087690+00:00URL=http://my.public.ip.address:28484
2023-04-17 09:33:04.087693+00:00HOME=/root
2023-04-17 09:33:04.087698+00:00APP_HOME_URL=http://my.public.ip.address:28484
2023-04-17 09:33:04.087701+00:00GRIST_FORCE_LOGIN=true
2023-04-17 09:33:04.087704+00:00INSECURE_COOKIE=true
2023-04-17 09:33:04.087708+00:00EXT_PORT=28484
2023-04-17 09:33:04.087711+00:00GRIST_IX_CHART_SERVICE_HOST=172.17.183.167
2023-04-17 09:33:04.087713+00:00PGID=1000
2023-04-17 09:33:04.087718+00:00GRIST_HIDE_UI_ELEMENTS=helpCenter,billing,templates,multiSite,multiAccounts
2023-04-17 09:33:04.087721+00:00GRIST_SANDBOX_FLAVOR=gvisor
2023-04-17 09:33:04.087723+00:00GRIST_FORWARD_AUTH_LOGOUT_PATH=_oauth/logout
2023-04-17 09:33:04.087726+00:00DEX_PORT=9999
2023-04-17 09:33:04.087731+00:00GRIST_IX_CHART_PORT=tcp://172.17.183.167:80
2023-04-17 09:33:04.087733+00:00GRIST_IX_CHART_SERVICE_PORT_IX_GRIST_28484_0=80
2023-04-17 09:33:04.087736+00:00GRIST_IX_CHART_SERVICE_PORT=80
2023-04-17 09:33:04.087739+00:00DEFAULT_PROVIDER=oidc
2023-04-17 09:33:04.087741+00:00KUBERNETES_PORT_443_TCP_ADDR=172.17.0.1
2023-04-17 09:33:04.087746+00:00PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-04-17 09:33:04.087748+00:00EMAIL=example@email.com
2023-04-17 09:33:04.087751+00:00KUBERNETES_PORT_443_TCP_PORT=443
2023-04-17 09:33:04.087753+00:00GRIST_DEFAULT_EMAIL=example@email.com
2023-04-17 09:33:04.087756+00:00GRIST_SINGLE_ORG=shanghai
2023-04-17 09:33:04.087760+00:00PROVIDERS_OIDC_CLIENT_ID=...
2023-04-17 09:33:04.087763+00:00KUBERNETES_PORT_443_TCP_PROTO=tcp
2023-04-17 09:33:04.087766+00:00PROVIDERS_OIDC_CLIENT_SECRET=...
2023-04-17 09:33:04.087769+00:00GRIST_IX_CHART_PORT_80_TCP_ADDR=172.17.183.167
2023-04-17 09:33:04.087774+00:00PUID=1000
2023-04-17 09:33:04.087776+00:00GRIST_IX_CHART_PORT_80_TCP_PORT=80
2023-04-17 09:33:04.087780+00:00SECRET=...
2023-04-17 09:33:04.087783+00:00GRIST_IX_CHART_PORT_80_TCP_PROTO=tcp
2023-04-17 09:33:04.087786+00:00GRIST_SESSION_SECRET=...
2023-04-17 09:33:04.087788+00:00KUBERNETES_PORT_443_TCP=tcp://172.17.0.1:443
2023-04-17 09:33:04.087792+00:00KUBERNETES_SERVICE_PORT_HTTPS=443
2023-04-17 09:33:04.087795+00:00KUBERNETES_SERVICE_HOST=172.17.0.1
2023-04-17 09:33:04.087797+00:00PWD=/
2023-04-17 09:33:04.087800+00:00TFA_PORT=17101
2023-04-17 09:33:04.087802+00:00PROVIDERS_OIDC_ISSUER_URL=http://my.public.ip.address:28484/dex
2023-04-17 09:33:04.087807+00:00GRIST_IX_CHART_PORT_80_TCP=tcp://172.17.183.167:80
2023-04-17 09:33:04.087810+00:00GRIST_FORWARD_AUTH_HEADER=X-Forwarded-User
2023-04-17 09:33:04.087812+00:00GRIST_PORT=17100
2023-04-17 09:33:04.087815+00:002023-04-17T09:33:04.087815565Z
2023-04-17 09:33:04.092762+00:00Found EMAIL without a matching PASSWORD, skipping
2023-04-17 09:33:04.092807+00:00Using /custom/dex.yaml
2023-04-17 09:33:04.099243+00:002023/04/17 09:33:04 Starting up on port 17102
2023-04-17 09:33:04.099285+00:00Checking dex... at http://my.public.ip.address:28484/dex/.well-known/openid-configuration
2023-04-17 09:33:04.216843+00:00time="2023-04-17T09:33:04Z" level=info msg="Dex Version: v2.33.1-dirty, Go Version: go1.18.4, Go OS/ARCH: linux amd64"
2023-04-17 09:33:04.216894+00:00time="2023-04-17T09:33:04Z" level=info msg="config using log level: debug"
2023-04-17 09:33:04.216900+00:00time="2023-04-17T09:33:04Z" level=info msg="config issuer: http://my.public.ip.address:28484/dex"
2023-04-17 09:33:04.218826+00:00time="2023-04-17T09:33:04Z" level=info msg="config storage: sqlite3"
2023-04-17 09:33:04.218843+00:00time="2023-04-17T09:33:04Z" level=info msg="config static client: Grist"
2023-04-17 09:33:04.218850+00:00time="2023-04-17T09:33:04Z" level=info msg="config connector: local passwords enabled"
2023-04-17 09:33:04.218865+00:00time="2023-04-17T09:33:04Z" level=info msg="config skipping approval screen"
2023-04-17 09:33:04.218901+00:00time="2023-04-17T09:33:04Z" level=info msg="config refresh tokens rotation enabled: true"
2023-04-17 09:33:04.220325+00:00time="2023-04-17T09:33:04Z" level=info msg="listening (http) on 0.0.0.0:9999"
2023-04-17 09:33:04.425249+00:00time="2023-04-17T09:33:04Z" level=info msg="Configuration loaded from flags."
2023-04-17 09:33:09.129459+00:00got: 200
2023-04-17 09:33:09.129586+00:00Happy with dex
2023-04-17 09:33:09.129767+00:00Starting traefik-forward-auth
2023-04-17 09:33:10.136383+00:00I think everything has started up now
2023-04-17 09:33:10.136606+00:00Listening internally on 80, externally at http://my.public.ip.address:28484
2023-04-17 10:03:37.456904+00:00time="2023-04-17T10:03:37Z" level=error msg="Error while Peeking first byte: read tcp 172.16.1.109:80->116.246.2.247:50689: read: connection timed out"
2023-04-17 10:41:34.815657+00:00time="2023-04-17T10:41:34Z" level=error msg="Error while Peeking first byte: read tcp 172.16.1.109:80->116.246.2.247:52504: read: connection timed out"
2023-04-17 10:47:18.882713+00:00time="2023-04-17T10:47:18Z" level=error msg="Error while Peeking first byte: read tcp 172.16.1.109:80->116.246.2.247:52763: read: connection timed out"

Perhaps it is due to /grist/sandbox/run.sh: line 6: ./sandbox/gvisor/update_engine_checkpoint.sh: No such file or directory. I set an environment variable GRIST_SANDBOX_FLAVOR=unsandboxed, and the error disappeared. However, the Bad Gateway is not solved. Here is the log:

2023-04-17 14:01:16.487024+00:00Calling traefik [
2023-04-17 14:01:16.487070+00:00'--providers.file.filename=/settings/traefik.yaml',
2023-04-17 14:01:16.487075+00:00'--entryPoints.web.address=:80'
2023-04-17 14:01:16.487079+00:00]
2023-04-17 14:01:16.490909+00:00LOGOUT_REDIRECT=http://my.public.ip.address:28484/signed-out
2023-04-17 14:01:16.490946+00:00TEAM=shanghai
2023-04-17 14:01:16.490950+00:00KUBERNETES_PORT=tcp://172.17.0.1:443
2023-04-17 14:01:16.490953+00:00KUBERNETES_SERVICE_PORT=443
2023-04-17 14:01:16.490956+00:00APP_HOST=my.public.ip.address
2023-04-17 14:01:16.490958+00:00GRIST_ORG_IN_PATH=false
2023-04-17 14:01:16.490961+00:00WHOAMI_PORT=17102
2023-04-17 14:01:16.490967+00:00HOSTNAME=grist-ix-chart-64bd49956f-m9kkb
2023-04-17 14:01:16.490969+00:00URL=http://my.public.ip.address:28484
2023-04-17 14:01:16.490972+00:00HOME=/root
2023-04-17 14:01:16.490974+00:00APP_HOME_URL=http://my.public.ip.address:28484
2023-04-17 14:01:16.490977+00:00GRIST_FORCE_LOGIN=true
2023-04-17 14:01:16.490981+00:00INSECURE_COOKIE=true
2023-04-17 14:01:16.490984+00:00EXT_PORT=28484
2023-04-17 14:01:16.490987+00:00GRIST_IX_CHART_SERVICE_HOST=172.17.183.167
2023-04-17 14:01:16.490989+00:00PGID=1000
2023-04-17 14:01:16.490992+00:00GRIST_HIDE_UI_ELEMENTS=helpCenter,billing,templates,multiSite,multiAccounts
2023-04-17 14:01:16.490998+00:00GRIST_SANDBOX_FLAVOR=unsandboxed
2023-04-17 14:01:16.491001+00:00GRIST_FORWARD_AUTH_LOGOUT_PATH=_oauth/logout
2023-04-17 14:01:16.491003+00:00DEX_PORT=9999
2023-04-17 14:01:16.491006+00:00GRIST_IX_CHART_PORT=tcp://172.17.183.167:80
2023-04-17 14:01:16.491009+00:00GRIST_IX_CHART_SERVICE_PORT=80
2023-04-17 14:01:16.491015+00:00GRIST_IX_CHART_SERVICE_PORT_IX_GRIST_28484_0=80
2023-04-17 14:01:16.491018+00:00DEFAULT_PROVIDER=oidc
2023-04-17 14:01:16.491020+00:00KUBERNETES_PORT_443_TCP_ADDR=172.17.0.1
2023-04-17 14:01:16.491023+00:00PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-04-17 14:01:16.491028+00:00EMAIL=example@email.com
2023-04-17 14:01:16.491030+00:00KUBERNETES_PORT_443_TCP_PORT=443
2023-04-17 14:01:16.491033+00:00GRIST_DEFAULT_EMAIL=example@email.com
2023-04-17 14:01:16.491035+00:00GRIST_SINGLE_ORG=shanghai
2023-04-17 14:01:16.491038+00:00PROVIDERS_OIDC_CLIENT_ID=pgbRRXfK1MnjyzXIGpHM
2023-04-17 14:01:16.491043+00:00KUBERNETES_PORT_443_TCP_PROTO=tcp
2023-04-17 14:01:16.491045+00:00PROVIDERS_OIDC_CLIENT_SECRET=......
2023-04-17 14:01:16.491048+00:00GRIST_IX_CHART_PORT_80_TCP_ADDR=172.17.183.167
2023-04-17 14:01:16.491050+00:00PUID=1000
2023-04-17 14:01:16.491054+00:00GRIST_IX_CHART_PORT_80_TCP_PORT=80
2023-04-17 14:01:16.491058+00:00SECRET=.....
2023-04-17 14:01:16.491061+00:00GRIST_IX_CHART_PORT_80_TCP_PROTO=tcp
2023-04-17 14:01:16.491064+00:00GRIST_SESSION_SECRET=.....
2023-04-17 14:01:16.491066+00:00KUBERNETES_SERVICE_PORT_HTTPS=443
2023-04-17 14:01:16.491069+00:00KUBERNETES_PORT_443_TCP=tcp://172.17.0.1:443
2023-04-17 14:01:16.491073+00:00KUBERNETES_SERVICE_HOST=172.17.0.1
2023-04-17 14:01:16.491076+00:00PWD=/
2023-04-17 14:01:16.491079+00:00TFA_PORT=17101
2023-04-17 14:01:16.491082+00:00PROVIDERS_OIDC_ISSUER_URL=http://my.public.ip.address:28484/dex
2023-04-17 14:01:16.491086+00:00GRIST_IX_CHART_PORT_80_TCP=tcp://172.17.183.167:80
2023-04-17 14:01:16.491090+00:00GRIST_FORWARD_AUTH_HEADER=X-Forwarded-User
2023-04-17 14:01:16.491093+00:00GRIST_PORT=17100
2023-04-17 14:01:16.491096+00:002023-04-17T14:01:16.491096426Z
2023-04-17 14:01:16.496572+00:00Using /custom/dex.yaml
2023-04-17 14:01:16.496585+00:00Found EMAIL without a matching PASSWORD, skipping
2023-04-17 14:01:16.499574+00:00Checking dex... at http://my.public.ip.address:28484/dex/.well-known/openid-configuration
2023-04-17 14:01:16.506648+00:002023/04/17 14:01:16 Starting up on port 17102
2023-04-17 14:01:16.530944+00:00internal/modules/cjs/loader.js:934
2023-04-17 14:01:16.530983+00:00throw err;
2023-04-17 14:01:16.530988+00:00^
2023-04-17 14:01:16.531003+00:002023-04-17T14:01:16.531003909Z
2023-04-17 14:01:16.531008+00:00Error: Cannot find module '/_build/stubs/app/server/server.js'
2023-04-17 14:01:16.531013+00:00at Function.Module._resolveFilename (internal/modules/cjs/loader.js:931:15)
2023-04-17 14:01:16.531017+00:00at Function.Module._load (internal/modules/cjs/loader.js:774:27)
2023-04-17 14:01:16.531024+00:00at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
2023-04-17 14:01:16.531029+00:00at internal/main/run_main_module.js:17:47 {
2023-04-17 14:01:16.531034+00:00code: 'MODULE_NOT_FOUND',
2023-04-17 14:01:16.531038+00:00requireStack: []
2023-04-17 14:01:16.531041+00:00}
2023-04-17 14:01:16.671835+00:00time="2023-04-17T14:01:16Z" level=info msg="Dex Version: v2.33.1-dirty, Go Version: go1.18.4, Go OS/ARCH: linux amd64"
2023-04-17 14:01:16.671870+00:00time="2023-04-17T14:01:16Z" level=info msg="config using log level: debug"
2023-04-17 14:01:16.671876+00:00time="2023-04-17T14:01:16Z" level=info msg="config issuer: http://my.public.ip.address:28484/dex"
2023-04-17 14:01:16.673715+00:00time="2023-04-17T14:01:16Z" level=info msg="config storage: sqlite3"
2023-04-17 14:01:16.673741+00:00time="2023-04-17T14:01:16Z" level=info msg="config static client: Grist"
2023-04-17 14:01:16.673746+00:00time="2023-04-17T14:01:16Z" level=info msg="config connector: local passwords enabled"
2023-04-17 14:01:16.673750+00:00time="2023-04-17T14:01:16Z" level=info msg="config skipping approval screen"
2023-04-17 14:01:16.673768+00:00time="2023-04-17T14:01:16Z" level=info msg="config refresh tokens rotation enabled: true"
2023-04-17 14:01:16.676265+00:00time="2023-04-17T14:01:16Z" level=info msg="listening (http) on 0.0.0.0:9999"
2023-04-17 14:01:16.783354+00:00time="2023-04-17T14:01:16Z" level=info msg="Configuration loaded from flags."
2023-04-17 14:01:21.544404+00:00got: 200
2023-04-17 14:01:21.544591+00:00Happy with dex
2023-04-17 14:01:21.544755+00:00Starting traefik-forward-auth
2023-04-17 14:01:22.552675+00:00I think everything has started up now
2023-04-17 14:01:22.552926+00:00Listening internally on 80, externally at http://my.public.ip.address:28484

Then I have no idea how to solve the problem… :thinking:

By the way, I can replicate the behavior @Ethan see by setting the path -v /.../dex.yaml:/settings/dex.yaml (proposed in Unable to open URL with grist-omnibus docker in TrueNAS). If you use the setting -v /.../dex.yaml:/custom/dex.yaml, No /custom/dex.yaml will change to Using /custom/dex.yaml. I think the path -v /.../dex.yaml:/settings/dex.yaml is no longer effective in the future version of grist.