Grist Omnibus Stop Working All Of A Sudden

Ask for Help
1

We have a grist omnibus instance running and it was working fine but all of a sudden we can’t access the website for some reason. Here is a log of what it is showing:

2024/09/06 18:33:56 Starting up on port 17101

time=“2024-09-06T18:33:56Z” level=info msg=“Configuration loaded from flags.”

info [grist-omnibus] Using /custom/dex.yaml

gvisor check ok (flags: -unprivileged -ignore-cgroups)

time=“2024-09-06T18:33:56Z” level=info msg=“Dex Version: v2.33.1-dirty, Go Version: go1.18.4, Go OS/ARCH: linux amd64”

time=“2024-09-06T18:33:56Z” level=info msg=“config using log level: debug”

time=“2024-09-06T18:33:56Z” level=info msg=“config issuer: https://grist.ej1899.com/dex”

time=“2024-09-06T18:33:56Z” level=info msg=“config storage: sqlite3”

invalid config: ID or IDEnv field is required for a client

error [grist-omnibus] dex failed: exited with 2

Running Grist as user 1001 with primary group 1001

info [grist-omnibus] Calling traefik [

‘–providers.file.filename=/settings/traefik.yaml’,

‘–entryPoints.web.address=:80’,

‘–certificatesResolvers.letsencrypt.acme.email=rhope@ej1899.com’,

‘–certificatesResolvers.letsencrypt.acme.storage=/persist/acme.json’,

‘–certificatesResolvers.letsencrypt.acme.tlschallenge=true’,

‘–entrypoints.websecure.address=:443’,

‘–entrypoints.web.http.redirections.entrypoint.scheme=https’,

‘–entrypoints.web.http.redirections.entrypoint.to=websecure’

]

2024/09/06 18:34:57 Starting up on port 17101

time=“2024-09-06T18:34:57Z” level=info msg=“Configuration loaded from flags.”

info [grist-omnibus] Using /custom/dex.yaml

time=“2024-09-06T18:34:57Z” level=info msg=“Dex Version: v2.33.1-dirty, Go Version: go1.18.4, Go OS/ARCH: linux amd64”

time=“2024-09-06T18:34:57Z” level=info msg=“config using log level: debug”

time=“2024-09-06T18:34:57Z” level=info msg=“config issuer: https://grist.ej1899.com/dex”

gvisor check ok (flags: -unprivileged -ignore-cgroups)

time=“2024-09-06T18:34:57Z” level=info msg=“config storage: sqlite3”

invalid config: ID or IDEnv field is required for a client

error [grist-omnibus] dex failed: exited with 2

2

invalid config: ID or IDEnv field is required for a client stands out as an error. I see the error case in dex here:

I’m not sure what could be causing it to be thrown for you. Could there have been any configuration change?

3

No configuration changes were made. It randomly happened yesterday even though we were able to use the hosted server in the morning time yesterday.

4

We tried restarting the instance and running through the same procedure but we see the same issue showing up for some odd reason.

5

Hi, I think you were bitten by a change in grist-omnibus. The way that dex should be configured has changed.

You will need to make some corresponding changes to your dex.yml.

Briefly, in the static clients section, make it look like this:

staticClients:
- id: '{{ getenv "GRIST_OIDC_IDP_CLIENT_ID" }}'
  redirectURIs:
  - '{{ getenv "APP_HOME_URL" }}/oauth2/callback'
  name: 'Grist'
  secret: '{{ getenv "GRIST_OIDC_IDP_CLIENT_SECRET" }}'
1 Like
6

Still getting the issue for some reason is it from the “dex-full.yaml”? where would the dex.yml be located?

7

Seems like restarting it makes it go back to the original staticClients configuration

8

I do not know what dex-full.yml is.

Can you give me some indication of how you configured omnibus and how you are running it?

If possible, can you also tell me how (or if) you upgraded it?

9

We actually ended up getting to this point thus far where we can’t seem to get to the actual work space :
image

This is the log:
2024-09-06 20:03:20.246 - debug: 3-pipe Sandbox started sandboxPid=228, flavor=gvisor, command=undefined, entryPoint=(default), docId=test
2024-09-06 20:03:20.248 - info: activity docCount=0, orgCount=2, orgInGoodStandingCount=2, userCount=8, userWithLoginCount=8
2024-09-06 20:03:20.283 - info: Sandbox stderr: run.py: sandbox/gvisor/run.py -E PYTHONPATH=/grist/sandbox/grist -E PIPE_MODE=minimal -m /grist/sandbox --restore /tmp/engine__grist python3 – /grist/sandbox/grist/main.py test sandboxPid=228, flavor=gvisor, command=undefined, entryPoint=(default), docId=test
2024-09-06 20:03:20.543 - debug: NSandbox pyCall sandboxPid=228, flavor=gvisor, command=undefined, entryPoint=(default), docId=test, funcName=get_version, loadMs=296
2024-09-06 20:03:20.544 - debug: Sandbox shutdown starting sandboxPid=228, flavor=gvisor, command=undefined, entryPoint=(default), docId=test
2024-09-06 20:03:20.690 - debug: Sandbox exited with code 0 signal null sandboxPid=228, flavor=gvisor, command=undefined, entryPoint=(default), docId=test
2024-09-06 20:03:21.255 - debug: Subprocess control failure: Error: not found flavor=gvisor, command=undefined, entryPoint=(default), docId=test
time=“2024-09-06T20:03:30Z” level=info msg=“login successful: connector "microsoft", username="Rohan Hope", preferred_username="", email="rhope@ej1899.com", groups=”
2024-09-06 20:03:30.547 - debug: Got tokenSet: {
access_token: ‘REDACTED’,
token_type: ‘bearer’,
expires_at: 1725739409,
id_token: ‘REDACTED’
}
2024-09-06 20:03:30.554 - debug: Got userinfo: {
iss: 'https://grist.ej1899.com/dex’,
sub: ‘CiQwY2FmYzg4NS00NTdkLTQ3ZjgtOWU0Ny1iYjYxYjNkZjM0Y2YSCW1pY3Jvc29mdA’,
aud: ‘XZEfWZNMdWdYgvF61B7l’,
exp: 1725739410,
iat: 1725653010,
at_hash: ‘E9ccDnaQPTXExLkh20Hejw’,
email: ‘rhope@ej1899.com’,
email_verified: true,
name: ‘Rohan Hope’
}
2024-09-06 20:03:30.555 - info: OIDCConfig: got OIDC response for rhope@ej1899.com (Rohan Hope) redirecting to Sign in to your account
2024-09-06 20:03:30.603 - debug: Auth[GET]: grist.ej1899.com / customHostSession=, method=GET, host=grist.ej1899.com, path=/, org=ej1899, email=rhope@ej1899.com, userId=6, altSessionId=bbLnXtdHe7bPj1RyALBMeN
2024-09-06 20:03:30.760 - debug: Auth[GET]: grist.ej1899.com /session/access/active customHostSession=, method=GET, host=grist.ej1899.com, path=/session/access/active, org=ej1899, email=rhope@ej1899.com, userId=6, altSessionId=bbLnXtdHe7bPj1RyALBMeN
2024-09-06 20:03:30.768 - debug: Auth[GET]: grist.ej1899.com /session/access/all customHostSession=, method=GET, host=grist.ej1899.com, path=/session/access/all, org=ej1899, email=rhope@ej1899.com, userId=6, altSessionId=bbLnXtdHe7bPj1RyALBMeN

10

We have it at version 1.1.17 currently and we are running ombinus through an AWS EC2 instance at the moment. We have client secret and client ID. Pretty much everything we did was from the tutorial and it was working. We changed the configuration within the dex-full.yml which has the staticClients and this is where are at right now with the image shown above.

11

I tried redeploying with a different domain and everything and made those changes on the dex.yaml and this is what I am getting on my logs:

image
image2468×723 200 KB

I am not too sure how to mitigate this at this point. We do use the omnibus for our work so needed this a bit urgently.

12

I would like to know,

  1. Are you running this via the AWS marketplace Grist offering?
  2. What tutorial did you follow to install Grist Omnibus? Is it this one?
  3. Did you recently restart Grist? I am trying to understand what could have gotten upgraded and how.
  4. What is in dex-full.yml, if anything? What is in dex.yml?
  5. Are you setting any other Grist configuration parameters anywhere? Perhaps as specified in the above tutorial?
13
  1. Yes we are using this one.
  2. Yes we used this tutorial.
  3. I did restart it after making the changes on the dex.yaml which is in the /home/ubuntu/grist directory
  4. This is a file in the /home/ubuntu/grist-persist folder which seems to mimick the dex.yaml
  5. No other configuration is being changed up. I am trying to just access the work space at this point and then can go from there.
14

Hi @Tazwar_Belal.

We identified an issue in grist-omnibus and just pushed up a fix. Could you try restarting your instance (sudo ~/grist/restartGrist) with your original configuration (i.e. without any of the suggestions from this thread) and see if that works?

George

1 Like
15

Yes it reverted back. Thank you.