Internal Server Error 500 while accessing $GITLAB/admin/runners
Solution 1
You did a restore, but probably did not restore the secrets store (/etc/gitlab/gitlab-secrets.json or $GITLAB_HOME/config/secrets.json). Same issue as gitlab 500 errors in the admin area
You can reset it like this from the rails console on the gitlab server:
root@gitlab:/# gitlab-rails console
-------------------------------------------------------------------------------------
GitLab: 11.5.1 (c90ae59)
GitLab Shell: 8.4.1
postgresql: 9.6.8
-------------------------------------------------------------------------------------
Both Deployment and its :status machine have defined a different default for "status". Use only one or the other for defining defaults to avoid unexpected behaviors.
Loading production environment (Rails 4.2.10)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true
Solution 2
I could resolve the issue by following official gitlab documentation:
gitlab-rails dbconsole
-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;
https://docs.gitlab.com/ee/raketasks/backup_restore.html#reset-runner-registration-tokens
Edit:
After clearing existing pipeline jobs (see above), I could still not open the ci settings page for some migrated projects where I had set environment variables. In this case try to remove them:
gitlab-rails dbconsole
SELECT * FROM ci_variables;
DELETE FROM ci_variables WHERE project_id='XX';
Hope that helps.
Solution 3
i've solved this,maybe it is usefal
a. Reset CI/CD variables
gitlab-rails dbconsole
SELECT * FROM public."ci_group_variables";
SELECT * FROM public."ci_variables";
DELETE FROM ci_group_variables;
DELETE FROM ci_variables;
b. Reset runner registration tokens
gitlab-rails dbconsole
-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;
c. Reset pending pipeline jobs
sudo gitlab-rails dbconsole
-- Clear build tokens
UPDATE ci_builds SET token = null, token_encrypted = null;
d. Fix project integrations
gitlab-rails dbconsole
-- truncate web_hooks table
TRUNCATE web_hooks CASCADE;
from gitlab
NerdSec
A little bit of python, elastic and security learnt every day!
Updated on June 08, 2022Comments
-
NerdSec almost 2 years
I have restored Gitlab from a backup, now every time I try to access the
runners
webpage, I get a Internal Server error.I have tried uninstalling all configured gitlab-runners and accessing it, the problem persists. Here is a trace from the gitlab-rails/production.log:
Started GET "/admin/runners" for 127.0.0.1 at 2019-01-16 07:17:12 -0500 Processing by Admin::RunnersController#index as HTML Completed 500 Internal Server Error in 66ms (ActiveRecord: 5.7ms) ActionView::Template::Error (): 37: 38: .col-sm-6 39: .bs-callout 40: = render partial: 'ci/runner/how_to_setup_runner', 41: locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token, 42: type: 'shared', 43: reset_token_url: reset_registration_token_admin_application_settings_path } lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt' app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token' app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!' app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field' app/models/application_setting.rb:409:in `runners_registration_token' lib/gitlab/current_settings.rb:19:in `method_missing' app/views/admin/runners/index.html.haml:40:in `_app_views_admin_runners_index_html_haml___977288809002382166_69944849285200' app/controllers/application_controller.rb:116:in `render' lib/gitlab/i18n.rb:55:in `with_locale' lib/gitlab/i18n.rb:61:in `with_user_locale' app/controllers/application_controller.rb:420:in `set_locale' lib/gitlab/middleware/multipart.rb:103:in `call' lib/gitlab/request_profiler/middleware.rb:16:in `call' lib/gitlab/middleware/go.rb:19:in `call' lib/gitlab/etag_caching/middleware.rb:13:in `call' lib/gitlab/middleware/correlation_id.rb:16:in `block in call' lib/gitlab/correlation_id.rb:15:in `use_id' lib/gitlab/middleware/correlation_id.rb:15:in `call' lib/gitlab/middleware/read_only/controller.rb:40:in `call' lib/gitlab/middleware/read_only.rb:18:in `call' lib/gitlab/middleware/basic_health_check.rb:25:in `call' lib/gitlab/request_context.rb:20:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call' lib/gitlab/middleware/release_env.rb:13:in `call' Started GET "/-/metrics" for 127.0.0.1 at 2019-01-16 07:17:24 -0500 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms)
-
NerdSec over 5 yearsYes, I did not restore the config. Is there a way I can reset it? I have lost the secrets.json.
-
Joost Evertse over 5 yearsI have updated the answer to show a way to reset the token
-
NerdSec over 5 yearsThis should ideally be added to the documentation.
-
Evan R. almost 5 yearsmake sure you type exit after. I just ran into this issue and initially still got 500 errors until I typed "exit" to get back out of the gitlab-rails console.
-
DependencyHell over 4 yearsThat did the trick! Thank you! For anyone on a docker install, use
docker exec -it gitlab_gitlab_1 bin/rails dbconsole
or more comfy :docker exec -it gitlab_gitlab_1 bash
then from the gitlab directory/home/git/gitlab
:bin/rails dbconsole
(and postgres password can be found in/home/git/gitlab/config/database.yml
) -
bbaassssiiee almost 3 yearsDid all that but: ``` ERROR: Checking for jobs... forbidden ```