Forbidden resource in API group at the cluster scope
User "system:serviceaccount:default:default" cannot list resource "endpoints" in API group "" at the cluster scope"
User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope"
User "system:serviceaccount:default:default" cannot list resource "services" in API group "" at the cluster scope"
Something running with ServiceAccount default
in namespace default
is doing things it does not have permissions for.
apiVersion: v1
kind: ServiceAccount
metadata:
name: monitoring-service-account
Here you create a specific ServiceAccount. You also give it some Cluster-wide permissions.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: default
You run Prometheus in namespace default
but do not specify a specific ServiceAccount, so it will run with ServiceAccount default
.
I think your problem is that you are supposed to set the ServiceAccount that you create in the Deployment-manifest for Prometheus.
Comments
-
BentCoder almost 2 years
I am unable to identify what the exact issue with the permissions with my setup as shown below. I've looked into all the similar QAs but still unable to solve the issue. The aim is to deploy Prometheus and let it scrape
/metrics
endpoints that my other applications in the cluster expose fine.Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope" Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" at the cluster scope" Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"services\" in API group \"\" at the cluster scope" ... ...
The command below returns
no
to all services, nodes, pods etc.kubectl auth can-i get services --as=system:serviceaccount:default:default -n default
Minikube
$ minikube start --vm-driver=virtualbox --extra-config=apiserver.Authorization.Mode=RBAC 😄 minikube v1.14.2 on Darwin 11.2 ✨ Using the virtualbox driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🔄 Restarting existing virtualbox VM for "minikube" ... 🐳 Preparing Kubernetes v1.19.2 on Docker 19.03.12 ... ▪ apiserver.Authorization.Mode=RBAC 🔎 Verifying Kubernetes components... 🌟 Enabled addons: storage-provisioner, default-storageclass, dashboard 🏄 Done! kubectl is now configured to use "minikube" by default
Roles
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: monitoring-cluster-role rules: - apiGroups: [""] resources: ["nodes", "services", "pods", "endpoints"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["configmaps"] verbs: ["get"] - apiGroups: ["extensions"] resources: ["deployments"] verbs: ["get", "list", "watch"]
apiVersion: v1 kind: ServiceAccount metadata: name: monitoring-service-account namespace: default
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: monitoring-cluster-role-binding roleRef: kind: ClusterRole name: monitoring-cluster-role apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: monitoring-service-account namespace: default
Prometheus
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config-map namespace: default data: prometheus.yml: | global: scrape_interval: 15s scrape_configs: - job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs: - role: endpoints relabel_configs: - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name
apiVersion: apps/v1 kind: Deployment metadata: name: prometheus-deployment namespace: default labels: app: prometheus spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus spec: containers: - name: prometheus image: prom/prometheus:latest ports: - name: http protocol: TCP containerPort: 9090 volumeMounts: - name: config mountPath: /etc/prometheus/ - name: storage mountPath: /prometheus/ volumes: - name: config configMap: name: prometheus-config-map - name: storage emptyDir: {}
apiVersion: v1 kind: Service metadata: name: prometheus-service namespace: default spec: type: NodePort selector: app: prometheus ports: - name: http protocol: TCP port: 80 targetPort: 9090