Why do my pods not fulfill the resource quota limits if the number match?

Question

After applying the following ResourceQuota compute-resources to my GKE Cluster

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    limits.cpu: "1"
    limits.memory: 1Gi

and updating a Deployment to

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
spec:
  selector:
    matchLabels:
      app: my-service
      tier: backend
      track: stable
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 50%
  template:
    metadata:
      labels:
        app: my-service
        tier: backend
        track: stable
    spec:
      containers:
        - name: my-service
          image: registry/namespace/my-service:latest
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              memory: "128Mi"
              cpu: "125m"
            limits:
              memory: "256Mi"
              cpu: "125m"

the scheduling fails 100% of tries due to pods "my-service-5bc4c68df6-4z8wp" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory. Since limits and requests are specified and they fulfill the limit, I don't see a reason why the pods should be forbidden.

How pod limits resource on kubernetes enforced when the pod exceed limits after pods is created ? is a different question.

I upgraded my cluster to 1.13.6-gke.0.

Solution

I was about to suggest to test within separate namespace, but see that you already tried.

As another workaround try to setup default limits by enabling LimitRanger admission controller and setting it up e.g.

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-limit-range
spec:
  limits:
  - default:
      memory: 256Mi
      cpu: 125m
    defaultRequest:
      cpu: 125m
      memory: 128Mi
    type: Container

Now if a Container is created in the default namespace, and the Container does not specify its own values for CPU request and CPU limit, the Container is given a default CPU limits of 125m and a default memory limit of 256Mi

Also, after setting up LimitRange, make sure you removed your deployment and there are no pods stuck in failed state.

Comments

[Suresh Vishnoi]

Hi, Can you try with 1 replica, hopefully, there is no overhead of init containers, as your 2 replicas are adding it to 250m core

[Kalle Richter]

@SureshVishnoi Thanks for you input. I set replicas to 1. I set cpu: "1" as well and reproduced the issue in a separate namespace. Do you have another idea?

[Kalle Richter]

Thanks for your input. I replaced ResourceQuota with LimitRange and changed it for the namespace with kubectl replace file.yml --namespace=mynamespace. I deleted the deployments (no pods were created). After recreation of the deployment the creation of pods still fails due to Error creating: pods "my-service-85d5cf694d-vmhzb" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=250m,limits.memory=64Mi, used: limits.cpu=2250m,limits.memory=2Gi, limited: limits.cpu=1,limits.memory=1Gi. The behaviour is the same on GKE as well as locally on microk8s on Ubuntu.

[A_Suh]

at least LimitRange has solved must specify limits.cpu,limits.memory issue. So now you have another issue - exceeded quota, which is expected because you have already used your quota ) Check the current usage kubectl describe ResourceQuota -n mynamespace

[Kalle Richter]

How to figure out where the quota values come from? I haven't configured them. The pod doesn't exceed the quota by it's specifications as well as by it runtime requirements (just idles). So, I don't see any reason why I should get this error.

[A_Suh]

@KarlRichter Is there anything kubectl get all -n mynamespace