upstream connect error or disconnect/reset before headers. reset reason: connection failure. Spring Boot and java 11
As @Victor mentioned the problem here was the wrong yaml file.
I solve it. In my case the yaml file was wrong. I reviewed it and the problem now is solved. Thank you guys., – Victor
If you're looking for yaml samples I would suggest to take a look at istio github samples.
As 503 upstream connect error or disconnect/reset before headers. reset reason: connection failure
occurs very often I set up little troubleshooting answer, there are another questions with 503 error which I encountered for several months with answers, useful informations from istio documentation and things I would check.
Examples with 503 error:
- Istio 503:s between (Public) Gateway and Service
- IstIO egress gateway gives HTTP 503 error
- Istio Ingress Gateway with TLS termination returning 503 service unavailable
- how to terminate ssl at ingress-gateway in istio?
- Accessing service using istio ingress gives 503 error when mTLS is enabled
Common cause of 503 errors from istio documentation:
- https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes
- https://istio.io/docs/ops/common-problems/network-issues/#503-errors-after-setting-destination-rule
- https://istio.io/latest/docs/concepts/traffic-management/#working-with-your-applications
Few things I would check first:
- Check services ports name, Istio can route correctly the traffic if it knows the protocol. It should be
<protocol>[-<suffix>]
as mentioned in istio documentation. - Check mTLS, if there are any problems caused by mTLS, usually those problems would result in error 503.
- Check if istio works, I would recommend to apply bookinfo application example and check if it works as expected.
- Check if your namespace is injected with
kubectl get namespace -L istio-injection
- If the VirtualService using the subsets arrives before the DestinationRule where the subsets are defined, the Envoy configuration generated by Pilot would refer to non-existent upstream pools. This results in HTTP 503 errors until all configuration objects are available to Pilot.
Related videos on Youtube
Victor
Updated on October 19, 2020Comments
-
Victor over 3 years
I'm having a problem migrating my pure Kubernetes app to an Istio managed. I'm using Google Cloud Platform (GCP), Istio 1.4, Google Kubernetes Engine (GKE), Spring Boot and JAVA 11.
I had the containers running in a pure GKE environment without a problem. Now I started the migration of my Kubernetes cluster to use Istio. Since then I'm getting the following message when I try to access the exposed service.
upstream connect error or disconnect/reset before headers. reset reason: connection failure
This error message looks like a really generic. I found a lot of different problems, with the same error message, but no one was related to my problem.
Bellow the version of the Istio:
client version: 1.4.10 control plane version: 1.4.10-gke.5 data plane version: 1.4.10-gke.5 (2 proxies)
Bellow my yaml files:
apiVersion: v1 kind: ServiceAccount metadata: labels: account: tree-guest name: tree-guest-service-account --- apiVersion: v1 kind: Service metadata: labels: app: tree-guest service: tree-guest name: tree-guest spec: ports: - name: http port: 8080 targetPort: 8080 selector: app: tree-guest --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: tree-guest version: v1 name: tree-guest-v1 spec: replicas: 1 selector: matchLabels: app: tree-guest version: v1 template: metadata: labels: app: tree-guestaz version: v1 spec: containers: - image: registry.hub.docker.com/victorsens/tree-quest:circle_ci_build_00923285-3c44-4955-8de1-ed578e23c5cf imagePullPolicy: IfNotPresent name: tree-guest ports: - containerPort: 8080 serviceAccount: tree-guest-service-account --- apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: tree-guest-gateway spec: selector: istio: ingressgateway # use istio default controller servers: - port: number: 80 name: http protocol: HTTP hosts: - "*" --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: tree-guest-virtual-service spec: hosts: - "*" gateways: - tree-guest-gateway http: - match: - uri: prefix: /v1 route: - destination: host: tree-guest port: number: 8080
To apply the yaml file I used the following argument:
kubectl apply -f <(istioctl kube-inject -f ./tree-guest.yaml)
Below the result of the Istio proxy argument, after deploying the application:
istio-ingressgateway-6674cc989b-vwzqg.istio-system SYNCED SYNCED SYNCED SYNCED istio-pilot-ff4489db8-2hx5f 1.4.10-gke.5 tree-guest-v1-774bf84ddd-jkhsh.default SYNCED SYNCED SYNCED SYNCED istio-pilot-ff4489db8-2hx5f 1.4.10-gke.5
If someone have a tip about what is going wrong, please let me know. I'm stuck in this problem for a couple of days.
Thanks.