How to solve 'upstream connect error or disconnect/reset before headers. reset reason: connection termination' in Istio?

Question

I try to setup an aws load balancer (ELB) with SSL follow the instruction at #6566

Certificate was attached on ELB.

However, I got the issue "upstream connect error or disconnect/reset before headers. reset reason: connection termination" when trying to access our web on browser.

Our setup without SSL has been worked before.

I use the custom values.yaml to install the istio (helm template):

helm template ./istio/install/kubernetes/helm/istio --name istio \
--namespace istio-system --values ./mesh/values.yaml | kubectl apply -f -

I have inserted below annotations to the gateways tag:

istio-ingressgateway:
    serviceAnnotations: 
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:ap-southeast-1:xxxxx:certificate/my-crt"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

Here is my gateway.yaml:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway #default istio ingressgateway
  servers:
  - port:
      number: 80
      name: http-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-gateway
spec:
  gateways:
  - istio-gateway
  hosts:
  - "*"
  http:
  - match:
    - uri:
        prefix: /socket.io/
    route:
    - destination:
        host: api-gateway-ws.default.svc.cluster.local
        port:
          number: 5001
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api-gateway.default.svc.cluster.local
        port:
          number: 5000

Solution 1

I encountered the same error but with a different issue. The Service port needed a name added to it. https://github.com/istio/istio/issues/19966. And they need to follow the format (protocol-suffix) https://istio.io/docs/ops/deployment/requirements/

ports:
    - name: https # Use http or https
       protocol: TCP
       port: 8080
       targetPort: 8080

Solution 2

I have resolved the issue by update the gateway manifest. Not sure why the error happen when adding multiple "match".

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway #default istio ingressgateway
  servers:
  - port:
      number: 80
      name: http-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-gateway
spec:
  gateways:
  - istio-gateway
  hosts:
  - "*"
  http:
  - match:
    - uri:
        prefix: "/socket.io"
    route:
    - destination:
        host: api-gateway-ws.default.svc.cluster.local
        port:
          number: 5001
    websocketUpgrade: true
  - route:
    - destination:
        host: api-gateway.default.svc.cluster.local
        port:
          number: 5000

Solution 3

This is most probably there is mismatch of the port that application running in the DOCKER is not the same port exposed in the DOCKER configuration

Docker file EXPOSE <PORT> should be same as the port the application is started.

Istio Ingress gateway can bind to the container port but can't communicate the application.

Solution 4

Split large virtual services and destination rules into multiple resources: The downside of your kind of configuration is that other configuration (e.g., route rules) for any of the underlying microservices, will need to also be included in this single configuration file. Reference: https://istio.io/docs/ops/best-practices/traffic-management/enter link description here

View more solutions

Comments

[Admin]

What does the log indicate?

[Admin]

@030: I think there is a problem with sync data between pilot and istio-proxy. The first time I start my services, I unable to login. However, If I delete all services and start its again, it worked !