Accessing service using istio ingress gives 503 error when mTLS is enabled
12,028
The problem is probably as follows: istio-ingressgateway initiates mTLS to hr--gateway-service on port 80, but hr--gateway-service expects plain HTTP connections.
There are multiple solutions:
- Define a DestinationRule to instruct clients to disable mTLS on calls to hr--gateway-service
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: hr--gateway-service-disable-mtls
spec:
host: hr--gateway-service.default.svc.cluster.local
trafficPolicy:
tls:
mode: DISABLE
- Instruct hr-gateway-service to accept mTLS connections. For that, configure the server TLS options on port 80 to be
MUTUALand to use Istio certificates and the private key. SpecifyserverCertificate,caCertificatesandprivateKeyto be/etc/certs/cert-chain.pem,/etc/certs/root-cert.pem,/etc/certs/key.pem, respectively.
Comments
-
Pasan W. almost 2 years
I have a mutual TLS enabled Istio mesh. My setup is as follows
- A service running inside a pod (Service container + envoy)
- An envoy gateway which stays in front of the above service. An Istio Gateway and Virtual Service attached to this. It routes
/info/route to the above service. - Another Istio Gateway configured for ingress using the default istio ingress pod. This also has Gateway+Virtual Service combination. The virtual service directs
/info/path to the service described in 2
I'm attempting to access the service from the ingress gateway using a curl command such as:
$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -vBut I'm getting a 503 not found error as below:
$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v Note: Unnecessary use of -X or --request, GET is already inferred. * Trying 10.105.138.94... * Connected to istio-ingressgateway.istio-system (10.105.138.94) port 80 (#0) > GET /info/ HTTP/1.1 > Host: istio-ingressgateway.istio-system > User-Agent: curl/7.47.0 > Accept: */* > Authorization: Bearer ... > < HTTP/1.1 503 Service Unavailable < content-length: 57 < content-type: text/plain < date: Sat, 12 Jan 2019 13:30:13 GMT < server: envoy < * Connection #0 to host istio-ingressgateway.istio-system left intactI checked the logs of
istio-ingressgatewaypod and the following line was logged there[2019-01-13T05:40:16.517Z] "GET /info/ HTTP/1.1" 503 UH 0 19 6 - "10.244.0.5" "curl/7.47.0" "da02fdce-8bb5-90fe-b422-5c74fe28759b" "istio-ingressgateway.istio-system" "-"If I logged into istio ingress pod and attempt to send the request with curl, I get a successful 200 OK.
# curl hr--gateway-service.default/info/ -H "Authorization: Bearer $token" -vAlso, I managed to get a successful response for the same curl command when the mesh was created in mTLS disabled mode. There are no conflicts shown in mTLS setup.
Here are the config details for my service mesh in case you need additional info.
Pods
$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default hr--gateway-deployment-688986c87c-z9nkh 1/1 Running 0 37m default hr--hr-deployment-596946948d-c89bn 2/2 Running 0 37m default hr--sts-deployment-694d7cff97-gjwdk 1/1 Running 0 37m ingress-nginx default-http-backend-6586bc58b6-8qss6 1/1 Running 0 42m ingress-nginx nginx-ingress-controller-6bd7c597cb-t4rwq 1/1 Running 0 42m istio-system grafana-85dbf49c94-lfpbr 1/1 Running 0 42m istio-system istio-citadel-545f49c58b-dq5lq 1/1 Running 0 42m istio-system istio-cleanup-secrets-bh5ws 0/1 Completed 0 42m istio-system istio-egressgateway-7d59954f4-qcnxm 1/1 Running 0 42m istio-system istio-galley-5b6449c48f-72vkb 1/1 Running 0 42m istio-system istio-grafana-post-install-lwmsf 0/1 Completed 0 42m istio-system istio-ingressgateway-8455c8c6f7-5khtk 1/1 Running 0 42m istio-system istio-pilot-58ff4d6647-bct4b 2/2 Running 0 42m istio-system istio-policy-59685fd869-h7v94 2/2 Running 0 42m istio-system istio-security-post-install-cqj6k 0/1 Completed 0 42m istio-system istio-sidecar-injector-75b9866679-qg88s 1/1 Running 0 42m istio-system istio-statsd-prom-bridge-549d687fd9-bspj2 1/1 Running 0 42m istio-system istio-telemetry-6ccf9ddb96-hxnwv 2/2 Running 0 42m istio-system istio-tracing-7596597bd7-m5pk8 1/1 Running 0 42m istio-system prometheus-6ffc56584f-4cm5v 1/1 Running 0 42m istio-system servicegraph-5d64b457b4-jttl9 1/1 Running 0 42m kube-system coredns-78fcdf6894-rxw57 1/1 Running 0 50m kube-system coredns-78fcdf6894-s4bg2 1/1 Running 0 50m kube-system etcd-ubuntu 1/1 Running 0 49m kube-system kube-apiserver-ubuntu 1/1 Running 0 49m kube-system kube-controller-manager-ubuntu 1/1 Running 0 49m kube-system kube-flannel-ds-9nvf9 1/1 Running 0 49m kube-system kube-proxy-r868m 1/1 Running 0 50m kube-system kube-scheduler-ubuntu 1/1 Running 0 49mServices
$ kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default hr--gateway-service ClusterIP 10.100.238.144 <none> 80/TCP,443/TCP 39m default hr--hr-service ClusterIP 10.96.193.43 <none> 80/TCP 39m default hr--sts-service ClusterIP 10.99.54.137 <none> 8080/TCP,8081/TCP,8090/TCP 39m default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 52m ingress-nginx default-http-backend ClusterIP 10.109.166.229 <none> 80/TCP 44m ingress-nginx ingress-nginx NodePort 10.108.9.180 192.168.60.3 80:31001/TCP,443:32315/TCP 44m istio-system grafana ClusterIP 10.102.141.231 <none> 3000/TCP 44m istio-system istio-citadel ClusterIP 10.101.128.187 <none> 8060/TCP,9093/TCP 44m istio-system istio-egressgateway ClusterIP 10.102.157.204 <none> 80/TCP,443/TCP 44m istio-system istio-galley ClusterIP 10.96.31.251 <none> 443/TCP,9093/TCP 44m istio-system istio-ingressgateway LoadBalancer 10.105.138.94 <pending> 80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31219/TCP,8060:31482/TCP,853:30034/TCP,15030:31544/TCP,15031:32652/TCP 44m istio-system istio-pilot ClusterIP 10.100.170.73 <none> 15010/TCP,15011/TCP,8080/TCP,9093/TCP 44m istio-system istio-policy ClusterIP 10.104.77.184 <none> 9091/TCP,15004/TCP,9093/TCP 44m istio-system istio-sidecar-injector ClusterIP 10.100.180.152 <none> 443/TCP 44m istio-system istio-statsd-prom-bridge ClusterIP 10.107.39.50 <none> 9102/TCP,9125/UDP 44m istio-system istio-telemetry ClusterIP 10.110.55.232 <none> 9091/TCP,15004/TCP,9093/TCP,42422/TCP 44m istio-system jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP 44m istio-system jaeger-collector ClusterIP 10.102.43.21 <none> 14267/TCP,14268/TCP 44m istio-system jaeger-query ClusterIP 10.104.182.189 <none> 16686/TCP 44m istio-system prometheus ClusterIP 10.100.0.70 <none> 9090/TCP 44m istio-system servicegraph ClusterIP 10.97.65.37 <none> 8088/TCP 44m istio-system tracing ClusterIP 10.109.87.118 <none> 80/TCP 44m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 52mGateway and virtual service described in point 2
$ kubectl describe gateways.networking.istio.io hr--gateway Name: hr--gateway Namespace: default API Version: networking.istio.io/v1alpha3 Kind: Gateway Metadata: ... Spec: Selector: App: hr--gateway Servers: Hosts: * Port: Name: http2 Number: 80 Protocol: HTTP2 Hosts: * Port: Name: https Number: 443 Protocol: HTTPS Tls: Mode: PASSTHROUGH $ kubectl describe virtualservices.networking.istio.io hr--gateway Name: hr--gateway Namespace: default Labels: app=hr--gateway Annotations: <none> API Version: networking.istio.io/v1alpha3 Kind: VirtualService Metadata: ... Spec: Gateways: hr--gateway Hosts: * Http: Match: Uri: Prefix: /info/ Rewrite: Uri: / Route: Destination: Host: hr--hr-serviceGateway and virtual service described in point 3
$ kubectl describe gateways.networking.istio.io ingress-gateway Name: ingress-gateway Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"ingress-gateway","namespace":"default"},"spec":{"sel... API Version: networking.istio.io/v1alpha3 Kind: Gateway Metadata: ... Spec: Selector: Istio: ingressgateway Servers: Hosts: * Port: Name: http2 Number: 80 Protocol: HTTP2 $ kubectl describe virtualservices.networking.istio.io hr--gateway-ingress-vs Name: hr--gateway-ingress-vs Namespace: default Labels: app=hr--gateway API Version: networking.istio.io/v1alpha3 Kind: VirtualService Metadata: Spec: Gateways: ingress-gateway Hosts: * Http: Match: Uri: Prefix: /info/ Route: Destination: Host: hr--gateway-service Events: <none> -
Pasan W. over 5 yearsSolution 1 works. However, I need to have mTLS between ingress and hr--gateway. So I attempted your second option. It gives me an error
server cannot have TLS settings for plain text HTTP portswhen trying to update the gateway. Can you please suggest why I get that? -
Vadim Eisenberg over 5 yearsTry to direct the traffic from ingress to hr--gateway thru port 443. -
Pasan W. over 5 yearsHi @Vadim. I changed the virtual service so that the traffic is routed to port 443 of hr--gateway and it worked fine. With this setup can we say the istio mesh is mTLS enabled? What I think is ingress to hr--gateway traffic is https instead of mTLS..
-
Vadim Eisenberg over 5 yearsIt is mTLS enabled if you specify
MUTUALin the server TLS options of hr--gateway. -
Vadim Eisenberg over 5 yearsTo be on the safe side, see the checks you can perform to ensure mTLS is enabled between the ingress gateway to hr gateway: preliminary.istio.io/docs/tasks/security/mutual-tls
-
Pasan W. over 5 yearsOne more question. Consider this scenario when the application(hr--hr-service) calls envoy (employee--gateway-service) with a curl command. Again this fails with a 503. employee--gateway-service is similar to hr--gateway-service and it routes to employee-employee-service.
-
Pasan W. over 5 yearsIf I explicitly route to port 443 with the curl command I get the expected result. Does this mean I have to change my application code to route to 443 when the mesh is mTLS enabled? Isn't envoy supposed to handle this without us having to change the code?
-
Vadim Eisenberg over 5 yearsI have a question - why do you need these gateways (hr-gateway and employee-gateway)? Regarding routing to port 443 - you can specify a VirtualService and rewrite the port by Istio, without changing the application. See an example of port rewriting by a VirtualService preliminary.istio.io/docs/examples/advanced-gateways/….
-
Vadim Eisenberg over 5 yearsI see, it makes sense.