I have a fresh kubernetes cluster (version 1.18) deployed using Rancher on a fresh Ubuntu Server 20.04. The problem I'm facing is that each https request made from any pod is failing with the following message:
curl -v https://pypi.org/simple/pip/
* Trying 44.227.65.245...
* Connected to pypi.org (44.227.65.245) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: Handshake failed
* Closing connection 0
curl: (35) gnutls_handshake() failed: Handshake failed
I've tested it using
kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.4
kubectl exec -it <pod_id> bash
I tried the same thing via ssh on the host system and everything was working properly. I also tried running standalone docker container (on the same machine) with the same image as pod (docker run -it --rm k8s.gcr.io/echoserver:1.4 bash
) and there were no problems here as well. So I'm guessing that this is something related to my kubernetes setup.
It happend only with https requests – http requests are working properly.
I tried to remove the cluster and prepare it from scratch but this didn't help.
UPDATE:
I'm receiving two types of messages, depending on the server I'm trying to connect:
root@hello-node-7bf657c596-smr2h:/# openssl s_client -connect google.com:443 -debug
CONNECTED(00000003)
write to 0xda15b0 [0xda1630] (305 bytes => 305 (0x131))
0000 - 16 03 01 01 2c 01 00 01-28 03 03 4e c0 cf d9 62 ....,...(..N...b
0010 - 58 6e 88 7b 28 7f e8 c1-62 c6 8f 23 60 86 b0 53 Xn.{(...b..#`..S
0020 - df 43 cd a7 58 52 bc 59-6a ae bf 00 00 aa c0 30 .C..XR.Yj......0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 11 c0 07 c0 0c c0 02 .<./...A........
00c0 - 00 05 00 04 c0 12 c0 08-00 16 00 13 00 10 00 0d ................
00d0 - c0 0d c0 03 00 0a 00 ff-01 00 00 55 00 0b 00 04 ...........U....
00e0 - 03 00 01 02 00 0a 00 1c-00 1a 00 17 00 19 00 1c ................
00f0 - 00 1b 00 18 00 1a 00 16-00 0e 00 0d 00 0b 00 0c ................
0100 - 00 09 00 0a 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... ....
0110 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................
0120 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................
0130 - 01 .
read from 0xda15b0 [0xda6b90] (7 bytes => 7 (0x7))
0000 - 15 00 00 00 02 02 28 ......(
140080050284184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1600154660
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
and
root@hello-node-7bf657c596-smr2h:/# openssl s_client -connect pypi.org:443 -debug
CONNECTED(00000003)
write to 0x24645b0 [0x2464630] (305 bytes => 305 (0x131))
0000 - 16 03 01 01 2c 01 00 01-28 03 03 0c 2b bf dc 71 ....,...(...+..q
0010 - d2 41 56 f3 40 e4 c5 3f-62 88 cf 2a a7 76 e4 a4 .AV.@..?b..*.v..
0020 - 08 96 13 89 30 13 fa 75-c9 2d 32 00 00 aa c0 30 ....0..u.-2....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 11 c0 07 c0 0c c0 02 .<./...A........
00c0 - 00 05 00 04 c0 12 c0 08-00 16 00 13 00 10 00 0d ................
00d0 - c0 0d c0 03 00 0a 00 ff-01 00 00 55 00 0b 00 04 ...........U....
00e0 - 03 00 01 02 00 0a 00 1c-00 1a 00 17 00 19 00 1c ................
00f0 - 00 1b 00 18 00 1a 00 16-00 0e 00 0d 00 0b 00 0c ................
0100 - 00 09 00 0a 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... ....
0110 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................
0120 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................
0130 - 01 .
read from 0x24645b0 [0x2469b90] (7 bytes => 7 (0x7))
0000 - 15 03 03 00 02 02 28 ......(
140284912154264:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1600154752
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
I found this two messages appearing together described here: https://blog.techstacks.com/2010/03/3-common-causes-of-unknown-ssl-protocol-errors-with-curl.html
The Destination Site Does Not Like the Cipher You could be trying to
connect to the site using an ssl cipher that the site is configured to
reject. For example, anonymous ciphers are typically disabled on
ssl-encrypted sites that are customer-facing. (Many of us set a
blanket rejection policy on any SSL-encrypted web site—regardless of
it's purpose.) The following command string "can" also result in the
curl (35) error:curl –ciphers ADH-RC4-MD5 https://some_web_site.some_domain.com/
Unfortunately, the type of error response you can get from curl
depends largely upon the ssl server. On some sites, you'll receive the
Unknown SSL Protocol error but on my techstacks-tools site, I get:curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3
alert handshake failure
Another thing I dicovered is that I prepare the same environment on the DigitalOcean droplet and ther were no problems there.
My bare metal environment is not available from the world, it stands inside internal network – maybe this is somehow related?
I also tried to setup k8s using microk8s but the problem still occurs so this is probably not related to rancher but only to my machine setup.
YAML of the pod:
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/podIP: 10.42.0.8/32
cni.projectcalico.org/podIPs: 10.42.0.8/32
creationTimestamp: "2020-09-14T19:51:57Z"
generateName: hello-node-7bf657c596-
labels:
app: hello-node
pod-template-hash: 7bf657c596
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:generateName: {}
f:labels:
.: {}
f:app: {}
f:pod-template-hash: {}
f:ownerReferences:
.: {}
k:{"uid":"dda770d0-3aa3-4fc9-97e1-c929fc3629e9"}:
.: {}
f:apiVersion: {}
f:blockOwnerDeletion: {}
f:controller: {}
f:kind: {}
f:name: {}
f:uid: {}
f:spec:
f:containers:
k:{"name":"echoserver"}:
.: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:dnsPolicy: {}
f:enableServiceLinks: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
manager: kube-controller-manager
operation: Update
time: "2020-09-14T19:51:57Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
.: {}
k:{"type":"PodScheduled"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
manager: kube-scheduler
operation: Update
time: "2020-09-14T19:51:57Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:cni.projectcalico.org/podIP: {}
f:cni.projectcalico.org/podIPs: {}
manager: calico
operation: Update
time: "2020-09-14T19:52:01Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"ContainersReady"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Initialized"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Ready"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
f:containerStatuses: {}
f:hostIP: {}
f:phase: {}
f:podIP: {}
f:podIPs:
.: {}
k:{"ip":"10.42.0.8"}:
.: {}
f:ip: {}
f:startTime: {}
manager: kubelet
operation: Update
time: "2020-09-14T19:52:01Z"
name: hello-node-7bf657c596-smr2h
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: hello-node-7bf657c596
uid: dda770d0-3aa3-4fc9-97e1-c929fc3629e9
resourceVersion: "3468"
selfLink: /api/v1/namespaces/default/pods/hello-node-7bf657c596-smr2h
uid: cf37b215-8269-42ce-9e70-750f9f862cac
spec:
containers:
- image: k8s.gcr.io/echoserver:1.4
imagePullPolicy: IfNotPresent
name: echoserver
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-9jlms
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: gepard
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-9jlms
secret:
defaultMode: 420
secretName: default-token-9jlms
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2020-09-14T19:52:00Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2020-09-14T19:52:01Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2020-09-14T19:52:01Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2020-09-14T19:52:00Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://2d2ddbf42fc4e6634a782c7036cf4a9a1d9f50a3d847bb5444932c701cd8186e
image: k8s.gcr.io/echoserver:1.4
imageID: docker-pullable://k8s.gcr.io/echoserver@sha256:5d99aa1120524c801bc8c1a7077e8f5ec122ba16b6dda1a5d3826057f67b9bcb
lastState: {}
name: echoserver
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2020-09-14T19:52:01Z"
hostIP: 10.0.0.3
phase: Running
podIP: 10.42.0.8
podIPs:
- ip: 10.42.0.8
qosClass: BestEffort
startTime: "2020-09-14T19:52:00Z"
Firewall is disabled at the moment.
Best Answer
The OP chose not to post an answer, but did say that they solved the problem: