As of Jan 29 2017, almost all Kubernetes networking documentation assume that the Kubernetes cluster runs in some kind of public or private cloud technology. For example, Openstack, AWS or GCE
What about Baremetal Kubernetes? Currently Mirantis seems to be the only company addressing the IP mobility problem in Baremetal Kubernetes.
What is the Problem?
One can deploy Kubernetes using a tool like Kargo onto baremetal CoreOS or Ubuntu hosts. Probably even on Redhat Atomic. But the problem is after you go through that work, how to do you configure IP Mobility for a pod within the cluster?
Kubernetes has the concept of configuring ExternalIPs on the Service Resources that you configure like so:
kind: Service
apiVersion: v1
metadata:
name: "frontend"
spec:
selector:
app: "frontend"
ports:
- protocol: "TCP"
port: 443
targetPort: 443
type: NodePort
externalIPs:
- 200.1.1.1
In the above example the frontend loadbalancer service is created for the frontend pod(app). The service is assigned 200.1.1.1 as the external facing IP. On baremetal Kubernetes, you the admin, are responsible for configuring the 200.1.1.1/32 IP address onto the appropriate Kubernete minions and redistributing this route into your network routing tables. Painful task!
Kubernetes clusters on top of Openstack, AWS or GCE have a solution for IP Mobility. It calls it cloud providers. Example:
kind: Service
apiVersion: v1
metadata:
name: "frontend"
spec:
selector:
app: "frontend"
ports:
- protocol: "TCP"
port: 443
targetPort: 443
type: LoadBalancer
The above configuration triggers the cloud provider code to give the service an externally facing IP. So almost by magic. providing the service with IP mobility. Watch Kelsey Hightower demonstrate a service created using a cloud provide loadbalancer. Notice in the video the External IP. It appears magically without any user intervention.
I’m not sure if you destroy and restore the service, if the cloud provider code gives the service the same IP each time..hmmm?
The IP Mobility Solution for Baremetal Kubernetes
Mirantis have developed a cloud provider of sorts. It is called the external IP controller. In a nutshell, its a set of apps that listens to service resource requests, i.e loadbalancer service requests, and adds the ExternalIP address to the interface of choice on the kubernetes minions. Using the example shown earlier, when the External IP controller is installed, it takes the External IP and adds it to an interface on the Kubernetes minion. Which Kubernetes minion does it add it to? The minion running a claimsController pod. The interface is determined by the external IP controller config.
So if you have multiple claimControllers you can place the external IP on multiple kubernetes minions and use ECMP routing instead of configuring a dedicated loadbalancer.
Where the claimsController is running
Notice it says the claimController is running on k8s6 and k8s9 nodes.
$ kubectl get po -l app=externalipcontroller -o wide
NAME READY STATUS RESTARTS AGE IP NODE
claimcontroller-711596365-1rv9j 1/1 Running 4 1d 10.1.3.6 k8s6
claimcontroller-711596365-b608s 1/1 Running 4 3d 10.1.3.9 k8s9
The number of claimController pods is managed by the claimController deployment
$ kubectl get deploy -l app=externalipcontroller
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
claimcontroller 2 2 2 2 6d
Where is the External IP placed?
Let’s review all the Kubernetes nodes in the cluster and examine their L3 config. The tool used to view L3 config is called netshow, a Cumulus Network app.
Notice that where the claimsController pod runs, the
200.1.1.1address is assigned to that interface.
$ ansible -m command -a 'netshow l3' all
k8s9 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ ---------------------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.28/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.9/32, 200.1.1.1/32, ::1/128
k8s1 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------- ------- ----- ------------ -------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.20/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.1/32, ::1/128
UP vagrant -1M 1500 Interface/L3 IP: 192.168.121.198/24
k8s6 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ ---------------------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.25/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.6/32, 200.1.1.1/32, ::1/128
k8s4 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ -------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.23/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.4/32, ::1/128
k8s3 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ -------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.22/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.3/32, ::1/128
k8s10 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ --------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.29/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.10/32, ::1/128
k8s11 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ --------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.30/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.11/32, ::1/128
k8s5 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ -------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.24/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.5/32, ::1/128
k8s2 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ -------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.21/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.2/32, ::1/128
k8s8 | SUCCESS | rc=0 >>
--------------------------------------------------------------------
Name Speed MTU Mode Summary
-- ------ ------- ----- ------------ -------------------------------------
UP eth0 -1M 1500 Interface/L3 IP: 192.168.200.27/24(DHCP)
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, 10.1.3.8/32, ::1/128
The External IP Controller can use any subnet mask you tell it and using IpPool definition files, those yaml files
that kubectl create accepts, you can vary the subnet mask based on the External IP selected.
The external IP controller can also auto-allocate the IP, just like the cloud provider code does. But after some time using it, it seems better to preset the IP using the ExternalIP keyword in the Services definition file.
Enabling IP External controller
There are instructions on the github page on how to configure it manually. It was easier through
to just include the kargo-compatible external IP controller role in the Kargo ansible playbook, then add this diff to the playbook. Then run ansible-playbook cluster.yml -t externalip to install the feature on the Kubernetes cluster.
iff --git a/inventory/group_vars/all.yml b/inventory/group_vars/all.yml
index d87d687..ec2a07f 100644
--- a/inventory/group_vars/all.yml
+++ b/inventory/group_vars/all.yml
@@ -1,3 +1,22 @@
+# External IP controller settings
+
+# Use the latest external ip controller code
+extip_image_tag: "latest"
+
+# Only create /32 entries
+extip_mask: 32
+
+# attach the created IPs to the lo interface
+extip_iface: lo
+
+# For ECMP place the generated IPs on 2 kubernetes minions
+extip_ctrl_replicas: 2
+
+# Apply IP on all claimcontrollers..use network ECMP
+extip_distribution: all
+
diff --git a/cluster.yml b/cluster.yml
index becad3b..d9225ea 100644
--- a/cluster.yml
+++ b/cluster.yml
@@ -26,7 +26,7 @@
- hosts: k8s-cluster:etcd:calico-rr
any_errors_fatal: true
roles:
- - { role: kubernetes/preinstall, tags: preinstall }
+ - { role: kubernetes/preinstall, tags: ['preinstall','externalip'] }
- { role: docker, tags: docker }
- { role: rkt, tags: rkt, when: "'rkt' in [ etcd_deployment_type, kubelet_deployment_type ]" }
@@ -58,6 +58,7 @@
roles:
- { role: dnsmasq, when: "dns_mode == 'dnsmasq_kubedns'", tags: dnsmasq }
- { role: kubernetes/preinstall, when: "dns_mode != 'none' and resolvconf_mode == 'host_resolvconf'", tags: resolvconf }
+ - { role: externalip , tags: 'externalip' }