F5与k8s配合-without F5 Container Connector（CC）

2017年07月12日 11367点热度 2人点赞 0条评论

思路：

k8s内使用headless service，service配置文件中配置selector，这样k8s内部的dns服务将把endpoint的IP作为 A记录到相关服务的名称下

让F5可以使用k8s的DNS 服务

在F5内使用FQDN方式配置pool，这样F5将自动获取所有k8s内的endpoint

优点：

不用关心容器内复杂的服务调度及iptables

无需使用api来创建F5配置

F5直接将流量送给endpoint效率高

endpoint内容器的健康性、扩展性由k8s负责，未被破坏,F5 monitor可以附加对应用的健康检查

pod漂移发生的地址变化，k8s内dns会自动更新，F5只需解析相关FQDN即可

F5只是从外部帮助引流。

无需F5 k8s-bigip-ctlr

要求：

endpoint可以直接通过宿主机外部访问到，比如flannel网络下，从一个k8s完全外部的client访问k8s创建的endpoint是可以的。

或者容器本身映射服务到宿主机，例如

apiVersion: v1
kind: Pod
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  containers:
  - name: webapp
    image: tomcat
    ports:
    - containerPort: 8080
      hostPort:8081

apiVersion: v1

kind: Pod

metadata:

labels:

app: webapp

spec:

containers:

- name: webapp

image: tomcat

ports:

- containerPort: 8080

hostPort:8081

或者pod使用hostNetwork（这种网络模型最容易），例如

apiVersion: v1
kind: Pod
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  hostNetwork: true
  containers:
  - name: webapp
    image: tomcat
    imagePullPolicy: Never
    ports:
    - containerPort: 8080

apiVersion: v1

kind: Pod

metadata:

labels:

app: webapp

spec:

hostNetwork: true

containers:

- name: webapp

image: tomcat

imagePullPolicy: Never

ports:

- containerPort: 8080

注意点：

F5 FQDN pool功能需要设置较小的dns查询价格，并设置忽略A记录的TTL时间

难点：

要确保kube-dns的对外服务IP，对F5来说能够恒定！

快速测试：

从flannel外部访问endpoint：

1.在外部机器上加入指向到endpoint网络的路由

BEI-ML-JLIN-:~ jlin$ sudo route add -net 10.2.4.0/24 172.16.199.27

2.验证路由

BEI-ML-JLIN-:~ jlin$ netstat -nr

Routing tables

Internet:

Destination        Gateway            Flags        Refs      Use   Netif Expire

default            192.168.0.1        UGSc           34        0     en0

default            link#12            UCSI            1        0 bridge1

10.2.4/24          172.16.199.27      UGSc            0        0  vmnet3

127                127.0.0.1          UCS             0        0     lo0

127.0.0.1          127.0.0.1          UH             35 10657422     lo0

169.254            link#5             UCS             0        0     en0

172.16.150/24      link#18            UC              1        0  vmnet8

172.16.199/24      link#15            UC              3        0  vmnet3

172.16.199.17      0:c:29:42:d:ac     UHLWIi          2    11146  vmnet3    835

172.16.199.27      0:c:29:ae:11:8d    UHLWIi          2     1381  vmnet3    754

172.16.199.37      0:c:29:7d:f5:be    UHLWIi          1     1148  vmnet3    806

BEI-ML-JLIN-:~ jlin$ netstat -nr

Routing tables

Internet:

Destination Gateway Flags Refs Use Netif Expire

default 192.168.0.1 UGSc 34 0 en0

default link#12 UCSI 1 0 bridge1

10.2.4/24 172.16.199.27 UGSc 0 0 vmnet3

127 127.0.0.1 UCS 0 0 lo0

127.0.0.1 127.0.0.1 UH 35 10657422 lo0

169.254 link#5 UCS 0 0 en0

172.16.150/24 link#18 UC 1 0 vmnet8

172.16.199/24 link#15 UC 3 0 vmnet3

172.16.199.17 0:c:29:42:d:ac UHLWIi 2 11146 vmnet3 835

172.16.199.27 0:c:29:ae:11:8d UHLWIi 2 1381 vmnet3 754

172.16.199.37 0:c:29:7d:f5:be UHLWIi 1 1148 vmnet3 806

3. k8s内设置

创建一个新的headless svc：

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: my-headless-service
  name: my-headless-service
  namespace: default
spec:
  clusterIP: None
  ports:
  - port: 9999
    targetPort: 80
  selector:
    run: k8s-nginx

kind: Service

apiVersion: v1

metadata:

labels:

k8s-app: my-headless-service

namespace: default

spec:

clusterIP: None

ports:

- port: 9999

targetPort: 80

selector:

run: k8s-nginx

此时kube-dns内的解析：

[root@docker3 ~]# dig @10.2.39.2 my-headless-service.default.svc.cluster.local

; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.3 <<>> @10.2.39.2 my-headless-service.default.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44048
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;my-headless-service.default.svc.cluster.local. IN A

;; ANSWER SECTION:
my-headless-service.default.svc.cluster.local. 30 IN A 10.2.39.3
my-headless-service.default.svc.cluster.local. 30 IN A 10.2.4.2

;; Query time: 0 msec
;; SERVER: 10.2.39.2#53(10.2.39.2)
;; WHEN: Wed Jul 12 09:23:28 CST 2017
;; MSG SIZE rcvd: 95

[root@docker3 ~]# dig @10.2.39.2 my-headless-service.default.svc.cluster.local

; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.3 <<>> @10.2.39.2 my-headless-service.default.svc.cluster.local

; (1 server found)

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44048

;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:

;my-headless-service.default.svc.cluster.local. IN A

;; ANSWER SECTION:

my-headless-service.default.svc.cluster.local. 30 IN A 10.2.39.3

my-headless-service.default.svc.cluster.local. 30 IN A 10.2.4.2

;; Query time: 0 msec

;; SERVER: 10.2.39.2#53(10.2.39.2)

;; WHEN: Wed Jul 12 09:23:28 CST 2017

;; MSG SIZE rcvd: 95

[root@docker1 app]# kubectl exec busybox ping my-headless-service.default
PING my-headless-service.default (10.2.39.3): 56 data bytes
64 bytes from 10.2.39.3: seq=0 ttl=62 time=1.186 ms
64 bytes from 10.2.39.3: seq=1 ttl=62 time=0.686 ms
^C
[root@docker1 app]# kubectl exec busybox ping my-headless-service.default
PING my-headless-service.default (10.2.4.2): 56 data bytes
64 bytes from 10.2.4.2: seq=0 ttl=64 time=0.462 ms
64 bytes from 10.2.4.2: seq=1 ttl=64 time=0.106 ms

[root@docker1 app]# kubectl exec busybox ping my-headless-service.default

PING my-headless-service.default (10.2.39.3): 56 data bytes

64 bytes from 10.2.39.3: seq=0 ttl=62 time=1.186 ms

64 bytes from 10.2.39.3: seq=1 ttl=62 time=0.686 ms

[root@docker1 app]# kubectl exec busybox ping my-headless-service.default

PING my-headless-service.default (10.2.4.2): 56 data bytes

64 bytes from 10.2.4.2: seq=0 ttl=64 time=0.462 ms

64 bytes from 10.2.4.2: seq=1 ttl=64 time=0.106 ms

下面10.2.4.2实际是k8s创建的一个pod的IP，可以看到从外部电脑可以访问pod

BEI-ML-JLIN-:~ jlin$ curl http://10.2.4.2

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

    body {

        width: 35em;

        margin: 0 auto;

        font-family: Tahoma, Verdana, Arial, sans-serif;

    }

</style>

</head>

<body>

<h1>Welcome to nginx!</h1>

<p>If you see this page, the nginx web server is successfully installed and

working. Further configuration is required.</p>

<p>For online documentation and support please refer to

<a href="http://nginx.org/">nginx.org</a>.<br/>

Commercial support is available at

<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

</body>

</html>

BEI-ML-JLIN-:~ jlin$ curl http://10.2.4.2

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

body {

width: 35em;

margin: 0 auto;

font-family: Tahoma, Verdana, Arial, sans-serif;

}

</style>

</head>

<body>

<h1>Welcome to nginx!</h1>

<p>If you see this page, the nginx web server is successfully installed and

working. Further configuration is required.</p>

<p>For online documentation and support please refer to

<a href="http://nginx.org/">nginx.org</a>.<br/>

Commercial support is available at

<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

</body>

</html>

外部DNS解析：

dig @10.2.39.2 my-headless-service.default.svc.cluster.local

; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.3 <<>> @10.2.39.2 my-headless-service.default.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44048
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;my-headless-service.default.svc.cluster.local. IN A

;; ANSWER SECTION:
my-headless-service.default.svc.cluster.local. 30 IN A 10.2.39.3
my-headless-service.default.svc.cluster.local. 30 IN A 10.2.4.2

;; Query time: 0 msec
;; SERVER: 10.2.39.2#53(10.2.39.2)
;; WHEN: Wed Jul 12 09:23:28 CST 2017
;; MSG SIZE  rcvd: 95