如何在F5 CIS方案中通过AS3声明式API暴露K8S服务

2019年04月19日 17664点热度 7人点赞 2条评论

Warning: Array to string conversion in /home/wwwroot/myf5/wp-content/plugins/wpjam-basic/public/wpjam-utils.php on line 496

对于k8s，openshift等PaaS平台，F5通过Container Ingress Services（CIS，以前叫Container Connector）解决方案实现通过F5 BIGIP将上述PaaS平台中需要对外暴露的服务发布到BIGIP上，从而借助BIGIP更多的应用服务交付能力，并解决原生平台在服务对外暴露上的一些问题。

在CIS解决方案中，通过运行一个bigip controller 容器（以下简称cc），实现对平台中发布的service等资源进行监听并将其转化为BIGIP上的配置。

在cc V1.9.0版本之前并不支持F5 Application Services 3(AS3)特性，因此在之前的版本上，如果需要配置cc本身还没有支持的一些ADC产品功能，就需要通过iApp模式来进行高级配置的服务发布，此时就需要用户能够熟悉iApp的编写，并将编写的iApp模板提前上传到BIGIP产品里，这将复杂化用户devops pipline以及需要开发人员熟悉与了解F5产品特性与配置，提高了用户门槛，整个实现方式也不够Cloud Native。

从CC V1.9.0开始支持AS3这一声明式API配置方式，这大大简化了配置方式，开发人员也无需了解很多F5产品细节，只需要通过申明需要创建的服务就可以实现自动化配置，在devops pipeline中也无需直接与设备进行交付并处理iApp相关配置，使得整个“source truth”都只存在于configmap中。

AS3配置方式与非AS3配置方式对比：

非AS3配置:

[root@k8s-master f5-k8s]# cat f5-vs.yaml 
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-deploy-svc.vs
  labels:
    f5type: virtual-server
data:
  # See the f5-schema table for schema-controller compatibility
  # https://clouddocs.f5.com/containers/latest/releases_and_versioning.html#f5-schema
  schema: "f5schemadb://bigip-virtual-server_v0.1.7.json"
  data: |
    {
      "virtualServer": {
        "backend": {
          "servicePort": 80,
          "serviceName": "nginx-deploy-svc",
          "healthMonitors": [{
            "interval": 15,
            "protocol": "http",
            "send": "GET / HTTP/1.1\r\nConnection: close\r\nHost: 1.1.1.1\r\n\r\n",
            "recv": "cka",
            "timeout": 30
          }]
        },
        "frontend": {
          "virtualAddress": {
            "port": 80,
            "bindAddr": "172.16.40.80"
          },
          "partition": "k8s",
          "balance": "least-connections-member",
          "mode": "http"
        }
      }
    }

可以看到在上述data部分下配置virtual server的frontend，backend两个部分，frontend部分主要配置关于VSIP，端口，profile，LB算法等，backend部分主要是monitor以及要自动化关注的哪个k8s service。如果使用了iApp，则还会包含iapp部分，例如:

kind: ConfigMap
apiVersion: v1
metadata:
  name: k8s.http
  namespace: default
  labels:
    f5type: virtual-server
data:
  # See the f5-schema table for schema-controller compatibility
  # https://clouddocs.f5.com/containers/latest/releases_and_versioning.html#f5-schema
  schema: "f5schemadb://bigip-virtual-server_v0.1.7.json"
  data: |
    {
      "virtualServer": {
        "backend": {
          "serviceName": "myService",
          "servicePort": 80
        },
        "frontend": {
          "partition": "k8s",
          "iapp": "/Common/f5.http",
          "iappPoolMemberTable": {
            "name": "pool__members",
            "columns": [
                {"name": "addr", "kind": "IPAddress"},
                {"name": "port", "kind": "Port"},
                {"name": "connection_limit", "value": "0"}
            ]
          },
          "iappOptions": {
            "description": "myService_f5.http iApp"
          },
          "iappVariables": {
            "monitor__monitor": "/#create_new#",
            "monitor__response": "none",
            "monitor__uri": "/",
            "net__client_mode": "wan",
            "net__server_mode": "lan",
            "pool__addr": "127.0.0.2",
            "pool__pool_to_use": "/#create_new#",
            "pool__port": "4430"
          }
        }
      }
    }

AS3配置方式：

AS3方式配置的configmap则只需要在data部分完全填入一个AS3的声明json template，其中CC会自动化的修改member部分的servicePort和serverAddresses的配置（如果已明确配置了上述内容，则CIS/CC不会修改）。这样只要AS3本身能够支持的就可以在configmap中使用，极大的丰富了配置能力，并使得客户可以统一化配置思想与方式，以code方式保持与管理F5配置。

[root@k8s-master f5-k8s]# cat f5-vs-as3.yaml 
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-as3
  labels:
    f5type: virtual-server
    as3: "true"
data:
  template: |
    {
    "class": "AS3",
    "action": "deploy",
    "persist": true,
    "declaration": {
        "class": "ADC",
        "schemaVersion": "3.10.0",
        "id": "123abc",
        "label": "k8s",
        "remark": "HTTPS with predictive-node pool and connection limit",
        "k8s": {
        "class": "Tenant",
        "nginxservice": {
            "class": "Application",
            "template": "https",
            "serviceMain": {
            "class": "Service_HTTPS",
            "virtualAddresses": [
                "192.0.2.11"
            ],
            "pool": "web_pool",
            "serverTLS": "webtls"
            },
            "web_pool": {
            "class": "Pool",
            "loadBalancingMode": "predictive-node",
            "monitors": [
                "http"
            ],
            "members": [{
                "servicePort": 80,
                "serverAddresses": [],
                "connectionLimit": 10
            }]
            },
            "webtls": {
            "class": "TLS_Server",
            "certificates": [{
                "certificate": "webcert"
            }]
            },
            "webcert": {
            "class": "Certificate",
            "remark": "in practice we recommend using a passphrase",
            "certificate": "-----BEGIN CERTIFICATE-----\nMIICnDCCAgWgAwIBAgIJAJ5n2b0OCEjwMA0GCSqGSIb3DQEBCwUAMGcxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApXYXNoaW5ndG9uMRAwDgYDVQQHDAdTZWF0dGxlMRQwEgYDVQQKDAtmNV9OZXR3b3JrczEbMBkGA1UEAwwSc2FtcGxlLmV4YW1wbGUubmV0MB4XDTE3MTEyNjE5NTAyNFoXDTE4MDIyNTE5NTAyNFowZzELMAkGA1UEBhMCVVMxEzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxFDASBgNVBAoMC2Y1X05ldHdvcmtzMRswGQYDVQQDDBJzYW1wbGUuZXhhbXBsZS5uZXQwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBALEsuXmSXVQpYjrZPW+WiTBjn491mwZYT7Q92V1HlSBtM6WdWlK1aZN5sovfKtOX7Yrm8xa+e4o/zJ2QYLyyv5O+t2EGN/4qUEjEAPY9mwJdfzRQy6Hyzm84J0QkTuUJ/EjNuPji3D0QJRALUTzu1UqqDCEtiN9OGyXEkh7uvb7BAgMBAAGjUDBOMB0GA1UdDgQWBBSVHPNrGWrjWyZvckQxFYWO59FRFjAfBgNVHSMEGDAWgBSVHPNrGWrjWyZvckQxFYWO59FRFjAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4GBAJeJ9SEckEwPhkXOm+IuqfbUS/RcziifBCTmVyE+Fa/j9pKSYTgiEBNdbJeBEa+gPMlQtbV7Y2dy8TKx/8axVBHiXC5geDML7caxOrAyHYBpnx690xJTh5OIORBBM/a/NvaR+P3CoVebr/NPRh9oRNxnntnqvqD7SW0U3ZPe3tJc\n-----END CERTIFICATE-----",
            "privateKey": "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-256-CBC,D8FFCE6B255601587CB54EC29B737D31\n\nkv4Fc3Jn0Ujkj0yRjt+gQQfBLSNF2aRLUENXnlr7Xpzqu0Ahr3jS1bAAnd8IWnsR\nyILqVmKsYF2DoHh0tWiEAQ7/y/fe5DTFhK7N4Wml6kp2yVMkP6KC4ssyYPw27kjK\nDBwBZ5O8Ioej08A5sgsLCmglbmtSPHJUn14pQnMTmLOpEtOsu6S+2ibPgSNpdg0b\nCAJNG/KHe+Vkx59qNDyDeKb7FZOlsX30+y67zUq9GQqJEDuysPJ2BUNP0IJXAjst\nFIt1qNoZew+5KDYs7u/lPxcMGTirUhgI84Jy4WcDvSOsP/tKlxj04TbIE3epmSKy\n+TihHkwY7ngIGtcm3Sfqk5jz2RXoj1/Ac3SW8kVTYaOUogBhn7zAq4Wju6Et4hQG\nRGapsJp1aCeZ/a4RCDTxspcKoMaRa97/URQb0hBRGx3DGUhzpmX9zl7JI2Xa5D3R\nmdBXtjLKYJTdIMdd27prBEKhMUpae2rz5Mw4J907wZeBq/wu+zp8LAnecfTe2nGY\nE32x1U7gSEdYOGqnwxsOexb1jKgCa67Nw9TmcMPV8zmH7R9qdvgxAbAtwBl1F9OS\nfcGaC7epf1AjJLtaX7krWmzgASHl28Ynh9lmGMdv+5QYMZvKG0LOg/n3m8uJ6sKy\nIzzvaJswwn0j5P5+czyoV5CvvdCfKnNb+3jUEN8I0PPwjBGKr4B1ojwhogTM248V\nHR69D6TxFVMfGpyJhCPkbGEGbpEpcffpgKuC/mEtMqyDQXJNaV5HO6HgAJ9F1P6v\n5ehHHTMRvzCCFiwndHdlMXUjqSNjww6me6dr6LiAPbejdzhL2vWx1YqebOcwQx3G\n-----END RSA PRIVATE KEY-----",
            "passphrase": {
                "ciphertext": "ZjVmNQ==",
                "protected": "eyJhbGciOiJkaXIiLCJlbmMiOiJub25lIn0"
            }
            }
        }
        }
    }
    }

上述cm配置中，可以看到serverAddresses数组是一个空数组，CC将会自动填写该部分配置。但是CC是如何知道应该将哪个k8s中的service所关联的pods ip（如果是以cluster方式发布服务）或者node ip（如果是以nodeport方式发布发布服务）填写到上述空数组里呢？

在上述配置中，有几个关键的用于标记一个应用的配置，分别是：
tenant信息，即表示对应一个BIGIP上的partition：

    "k8s": {
    "class": "Tenant",

应用名称信息 nginxservice，该名称将与partition名称一起共同组成的配置对象的真实path： k8s/nginxservice

"nginxservice": {
    "class": "Application",

pool名称信息：

            "web_pool": {
            "class": "Pool",

通过将一个k8s service打上与上述三个信息关联的label，从而可以让CC了解上述AS3应该与哪个service产生关联。

因此在以AS3模式发布下，需要在发布k8s service时候在yaml文件里打上一些labels：

cis.f5.com/as3-app: nginxservice
cis.f5.com/as3-pool: web_pool
cis.f5.com/as3-tenant: k8s

[root@k8s-master f5-k8s]# kubectl get svc  nginx-deploy-svc -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-12-14T12:14:56Z
  labels:
    cis.f5.com/as3-app: nginxservice
    cis.f5.com/as3-pool: web_pool
    cis.f5.com/as3-tenant: k8s
  name: nginx-deploy-svc
  namespace: default
  resourceVersion: "20470610"
  selfLink: /api/v1/namespaces/default/services/nginx-deploy-svc
  uid: dead50a0-ff99-11e8-9344-005056b34685
spec:
  clusterIP: 10.250.0.75
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx-deploy
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

需要注意的是，对于同一个Service不能由多个configmap来对其进行配置，这将导致冲突，比如不可以同时使用传统配置模式又附加AS3配置模式对同一个服务进行配置。
另外，
当前CC的AS3模式不支持对同一个VS声明多个pool
vxlan中产生的静态arp条目在AS3删除配置后并不会被删除

上述AS3最终产生的F5配置:

root@(v13-common)(cfg-sync Changes Pending)(Active)(/k8s/nginxservice)(tmos)# list ltm
ltm pool web_pool {
    load-balancing-mode predictive-node
    members {
        /k8s/10.244.0.78:http {
            address 10.244.0.78
            connection-limit 10
            session monitor-enabled
            state up
            metadata {
                source {
                    value declaration
                }
            }
        }
        /k8s/10.244.1.35:http {
            address 10.244.1.35
            connection-limit 10
            session monitor-enabled
            state up
            metadata {
                source {
                    value declaration
                }
            }
        }
    }
    min-active-members 1
    monitor min 1 of { /Common/http }
    partition k8s
}
ltm profile client-ssl webtls {
    allow-expired-crl disabled
    app-service none
    authenticate once
    c3d-drop-unknown-ocsp-status drop
    c3d-ocsp none
    ca-file none
    cert webcert.crt
    cert-key-chain {
        set0 {
            cert webcert.crt
            key webcert.key
            passphrase $M$NH$TKVv9Ro1/CpcF/9+dkUcJg==
        }
    }
    chain none
    cipher-group none
    ciphers DEFAULT
    client-cert-ca none
    crl-file none
    description none
    inherit-certkeychain false
    key webcert.key
    passphrase $M$V3$cTTm7s0n49/MeYNetBFRfg==
    peer-cert-mode ignore
    server-name none
    sni-default true
    sni-require false
    ssl-c3d disabled
}
ltm tacdb licenseddb /Common/licensed-tacdb {
    partition none
}
ltm virtual serviceMain {
    description nginxservice
    destination /k8s/192.0.2.11:https
    ip-protocol tcp
    mask 255.255.255.255
    partition k8s
    persist {
        /Common/cookie {
            default yes
        }
    }
    pool web_pool
    profiles {
        /Common/f5-tcp-progressive { }
        /Common/http { }
        webtls {
            context clientside
        }
    }
    source 0.0.0.0/0
    source-address-translation {
        type automap
    }
    translate-address enabled
    translate-port enabled
    vs-index 92
}
ltm virtual serviceMain-Redirect {
    description nginxservice
    destination /k8s/192.0.2.11:http
    ip-protocol tcp
    mask 255.255.255.255
    partition k8s
    profiles {
        /Common/f5-tcp-progressive { }
        /Common/http { }
    }
    rules {
        /Common/_sys_https_redirect
    }
    source 0.0.0.0/0
    source-address-translation {
        type automap
    }
    translate-address enabled
    translate-port enabled
    vs-index 93
}

需要特别注意的是，在通过as3-configmap配置方式进行配置的时候，as3中的tenant名称不要与CC里配置的bigip partition名称重复。也就是说，cc里配置的bigip partition是非as3方式配置的configmap所配置的服务存储位置，而as3声明的configmap则根据as3的特性自动化在BIGIP上创建新的对应的tenant，两者是相互排斥的。所以一个cc + as3-configmap 则意味着在BIGIP上存在两个不同的partition。

另外，需要注意vxlan模型下AS3实例支持不能超过1个。

cc，cc-bigip-partition, as3 tenant，bigip partition 之间配置关系为：

bigip-ctrl in kube-system ns, monitor default ns and map to k8s partition
svc1 in default ns ---> configmap1(include as3) in default ns -----> k8sas3 partition
svc2 in cka  ns ------> configmap2(include as3) in defualt ns ----->ckaas3 partition
----> good, both svc are created, and no obvious error in cc logs


bigip-ctrl in kube-system ns, monitor default ns and map to k8s partition
svc1 in default ns ----> configmap1(include as3) in default ns ---> k8sas3 partition(tenant)
svc2 in cka  ns -------> configmap2(include as3) in cka ns -------> ckaas3 partition(tenant)
---->the configmap2 does not create bigip objects


bigip-ctrl in kube-system ns, monitor default ns and map to k8s partition
+ bigip-ctrl in cka ns, monitor cka ns and map to cka partition
svc1 in default ns ---> configmap1(include as3) in default ns ---> k8sas3 partition(tenant)
svc2 in cka  ns ------> configmap2(include as3) in cka ns -------> ckaas3 partition(tenant)
---->good, both svc are created, and no obvious error in cc logs

附：如果同一个k8s partition下发布2个服务，则AS3格式为：

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-as3
  labels:
    f5type: virtual-server
    as3: "true"
data:
  template: |
    {
    "class": "AS3",
    "action": "deploy",
    "persist": true,
    "declaration": {
        "class": "ADC",
        "schemaVersion": "3.10.0",
        "id": "123abc",
        "label": "k8s",
        "remark": "HTTPS with predictive-node pool and connection limit",
        "k8s": {
        "class": "Tenant",
        "nginxservice": {
            "class": "Application",
            "template": "https",
            "serviceMain": {
            "class": "Service_HTTPS",
            "virtualAddresses": [
                "192.0.2.11"
            ],
            "pool": "web_pool",
            "serverTLS": "webtls"
            },
            "web_pool": {
            "class": "Pool",
            "loadBalancingMode": "predictive-node",
            "monitors": [
                "http"
            ],
            "members": [{
                "servicePort": 80,
                "serverAddresses": [],
                "connectionLimit": 10
            }]
            },
            "webtls": {
            "class": "TLS_Server",
            "certificates": [{
                "certificate": "webcert"
            }]
            },
            "webcert": {
            "class": "Certificate",
            "remark": "in practice we recommend using a passphrase",
            "certificate": "-----BEGIN CERTIFICATE-----\nMIICnDCCAgWgAwIBAgIJAJ5n2b0OCEjwMA0GCSqGSIb3DQEBCwUAMGcxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApXYXNoaW5ndG9uMRAwDgYDVQQHDAdTZWF0dGxlMRQwEgYDVQQKDAtmNV9OZXR3b3JrczEbMBkGA1UEAwwSc2FtcGxlLmV4YW1wbGUubmV0MB4XDTE3MTEyNjE5NTAyNFoXDTE4MDIyNTE5NTAyNFowZzELMAkGA1UEBhMCVVMxEzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxFDASBgNVBAoMC2Y1X05ldHdvcmtzMRswGQYDVQQDDBJzYW1wbGUuZXhhbXBsZS5uZXQwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBALEsuXmSXVQpYjrZPW+WiTBjn491mwZYT7Q92V1HlSBtM6WdWlK1aZN5sovfKtOX7Yrm8xa+e4o/zJ2QYLyyv5O+t2EGN/4qUEjEAPY9mwJdfzRQy6Hyzm84J0QkTuUJ/EjNuPji3D0QJRALUTzu1UqqDCEtiN9OGyXEkh7uvb7BAgMBAAGjUDBOMB0GA1UdDgQWBBSVHPNrGWrjWyZvckQxFYWO59FRFjAfBgNVHSMEGDAWgBSVHPNrGWrjWyZvckQxFYWO59FRFjAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4GBAJeJ9SEckEwPhkXOm+IuqfbUS/RcziifBCTmVyE+Fa/j9pKSYTgiEBNdbJeBEa+gPMlQtbV7Y2dy8TKx/8axVBHiXC5geDML7caxOrAyHYBpnx690xJTh5OIORBBM/a/NvaR+P3CoVebr/NPRh9oRNxnntnqvqD7SW0U3ZPe3tJc\n-----END CERTIFICATE-----",
            "privateKey": "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-256-CBC,D8FFCE6B255601587CB54EC29B737D31\n\nkv4Fc3Jn0Ujkj0yRjt+gQQfBLSNF2aRLUENXnlr7Xpzqu0Ahr3jS1bAAnd8IWnsR\nyILqVmKsYF2DoHh0tWiEAQ7/y/fe5DTFhK7N4Wml6kp2yVMkP6KC4ssyYPw27kjK\nDBwBZ5O8Ioej08A5sgsLCmglbmtSPHJUn14pQnMTmLOpEtOsu6S+2ibPgSNpdg0b\nCAJNG/KHe+Vkx59qNDyDeKb7FZOlsX30+y67zUq9GQqJEDuysPJ2BUNP0IJXAjst\nFIt1qNoZew+5KDYs7u/lPxcMGTirUhgI84Jy4WcDvSOsP/tKlxj04TbIE3epmSKy\n+TihHkwY7ngIGtcm3Sfqk5jz2RXoj1/Ac3SW8kVTYaOUogBhn7zAq4Wju6Et4hQG\nRGapsJp1aCeZ/a4RCDTxspcKoMaRa97/URQb0hBRGx3DGUhzpmX9zl7JI2Xa5D3R\nmdBXtjLKYJTdIMdd27prBEKhMUpae2rz5Mw4J907wZeBq/wu+zp8LAnecfTe2nGY\nE32x1U7gSEdYOGqnwxsOexb1jKgCa67Nw9TmcMPV8zmH7R9qdvgxAbAtwBl1F9OS\nfcGaC7epf1AjJLtaX7krWmzgASHl28Ynh9lmGMdv+5QYMZvKG0LOg/n3m8uJ6sKy\nIzzvaJswwn0j5P5+czyoV5CvvdCfKnNb+3jUEN8I0PPwjBGKr4B1ojwhogTM248V\nHR69D6TxFVMfGpyJhCPkbGEGbpEpcffpgKuC/mEtMqyDQXJNaV5HO6HgAJ9F1P6v\n5ehHHTMRvzCCFiwndHdlMXUjqSNjww6me6dr6LiAPbejdzhL2vWx1YqebOcwQx3G\n-----END RSA PRIVATE KEY-----",
            "passphrase": {
                "ciphertext": "ZjVmNQ==",
                "protected": "eyJhbGciOiJkaXIiLCJlbmMiOiJub25lIn0"
            }
            }
        },
        "nginxstatefulsvc": {
            "class": "Application",
            "template": "http",
            "serviceMain": {
               "class": "Service_HTTP",
               "virtualAddresses": [
                  "172.16.40.80"
               ],
               "pool": "nginx_stateful_pool"
            },
            "nginx_stateful_pool": {
               "class": "Pool",
               "monitors": [
                  "http"
               ],
               "members": [{
                  "servicePort": 80,
                  "serverAddresses": []
               }]
            }
         }
        }
    }
    }

附:如何清除采用as3-configmap所产生的配置

在非as3 configmap配置模式下，只要在k8s里删除configmap，对应F5 bigip配置就会被自动删除掉。

但是如果采用了as3 configmap模式，删除configmap是无法清除对应的F5 bigip配置的。需要在删除对应的as3-configmap后，部署一个空as3的as3-configmap，这样才可以清空F5上的配置。然后记得别忘记删除刚才部署的空configmap。所以步骤是

kubectl delete -f your-as3-configmap

kubectl create -f blank-as3-configmap

kubectl delete -f blank-as3-configmap

空as3-configmap样例如下，注意修改tenant为实际对应的tenant

kind: ConfigMap
apiVersion: v1
metadata:
  name: f5-as3-declaration
  namespace: default
  labels:
    f5type: virtual-server
    as3: "true"
data:
  template: |
    {
      "class": "AS3",
      "declaration": {
        "class": "ADC",
        "schemaVersion": "3.10.0",
        "id": "urn:uuid:33045210-3ab8-4636-9b2a-c98d22ab915d",
        "label": "Sample 1",
        "remark": "Remove AS3 declaration",
        "AS3": {
          "class": "Tenant"
        }
      }
    }

Array

本作品采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可