-# sdewan-controller
+# Sdewan operator
+
+The sdewan operator is developed under kubebuilder framework
+
+## Deployment Guide
+
+The API admission webhook depends on cert-manager so we need to install cert-manager first.
+
+We have the image built and published at `integratedcloudnative/sdewan-controller:dev`. The openwrt
+docker image we used for test is at `integratedcloudnative/openwrt:dev`. To use some other images,
+we need to make changes in deployment yaml file.
+
+After clone the repo, please change into directory `platform/crd-ctrlr`.
+We are going to run command from this directory in the deployment guide.
+
+The installation steps for Sdewan operator:
+1. kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.11.0/cert-manager.yaml --validate=false
+2. kubectl apply -f examples/sdewan-controller.yaml
+
+Sample deployment of CNF:
+1. Setup ovn networks
+ ```
+ kubectl apply -f examples/attach-network-ovn.yaml
+ kubectl apply -f examples/ovn-net1.yaml
+ kubectl apply -f examples/ovn-net2.yml
+ ```
+2. Launch CNF deployment. **NOTE:** CNF deployment is supposed to bind to a Node.
+ For the sample cnf yaml, we bind it to master node. You can bind to other node by modifying the `nodeSelector`.
+ ```
+ kubectl apply -f examples/cnf-deployment.yaml # for kubernetes older than 1.16, please use cnf-deployment-older-than-1.16.yaml
+ ```
+3. Create rule for the CNF
+ ```
+ kubectl apply -f src/config/samples/batch_v1alpha1_mwan3policy.yaml
+ ```
+4. Verify that the policy is applied with the CNF by checking the last lines of mwan3 file. It should contains line `config policy 'balance1'`
+ ```
+ kubectl get pod |grep cnf1
+ kubectl exec cnf1-6d759f9b4b-fbqrr -- cat /etc/config/mwan3
+ # or you can merge these two commands: kubectl exec `kubectl get pod |grep cnf1 |head -n1 | awk '{print $1}'` -- cat /etc/config/mwan3
+ ```
+
+
+## Developer Guide
+
+Project initialization(mostly, developers should not execute this step)
+```
+go mod init sdewan.akraino.org/sdewan
+kubebuilder init --domain sdewan.akraino.org
+```
+
+To create new CRD and controller
+```
+kubebuilder create api --group batch --version v1alpha1 --kind Mwan3Policy
+```
+
+To run local controller without webhook(For test/debug purpose)
+```
+make install
+make run ENABLE_WEBHOOKS=false
+```
+
+To build controller docker image
+```
+make docker-build IMG="integratedcloudnative/sdewan-controller:dev"
+```
+
+To generate yaml file for controller deployment
+```
+make gen-yaml IMG="integratedcloudnative/sdewan-controller:dev"
+```
+
+### Controller Implementation
+
+
+
+- One CRD one controller
+- Controller watches itself CR and the Deployment(ready status only)
+- Reconcile calls WrtProvider to add/update/delete rules for CNF
+- CnfProvider interfaces defines the function CNF function calls. WrtProvider is one implementation of CnfProvider
+- For the users, CNF rules are CRs. But for openwrt, the rules are openwrt rule entities. We can pass the CRs to OpenWRT API. Instead, we need to convert the CRs to OpenWRT entities.
+- Finalizer should be added to CR only when AddUpdate call succeed. Likewise, finalizer should be removed from CR only when Delete call succeed.
+- **As we have many CRDs, so there could be many duplicate code. For example, convertCrd, AddUpdateXX, and even reconcile logic. So we need to extract the similar logic into functions to reduce the duplicatioin.**
+
+### What we have implemented
+
+- CNF image built from HuiFeng's script. I have uploaded the image at `integratedcloudnative/openwrt:dev`
+- The CNF sample deployment yaml file under sample directory (together with configmap and ovn network yaml files)
+- A runable framework with Mwan3Policy CRD and controller implemented. It means we can run the controller and add/update/delete mwan3policy rules.
+
+### What we don't have yet
+
+- Add a watch for deployment, so that the controller can get the CNF ready status change. [predicate feature](https://godoc.org/sigs.k8s.io/controller-runtime/pkg/predicate#example-Funcs) should be used to filter no-status event.
+- Implemente the remain CRDs/controllers. As all the controller logics are almost the same, some workload will be the extracting of the similar logic and make them functions.
+- Add raw webhook to implemente the label based permission system
+- Add validation webhook to validate CR
+
+## References
+
+- https://book.kubebuilder.io/
+- https://openwrt.org/
--- /dev/null
+<mxfile host="app.diagrams.net" modified="2020-04-21T08:22:30.961Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36" etag="vbx_huyF-hUeLDkj3lo7" version="12.9.14" type="device"><diagram id="f-1GySHZMf33IIK9-Ksn" name="Page-1">7V1bc+I2G/41zPS7yI5Pss1lQpJ2p9kt36ad7V4KW4AbY1FbBNhfX8m28EECTBbZ3kCSydhCNvh53pNevRIDc7TY/BrD5fwT9lE4MDR/MzDvB4bhuAb9zxq2WYMOTCdrmcWBn7cVDc/Bd5Q3annrKvBRUulIMA5JsKw2ejiKkEcqbTCO8brabYrD6rsu4QwJDc8eDMXWr4FP5lmrC7Si/TcUzOb8nXUtf2UBeee8IZlDH69LTebDwBzFGJPsaLEZoZCBx3HJrnvc8+rug8UoIk0u2Gibf7/9gcfa95enf+azxNBWTzf5XV5huMofeGDYIb3f3YQezNjB6PMj7XGPliHeLth75R3oO+365E9Ithy2GK8iH7F31ujL63lA0PMSeuzVNRUU2jYni5Ce6fTQw4vAy4+nQRiOcIjj9D4m0n2AHNqekBi/oNIrQ9sxob1771cUE7TZC42+A5xKKsILROIt7ZJfMOQc5VJ642an64LxHa/zEttcCmAuZLPdnQse6EFOxSm08HcrQYp8Kpj5KY7JHM9wBMOHovXOW8WvKeYMyCoDxQVPGC/zLv8gQra5ysEVwVVW0CYgf5eOv7FbfTBAfnq/yW+dnmz5SUQf/+/ySXYZ4KfFZelZcZ1/y7SVnnohTBImD6zxMQj558kAYSgcZpmChlexhw7Cm1sSGM8QOdDR1uRyE6MQkuC1+kkUSIFEOxUpGlUz17dkiuYaE9M+k6I5VlXRdH5+TNN0ZaoGDAHkW9//a+lDgj6tYWSOcRh4251ZjLnNu0chqnU5FzcwDGYRPQ7RlKixcDuQy8BbEuAtZSaugeupgKt9QdTJewFFt4kHoliRKsJVyY5whGpqkDdx9D2KLaLtdwz5gIYEt/kLi8D3U4Mro7RKepnVc+iPU6NRF2mUsWgoY9F+g586m2vK/Qp3Tg09U+GMvvH7HfBMFad6Th9kNfVBXbogHq33IA6pkq03JLsShhyhugt2TdBphGEJRngNiTc/4Mr0Hwoz9AnUkSELMzTNfrh9ZFfgiJTap+nPecwn0Krmc9ix9TSHvdGuapSvJsg/o36ZbkP92iMQ7aiX1Z9RXGnkdoL51AfdmM/G9HZrPvnHvBDz6dYGEU631tMyJSOIGvAzivxy76PnSUM44d21UyEB9cwR/wzlcZUhQcUBilCxj4Py5nFpIxk6wNVeGGsoGg2zAsqSAmJY9BOAqNeH+B2DKCZWzgdiPXsFkTv1ZHbR9lw0aWz/ToK9M2AdCbCHMifvN1vyFjUxa2pid+vFxBBCIPPx65XDg/7CAt1yODzO4cfxM/KuNB5URdAxjVyMugworWHNPJkSN24AERV1MyRmA1jO5cingP1KHXn6Iw5wsp8fk9I9g8j9HkLOgGkpYkCXOYkexKN7cOPaDSqSLBFk2cBI3Zy6zEorEuMJ9FzflImxYVoW8FuE3WkGu65sFNDiJPYU+kNfaj0gsnTTaBF2u2vYZdOrqkZfmofkWSlgA91BLcIOuoZd5aBXGYh2vyy10SCx9x4std0zS60y69UjS233zFL3NGN7EoiyeiIZisrKibjN6kMhStPahGohyrHptVMLUQ4OcY/OwHEbfHQGzui0PsVoMajvMsypq5skud+q0TLF8PI+iIlYA/muslmHOTKqQZSYNTAlDKmrLREj0f+vED28YIZqubXOKRLj3HGMPZQkQTS7ZJ6EYuWuiWo3MnagdCpzYtlAs9r0MpJ5+Ha9jCw0rk2srHH88m9m2C5tcuUwmZbzQZxPaVVtLHGiOndB2qFlEz8pR4fndWy3atMko05pnZAycsBbBky9rexutfY0V6vjtaedrh4DkhAd3eSmkt5Si+ACJUwFPtODC9FCvebhhqKJ1Fu1kUAWWtRX4EZTGhe+Bj5F09A+MlSnDLN37PBOW83GV5l3xqLt9tSYqlkTVVqtOwmx9zIor9VlHXyYzFUsmgJOQ9ObrerszPSKgc/XmBQ6XBOVZB0sQpjpIY4Ilxq9iRISJh533jwI/Se4xSsGSkKg98LP7uY4Dr7T28LCAMCY5HJk2JUez+zKnMMYJbTPmLOk15o+wU2l4xNMyE7bwxAuk2Cye4wF5SuI7jAheNGOSdAlsym6LRu8KEtJA7Fq4zN3uQOWNjV1BsSzj2gwPF7FS5yU2393k1EYpBsuyG36qYY8X00slyD2qYJo9pT2ubeKli85VKlZotdOw1Tv59T4oyg1XATygifWZ4mDiKRQgjv6R8EdMesC7lnQAe704pz+se4xGeGIPgsMUqYRlaM1SkjxCSoR3FsGSpk67q3vMST1PTJJsVSV94AGSewwSLnMOOc7kdTTB80I3/ntHN8/Uw9zowtSYIpSYEoYD+EEhWOcBCTA7P5x1rcmCcfIVkZvU0PgqooNZAUw9HE12TYDv/yvH9puvVttdyR7u8jEgevg+eVBVpnD5EHYUOIqDKqFQboNhlQa1O30I4rDzSCtJGbPO4r9qxQolwLZerNWpcBusIJg9CW53BE/ANaHaurmRrJ1UKtjfrdBUbbq1QK6VpvSdGW16m0uP3VVFmS83QJkZDWdcOx8/elQFjP2HsW+LUAdtlgDbfuuY1uCsWULVG6BZeqDH1yZfxIf3SFuHPdjl7w09YgC9Wxp6lBWh31dm3qaL+l8bWqDCabLXJx6mjJ2vTiVz1Qe4vGPJYrWMaG9HiISkABd8IhBCI1leUAN8FFFKxw6YkDCKPsaizn+65zQGURAKCSUTfcDiRbr3PWeXwQarG265vqPaNBJUzkyflXl+h3R2d76foNit2sq7zwCwCt1jwkA17rzS4DopiV7SV+FQKUQgIZWQJ2fF+s/ju8WfhWCswqB07kQiPnhXxG5SkBrEjDsWgKGb9kcvQ/b+7b5XRx8nu1oSd8+thuX9KWX0ieD21KHXKyLO49ZQ2kModey2oYzLAvG8Qv4TOL+C+pbPp16gQHMmqxmT1lI7g6uHxBm0Zy97K1QO9d+t1PXQ558MZQLLNDYfBz5VgVQI0CS62l1Onj4k+8LXt02Ot95ujff/uM03Zu/073DdU2cS8m+iut2/JH+H0F2j4vI5NXX7UomuYfnUU96WnwhW2Y5i6+1Mx/+Aw==</diagram></mxfile>
\ No newline at end of file
# SDEWAN crd-ctrlr examples
+
+## To deploy an example CNF
+
+```shell
+kubectl apply -f attach-network-ovn.yaml
+kubectl apply -f ovn-net1.yaml ovn-net2.yml ovn-provnet.yaml sdewan-cm.yaml
+kubectl apply -f sdewan-deployment.yaml
+```
--- /dev/null
+---
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+ name: ovn-networkobj
+spec:
+ config: '{
+ "cniVersion": "0.3.1",
+ "name": "ovn4nfv-k8s-plugin",
+ "type": "ovn4nfvk8s-cni"
+ }'
--- /dev/null
+---
+apiVersion: v1
+data:
+ entrypoint.sh: |-
+ #!/bin/bash
+ # Always exit on errors.
+ set -ex
+ echo "" > /etc/config/network
+ cat > /etc/config/mwan3 <<EOF
+ config globals 'globals'
+ option mmx_mask '0x3F00'
+ option local_source 'lan'
+ EOF
+ eval "networks=$(grep nfn-network /tmp/podinfo/annotations | awk -F '=' '{print $2}')"
+ for net in $(echo -e $networks | jq -c ".interface[]")
+ do
+ interface=$(echo $net | jq -r .interface)
+ ipaddr=$(ifconfig $interface | awk '/inet/{print $2}' | cut -f2 -d ":" | awk 'NR==1 {print $1}')
+ vif="$interface"
+ cat >> /etc/config/network <<EOF
+ config interface '$vif'
+ option ifname '$interface'
+ option proto 'static'
+ option ipaddr '$ipaddr'
+ option netmask '$netmask'
+ EOF
+ cat >> /etc/config/mwan3 <<EOF
+ config interface '$vif'
+ option enabled '1'
+ option family 'ipv4'
+ option reliability '2'
+ option count '1'
+ option timeout '2'
+ option failure_latency '1000'
+ option recovery_latency '500'
+ option failure_loss '20'
+ option recovery_loss '5'
+ option interval '5'
+ option down '3'
+ option up '8'
+ EOF
+ done
+ /sbin/procd &
+ /sbin/ubusd &
+ iptables -S
+ sleep 1
+ /etc/init.d/rpcd start
+ /etc/init.d/dnsmasq start
+ /etc/init.d/network start
+ /etc/init.d/odhcpd start
+ /etc/init.d/uhttpd start
+ /etc/init.d/log start
+ /etc/init.d/dropbear start
+ /etc/init.d/mwan3 restart
+ echo "Entering sleep... (success)"
+ # Sleep forever.
+ while true; do sleep 100; done
+kind: ConfigMap
+metadata:
+ name: sdewan-sh
+ namespace: default
+...
+
+---
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+ name: cnf1
+ namespace: default
+spec:
+ progressDeadlineSeconds: 600
+ replicas: 2
+ selector:
+ matchLabels:
+ sdewanPurpose: cnf1
+ strategy:
+ rollingUpdate:
+ maxSurge: 25%
+ maxUnavailable: 25%
+ type: RollingUpdate
+ template:
+ metadata:
+ annotations:
+ k8s.plugin.opnfv.org/nfn-network: |-
+ { "type": "ovn4nfv", "interface": [
+ {
+ "defaultGateway": "false",
+ "interface": "net1",
+ "name": "ovn-net1"
+ },
+ {
+ "defaultGateway": "false",
+ "interface": "net2",
+ "name": "ovn-net2"
+ }
+ ]}
+ k8s.v1.cni.cncf.io/networks: '[{ "name": "ovn-networkobj"}]'
+ labels:
+ sdewanPurpose: cnf1
+ spec:
+ containers:
+ - command:
+ #- sleep
+ #- "3600"
+ - /bin/sh
+ - /tmp/sdewan/entrypoint.sh
+ image: integratedcloudnative/openwrt:dev
+ imagePullPolicy: IfNotPresent
+ name: sdewan
+ readinessProbe:
+ failureThreshold: 5
+ httpGet:
+ path: /
+ port: 80
+ scheme: HTTP
+ initialDelaySeconds: 5
+ periodSeconds: 5
+ successThreshold: 1
+ timeoutSeconds: 1
+ securityContext:
+ privileged: true
+ procMount: Default
+ volumeMounts:
+ - mountPath: /tmp/sdewan
+ name: sdewan-sh
+ readOnly: true
+ - mountPath: /tmp/podinfo
+ name: podinfo
+ readOnly: true
+ nodeSelector:
+ node-role.kubernetes.io/master: ""
+ restartPolicy: Always
+ volumes:
+ - configMap:
+ defaultMode: 420
+ name: sdewan-sh
+ name: sdewan-sh
+ - name: podinfo
+ downwardAPI:
+ items:
+ - path: "annotations"
+ fieldRef:
+ fieldPath: metadata.annotations
--- /dev/null
+---
+apiVersion: v1
+data:
+ entrypoint.sh: |-
+ #!/bin/bash
+ # Always exit on errors.
+ set -ex
+ echo "" > /etc/config/network
+ cat > /etc/config/mwan3 <<EOF
+ config globals 'globals'
+ option mmx_mask '0x3F00'
+ option local_source 'lan'
+ EOF
+ eval "networks=$(grep nfn-network /tmp/podinfo/annotations | awk -F '=' '{print $2}')"
+ for net in $(echo -e $networks | jq -c ".interface[]")
+ do
+ interface=$(echo $net | jq -r .interface)
+ ipaddr=$(ifconfig $interface | awk '/inet/{print $2}' | cut -f2 -d ":" | awk 'NR==1 {print $1}')
+ vif="$interface"
+ cat >> /etc/config/network <<EOF
+ config interface '$vif'
+ option ifname '$interface'
+ option proto 'static'
+ option ipaddr '$ipaddr'
+ option netmask '$netmask'
+ EOF
+ cat >> /etc/config/mwan3 <<EOF
+ config interface '$vif'
+ option enabled '1'
+ option family 'ipv4'
+ option reliability '2'
+ option count '1'
+ option timeout '2'
+ option failure_latency '1000'
+ option recovery_latency '500'
+ option failure_loss '20'
+ option recovery_loss '5'
+ option interval '5'
+ option down '3'
+ option up '8'
+ EOF
+ done
+ /sbin/procd &
+ /sbin/ubusd &
+ iptables -S
+ sleep 1
+ /etc/init.d/rpcd start
+ /etc/init.d/dnsmasq start
+ /etc/init.d/network start
+ /etc/init.d/odhcpd start
+ /etc/init.d/uhttpd start
+ /etc/init.d/log start
+ /etc/init.d/dropbear start
+ /etc/init.d/mwan3 restart
+ echo "Entering sleep... (success)"
+ # Sleep forever.
+ while true; do sleep 100; done
+kind: ConfigMap
+metadata:
+ name: sdewan-sh
+ namespace: default
+...
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: cnf1
+ namespace: default
+ labels:
+ sdewanPurpose: cnf1
+spec:
+ progressDeadlineSeconds: 600
+ replicas: 2
+ selector:
+ matchLabels:
+ sdewanPurpose: cnf1
+ strategy:
+ rollingUpdate:
+ maxSurge: 25%
+ maxUnavailable: 25%
+ type: RollingUpdate
+ template:
+ metadata:
+ annotations:
+ k8s.plugin.opnfv.org/nfn-network: |-
+ { "type": "ovn4nfv", "interface": [
+ {
+ "defaultGateway": "false",
+ "interface": "net1",
+ "name": "ovn-net1"
+ },
+ {
+ "defaultGateway": "false",
+ "interface": "net2",
+ "name": "ovn-net2"
+ }
+ ]}
+ k8s.v1.cni.cncf.io/networks: '[{ "name": "ovn-networkobj"}]'
+ labels:
+ sdewanPurpose: cnf1
+ spec:
+ containers:
+ - command:
+ #- sleep
+ #- "3600"
+ - /bin/sh
+ - /tmp/sdewan/entrypoint.sh
+ image: integratedcloudnative/openwrt:dev
+ imagePullPolicy: IfNotPresent
+ name: sdewan
+ readinessProbe:
+ failureThreshold: 5
+ httpGet:
+ path: /
+ port: 80
+ scheme: HTTP
+ initialDelaySeconds: 5
+ periodSeconds: 5
+ successThreshold: 1
+ timeoutSeconds: 1
+ securityContext:
+ privileged: true
+ procMount: Default
+ volumeMounts:
+ - mountPath: /tmp/sdewan
+ name: sdewan-sh
+ readOnly: true
+ - mountPath: /tmp/podinfo
+ name: podinfo
+ readOnly: true
+ nodeSelector:
+ node-role.kubernetes.io/master: ""
+ restartPolicy: Always
+ volumes:
+ - configMap:
+ defaultMode: 420
+ name: sdewan-sh
+ name: sdewan-sh
+ - name: podinfo
+ downwardAPI:
+ items:
+ - path: "annotations"
+ fieldRef:
+ fieldPath: metadata.annotations
--- /dev/null
+apiVersion: k8s.plugin.opnfv.org/v1alpha1
+kind: Network
+metadata:
+ name: ovn-net1
+spec:
+ cniType : ovn4nfv
+ ipv4Subnets:
+ - subnet: 172.16.44.0/24
+ name: subnet1
+ gateway: 172.16.44.1/24
--- /dev/null
+apiVersion: k8s.plugin.opnfv.org/v1alpha1
+kind: Network
+metadata:
+ name: ovn-net2
+spec:
+ cniType : ovn4nfv
+ ipv4Subnets:
+ - subnet: 172.16.45.0/24
+ name: subnet2
+ gateway: 172.16.45.1/24
--- /dev/null
+apiVersion: k8s.plugin.opnfv.org/v1alpha1
+kind: ProviderNetwork
+metadata:
+ name: pnetwork
+spec:
+ cniType: ovn4nfv
+ ipv4Subnets:
+ - subnet: 172.16.34.0/24
+ name: subnet1
+ gateway: 172.16.34.1/24
+ excludeIps: 172.16.34.2 172.16.34.5..172.16.34.10
+ providerNetType: VLAN
+ vlan:
+ vlanId: "100"
+ providerInterfaceName: eth1
+ logicalInterfaceName: eth1.100
+ vlanNodeSelector: specific
+ nodeLabelList:
+ - kubernetes.io/hostname=ubuntu18
--- /dev/null
+apiVersion: v1
+kind: Namespace
+metadata:
+ labels:
+ control-plane: controller-manager
+ name: sdewan-system
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+ annotations:
+ controller-gen.kubebuilder.io/version: v0.2.4
+ creationTimestamp: null
+ name: mwan3policies.batch.sdewan.akraino.org
+spec:
+ group: batch.sdewan.akraino.org
+ names:
+ kind: Mwan3Policy
+ listKind: Mwan3PolicyList
+ plural: mwan3policies
+ singular: mwan3policy
+ scope: Namespaced
+ subresources:
+ status: {}
+ validation:
+ openAPIV3Schema:
+ description: Mwan3Policy is the Schema for the mwan3policies API
+ properties:
+ apiVersion:
+ description: 'APIVersion defines the versioned schema of this representation
+ of an object. Servers should convert recognized schemas to the latest
+ internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+ type: string
+ kind:
+ description: 'Kind is a string value representing the REST resource this
+ object represents. Servers may infer this from the endpoint the client
+ submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+ type: string
+ metadata:
+ type: object
+ spec:
+ properties:
+ members:
+ items:
+ description: Mwan3PolicySpec defines the desired state of Mwan3Policy
+ properties:
+ metric:
+ type: integer
+ network:
+ description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of
+ cluster Important: Run "make" to regenerate code after modifying
+ this file'
+ type: string
+ weight:
+ type: integer
+ required:
+ - metric
+ - network
+ - weight
+ type: object
+ type: array
+ required:
+ - members
+ type: object
+ status:
+ description: status subsource used for Sdewan rule CRDs
+ properties:
+ appliedTime:
+ format: date-time
+ type: string
+ appliedVersion:
+ description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
+ of cluster Important: Run "make" to regenerate code after modifying
+ this file'
+ type: string
+ inSync:
+ type: boolean
+ required:
+ - appliedTime
+ - appliedVersion
+ - inSync
+ type: object
+ type: object
+ version: v1alpha1
+ versions:
+ - name: v1alpha1
+ served: true
+ storage: true
+status:
+ acceptedNames:
+ kind: ""
+ plural: ""
+ conditions: []
+ storedVersions: []
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+ name: sdewan-leader-election-role
+ namespace: sdewan-system
+rules:
+- apiGroups:
+ - ""
+ resources:
+ - configmaps
+ verbs:
+ - get
+ - list
+ - watch
+ - create
+ - update
+ - patch
+ - delete
+- apiGroups:
+ - ""
+ resources:
+ - configmaps/status
+ verbs:
+ - get
+ - update
+ - patch
+- apiGroups:
+ - ""
+ resources:
+ - events
+ verbs:
+ - create
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ creationTimestamp: null
+ name: sdewan-manager-role
+rules:
+- apiGroups:
+ - batch.sdewan.akraino.org
+ resources:
+ - mwan3policies
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - batch.sdewan.akraino.org
+ resources:
+ - mwan3policies/status
+ verbs:
+ - get
+ - patch
+ - update
+- apiGroups:
+ - extensions
+ resources:
+ - deployments
+ verbs:
+ - get
+ - list
+ - watch
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ name: sdewan-proxy-role
+rules:
+- apiGroups:
+ - authentication.k8s.io
+ resources:
+ - tokenreviews
+ verbs:
+ - create
+- apiGroups:
+ - authorization.k8s.io
+ resources:
+ - subjectaccessreviews
+ verbs:
+ - create
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+ name: sdewan-leader-election-rolebinding
+ namespace: sdewan-system
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: Role
+ name: sdewan-leader-election-role
+subjects:
+- kind: ServiceAccount
+ name: default
+ namespace: sdewan-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+ name: sdewan-manager-rolebinding
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: sdewan-manager-role
+subjects:
+- kind: ServiceAccount
+ name: default
+ namespace: sdewan-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+ name: sdewan-proxy-rolebinding
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: sdewan-proxy-role
+subjects:
+- kind: ServiceAccount
+ name: default
+ namespace: sdewan-system
+---
+apiVersion: v1
+kind: Service
+metadata:
+ labels:
+ control-plane: controller-manager
+ name: sdewan-controller-manager-metrics-service
+ namespace: sdewan-system
+spec:
+ ports:
+ - name: https
+ port: 8443
+ targetPort: https
+ selector:
+ control-plane: controller-manager
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ labels:
+ control-plane: controller-manager
+ name: sdewan-controller-manager
+ namespace: sdewan-system
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ control-plane: controller-manager
+ template:
+ metadata:
+ labels:
+ control-plane: controller-manager
+ spec:
+ containers:
+ - args:
+ - --secure-listen-address=0.0.0.0:8443
+ - --upstream=http://127.0.0.1:8080/
+ - --logtostderr=true
+ - --v=10
+ image: gcr.io/kubebuilder/kube-rbac-proxy:v0.4.1
+ name: kube-rbac-proxy
+ ports:
+ - containerPort: 8443
+ name: https
+ - args:
+ - --metrics-addr=127.0.0.1:8080
+ - --enable-leader-election
+ command:
+ - /manager
+ image: integratedcloudnative/sdewan-controller:dev
+ name: manager
+ resources:
+ limits:
+ cpu: 100m
+ memory: 30Mi
+ requests:
+ cpu: 100m
+ memory: 20Mi
+ terminationGracePeriodSeconds: 10
--- /dev/null
+# Build the manager binary
+FROM golang:1.13 as builder
+
+WORKDIR /workspace
+# Copy the Go Modules manifests
+COPY go.mod go.mod
+COPY go.sum go.sum
+# cache deps before building and copying source so that we don't need to re-download as much
+# and so that source changes don't invalidate our downloaded layer
+RUN go mod download
+
+# Copy the go source
+COPY main.go main.go
+COPY api/ api/
+COPY controllers/ controllers/
+COPY cnfprovider/ cnfprovider/
+COPY openwrt/ openwrt/
+
+# Build
+RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go
+
+# Use distroless as minimal base image to package the manager binary
+# Refer to https://github.com/GoogleContainerTools/distroless for more details
+FROM gcr.io/distroless/static:nonroot
+WORKDIR /
+COPY --from=builder /workspace/manager .
+USER nonroot:nonroot
+
+ENTRYPOINT ["/manager"]
--- /dev/null
+
+# Image URL to use all building/pushing image targets
+IMG ?= integratedcloudnative/sdewan-controller:dev
+# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
+CRD_OPTIONS ?= "crd:trivialVersions=true"
+
+# Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set)
+ifeq (,$(shell go env GOBIN))
+GOBIN=$(shell go env GOPATH)/bin
+else
+GOBIN=$(shell go env GOBIN)
+endif
+
+all: manager
+
+# Run tests
+test: generate fmt vet manifests
+ go test ./... -coverprofile cover.out
+
+# Build manager binary
+manager: generate fmt vet
+ go build -o bin/manager main.go
+
+# Run against the configured Kubernetes cluster in ~/.kube/config
+run: generate fmt vet manifests
+ go run ./main.go
+
+# Install CRDs into a cluster
+install: manifests
+ kustomize build config/crd | kubectl apply -f -
+
+# Uninstall CRDs from a cluster
+uninstall: manifests
+ kustomize build config/crd | kubectl delete -f -
+
+# Deploy controller in the configured Kubernetes cluster in ~/.kube/config
+deploy: manifests
+ cd config/manager && kustomize edit set image controller=${IMG}
+ kustomize build config/default | kubectl apply -f -
+
+# Deploy controller in the configured Kubernetes cluster in ~/.kube/config
+gen-yaml: manifests
+ cd config/manager && kustomize edit set image controller=${IMG}
+ kustomize build config/default > ../examples/sdewan-controller.yaml
+
+# Generate manifests e.g. CRD, RBAC etc.
+manifests: controller-gen
+ $(CONTROLLER_GEN) $(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
+
+# Run go fmt against code
+fmt:
+ go fmt ./...
+
+# Run go vet against code
+vet:
+ go vet ./...
+
+# Generate code
+generate: controller-gen
+ $(CONTROLLER_GEN) object:headerFile=./hack/boilerplate.go.txt paths="./..."
+
+# Build the docker image
+docker-build: test
+ docker build . -t ${IMG}
+
+# Push the docker image
+docker-push:
+ docker push ${IMG}
+
+# find or download controller-gen
+# download controller-gen if necessary
+controller-gen:
+ifeq (, $(shell which controller-gen))
+ @{ \
+ set -e ;\
+ CONTROLLER_GEN_TMP_DIR=$$(mktemp -d) ;\
+ cd $$CONTROLLER_GEN_TMP_DIR ;\
+ go mod init tmp ;\
+ go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.2.4 ;\
+ rm -rf $$CONTROLLER_GEN_TMP_DIR ;\
+ }
+CONTROLLER_GEN=$(GOBIN)/controller-gen
+else
+CONTROLLER_GEN=$(shell which controller-gen)
+endif
--- /dev/null
+domain: sdewan.akraino.org
+repo: sdewan.akraino.org/sdewan
+resources:
+- group: batch
+ kind: Mwan3Policy
+ version: v1alpha1
+version: "2"
+++ /dev/null
-# Sdewan operator
-
-The sdewan operator is developed under kubebuilder framework
-
-We define two CRDs in this patch: Sdewan and Mwan3Conf.
-
-Sdewan defines the CNF base info, which node we should deploy the CNF on,
-which network should the CNF use with multus CNI, etc.
-
-The Mwan3Conf defines the mwan3 rules. In the next step, we are going to
-develop the firewall and the ipsec functions. Mwan3Conf is validated by k8s
-api admission webhook.
-
-For each created Sdewan instance, the controller creates a pod, a configmap
-and a service for the instance. The pod runs openwrt which provides network
-services, i.e. sdwan, firewall, ipsec etc.
-
-The configmap stores the network interface information and the entrypoint.sh.
-The network interface information has the following format:
-```
-[
- {
- "name": "ovn-priv-net",
- "isProvider": false,
- "interface": "net0",
- "defaultGateway": false
- }
-]
-```
-
-The service created by the controller is used for openwrt api access.
-We call this svc to apply rules, get openwrt info, restart openwrt service.
-
-After the openwrt pod ready, the Sdewan controller apply the configured mwan3 rules.
-mwan3 rule details are configured in Mwan3Conf CR, which is referenced by Sdewan.Spec.Mwan3Conf
-Every time the Mwan3Conf instance changes, the controller re-apply the new rules by calling opwnrt
-api. We can also change the rule refernce at the runtime.
-
-## Deployment
-
-The API admission webhook depends on cert-manager so we need to install cert-manager first.
-
-To install the CRD and the controller, we can follow this guide.
-https://book.kubebuilder.io/cronjob-tutorial/running-webhook.html
-
-We have the image built and published at `integratedcloudnative/sdewan-controller:dev`. The openwrt
-docker image we used for test is at `integratedcloudnative/openwrt:dev`. To use some other images,
-we need to make configuration in `config/default/manager_image_patch.yaml`
-
-The simple installation steps:
-1. kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.11.0/cert-manager.yaml
-2. kubectl apply -f sdewan-deploy.yaml
-
-## Create Sdewan CNF docker image
-1. update build/set_proxy file with required proxy for docker build
-2. execute below commands to generate Sdewan CNF docker image which tagged with 'openwrt-1806-mwan3'
-```
-cd build
-sudo bash build_image.sh
-```
-
-## References
-
-- https://book.kubebuilder.io/
-- https://openwrt.org/
--- /dev/null
+/*
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// status subsource used for Sdewan rule CRDs
+type SdewanStatus struct {
+ // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
+ // Important: Run "make" to regenerate code after modifying this file
+ AppliedVersion string `json:"appliedVersion"`
+ AppliedTime *metav1.Time `json:"appliedTime"`
+ InSync bool `json:"inSync"`
+}
--- /dev/null
+/*
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Package v1alpha1 contains API Schema definitions for the batch v1alpha1 API group
+// +kubebuilder:object:generate=true
+// +groupName=batch.sdewan.akraino.org
+package v1alpha1
+
+import (
+ "k8s.io/apimachinery/pkg/runtime/schema"
+ "sigs.k8s.io/controller-runtime/pkg/scheme"
+)
+
+var (
+ // GroupVersion is group version used to register these objects
+ GroupVersion = schema.GroupVersion{Group: "batch.sdewan.akraino.org", Version: "v1alpha1"}
+
+ // SchemeBuilder is used to add go types to the GroupVersionKind scheme
+ SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
+
+ // AddToScheme adds the types in this group-version to the given scheme.
+ AddToScheme = SchemeBuilder.AddToScheme
+)
--- /dev/null
+/*
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN!
+// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized.
+
+// Mwan3PolicySpec defines the desired state of Mwan3Policy
+type Mwan3PolicyMember struct {
+ // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
+ // Important: Run "make" to regenerate code after modifying this file
+ Network string `json:"network"`
+ Metric int `json:"metric"`
+ Weight int `json:"weight"`
+}
+
+type Mwan3PolicySpec struct {
+ Members []Mwan3PolicyMember `json:"members"`
+}
+
+// +kubebuilder:object:root=true
+// +kubebuilder:subresource:status
+
+// Mwan3Policy is the Schema for the mwan3policies API
+type Mwan3Policy struct {
+ metav1.TypeMeta `json:",inline"`
+ metav1.ObjectMeta `json:"metadata,omitempty"`
+
+ Spec Mwan3PolicySpec `json:"spec,omitempty"`
+ Status SdewanStatus `json:"status,omitempty"`
+}
+
+// +kubebuilder:object:root=true
+
+// Mwan3PolicyList contains a list of Mwan3Policy
+type Mwan3PolicyList struct {
+ metav1.TypeMeta `json:",inline"`
+ metav1.ListMeta `json:"metadata,omitempty"`
+ Items []Mwan3Policy `json:"items"`
+}
+
+func init() {
+ SchemeBuilder.Register(&Mwan3Policy{}, &Mwan3PolicyList{})
+}
--- /dev/null
+// +build !ignore_autogenerated
+
+/*
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by controller-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+ runtime "k8s.io/apimachinery/pkg/runtime"
+)
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Mwan3Policy) DeepCopyInto(out *Mwan3Policy) {
+ *out = *in
+ out.TypeMeta = in.TypeMeta
+ in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+ in.Spec.DeepCopyInto(&out.Spec)
+ in.Status.DeepCopyInto(&out.Status)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Mwan3Policy.
+func (in *Mwan3Policy) DeepCopy() *Mwan3Policy {
+ if in == nil {
+ return nil
+ }
+ out := new(Mwan3Policy)
+ in.DeepCopyInto(out)
+ return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *Mwan3Policy) DeepCopyObject() runtime.Object {
+ if c := in.DeepCopy(); c != nil {
+ return c
+ }
+ return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Mwan3PolicyList) DeepCopyInto(out *Mwan3PolicyList) {
+ *out = *in
+ out.TypeMeta = in.TypeMeta
+ in.ListMeta.DeepCopyInto(&out.ListMeta)
+ if in.Items != nil {
+ in, out := &in.Items, &out.Items
+ *out = make([]Mwan3Policy, len(*in))
+ for i := range *in {
+ (*in)[i].DeepCopyInto(&(*out)[i])
+ }
+ }
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Mwan3PolicyList.
+func (in *Mwan3PolicyList) DeepCopy() *Mwan3PolicyList {
+ if in == nil {
+ return nil
+ }
+ out := new(Mwan3PolicyList)
+ in.DeepCopyInto(out)
+ return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *Mwan3PolicyList) DeepCopyObject() runtime.Object {
+ if c := in.DeepCopy(); c != nil {
+ return c
+ }
+ return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Mwan3PolicyMember) DeepCopyInto(out *Mwan3PolicyMember) {
+ *out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Mwan3PolicyMember.
+func (in *Mwan3PolicyMember) DeepCopy() *Mwan3PolicyMember {
+ if in == nil {
+ return nil
+ }
+ out := new(Mwan3PolicyMember)
+ in.DeepCopyInto(out)
+ return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Mwan3PolicySpec) DeepCopyInto(out *Mwan3PolicySpec) {
+ *out = *in
+ if in.Members != nil {
+ in, out := &in.Members, &out.Members
+ *out = make([]Mwan3PolicyMember, len(*in))
+ copy(*out, *in)
+ }
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Mwan3PolicySpec.
+func (in *Mwan3PolicySpec) DeepCopy() *Mwan3PolicySpec {
+ if in == nil {
+ return nil
+ }
+ out := new(Mwan3PolicySpec)
+ in.DeepCopyInto(out)
+ return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *SdewanStatus) DeepCopyInto(out *SdewanStatus) {
+ *out = *in
+ if in.AppliedTime != nil {
+ in, out := &in.AppliedTime, &out.AppliedTime
+ *out = (*in).DeepCopy()
+ }
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SdewanStatus.
+func (in *SdewanStatus) DeepCopy() *SdewanStatus {
+ if in == nil {
+ return nil
+ }
+ out := new(SdewanStatus)
+ in.DeepCopyInto(out)
+ return out
+}
--- /dev/null
+package cnfprovider
+
+import (
+ sdewanv1alpha1 "sdewan.akraino.org/sdewan/api/v1alpha1"
+)
+
+type CnfProvider interface {
+ AddUpdateMwan3Policy(*sdewanv1alpha1.Mwan3Policy) error
+ DeleteMwan3Policy(*sdewanv1alpha1.Mwan3Policy) error
+ // TODO: Add more Interfaces here
+ IsCnfReady() (bool, error)
+}
--- /dev/null
+package cnfprovider
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ appsv1 "k8s.io/api/apps/v1"
+ corev1 "k8s.io/api/core/v1"
+ "reflect"
+ sdewanv1alpha1 "sdewan.akraino.org/sdewan/api/v1alpha1"
+ "sdewan.akraino.org/sdewan/openwrt"
+ "sigs.k8s.io/controller-runtime/pkg/client"
+ logf "sigs.k8s.io/controller-runtime/pkg/log"
+ "strconv"
+)
+
+var log = logf.Log.WithName("wrt_provider")
+
+type WrtProvider struct {
+ Namespace string
+ SdewanPurpose string
+ Deployment appsv1.Deployment
+ K8sClient client.Client
+}
+
+func NewWrt(namespace string, sdewanPurpose string, k8sClient client.Client) (*WrtProvider, error) {
+ reqLogger := log.WithValues("namespace", namespace, "sdewanPurpose", sdewanPurpose)
+ ctx := context.Background()
+ deployments := &appsv1.DeploymentList{}
+ err := k8sClient.List(ctx, deployments, client.MatchingLabels{"sdewanPurpose": sdewanPurpose})
+ if err != nil {
+ reqLogger.Error(err, "Failed to get cnf deployment")
+ return nil, client.IgnoreNotFound(err)
+ }
+ if len(deployments.Items) <= 0 {
+ reqLogger.Info("No deployment exists")
+ return nil, nil
+ }
+ if len(deployments.Items) > 1 {
+ reqLogger.Error(nil, "More than one deployment exists")
+ return nil, errors.New("More than one deployment exists")
+ }
+
+ return &WrtProvider{namespace, sdewanPurpose, deployments.Items[0], k8sClient}, nil
+}
+
+func (p *WrtProvider) net2iface(net string) (string, error) {
+ type Iface struct {
+ DefaultGateway bool `json:"defaultGateway,string"`
+ Interface string
+ Name string
+ }
+ type NfnNet struct {
+ Type string
+ Interface []Iface
+ }
+ ann := p.Deployment.Spec.Template.Annotations
+ nfnNet := NfnNet{}
+ err := json.Unmarshal([]byte(ann["k8s.plugin.opnfv.org/nfn-network"]), &nfnNet)
+ if err != nil {
+ return "", err
+ }
+ for _, iface := range nfnNet.Interface {
+ if iface.Name == net {
+ return iface.Interface, nil
+ }
+ }
+ return "", errors.New(fmt.Sprintf("No matched network in annotation: %s", net))
+
+}
+
+func (p *WrtProvider) convertCrd(mwan3Policy *sdewanv1alpha1.Mwan3Policy) (*openwrt.SdewanPolicy, error) {
+ members := make([]openwrt.SdewanMember, len(mwan3Policy.Spec.Members))
+ for i, membercr := range mwan3Policy.Spec.Members {
+ iface, err := p.net2iface(membercr.Network)
+ if err != nil {
+ return nil, err
+ }
+ members[i] = openwrt.SdewanMember{
+ Interface: iface,
+ Metric: strconv.Itoa(membercr.Metric),
+ Weight: strconv.Itoa(membercr.Weight),
+ }
+ }
+ return &openwrt.SdewanPolicy{Name: mwan3Policy.Name, Members: members}, nil
+
+}
+
+func (p *WrtProvider) AddUpdateMwan3Policy(mwan3Policy *sdewanv1alpha1.Mwan3Policy) (bool, error) {
+ reqLogger := log.WithValues("Mwan3Policy", mwan3Policy.Name, "cnf", p.Deployment.Name)
+ ctx := context.Background()
+ podList := &corev1.PodList{}
+ err := p.K8sClient.List(ctx, podList, client.MatchingLabels{"sdewanPurpose": p.SdewanPurpose})
+ if err != nil {
+ reqLogger.Error(err, "Failed to get cnf pod list")
+ return false, err
+ }
+ policy, err := p.convertCrd(mwan3Policy)
+ if err != nil {
+ reqLogger.Error(err, "Failed to convert mwan3Policy CR")
+ return false, err
+ }
+ cnfChanged := false
+ for _, pod := range podList.Items {
+ openwrtClient := openwrt.NewOpenwrtClient(pod.Status.PodIP, "root", "")
+ mwan3 := openwrt.Mwan3Client{OpenwrtClient: openwrtClient}
+ service := openwrt.ServiceClient{OpenwrtClient: openwrtClient}
+ runtimePolicy, _ := mwan3.GetPolicy(policy.Name)
+ changed := false
+ if runtimePolicy == nil {
+ _, err := mwan3.CreatePolicy(*policy)
+ if err != nil {
+ reqLogger.Error(err, "Failed to create policy")
+ return false, err
+ }
+ changed = true
+ } else if reflect.DeepEqual(*runtimePolicy, *policy) {
+ reqLogger.Info("Equal to the runtime policy, so no update")
+ } else {
+ _, err := mwan3.UpdatePolicy(*policy)
+ if err != nil {
+ reqLogger.Error(err, "Failed to update policy")
+ return false, err
+ }
+ changed = true
+ }
+ if changed {
+ _, err = service.ExecuteService("mwan3", "restart")
+ if err != nil {
+ reqLogger.Error(err, "Failed to restart mwan3 service")
+ return changed, err
+ }
+ cnfChanged = true
+ }
+ }
+ // We say the AddUpdate succeed only when the add/update for all pods succeed
+ return cnfChanged, nil
+}
+
+func (p *WrtProvider) DeleteMwan3Policy(mwan3Policy *sdewanv1alpha1.Mwan3Policy) (bool, error) {
+ reqLogger := log.WithValues("Mwan3Policy", mwan3Policy.Name, "cnf", p.Deployment.Name)
+ ctx := context.Background()
+ podList := &corev1.PodList{}
+ err := p.K8sClient.List(ctx, podList, client.MatchingLabels{"sdewanPurpose": p.SdewanPurpose})
+ if err != nil {
+ reqLogger.Error(err, "Failed to get pod list")
+ return false, err
+ }
+ cnfChanged := false
+ for _, pod := range podList.Items {
+ openwrtClient := openwrt.NewOpenwrtClient(pod.Status.PodIP, "root", "")
+ mwan3 := openwrt.Mwan3Client{OpenwrtClient: openwrtClient}
+ service := openwrt.ServiceClient{OpenwrtClient: openwrtClient}
+ runtimePolicy, _ := mwan3.GetPolicy(mwan3Policy.Name)
+ if runtimePolicy == nil {
+ reqLogger.Info("Runtime policy doesn't exist, so don't have to delete")
+ } else {
+ err = mwan3.DeletePolicy(mwan3Policy.Name)
+ if err != nil {
+ reqLogger.Error(err, "Failed to delete policy")
+ return false, err
+ }
+ _, err = service.ExecuteService("mwan3", "restart")
+ if err != nil {
+ reqLogger.Error(err, "Failed to restart mwan3 service")
+ return false, err
+ }
+ cnfChanged = true
+ }
+ }
+ // We say the deletioni succeed only when the deletion for all pods succeed
+ return cnfChanged, nil
+}
--- /dev/null
+# The following manifests contain a self-signed issuer CR and a certificate CR.
+# More document can be found at https://docs.cert-manager.io
+# WARNING: Targets CertManager 0.11 check https://docs.cert-manager.io/en/latest/tasks/upgrading/index.html for breaking changes
+apiVersion: cert-manager.io/v1alpha2
+kind: Issuer
+metadata:
+ name: selfsigned-issuer
+ namespace: system
+spec:
+ selfSigned: {}
+---
+apiVersion: cert-manager.io/v1alpha2
+kind: Certificate
+metadata:
+ name: serving-cert # this name should match the one appeared in kustomizeconfig.yaml
+ namespace: system
+spec:
+ # $(SERVICE_NAME) and $(SERVICE_NAMESPACE) will be substituted by kustomize
+ dnsNames:
+ - $(SERVICE_NAME).$(SERVICE_NAMESPACE).svc
+ - $(SERVICE_NAME).$(SERVICE_NAMESPACE).svc.cluster.local
+ issuerRef:
+ kind: Issuer
+ name: selfsigned-issuer
+ secretName: webhook-server-cert # this secret will not be prefixed, since it's not managed by kustomize
--- /dev/null
+resources:
+- certificate.yaml
+
+configurations:
+- kustomizeconfig.yaml
--- /dev/null
+# This configuration is for teaching kustomize how to update name ref and var substitution
+nameReference:
+- kind: Issuer
+ group: cert-manager.io
+ fieldSpecs:
+ - kind: Certificate
+ group: cert-manager.io
+ path: spec/issuerRef/name
+
+varReference:
+- kind: Certificate
+ group: cert-manager.io
+ path: spec/commonName
+- kind: Certificate
+ group: cert-manager.io
+ path: spec/dnsNames
--- /dev/null
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+ annotations:
+ controller-gen.kubebuilder.io/version: v0.2.4
+ creationTimestamp: null
+ name: mwan3policies.batch.sdewan.akraino.org
+spec:
+ group: batch.sdewan.akraino.org
+ names:
+ kind: Mwan3Policy
+ listKind: Mwan3PolicyList
+ plural: mwan3policies
+ singular: mwan3policy
+ scope: Namespaced
+ subresources:
+ status: {}
+ validation:
+ openAPIV3Schema:
+ description: Mwan3Policy is the Schema for the mwan3policies API
+ properties:
+ apiVersion:
+ description: 'APIVersion defines the versioned schema of this representation
+ of an object. Servers should convert recognized schemas to the latest
+ internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+ type: string
+ kind:
+ description: 'Kind is a string value representing the REST resource this
+ object represents. Servers may infer this from the endpoint the client
+ submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+ type: string
+ metadata:
+ type: object
+ spec:
+ properties:
+ members:
+ items:
+ description: Mwan3PolicySpec defines the desired state of Mwan3Policy
+ properties:
+ metric:
+ type: integer
+ network:
+ description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of
+ cluster Important: Run "make" to regenerate code after modifying
+ this file'
+ type: string
+ weight:
+ type: integer
+ required:
+ - metric
+ - network
+ - weight
+ type: object
+ type: array
+ required:
+ - members
+ type: object
+ status:
+ description: status subsource used for Sdewan rule CRDs
+ properties:
+ appliedTime:
+ format: date-time
+ type: string
+ appliedVersion:
+ description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
+ of cluster Important: Run "make" to regenerate code after modifying
+ this file'
+ type: string
+ inSync:
+ type: boolean
+ required:
+ - appliedTime
+ - appliedVersion
+ - inSync
+ type: object
+ type: object
+ version: v1alpha1
+ versions:
+ - name: v1alpha1
+ served: true
+ storage: true
+status:
+ acceptedNames:
+ kind: ""
+ plural: ""
+ conditions: []
+ storedVersions: []
--- /dev/null
+# This kustomization.yaml is not intended to be run by itself,
+# since it depends on service name and namespace that are out of this kustomize package.
+# It should be run by config/default
+resources:
+- bases/batch.sdewan.akraino.org_mwan3policies.yaml
+# +kubebuilder:scaffold:crdkustomizeresource
+
+patchesStrategicMerge:
+# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix.
+# patches here are for enabling the conversion webhook for each CRD
+#- patches/webhook_in_mwan3policies.yaml
+# +kubebuilder:scaffold:crdkustomizewebhookpatch
+
+# [CERTMANAGER] To enable webhook, uncomment all the sections with [CERTMANAGER] prefix.
+# patches here are for enabling the CA injection for each CRD
+#- patches/cainjection_in_mwan3policies.yaml
+# +kubebuilder:scaffold:crdkustomizecainjectionpatch
+
+# the following config is for teaching kustomize how to do kustomization for CRDs.
+configurations:
+- kustomizeconfig.yaml
--- /dev/null
+# This file is for teaching kustomize how to substitute name and namespace reference in CRD
+nameReference:
+- kind: Service
+ version: v1
+ fieldSpecs:
+ - kind: CustomResourceDefinition
+ group: apiextensions.k8s.io
+ path: spec/conversion/webhookClientConfig/service/name
+
+namespace:
+- kind: CustomResourceDefinition
+ group: apiextensions.k8s.io
+ path: spec/conversion/webhookClientConfig/service/namespace
+ create: false
+
+varReference:
+- path: metadata/annotations
--- /dev/null
+# The following patch adds a directive for certmanager to inject CA into the CRD
+# CRD conversion requires k8s 1.13 or later.
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+ annotations:
+ cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+ name: mwan3policies.batch.sdewan.akraino.org
--- /dev/null
+# The following patch enables conversion webhook for CRD
+# CRD conversion requires k8s 1.13 or later.
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+ name: mwan3policies.batch.sdewan.akraino.org
+spec:
+ conversion:
+ strategy: Webhook
+ webhookClientConfig:
+ # this is "\n" used as a placeholder, otherwise it will be rejected by the apiserver for being blank,
+ # but we're going to set it later using the cert-manager (or potentially a patch if not using cert-manager)
+ caBundle: Cg==
+ service:
+ namespace: system
+ name: webhook-service
+ path: /convert
--- /dev/null
+# Adds namespace to all resources.
+namespace: sdewan-system
+
+# Value of this field is prepended to the
+# names of all resources, e.g. a deployment named
+# "wordpress" becomes "alices-wordpress".
+# Note that it should also match with the prefix (text before '-') of the namespace
+# field above.
+namePrefix: sdewan-
+
+# Labels to add to all resources and selectors.
+#commonLabels:
+# someName: someValue
+
+bases:
+- ../crd
+- ../rbac
+- ../manager
+# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in crd/kustomization.yaml
+#- ../webhook
+# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
+#- ../certmanager
+# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
+#- ../prometheus
+
+patchesStrategicMerge:
+ # Protect the /metrics endpoint by putting it behind auth.
+ # Only one of manager_auth_proxy_patch.yaml and
+ # manager_prometheus_metrics_patch.yaml should be enabled.
+- manager_auth_proxy_patch.yaml
+ # If you want your controller-manager to expose the /metrics
+ # endpoint w/o any authn/z, uncomment the following line and
+ # comment manager_auth_proxy_patch.yaml.
+ # Only one of manager_auth_proxy_patch.yaml and
+ # manager_prometheus_metrics_patch.yaml should be enabled.
+#- manager_prometheus_metrics_patch.yaml
+
+# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in crd/kustomization.yaml
+#- manager_webhook_patch.yaml
+
+# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
+# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
+# 'CERTMANAGER' needs to be enabled to use ca injection
+#- webhookcainjection_patch.yaml
+
+# the following config is for teaching kustomize how to do var substitution
+vars:
+# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
+#- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
+# objref:
+# kind: Certificate
+# group: cert-manager.io
+# version: v1alpha2
+# name: serving-cert # this name should match the one in certificate.yaml
+# fieldref:
+# fieldpath: metadata.namespace
+#- name: CERTIFICATE_NAME
+# objref:
+# kind: Certificate
+# group: cert-manager.io
+# version: v1alpha2
+# name: serving-cert # this name should match the one in certificate.yaml
+#- name: SERVICE_NAMESPACE # namespace of the service
+# objref:
+# kind: Service
+# version: v1
+# name: webhook-service
+# fieldref:
+# fieldpath: metadata.namespace
+#- name: SERVICE_NAME
+# objref:
+# kind: Service
+# version: v1
+# name: webhook-service
--- /dev/null
+# This patch inject a sidecar container which is a HTTP proxy for the controller manager,
+# it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: controller-manager
+ namespace: system
+spec:
+ template:
+ spec:
+ containers:
+ - name: kube-rbac-proxy
+ image: gcr.io/kubebuilder/kube-rbac-proxy:v0.4.1
+ args:
+ - "--secure-listen-address=0.0.0.0:8443"
+ - "--upstream=http://127.0.0.1:8080/"
+ - "--logtostderr=true"
+ - "--v=10"
+ ports:
+ - containerPort: 8443
+ name: https
+ - name: manager
+ args:
+ - "--metrics-addr=127.0.0.1:8080"
+ - "--enable-leader-election"
--- /dev/null
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: controller-manager
+ namespace: system
+spec:
+ template:
+ spec:
+ containers:
+ - name: manager
+ ports:
+ - containerPort: 9443
+ name: webhook-server
+ protocol: TCP
+ volumeMounts:
+ - mountPath: /tmp/k8s-webhook-server/serving-certs
+ name: cert
+ readOnly: true
+ volumes:
+ - name: cert
+ secret:
+ defaultMode: 420
+ secretName: webhook-server-cert
--- /dev/null
+# This patch add annotation to admission webhook config and
+# the variables $(CERTIFICATE_NAMESPACE) and $(CERTIFICATE_NAME) will be substituted by kustomize.
+apiVersion: admissionregistration.k8s.io/v1beta1
+kind: MutatingWebhookConfiguration
+metadata:
+ name: mutating-webhook-configuration
+ annotations:
+ cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+---
+apiVersion: admissionregistration.k8s.io/v1beta1
+kind: ValidatingWebhookConfiguration
+metadata:
+ name: validating-webhook-configuration
+ annotations:
+ cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
--- /dev/null
+resources:
+- manager.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+images:
+- name: controller
+ newName: integratedcloudnative/sdewan-controller
+ newTag: dev
--- /dev/null
+apiVersion: v1
+kind: Namespace
+metadata:
+ labels:
+ control-plane: controller-manager
+ name: system
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: controller-manager
+ namespace: system
+ labels:
+ control-plane: controller-manager
+spec:
+ selector:
+ matchLabels:
+ control-plane: controller-manager
+ replicas: 1
+ template:
+ metadata:
+ labels:
+ control-plane: controller-manager
+ spec:
+ containers:
+ - command:
+ - /manager
+ args:
+ - --enable-leader-election
+ image: controller:latest
+ name: manager
+ resources:
+ limits:
+ cpu: 100m
+ memory: 30Mi
+ requests:
+ cpu: 100m
+ memory: 20Mi
+ terminationGracePeriodSeconds: 10
--- /dev/null
+resources:
+- monitor.yaml
--- /dev/null
+
+# Prometheus Monitor Service (Metrics)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+ labels:
+ control-plane: controller-manager
+ name: controller-manager-metrics-monitor
+ namespace: system
+spec:
+ endpoints:
+ - path: /metrics
+ port: https
+ selector:
+ control-plane: controller-manager
--- /dev/null
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ name: proxy-role
+rules:
+- apiGroups: ["authentication.k8s.io"]
+ resources:
+ - tokenreviews
+ verbs: ["create"]
+- apiGroups: ["authorization.k8s.io"]
+ resources:
+ - subjectaccessreviews
+ verbs: ["create"]
--- /dev/null
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+ name: proxy-rolebinding
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: proxy-role
+subjects:
+- kind: ServiceAccount
+ name: default
+ namespace: system
--- /dev/null
+apiVersion: v1
+kind: Service
+metadata:
+ labels:
+ control-plane: controller-manager
+ name: controller-manager-metrics-service
+ namespace: system
+spec:
+ ports:
+ - name: https
+ port: 8443
+ targetPort: https
+ selector:
+ control-plane: controller-manager
--- /dev/null
+resources:
+- role.yaml
+- role_binding.yaml
+- leader_election_role.yaml
+- leader_election_role_binding.yaml
+# Comment the following 3 lines if you want to disable
+# the auth proxy (https://github.com/brancz/kube-rbac-proxy)
+# which protects your /metrics endpoint.
+- auth_proxy_service.yaml
+- auth_proxy_role.yaml
+- auth_proxy_role_binding.yaml
--- /dev/null
+# permissions to do leader election.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+ name: leader-election-role
+rules:
+- apiGroups:
+ - ""
+ resources:
+ - configmaps
+ verbs:
+ - get
+ - list
+ - watch
+ - create
+ - update
+ - patch
+ - delete
+- apiGroups:
+ - ""
+ resources:
+ - configmaps/status
+ verbs:
+ - get
+ - update
+ - patch
+- apiGroups:
+ - ""
+ resources:
+ - events
+ verbs:
+ - create
--- /dev/null
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+ name: leader-election-rolebinding
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: Role
+ name: leader-election-role
+subjects:
+- kind: ServiceAccount
+ name: default
+ namespace: system
--- /dev/null
+# permissions to do edit mwan3policies.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ name: mwan3policy-editor-role
+rules:
+- apiGroups:
+ - batch.sdewan.akraino.org
+ resources:
+ - mwan3policies
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - batch.sdewan.akraino.org
+ resources:
+ - mwan3policies/status
+ verbs:
+ - get
+ - patch
+ - update
--- /dev/null
+# permissions to do viewer mwan3policies.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ name: mwan3policy-viewer-role
+rules:
+- apiGroups:
+ - batch.sdewan.akraino.org
+ resources:
+ - mwan3policies
+ verbs:
+ - get
+ - list
+ - watch
+- apiGroups:
+ - batch.sdewan.akraino.org
+ resources:
+ - mwan3policies/status
+ verbs:
+ - get
--- /dev/null
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ creationTimestamp: null
+ name: manager-role
+rules:
+- apiGroups:
+ - batch.sdewan.akraino.org
+ resources:
+ - mwan3policies
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - batch.sdewan.akraino.org
+ resources:
+ - mwan3policies/status
+ verbs:
+ - get
+ - patch
+ - update
+- apiGroups:
+ - extensions
+ resources:
+ - deployments
+ verbs:
+ - get
+ - list
+ - watch
--- /dev/null
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+ name: manager-rolebinding
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: manager-role
+subjects:
+- kind: ServiceAccount
+ name: default
+ namespace: system
--- /dev/null
+apiVersion: batch.sdewan.akraino.org/v1alpha1
+kind: Mwan3Policy
+metadata:
+ name: balance1
+ namespace: default
+ labels:
+ sdewanPurpose: cnf1
+spec:
+ members:
+ - network: ovn-net1
+ weight: 4
+ metric: 2
+ - network: ovn-net2
+ weight: 3
+ metric: 3
--- /dev/null
+resources:
+- manifests.yaml
+- service.yaml
+
+configurations:
+- kustomizeconfig.yaml
--- /dev/null
+# the following config is for teaching kustomize where to look at when substituting vars.
+# It requires kustomize v2.1.0 or newer to work properly.
+nameReference:
+- kind: Service
+ version: v1
+ fieldSpecs:
+ - kind: MutatingWebhookConfiguration
+ group: admissionregistration.k8s.io
+ path: webhooks/clientConfig/service/name
+ - kind: ValidatingWebhookConfiguration
+ group: admissionregistration.k8s.io
+ path: webhooks/clientConfig/service/name
+
+namespace:
+- kind: MutatingWebhookConfiguration
+ group: admissionregistration.k8s.io
+ path: webhooks/clientConfig/service/namespace
+ create: true
+- kind: ValidatingWebhookConfiguration
+ group: admissionregistration.k8s.io
+ path: webhooks/clientConfig/service/namespace
+ create: true
+
+varReference:
+- path: metadata/annotations
--- /dev/null
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: webhook-service
+ namespace: system
+spec:
+ ports:
+ - port: 443
+ targetPort: 9443
+ selector:
+ control-plane: controller-manager
--- /dev/null
+/*
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controllers
+
+import (
+ "context"
+ "time"
+
+ "github.com/go-logr/logr"
+ "k8s.io/apimachinery/pkg/api/errors"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/apimachinery/pkg/runtime"
+ ctrl "sigs.k8s.io/controller-runtime"
+ "sigs.k8s.io/controller-runtime/pkg/client"
+
+ batchv1alpha1 "sdewan.akraino.org/sdewan/api/v1alpha1"
+ "sdewan.akraino.org/sdewan/cnfprovider"
+)
+
+// Mwan3PolicyReconciler reconciles a Mwan3Policy object
+type Mwan3PolicyReconciler struct {
+ client.Client
+ Log logr.Logger
+ Scheme *runtime.Scheme
+}
+
+// +kubebuilder:rbac:groups=batch.sdewan.akraino.org,resources=mwan3policies,verbs=get;list;watch;create;update;patch;delete
+// +kubebuilder:rbac:groups=batch.sdewan.akraino.org,resources=mwan3policies/status,verbs=get;update;patch
+// +kubebuilder:rbac:groups=extensions,resources=deployments,verbs=get;list;watch
+
+func (r *Mwan3PolicyReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
+ ctx := context.Background()
+ log := r.Log.WithValues("mwan3policy", req.NamespacedName)
+
+ // your logic here
+ during, _ := time.ParseDuration("5s")
+ instance := &batchv1alpha1.Mwan3Policy{}
+ err := r.Get(ctx, req.NamespacedName, instance)
+ if err != nil {
+ if errors.IsNotFound(err) {
+ // No instance
+ return ctrl.Result{}, nil
+ }
+ // Error reading the object - requeue the request.
+ return ctrl.Result{RequeueAfter: during}, nil
+ }
+ cnf, err := cnfprovider.NewWrt(req.NamespacedName.Namespace, instance.Labels["sdewanPurpose"], r.Client)
+ if err != nil {
+ log.Error(err, "Failed to get cnf")
+ // A new event are supposed to be received upon cnf ready
+ // so not requeue
+ return ctrl.Result{}, nil
+ }
+ finalizerName := "rule.finalizers.sdewan.akraino.org"
+ if instance.ObjectMeta.DeletionTimestamp.IsZero() {
+ // creating or updating CR
+ if cnf == nil {
+ // no cnf exists
+ log.Info("No cnf exist, so not create/update mwan3 policy")
+ return ctrl.Result{}, nil
+ }
+ changed, err := cnf.AddUpdateMwan3Policy(instance)
+ if err != nil {
+ log.Error(err, "Failed to add/update mwan3 policy")
+ return ctrl.Result{RequeueAfter: during}, nil
+ }
+ if !containsString(instance.ObjectMeta.Finalizers, finalizerName) {
+ log.Info("Adding finalizer for mwan3 policy")
+ instance.ObjectMeta.Finalizers = append(instance.ObjectMeta.Finalizers, finalizerName)
+ if err := r.Update(ctx, instance); err != nil {
+ return ctrl.Result{}, err
+ }
+ }
+ if changed {
+ instance.Status.AppliedVersion = instance.ResourceVersion
+ instance.Status.AppliedTime = &metav1.Time{Time: time.Now()}
+ instance.Status.InSync = true
+ err = r.Status().Update(ctx, instance)
+ if err != nil {
+ log.Error(err, "Failed to update mwan3 policy status")
+ return ctrl.Result{}, err
+ }
+ }
+ } else {
+ // deletin CR
+ if cnf == nil {
+ // no cnf exists
+ if containsString(instance.ObjectMeta.Finalizers, finalizerName) {
+ instance.ObjectMeta.Finalizers = removeString(instance.ObjectMeta.Finalizers, finalizerName)
+ if err := r.Update(ctx, instance); err != nil {
+ return ctrl.Result{}, err
+ }
+ }
+ return ctrl.Result{}, nil
+ }
+ _, err := cnf.DeleteMwan3Policy(instance)
+ if err != nil {
+ log.Error(err, "Failed to delete mwan3 policy")
+ return ctrl.Result{RequeueAfter: during}, nil
+ }
+ if containsString(instance.ObjectMeta.Finalizers, finalizerName) {
+ instance.ObjectMeta.Finalizers = removeString(instance.ObjectMeta.Finalizers, finalizerName)
+ if err := r.Update(ctx, instance); err != nil {
+ return ctrl.Result{}, err
+ }
+ }
+ }
+
+ return ctrl.Result{}, nil
+}
+
+func (r *Mwan3PolicyReconciler) SetupWithManager(mgr ctrl.Manager) error {
+ return ctrl.NewControllerManagedBy(mgr).
+ For(&batchv1alpha1.Mwan3Policy{}).
+ Complete(r)
+}
+
+// Helper functions to check and remove string from a slice of strings.
+func containsString(slice []string, s string) bool {
+ for _, item := range slice {
+ if item == s {
+ return true
+ }
+ }
+ return false
+}
+
+func removeString(slice []string, s string) (result []string) {
+ for _, item := range slice {
+ if item == s {
+ continue
+ }
+ result = append(result, item)
+ }
+ return
+}
--- /dev/null
+/*
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controllers
+
+import (
+ "path/filepath"
+ "testing"
+
+ . "github.com/onsi/ginkgo"
+ . "github.com/onsi/gomega"
+
+ "k8s.io/client-go/kubernetes/scheme"
+ "k8s.io/client-go/rest"
+ batchv1alpha1 "sdewan.akraino.org/sdewan/api/v1alpha1"
+ "sigs.k8s.io/controller-runtime/pkg/client"
+ "sigs.k8s.io/controller-runtime/pkg/envtest"
+ logf "sigs.k8s.io/controller-runtime/pkg/log"
+ "sigs.k8s.io/controller-runtime/pkg/log/zap"
+ // +kubebuilder:scaffold:imports
+)
+
+// These tests use Ginkgo (BDD-style Go testing framework). Refer to
+// http://onsi.github.io/ginkgo/ to learn more about Ginkgo.
+
+var cfg *rest.Config
+var k8sClient client.Client
+var testEnv *envtest.Environment
+
+func TestAPIs(t *testing.T) {
+ RegisterFailHandler(Fail)
+
+ RunSpecsWithDefaultAndCustomReporters(t,
+ "Controller Suite",
+ []Reporter{envtest.NewlineReporter{}})
+}
+
+var _ = BeforeSuite(func(done Done) {
+ logf.SetLogger(zap.LoggerTo(GinkgoWriter, true))
+
+ By("bootstrapping test environment")
+ testEnv = &envtest.Environment{
+ CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
+ }
+
+ var err error
+ cfg, err = testEnv.Start()
+ Expect(err).ToNot(HaveOccurred())
+ Expect(cfg).ToNot(BeNil())
+
+ err = batchv1alpha1.AddToScheme(scheme.Scheme)
+ Expect(err).NotTo(HaveOccurred())
+
+ // +kubebuilder:scaffold:scheme
+
+ k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
+ Expect(err).ToNot(HaveOccurred())
+ Expect(k8sClient).ToNot(BeNil())
+
+ close(done)
+}, 60)
+
+var _ = AfterSuite(func() {
+ By("tearing down the test environment")
+ err := testEnv.Stop()
+ Expect(err).ToNot(HaveOccurred())
+})
--- /dev/null
+module sdewan.akraino.org/sdewan
+
+go 1.14
+
+require (
+ github.com/go-logr/logr v0.1.0
+ github.com/onsi/ginkgo v1.8.0
+ github.com/onsi/gomega v1.5.0
+ k8s.io/api v0.0.0-20190918155943-95b840bb6a1f
+ k8s.io/apimachinery v0.0.0-20190913080033-27d36303b655
+ k8s.io/client-go v0.0.0-20190918160344-1fbdaa4c8d90
+ sigs.k8s.io/controller-runtime v0.4.0
+)
--- /dev/null
+cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+cloud.google.com/go v0.38.0 h1:ROfEUZz+Gh5pa62DJWXSaonyu3StP6EA6lPEXPI6mCo=
+cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
+github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
+github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI=
+github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0=
+github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA=
+github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0=
+github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0=
+github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc=
+github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk=
+github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
+github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
+github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
+github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
+github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
+github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
+github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
+github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
+github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
+github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
+github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLMYoU8P317H5OQ+Via4RmuPwCS0=
+github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
+github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
+github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/coreos/bbolt v1.3.1-coreos.6/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
+github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
+github.com/coreos/etcd v3.3.15+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
+github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
+github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
+github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
+github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
+github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
+github.com/coreos/pkg v0.0.0-20180108230652-97fdf19511ea/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
+github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
+github.com/davecgh/go-spew v0.0.0-20151105211317-5215b55f46b2/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
+github.com/docker/docker v0.7.3-0.20190327010347-be7ac8be2ae0/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
+github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
+github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
+github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
+github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
+github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
+github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
+github.com/evanphx/json-patch v4.2.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
+github.com/evanphx/json-patch v4.5.0+incompatible h1:ouOWdg56aJriqS0huScTkVXPC5IcNrDCXZ6OoTAWu7M=
+github.com/evanphx/json-patch v4.5.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
+github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
+github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
+github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
+github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
+github.com/go-logr/logr v0.1.0 h1:M1Tv3VzNlEHg6uyACnRdtrploV2P7wZqH8BoQMtz0cg=
+github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
+github.com/go-logr/zapr v0.1.0 h1:h+WVe9j6HAA01niTJPA/kKH0i7e0rLZBCwauQFcRE54=
+github.com/go-logr/zapr v0.1.0/go.mod h1:tabnROwaDl0UNxkVeFRbY8bwB37GwRv0P8lg6aAiEnk=
+github.com/go-openapi/analysis v0.0.0-20180825180245-b006789cd277/go.mod h1:k70tL6pCuVxPJOHXQ+wIac1FUrvNkHolPie/cLEU6hI=
+github.com/go-openapi/analysis v0.17.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik=
+github.com/go-openapi/analysis v0.18.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik=
+github.com/go-openapi/analysis v0.19.2/go.mod h1:3P1osvZa9jKjb8ed2TPng3f0i/UY9snX6gxi44djMjk=
+github.com/go-openapi/errors v0.17.0/go.mod h1:LcZQpmvG4wyF5j4IhA73wkLFQg+QJXOQHVjmcZxhka0=
+github.com/go-openapi/errors v0.18.0/go.mod h1:LcZQpmvG4wyF5j4IhA73wkLFQg+QJXOQHVjmcZxhka0=
+github.com/go-openapi/errors v0.19.2/go.mod h1:qX0BLWsyaKfvhluLejVpVNwNRdXZhEbTA4kxxpKBC94=
+github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0=
+github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M=
+github.com/go-openapi/jsonpointer v0.18.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M=
+github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg=
+github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg=
+github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I=
+github.com/go-openapi/jsonreference v0.18.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I=
+github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc=
+github.com/go-openapi/loads v0.17.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU=
+github.com/go-openapi/loads v0.18.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU=
+github.com/go-openapi/loads v0.19.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU=
+github.com/go-openapi/loads v0.19.2/go.mod h1:QAskZPMX5V0C2gvfkGZzJlINuP7Hx/4+ix5jWFxsNPs=
+github.com/go-openapi/runtime v0.0.0-20180920151709-4f900dc2ade9/go.mod h1:6v9a6LTXWQCdL8k1AO3cvqx5OtZY/Y9wKTgaoP6YRfA=
+github.com/go-openapi/runtime v0.19.0/go.mod h1:OwNfisksmmaZse4+gpV3Ne9AyMOlP1lt4sK4FXt0O64=
+github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc=
+github.com/go-openapi/spec v0.17.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI=
+github.com/go-openapi/spec v0.18.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI=
+github.com/go-openapi/spec v0.19.2/go.mod h1:sCxk3jxKgioEJikev4fgkNmwS+3kuYdJtcsZsD5zxMY=
+github.com/go-openapi/strfmt v0.17.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU=
+github.com/go-openapi/strfmt v0.18.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU=
+github.com/go-openapi/strfmt v0.19.0/go.mod h1:+uW+93UVvGGq2qGaZxdDeJqSAqBqBdl+ZPMF/cC8nDY=
+github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I=
+github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg=
+github.com/go-openapi/swag v0.18.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg=
+github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
+github.com/go-openapi/validate v0.18.0/go.mod h1:Uh4HdOzKt19xGIGm1qHf/ofbX1YQ4Y+MYsct2VUrAJ4=
+github.com/go-openapi/validate v0.19.2/go.mod h1:1tRCw7m3jtI8eNWEEliiAqUIcBztB2KDnRCRMUi7GTA=
+github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d h1:3PaI8p3seN09VjbTYC/QWlUZdZ1qS1zGjy7LH2Wt07I=
+github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
+github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/groupcache v0.0.0-20180513044358-24b0969c4cb7 h1:u4bArs140e9+AfE52mFHOXVFnOSBJBRlzTHrOPLOIhE=
+github.com/golang/groupcache v0.0.0-20180513044358-24b0969c4cb7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.0.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
+github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
+github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
+github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY=
+github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
+github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI=
+github.com/google/gofuzz v1.0.0 h1:A8PeW59pxE9IoFRqBp37U+mSNaQoZ46F1f0f863XSXw=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
+github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
+github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
+github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
+github.com/googleapis/gnostic v0.3.1 h1:WeAefnSUHlBb0iJKwxFDZdbfGwkd7xRNuV+IpXMJhYk=
+github.com/googleapis/gnostic v0.3.1/go.mod h1:on+2t9HRStVgn95RSsFWFz+6Q0Snyqv1awfrALZdbtU=
+github.com/gophercloud/gophercloud v0.1.0/go.mod h1:vxM41WHh5uqHVBMZHzuwNOHh8XEoIEcSTewFxm1c5g8=
+github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
+github.com/gregjones/httpcache v0.0.0-20170728041850-787624de3eb7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
+github.com/grpc-ecosystem/go-grpc-middleware v0.0.0-20190222133341-cfaf5686ec79/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
+github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
+github.com/grpc-ecosystem/grpc-gateway v1.3.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
+github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU=
+github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
+github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
+github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
+github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
+github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
+github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
+github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
+github.com/json-iterator/go v0.0.0-20180612202835-f2b4162afba3/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
+github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
+github.com/json-iterator/go v1.1.7 h1:KfgG9LzI+pYjr4xvmz/5H4FXjokeP+rlHLhv3iH62Fo=
+github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
+github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
+github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA=
+github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
+github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
+github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180320133207-05fbef0ca5da/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
+github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
+github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v1.4.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
+github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
+github.com/onsi/gomega v1.3.0/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
+github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo=
+github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
+github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
+github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
+github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
+github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA=
+github.com/prometheus/client_golang v0.9.2 h1:awm861/B8OKDd2I/6o1dy3ra4BamzKhYOiGItCeZ740=
+github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM=
+github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 h1:idejC8f05m9MGOsuEi1ATq9shN03HrxNkD/luQvxCv8=
+github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
+github.com/prometheus/common v0.0.0-20181126121408-4724e9255275 h1:PnBWHBf+6L0jOqq0gIVUe6Yk0/QMZ640k6NvkxcBf+8=
+github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
+github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a h1:9a8MnZMP0X2nLJdBg+pBmGgkJlSaKC2KaQmTCk1XDtE=
+github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
+github.com/remyoudompheng/bigfft v0.0.0-20170806203942-52369c62f446/go.mod h1:uYEyJGbgTkfkS4+E/PavXkNJcbFIpEtjt2B0KDQ5+9M=
+github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
+github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
+github.com/soheilhy/cmux v0.1.3/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
+github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
+github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
+github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
+github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
+github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
+github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
+github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE=
+github.com/stretchr/testify v0.0.0-20151208002404-e3a8ff8ce365/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
+github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
+github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
+github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
+go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
+go.uber.org/atomic v0.0.0-20181018215023-8dc6146f7569/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
+go.uber.org/atomic v1.3.2 h1:2Oa65PReHzfn29GpvgsYwloV9AVFHPDk8tYxt2c2tr4=
+go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
+go.uber.org/multierr v0.0.0-20180122172545-ddea229ff1df/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
+go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI=
+go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
+go.uber.org/zap v0.0.0-20180814183419-67bc79d13d15/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
+go.uber.org/zap v1.9.1 h1:XCJQEf3W6eZaVwhRBof6ImoYGJSITeKWsyeh3HFu/5o=
+go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
+golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8 h1:1wopBVtVdWnn03fZelqdXTqk7U7zPQCb+T4rbU9ZEoU=
+golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
+golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
+golang.org/x/exp v0.0.0-20190312203227-4b39c73a6495/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
+golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
+golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
+golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
+golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
+golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
+golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180112015858-5ccada7d0a7b/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20181005035420-146acd28ed58/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190812203447-cdfb69ac37fc h1:gkKoSkUmnU6bpS/VhkuO27bzQeSA51uaEfbOW5dNb68=
+golang.org/x/net v0.0.0-20190812203447-cdfb69ac37fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 h1:SVwTIAaPC2U/AvvLNZ2a7OVsmBpC8L5BlwK1whH3hm0=
+golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180117170059-2c42eef0765b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190209173611-3b5209105503/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190321052220-f7bb7a8bee54/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f h1:25KHgbfyiSm6vwQLbM3zZIe1v9p/3ea4Rz+nnM5K/i4=
+golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.1-0.20171227012246-e19ae1496984/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/time v0.0.0-20181108054448-85acf8d2951c h1:fqgJT0MGcGpPgpWU7VRdRjuArfcOvC4AoJmILihzhDg=
+golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190206041539-40960b6deb8e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
+golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gomodules.xyz/jsonpatch/v2 v2.0.1 h1:xyiBuvkD2g5n7cYzx6u2sxQvsAy4QJsZFCzGVdzOXZ0=
+gomodules.xyz/jsonpatch/v2 v2.0.1/go.mod h1:IhYNNY4jnS53ZnfE4PAmpKtDpTCj1JFXc+3mwe7XcUU=
+gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485/go.mod h1:2ltnJ7xHfj0zHS40VVPYEAAMTa3ZGguvHGBSJeRWqE0=
+gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0/go.mod h1:wa6Ws7BG/ESfp6dHfk7C6KdzKA7wR7u/rKwOGE66zvw=
+gonum.org/v1/netlib v0.0.0-20190331212654-76723241ea4e/go.mod h1:kS+toOQn6AQKjmKJ7gzohV1XkqsFehRA2FbsbkopSuQ=
+google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
+google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
+google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/appengine v1.5.0 h1:KxkO13IPW4Lslp2bz+KHP2E3gtFlrIGNThxkZQ3g+4c=
+google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
+google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
+google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
+google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
+google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
+google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
+gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
+gopkg.in/inf.v0 v0.9.0/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
+gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k=
+gopkg.in/square/go-jose.v2 v2.2.2/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
+gopkg.in/yaml.v2 v2.0.0/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
+gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
+honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+k8s.io/api v0.0.0-20190918155943-95b840bb6a1f h1:8FRUST8oUkEI45WYKyD8ed7Ad0Kg5v11zHyPkEVb2xo=
+k8s.io/api v0.0.0-20190918155943-95b840bb6a1f/go.mod h1:uWuOHnjmNrtQomJrvEBg0c0HRNyQ+8KTEERVsK0PW48=
+k8s.io/apiextensions-apiserver v0.0.0-20190918161926-8f644eb6e783 h1:V6ndwCPoao1yZ52agqOKaUAl7DYWVGiXjV7ePA2i610=
+k8s.io/apiextensions-apiserver v0.0.0-20190918161926-8f644eb6e783/go.mod h1:xvae1SZB3E17UpV59AWc271W/Ph25N+bjPyR63X6tPY=
+k8s.io/apimachinery v0.0.0-20190913080033-27d36303b655 h1:CS1tBQz3HOXiseWZu6ZicKX361CZLT97UFnnPx0aqBw=
+k8s.io/apimachinery v0.0.0-20190913080033-27d36303b655/go.mod h1:nL6pwRT8NgfF8TT68DBI8uEePRt89cSvoXUVqbkWHq4=
+k8s.io/apiserver v0.0.0-20190918160949-bfa5e2e684ad/go.mod h1:XPCXEwhjaFN29a8NldXA901ElnKeKLrLtREO9ZhFyhg=
+k8s.io/client-go v0.0.0-20190918160344-1fbdaa4c8d90 h1:mLmhKUm1X+pXu0zXMEzNsOF5E2kKFGe5o6BZBIIqA6A=
+k8s.io/client-go v0.0.0-20190918160344-1fbdaa4c8d90/go.mod h1:J69/JveO6XESwVgG53q3Uz5OSfgsv4uxpScmmyYOOlk=
+k8s.io/code-generator v0.0.0-20190912054826-cd179ad6a269/go.mod h1:V5BD6M4CyaN5m+VthcclXWsVcT1Hu+glwa1bi3MIsyE=
+k8s.io/component-base v0.0.0-20190918160511-547f6c5d7090/go.mod h1:933PBGtQFJky3TEwYx4aEPZ4IxqhWh3R6DCmzqIn1hA=
+k8s.io/gengo v0.0.0-20190128074634-0689ccc1d7d6/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
+k8s.io/gengo v0.0.0-20190822140433-26a664648505/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
+k8s.io/klog v0.0.0-20181102134211-b9b56d5dfc92/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk=
+k8s.io/klog v0.3.0/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk=
+k8s.io/klog v0.4.0 h1:lCJCxf/LIowc2IGS9TPjWDyXY4nOmdGdfcwwDQCOURQ=
+k8s.io/klog v0.4.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I=
+k8s.io/kube-openapi v0.0.0-20190816220812-743ec37842bf h1:EYm5AW/UUDbnmnI+gK0TJDVK9qPLhM+sRHYanNKw0EQ=
+k8s.io/kube-openapi v0.0.0-20190816220812-743ec37842bf/go.mod h1:1TqjTSzOxsLGIKfj0lK8EeCP7K1iUG65v09OM0/WG5E=
+k8s.io/utils v0.0.0-20190801114015-581e00157fb1 h1:+ySTxfHnfzZb9ys375PXNlLhkJPLKgHajBU0N62BDvE=
+k8s.io/utils v0.0.0-20190801114015-581e00157fb1/go.mod h1:sZAwmy6armz5eXlNoLmJcl4F1QuKu7sr+mFQ0byX7Ew=
+modernc.org/cc v1.0.0/go.mod h1:1Sk4//wdnYJiUIxnW8ddKpaOJCF37yAdqYnkxUpaYxw=
+modernc.org/golex v1.0.0/go.mod h1:b/QX9oBD/LhixY6NDh+IdGv17hgB+51fET1i2kPSmvk=
+modernc.org/mathutil v1.0.0/go.mod h1:wU0vUrJsVWBZ4P6e7xtFJEhFSNsfRLJ8H458uRjg03k=
+modernc.org/strutil v1.0.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs=
+modernc.org/xc v1.0.0/go.mod h1:mRNCo0bvLjGhHO9WsyuKVU4q0ceiDDDoEeWDJHrNx8I=
+sigs.k8s.io/controller-runtime v0.4.0 h1:wATM6/m+3w8lj8FXNaO6Fs/rq/vqoOjO1Q116Z9NPsg=
+sigs.k8s.io/controller-runtime v0.4.0/go.mod h1:ApC79lpY3PHW9xj/w9pj+lYkLgwAAUZwfXkME1Lajns=
+sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e/go.mod h1:wWxsB5ozmmv/SG7nM11ayaAW51xMvak/t1r0CSlcokI=
+sigs.k8s.io/structured-merge-diff v0.0.0-20190817042607-6149e4549fca/go.mod h1:IIgPezJWb76P0hotTxzDbWsMYB8APh18qZnxkomBpxA=
+sigs.k8s.io/testing_frameworks v0.1.2 h1:vK0+tvjF0BZ/RYFeZ1E6BYBwHJJXhjuZ3TdsEKH+UQM=
+sigs.k8s.io/testing_frameworks v0.1.2/go.mod h1:ToQrwSC3s8Xf/lADdZp3Mktcql9CG0UAmdJG9th5i0w=
+sigs.k8s.io/yaml v1.1.0 h1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs=
+sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o=
--- /dev/null
+/*
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
\ No newline at end of file
--- /dev/null
+/*
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+ "flag"
+ "os"
+
+ "k8s.io/apimachinery/pkg/runtime"
+ clientgoscheme "k8s.io/client-go/kubernetes/scheme"
+ _ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
+ batchv1alpha1 "sdewan.akraino.org/sdewan/api/v1alpha1"
+ "sdewan.akraino.org/sdewan/controllers"
+ ctrl "sigs.k8s.io/controller-runtime"
+ "sigs.k8s.io/controller-runtime/pkg/log/zap"
+ // +kubebuilder:scaffold:imports
+)
+
+var (
+ scheme = runtime.NewScheme()
+ setupLog = ctrl.Log.WithName("setup")
+)
+
+func init() {
+ _ = clientgoscheme.AddToScheme(scheme)
+
+ _ = batchv1alpha1.AddToScheme(scheme)
+ // +kubebuilder:scaffold:scheme
+}
+
+func main() {
+ var metricsAddr string
+ var enableLeaderElection bool
+ flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "The address the metric endpoint binds to.")
+ flag.BoolVar(&enableLeaderElection, "enable-leader-election", false,
+ "Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.")
+ flag.Parse()
+
+ ctrl.SetLogger(zap.New(func(o *zap.Options) {
+ o.Development = true
+ }))
+
+ mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
+ Scheme: scheme,
+ MetricsBindAddress: metricsAddr,
+ LeaderElection: enableLeaderElection,
+ Port: 9443,
+ })
+ if err != nil {
+ setupLog.Error(err, "unable to start manager")
+ os.Exit(1)
+ }
+
+ if err = (&controllers.Mwan3PolicyReconciler{
+ Client: mgr.GetClient(),
+ Log: ctrl.Log.WithName("controllers").WithName("Mwan3Policy"),
+ Scheme: mgr.GetScheme(),
+ }).SetupWithManager(mgr); err != nil {
+ setupLog.Error(err, "unable to create controller", "controller", "Mwan3Policy")
+ os.Exit(1)
+ }
+ // +kubebuilder:scaffold:builder
+
+ setupLog.Info("starting manager")
+ if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
+ setupLog.Error(err, "problem running manager")
+ os.Exit(1)
+ }
+}
--- /dev/null
+package openwrt
+
+import (
+ "encoding/json"
+)
+
+const (
+ firewallBaseURL = "sdewan/firewall/v1/"
+)
+
+type FirewallClient struct {
+ OpenwrtClient *openwrtClient
+}
+
+// Firewall Zones
+type SdewanFirewallZone struct {
+ Name string `json:"name"`
+ Network []string `json:"network"`
+ Masq string `json:"masq"`
+ MasqSrc []string `json:"masq_src"`
+ MasqDest []string `json:"masq_dest"`
+ MasqAllowInvalid string `json:"masq_allow_invalid"`
+ MtuFix string `json:"mtu_fix"`
+ Input string `json:"input"`
+ Forward string `json:"forward"`
+ Output string `json:"output"`
+ Family string `json:"family"`
+ Subnet []string `json:"subnet"`
+ ExtraSrc string `json:"extra_src"`
+ ExtraDest string `json:"etra_dest"`
+}
+
+type SdewanFirewallZones struct {
+ Zones []SdewanFirewallZone `json:"zones"`
+}
+
+// Firewall Forwarding
+type SdewanFirewallForwarding struct {
+ Name string `json:"name"`
+ Src string `json:"src"`
+ Dest string `json:"dest"`
+ Family string `json:"family"`
+}
+
+type SdewanFirewallForwardings struct {
+ Forwardings []SdewanFirewallForwarding `json:"forwardings"`
+}
+
+// Firewall Rule
+type SdewanFirewallRule struct {
+ Name string `json:"name"`
+ Src string `json:"src"`
+ SrcIp string `json:"src_ip"`
+ SrcMac string `json:"src_mac"`
+ SrcPort string `json:"src_port"`
+ Proto string `json:"proto"`
+ IcmpType []string `json:"icmp_type"`
+ Dest string `json:"dest"`
+ DestIp string `json:"dest_ip"`
+ DestPort string `json:"dest_port"`
+ Mark string `json:"mark"`
+ Target string `json:"target"`
+ SetMark string `json:"set_mark"`
+ SetXmark string `json:"set_xmark"`
+ Family string `json:"family"`
+ Extra string `json:"extra"`
+}
+
+type SdewanFirewallRules struct {
+ Rules []SdewanFirewallRule `json:"rules"`
+}
+
+// Firewall Redirect
+type SdewanFirewallRedirect struct {
+ Name string `json:"name"`
+ Src string `json:"src"`
+ SrcIp string `json:"src_ip"`
+ SrcDIp string `json:"src_dip"`
+ SrcMac string `json:"src_mac"`
+ SrcPort string `json:"src_port"`
+ SrcDPort string `json:"src_dport"`
+ Proto string `json:"proto"`
+ Dest string `json:"dest"`
+ DestIp string `json:"dest_ip"`
+ DestPort string `json:"dest_port"`
+ Mark string `json:"mark"`
+ Target string `json:"target"`
+ Family string `json:"family"`
+}
+
+type SdewanFirewallRedirects struct {
+ Redirects []SdewanFirewallRedirect `json:"redirects"`
+}
+
+// Zone APIs
+// get zones
+func (f *FirewallClient) GetZones() (*SdewanFirewallZones, error) {
+ var response string
+ var err error
+ response, err = f.OpenwrtClient.Get(firewallBaseURL + "zones")
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallZones SdewanFirewallZones
+ err = json.Unmarshal([]byte(response), &sdewanFirewallZones)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallZones, nil
+}
+
+// get zone
+func (m *FirewallClient) GetZone(zone string) (*SdewanFirewallZone, error) {
+ var response string
+ var err error
+ response, err = m.OpenwrtClient.Get(firewallBaseURL + "zone/" + zone)
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallZone SdewanFirewallZone
+ err = json.Unmarshal([]byte(response), &sdewanFirewallZone)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallZone, nil
+}
+
+// create zone
+func (m *FirewallClient) CreateZone(zone SdewanFirewallZone) (*SdewanFirewallZone, error) {
+ var response string
+ var err error
+ zone_obj, _ := json.Marshal(zone)
+ response, err = m.OpenwrtClient.Post(firewallBaseURL+"zone", string(zone_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallZone SdewanFirewallZone
+ err = json.Unmarshal([]byte(response), &sdewanFirewallZone)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallZone, nil
+}
+
+// delete zone
+func (m *FirewallClient) DeleteZone(zone_name string) error {
+ _, err := m.OpenwrtClient.Delete(firewallBaseURL + "zone/" + zone_name)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// update zone
+func (m *FirewallClient) UpdateZone(zone SdewanFirewallZone) (*SdewanFirewallZone, error) {
+ var response string
+ var err error
+ zone_obj, _ := json.Marshal(zone)
+ zone_name := zone.Name
+ response, err = m.OpenwrtClient.Put(firewallBaseURL+"zone/"+zone_name, string(zone_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallZone SdewanFirewallZone
+ err = json.Unmarshal([]byte(response), &sdewanFirewallZone)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallZone, nil
+}
+
+// Rule APIs
+// get rules
+func (f *FirewallClient) GetRules() (*SdewanFirewallRules, error) {
+ var response string
+ var err error
+ response, err = f.OpenwrtClient.Get(firewallBaseURL + "rules")
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallRules SdewanFirewallRules
+ err = json.Unmarshal([]byte(response), &sdewanFirewallRules)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallRules, nil
+}
+
+// get rule
+func (m *FirewallClient) GetRule(rule string) (*SdewanFirewallRule, error) {
+ var response string
+ var err error
+ response, err = m.OpenwrtClient.Get(firewallBaseURL + "rule/" + rule)
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallRule SdewanFirewallRule
+ err = json.Unmarshal([]byte(response), &sdewanFirewallRule)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallRule, nil
+}
+
+// create rule
+func (m *FirewallClient) CreateRule(rule SdewanFirewallRule) (*SdewanFirewallRule, error) {
+ var response string
+ var err error
+ rule_obj, _ := json.Marshal(rule)
+ response, err = m.OpenwrtClient.Post(firewallBaseURL+"rule", string(rule_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallRule SdewanFirewallRule
+ err = json.Unmarshal([]byte(response), &sdewanFirewallRule)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallRule, nil
+}
+
+// delete rule
+func (m *FirewallClient) DeleteRule(rule_name string) error {
+ _, err := m.OpenwrtClient.Delete(firewallBaseURL + "rule/" + rule_name)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// update rule
+func (m *FirewallClient) UpdateRule(rule SdewanFirewallRule) (*SdewanFirewallRule, error) {
+ var response string
+ var err error
+ rule_obj, _ := json.Marshal(rule)
+ rule_name := rule.Name
+ response, err = m.OpenwrtClient.Put(firewallBaseURL+"rule/"+rule_name, string(rule_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallRule SdewanFirewallRule
+ err = json.Unmarshal([]byte(response), &sdewanFirewallRule)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallRule, nil
+}
+
+// Forwarding APIs
+// get forwardings
+func (f *FirewallClient) GetForwardings() (*SdewanFirewallForwardings, error) {
+ var response string
+ var err error
+ response, err = f.OpenwrtClient.Get(firewallBaseURL + "forwardings")
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallForwardings SdewanFirewallForwardings
+ err = json.Unmarshal([]byte(response), &sdewanFirewallForwardings)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallForwardings, nil
+}
+
+// get forwarding
+func (m *FirewallClient) GetForwarding(forwarding string) (*SdewanFirewallForwarding, error) {
+ var response string
+ var err error
+ response, err = m.OpenwrtClient.Get(firewallBaseURL + "forwarding/" + forwarding)
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallForwarding SdewanFirewallForwarding
+ err = json.Unmarshal([]byte(response), &sdewanFirewallForwarding)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallForwarding, nil
+}
+
+// create forwarding
+func (m *FirewallClient) CreateForwarding(forwarding SdewanFirewallForwarding) (*SdewanFirewallForwarding, error) {
+ var response string
+ var err error
+ forwarding_obj, _ := json.Marshal(forwarding)
+ response, err = m.OpenwrtClient.Post(firewallBaseURL+"forwarding", string(forwarding_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallForwarding SdewanFirewallForwarding
+ err = json.Unmarshal([]byte(response), &sdewanFirewallForwarding)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallForwarding, nil
+}
+
+// delete forwarding
+func (m *FirewallClient) DeleteForwarding(forwarding_name string) error {
+ _, err := m.OpenwrtClient.Delete(firewallBaseURL + "forwarding/" + forwarding_name)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// update forwarding
+func (m *FirewallClient) UpdateForwarding(forwarding SdewanFirewallForwarding) (*SdewanFirewallForwarding, error) {
+ var response string
+ var err error
+ forwarding_obj, _ := json.Marshal(forwarding)
+ forwarding_name := forwarding.Name
+ response, err = m.OpenwrtClient.Put(firewallBaseURL+"forwarding/"+forwarding_name, string(forwarding_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallForwarding SdewanFirewallForwarding
+ err = json.Unmarshal([]byte(response), &sdewanFirewallForwarding)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallForwarding, nil
+}
+
+// Redirect APIs
+// get redirects
+func (f *FirewallClient) GetRedirects() (*SdewanFirewallRedirects, error) {
+ var response string
+ var err error
+ response, err = f.OpenwrtClient.Get(firewallBaseURL + "redirects")
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallRedirects SdewanFirewallRedirects
+ err = json.Unmarshal([]byte(response), &sdewanFirewallRedirects)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallRedirects, nil
+}
+
+// get redirect
+func (m *FirewallClient) GetRedirect(redirect string) (*SdewanFirewallRedirect, error) {
+ var response string
+ var err error
+ response, err = m.OpenwrtClient.Get(firewallBaseURL + "redirect/" + redirect)
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallRedirect SdewanFirewallRedirect
+ err = json.Unmarshal([]byte(response), &sdewanFirewallRedirect)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallRedirect, nil
+}
+
+// create redirect
+func (m *FirewallClient) CreateRedirect(redirect SdewanFirewallRedirect) (*SdewanFirewallRedirect, error) {
+ var response string
+ var err error
+ redirect_obj, _ := json.Marshal(redirect)
+ response, err = m.OpenwrtClient.Post(firewallBaseURL+"redirect", string(redirect_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallRedirect SdewanFirewallRedirect
+ err = json.Unmarshal([]byte(response), &sdewanFirewallRedirect)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallRedirect, nil
+}
+
+// delete redirect
+func (m *FirewallClient) DeleteRedirect(redirect_name string) error {
+ _, err := m.OpenwrtClient.Delete(firewallBaseURL + "redirect/" + redirect_name)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// update redirect
+func (m *FirewallClient) UpdateRedirect(redirect SdewanFirewallRedirect) (*SdewanFirewallRedirect, error) {
+ var response string
+ var err error
+ redirect_obj, _ := json.Marshal(redirect)
+ redirect_name := redirect.Name
+ response, err = m.OpenwrtClient.Put(firewallBaseURL+"redirect/"+redirect_name, string(redirect_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanFirewallRedirect SdewanFirewallRedirect
+ err = json.Unmarshal([]byte(response), &sdewanFirewallRedirect)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanFirewallRedirect, nil
+}
--- /dev/null
+package openwrt
+
+import (
+ "encoding/json"
+)
+
+const (
+ ipsecBaseURL = "sdewan/ipsec/v1/"
+)
+
+type IpsecClient struct {
+ OpenwrtClient *openwrtClient
+}
+
+// Proposals
+type SdewanIpsecProposal struct {
+ Name string `json:"name"`
+ EncryptionAlgorithm string `json:"encryption_algorithm"`
+ HashAlgorithm string `json:"hash_algorithm"`
+ DhGroup string `json:"dh_group"`
+}
+
+type SdewanIpsecProposals struct {
+ Proposals []SdewanIpsecProposal `json:"proposals"`
+}
+
+// Sites
+type SdewanIpsecConnection struct {
+ Name string `json:"name"`
+ Type string `json:"type"`
+ Mode string `json:"mode"`
+ LocalSubnet string `json:"local_subnet"`
+ LocalNat string `json:"local_nat"`
+ LocalSourceip string `json:"local_sourceip"`
+ LocalUpdown string `json:"local_updown"`
+ LocalFirewall string `json:"local_firewall"`
+ RemoteSubnet string `json:"remote_subnet"`
+ RemoteSourceip string `json:"remote_sourceip"`
+ RemoteUpdown string `json:"remote_updown"`
+ RemoteFirewall string `json:"remote_firewall"`
+ CryptoProposal []string `json:"crypto_proposal"`
+}
+
+type SdewanIpsecSite struct {
+ Name string `json:"name"`
+ Gateway string `json:"gateway"`
+ PreSharedKey string `json:"pre_shared_key"`
+ AuthenticationMethod string `json:"authentication_method"`
+ LocalIdentifier string `json:"local_identifier"`
+ RemoteIdentifier string `json:"remote_identifier"`
+ CryptoProposal []string `json:"crypto_proposal"`
+ ForceCryptoProposal string `json:"force_crypto_proposal"`
+ LocalPublicCert string `json:"local_public_cert"`
+ LocalPrivateCert string `json:"local_private_cert"`
+ SharedCa string `json:"shared_ca"`
+ Connections []SdewanIpsecConnection `json:"connections"`
+}
+
+type SdewanIpsecSites struct {
+ Sites []SdewanIpsecSite `json:"sites"`
+}
+
+// Proposal APIs
+// get proposals
+func (f *IpsecClient) GetProposals() (*SdewanIpsecProposals, error) {
+ var response string
+ var err error
+ response, err = f.OpenwrtClient.Get(ipsecBaseURL + "proposals")
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanIpsecProposals SdewanIpsecProposals
+ err = json.Unmarshal([]byte(response), &sdewanIpsecProposals)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanIpsecProposals, nil
+}
+
+// get proposal
+func (m *IpsecClient) GetProposal(proposal string) (*SdewanIpsecProposal, error) {
+ var response string
+ var err error
+ response, err = m.OpenwrtClient.Get(ipsecBaseURL + "proposal/" + proposal)
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanIpsecProposal SdewanIpsecProposal
+ err = json.Unmarshal([]byte(response), &sdewanIpsecProposal)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanIpsecProposal, nil
+}
+
+// create proposal
+func (m *IpsecClient) CreateProposal(proposal SdewanIpsecProposal) (*SdewanIpsecProposal, error) {
+ var response string
+ var err error
+ proposal_obj, _ := json.Marshal(proposal)
+ response, err = m.OpenwrtClient.Post(ipsecBaseURL+"proposal", string(proposal_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanIpsecProposal SdewanIpsecProposal
+ err = json.Unmarshal([]byte(response), &sdewanIpsecProposal)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanIpsecProposal, nil
+}
+
+// delete proposal
+func (m *IpsecClient) DeleteProposal(proposal_name string) error {
+ _, err := m.OpenwrtClient.Delete(ipsecBaseURL + "proposal/" + proposal_name)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// update proposal
+func (m *IpsecClient) UpdateProposal(proposal SdewanIpsecProposal) (*SdewanIpsecProposal, error) {
+ var response string
+ var err error
+ proposal_obj, _ := json.Marshal(proposal)
+ proposal_name := proposal.Name
+ response, err = m.OpenwrtClient.Put(ipsecBaseURL+"proposal/"+proposal_name, string(proposal_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanIpsecProposal SdewanIpsecProposal
+ err = json.Unmarshal([]byte(response), &sdewanIpsecProposal)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanIpsecProposal, nil
+}
+
+// Site APIs
+// get sites
+func (f *IpsecClient) GetSites() (*SdewanIpsecSites, error) {
+ var response string
+ var err error
+ response, err = f.OpenwrtClient.Get(ipsecBaseURL + "sites")
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanIpsecSites SdewanIpsecSites
+ err = json.Unmarshal([]byte(response), &sdewanIpsecSites)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanIpsecSites, nil
+}
+
+// get site
+func (m *IpsecClient) GetSite(site string) (*SdewanIpsecSite, error) {
+ var response string
+ var err error
+ response, err = m.OpenwrtClient.Get(ipsecBaseURL + "site/" + site)
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanIpsecSite SdewanIpsecSite
+ err = json.Unmarshal([]byte(response), &sdewanIpsecSite)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanIpsecSite, nil
+}
+
+// create site
+func (m *IpsecClient) CreateSite(site SdewanIpsecSite) (*SdewanIpsecSite, error) {
+ var response string
+ var err error
+ site_obj, _ := json.Marshal(site)
+ response, err = m.OpenwrtClient.Post(ipsecBaseURL+"site", string(site_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanIpsecSite SdewanIpsecSite
+ err = json.Unmarshal([]byte(response), &sdewanIpsecSite)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanIpsecSite, nil
+}
+
+// delete site
+func (m *IpsecClient) DeleteSite(site_name string) error {
+ _, err := m.OpenwrtClient.Delete(ipsecBaseURL + "site/" + site_name)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// update site
+func (m *IpsecClient) UpdateSite(site SdewanIpsecSite) (*SdewanIpsecSite, error) {
+ var response string
+ var err error
+ site_obj, _ := json.Marshal(site)
+ site_name := site.Name
+ response, err = m.OpenwrtClient.Put(ipsecBaseURL+"site/"+site_name, string(site_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanIpsecSite SdewanIpsecSite
+ err = json.Unmarshal([]byte(response), &sdewanIpsecSite)
+ if err != nil {
+ return nil, err
+ }
+
+ return &sdewanIpsecSite, nil
+}
--- /dev/null
+package openwrt
+
+import (
+ "encoding/json"
+)
+
+const (
+ mwan3BaseURL = "sdewan/mwan3/v1/"
+)
+
+type Mwan3Client struct {
+ OpenwrtClient *openwrtClient
+}
+
+// MWAN3 interface status
+type IpStatus struct {
+ Status string `json:"status"`
+ Latency int `json:"latency"`
+ Packetloss int `json:"packetloss"`
+ Ip string `json:"ip"`
+}
+
+type WanInterfaceStatus struct {
+ Running bool `json:"running"`
+ Score int `json:"score"`
+ Lost int `json:"lost"`
+ Status string `json:"status"`
+ Age int `json:"age"`
+ Turn int `json:"turn"`
+ Ips []IpStatus `json:"track_ip"`
+}
+
+type InterfaceStatus struct {
+ Interfaces map[string]WanInterfaceStatus `json:"interfaces"`
+ Connected map[string][]string `json:"connected"`
+}
+
+// MWAN3 Policy
+type SdewanMember struct {
+ Interface string `json:"interface"`
+ Metric string `json:"metric"`
+ Weight string `json:"weight"`
+}
+
+type SdewanPolicy struct {
+ Name string `json:"name"`
+ Members []SdewanMember `json:"members"`
+}
+
+type SdewanPolicies struct {
+ Policies []SdewanPolicy `json:"policies"`
+}
+
+// MWAN3 Rule
+type SdewanRule struct {
+ Name string `json:"name"`
+ Policy string `json:"policy"`
+ SrcIp string `json:"src_ip"`
+ SrcPort string `json:"src_port"`
+ DestIp string `json:"dest_ip"`
+ DestPort string `json:"dest_port"`
+ Proto string `json:"proto"`
+ Family string `json:"family"`
+ Sticky string `json:"sticky"`
+ Timeout string `json:"timeout"`
+}
+
+type SdewanRules struct {
+ Rules []SdewanRule `json:"rules"`
+}
+
+// get interface status
+func (m *Mwan3Client) GetInterfaceStatus() (*InterfaceStatus, error) {
+ response, err := m.OpenwrtClient.Get("admin/status/mwan/interface_status")
+ if err != nil {
+ return nil, err
+ }
+
+ var interfaceStatus InterfaceStatus
+ err2 := json.Unmarshal([]byte(response), &interfaceStatus)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &interfaceStatus, nil
+}
+
+// Policy APIs
+// get policies
+func (m *Mwan3Client) GetPolicies() (*SdewanPolicies, error) {
+ response, err := m.OpenwrtClient.Get(mwan3BaseURL + "policies")
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanPolicies SdewanPolicies
+ err2 := json.Unmarshal([]byte(response), &sdewanPolicies)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &sdewanPolicies, nil
+}
+
+// get policy
+func (m *Mwan3Client) GetPolicy(policy_name string) (*SdewanPolicy, error) {
+ response, err := m.OpenwrtClient.Get(mwan3BaseURL + "policies/" + policy_name)
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanPolicy SdewanPolicy
+ err2 := json.Unmarshal([]byte(response), &sdewanPolicy)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &sdewanPolicy, nil
+}
+
+// create policy
+func (m *Mwan3Client) CreatePolicy(policy SdewanPolicy) (*SdewanPolicy, error) {
+ policy_obj, _ := json.Marshal(policy)
+ response, err := m.OpenwrtClient.Post(mwan3BaseURL+"policies", string(policy_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanPolicy SdewanPolicy
+ err2 := json.Unmarshal([]byte(response), &sdewanPolicy)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &sdewanPolicy, nil
+}
+
+// delete policy
+func (m *Mwan3Client) DeletePolicy(policy_name string) error {
+ _, err := m.OpenwrtClient.Delete(mwan3BaseURL + "policies/" + policy_name)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// update policy
+func (m *Mwan3Client) UpdatePolicy(policy SdewanPolicy) (*SdewanPolicy, error) {
+ policy_obj, _ := json.Marshal(policy)
+ policy_name := policy.Name
+ response, err := m.OpenwrtClient.Put(mwan3BaseURL+"policies/"+policy_name, string(policy_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanPolicy SdewanPolicy
+ err2 := json.Unmarshal([]byte(response), &sdewanPolicy)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &sdewanPolicy, nil
+}
+
+// Rule APIs
+// get rules
+func (m *Mwan3Client) GetRules() (*SdewanRules, error) {
+ response, err := m.OpenwrtClient.Get(mwan3BaseURL + "rules")
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanRules SdewanRules
+ err2 := json.Unmarshal([]byte(response), &sdewanRules)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &sdewanRules, nil
+}
+
+// get rule
+func (m *Mwan3Client) GetRule(rule string) (*SdewanRule, error) {
+ response, err := m.OpenwrtClient.Get(mwan3BaseURL + "rules/" + rule)
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanRule SdewanRule
+ err2 := json.Unmarshal([]byte(response), &sdewanRule)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &sdewanRule, nil
+}
+
+// create rule
+func (m *Mwan3Client) CreateRule(rule SdewanRule) (*SdewanRule, error) {
+ rule_obj, _ := json.Marshal(rule)
+ response, err := m.OpenwrtClient.Post(mwan3BaseURL+"rules", string(rule_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanRule SdewanRule
+ err2 := json.Unmarshal([]byte(response), &sdewanRule)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &sdewanRule, nil
+}
+
+// delete rule
+func (m *Mwan3Client) DeleteRule(rule_name string) error {
+ _, err := m.OpenwrtClient.Delete(mwan3BaseURL + "rules/" + rule_name)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// update rule
+func (m *Mwan3Client) UpdateRule(rule SdewanRule) (*SdewanRule, error) {
+ rule_obj, _ := json.Marshal(rule)
+ rule_name := rule.Name
+ response, err := m.OpenwrtClient.Put(mwan3BaseURL+"rules/"+rule_name, string(rule_obj))
+ if err != nil {
+ return nil, err
+ }
+
+ var sdewanRule SdewanRule
+ err2 := json.Unmarshal([]byte(response), &sdewanRule)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &sdewanRule, nil
+}
--- /dev/null
+package openwrt
+
+import (
+ "bytes"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "runtime"
+ "strings"
+)
+
+type OpenwrtError struct {
+ Code int
+ Message string
+}
+
+func (e *OpenwrtError) Error() string {
+ return fmt.Sprintf("Error Code: %d, Error Message: %s", e.Code, e.Message)
+}
+
+type openwrtClient struct {
+ ip string
+ user string
+ password string
+ token string
+}
+
+func CloseClient(o *openwrtClient) {
+ o.logout()
+ runtime.SetFinalizer(o, nil)
+}
+
+func NewOpenwrtClient(ip string, user string, password string) *openwrtClient {
+ client := &openwrtClient{
+ ip: ip,
+ user: user,
+ password: password,
+ token: "",
+ }
+
+ runtime.SetFinalizer(client, CloseClient)
+ return client
+}
+
+// openwrt base URL
+func (o *openwrtClient) getBaseURL() string {
+ return "http://" + o.ip + "/cgi-bin/luci/"
+}
+
+// login to openwrt http server
+func (o *openwrtClient) login() error {
+ client := &http.Client{
+ // block redirect
+ CheckRedirect: func(req *http.Request, via []*http.Request) error {
+ return http.ErrUseLastResponse
+ },
+ }
+
+ // login
+ login_info := "luci_username=" + o.user + "&luci_password=" + o.password
+ var req_body = []byte(login_info)
+ req, _ := http.NewRequest("POST", o.getBaseURL(), bytes.NewBuffer(req_body))
+ req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
+
+ resp, err := client.Do(req)
+ if resp != nil {
+ defer resp.Body.Close()
+ }
+
+ if err != nil {
+ return err
+ } else if resp.StatusCode != 302 {
+ // fail to auth
+ return &OpenwrtError{Code: resp.StatusCode, Message: "Unauthorized"}
+ } else {
+ // get token
+ res_cookie := resp.Header["Set-Cookie"][0]
+ res_cookies := strings.Split(res_cookie, ";")
+ for _, cookie := range res_cookies {
+ cookie := strings.TrimSpace(cookie)
+ index := strings.Index(cookie, "=")
+ var key = cookie
+ var value = ""
+ if index != -1 {
+ key = cookie[:index]
+ value = cookie[index+1:]
+ }
+
+ if key == "sysauth" {
+ o.token = value
+ break
+ }
+ }
+ }
+
+ return nil
+}
+
+// logout to openwrt http server
+func (o *openwrtClient) logout() error {
+ if o.token != "" {
+ _, err := o.Get("admin/logout")
+ o.token = ""
+ return err
+ }
+
+ return nil
+}
+
+// call openwrt restful API
+func (o *openwrtClient) call(method string, url string, request string) (string, error) {
+ for i := 0; i < 2; i++ {
+ if o.token == "" {
+ err := o.login()
+ if err != nil {
+ return "", err
+ }
+ }
+
+ client := &http.Client{}
+ req_body := bytes.NewBuffer([]byte(request))
+ req, _ := http.NewRequest(method, o.getBaseURL()+url, req_body)
+ req.Header.Add("Cookie", "sysauth="+o.token)
+ resp, err := client.Do(req)
+ if err != nil {
+ return "", err
+ }
+ defer resp.Body.Close()
+
+ body, _ := ioutil.ReadAll(resp.Body)
+ if resp.StatusCode >= 400 {
+ if resp.StatusCode == 403 {
+ // token expired, retry
+ o.token = ""
+ continue
+ } else {
+ // error request
+ return "", &OpenwrtError{Code: resp.StatusCode, Message: string(body)}
+ }
+ }
+
+ return string(body), nil
+ }
+
+ return "", nil
+}
+
+// call openwrt Get restful API
+func (o *openwrtClient) Get(url string) (string, error) {
+ return o.call("GET", url, "")
+}
+
+// call openwrt restful API
+func (o *openwrtClient) Post(url string, request string) (string, error) {
+ return o.call("POST", url, request)
+}
+
+// call openwrt restful API
+func (o *openwrtClient) Put(url string, request string) (string, error) {
+ return o.call("PUT", url, request)
+}
+
+// call openwrt restful API
+func (o *openwrtClient) Delete(url string) (string, error) {
+ return o.call("DELETE", url, "")
+}
--- /dev/null
+package openwrt
+
+import (
+ "encoding/json"
+)
+
+const (
+ serviceBaseURL = "sdewan/v1/"
+)
+
+var available_Services = []string{"mwan3", "firewall", "ipsec"}
+
+type ServiceClient struct {
+ OpenwrtClient *openwrtClient
+}
+
+// Service API struct
+type AvailableServices struct {
+ Services []string `json:"services"`
+}
+
+// get available services
+func (s *ServiceClient) GetAvailableServices() (*AvailableServices, error) {
+ response, err := s.OpenwrtClient.Get(serviceBaseURL + "services")
+ if err != nil {
+ return nil, err
+ }
+
+ var servs AvailableServices
+ err2 := json.Unmarshal([]byte(response), &servs)
+ if err2 != nil {
+ return nil, err2
+ }
+
+ return &servs, nil
+}
+
+func (s *ServiceClient) formatExecuteServiceBody(operation string) string {
+ return "{\"action\":\"" + operation + "\"}"
+}
+
+// execute operation on service
+func (s *ServiceClient) ExecuteService(service string, operation string) (bool, error) {
+ if !IsContained(available_Services, service) {
+ return false, &OpenwrtError{Code: 400, Message: "Bad Request: not supported service(" + service + ")"}
+ }
+
+ _, err := s.OpenwrtClient.Put(serviceBaseURL+"service/"+service, s.formatExecuteServiceBody(operation))
+ if err != nil {
+ return false, err
+ }
+
+ return true, nil
+}
--- /dev/null
+package openwrt
+
+import (
+ "reflect"
+)
+
+// util function to check whether items contains item
+func IsContained(items interface{}, item interface{}) bool {
+ switch reflect.TypeOf(items).Kind() {
+ case reflect.Slice:
+ v := reflect.ValueOf(items)
+ for i := 0; i < v.Len(); i++ {
+ if reflect.DeepEqual(item, v.Index(i).Interface()) {
+ return true
+ }
+ }
+ default:
+ return false
+ }
+
+ return false
+}
Code Review / icn / sdwan.git / commitdiff
