--- /dev/null
+# Multus/SRIOV CNI&Device Plugin Support for IEC
+
+
+## Introduction
+This commit provides Kubernetes networking support for Multus with SRIOV CNI support both on arm64 and amd64. A special configuration file for Broadcom smartNIC Stingray PS225 is provided as an example. So it can not only be used as a sample container networking support for Stingray PS225 with its sriov interfaces, but also a generic SRIOV support for various ethernet NICs.
+Here we would like to provide 2 types legacy CNIs besides the SRIOV-CNI by Multus, which are [Flannel](https://coreos.com/flannel/docs/latest/kubernetes.html), and [Calico](https://docs.projectcalico.org/v3.11/introduction/).
+They would be provided as the default CNIs for any pods without explicit annotations.
+
+The work here is based on the following open source projects:
+1. [Multus](https://github.com/intel/multus-cni)
+1. [SRIOV Device Plugin](https://github.com/intel/sriov-network-device-plugin)
+1. [SRIOV CNI](https://github.com/intel/sriov-cni)
+
+For Broadcom Stingray PS225, please refer its [documents](https://github.com/CCX-Stingray/Documentation).
+
+
+## Initial setup
+
+The SRIOV interfaces should be created before using the SRIOV CNI, for example, if the PF name of one of the Stingray PS225 ethernet NICs in your system is enp8s0f0np0, then you can lookup the maximum supported number of VFs under the PF and create the VFs with the command:
+
+```
+#ip link set enp8s0f0np0 up
+#cat /sys/class/net/enp8s0f0np0/device/sriov_totalvfs
+16
+#echo 16 > /sys/class/net/enp8s0f0np0/device/sriov_numvfs
+```
+For SRIOV CNI with DPDK type drivers, such as vfio-pci, uio_pci_generic, please bind the driver by dpdk-devbind besides creating the VFs, for example:
+```dpdk-devbind.py -b vfio-pci enp12s2```
+The `enp12s2` is the name of one of VFs of a ethernet NIC.
+
+For more information, please refer the above links in the [Introduction](#Introduction).
+
+##Installation
+
+There are 4 yaml files give:
+1. configMap.yaml:
+The resource list configuration file for SRIOV device plugin
+1. multus-sriov-flannel-daemonsets.yaml
+The Multus, SRIOV device plugin&CNI configuration file
+1. flannel-daemonset.yml
+The Flannel CNI installation file
+1. sriov-crd.yaml
+The SRIOV CNI configuration file for the attached SRIOV interface resource.
+
+Usually users should modify the `configMap.yaml` and `sriov-crd.yaml` with their own corresponding networking configuration before doing the installation.
+
+A quick installation script is given as `install.sh`, and the uninstallation could be done by call the `uninstall.sh`.
+
+**The `install.sh` should be called after the Kubernetes cluster had been installed but before installing the CNIs.**
+
+2 sample user pods yaml files using the SRIOV CNI are given also, please see `pod1.yaml` and `pod2.yaml` in the directory, and 2 iperfv2 based yaml files are also provided to facilitate the performance test. We choose IPerfv2 instead of the IPerfv3 because it seems can support multi-threaded sending/receiving. The 4 yaml files here are provided just for arm64 platform and they could easily be adapted to amd64 platform with corresponding images with a docker building based on the provided Dockerfile.iperf2 file.
+
+
+## Notes
+Current installation files are suitable for K8s version less than 1.6, for that equal to or greater than 1.6, some modifications to the installation files should be done, which would adapt to the K8s api changes for newer versions.
+We would try to give a sample installation file for k8s 1.6 or above in the future, but it had not tested now.
+
+The command `kubectl get node $(hostname) -o json | jq '.status.allocatable'` can be used to check whether the allocated resources are available to use or not.
+
+## Other References
+[Advanced Networking Features in
+Kubernetes and Container Bare Metal](https://builders.intel.com/docs/networkbuilders/adv-network-features-in-kubernetes-app-note.pdf)
+
+[Kubernetes: Multus + SRIOV quickstart](https://zshisite.wordpress.com/2018/11/15/kubernetes-multus-sriov-quickstart/)
+
--- /dev/null
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: sriovdp-config
+ namespace: kube-system
+data:
+ config.json: |
+ {
+ "resourceList": [{
+ "resourceName": "ps225_sriov_netdevice",
+ "selectors": {
+ "vendors": ["14e4"],
+ "devices": ["d800"],
+ "drivers": ["bnxt_en"],
+ "pfNames": ["enp8s0f0np0"]
+ }
+ },
+ {
+ "resourceName": "intel_sriov_netdevice",
+ "selectors": {
+ "vendors": ["8086"],
+ "devices": ["154c"],
+ "drivers": ["i40evf"],
+ "pfNames": ["enp12s0f0"]
+ }
+ },
+ {
+ "resourceName": "intel_sriov_dpdk",
+ "selectors": {
+ "vendors": ["8086"],
+ "devices": ["154c"],
+ "drivers": ["vfio-pci"],
+ "pfNames": ["enp12s0f1"]
+ }
+ }
+ ]
+ }
--- /dev/null
+# yamllint disable
+# This is a modified Flannel daemonset.
+# it is based on: https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
+# Notably, it removes the creation of an configuration file in/etc/cni/net.d/
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+ name: flannel
+rules:
+ - apiGroups:
+ - ""
+ resources:
+ - pods
+ verbs:
+ - get
+ - apiGroups:
+ - ""
+ resources:
+ - nodes
+ verbs:
+ - list
+ - watch
+ - apiGroups:
+ - ""
+ resources:
+ - nodes/status
+ verbs:
+ - patch
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+ name: flannel
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: flannel
+subjects:
+ - kind: ServiceAccount
+ name: flannel
+ namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: flannel
+ namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+ name: kube-flannel-cfg
+ namespace: kube-system
+ labels:
+ tier: node
+ app: flannel
+data:
+ # ------------------------------- Intentionally removed, Multus daemonset configures /etc/cni/net.d
+ #cni-conf.json: |
+ # {
+ # "name": "cbr0",
+ # "plugins": [
+ # {
+ # "type": "flannel",
+ # "delegate": {
+ # "hairpinMode": true,
+ # "isDefaultGateway": true
+ # }
+ # },
+ # {
+ # "type": "portmap",
+ # "capabilities": {
+ # "portMappings": true
+ # }
+ # }
+ # ]
+ # }
+ net-conf.json: |
+ {
+ "Network": "10.244.0.0/16",
+ "Backend": {
+ "Type": "vxlan",
+ "Port": 18989
+ }
+ }
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-flannel-ds-amd64
+ namespace: kube-system
+ labels:
+ tier: node
+ app: flannel
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: flannel
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: amd64
+ tolerations:
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: flannel
+ # ------------------------------- Intentionally removed, Multus daemonset configures /etc/cni/net.d
+ # initContainers:
+ # - name: install-cni
+ # image: quay.io/coreos/flannel:v0.10.0-amd64
+ # command:
+ # - cp
+ # args:
+ # - -f
+ # - /etc/kube-flannel/cni-conf.json
+ # - /etc/cni/net.d/10-flannel.conflist
+ # volumeMounts:
+ # - name: cni
+ # mountPath: /etc/cni/net.d
+ # - name: flannel-cfg
+ # mountPath: /etc/kube-flannel/
+ containers:
+ - name: kube-flannel
+ image: quay.io/coreos/flannel:v0.10.0-amd64
+ imagePullPolicy: IfNotPresent
+ command:
+ - /opt/bin/flanneld
+ args:
+ - --ip-masq
+ - --kube-subnet-mgr
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ securityContext:
+ privileged: true
+ env:
+ - name: POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
+ volumeMounts:
+ - name: run
+ mountPath: /run
+ - name: flannel-cfg
+ mountPath: /etc/kube-flannel/
+ volumes:
+ - name: run
+ hostPath:
+ path: /run
+ - name: cni
+ hostPath:
+ path: /etc/cni/net.d
+ - name: flannel-cfg
+ configMap:
+ name: kube-flannel-cfg
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-flannel-ds-arm64
+ namespace: kube-system
+ labels:
+ tier: node
+ app: flannel
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: flannel
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: arm64
+ tolerations:
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: flannel
+ # ------------------------------- Intentionally removed, Multus daemonset configures /etc/cni/net.d
+ # initContainers:
+ # - name: install-cni
+ # image: quay.io/coreos/flannel:v0.10.0-arm64
+ # command:
+ # - cp
+ # args:
+ # - -f
+ # - /etc/kube-flannel/cni-conf.json
+ # - /etc/cni/net.d/10-flannel.conflist
+ # volumeMounts:
+ # - name: cni
+ # mountPath: /etc/cni/net.d
+ # - name: flannel-cfg
+ # mountPath: /etc/kube-flannel/
+ containers:
+ - name: kube-flannel
+ image: quay.io/coreos/flannel:v0.10.0-arm64
+ command:
+ - /opt/bin/flanneld
+ args:
+ - --ip-masq
+ - --kube-subnet-mgr
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ securityContext:
+ privileged: true
+ env:
+ - name: POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
+ volumeMounts:
+ - name: run
+ mountPath: /run
+ - name: flannel-cfg
+ mountPath: /etc/kube-flannel/
+ volumes:
+ - name: run
+ hostPath:
+ path: /run
+ - name: cni
+ hostPath:
+ path: /etc/cni/net.d
+ - name: flannel-cfg
+ configMap:
+ name: kube-flannel-cfg
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-flannel-ds-arm
+ namespace: kube-system
+ labels:
+ tier: node
+ app: flannel
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: flannel
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: arm
+ tolerations:
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: flannel
+ initContainers:
+ - name: install-cni
+ image: quay.io/coreos/flannel:v0.10.0-arm
+ command:
+ - cp
+ args:
+ - -f
+ - /etc/kube-flannel/cni-conf.json
+ - /etc/cni/net.d/10-flannel.conflist
+ volumeMounts:
+ - name: cni
+ mountPath: /etc/cni/net.d
+ - name: flannel-cfg
+ mountPath: /etc/kube-flannel/
+ containers:
+ - name: kube-flannel
+ image: quay.io/coreos/flannel:v0.10.0-arm
+ command:
+ - /opt/bin/flanneld
+ args:
+ - --ip-masq
+ - --kube-subnet-mgr
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ securityContext:
+ privileged: true
+ env:
+ - name: POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
+ volumeMounts:
+ - name: run
+ mountPath: /run
+ - name: flannel-cfg
+ mountPath: /etc/kube-flannel/
+ volumes:
+ - name: run
+ hostPath:
+ path: /run
+ - name: cni
+ hostPath:
+ path: /etc/cni/net.d
+ - name: flannel-cfg
+ configMap:
+ name: kube-flannel-cfg
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-flannel-ds-ppc64le
+ namespace: kube-system
+ labels:
+ tier: node
+ app: flannel
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: flannel
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: ppc64le
+ tolerations:
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: flannel
+ initContainers:
+ - name: install-cni
+ image: quay.io/coreos/flannel:v0.10.0-ppc64le
+ command:
+ - cp
+ args:
+ - -f
+ - /etc/kube-flannel/cni-conf.json
+ - /etc/cni/net.d/10-flannel.conflist
+ volumeMounts:
+ - name: cni
+ mountPath: /etc/cni/net.d
+ - name: flannel-cfg
+ mountPath: /etc/kube-flannel/
+ containers:
+ - name: kube-flannel
+ image: quay.io/coreos/flannel:v0.10.0-ppc64le
+ command:
+ - /opt/bin/flanneld
+ args:
+ - --ip-masq
+ - --kube-subnet-mgr
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ securityContext:
+ privileged: true
+ env:
+ - name: POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
+ volumeMounts:
+ - name: run
+ mountPath: /run
+ - name: flannel-cfg
+ mountPath: /etc/kube-flannel/
+ volumes:
+ - name: run
+ hostPath:
+ path: /run
+ - name: cni
+ hostPath:
+ path: /etc/cni/net.d
+ - name: flannel-cfg
+ configMap:
+ name: kube-flannel-cfg
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-flannel-ds-s390x
+ namespace: kube-system
+ labels:
+ tier: node
+ app: flannel
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: flannel
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: s390x
+ tolerations:
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: flannel
+ initContainers:
+ - name: install-cni
+ image: quay.io/coreos/flannel:v0.10.0-s390x
+ command:
+ - cp
+ args:
+ - -f
+ - /etc/kube-flannel/cni-conf.json
+ - /etc/cni/net.d/10-flannel.conflist
+ volumeMounts:
+ - name: cni
+ mountPath: /etc/cni/net.d
+ - name: flannel-cfg
+ mountPath: /etc/kube-flannel/
+ containers:
+ - name: kube-flannel
+ image: quay.io/coreos/flannel:v0.10.0-s390x
+ command:
+ - /opt/bin/flanneld
+ args:
+ - --ip-masq
+ - --kube-subnet-mgr
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ securityContext:
+ privileged: true
+ env:
+ - name: POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
+ volumeMounts:
+ - name: run
+ mountPath: /run
+ - name: flannel-cfg
+ mountPath: /etc/kube-flannel/
+ volumes:
+ - name: run
+ hostPath:
+ path: /run
+ - name: cni
+ hostPath:
+ path: /etc/cni/net.d
+ - name: flannel-cfg
+ configMap:
+ name: kube-flannel-cfg
--- /dev/null
+#!/bin/bash -ex
+# shellcheck disable=SC2016,SC2046
+
+#SCRIPTS_DIR=$(dirname "${BASH_SOURCE[0]}")
+
+function wait_for {
+ # Execute in a subshell to prevent local variable override during recursion
+ (
+ local total_attempts=$1; shift
+ local cmdstr=$*
+ local sleep_time=2
+ echo -e "\n[wait_for] Waiting for cmd to return success: ${cmdstr}"
+ # shellcheck disable=SC2034
+ for attempt in $(seq "${total_attempts}"); do
+ echo "[wait_for] Attempt ${attempt}/${total_attempts%.*} for: ${cmdstr}"
+ # shellcheck disable=SC2015
+ eval "${cmdstr}" && echo "[wait_for] OK: ${cmdstr}" && return 0 || true
+ sleep "${sleep_time}"
+ done
+ echo "[wait_for] ERROR: Failed after max attempts: ${cmdstr}"
+ return 1
+ )
+}
+
+
+kubectl create -f configMap.yaml
+wait_for 5 'test $(kubectl get configmap -n kube-system | grep sriovdp-config -c ) -eq 1'
+
+kubectl create -f multus-sriov-flannel-daemonsets.yaml
+wait_for 100 'test $(kubectl get pods -n kube-system | grep -e "kube-multus-ds" | grep "Running" -c) -ge 1'
+wait_for 20 'test $(kubectl get pods -n kube-system | grep -e "kube-sriov-cni" | grep "Running" -c) -ge 1'
+wait_for 20 'test $(kubectl get pods -n kube-system | grep -e "kube-sriov-device-plugin" | grep "Running" -c) -ge 1'
+
+kubectl create -f flannel-daemonset.yml
+wait_for 100 'test $(kubectl get pods -n kube-system | grep -e "kube-flannel-ds" | grep "Running" -c) -ge 1'
+
+kubectl create -f sriov-crd.yaml
+wait_for 5 'test $(kubectl get crd | grep -e "network-attachment-definitions" -c) -ge 1'
+
+
+kubectl get node $(hostname) -o json | jq '.status.allocatable'
--- /dev/null
+# yamllint disable
+# This yaml file contains necessary configuration to setup
+# a demo environment for Multus + SR-IOV, the config includes
+# the following pieces:
+# 1. Multus ConfigMap
+# 2. Network Plumbing Working Group Spec Version 1 CustomerResourceDefinition
+# 3. Multus ClusterRole & ClusterRoleBinding
+# 4. Multus & SR-IOV Device Plugin ServiceAccounts
+# 5. Multus & SR-IOV Device Plugin & SR-IOV CNI DaemonSets
+
+# Note: This yaml file will not create customer SR-IOV CRD
+# which will be specified in Pod spec annotation. Below is
+# an example of SR-IOV CRD:
+#
+# apiVersion: "k8s.cni.cncf.io/v1"
+# kind: NetworkAttachmentDefinition
+# metadata:
+# name: sriov-net1
+# annotations:
+# k8s.v1.cni.cncf.io/resourceName: intel.com/sriov
+# spec:
+# config: '{
+# "type": "sriov",
+# "name": "sriov-network",
+# "ipam": {
+# "type": "host-local",
+# "subnet": "10.56.217.0/24",
+# "routes": [{
+# "dst": "0.0.0.0/0"
+# }],
+# "gateway": "10.56.217.1"
+# }
+# }'
+
+# An example of Pod spec using above SR-IOV CRD:
+#
+# apiVersion: v1
+# kind: Pod
+# metadata:
+# name: testpod1
+# labels:
+# env: test
+# annotations:
+# k8s.v1.cni.cncf.io/networks: sriov-net1
+# spec:
+# containers:
+# - name: appcntr1
+# image: centos/tools
+# imagePullPolicy: IfNotPresent
+# command: [ "/bin/bash", "-c", "--" ]
+# args: [ "while true; do sleep 300000; done;" ]
+# resources:
+# requests:
+# intel.com/sriov: '1'
+# limits:
+# intel.com/sriov: '1'
+
+
+# --------------------------------------------------------------------
+
+# 1. Multus ConfigMap
+#
+# This configMap assumes that:
+# - Kubeconfig file is located at "/etc/kubernetes/admin.conf" on host
+# - Default master plugin for Multus is set to flannel
+#
+# Note: If either of above is not True in your environment
+# make sure they are properly set to the corrent values.
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+ name: multus-cni-config
+ namespace: kube-system
+ labels:
+ tier: node
+ app: multus
+data:
+ cni-conf.json: |
+ {
+ "name": "multus-cni-network",
+ "type": "multus",
+ "capabilities": {
+ "portMappings": true
+ },
+ "delegates": [
+ {
+ "cniVersion": "0.3.1",
+ "name": "default-cni-network",
+ "plugins": [
+ {
+ "type": "flannel",
+ "name": "flannel.1",
+ "delegate": {
+ "isDefaultGateway": true,
+ "hairpinMode": true
+ }
+ },
+ {
+ "type": "portmap",
+ "capabilities": {
+ "portMappings": true
+ }
+ }
+ ]
+ }
+ ],
+ "kubeconfig": "/etc/kubernetes/admin.conf"
+ }
+
+
+# 2. NPWG spec v1 Network Attachment Definition
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+ name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+ group: k8s.cni.cncf.io
+ version: v1
+ scope: Namespaced
+ names:
+ plural: network-attachment-definitions
+ singular: network-attachment-definition
+ kind: NetworkAttachmentDefinition
+ shortNames:
+ - net-attach-def
+ validation:
+ openAPIV3Schema:
+ properties:
+ spec:
+ properties:
+ config:
+ type: string
+
+
+# 3.1 Multus Cluster Role
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+ name: multus
+rules:
+ - apiGroups: ["k8s.cni.cncf.io"]
+ resources:
+ - '*'
+ verbs:
+ - '*'
+ - apiGroups:
+ - ""
+ resources:
+ - pods
+ - pods/status
+ verbs:
+ - get
+ - update
+
+# 3.2 Multus Cluster Role Binding
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+ name: multus
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: multus
+subjects:
+- kind: ServiceAccount
+ name: multus
+ namespace: kube-system
+
+# 4.1 SR-IOV Device Plugin ServiceAccount
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: sriov-device-plugin
+ namespace: kube-system
+
+# 4.2 Multus ServiceAccount
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: multus
+ namespace: kube-system
+
+# 5.1 SR-IOV Device Plugin DaemonSet
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-sriov-device-plugin-amd64
+ namespace: kube-system
+ labels:
+ tier: node
+ app: sriovdp
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: sriovdp
+ spec:
+ hostNetwork: true
+ hostPID: true
+ nodeSelector:
+ beta.kubernetes.io/arch: amd64
+ tolerations:
+ #- key: node-role.kubernetes.io/master
+ # operator: Exists
+ # effect: NoSchedule
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: sriov-device-plugin
+ containers:
+ - name: kube-sriovdp
+ image: nfvpe/sriov-device-plugin
+ imagePullPolicy: IfNotPresent
+ args:
+ - --log-dir=sriovdp
+ - --log-level=10
+ - --resource-prefix=arm.com
+ securityContext:
+ privileged: true
+ volumeMounts:
+ - name: devicesock
+ mountPath: /var/lib/kubelet/
+ readOnly: false
+ - name: log
+ mountPath: /var/log
+ - name: config-volume
+ mountPath: /etc/pcidp
+ volumes:
+ - name: devicesock
+ hostPath:
+ path: /var/lib/kubelet/
+ - name: log
+ hostPath:
+ path: /var/log
+ - name: config-volume
+ configMap:
+ name: sriovdp-config
+ items:
+ - key: config.json
+ path: config.json
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-sriov-device-plugin-arm64
+ namespace: kube-system
+ labels:
+ tier: node
+ app: sriovdp
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: sriovdp
+ spec:
+ hostNetwork: true
+ hostPID: true
+ nodeSelector:
+ beta.kubernetes.io/arch: arm64
+ tolerations:
+ #- key: node-role.kubernetes.io/master
+ # operator: Exists
+ # effect: NoSchedule
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: sriov-device-plugin
+ containers:
+ - name: kube-sriovdp
+ #image: nfvpe/sriov-device-plugin
+ image: iecedge/sriov-device-plugin-arm64
+ imagePullPolicy: IfNotPresent
+ #imagePullPolicy: Never
+ args:
+ - --log-dir=sriovdp
+ - --log-level=10
+ - --resource-prefix=arm.com
+ securityContext:
+ privileged: true
+ volumeMounts:
+ - name: devicesock
+ mountPath: /var/lib/kubelet/
+ readOnly: false
+ - name: log
+ mountPath: /var/log
+ - name: config-volume
+ mountPath: /etc/pcidp
+ volumes:
+ - name: devicesock
+ hostPath:
+ path: /var/lib/kubelet/
+ - name: log
+ hostPath:
+ path: /var/log
+ - name: config-volume
+ configMap:
+ name: sriovdp-config
+ items:
+ - key: config.json
+ path: config.json
+
+# 5.2 SR-IOV CNI DaemonSet
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-sriov-cni-ds-amd64
+ namespace: kube-system
+ labels:
+ tier: node
+ app: sriov-cni
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: sriov-cni
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: amd64
+ tolerations:
+ - key: node-role.kubernetes.io/master
+ operator: Exists
+ effect: NoSchedule
+ containers:
+ - name: kube-sriov-cni
+ image: nfvpe/sriov-cni:latest
+ imagePullPolicy: IfNotPresent
+ securityContext:
+ privileged: true
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ volumeMounts:
+ - name: cnibin
+ mountPath: /host/opt/cni/bin
+ volumes:
+ - name: cnibin
+ hostPath:
+ path: /opt/cni/bin
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+ name: kube-sriov-cni-ds-arm64
+ namespace: kube-system
+ labels:
+ tier: node
+ app: sriov-cni
+spec:
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: sriov-cni
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: arm64
+ tolerations:
+ #- key: node-role.kubernetes.io/master
+ # operator: Exists
+ # effect: NoSchedule
+ - operator: Exists
+ effect: NoSchedule
+ containers:
+ - name: kube-sriov-cni
+ #image: nfvpe/sriov-cni-arm64:latest
+ image: iecedge/sriov-cni-arm64:latest
+ imagePullPolicy: IfNotPresent
+ securityContext:
+ privileged: true
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ volumeMounts:
+ - name: cnibin
+ mountPath: /host/opt/cni/bin
+ volumes:
+ - name: cnibin
+ hostPath:
+ path: /opt/cni/bin
+
+# 5.3 Multus DaemonSet
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kube-multus-ds-amd64
+ namespace: kube-system
+ labels:
+ tier: node
+ app: multus
+ name: multus
+spec:
+ selector:
+ matchLabels:
+ name: multus
+ updateStrategy:
+ type: RollingUpdate
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: multus
+ name: multus
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: amd64
+ tolerations:
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: multus
+ containers:
+ - name: kube-multus
+ #image: nfvpe/multus:v3.3
+ #- "--multus-conf-file=auto"
+ #- "--cni-version=0.3.1"
+ #image: nfvpe/multus:v3.4
+ image: iecedge/multus-amd64:v3.4
+ imagePullPolicy: IfNotPresent
+ command: ["/entrypoint.sh"]
+ args:
+ - "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ securityContext:
+ privileged: true
+ volumeMounts:
+ - name: cni
+ mountPath: /host/etc/cni/net.d
+ - name: cnibin
+ mountPath: /host/opt/cni/bin
+ - name: multus-cfg
+ mountPath: /tmp/multus-conf
+ - name: kubernetes-cfg-dir
+ mountPath: /etc/kubernetes
+ volumes:
+ - name: cni
+ hostPath:
+ path: /etc/cni/net.d
+ - name: cnibin
+ hostPath:
+ path: /opt/cni/bin
+ - name: multus-cfg
+ configMap:
+ name: multus-cni-config
+ items:
+ - key: cni-conf.json
+ path: 70-multus.conf
+ - name: kubernetes-cfg-dir
+ hostPath:
+ path: /etc/kubernetes
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kube-multus-ds-arm64
+ namespace: kube-system
+ labels:
+ tier: node
+ app: multus
+ name: multus
+spec:
+ selector:
+ matchLabels:
+ name: multus
+ updateStrategy:
+ type: RollingUpdate
+ template:
+ metadata:
+ labels:
+ tier: node
+ app: multus
+ name: multus
+ spec:
+ hostNetwork: true
+ nodeSelector:
+ beta.kubernetes.io/arch: arm64
+ tolerations:
+ - operator: Exists
+ effect: NoSchedule
+ serviceAccountName: multus
+ containers:
+ - name: kube-multus
+ #image: nfvpe/multus:v3.3
+ #image: iecedge/multus-arm64:latest
+ #- "--multus-conf-file=auto"
+ #- "--cni-version=0.3.1"
+ image: iecedge/multus-arm64:v3.4
+ imagePullPolicy: IfNotPresent
+ command: ["/entrypoint.sh"]
+ args:
+ - "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "50Mi"
+ limits:
+ cpu: "100m"
+ memory: "50Mi"
+ securityContext:
+ privileged: true
+ volumeMounts:
+ - name: cni
+ mountPath: /host/etc/cni/net.d
+ - name: cnibin
+ mountPath: /host/opt/cni/bin
+ - name: multus-cfg
+ mountPath: /tmp/multus-conf
+ - name: kubernetes-cfg-dir
+ mountPath: /etc/kubernetes
+ volumes:
+ - name: cni
+ hostPath:
+ path: /etc/cni/net.d
+ - name: cnibin
+ hostPath:
+ path: /opt/cni/bin
+ - name: multus-cfg
+ configMap:
+ name: multus-cni-config
+ items:
+ - key: cni-conf.json
+ path: 70-multus.conf
+ - name: kubernetes-cfg-dir
+ hostPath:
+ path: /etc/kubernetes
+
--- /dev/null
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+ name: sriov-net1
+ annotations:
+ k8s.v1.cni.cncf.io/resourceName: arm.com/ps225_sriov_netdevice
+ # "vlan": 1000,
+spec:
+ config: '{
+ "type": "sriov",
+ "cniVersion": "0.3.1",
+ "name": "sriov-network",
+ "ipam": {
+ "type": "host-local",
+ "subnet": "10.56.217.0/24",
+ "rangeStart": "10.56.217.11",
+ "rangeEnd": "10.56.217.181",
+ "routes": [{
+ "dst": "0.0.0.0/0"
+ }],
+ "gateway": "10.56.217.1"
+ }
+}'
--- /dev/null
+#!/bin/bash -ex
+# shellcheck disable=SC1073,SC1072,SC1039,SC2059,SC2046
+
+
+kubectl delete -f sriov-crd.yaml
+sleep 2
+kubectl delete -f flannel-daemonset.yml
+sleep 5
+kubectl delete -f multus-sriov-flannel-daemonsets.yaml
+sleep 5
+kubectl delete -f configMap.yaml
+sleep 2
+
+kubectl get node $(hostname) -o json | jq '.status.allocatable'
--- /dev/null
+FROM ubuntu:18.04
+MAINTAINER The IPerf Project <trevor.tao@arm.com>
+
+RUN apt-get update && apt-get install -y iperf inetutils-ping iproute2
+
+CMD ["iperf -s &"]
--- /dev/null
+# yamllint disable
+apiVersion: v1
+kind: Pod
+metadata:
+ name: iperfv2-client-sriov
+ labels:
+ app: iperfv2-client-sriov
+ annotations:
+ k8s.v1.cni.cncf.io/networks: sriov-net1
+ #spec:
+ # replicas: 1
+ # template:
+ #metadata:
+ # labels:
+ # app: iperf-client
+ # ldpreload-related labels
+spec:
+ containers:
+ - image: iecedge/iperf-arm64:latest
+ imagePullPolicy: IfNotPresent
+ name: iperfv2-client-sriov
+ command: ["bash"]
+ args: ["-c", "while true; do sleep 30; done;"]
+ resources:
+ requests:
+ arm.com/ps225_sriov_netdevice: '1'
+ limits:
+ arm.com/ps225_sriov_netdevice: '1'
+# securityContext:
+# privileged: true
--- /dev/null
+# yamllint disable
+#apiVersion: extensions/v1beta1
+apiVersion: apps/v1
+kind: Service
+apiVersion: v1
+metadata:
+ name: iperfv2-server-sriov
+spec:
+ type: NodePort
+ ports:
+ - protocol: TCP
+ port: 5001
+ targetPort: 5001
+ #nodePort: 31050
+ selector:
+ app: iperfv2-server-sriov
+---
+apiVersion: v1
+#kind: Deployment
+kind: Pod
+metadata:
+ name: iperfv2-server-sriov
+ labels:
+ app: iperfv2-server-sriov
+ annotations:
+ k8s.v1.cni.cncf.io/networks: sriov-net1
+spec:
+ containers:
+ - image: iecedge/iperf-arm64
+ imagePullPolicy: IfNotPresent
+ name: iperfv2-server-sriov
+ ports:
+ - containerPort: 5001
+ command: ["iperf"]
+ args: ["-s"]
+ resources:
+ requests:
+ arm.com/ps225_sriov_netdevice: '1'
+ limits:
+ arm.com/ps225_sriov_netdevice: '1'
+# securityContext:
+# privileged: true
--- /dev/null
+# yamllint disable
+# An example of Pod spec using above SR-IOV CRD:
+apiVersion: v1
+kind: Pod
+metadata:
+ name: testpod1
+ labels:
+ env: test
+ annotations:
+ k8s.v1.cni.cncf.io/networks: sriov-net1
+spec:
+ containers:
+ - name: appcntr1
+ image: iecedge/centos-tools-arm64
+ imagePullPolicy: IfNotPresent
+ command: [ "/bin/bash", "-c", "--" ]
+ args: [ "while true; do sleep 300000; done;" ]
+ resources:
+ requests:
+ arm.com/ps225_sriov_netdevice: '1'
+ limits:
+ arm.com/ps225_sriov_netdevice: '1'
+
--- /dev/null
+# yamllint disable
+# An example of Pod spec using above SR-IOV CRD:
+apiVersion: v1
+kind: Pod
+metadata:
+ name: testpod2
+ labels:
+ env: test
+ annotations:
+ k8s.v1.cni.cncf.io/networks: sriov-net1
+spec:
+ containers:
+ - name: appcntr2
+ image: iecedge/centos-tools-arm64
+ imagePullPolicy: IfNotPresent
+ command: [ "/bin/bash", "-c", "--" ]
+ args: [ "while true; do sleep 300000; done;" ]
+ resources:
+ requests:
+ arm.com/ps225_sriov_netdevice: '1'
+ limits:
+ arm.com/ps225_sriov_netdevice: '1'
+
}
+install_multus_sriov_flannel(){
+
+ sed -i "s@10.244.0.0/16@${POD_NETWORK_CIDR}@" "${SCRIPTS_DIR}/cni/multus/multus-sriov-flannel/flannel-daemonset.yml"
+ # Install Multus Flannel+SRIOV by yaml files
+ # shellcheck source=/dev/null
+ source ${SCRIPTS_DIR}/cni/multus/multus-sriov-flannel/install.sh
+
+}
+
install_danm(){
${SCRIPTS_DIR}/cni/danm/danm_install.sh
echo "Install Ovn-Kubernetes ..."
install_ovn_kubernetes
;;
+ 'multus-flannel-sriov')
+ echo "Install Flannel with SRIOV CNI by Multus-CNI ..."
+ install_multus_sriov_flannel
+ ;;
'danm')
echo "Install danm ..."
install_danm
Code Review / iec.git / commitdiff