4 This document describes how to deployment IEC platform with Contiv-VPP
5 networking on bare metal hosts.The automatic deployment script provided
6 by IEC uses calico CNI by default. To enable Contiv-VPP network solution
7 for Kubernetes, you need to make some minor modifications.Now the IEC
8 only supports multiple NICs deployment, and does not support Configuring
9 STN for the time being.In addition, the deployment methods of IEC type1
10 and type2 are slightly different, and will be introduced in different
13 Setting up for IEC type2
14 ------------------------
16 IEC type2 deploy on large and powerful business servers.The main
17 installation steps as following:
22 ALL port that are to be used by an DPDK aplication must be bound to the
23 uio_pci_generic, igb_uio or vfio-pci module before the application is
24 run, more detail info please refer `DPDK DOC`_.
26 The following guide will use vfio-pci. Load kernel module
30 $ sudo modprobe vfio-pci
32 Verify that PCI driver has loaded successfully
38 vfio_virqfd 16384 1 vfio_pci
39 vfio_iommu_type1 24576 0
40 vfio 40960 2 vfio_iommu_type1,vfio_pci
42 Determining network adapter that vpp to use
46 $ sudo lshw -class network -businfo
47 Bus info Device Class Description
48 ================================================
49 pci@0000:89:00.0 enp137s0f0 network Ethernet Controller X710 for 10GbE SFP+
50 pci@0000:89:00.1 enp137s0f1 network Ethernet Controller X710 for 10GbE SFP+
52 In this example, enp137s0f1 used by vpp and binding to kernel module:
56 $ sudo ~/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci enp137s0f1
58 The script dpdk-devbind.py in `DPDK`_ repo.
63 As a minimum requirement 3 nodes are needed: jumpserver, master node and
64 worker node. The two kinds nodes are configured by different script.
66 - Modify default network solution
70 --- a/src/foundation/scripts/config
71 +++ b/src/foundation/scripts/config
72 @@ -30,7 +30,7 @@ K8S_WORKER_GROUP=(
73 CLUSTER_IP=172.16.1.136 # Align with the value in our K8s setup script
74 POD_NETWORK_CIDR=192.168.0.0/16
75 #IEC support three kinds network solution for Kubernetes: calico,flannel,contivpp
78 #kubernetes-cni version 0.7.5/ 0.6.0
81 - Master node configuration
83 Initialize DEV_NAME for master node,Instantiate the fourth argument of the setup-cni.sh script
87 --- a/src/foundation/scripts/startup.sh
88 +++ b/src/foundation/scripts/startup.sh
89 @@ -99,7 +99,7 @@ deploy_k8s () {
90 #Deploy etcd & CNI from master node
91 - SETUP_CNI="cd iec/src/foundation/scripts && source setup-cni.sh $CLUSTER_IP $POD_NETWORK_CIDR $CNI_TYPE"
92 + SETUP_CNI="cd iec/src/foundation/scripts && source setup-cni.sh $CLUSTER_IP $POD_NETWORK_CIDR $CNI_TYPE enp137s0f1"
93 sshpass -p ${K8S_MASTERPW} ssh -o StrictHostKeyChecking=no ${HOST_USER}@${K8S_MASTER_IP} ${SETUP_CNI}
94 SETUP_HELM="cd iec/src/foundation/scripts && source helm.sh"
96 The modified result is as follows
100 --- a/src/foundation/scripts/setup-cni.sh
101 +++ b/src/foundation/scripts/setup-cni.sh
103 CLUSTER_IP=${1:-172.16.1.136} # Align with the value in our K8s setup script
104 POD_NETWORK_CIDR=${2:-192.168.0.0/16}
105 CNI_TYPE=${3:-calico}
107 +DEV_NAME=${4:-enp137s0f1}
109 - Worker node configuration
111 The same as master node, worker node need setting up DPDK and
112 determining network adapter. Initialize DEV_NAME for work node
116 --- a/src/foundation/scripts/config
117 +++ b/src/foundation/scripts/config
118 @@ -42,4 +42,4 @@ KUBE_VERSION=1.13.0
119 # [10.169.40.106]="enp137s0f0"
123 +DEV_NAME=([10.169.40.106]="enp137s0f0")
125 DEV_NAME is an associative array, list network interface device names
126 used by contivpp. Use IP address of K8S_WORKER_GROUP as key.
130 Simply start the installation script startup.sh on jumpserver:
134 jenkins@jumpserver:~/iec/src/foundation/scripts$ ./startup.sh
136 for more details and information refer to `installation.instruction.rst`_
138 Setting up for IEC type1
139 ------------------------
141 IEC type1 device is suitable for low power device.Now we choose
142 `MACCHIATObin`_ board as the main hardware
148 Marvell User-Space SDK(`MUSDK`_)
149 is a light-weight user-space I/O driver for Marvell's Embedded
150 Networking SoC's, more detail info please refer `VPP Marvell plugin`_
152 Automation deployment
153 ~~~~~~~~~~~~~~~~~~~~~
155 - Modify default yaml
159 diff --git a/src/foundation/scripts/setup-cni.sh b/src/foundation/scripts/setup-cni.sh
160 index d466831..6993006 100755
161 --- a/src/foundation/scripts/setup-cni.sh
162 +++ b/src/foundation/scripts/setup-cni.sh
163 @@ -43,7 +43,7 @@ install_contivpp(){
165 # Install contivpp CNI
166 sed -i "s@10.1.0.0/16@${POD_NETWORK_CIDR}@" "${SCRIPTS_DIR}/cni/contivpp/contiv-vpp.yaml"
167 - kubectl apply -f "${SCRIPTS_DIR}/cni/contivpp/contiv-vpp.yaml"
168 + kubectl apply -f "${SCRIPTS_DIR}/cni/contivpp/contiv-vpp-macbin.yaml"
173 To configure a PP2 interface, MainVppInterface with the prefix mv-ppio-
174 must be configured in the NodeConfig section of the deployment yaml.
175 mv-ppio-X/Y is VPP interface name where X is PP2 device ID and Y is PPIO
176 ID Interface needs to be assigned to MUSDK in FDT configuration and
177 linux interface state must be up. Example configuration:
181 ~/iec/src/foundation/scripts/cni/contivpp/contiv-vpp-macbin.yaml
183 - nodeName: net-arm-mcbin-iec
185 interfaceName: mv-ppio-0/0
186 - nodeName: net-arm-mcbin-iec-1
188 interfaceName: mv-ppio-0/0
190 PP2 doesn't have any dependency on DPDK or DPDK plugin but it can work
191 with DPDK plugin enabled or disabled.It is observed that performace is
192 better around 30% when DPDK plugin is disabled. DPKD plugin can be
193 disabled by adding following config to the contiv-vswitch.conf.
197 --- a/src/foundation/scripts/cni/contivpp/contiv-vswitch.conf
198 +++ b/src/foundation/scripts/cni/contivpp/contiv-vswitch.conf
199 @@ -24,3 +24,7 @@ socksvr {
204 + plugin vpp_plugin.so { enable }
205 + plugin dpdk_plugin.so { disable }
210 It`s necessary to modify relevant script as IEC type2 to support automatic deployment.
214 Simply start the installation script startup.sh on jumpserver:
218 jenkins@jumpserver:~/iec/src/foundation/scripts$ ./startup.sh
220 for more details and information refer to
221 `installation.instruction.rst`_
223 Deployment Verification
224 -----------------------
226 invok ./src/foundation/scripts/nginx.sh install nginx; to test if CNI
229 Uninstalling Contiv-VPP
230 -----------------------
232 To uninstall the network plugin for type2:
236 kubectl delete -f ./iec/src/foundation/scripts/cni/contivpp/contiv-vpp.yaml
238 To uninstall the network plugin for type1:
242 kubectl delete -f ./iec/src/foundation/scripts/cni/contivpp/contiv-vpp-macbin.yaml
244 In order to remove the persisted config, cleanup the bolt and ETCD
249 rm -rf /var/etcd/contiv-data
251 .. All links go below this line
252 .. _`DPDK DOC`: https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules
253 .. _`DPDK`: https://github.com/DPDK/dpdk/blob/master/usertools/dpdk-devbind.py
254 .. _`installation.instruction.rst`: ./installation.instruction.rst
255 .. _`MACCHIATObin`: http://macchiatobin.net
256 .. _`MUSDK`: https://github.com/MarvellEmbeddedProcessors/musdk-marvell
257 .. _`VPP Marvell plugin`: https://github.com/FDio/vpp/blob/master/src/plugins/marvell/README.md