From: Trevor Tao Date: Fri, 15 Feb 2019 15:21:10 +0000 (+0800) Subject: Add IEC Foundation Installation Document X-Git-Tag: 1.0.0~19^2 X-Git-Url: https://gerrit.akraino.org/r/gitweb?a=commitdiff_plain;h=14f3bb8b0598b9a6f4164dac52a2cb8e628bd57d;p=iec.git Add IEC Foundation Installation Document Add IEC reference framework installation document, including the installation description for docker, Kubernetes, Calico and Helm on arm, which would be used as a basis for our development work. Change-Id: I1400710c54f972398559324f0cb71187bf6282a3 Signed-off-by: Trevor Tao --- diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..e40fa81 --- /dev/null +++ b/README.rst @@ -0,0 +1,286 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Integrated Edge Cloud documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + + +================================= +IEC Reference Foundation Overview +================================= + +This document provides a general description about the reference foundation of IEC. +The Integrated Edge Cloud (IEC) will enable new functionalities and business models +on the network edge. The benefits of running applications on the network edge are +- Better latencies for end users +- Less load on network since more data can be processed locally +- Fully utilize the computation power of the edge devices + +.. _Kubernetes: https://kubernetes.io/ +.. _Calico: https://www.projectcalico.org/ +.. _Contiv: https://github.com/contiv/vpp +.. _OVN-kubernetes: https://github.com/openvswitch/ovn-kubernetes + +Currently, the chosen operating system(OS) is Ubuntu 16.04 and/or 18.04. +The infrastructure orchestration of IEC is based on Kubernetes_, which is a +production-grade container orchestration with rich running eco-system. +The current container networking mechanism (CNI) choosed for Kubernetes is project +Calico, which is a high performance, scalable, policy enabled and widely used container +networking solution with rather easy installation and arm64 support. In the future, +Contiv/VPP or OVN-Kubernetes would also be candidates for Kubernetes networking. + + +Kubernetes Install for Ubuntu +----------------------------- + +Install Docker as Prerequisite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. _Docker: https://www.docker.com/ +.. _install: https://docs.docker.com/install/linux/docker-ce/ubuntu/ + +Docker_ is used for Kuberntes docker images management. The installation script for docker +version 18.06 is given below. More docker install information can be found in the install_ +guide:: + DOCKER_VERSION=18.06.1 + ARCH=arm64 + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - + sudo apt-key fingerprint 0EBFCD88 + sudo add-apt-repository \ + "deb [arch=${ARCH}] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) \ + stable" + sudo apt-get update + sudo apt-get install -y docker-ce=${DOCKER_VERSION}~ce~3-0~ubuntu + + +Disable swap on your machine +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Turn off all swap devices and files with:: + + sudo swapoff -a + +.. _kubeadm: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ + + +Install Kubernetes with Kubeadm +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +kubeadm_ helps you bootstrap a minimum viable Kubernetes cluster that conforms +to best practices which a preferred installation method for IEC currently. +Now we choose v1.13.0 as a current stable version of Kubernetes for arm64. +Usually the current host(edge server/gateway)'s management interface is chosen as +the Kubeapi-server advertise address which is indicated here as ``$MGMT_IP``. + +The common installation steps for both Kubernetes master and slave node are given +as Linux shell scripts:: + + sudo bash + apt-get update && apt-get install -y apt-transport-https curl + curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - + cat </etc/apt/sources.list.d/kubernetes.list + deb https://apt.kubernetes.io/ kubernetes-xenial main + EOF + apt-get update + apt-get install -y kubelet=1.13.0-00 kubeadm=1.13.0-00 kubectl=1.13.0-00 + apt-mark hold kubelet kubeadm kubectl + sysctl net.bridge.bridge-nf-call-iptables=1 + +For host setup as Kubernetes `master`:: + + sudo kubeadm config images pull + sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=$MGMT_IP \ + --service-cidr=172.16.1.0/24 + +To start using your cluster, you need to run (as a regular user):: + + mkdir -p $HOME/.kube + sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config + sudo chown $(id -u):$(id -g) $HOME/.kube/config + +or if you are the ``root`` user:: + + export KUBECONFIG=/etc/kubernetes/admin.conf + +For hosts setup as Kubernetes `slave`:: + + kubeadm join --token :6443 --discovery-token-ca-cert-hash sha256: + +in which the token is given in the master's ``kubeadm init``. + +or using following command which will skip ca-cert verification:: + + kubeadm join --token :6443 --discovery-token-unsafe-skip-ca-verification + +After the `slave` joining the Kubernetes cluster, in the master node, you could check the cluster +node with the command:: + kubectl get nodes + + +Install the Calico CNI Plugin to Kubernetes Cluster +--------------------------------------------------- + +Now we install a Calico_ network add-on so that Kubernetes pods can communicate with each other. +The network must be deployed before any applications. Kubeadm only supports Container Networking +Interface(CNI) based networks for which Calico has supported. + +Install the Etcd Database +~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + kubectl apply -f https://raw.githubusercontent.com/Jingzhao123/arm64TemporaryCalico/temporay_arm64/ + v3.3/getting-started/kubernetes/installation/hosted/etcd-arm64.yaml + +Install the RBAC Roles required for Calico +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/rbac.yaml + +Install Calico to system +~~~~~~~~~~~~~~~~~~~~~~~~ + +Firstly, we should get the configuration file from web site and modify the corresponding image +from amd64 to arm64 version. Then, by using kubectl, the calico pod will be created. +:: + wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/calico.yaml + +Since the "quay.io/calico" image repo does not support does not multi-arch, we have +to replace the “quay.io/calico” image path to "calico" which supports multi-arch. +:: + sed -i "s/quay.io\/calico/calico/" calico.yaml + +Deploy the Calico using following command:: + + kubectl apply -f calico.yaml + +.. Attention:: + + In calico.yaml file, there is an option "IP_AUTODETECTION_METHOD" about choosing + network interface. The default value is "first-found" which means the first valid + IP address (except local interface, docker bridge). So if the number of network-interface + is more than 1 on your server, you should configure it depends on your networking + environments. If it does not configure it properly, there are some error about + calico-node pod: "BGP not established with X.X.X.X". + + +Remove the taints on master node +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + kubectl taint nodes --all node-role.kubernetes.io/master- + + +Verification for the Work of Kubernetes +--------------------------------------- + +Now we can verify the work of Kubernetes and Calico with Kubernets pod and service creation and accessing +based on Nginx which is a widely used web server. + +Firstly, create a file named nginx-app.yaml to describe a Pod and service by:: + + $ cat <~/nginx-app.yaml + apiVersion: v1 + kind: Service + metadata: + name: nginx + labels: + app: nginx + spec: + type: NodePort + ports: + - port: 80 + protocol: TCP + name: http + selector: + app: nginx + --- + apiVersion: v1 + kind: ReplicationController + metadata: + name: nginx + spec: + replicas: 2 + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx + ports: + - containerPort: 80 + EOF + +then test the Kubernetes working status with the script:: + + set -ex + kubectl create -f ~/nginx-app.yaml + kubectl get nodes + kubectl get services + kubectl get pods + kubectl get rc + r="0" + while [ $r -ne "2" ] + do + r=$(kubectl get pods | grep Running | wc -l) + sleep 60 + done + svcip=$(kubectl get services nginx -o json | grep clusterIP | cut -f4 -d'"') + sleep 10 + wget http://$svcip + kubectl delete -f ./examples/nginx-app.yaml + kubectl delete -f ./nginx-app.yaml + kubectl get rc + kubectl get pods + kubectl get services + +.. _Helm: https://github.com/helm/helm + +Helm Install on Arm64 +--------------------- + +Helm_ is a tool for managing Kubernetes charts. Charts are packages of pre-configured +Kubernetes resources. The installation of Helm on arm64 is as followes:: + + wget https://storage.googleapis.com/kubernetes-helm/helm-v2.12.3-linux-arm64.tar.gz + xvf helm-v2.12.3-linux-arm64.tar.gz + sudo cp linux-arm64/helm /usr/bin + sudo cp linux-arm64/tiller /usr/bin + + +Further Information +------------------- + +We would like to provide a walk through shell script to automate the installation of Kubernetes +and Calico in the future. But this README is still useful for IEC developers and users. + +For issues or anything on the reference foundation stack of IEC, you could contact: + +Trevor Tao: trevor.tao@arm.com + +Jingzhao Ni: jingzhao.ni@arm.com + +Jianlin Lv: jianlin.lv@arm.com