..
      Licensed under the Apache License, Version 2.0 (the "License"); you may
      not use this file except in compliance with the License. You may obtain
      a copy of the License at

          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
      License for the specific language governing permissions and limitations
      under the License.

      Convention for heading levels in Integrated Edge Cloud documentation:

      =======  Heading 0 (reserved for the title in a document)
      -------  Heading 1
      ~~~~~~~  Heading 2
      +++++++  Heading 3
      '''''''  Heading 4

      Avoid deeper levels because they do not render well.


=================================
IEC Reference Foundation Overview
=================================

This document provides a general description about the reference foundation of IEC.
The Integrated Edge Cloud (IEC) will enable new functionalities and business models
on the network edge. The benefits of running applications on the network edge are
- Better latencies for end users
- Less load on network since more data can be processed locally
- Fully utilize the computation power of the edge devices

.. _Kubernetes: https://kubernetes.io/
.. _Calico: https://www.projectcalico.org/
.. _Contiv: https://github.com/contiv/vpp
.. _OVN-kubernetes: https://github.com/openvswitch/ovn-kubernetes

Currently, the chosen operating system(OS) is Ubuntu 16.04 and/or 18.04.
The infrastructure orchestration of IEC is based on Kubernetes_, which is a
production-grade container orchestration with rich running eco-system.
The current container networking mechanism (CNI) choosed for Kubernetes is project
Calico, which is a high performance, scalable, policy enabled and widely used container
networking solution with rather easy installation and arm64 support. In the future,
Contiv/VPP or OVN-Kubernetes would also be candidates for Kubernetes networking.


Kubernetes Install for Ubuntu
-----------------------------

Install Docker as Prerequisite
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. _Docker: https://www.docker.com/
.. _install: https://docs.docker.com/install/linux/docker-ce/ubuntu/

Docker_ is used for Kuberntes docker images management. The installation script for docker
version 18.06 is given below. More docker install information can be found in the install_
guide::
    DOCKER_VERSION=18.06.1
    ARCH=arm64
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    sudo apt-key fingerprint 0EBFCD88
    sudo add-apt-repository \
     "deb [arch=${ARCH}] https://download.docker.com/linux/ubuntu \
     $(lsb_release -cs) \
     stable"
    sudo apt-get update
    sudo apt-get install -y docker-ce=${DOCKER_VERSION}~ce~3-0~ubuntu


Disable swap on your machine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Turn off all swap devices and files with::

   sudo swapoff -a

.. _kubeadm: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/


Install Kubernetes with Kubeadm
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

kubeadm_ helps you bootstrap a minimum viable Kubernetes cluster that conforms
to best practices which a preferred installation method for IEC currently.
Now we choose v1.13.0 as a current stable version of Kubernetes for arm64.
Usually the current host(edge server/gateway)'s management interface is chosen as
the Kubeapi-server advertise address which is indicated here as ``$MGMT_IP``.

The common installation steps for both Kubernetes master and slave node are given
as Linux shell scripts::

   sudo bash
   apt-get update && apt-get install -y apt-transport-https curl
   curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
   cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
   deb https://apt.kubernetes.io/ kubernetes-xenial main
   EOF
   apt-get update
   apt-get install -y kubelet=1.13.0-00 kubeadm=1.13.0-00 kubectl=1.13.0-00
   apt-mark hold kubelet kubeadm kubectl
   sysctl net.bridge.bridge-nf-call-iptables=1

For host setup as Kubernetes `master`::

   sudo kubeadm config images pull
   sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=$MGMT_IP \
   --service-cidr=172.16.1.0/24

To start using your cluster, you need to run (as a regular user)::

   mkdir -p $HOME/.kube
   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   sudo chown $(id -u):$(id -g) $HOME/.kube/config

or if you are the ``root`` user::

   export KUBECONFIG=/etc/kubernetes/admin.conf

For hosts setup as Kubernetes `slave`::

   kubeadm join --token <token> <master-ip>:6443 --discovery-token-ca-cert-hash sha256:<hash>

in which the token is given in the master's ``kubeadm init``.

or using following command which will skip ca-cert verification::

   kubeadm join --token <token> <master_ip>:6443 --discovery-token-unsafe-skip-ca-verification

After the `slave` joining the Kubernetes cluster, in the master node, you could check the cluster
node with the command::
   kubectl get nodes


Install the Calico CNI Plugin to Kubernetes Cluster
---------------------------------------------------

Now we install a Calico_ network add-on so that Kubernetes pods can communicate with each other.
The network must be deployed before any applications. Kubeadm only supports Container Networking
Interface(CNI) based networks for which Calico has supported.

Install the Etcd Database
~~~~~~~~~~~~~~~~~~~~~~~~~

::

   kubectl apply -f https://raw.githubusercontent.com/Jingzhao123/arm64TemporaryCalico/temporay_arm64/
   v3.3/getting-started/kubernetes/installation/hosted/etcd-arm64.yaml

Install the RBAC Roles required for Calico
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

   kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/rbac.yaml

Install Calico to system
~~~~~~~~~~~~~~~~~~~~~~~~

Firstly, we should get the configuration file from web site and modify the corresponding image
from amd64 to arm64 version. Then, by using kubectl, the calico pod will be created.
::
   wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/calico.yaml

Since the "quay.io/calico" image repo does not support does not multi-arch, we have
to replace the “quay.io/calico” image path to "calico" which supports multi-arch.
::
   sed -i "s/quay.io\/calico/calico/" calico.yaml

Deploy the Calico using following command::

   kubectl apply -f calico.yaml

.. Attention::

   In calico.yaml file, there is an option "IP_AUTODETECTION_METHOD" about choosing
   network interface. The default value is "first-found" which means the first valid
   IP address (except local interface, docker bridge). So if the number of network-interface
   is more than 1 on your server, you should configure it depends on your networking
   environments. If it does not configure it properly, there are some error about
   calico-node pod: "BGP not established with X.X.X.X".


Remove the taints on master node
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

   kubectl taint nodes --all node-role.kubernetes.io/master-


Verification for the Work of Kubernetes
---------------------------------------

Now we can verify the work of Kubernetes and Calico with Kubernets pod and service creation and accessing
based on Nginx which is a widely used web server.

Firstly, create a file named nginx-app.yaml to describe a Pod and service by::

  $ cat <<EOF >~/nginx-app.yaml
  apiVersion: v1
  kind: Service
  metadata:
    name: nginx
    labels:
      app: nginx
  spec:
    type: NodePort
    ports:
    - port: 80
      protocol: TCP
      name: http
    selector:
      app: nginx
  ---
  apiVersion: v1
  kind: ReplicationController
  metadata:
    name: nginx
  spec:
    replicas: 2
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx
          ports:
          - containerPort: 80
  EOF

then test the Kubernetes working status with the script::

   set -ex
   kubectl create -f ~/nginx-app.yaml
   kubectl get nodes
   kubectl get services
   kubectl get pods
   kubectl get rc
   r="0"
   while [ $r -ne "2" ]
   do
      r=$(kubectl get pods | grep Running | wc -l)
      sleep 60
   done
   svcip=$(kubectl get services nginx  -o json | grep clusterIP | cut -f4 -d'"')
   sleep 10
   wget http://$svcip
   kubectl delete -f ./examples/nginx-app.yaml
   kubectl delete -f ./nginx-app.yaml
   kubectl get rc
   kubectl get pods
   kubectl get services

.. _Helm: https://github.com/helm/helm

Helm Install on Arm64
---------------------

Helm_ is a tool for managing Kubernetes charts. Charts are packages of pre-configured
Kubernetes resources. The installation of Helm on arm64 is as followes::

   wget https://storage.googleapis.com/kubernetes-helm/helm-v2.12.3-linux-arm64.tar.gz
   xvf helm-v2.12.3-linux-arm64.tar.gz
   sudo cp linux-arm64/helm /usr/bin
   sudo cp linux-arm64/tiller /usr/bin


Further Information
-------------------

We would like to provide a walk through shell script to automate the installation of Kubernetes
and Calico in the future. But this README is still useful for IEC developers and users.

For issues or anything on the reference foundation stack of IEC, you could contact:

Trevor Tao: trevor.tao@arm.com

Jingzhao Ni: jingzhao.ni@arm.com

Jianlin Lv:  jianlin.lv@arm.com