## Intel Rook infrastructure for Ceph cluster deployment

By default create osd on folder /var/lib/rook/storage-dir, and Ceph cluster
information on /var/lib/rook.

# Precondition

1. Compute node disk space: 20GB+ free disk space.

2. Kubernetes version: Kubernetes version >= 1.13 required by Ceph CSI v1.0.
Following is the upgrade patch in kud github: https://github.com/onap/multicloud-k8s

```
$ git diff
diff --git a/kud/deployment_infra/playbooks/kud-vars.yml b/kud/deployment_infra/playbooks/kud-vars.yml
index 9b36547..5c29fa4 100644
--- a/kud/deployment_infra/playbooks/kud-vars.yml
+++ b/kud/deployment_infra/playbooks/kud-vars.yml
@@ -58,7 +58,7 @@ ovn4nfv_version: adc7b2d430c44aa4137ac7f9420e14cfce3fa354
 ovn4nfv_url: "https://git.opnfv.org/ovn4nfv-k8s-plugin/"

 go_version: '1.12.5'
-kubespray_version: 2.8.2
-helm_client_version: 2.9.1
+kubespray_version: 2.9.0
+helm_client_version: 2.13.1
 # kud playbooks not compatible with 2.8.0 - see MULTICLOUD-634
 ansible_version: 2.7.10
diff --git a/kud/hosting_providers/vagrant/inventory/group_vars/k8s-cluster.yml b/kud/hosting_providers/vagrant/inventory/group_vars/k8s-cluster.yml
index 9966ba8..cacb4b3 100644
--- a/kud/hosting_providers/vagrant/inventory/group_vars/k8s-cluster.yml
+++ b/kud/hosting_providers/vagrant/inventory/group_vars/k8s-cluster.yml
@@ -48,7 +48,7 @@ local_volumes_enabled: true
 local_volume_provisioner_enabled: true

 ## Change this to use another Kubernetes version, e.g. a current beta release
-kube_version: v1.12.3
+kube_version: v1.13.5

 # Helm deployment
 helm_enabled: true
```

After upgraded, the Kubernetes version as following:
```
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
```

If something is wrong with Kubectl server version, you can manually upgrade as
command:
```console
$ kubeadm upgrade apply v1.13.5
```

# Deployment

To bring up Rook operator(v1.0) and Ceph cluster(Mimic 13.2.2) as following:

```console
cd yaml
./install.sh
```

# Test

If you want to make a test on the ceph sample workload, check as following:

1. Bring up Rook operator and Ceph cluster.
2. Goto Create storage class.

```console
kubectl create -f ./test/rbd/storageclass.yaml
```

3. Create RBD secret.
```console
kubectl exec -ti -n rook-ceph rook-ceph-operator-948f8f84c-749zb -- bash -c 
"ceph -c /var/lib/rook/rook-ceph/rook-ceph.config auth get-or-create-key client.kube mon \"allow profile rbd\" osd \"profile rbd pool=rbd\""
```
   You need to replace the pod name with your own rook-operator, refer: kubetl get pod -n rook-ceph
   Then get secret of admin and client user key by go into operator pod and execute:
```console
ceph auth get-key client.admin|base64
ceph auth get-key client.kube|base64
```
  Then fill the key into secret.yaml
```console
kubectl create -f ./test/rbd/secret.yaml
```
4. Create RBD Persistent Volume Claim
```console
kubectl create -f ./test/rbd/pvc.yaml
```
5. Create RBD demo pod
```console
kubectl creaet -f ./test/rbd/pod.yaml
```
6. Check the Volumes created and application mount status
```console
tingjie@ceph4:~/bohemian/workspace/rook/Documentation$ kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-pvc   Bound    pvc-98f50bec-8a4f-434d-8def-7b69b628d427   1Gi        RWO            csi-rbd        84m
tingjie@ceph4:~/bohemian/workspace/rook/Documentation$ kubectl get pod
NAME              READY   STATUS    RESTARTS   AGE
csirbd-demo-pod   1/1     Running   0          84m
tingjie@ceph4:~/bohemian/workspace/rook/Documentation$ kubectl exec -ti csirbd-demo-pod -- bash
root@csirbd-demo-pod:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         733G   35G  662G   5% /
tmpfs            64M     0   64M   0% /dev
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/sda2       733G   35G  662G   5% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/rbd0       976M  2.6M  958M   1% /var/lib/www/html
tmpfs            32G   12K   32G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            32G     0   32G   0% /proc/acpi
tmpfs            32G     0   32G   0% /proc/scsi
tmpfs            32G     0   32G   0% /sys/firmware
```
7. Create RBD snapshot-class
```console
kubectl create -f ./test/rbd/snapshotclass.yaml
```
8. Create Volume snapshot and verify
```console
kubectl create -f ./test/rbd/snapshot.yaml

$ kubectl get volumesnapshotclass
NAME                      AGE
csi-rbdplugin-snapclass   51s
$ kubectl get volumesnapshot
NAME               AGE
rbd-pvc-snapshot   33s

```
9. Restore the snapshot to a new PVC and verify
```console
kubectl create -f ./test/rbd/pvc-restore.yaml

$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-pvc           Bound    pvc-98f50bec-8a4f-434d-8def-7b69b628d427   1Gi        RWO            csi-rbd        42h
rbd-pvc-restore   Bound    pvc-530a4939-e4c0-428d-a072-c9c39d110d7a   1Gi        RWO            csi-rbd        5s
```