1 ## Intel Rook infrastructure for Ceph cluster deployment
3 By default create osd on folder /var/lib/rook/storage-dir, and Ceph cluster
4 information on /var/lib/rook.
8 1. Compute node disk space: 20GB+ free disk space.
10 2. Kubernetes version: Kubernetes version >= 1.13 required by Ceph CSI v1.0.
11 Following is the upgrade patch in kud github: https://github.com/onap/multicloud-k8s
15 diff --git a/kud/deployment_infra/playbooks/kud-vars.yml b/kud/deployment_infra/playbooks/kud-vars.yml
16 index 9b36547..5c29fa4 100644
17 --- a/kud/deployment_infra/playbooks/kud-vars.yml
18 +++ b/kud/deployment_infra/playbooks/kud-vars.yml
19 @@ -58,7 +58,7 @@ ovn4nfv_version: adc7b2d430c44aa4137ac7f9420e14cfce3fa354
20 ovn4nfv_url: "https://git.opnfv.org/ovn4nfv-k8s-plugin/"
23 -kubespray_version: 2.8.2
24 -helm_client_version: 2.9.1
25 +kubespray_version: 2.9.0
26 +helm_client_version: 2.13.1
27 # kud playbooks not compatible with 2.8.0 - see MULTICLOUD-634
28 ansible_version: 2.7.10
29 diff --git a/kud/hosting_providers/vagrant/inventory/group_vars/k8s-cluster.yml b/kud/hosting_providers/vagrant/inventory/group_vars/k8s-cluster.yml
30 index 9966ba8..cacb4b3 100644
31 --- a/kud/hosting_providers/vagrant/inventory/group_vars/k8s-cluster.yml
32 +++ b/kud/hosting_providers/vagrant/inventory/group_vars/k8s-cluster.yml
33 @@ -48,7 +48,7 @@ local_volumes_enabled: true
34 local_volume_provisioner_enabled: true
36 ## Change this to use another Kubernetes version, e.g. a current beta release
37 -kube_version: v1.12.3
38 +kube_version: v1.13.5
44 After upgraded, the Kubernetes version as following:
47 Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
48 Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
51 If something is wrong with Kubectl server version, you can manually upgrade as
54 $ kubeadm upgrade apply v1.13.5
59 To bring up Rook operator(v1.0) and Ceph cluster(Mimic 13.2.2) as following:
68 If you want to make a test on the ceph sample workload, check as following:
70 1. Bring up Rook operator and Ceph cluster.
71 2. Goto Create storage class.
74 kubectl create -f ./test/rbd/storageclass.yaml
79 kubectl exec -ti -n rook-ceph rook-ceph-operator-948f8f84c-749zb -- bash -c
80 "ceph -c /var/lib/rook/rook-ceph/rook-ceph.config auth get-or-create-key client.kube mon \"allow profile rbd\" osd \"profile rbd pool=rbd\""
82 You need to replace the pod name with your own rook-operator, refer: kubetl get pod -n rook-ceph
83 Then get secret of admin and client user key by go into operator pod and execute:
85 ceph auth get-key client.admin|base64
86 ceph auth get-key client.kube|base64
88 Then fill the key into secret.yaml
90 kubectl create -f ./test/rbd/secret.yaml
92 4. Create RBD Persistent Volume Claim
94 kubectl create -f ./test/rbd/pvc.yaml
96 5. Create RBD demo pod
98 kubectl creaet -f ./test/rbd/pod.yaml
100 6. Check the Volumes created and application mount status
102 tingjie@ceph4:~/bohemian/workspace/rook/Documentation$ kubectl get pvc
103 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
104 rbd-pvc Bound pvc-98f50bec-8a4f-434d-8def-7b69b628d427 1Gi RWO csi-rbd 84m
105 tingjie@ceph4:~/bohemian/workspace/rook/Documentation$ kubectl get pod
106 NAME READY STATUS RESTARTS AGE
107 csirbd-demo-pod 1/1 Running 0 84m
108 tingjie@ceph4:~/bohemian/workspace/rook/Documentation$ kubectl exec -ti csirbd-demo-pod -- bash
109 root@csirbd-demo-pod:/# df -h
110 Filesystem Size Used Avail Use% Mounted on
111 overlay 733G 35G 662G 5% /
112 tmpfs 64M 0 64M 0% /dev
113 tmpfs 32G 0 32G 0% /sys/fs/cgroup
114 /dev/sda2 733G 35G 662G 5% /etc/hosts
115 shm 64M 0 64M 0% /dev/shm
116 /dev/rbd0 976M 2.6M 958M 1% /var/lib/www/html
117 tmpfs 32G 12K 32G 1% /run/secrets/kubernetes.io/serviceaccount
118 tmpfs 32G 0 32G 0% /proc/acpi
119 tmpfs 32G 0 32G 0% /proc/scsi
120 tmpfs 32G 0 32G 0% /sys/firmware
122 7. Create RBD snapshot-class
124 kubectl create -f ./test/rbd/snapshotclass.yaml
126 8. Create Volume snapshot and verify
128 kubectl create -f ./test/rbd/snapshot.yaml
130 $ kubectl get volumesnapshotclass
132 csi-rbdplugin-snapclass 51s
133 $ kubectl get volumesnapshot
138 9. Restore the snapshot to a new PVC and verify
140 kubectl create -f ./test/rbd/pvc-restore.yaml
143 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
144 rbd-pvc Bound pvc-98f50bec-8a4f-434d-8def-7b69b628d427 1Gi RWO csi-rbd 42h
145 rbd-pvc-restore Bound pvc-530a4939-e4c0-428d-a072-c9c39d110d7a 1Gi RWO csi-rbd 5s