Running Rook with k3s
Rancher has a light weight Kubernetes distribution called k3s. It’s a full compliant Kubernetes distribution with a minimal footprint. So small, in fact, that it can run on raspberry Pi’s. It would be really great if we could also run Ceph on it to create a clustered storage system. Luckily for Ceph we have Rook, but does it run on k3s?
To test it out we create 8 nodes running Debian x86_64, of which 4 will have an extra storage device (/dev/sda) attached to them.
Install K3s
First thing to do is install the server. We will do this on node0:
$ curl -sfL https://get.k3s.io | sh -This will run an install script and launch the server processes. Once completed, the server should be up and running. Now we can install the agents on each remaining node:
$ curl -sfL https://get.k3s.io | K3S_URL=https://example-url:6443 K3S_TOKEN=XXX sh -Where K3S_URL is the url to the master server, and K3S_TOKEN is the server token that you can find on the server at /var/lib/rancher/k3s/server/node-token.
Now that we have the server and agents running, check out the status:
$ kubectl get node
NAME STATUS ROLES AGE VERSION
node0 Ready <none> 22m v1.14.1-k3s.4
node1 Ready <none> 12m v1.14.1-k3s.4
node2 Ready <none> 4m34s v1.14.1-k3s.4
node3 Ready <none> 4m40s v1.14.1-k3s.4
node4 Ready <none> 4m44s v1.14.1-k3s.4
node5 Ready <none> 4m38s v1.14.1-k3s.4
node6 Ready <none> 111s v1.14.1-k3s.4
node7 Ready <none> 2m v1.14.1-k3s.4Looks like all the nodes are up!
Now label the respective instances as server or node. Lets start by labeling node0 as server:
$ kubectl label node node0 kubernetes.io/role=master
$ kubectl label node node0 node-role.kubernetes.io/master=""Label node1 to node7 as node. Start with node1:
$ kubectl label node node1 kubernetes.io/role=node
$ kubectl label node node1 node-role.kubernetes.io/node=""and do the same for node2 to node7. The result should be as follows:
$ kubectl get node
NAME STATUS ROLES AGE VERSION
node0 Ready master 17h v1.14.1-k3s.4
node1 Ready node 17h v1.14.1-k3s.4
node2 Ready node 37m v1.14.1-k3s.4
node3 Ready node 37m v1.14.1-k3s.4
node4 Ready node 37m v1.14.1-k3s.4
node5 Ready node 37m v1.14.1-k3s.4
node6 Ready node 34m v1.14.1-k3s.4
node7 Ready node 35m v1.14.1-k3s.4Install Rook
Now that our Kubernetes cluster is up and running, we can start to deploy Ceph on it via Rook. Rook requires a few installation steps, the first is to install the common components:
$ wget https://raw.githubusercontent.com/rook/rook/blob/release-1.0/cluster/examples/kubernetes/ceph/common.yaml
$ kubectl create -f common.yamlNext install the operator portion:
$ wget https://raw.githubusercontent.com/rook/rook/release-1.0/cluster/examples/kubernetes/ceph/operator.yaml
$ kubectl create -f operator.yamlNow that the operator is in place, we can go ahead and install the cluster. Create a file called cluster.yaml with the following content:
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: ceph/ceph:v14.2.1-20190430
dataDirHostPath: /var/lib/rook
mon:
count: 3
allowMultiplePerNode: true
dashboard:
enabled: true
storage:
useAllNodes: true
useAllDevices: true
config:
osdsPerDevice: "1"You must make sure the devices you want to use as OSD (Ceph physical storage device) have NO partitions in them! If the disk has old data then Rook will not include it in the provisioning of the OSDs.
Once created, you can check out the status of the deployment as follows:
$ kubectl -n rook-ceph get pod
NAME READY STATUS RESTARTS AGE
rook-ceph-agent-7f9s6 1/1 Running 0 9m 10.0.2.123 node3 <none> <none>
rook-ceph-agent-9746b 1/1 Running 0 9m 10.0.2.127 node7 <none> <none>
rook-ceph-agent-c9hv6 1/1 Running 0 9m 10.0.2.122 node2 <none> <none>
rook-ceph-agent-ckwfc 1/1 Running 0 9m 10.0.2.124 node4 <none> <none>
rook-ceph-agent-gcvbp 1/1 Running 0 9m 10.0.2.126 node6 <none> <none>
rook-ceph-agent-kl492 1/1 Running 0 9m 10.0.2.125 node5 <none> <none>
rook-ceph-agent-whl66 1/1 Running 0 9m 10.0.2.121 node1 <none> <none>
rook-ceph-agent-wkqtl 1/1 Running 0 9m 10.0.2.120 node0 <none> <none>
rook-ceph-mgr-a-7c86d8c55b-29n8b 1/1 Running 0 7m56s 10.42.9.23 node7 <none> <none>
rook-ceph-mon-a-b969d4775-xckl7 1/1 Running 0 8m40s 10.42.0.29 node0 <none> <none>
rook-ceph-mon-b-865f89ffc4-vn8ns 1/1 Running 0 8m32s 10.42.1.24 node1 <none> <none>
rook-ceph-mon-c-9f74fd6bc-vs9g4 1/1 Running 0 8m18s 10.42.7.22 node2 <none> <none>
rook-ceph-operator-7bbb59d7bd-kxn2x 1/1 Running 0 9m2s 10.42.10.22 node6 <none> <none>
rook-ceph-osd-0-77464bc789-rdp6b 1/1 Running 0 6m52s 10.42.2.24 node4 <none> <none>
rook-ceph-osd-1-55569b7749-hvcd6 1/1 Running 0 6m50s 10.42.6.25 node5 <none> <none>
rook-ceph-osd-2-585759759-4qp7k 1/1 Running 0 6m45s 10.42.10.25 node6 <none> <none>
rook-ceph-osd-3-7d776bc679-62vrj 1/1 Running 0 6m47s 10.42.9.25 node7 <none> <none>
rook-ceph-osd-prepare-node0-kpzdh 0/2 Completed 0 7m30s 10.42.0.30 node0 <none> <none>
rook-ceph-osd-prepare-node1-w7tcp 0/2 Completed 0 7m30s 10.42.1.25 node1 <none> <none>
rook-ceph-osd-prepare-node2-z9nnz 0/2 Completed 0 7m30s 10.42.7.23 node2 <none> <none>
rook-ceph-osd-prepare-node3-fz6br 0/2 Completed 0 7m30s 10.42.4.16 node3 <none> <none>
rook-ceph-osd-prepare-node4-wjjzk 0/2 Completed 0 7m29s 10.42.2.23 node4 <none> <none>
rook-ceph-osd-prepare-node5-927jh 0/2 Completed 0 7m29s 10.42.6.24 node5 <none> <none>
rook-ceph-osd-prepare-node6-849h5 0/2 Completed 0 7m28s 10.42.10.24 node6 <none> <none>
rook-ceph-osd-prepare-node7-kn25p 0/2 Completed 0 7m27s 10.42.9.24 node7 <none> <none>
rook-ceph-tools-b8c679f95-rkn4r 1/1 Running 0 9m1s 10.0.2.124 node4 <none> <none>
rook-discover-5thnt 1/1 Running 0 9m 10.42.2.22 node4 <none> <none>
rook-discover-7w5j5 1/1 Running 0 9m 10.42.1.23 node1 <none> <none>
rook-discover-9j97b 1/1 Running 0 9m 10.42.0.28 node0 <none> <none>
rook-discover-ccggp 1/1 Running 0 9m 10.42.4.15 node3 <none> <none>
rook-discover-cgmbz 1/1 Running 0 9m 10.42.10.23 node6 <none> <none>
rook-discover-cmqww 1/1 Running 0 9m 10.42.7.21 node2 <none> <none>
rook-discover-ggm7q 1/1 Running 0 9m 10.42.9.22 node7 <none> <none>
rook-discover-j7xjr 1/1 Running 0 9m 10.42.6.23 node5 <none> <none>If you want to run various ceph commands, the Ceph Toolbox will come in handy. You can deploy it as follows:
$ wget https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/toolbox.yaml
$ kubectl create -f toolbox.yamlOnce it is up and running you can access the toolbox container as follows:
$ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bashOnce in the container you can run commands like the following to observe the OSD status:
$ ceph osd treeAfter confirming everything is ok, lets deploy the CephFS service:
$ wget https://raw.githubusercontent.com/rook/rook/blob/release-1.0/cluster/examples/kubernetes/ceph/filesystem.yaml
$ kubectl create -f filesystem.yamlWith CephFS now deployed, lets view all the services that are now running:
$ kubectl -n rook-ceph get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.43.49.73 <none> 9283/TCP 15m
rook-ceph-mgr-dashboard ClusterIP 10.43.68.50 <none> 8443/TCP 15m
rook-ceph-mon-a ClusterIP 10.43.214.249 <none> 6789/TCP,3300/TCP 16m
rook-ceph-mon-b ClusterIP 10.43.229.139 <none> 6789/TCP,3300/TCP 16m
rook-ceph-mon-c ClusterIP 10.43.57.138 <none> 6789/TCP,3300/TCP 16mWe see that the dashboard is available, but has no external IP yet. Let’s create one now. Create a file dashboard-external-https.yamlwith the following content:
apiVersion: v1
kind: Service
metadata:
name: rook-ceph-mgr-dashboard-external-https
namespace: rook-ceph
labels:
app: rook-ceph-mgr
rook_cluster: rook-ceph
spec:
ports:
- name: dashboard
port: 8443
protocol: TCP
targetPort: 8443
selector:
app: rook-ceph-mgr
rook_cluster: rook-ceph
sessionAffinity: None
type: NodePortand create it in the cluster:
$ kubectl create -f dashboard-external-https.yamlTo access the website, we need to retrieve the secret for the user admin. You can do that as follows:
$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echoList the services again so we find where NodePort is now running:
$ kubectl -n rook-ceph get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.43.49.73 <none> 9283/TCP 26m
rook-ceph-mgr-dashboard ClusterIP 10.43.68.50 <none> 8443/TCP 26m
rook-ceph-mgr-dashboard-external-https NodePort 10.43.114.102 <none> 8443:32411/TCP 6m9s
rook-ceph-mon-a ClusterIP 10.43.214.249 <none> 6789/TCP,3300/TCP 27m
rook-ceph-mon-b ClusterIP 10.43.229.139 <none> 6789/TCP,3300/TCP 27m
rook-ceph-mon-c ClusterIP 10.43.57.138 <none> 6789/TCP,3300/TCP 26mNow we can access the dashboard by using the port number provided by NodePort, but we first need to find out on which node the dashboard service actually runs. Since the dashboard is run by the Ceph Manager, we need to look for the pod that runs that:
$ kubectl -n rook-ceph get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
...
rook-ceph-mgr-a-7c86d8c55b-29n8b 1/1 Running 0 3d18h 10.42.9.23 node7 <none> <none>
...In our case it runs on node7, so we can access the dashboard from https://node7:32411.
Deploy WebDAV
Lets test our CephFS installation by running a webdav container on top of it. Create a webdav-test.yaml file with the following content:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: null
generation: 1
labels:
name: webdav-rook
name: webdav-rook
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/webdav
spec:
progressDeadlineSeconds: 2147483647
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
name: webdav-rook
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
name: webdav-rook
spec:
containers:
- image: atlp/esp-webdav
imagePullPolicy: Always
name: webdav-rook
volumeMounts:
- mountPath: "/shared"
name: volume
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: volume
flexVolume:
driver: ceph.rook.io/rook
fsType: ceph
options:
fsName: myfs # name of fs specified in filesystem CRD.
clusterNamespace: rook-cephWe will use our WebDAV image which will serve the /shared folder. This folder in turn is mounted using our CephFS myfs filesystem. Deploy the container as follows:
$ kubectl create -f webdav-test.yamlOnce the webDAV container is running, expose the service using a LoadBalancer. Create a file called webdav-service.yaml with the following content:
apiVersion: v1
kind: Service
metadata:
name: webdav-service
labels:
app: webdav-service
spec:
ports:
- port: 8082
targetPort: 80
type: LoadBalancer
selector:
name: webdav-rookWe expose our webdav service on port 8082 to the outside world. Deploy it as follows:
$ kubectl create -f webdav-service.yamlThe service should now be available externally:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 4d18h
webdav-service LoadBalancer 10.43.136.21 10.0.2.120,10.0.2.121,10.0.2.122,... 8082:30158/TCP 10mThe webDAV service is now accessible at http://10.0.2.120:8082/shared.
