TheLazy.Dev
Portfolio GitHub

Blog

ceph |linux |s3 |devops

Exploring Ceph in a multi-node setup

akhal3d96

Ahmed Khaled

Posted on October 10, 2021

Exploring Ceph in a multi-node setup

Introduction

Caph is the future of data storage whether it was objects, blocks or a file system. It provides a great alternative to self-hosted solutions such as GlusterFS in the realm of file systems and to managed services such as AWS S3 as an object storage where it makes more sense to host your own data because you have to store it on private cloud or simply to reduce costs as S3 costs sometimes grow to be pretty insane.

Environment setup

I used Vagrant to create a declarative configuration of my environment.

My setup consist of three virtual machines with Ubuntu 20.04 and attached disk storage of 10 GiB to each one of them (except the first machine it has two disks). Ceph documentation says the minimum disk size is 5 GiB but I had problems in the past working with storage less than 10 GiB so I wanted to be in the safe side.

I gave the admin host more memory than the others because it's the one I mainly interact with and I wanted to make it fast and smooth as possible.



# Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  num_of_nodes = 3
  (0..(num_of_nodes - 1)).each do |node|

    node_name = "ceph-#{node + 1}"

    config.vm.define node_name do |ceph_node|
      ceph_node.vm.box = "ubuntu/focal64"
      ceph_node.vm.hostname = node_name
      ceph_node.vm.network "private_network", type: "dhcp"

      ceph_node.vm.disk :disk, size: "10GB", name: "ceph_storage"

      # CephFS
      ceph_node.vm.disk :disk, size: "10GB", name: "ceph_storage_extra" if node == 0

      ceph_node.vm.provision "shell", path: "install-docker.sh"
      ceph_node.vm.provision "shell", inline: "timedatectl set-ntp on"

      ceph_node.vm.provider "virtualbox" do |vb|
        vb.memory = node == 0 ? "3072" : "2048"
      end
    end
  end
end
Enter fullscreen mode Exit fullscreen mode

I used cephadm to create my Ceph cluster, which requires Docker (or Podman) to be installed on the machine I want to run it on so I made Vagrant execute install-docker.sh on all the virtual machines it creates but I edited it to suit my needs.



# install-docker.sh

#!/bin/sh

set -o errexit
set -o nounset

IFS=$(printf '\n\t')

apt remove --yes docker docker-engine docker.io containerd runc || true
apt update
apt --yes --no-install-recommends install apt-transport-https ca-certificates
wget --quiet --output-document=- https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository --yes "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release --codename --short) stable"
apt update
apt --yes --no-install-recommends install docker-ce docker-ce-cli containerd.io
systemctl enable docker
printf '\nDocker installed successfully\n\n'
Enter fullscreen mode Exit fullscreen mode

I also activated time synchronization on all the hosts in the Vagrantfile because it was disabled for some reason and Ceph requires it to be activated.



$ timedatectl set-ntp on
Enter fullscreen mode Exit fullscreen mode

After running vagrant up you should have three virtual machines on a private network which is also accessible from your host machine in something like this:



ceph-1 (admin)
IPv4 address for enp0s8: 172.28.128.7


ceph-2
IPv4 address for enp0s8: 172.28.128.8


ceph-3
IPv4 address for enp0s8: 172.28.128.9
Enter fullscreen mode Exit fullscreen mode

Creating the cluster

On the host machine:



vagrant ssh ceph-1
Enter fullscreen mode Exit fullscreen mode

To create our Ceph cluster using cephadm we have to download the script from Github on the admin host machine and make it executable:



curl --silent --remote-name --location https://github.com/ceph/ceph/raw/pacific/src/cephadm/cephadm

chmod +x cephadm
Enter fullscreen mode Exit fullscreen mode

The script itself is sufficient enough to bootstrap the cluster but it's more convenient to install it as a package on the system.



./cephadm add-repo --release pacific
./cephadm install
Enter fullscreen mode Exit fullscreen mode

After that we can bootstrap our cluster safely:



cephadm bootstrap --mon-ip 172.28.128.7 --ssh-user vagrant
Enter fullscreen mode Exit fullscreen mode

Obviously 172.28.128.7 is the IP of the admin host (current host).

From Ceph documentation:

The --ssh-user option makes it possible to choose which ssh user cephadm will use to connect to hosts. The associated ssh key will be added to /home//.ssh/authorized_keys. The user that you designate with this option must have passwordless sudo access.

This is slightly off-topic but Vagrant by default creates a passwordless sudo and passwordless SSH access to the virtual machines in creates, the SSH access is possible through private key which it creates and puts in .vagrant/machines/*<machine name>*/virtualbox/private_key in the host Vagrantfile directory. Vagrant also maps Vagrantfile directory to /vagrant in the guest virtual machine, so for example ceph-3 host private SSH key is located in ceph-1 host in directory /vagrant/.vagrant/machines/ceph-3/virtualbox/private_key

Anyway, back to Ceph, after running the bootstrap command you should get your dashboard credentials:

ceph initial credentials

After that, from your host machine, replace ceph-1 with the IP of the virtual machine of your admin host in order to access the dashboard. The dashboard will ask you to change your password to a new one do it and make sure you can access the dashboard successfully.

If everything run smoothly we can now add new hosts to the cluster.

First we run ssh-agent



eval $(ssh-agent)
Enter fullscreen mode Exit fullscreen mode

Then we add the private keys of the other virtual machines (ceph-2 and ceph-3) we want to add to the cluster.



ssh-add /vagrant/.vagrant/machines/ceph-2/virtualbox/private_key

ssh-add /vagrant/.vagrant/machines/ceph-3/virtualbox/private_key
Enter fullscreen mode Exit fullscreen mode

We now copy Ceph public keys to authorized_keys in the other virtual machines to make Ceph able to access those machines and add them to the cluster.



ssh-copy-id -f -i /etc/ceph/ceph.pub vagrant@172.28.128.9
ssh-copy-id -f -i /etc/ceph/ceph.pub vagrant@172.28.128.8
Enter fullscreen mode Exit fullscreen mode

Add them to the clusters.



sudo ceph orch host add ceph-2 172.28.128.8
sudo ceph orch host add ceph-3 172.28.128.9
Enter fullscreen mode Exit fullscreen mode

Notice that the hostname should match the hostname on those machines.

It might take some time (couple of minutes in my case) for Ceph to pull all the images and run all the containers on those machines in order to be added to the cluster.

Make sure all the other hosts are added you should open the dashboard and get something like this:

ceph dashboard after adding the three hosts

Notice that the cluster health is yellow because there's no storage yet Ceph can use to store data in it.

Finally in order to add the disks we attached earlier we have to deploy OSD Service.



sudo ceph orch apply osd --all-available-devices
Enter fullscreen mode Exit fullscreen mode

It might take some time (couple of minutes too) until it propagate in the three hosts and add their attached disks.

After that your cluster status should be healthy and you can run object storage services and file system services in your cluster.

ceph dashboard showing a healthy ceph cluster

Object Storage

To use the Object Gateway a RGW service must be running.



ceph orch apply rgw s3
Enter fullscreen mode Exit fullscreen mode

You should also create a radosgw user in order to access the object gateway service whether it was from an API or from Ceph dashboard.



ceph dashboard set-rgw-credentials

radosgw-admin user create --uid="ahmed" --display-name="Ahmed Khaled" --system
Enter fullscreen mode Exit fullscreen mode

You should get a result similar to this:



{
    "user_id": "ahmed",
    "display_name": "Ahmed Khaled",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "ahmed",
            "access_key": "05LT3KPECW7E30X2P4UO",
            "secret_key": "bBEJBxiM6JkRDUzCpGzN7EkGaVr9qSjqlDdNZWMp"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "system": "true",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}
Enter fullscreen mode Exit fullscreen mode

The access_key and secret_key are what we care about the most in the result.

In order to access the object gateway from the dashboard we should import the access key and the secret key.



echo 05LT3KPECW7E30X2P4UO > /tmp/access_key
echo bBEJBxiM6JkRDUzCpGzN7EkGaVr9qSjqlDdNZWMp > /tmp/secret_key

ceph dashboard set-rgw-api-access-key -i /tmp/access_key 
ceph dashboard set-rgw-api-secret-key -i /tmp/secret_key
Enter fullscreen mode Exit fullscreen mode

ceph dashbaord showing the radosgw user created

Test Ceph S3-like API using boto

In your host machine setup a virtual Python environment using virtualenv in any directory you want.



virtualenv --python=python2.7 .venv
source .venv/bin/activate
pip install boto
Enter fullscreen mode Exit fullscreen mode

I used Python 2.7 because it's what the official documentation supports but I do't think there would ny any issue for Python 3 and boto3 to work with Ceph API.

To make it more real-world-like I added the IP of the Ceph admin virtual machine to my /etc/hosts and named it 'objects.lan'

For more information about private networks and private domains check my article on this topic.



172.28.128.7    objects.lan
Enter fullscreen mode Exit fullscreen mode

After that create the Python test file.



# s3.py

import boto
import boto.s3.connection

# This is for quick demonstration purpose 
# Never write your credentials in a code
access_key = '05LT3KPECW7E30X2P4UO'
secret_key = 'bBEJBxiM6JkRDUzCpGzN7EkGaVr9qSjqlDdNZWMp'


host = 'objects.lan' # /etc/hosts

conn = boto.connect_s3(
        aws_access_key_id = access_key,
        aws_secret_access_key = secret_key,
        host = host,
        is_secure=False,               # uncomment if you are not using ssl
        calling_format = boto.s3.connection.OrdinaryCallingFormat(),
        )

# Create a new nucket
bucket = conn.create_bucket('my-new-bucket')


# List all buckets
for bucket in conn.get_all_buckets():
        print("{name}\t{created}".format(
                name = bucket.name,
                created = bucket.creation_date,
        ))


# Add object to bucket
key = bucket.new_key('hello.txt')
key.set_contents_from_string('Hello World!\n')

# Generate a presigned URL
object_key = bucket.get_key('hello.txt')

# Make the url available for 20 seconds only
object_url = object_key.generate_url(20, query_auth=True, force_http=True)
print(object_url)
Enter fullscreen mode Exit fullscreen mode

Run it and you should get a similar result.



$ python s3.py 
my-new-bucket   2021-10-08T13:37:03.013Z
http://objects.lan/my-new-bucket/hello.txt?Signature=4ukxcnWqcNjbRKbF1gJ8%2FQ9eJMI%3D&Expires=1633873445&AWSAccessKeyId=05LT3KPECW7E30X2P4UO
Enter fullscreen mode Exit fullscreen mode

Test it



$ for i in {1..5}; do curl 'http://objects.lan/my-new-bucket/hello.txt?Signature=4ukxcnWqcNjbRKbF1gJ8%2FQ9eJMI%3D&Expires=1633873445&AWSAccessKeyId=05LT3KPECW7E30X2P4UO'; sleep 5; done

Hello World!
Hello World!
Hello World!
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx000000000000000000043-006162f2c5-455c5-default</RequestId><HostId>455c5-default-default</HostId></Error><?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx000000000000000000044-006162f2ca-455c5-default</RequestId><HostId>455c5-default-default</HostId></Error
Enter fullscreen mode Exit fullscreen mode

File System

Creating and testing the files system is much easier and more straightforward.

To create a file system named cfs



ceph fs volume create cfs
Enter fullscreen mode Exit fullscreen mode

Create a client to access the file system



ceph fs authorize cfs client.cfs / rw > /etc/ceph/ceph.client.cfs.keyring
Enter fullscreen mode Exit fullscreen mode

Test it



mkdir /mnt/mycephfs/test
Enter fullscreen mode Exit fullscreen mode

Now in the dashboard:
ceph dashboard showing the directory we created

Notes

  • You can send commands to Ceph clusters from a client host.

  • AWS uses SSD (as in gp2 and gp3 types) doesn't work natievly with Ceph but there's a workaround for this using ceph-volume.

  • Ceph is designed by default to be highly available so it by default expects multiple nodes with multiple disks however it can be configured to work in a single node environment.

💖 💪 🙅 🚩
akhal3d96
Ahmed Khaled

Posted on October 10, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Exploring Ceph in a multi-node setup
ceph Exploring Ceph in a multi-node setup

October 10, 2021