Deploying a Ceph Cluster using Ceph-Ansible

Ceph is an open-source distributed storage system that provides object, block, and file system storage simultaneously. As a result, it is widely used in OpenStack deployments to provide backends for Swift, Cinder, and Manila. This article will introduce how to deploy a production-ready, high-availability Ceph cluster using Ceph-Ansible.

Basic Architecture

A basic Ceph cluster essentially consists of three components: Ceph Monitor, Ceph OSD, and Ceph Manager Daemon.

Ceph Monitor

The Ceph Monitor is a core component of Ceph, responsible for maintaining maps of the cluster state, including the states of monitors, managers, OSDs, and CRUSH. This data is essential for Ceph's operation. Additionally, the Monitor handles authentication between daemons and clients. A high-availability Ceph cluster requires at least three Monitors.

Ceph OSD

Ceph OSDs are responsible for storing data and handling data replication, recovery, and rebalancing. They also provide monitoring information to the Monitor and Manager by checking the heartbeats of other Ceph OSDs. A high-availability Ceph cluster requires at least three OSDs.

Ceph Manager Daemon

The Ceph Manager Daemon is typically installed on the same nodes as the Ceph Monitor. It monitors the overall status of the Ceph cluster, including storage utilization, performance, and system load. Its primary purpose is to provide additional monitoring and interfaces to facilitate the setup of external monitoring and management systems. Starting from Ceph Luminous, it has become a mandatory part of the cluster. A high-availability Manager Daemon setup requires two nodes.

Based on the introduction above, we can see that at least three nodes are required to deploy a high-availability Ceph cluster.

Network

Ceph's network architecture is fundamentally simple, consisting of a cluster network and a public network.

Cluster Network

The Cluster Network is used for data backup and replication between internal Ceph nodes. It is generally recommended to use a network interface of 10G or higher.

Public Network

The Public Network is used when Ceph provides data to external services (e.g., Cinder). It is also generally recommended to use a network interface of 10G or higher.

Prerequisites

First, we need to download the Ceph-Ansible source code and switch to a stable branch.

git clone https://github.com/ceph/ceph-ansible
cd ceph-ansible
git checkout stable-3.1
cp site.yml.example site.yml

Install Ansible

You can install Ansible directly using the package manager of your Linux distribution.

CentOS

sudo yum install epel-release
sudo yum install ansible

Ubuntu

sudo apt install ansible

Environment Configuration

All Ceph-Ansible environment configuration files are located in group_vars/<code> 下，預設會提供 </code>sample<code> 結尾的範例檔，會需要把 sample 拿掉。簡單的部署需要設定的檔案只有 </code>all.yml<code> 跟 </code>osds.yml

cp group_vars/all.yml.sample group_vars/all.yml
cp group_vars/osds.yml.sample group_vars/osds.yml

all.yml Configuration

all.yml defines some basic parameters, which are typically applied to every node in the entire cluster.

ceph_origin: repository
ceph_repository: community
ceph_stable_release: luminous
public_network: "192.168.113.0/24"
cluster_network: "172.168.113.0/24"
monitor_interface: eth1

ceph_origin: The Ceph installation method; you can choose from three options: repository, distro, or local.

• repository: Use the Ceph upstream repository
• distro: Use the Ceph package from the Linux distro
• local: Use locally compiled Ceph binaries

ceph_stable_release As the name suggests, this is the Ceph release version. Luminous LTS is used here, while the latest version is currently Mimic.

public_network<code> 與 </code>cluster_network As described in the network architecture section above, enter the CIDRs for the two network segments.

monitor_interface is the interface used for monitor communication.

osds.yml Configuration

osds.yml defines some OSD parameters, which are typically applied across the entire OSD node.

devices:
  - '/dev/sdb'
  - '/dev/sdc'
osd_scenario: collocated

devices<code>: 用來儲存的裝置，可以定義多個，如果每個 node 並不相同的話，可以嘗試使用 </code>osd_auto_discovery<code>，將其設為 </code>true。

osd_scenario: OSD deployment methods, with three options: collocated, non-collocated, and lvm.

• collocated<code>: 將 </code>ceph data<code>、</code>ceph block<code>、</code>ceph block.db<code>跟</code>ceph block.wal Placed on the same device.
• non-collocated<code>: 會將 </code>ceph data<code>跟</code>ceph block<code> 放在 </code>devices<code> 上，並且將 </code>ceph block.db<code>跟</code>ceph block.wal<code> 放在額外設定的 </code>dedicated_devices On.
• lvm<code>: 需要設定 </code>data<code>、</code>wal<code>跟</code>db<code> 的 </code>lv name<code> 跟 </code>vg group, only data is a required field.

Ansible Inventory Configuration

Fill in the Ansible Inventory with the hostnames or IP addresses of your chosen nodes; this example includes three Ceph nodes. ceph01, ceph02, ceph03<code>，所有 ceph 元件在這三個 node 上皆會安裝。會將以下資訊寫入 </code>hosts Inside the file

[mons]
ceph01
ceph02
ceph03

[osds]
ceph01
ceph02
ceph03

[mgrs]
ceph01
ceph02
ceph03

# 實際部署

實際部署非常簡單，只需要跑 Ansible-Playbook 即可，指令如下：

```bash
ansible-playbook -i hosts site.yml

If the process completes without issues, you can connect to one of the Ceph nodes and use the command to check the Ceph cluster status:

ceph -s

If HEALTH_OK is displayed correctly, it means the Ceph cluster has been successfully deployed!

Reference

Ceph Ansible Documentation

---

Copyright Notice: All articles on this blog are licensed under CC BY-NC-SA 4.0 unless otherwise stated.