Deploy Ceph Cluster using Ceph-Ansible

Ceph is an open source distributed storage system that can simultaneously provide object, block, and file storage. As a result, it is widely used in OpenStack deployments, serving as the backend for Swift, Cinder, and Manila. This article will explain how to deploy a Production-Ready, High-Availability Ceph Cluster using Ceph-Ansible.

Basic Architecture

A basic Ceph Cluster includes three components: Ceph Monitor, Ceph OSD, and Ceph Manager Daemon

Ceph Monitor

The Ceph Monitor is the core component of Ceph. It maintains a real-time map of the entire cluster, including information on monitors, managers, OSDs, and CRUSH. These data are essential for Ceph's operation. Additionally, the Monitor handles authentication between daemons and clients. A High-Availability Ceph Cluster requires a minimum of three Monitors

Ceph OSD

The Ceph OSD is responsible for storing data, managing data replication, recovery, and rebalancing. It also sends heartbeat signals to other Ceph OSDs, providing status information to the Monitor and Manager. A High-Availability Ceph cluster requires a minimum of three OSDs

Ceph Manager Daemon

The Ceph Manager Daemon is typically deployed alongside the Ceph Monitor on the same nodes. It monitors the overall status of the Ceph Cluster, including storage utilization, performance metrics, and system load. Its main purpose is to provide additional visibility and simplify external monitoring and management. Since Ceph Luminous, it has become an essential part of the cluster. A High-Availability Manager Daemon requires two nodes

From the above description, it is clear that we need a minimum of three nodes to deploy a High-Availability Ceph Cluster

network

Ceph's network architecture is relatively simple, consisting of a cluster network and a public network

Cluster Network

The Cluster Network is used by Ceph nodes to transfer data during replication and recovery. It is typically recommended to use a 10G or higher network interface

Public Network

The Public Network is used by Ceph to serve data to external services (e.g., Cinder). It is also typically recommended to use a 10G or higher network interface

Preparation

First, we need to download the Ceph-Ansible repository and switch to the stable branch

git clone https://github.com/ceph/ceph-ansible
cd ceph-ansible
git checkout stable-3.1
cp site.yml.example site.yml

Install Ansible

Ansible can be installed directly using the package manager of the Linux distribution.

CentOS

sudo yum install epel-release
sudo yum install ansible

Ubuntu

sudo apt install ansible

Environment Setup

All Ceph-Ansible configuration files are located in group_vars/<code> 下，預設會提供 </code>sample<code> 結尾的範例檔，會需要把 sample 拿掉。簡單的部署需要設定的檔案只有 </code>all.yml<code> 跟 </code>osds.yml

cp group_vars/all.yml.sample group_vars/all.yml
cp group_vars/osds.yml.sample group_vars/osds.yml

all.yml Configuration

all.yml The default values are basic settings commonly used across all nodes in the cluster

ceph_origin: repository
ceph_repository: community
ceph_stable_release: luminous
public_network: "192.168.113.0/24"
cluster_network: "172.168.113.0/24"
monitor_interface: eth1

ceph_originCeph installation method, which allows choosing among repository, distro, and local installation options

• repository: use Ceph upstream repository
• distro: use Ceph from Linux distribution packages
• local: use locally compiled Ceph binary

ceph_stable_release Ceph's release version is codenamed Luminous LTS, and the latest version available is Mimic

public_network<code> 與 </code>cluster_network As described in the network topology above, input two CIDR blocks for the network interfaces

monitor_interface This is the interface used by the monitor nodes

osds.yml Configuration

osds.yml The default settings are some OSD parameters, usually applied across the entire OSD daemon

devices:
  - '/dev/sdb'
  - '/dev/sdc'
osd_scenario: collocated

devices<code>: 用來儲存的裝置，可以定義多個，如果每個 node 並不相同的話，可以嘗試使用 </code>osd_auto_discovery<code>，將其設為 </code>true。

osd_scenarioOSD deployment modes include collocated, non-collocated, and LVM

• collocated<code>: 將 </code>ceph data<code>、</code>ceph block<code>、</code>ceph block.db<code>跟</code>ceph block.wal placed on the same deployment.
• non-collocated<code>: 會將 </code>ceph data<code>跟</code>ceph block<code> 放在 </code>devices<code> 上，並且將 </code>ceph block.db<code>跟</code>ceph block.wal<code> 放在額外設定的 </code>dedicated_devices .
• lvm<code>: 需要設定 </code>data<code>、</code>wal<code>跟</code>db<code> 的 </code>lv name<code> 跟 </code>vg group, only data is required.

Ansible Inventory Configuration

Ansible Inventory is based on the selected hostnames or IP addresses of each node; this example will include three Ceph nodes ceph01, ceph02, ceph03<code>，所有 ceph 元件在這三個 node 上皆會安裝。會將以下資訊寫入 </code>hosts File content

[mons]
ceph01
ceph02
ceph03

[osds]
ceph01
ceph02
ceph03

[mgrs]
ceph01
ceph02
ceph03

# 實際部署

實際部署非常簡單，只需要跑 Ansible-Playbook 即可，指令如下：

```bash
ansible-playbook -i hosts site.yml

If the deployment completes without issues, you can connect to one of the Ceph nodes and use commands to check the Ceph cluster status:

ceph -s

A correct display of HEALTH_OK indicates a successful Ceph cluster deployment!

Reference

Ceph Ansible Documentation

---

Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 4.0 unless stated otherwise.