Last week, we introduced Ceph and OpenStack – Best Practices Part I, and this time we will follow up on previous suggestions to introduce a few more best practices for integrating Ceph and OpenStack. Using these additional settings can further enhance the integration of OpenStack and Ceph.
Table of Contents
Using RAW Images
In the previous article, Ceph and OpenStack – Best Practices Part I we introduced that RBD itself has a layering feature, but this feature has some limitations. Images uploaded to OpenStack must be in RAW format to utilize RBD layering. Therefore, it is recommended to convert all uploaded images to RAW format first. You can easily convert various formats using the qemu-img command:
qemu-img convert image.qcow2 image.rawAfter that, uploading the RAW format image to OpenStack will enable the use of the RBD layering feature.
Configure images to use Virtio-SCSI for VMs
Whether for ephemeral disks or volumes, Nova VMs with an RBD backend use the Virtio-blk driver by default to allow guests to access RBD images. In this case, VirtIO provides a paravirtual I/O bus, and devices are named /dev/vda, /dev/vdb, and so on. Although VirtIO block devices are lightweight and efficient, they do not support the discard command.
The inability to use discard means the guest cannot use the discard option when mounting disks, which also means the system cannot use fstrim to clear blocks that have already been freed. This causes some security issues because the data users intend to delete is not actually deleted or overwritten. In addition to security concerns, this also creates operational problems.
Lack of discard support also means that the RADOS objects of an RBD image will not be deleted throughout the image's entire lifecycle—they will persist until the entire image is deleted. Consequently, there may be tens of thousands of RADOS objects in the Ceph cluster that exist but will never be used.
Fortunately, another VirtIO disk driver supports the discard command: the VirtIO SCSI controller, or virtio-scsi.
To enable the VirtIO SCSI controller on OpenStack, you need to configure some image properties in Glance, specifically hw_scsi_model and hw_disk_bus. This can be easily set via the OpenStack CLI command:
openstack image set
--property hw_scsi_model=virtio-scsi
--property hw_disk_bus=scsi
<image 的名稱或 ID>Or, if there are multiple images, you can use a simple script:
for i in $(openstack image list | awk '{print $2}' | sed "s/ID//g");
do
openstack image set --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi \
--property hw_qemu_guest_agent=yes --property os_require_quiesce=yes $i
doneAt this point, if you create an instance using this image, you will find that the device names, previously /dev/vdX, will change to /dev/sdX, and you will also gain access to all the information from the SCSI stack. For example, /proc/scsi/scsi will be available to provide information about the SCSI bus and controller, and you can use lssci to identify LUs, etc.
Not only is the boot volume provided to the instance via the SCSI controller, but all subsequently attached volumes will also use the SCSI controller. Consequently, these volumes can also utilize the discard operation for free block reclamation.
Differentiate Ceph Pools using Cinder multi-backend and Volume Types
When managing two Ceph pools simultaneously—one All-flash and one HDD—Cinder multi-backend and Volume Types are required to distinguish between them. This involves modifying the volume backend settings in cinder.conf:
[DEFAULT]
enabled_backends=rbd-hdd,rbd-ssd
scheduler_driver=cinder.scheduler.filter_scheduler.FilterScheduler
default_volume_type = hdd
[rbd-hdd]
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
backend_host=rbd:volumes
rbd_pool=volumes
volume_backend_name=rbd-hdd
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid = {{ cinder_rbd_secret_uuid }}
[rbd-ssd]
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
backend_host=rbd:volumes
rbd_pool=volumes-ssd
volume_backend_name=rbd-ssd
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid = {{ cinder_rbd_secret_uuid }}In the configuration above, we have enabled two Cinder backends:rbd-ssd<code> 跟 </code>rbd-hdd<code>。</code>rbd-ssd<code> 會使用 Ceph 的 </code>volumes-ssd<code> 這個 pool，而 </code>rbd-hdd<code> 使用 </code>volumes
Finally, use the OpenStack CLI to create the corresponding volume types for the different backends:
openstack --os-username admin --os-tenant-name admin volume type create ssd
openstack --os-username admin --os-tenant-name admin volume type set ssd \
--property volume_backend_name=rbd-ssd
openstack --os-username admin --os-tenant-name admin volume type create hdd
openstack --os-username admin --os-tenant-name admin volume type set hdd \
--property volume_backend_name=rbd-hddYou can then specify the volume type when creating a volume to utilize the corresponding Ceph pool.
Implement QoS using Cinder Volume Types
Volume types are typically used to distinguish between different Cinder backends—for instance, allowing users to choose between Ceph RBD or NetApp for volume creation. Furthermore, administrators can leverage this feature to configure different QoS profiles on the same backend. Cinder QoS can limit parameters such as maximum IOPS or throughput for a volume; these can be configured via the OpenStack CLI or the Horizon Dashboard.
To limit sequential read/write speeds to 150MB/s and random read/write speeds to 1000 IOPS:
openstack volume qos create \
--consumer front-end \
--property total_bytes_sec=$((100<<20)) \
--property total_iops_sec=1500 \
"volume_qos"Next, you will need to define a corresponding volume type and apply the QoS specification:
openstack volume type create --public "volume_qos"
openstack volume qos associate "volume_qos" "volume_qos"Finally, the volume type can be set to public or restricted to specific projects.
讀 Part I
Reference
- Additional recommendations for Ceph and OpenStack
- OpenStack Documentation: Configuring multiple storage backends
Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 4.0 unless otherwise stated.
