I have 3 servers that I want to use as a storage cluster in a PoC Openstack set up.
Each server has a small /dev/sda for the OS, etc. and a large /dev/sdb I hope to use as storage by Ceph.
I can use the Juju Ceph charm to deploy Ceph to each server.
As expected the Ceph charm sets up a Ceph Monitor on each server and reports back that a cluster with quorum has been formed.
The docs say that the main Ceph charm also sets up Ceph-OSD for any disks it finds. In my case it only sets up an OSD on a single server.
Have I understood the main Ceph charm correctly? Should it set up the 3 Ceph Mons and also 3 Ceph-OSDs (configured for /dev/sdb)?
My Ceph knowledge is minimal so I'm still piecing together the terminology and concepts. Am I correct in thinking that a single Ceph-OSD can manage block AND object storage or does Ceph allocate some OSDs solely for block and others solely for object storage?
2 Answers
Would you mind posting the config file that you used for your deployment? That would be helpful.
Given your details about the hardware I would think that the charm would setup 3 OSD's (1 per server on /dev/sdb).
You're correct about Ceph managing a single block and storage pool.
This can appear slightly confusing at first, the 'ceph' charm sets up both a mon as well as an OSD. The ceph-osd charm only sets up an OSD (no mon), but otherwise works the same and with the same options. You will mix them depending on your deployment, and you can relate the ceph and ceph-osd charms to each other.
For a minimal cluster of 3 servers (such as you have), you need 3 mons and thus you would simply create 3 units of the 'ceph' charm which will configure both the required mons & an OSD on each unit. If you had more servers (e.g. 6) you might setup 3 units of ceph and 3 units of ceph-osd.
The storage used by ceph can be configured with the osd-devices option on the charm, it can accept either device path(s) in /dev or a filesystem location to place the files which defaults to /dev/vdb - you'll likely need to change it. Refer to the documentation here:
Lastly with regards to block and object storage, ceph at a lower level provides a "rados" service which stores objects and is accessed using a low level rados API. Within the Ceph cluster (which has multiple OSDs) you can create multiple 'pools' to hold those objects. Each pool has a name, and the names of objects are unique within a pool (but you can use the same object name in different pools, they are isolated from each other) -- you can also set various options on each pool, such as the number of copies to store each piece of data. Be mindful though, there is an overhead to each pool (largely in terms of placement groups) and you can't create them wildly, but you are likely to have a few pools in your installation.
On top of that Ceph cluster, you can build a number of services including block storage (using RBD - Rados Block Device) and object storage with a REST API (using radosgw). Both of these services use the rados API at the low level for storing and manipulating the objects, but provide a higher level service such as a block device or HTTP REST interface to the objects.
With the object storage gateway (radosgw) your objects are generally named and created in the same way they are within the gateway, which may be obvious, however the RBD rados block device splits the storage volume up into chunks (e.g. 4MB chunks) and creates an object for each chunk and some additional objects which store metadata about the entire block device.
You can run both services (or more) on the same ceph cluster and OSDs, but will generally put each service into it's own pool (radosgw actually uses several pools).
I hope that helps get you started with the charm itself, some basics about ceph and how both block and file storage can co-exist on a ceph cluster. It may still be slightly confusing in terms of the terminology still, so I'd suggest having a browse through the manual and search youtube for one of the various talks introducing ceph which will really help you get a handle on the terminology.
2
