![]() Mon-secret: AQAhT19dlUz0LhBBINv5M5G4YyBswyU43RsLxA= Key = AQBZR19dbVeaIhBBXFYyxGyusGf8x1bNQunuew=Įventually, we discover more secrets in the rook-ceph-mon ConfigMap: kind: SecretĪdmin-secret: AQAhT19d9MMEMRGG+wxIwDqWO1aZiZGcGlSMKp=įsid: ZmZiYjliZDMtODRkOS00ZDk1LTczNTItYWY4MzZhOGJkNDJhCg= Keeping on looking… For example, here is the rook-ceph-mgr-a-keyring secret: Now let’s analyze the contents of the rook-ceph-admin-keyring secret: kind: Secret Key = AQAhT19d9MMEMRGG+wxIwDqWO1aZiZGcGlSMKp= Key = AQAhT19dlUz0LhBBINv5M5G4YyBswyU43RsLxA= Upon decoding it, we’ll get the regular keyring with permissions for the administrator and monitors: Let’s take a closer look at the contents of the rook-ceph-mons-keyring secret: kind: Secret var/lib/ceph/mon/ceph-a from ceph-daemon-data (rw) etc/ceph/keyring-store/ from rook-ceph-mons-keyring (ro) Type: HostPath (bare host directory volume) Type: Secret (a volume populated by a Secret) Type: ConfigMap (a volume populated by a ConfigMap) Rook mounts the following components into the monitor’s pod: Volumes: This step is necessary to unmount all mounted RBD images since a regular reboot will not work in this case (the system will be unsuccessfully trying to unmount images normally).Īs you know, the running monitor daemon is the prerequisite for any Ceph cluster. You can do it with sysrq (or “manually” in your data center). To proceed, we have to do a hard reboot of all servers with mounted RBD images ( ls /dev/rbd*). NB: In new versions of Rook (after this PR was accepted), ConfigMaps ceased to be an indicator of the successful cluster deployment. They are being created upon the successful deployment of the cluster. A bit of Rook internals, or The long way Looking around and restoring Ceph monitorsįirstly, we have to examine the list of ConfigMaps: there are required rook-ceph-config and rook-config-override. As you know, there are two types of administrators: those who do not yet use backups, and those who have painfully learned to use them always (we’ll talk about this in a bit). Obviously, there is a shorter and proper way: you can use backups. Let’s start with a longer and more entertaining way by investigating Rook internals and restoring its components manually step by step. Why? The rook-operator has suddenly decided to make a new cluster! So, how do we restore an old cluster and its data? Now it’s time to answer the question, when was the rook-ceph-operator pod started for the last time? It turns out this happened quite recently. Neither monitors nor OSD/MGR pods are operational in the cluster.You cannot read them which means that monitors are unavailable This suggests that something is wrong with RBD images mounted on nodes. Commands like lsblk and df do not work on Kubernetes nodes. ![]() New pods cannot mount RBD images from Ceph. ![]() You’ve been pleased with its operation, and then at some point, this is what happens: Imagine that you have configured and started Rook in your K8s cluster. To add some spice to this story, let us suppose that we have just experienced a (hypothetical) problem with the cluster… Skating on thin ice We hope this article will help you to avoid many of those complexities before they manifest themselves. However, this simplicity brings some complexities. ![]() We have already explained how/why we like Rook: working with some kinds of storage in the Kubernetes cluster becomes a lot easier. By Ruslan Baimuhametov, software engineer
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |