Resetting an existing BlueStore OSD

During an attempt to migrate some OSDs’ BlueStore RocksDB to a different block device, we noticed (previously undetected) fatal read errors on the existing RocksDB. The only way to recover from this situation is to remove the OSD and rebuild its content from the other copies.

There are standard procedures to delete and to create OSDs, BlueStore and FileStore. But during our transition from FileStore to BlueStore, we came across a problem where we could not specify the new OSD’s id and had other minor difficulties. And we now wanted to cause the least data movement possible. All this while replacing the RocksDB block device.

To make a long story short: We were looking for a “mkfs”-style approach. Continue reading

Posted in BlueStore, Ceph | Leave a comment

Highly available database: MariaDB with Galera in a two-node cluster

This article originates from our work with OpenStack, but Galera also can be used as a standalone solution for highly available databases, of course. It doesn’t seem that difficult to set it up reading the OpenStack HA-Guide, but of course there are some things to be aware of to get it up and running. Continue reading

Posted in High Availability, OpenStack | Tagged , , , , | Leave a comment

Obstacles for your OpenStack cloud

This short post is for all OpenStack users or administrators who like to use tiny base images to test the functionality of their OpenStack environment. I really like the very small CirrOS image, it allows you to launch instances very quickly to test basic functionality of your image, network and compute services.

But unfortunately not all of the available images are suitable for OpenStack. Some of them don’t have a (readable?) partition table which makes it very unlikely that it will boot successfully. Just check the raw image for an existing partition table before you upload it to Glance.

Just recently I wanted to test the latest release of CirrOS:

test:~ # fdisk -l Downloads/cirros-0.4.0~pre1-i386-disk.img
Disk Downloads/cirros-0.4.0~pre1-i386-disk.img: 11.6 MiB, 12139008 bytes, 23709 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

This image didn’t work in my OpenStack cloud, so I had to use an old CirrOS image:

test:~ # fdisk -l Downloads/cirros

Disk Downloads/cirros: 39.2 MiB, 41126400 bytes, 80325 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
Downloads/cirros1 * 16065 80324 64260 31.4M 83 Linux

This image was uploaded to our OpenStack cloud about two years ago and works perfectly fine, but unfortunately I can’t tell you the exact version. It was downloaded from Glance to show you the partition table, and that’s also the reason why this image is a little bigger than cirros-0.4.0.
So be aware of this if you can’t launch instances.

Posted in OpenStack, SUSE Cloud, Virtualisation | Tagged , , , | Leave a comment