CURRENT: Ceph problems in Karlsruhe

For a few days now, increased storage problems have been occurring in the bwCloud region of Karlsruhe. These problems are still ongoing at the moment (Tuesday 27.10.2020, 10 am). The bwCloud operations group has been in operation since the first reports and is in the process of identifying and solving the problems.

What exactly is affected?

Apparently there are technical problems in the area of storage (Ceph). The cause is still being determined. As a result, the entire system is exposed to an increased internal system load because ceph has "self-healing mechanisms". Among other things, these mechanisms have the effect that affected subsystems (e.g. hard disk) are avoided. As a result, the data is internally rearranged ("re-balanced"), which leads to a higher internal network load, which in turn ensures that the running VMs do not respond as quickly as usual.

Are the other bwCloud regions affected?

No, only the Karlsruhe region is affected by the storage problems.

Are data / instances in Karlsruhe affected?

We try to avoid data loss under all circumstances. The data is stored internally on the storage media in multiple redundant systems, so a loss is very unlikely. In order to guarantee the integrity of the data, our bwCloud operations group works very carefully to solve the problems - which also explains the longer processing time.

Where can I get updates and new information?

Here on this website we continuously update as soon as we have new information.