One of our Hypervisor Nodes is in failed state. We are investigating the issue right now.
All Virtual Machines which have been created on this particular host are affected.
The server is back online. All remaining virtual machines are now running again. Due to some hardware defect it took us quite some time to tackle down the issue.
We are sorry for any inconveniences made.
We are still struggling bringing the affected node back online. We will try some more possible solutions, before we need to redeploy the whole system.
The data of the remaining non-recoverable virtual machines are probably ok.
We have been able to recover all virtual machines, that are running with ceph based root filesystems.
There are still VMs, which cannot be recovered due to local NVMe disks.
We will post updates as soon as possible.
3rd September 2020
Hypervisorsnet-openstack-hv08 hardware failure
We expiering some trouble with one of our openstack-hypervisors since 3:00 AM , so some virtual machines are not available yet. . We are investigating for a possible migration of the virtual machines to another hypervisor and fixing the hardware.