Past Incidents

Due to temporary problems in the morning of the 9th of January, we were not able to create snapshots for our customer servers. The problems were already resolved and all services run as usually.

We are currently experiencing a failure of a cloud node, which is responsible for the NWS-SaaS apps. The problem has been identified and the affected apps are in the process of being re-deployed.

Start Outage: 11:12 CET

End Outage: 14:35 - 16:00 CET depending on machine

11:25 One of our hypervisors is unable to start. Ceph-based machines have already been migrated to other hypervisors and are up and running again. SSD-based machines are currently locked to the affected host. Our technicians are on their way to the data-center to carry out tests directly on the hypervisor. We will inform you immediately if progress can be foreseen.

12:00 The on-site analysis showed that the host is dead; we are currently trying to rescue the SSD images and copy them to other hypervisors.

13:15 In the meantime the SSDs have been removed from the defective hypervisor, we are now trying to install them in another server and migrate the data.

14:30 We found a way to access the images. We are currently copying the images to other hypervisors and trying to start the affected machines

15:30 Most of the machines are already running on other hypervisors. The remaining machines are still being copied and started. Customers with faulty machines will be contacted again separately. No snapshots were imported, but the images were restored from earlier today (latest data on machines).

16:06 All machines have been restored

We are facing some hardware related disk issues on one of our Cloud nodes. Our Engineers have already identified the issue and are fixing virtual machines that have been affected. We apologize for the inconvenience.