Infra outage...

Hi *,

sorry, but we have a networking/routing problem that makes our
services inaccessible at the moment.

Please bear with us...

ciao
Christian

Hello,

sorry, but we have a networking/routing problem that makes our
services inaccessible at the moment.

Please bear with us...

a quick update:

Systems are beginning to stabilize. An extremely big thank you to Alex, Alin, Cloph and Robert for their immediate availability and their support - I am *so* glad to have all of them on board.

It's too early to see what the issue was. Right now it looks like an internal network connectivity issue. Gluster healed itself, and one host machine rebooted unexpectedly, which killed several VMs and the HostedEngine.

The good news is, so to speak, is that the self-healing has worked, it just took a while, as in the end the systems came back without anything we actively did/could do.

About consequences and lessons learned, infra will follow-up later, definitely some changes in our approach are due now.

Florian