in short: our server crashed the other day and we’re not really sure why, so we had to restore a backup
it went down at about 1:40 pm yesterday and we gave it until early this morning to diagnose what the problem was ourselves and get feedback from our server host (Digital Ocean) on what the problem might be. unfortunately we couldn’t do that and none of their suggestions helped, so this morning we pulled the trigger on restoring the backup. as a consequence a few days of posts and registrations are now lost.
so: if you registered some time in the past 7 days, you will need to re-register.
(as a registration aside, we approve registrations very quickly so if you have registered your account should be approved within 24 hours of joining the site)
it’s unfortunate to lose all the posts and registrations of a week but we don’t pretend to have 100% uptime (nor do we really have the funding to guarantee that, lol) so it is what it is. thanks for using the site, folks, hopefully this won’t happen again anytime soon
Curiously, because I access Beehaw through my own instance, I didn’t notice. Those posts and replies are still there in my database. Makes me wish ActivityPub applications were truly distributed/decentralized and it could recover from this.
Damn that’s weird. feddit.de recently had a crash too which resulted in the loss of a few posts. Do we already know the reason behind the crash @wintermute@feddit.de ? (Sorry I can’t check it on Matrix. It’s annoying that Element supports only one account and I hate other clients too much, and I am too lazy to login to the web client :D)
I can add what the Digital Ocean support team told me. Namely, the server suffered from some type of “kernel panic” that is very rare. The Digital Ocean support team do not have access to our back-end user interface nor can they access our server otherwise.
They guided me, step by step, through several attempts to restore the server from this “kernel panic”. We made eight different attempts which all failed and we don’t know what caused this issue in the first place.
Luckily, we have server backups enabled and this was the last resort.
As a sysadmin i feel your pain
Oof, that’s rough. Welcome back.
https://media.tenor.com/bVUh2pUkmZMAAAAC/futurama-were-back-baby.gif
Very glad to have you back.
Curiously, because I access Beehaw through my own instance, I didn’t notice. Those posts and replies are still there in my database. Makes me wish ActivityPub applications were truly distributed/decentralized and it could recover from this.
Thanks for running Beehaw btw.
Damn that’s weird. feddit.de recently had a crash too which resulted in the loss of a few posts. Do we already know the reason behind the crash @wintermute@feddit.de ? (Sorry I can’t check it on Matrix. It’s annoying that Element supports only one account and I hate other clients too much, and I am too lazy to login to the web client :D)
The pictrs container filled up the disk with temp files.
The issue was closed, but the problem still persists.
Does it make sense to repost?
if there are articles that got eaten by the past 7 days that you think are worth reposting yeah, i’d repost them
Thanks for the updates!
… yeah, and keep up the good work 😅
I can add what the Digital Ocean support team told me. Namely, the server suffered from some type of “kernel panic” that is very rare. The Digital Ocean support team do not have access to our back-end user interface nor can they access our server otherwise.
They guided me, step by step, through several attempts to restore the server from this “kernel panic”. We made eight different attempts which all failed and we don’t know what caused this issue in the first place.
Luckily, we have server backups enabled and this was the last resort.
Was the filesystem corrupted?
More than likely.
A kernel panic may prevent a VM from booting but shouldn’t prevent accessing the disk image for recovery purpose.
Maybe the kernel panic corrupted the disk image, or maybe Digital Ocean doesn’t have a recovery mechanism that work when the VM doesn’t boot.
More than likely, this is what happened.
deleted by creator