View previous topic :: View next topic |
Author |
Message |
Nycticora
Joined: 09 Feb 2013 Posts: 2277
|
Posted: Tue Jun 25, 2019 5:06 am Post subject: Disaster in progress |
|
|
Hi all,
There was a huge disaster today. The server the game runs on is basically broken, and can't be fixed. It's in a half-working state right now.
Basically what happened was that our hosting provider wanted to update the machine to apply a security fix. But the operating system itself, Debian, is broken upstream. The attempt to update the package broke the installation, and it can't be fixed because the stable version of Debian is broken. I tried skipping the stable version and jumping to the testing version, but that isn't really a valid thing to do and broke the system even harder than it was before.
Luckily Davairus pays extra for backups and we were able to restore the game from a backup. Progress made in-game today during the day was lost, but any progress from yesterday before the crash or earlier is intact.
The server is still very broken and if there's a reboot it will not start without intervention (it needs /sbin/ifup eth0 on every boot because startup scripts are totally hosed).
We'll need to switch to another operating system entirely as Debian is no longer usable. We'll probably have to hold off on that until either Davairus or I get some free time. The game's in a highly volatile state right now and we will need to have some extended downtime soon.
Sorry about the missing forum messages and the loss of progress this afternoon. |
|
Back to top |
|
|
Davairus Implementor
Joined: 16 Jan 2004 Posts: 10368 Location: 0x0000
|
Posted: Tue Jun 25, 2019 8:23 am Post subject: |
|
|
Luck favors the prepared!
I know everybody appreciates the effort. Throwing down a big thank you for lending the rad skills |
|
Back to top |
|
|
Nycticora
Joined: 09 Feb 2013 Posts: 2277
|
Posted: Tue Jun 25, 2019 1:27 pm Post subject: |
|
|
I think the game is stable for now, the plan is we'll bring up another server on a different OS (Slackware) and move the game over to it. We'll probably announce a brawl day soon to test it out. |
|
Back to top |
|
|
Cartsel
Joined: 09 Jul 2016 Posts: 9
|
Posted: Wed Jun 26, 2019 3:22 am Post subject: Mad props |
|
|
The amount of work that went into getting things to the stage it is now, I cannot say enough about all y'all have done. i.e. you can actually play the game. Thanks for all the hard work! |
|
Back to top |
|
|
Olyn Immortal
Joined: 23 Jul 2008 Posts: 3252 Location: Pennsylvania
|
Posted: Wed Jun 26, 2019 10:49 am Post subject: |
|
|
Scheduled maintenance at Linode may bring us down overnight. It is scheduled to begin tomorrow at 7am Eastern / 4am Pacific. Downtime should be minimal, but we may need to start back up manually like we did a few days ago. |
|
Back to top |
|
|
Davairus Implementor
Joined: 16 Jan 2004 Posts: 10368 Location: 0x0000
|
Posted: Wed Jun 26, 2019 6:21 pm Post subject: |
|
|
Hold onto your butts... |
|
Back to top |
|
|
Nycticora
Joined: 09 Feb 2013 Posts: 2277
|
Posted: Mon Oct 21, 2019 1:38 pm Post subject: |
|
|
Last night's outage was due to this - our host rebooted our machine to apply some updates and it didn't come back up until I kicked some services manually. |
|
Back to top |
|
|
|