-LOCKED - Today's Downtime.
Posted on 2017-05-02 08:57:44
Hey all. Thank you so much for your patience today regarding the downtime.

What was the downtime for?
We received an email last week notifying us that some time over the next few months our servers would be physically moved from one location to another, and they promised us ample time to prepare for this move. We then received an email a few days later stating that our servers would be moved the following day, and that a time slot of 12 hours was designated although they predicted only 2-3 hours of actual downtime would be needed. We were alarmed that it was so short notice and the window in which the downtime could happen was so long. We luckily managed to rearrange the servers to be moved on 1st May. This was the main reason for the downtime.

So what happened?
1st May rolled around and we waited with anticipation for the servers to shut down. As both coders are in the UK we were hoping that they would take the servers down just in time for us to get online and handle the aftermath. What we didn't expect was just how messed up the aftermath would be.

What should have been a simple 3 hour job turned into a nightmare day. The actual moving of the servers was over and done with early on in the day, but when we logged in we discovered that pretty much the entire database was corrupted. We investigated the logs on the servers to find that the host had started a graceful shutdown (to those who don't know, this means that they were closing everything down before turning the server off) but as time went on we believe they became impatient with it taking so long. The startup logs showed that the "Database was not shut down normally".

We have seen almost an entire day of downtime not because of the initial move, but because we have been trying to restore all the data that was corrupted.

What does this mean?
We have essentially had to do a minor rollback of the game. We store regular backups of our most important data so lions, members and hoards are the most recent - although there may be some time disconnects as lions were backed up most recently, then before that was hoard, then before that was members, etc.

Unfortunately the one table we were unable to restore has been messages. The inbox table was completely corrupted and we have no workable backup. We are continuing to try and restore everything, and the message system will continue working as it did before, but until further notice we are unable to retrieve your messages. We will be working through the night to try and get inboxes back together, but we have no guarantee that we can get this working. We’ll do our very best however!

You may notice that there was no rollover today - this failed to run regardless as it was due to run in the middle of the time the hosts took our servers offline. Due to the large amount of downtime today, we decided not to run it again, so nobody will have a rollover until tomorrow, as you normally would.

Here is a list of the backups and when they were taken to give you an idea of what data might not quite line up for you:
* Lions may not be as 100% recent as you may have wanted if you were on and making changes to lions (claiming new, customising, breeding etc) close to when we went down. We imported a backup very close to the downtime, which was taken approximately 1 hour and 20 minutes before the server went down.
* The same goes for hoard, which is a backup taken 25 minutes before the lions backup.
* User logs were imported from a backup taken 45 minutes before hoard.
* The rest of the database is imported from a backup taken 30 minutes before the user log backup.

This means that any activity taken after these backups is now lost. We hope you can appreciate how close the backups were to downtime to avoid further loss.

Lion counts may appear incorrect for those who had new lion activity between the main database backup and the lion backup (1 hour and 40 minutes gap), but this should fix itself on your next rollover.

GB purchases are of particular importance, and any that do not show up in your purchase log on the game or appear to have been credited to your account should be reported. Please send your PMs to Katze (#3) regarding this and they’ll get recredited. Mention in the PM the time/date of purchase, transaction ID, amount bought and the email address you used via Paypal.

We’d just like to reiterate that we will be spending a considerable amount of time and effort trying to get inboxes back for you guys, but we can’t make promises. We’d also like to apologise for all of this mess - sadly it was out of our control, but we’re doing our very best to rectify everything!



Hrt Icon 0 players like this post! Like?

Edited on 02/05/17 @ 16:01:20 by Kitty (#2)

Jacimagicwhitedeer (#87017)

Amazing
View Forum Posts


Posted on
2017-05-02 11:05:02
Thank you for explanign to me what happend during the down time , and why are inboxes are empty.



Hrt Icon 0 players like this post! Like?

ModbotFanClub [Jean] (#110629)

Usual
View Forum Posts


Posted on
2017-05-02 11:07:46
Ackkk!!!!! Yes!!!! Lioden is back and running. Thank you guys so much for working so tirelessly for us! Lol every single website I've been on always has really long down times, so I'm not that surprised. Thank goodness nothing worse happened though.



Hrt Icon 0 players like this post! Like?

On Hiatus (#52897)

Usual
View Forum Posts


Posted on
2017-05-02 11:08:58
Easy for you to say, I lost a lot of important information....!



Hrt Icon 0 players like this post! Like?


𝔻𝔼𝕃𝕋𝔸
(#105908)

Impeccable
View Forum Posts


Posted on
2017-05-02 11:18:34
thanks guys! we all love you!!



Hrt Icon 0 players like this post! Like?

Red Wren 🐦 (#95058)

Prince of the Savannah
View Forum Posts


Posted on
2017-05-02 11:22:19
Kudos to the hard working staff, you all deserve a drink and a long day off.



Hrt Icon 0 players like this post! Like?

Lurgle (#49415)

Aztec Knight
View Forum Posts


Posted on
2017-05-02 11:25:45
Gosh, I have to agree with that one, @Eyebone
Thank you so much for working so hard for Lioden. I've probably never seen a more committed staff, I'm so glad we have you guys.
I'm just glad that we didn't lose Lioden as a whole.. :)



Hrt Icon 0 players like this post! Like?

Rogue (#93122)

Impeccable
View Forum Posts


Posted on
2017-05-02 11:27:11
I'm glad u decided not to do rollover today and it wont count towards our login bonuses



Hrt Icon 0 players like this post! Like?

SeptSapphire (#111765)

Divine
View Forum Posts


Posted on
2017-05-02 11:27:58
Thanks for the work you all put into getting us back online and restoring what you could. I've seen server moving nightmares before that had similar, if not worse, consequences (for one, the site came back up with all of our creatures from the last 6 months or so just wiped out due to a lack of backups), and I think you handled all this well, everything considered. We all appreciate the time you put into getting us back online!



Hrt Icon 0 players like this post! Like?

Jacimagicdeer(side) (#108127)

King of the Jungle
View Forum Posts


Posted on
2017-05-02 11:35:57
I'm glad that my lions are okay on both of my acocunts.



Hrt Icon 0 players like this post! Like?

Ankokou (#3124)

Merciful
View Forum Posts


Posted on
2017-05-02 11:38:27
Thank you for all your hard work, I'm so sorry this has happened to you guys.

You did the best you can, it isn't your fault the server host is awful (seriously, what the hell? That's extremely unprofessional on their part and completely out of your hands.)

I hope everything is running smoothly now and you get some well deserved rest.
Its good you had such a dedicated backup schedule!



Hrt Icon 0 players like this post! Like?


Edited on 02/05/17 @ 18:39:53 by Ankokou (#3124)

Nebula (#100935)

Vicious
View Forum Posts


Posted on
2017-05-02 11:43:12
thank you for all the time you spent getting the server back on track <3



Hrt Icon 0 players like this post! Like?

cool name (#92019)

Scourge of Lions
View Forum Posts


Posted on
2017-05-02 11:50:33
Ah man, it's a shame it all happened and you guys had to spend so much time repairing stuff, but thank you so much for the hard work and getting Lioden back for us to play! I bet every player appreciates that. Even if the old messages won't be back, you still did great job, so thanks again! Hope something like that never happens again and we can complain about just minor stuff as always. <3



Hrt Icon 0 players like this post! Like?

Indy .: Hard Mode :. (#86046)

Protector
View Forum Posts


Posted on
2017-05-02 12:18:02
Thank you so much for your hard work! I'm sorry it's been so rough. I hope everyone is understanding.



Hrt Icon 0 players like this post! Like?

ℍ𝕒𝕕𝕖𝕤 (#98157)

Total Chad
View Forum Posts


Posted on
2017-05-02 12:49:16
Thank you guys so much for all the time and work you've put and will be putting into getting the site back on it's feet and keeping us updated and entertained on facebook! I've never run into a more dedicated group of admins and I'm really glad to have been a part of a site so well looked after!



Hrt Icon 0 players like this post! Like?

cleverest [main] G2
2Ros Nacre (#106440)

Lone Wanderer
View Forum Posts


Posted on
2017-05-02 12:53:23
Glad to see Lioden back up!!
You all have done such a wonderful job, spending all this time fighting to get everything back up as soon as possible. Thank you so much for you dedication!



Hrt Icon 0 players like this post! Like?



Topic is locked




Memory Used: 633.73 KB - Queries: 2 - Query Time: 0.00084 - Total Time: 0.00595s