What happened!??

[Schweddy]

Short version: The ISP was painting their datacenter and the painter accidentally 'elbowed' the emergency power off switch. Databases do not like instant power off. It needed to be repaired. We lost the posts while the server was confused.

Long technical version: The above statement, plus... During the data recovery the server had mixed tokens and session ID's in the system after reboot. That enabled a few to be able to log in and possibly post. However, I was unable to turn off the forum software as the switch is in the database. We were also unable to view/update via phpmyadmin. During this time, the troubleshooters attempted repair from daily and weekly backups and were unsuccessful (due to the errors stated previously). Command line on the system appeared great resulting in the technicians having quick judgments to the current status. When attempting to contact the ports for the db via 80, it would go nuts. The errors (5 of them) combined with our multiple cache system made it even more of a challenge to find the root cause.

Results: The database was eventually recovered from the latest daily backup successfully. Posts after 2:30am Saturday are lost.

Please accept our apology and thank you for your patience during the last two days.

 

[RDS]

Glad it's back to normal now. Thanks a million.

 

[Carjosse]

My donations seem to have disappeared aswell on the bar and on my profile apparently.

 

[Schweddy]

My donations seem to have disappeared aswell on the bar and on my profile apparently.

Would you PM me the details and I will make sure it gets into the system.

 

[Dr. Chuckles]

The database was eventually recovered from the latest daily backup successfully. Posts after 2:30am Saturday are lost.

12345

 

[soccerboy22]

Here is my theory: "So because the Cowboys looked like they were doing well the world was starting to end and the beginning of the end was DP shutting down. Now that the Cowboys lost everything in the world is correcting itself and therefore DP is returning."

 

[Superfly]

I donated $1.5 million and was made Supreme Benevolent Moderator Extraordinaire.

Just wanted to refresh your memory in case you forgot.

 

[APACHERAT]

[ Posts after 2:30am Saturday are lost.

.

I'm guessing all of the post after the 26th (Thursday) were lost because all of my post I posted on the 27th and 28th and the thread I started on the 27th seem to have disappeared into a black hole.

 

[SheWolf]

Thanks for the explanation. I thought a bunch of the posts were gone

 

[Josie]

I was having a slight meltdown. I'm so happy to see you all again!

 

[polgara]

I'm guessing all of the post after the 26th (Thursday) were lost because all of my post I posted on the 27th and 28th and the thread I started on the 27th seem to have disappeared into a black hole.

:agree: The backlog must have been enormous!

Good evening, APACHERAT! :2wave: Glad to see you made it!

 

[MaggieD]

My donations seem to have disappeared aswell on the bar and on my profile apparently.

Several times I thought about your generous donation and wondered what you thought about the long blackout. You still rock.

 

[DiAnna]

I'm so glad we're all back, and many thanks to the admins who gave up a large chunk of their weekend to get us up and running.

That said, I had to stifle a snort of laughter as the image of a guy in paint-stained overalls "elbowing" a power-off switch, horrified by the sounds of servers stuttering into agonized silence, followed by the anguished scream of "Arghhhhhhhh, NOOOOOOO!" coming from the other room!

...I know, I'm evil. I'm so ashamed. :3oops:

 

[clownboy]

Time to childproof the datacenter. :mrgreen:

Thanks for all the very hard work in getting us back and running.

 

[Green Balls]

This reminds me of working at a pizza place in the '90s. One of the new kids was mopping the floor. I hear a loud "Oh sh--!". I turned around to see the emergency fire foam trigger hanging by the plastic bar, almost out far enough for the system to activate. He hit the trigger with a mop handle. Luckily we pushed it back in and no harm was done, but almost a disaster.

 

[Van Basten]

Happy things are back up and running.

 

[American]

Short version: The ISP was painting their datacenter and the painter accidentally 'elbowed' the emergency power off switch. Databases do not like instant power off. It needed to be repaired. We lost the posts while the server was confused.

Long technical version: The above statement, plus... During the data recovery the server had mixed tokens and session ID's in the system after reboot. That enabled a few to be able to log in and possibly post. However, I was unable to turn off the forum software as the switch is in the database. We were also unable to view/update via phpmyadmin. During this time, the troubleshooters attempted repair from daily and weekly backups and were unsuccessful (due to the errors stated previously). Command line on the system appeared great resulting in the technicians having quick judgments to the current status. When attempting to contact the ports for the db via 80, it would go nuts. The errors (5 of them) combined with our multiple cache system made it even more of a challenge to find the root cause.

Results: The database was eventually recovered from the latest daily backup successfully. Posts after 2:30am Saturday are lost.

Please accept our apology and thank you for your patience during the last two days.

And your apologizing why? Geez, it's a messageboard, something we can all live with. You do good work, and it's voluntary.....and much appreciated.

 

[The Mark]

There's an emergency power-off switch or 3 around the edges of the datacenter I work at....not sure they're connected though.
They're orange, surrounded by alternating black/yellow slashes and have clear plastic covers to prevent accidents.

 

[DDD]

I'm guessing all of the post after the 26th (Thursday) were lost because all of my post I posted on the 27th and 28th and the thread I started on the 27th seem to have disappeared into a black hole.

What the heck did you posted to have caused all of this!!?

 

[The Mark]

What the heck did you posted to have caused all of this!!?

Umm....Read op

 

[DDD]

Umm....Read op

I did! The thing is the OP does not states or even hints that Apacherat posted NSA sensitive information that caused it to appear that DP had somehow "accidentally" switched the power off button! It is a conspiracy!

Hence now all his posts together with our posts (lateral damage) were removed from Saturday!

 

[The Mark]

I did! The thing is the OP does not states or even hints that Apacherat posted NSA sensitive information that caused it to appear that DP had somehow "accidentally" switched the power off button! It is a conspiracy!

Hence now all his posts together with our posts (lateral damage) were removed from Saturday!

Oh...

 

[minnie616]

When I click read my posts I only get the posts I posted after the board was up and up and running again last night the rest of the posts are posts from many other members.
Is anyone else having the same problem?

 

[CaptainCourtesy]

I save a copy of pretty much every post I make for just such an emergency as what happened.

 

[DDD]

I save a copy of pretty much every post I make for just such an emergency as what happened.

For what? Re-posting them?