• Hi Guest - Come check out all of the new CP Merch Shop! Now you can support CigarPass buy purchasing hats, apparel, and more...
    Click here to visit! here...

We're back...

Rod

Rod

Administrator
Staff member
Joined
Jan 4, 2001
Messages
10,333
First, let me apologize for the downtime that occurred today. I had our forum vendor perform an upgrade to the gallery, however something went wrong during the process which corrupted the database.

After having spent a few hours trying to fix the issue, I decided it was best to perform a restore from 3:17AM this morning. Due to the backup being on a server offsite, it took 7 hours to pull the backup and then perform the restore. At around 6:30 PM everything seemed to be working great, until the members table in the SQL database became corrupt. No big deal, I ran a repair on the db, which resolved the issue.

At around 7:30 PM I was reading a PM, and noticed that one of the members on the PM had been deleted. Only a few of his posts were left behind, however all identifying data was removed. Turns out, when I repaired the db, this member account had become corrupt. When the repair ran, it removed the account.

At around 8:15 PM I decided enough was enough. I decided to pull a backup from Friday night, and restore that one. Best to go back to a date that I knew was stable.

Long story short, I restored a backup that is 4 days old. I'd rather make 100% sure that all data is intact before it's too late, so we lose 4 days worth of posts. Upside? We know our db is functioning and intact. This time next month, no one will even notice the few days of missing data.

So there you have it. I need a break... and a beer...
 
Beers and cigar for ROD!

I heard you had a nice Punch ready for the occasion :rolleyes: hehehehe
 
Your efforts are appreciated, Rod. Thank you much.
 
Thanks for all your hard work Rod its very much appreciated
 
A big Thank You! for coming to get one of your lost sheep, Rod. :thumbs:
 
broblues said:
Out of curiosity, what corrupted the site?
Click to expand...

After the restore from 3:17AM was completed, I was still getting database errors. Everything seemed to clear up after about 10 min. I ran a db check just to be sure, and it came back fine, so I executed a full backup (usually takes 40 min to complete). The CPU usage pegged at 100%, and during the sql dump the members table was stuck open. One of the members accounts did not close, so when I ran a repair on SQL it repaired the db by actually removing the account, and leaving remnants behind. This did not sit well with me, so I pulled an earlier restore from Friday night. That restore worked great.

I am in the middle of revamping my backup procedure, and adding another server -- this one will be in England (DR site). In case we happen to have a major server failure, I will have 3 backups to choose from. The England site will replicate daily, and make restores much more seamless and quicker. I'm trying to get away from having restores take hours, when they should take minutes. Another big advantage is that in case the data center we're housed in goes under, I will be able to connect to the England DR site and restore the entire site and db's to any data center of my choosing.

Thanks,
Rod
 
Rod said:
Long story short, I restored a backup that is 4 days old. I'd rather make 100% sure that all data is intact before it's too late, so we lose 4 days worth of posts. Upside? We know our db is functioning and intact. This time next month, no one will even notice the few days of missing data.

So there you have it. I need a break... and a beer...
Click to expand...
Oh no! does this mean that Stogieman will have a 4 day postus interuptus? He's like the Cal Ripkin of CP. He going to have to start a new continuous posting streak. :laugh:

Seriously though, thanks for all the hard work in keeping this amazing site pulled together.
 
Wait, so it's like this weekend never happened? Feels just like my life.
 
It's kinda like the movie "Ground Hog Day" :laugh:

Thanks Rod for all your work.


*edit for spelling. Check it before you post it.
 
Good to hear, Rod. I was puzzled by the disappearance of a comment I'd made and thought maybe my posting privileges had been revoked!
 
You must log in or register to reply here.
Top