Elsmar Cove BCP - Continuity of the forum when disaster strikes

[Richard Regalado]

Do we have one to ensure continuity of the forum when disaster strikes?

 

[Marc]

1. The main server has 4 drives. It uses two with a hardware RAID 1 Mirror with auto-fail over to the second drives.
2. Other than the OS, I maintain backups of every file on the server here in my home "office" (including databases) in southern Ohio.
3. The server is located in Softlayer's DAL 05 data center in Dallas, TX. It is backed up to a server in Scotland every 2 hours, 24 hour consolidation every day, monthly consolidation for 3 months. This backup includes the OS and everything on the drives.

No - There is no auto-failover to a data center in another location. I could set that up but it would be expensive. As it is, should Dallas be wiped out (hurricane, tornado, data center catches fire or whatever), I can get a server in another data center online and restore from the Scotland backup within 24 hours.

The plan has been in place since 2007.

 

[Richard Regalado]

Thank you Marc.

What if you're not around to do the backups or to get that new server? Who's your backup?

 

[Marc]

I have no backup plan for if I die or become too sick to work a keyboard. There is no one else who has any idea about all the things I do, from helping people with problems such as emails asking for help resetting their password (and many things that are well documented in the instructions forum here), to backups to here, or server maintenance I perform.

The forum would continue to run for several months at least, though, and in my Will I have 5 people named who, should they want it, would be given access to the databases and the vBulletin software.

Unfortunate, but this isn't a big money forum. I can't afford to hire someone to do everything I do. If it made that much money, believe me I'd do it and relieve myself of a lot of time I spend doing everything I do. The time I spend every day doing "what I do" is becoming more onerous every year. I'm getting old (sigh...).

If you have any suggestions, I'm open to hearing them.

 

[Richard Regalado]

We can start by documenting the tasks you're doing behind the limelight if it has not been done yet. The nominated potential heirs would appreciate having a manual of sorts.

If allowed, I'd like to open the floor for suggestions from the distinguished members of the forum who have been here longer than I am and to other BCM professionals.

 

[aliasJohnQ]

I wonder if the U.S. government has as good a system as Marc has............

 

[aliasJohnQ]

Well, Marc, I hope I can speak for everyone here, that I hope and pray you live to at least 120......Great Job!

 

[Marc]

We can start by documenting the tasks you're doing behind the limelight if it has not been done yet. The nominated potential heirs would appreciate having a manual of sorts.

If allowed, I'd like to open the floor for suggestions from the distinguished members of the forum who have been here longer than I am and to other BCM professionals.

It's not so much writing instructions for someone to follow. It's having someone with the expertise in the aspects I deal with.

Some of it appears somewhat simple. I get email requests every day for things like help with resetting someone's password. Then again, often times a company's firewall and/or email screening software screens out the forum email response. You have to have a person who knows how to recognize what's going on and to get around that.

I guess it boils down to something akin to being a pilot. You can read all the books you want, but you need to fly with an instructor for at least 40 hours (that's just for a VFR Private license) before you can safely do it on your own. As I related in another thread, I have a cousin who came to me a year or so ago wanting to start a web site. I told him he'd have to come up here and spend at least 3 full days for me to show him the basics. And from there he would have to understand it's a *job* where you keep learning every day. He did come up for a day, or most of a day, and decided it was way beyond what he thought it would be. Like many people, he thought he could throw up a web site, it would run on it's own, and the money would just start pouring in. He was so naive that he paid GoDaddy US$40 for "visibility" which he took to be getting his site ranked highly in Google and other search engines. Ha! A totally wasted 40 bucks.

Now - If you get the right person, such as an experienced "web master", they should be able to sit down at my computer here at my home (or ship the computer to them) and pretty quickly understand what's going on and what I do by looking at the files I have and by looking through my emails. I do have a spreadsheet with all my passwords, user names, software license numbers, and all that stuff. Everything is categorized in appropriately named directories. If they are really qualified, within a few days to a week the person should easily be able to understand what I do and how I'm doing it.

Now - When I say "experienced web master", I mean:

1. They can administrate a server. I did it myself for years but these days I do have a professional company doing my server admin and monitoring. Believe me - It's expensive. But to me these days it's worth the cost. I know quite a lot but I'm not a professional. It's just stuff I learned over the years. I hired a professional because things come up that I don't know how to fix. Nor am I up to date on server security - So I pay a professional company to deal with server stuff in general these days. This includes understanding the details like the domain name registrars and how to make changes including domain transfers to new owners.

2. The person has to be proficient in vBulletin software. vBulletin is a lot more complex than it appears on the surface. Think of it this way - You have Minitab software. Big deal. You may understand the math (specifically statistical analysis), but learning minitab software (which is in its self complex software that one doesn't just sit down and understand how to use right off the bat) is a whole different aspect. In short, on your end you see a fair amount of complexity when you use the forum, but you should see the admin control panel. This is not to mention all the "mods" and "plug-ins" I have installed, templates I have tailored, php files I have made changes to... (etc., etc.) Yes - I do have records of all changes I have made over the years.

3. The person has to understand databases. And the person should have some knowledge of the vBulletin database schema. The person should have experience with MySQL and know how to dump and restore databases. This is stuff you have to ssh in to do in the CLI - It's not browser based (aka GUI) stuff. The person has to be proficient enough to know what to do when a database starts throwing errors, which usually tie into a PHP issue, and how to diagnose the problem quickly.

4. The person has to understand a fair amount of PHP and Apache (not to mention nginx these days - It's running on this server as a front end which feeds Apache), especially configuration. While the person doesn't have to be a PHP coder, they have to understand the basics.

5. The person has to have some SEO optimization knowledge and follow search engine trends and have some idea what, if anything, should be changed after one of Google's major algorithm changes. I'm dealing with that right now. Traffic is lower than it "should" be. But even there - I think part of the low traffic is a factor of the increasing popularity of LinkedIn. It gets pretty complex trying to "read" all the measurables and what is happening in general to try to not only make sense of what's happening, but also what, if anything, can be done about it.

I could name a few more things but...

To some degree there is also the aspect that I do things someone else may not. My server admin thinks I'm going overboard by backing up databases here, for example. He is right - They're backing up every 2 hours to Scotland. But that's why I have some data backed up on 3 different drives here at home. It's just my way, my personality - Multiple backups are good. Backups *do* fail and unless they check every one (which is time consuming and expensive) they won't know until the disaster occurs and they try to load the backup on a new server and find the backup is bad. Since I back up at least once a week, there is the potential to lose up to a week of posts should the server admin's backup fail. But - That's better than losing posts going back to 1998.

Ultimately, if I would die today the forum would go on for several months without anyone doing anything. What you would see is (well, you personally wouldn't see some things, like cessation of replying to emails for help and such), fewer thread titles cleaned up (I want to put in my Thanks here to Jan, Harry and a few other mods who help keep thread titles appropriate, which I very much appreciate the help with) and you'd notice few threads would be tagged. You would also notice most threads would have generic icons (not a significant problem - Essentially aesthetics). If you see those things start happening I'm either traveling (which is rare for me and I try to give notice when I do) or I'm seriously disabled or dead (to be blunt).

Worst case would be death, and in that case it would be up to my heirs to try to figure out what to do. I've put as much in the way of instructions in my will as is feasible, but they would have to decide what to do. I imagine they would try to sell the site and the associated domain names to a company or an individual who is in "the business" so to speak. Like I say - I have put 5 people here who are "regulars" in my Will in that for free one of them can have copies of the vBulletin database and software license. I feel I owe it to all the people who have spent thousands of hours here that the data be *public* without any copyright BS or anything like that. The domain name is another issue all together. To be honest I doubt any of the 5 would really want to take over and try to keep the site online, but you never know.

So - That's pretty much the scoop. It's not really a matter of my writing out a set of instructions. A couple of years ago I did look into getting someone to "manage" the forum and website as a whole. The cheapest person, with no significant references, wanted about US$80K. A good candidate wanted US$120K+. If I had wanted a good candidate I would have been paying a lot out of my pocket - The forum simply doesn't make that much money. Obviously I decided to keep on doing this myself.

What will happen? I don't know. But - I think I've given everyone enough information that if anyone has any advice I'm happy to hear suggestions.

 

[Peter Fraser]

Marc

<backed up to a server in Scotland every 2 hours...>
Have you made any allowance for what would happen if Scotland gets its independence in 2014?

 

[Marc]

Nope - I'd be surprised if it makes any difference. Not to mention one never knows. In 2 years I may have a different server admin in another country. Technically I have an "emergency" backup server admin here in the US, but they're cheap because it's "emergency" service in case for some reason I couldn't get my admin in Scotland fast enough should a serious problem occur. AdminGeeks is the main admin group. ServerWizards are my backup. As it is with data, I typically have a backup of some type for most things.