Hopefully This Will Not Happen To You.
Sorry for the delay in posting anything in my blog; the good news is that the intervening period has been very busy, good news in the sense that more experience stacks on our shelves and far more important, we helped several clients.
We had one client suffer a misfortune, firstly they have only one production CF-web server and one production database server. There have been many presentations where some have said, "we don't need high-availability (HA)". My response is always a question "does anyone rely on your web site" the answer almost always is "yes of course" and my answer then is "you need HA". This is a critically important point and I have this recommendation, if you are planning on offering a service based on server-side reliability, then plan to have redundancy from day one; build it into your costs; this is really important. There is nothing more meaningful than going from one to two of anything, where redundancy is concerned.
So back to our client misfortune, they are hosted with a well-known data-center provider, who assured them that there were regular back-ups and herein lies another point of importance; back-ups need the fine print clearly stated. For instance, there is a world of difference between backing up web servers and database servers. Even if web site code is changed 4 times a day (not advisable with good change-release management) the data tier is changing second by second in most cases; this needs to be taken account of. The back-up regime for CF-web servers needs to be very different to that for databases.
One more critical point, do not wait to test back-up and recovery until you actually need it, that is far too late. Our advice is to test that mechanism at least once a month, yes it might cost a little more but the cost is nothing compared to a total failure, which is what our client suffered and why they called us in.
So we really hope this will never happen to you.