[aclug-L] Status of aclug.org

[John Goerzen <jgoerzen@xxxxxxxxxxxx>]

Hello,

On Thursday night at about 9:00 PM, the server for complete.org (which hosts
aclug.org) lost connectivity.  The colocation facility had a billing
misunderstanding with their upstream and got cut off.  In fact, they are
still down.  Unfortunately, Jonathan Hall (which does backup MX and DNS for
the aclug.org domain) also lost connectivity earlier this week, so there was
no backup service in place.

As it became apparent on Friday that this problem was not going to be
resolved quickly, I set up a new server at a far better location (see
details below).  The new server was activated this morning and I have been
working for nearly 12 hours to restore services from the last off-site
backup of pi.complete.org, taken 10 days ago.

At this time, the ACLUG mailing lists have been restored to operational
status for *some* people.  Some people will continue to get bounced messages
or DNS errors for the next 12 ro 24 hours as the updated DNS information
propogates.  If you see this message, but cannot send messages to the ACLUG
lists, this is probably the case for you.  There is nothing I can do about
this, although if you run your own nameserver you can try restarting bind.

In all, it took only 48 hours to go from server-down emergency to new server
in place with basic services restored.

At this time, the list archives are down and will remain so until I can get
access to the data for time between the last offsite backup and today.  The
ACLUG web site is not up yet, but I hope to have it restored tomorrow. 
Until it is, you will see the status page at www.aclug.org.

I have created a site for the status of the service move, which will have
updates as each service is brought up.  You can see it at
http://198.78.66.183/ or at http://www.complete.org/.

I am happy with the new colocation facility.  They have three separate
upstream connections, a climate-controlled facility with Diessel backup
generators.  All their machines contain three redundant power supplies from
separate circuits, multiple redundant fans.  All drives are RAID-mirrored
and on-site backups are performed nightly onto another live filesystem, so a
complete restore takes about 10 minutes.  In other words, this is a much
different situation than just plugging a box into a 'net connection.  I was
planning a move already, but these events forced me to do it a lot sooner!

Thanks,
John Goerzen

-- 
John Goerzen <jgoerzen@xxxxxxxxxxxx>                       www.complete.org
-- This is the discussion@xxxxxxxxx list.  To unsubscribe,
visit http://www.complete.org/cgi-bin/listargate-aclug.cgi