[webdev] log rotate (ex: Next Steps)

[Tom Hull <thull2@xxxxxxx>]

This just responds to the backup issue. The rest of the thread
I'll address separately.

Jeff Vian wrote:
> Specifically, what is the problem with aolserver and log rotation?

Aolserver currently creates two log files:

   1) /usr/local/aolserver/log/server.log

   2) /usr/local/aolserver/servers/server1/modules/nslog/access.log

#1 is mostly a debugging log. It quickly grows very big, and
doesn't rotate automatically. (Also, I'm not real happy with the
quality of information there -- too much noise.)

#2 is the access log. This grows by one line per file, and it
appears to be rotating once-daily, with five days total kept.

This strikes me as the wrong policy: I'd like to keep the access
logs almost indefinitely, but the debug log is either worthless
or mostly worthless (should be boiled down) after a couple of
days.

The first thing to do here is to understand the config file. In
the config file, the important lines are:

   ns_section "ns/parameters"
     ns_param ServerLog ${homedir}/log/server.log
     ns_param LogRoll on

   ns_section "ns/db/pools"
     ns_param log "ACLUG Log Pool"

   ns_section "ns/db/pool/main"
     ns_param Verbose Off
     ns_param LogSQLErrors On
     ns_param ExtendedTableInfo On

   ns_section "ns/db/pool/log"
     ns_param Verbose On
     ns_param LogSQLErrors On
     ns_param ExtendedTableInfo On

   ns_section "ns/db/pool/subquery"
     ns_param Verbose On
     ns_param LogSQLErrors On
     ns_param ExtendedTableInfo On

   ns_section "ns/server/${server}/module/nslog"
     ns_param rolllog true
     ns_param rollonsignal true
     ns_param rollhour 0
     ns_param maxbackup 5

The other little piece of information here is that when you
send a SIGHUP to aolserver it rotates both logs. The signal
command would be:

   kill -HUP $(cat /usr/local/aolserver/log/nspid.server1)

I think the best solution here would be to find the right
configuration variables to get aolserver to implement the
right rotation policy. Another approach would be to disable
automatic log rotation (remove the rollhour parameter?),
and use a cron script to force rotation with SIGHUP. The
advantage of the latter approach is that the script can
process and backup the rotated logs right after they have
been created.

The likely processing we would want to do would be to boil
the server.log down -- try to preserve error messages but
not the usual verbose blather -- and to consolidate the
access.log files so they can be used by some sort of web
stats package (IIRC, they should be in the same format as
apache).

Obviously, a little more research needs to be done so we
can figure out just what the right thing to do is. This
I think should be done before we "go live", but until
then I'm not worried about accidentally losing any of
the log info (especially the old-server.log file that
I manually renamed a while back).

The nslog source files are in:

   /home/tom/loot/aolserver-3.4.2/nslog/

Also look in:

   /home/tom/loot/aolserver-3.4.2/nsd/log.c

for something on server.log.

 > I can work that out if I know what you want. (and have either root
 > access or sudo access)

AFAIK, no one has set up sudo. You can get the root password from
me, Dale, or John A.

Dale can fill you in on the backup script, which I think is in:

   /home/nsadmin/aclug-backup

As I mentioned before, a comprehensive backup will also have to back
up the bugzilla database, the cvs repository, and some other files.
(Presumably the document root can be reconstructed from the cvs
repository, but openacs probably provides paths that violate that
assumption.)

We might consider copying the backup to another of John Alexander's
machines and/or rsync'ing it offsite.

-- 
/*
  *  Tom Hull * thull2 at cox.net * http://www.tomhull.com/
  */