Re: central logfile / real time logging

Andreas Klemm wrote:
Hi,

do you have an idea how to store application generated logfiles
(i.e. apache, ... ) from 1000 Solaris "zones" running on 250
physical Sun server onto a central FreeBSD logserver ?

There are 6 strong demands, which makes this setup very special
and tricky.

After many thinking and googeling I didn't find any OpenSource or
at least "not too expensive" solution which fulfills all of these
demands:

- "reliability", no logmessage may get lost
- on the central logserver the logfiles need to stay in separate files
per virtual zone and application (can be the case the zone runs
multiple services)
- "real-time logging", as soon as an application logrecord gets written
on one of the 1000 zones this logmessage should be written immediately
onto the central logserver. So its not sufficient to make this batched
on an hourly base.
- applications may not write to NFS directly to prevent locking of apps
if NFS server is unavailable (if solution should be NFS based).
- this solution needs to be aware of logfile rotation
on the application server side (the 1000 zones)
- this logging needs to be done from the "global zone" of a Solaris server.


I think you are heading in the right direction here.

Syslog-ng should be able to do all of this for you. The premium edition will give you disk based buffering on you global zone relay's if you have money to spend that is....

To stop message loss you can use TCP as the transport.

You should be able to configure syslog-ng to listen on a different port for every file you want to transport.

It will also be able to do automagic rotation of the files on the centeral server for you such as {sourcehost}/{year}/{month}/{day}.log etc....

There are some tools that you can use with apache for example which will allow apache to log directly to syslog so that you don't have to worry about the source files. Or you could get syslog-ng to setup a collection of named pipes that your processes write their logs to etc etc, there are a huge number of solutions.


I have attached an example config that we use to collect logs from our switches and other hosts, it may interest you.

Some docs that may help:

http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/index.html or PDF version: http://www.balabit.com/dl/guides/syslog-ng-v2.0-guide-admin-en.pdf
http://www.campin.net/syslog-ng/faq.html


Good luck

Tom
Some people seem to use syslog-ng for this on client and central
repository side.

But I see here some problems. I need to convert apache logs from
normal logfiles. This must happen from the Global Zones.

Then the syslog-ng on the central side has no chance (I think)
to demultiplex the logfile stream into logs separated by
the different zone and application.

On the Global Zones I need a process that tracks some logfiles
that are from different virtual zones.

I need to open one tcp session to central archive per logfile.

So ~250 global zones open ~1000 tcp sessions towards the central
FreeBSD server.

There must be a way to tell the application on central FreeBSD
Server, that this stream now should be written to a certain
subdirectory and filename, so that all is separate.

This application need to buffer on the Global zone, if the server
is not available. Or at least keep track, what hasn't been written
yet. And must be aware of logfile rotation.

Any idea ?

Andreas ///


#
# This sample configuration file is essentially equilivent to the stock
# FreeBSD /etc/syslog.conf file.
#

#
# options
#
options { long_hostnames(off); sync(0); create_dirs(yes);};

#
# sources
#
source src { unix-dgram("/var/run/log");
unix-dgram("/var/run/logpriv" perm(0600));
internal(); file("/dev/klog"); };

source switchs {
udp(port(9999));
};

source r_hosts {
udp(port(9998));
};

source r_workstations {
udp(port(514));
};
#
# destinations
#
destination messages { file("/var/log/messages"); };
destination security { file("/var/log/security"); };
destination authlog { file("/var/log/auth.log"); };
destination maillog { file("/var/log/maillog"); };
destination lpd-errs { file("/var/log/lpd-errs"); };
destination xferlog { file("/var/log/xferlog"); };
destination cron { file("/var/log/cron"); };
destination debuglog { file("/var/log/debug.log"); };
destination consolelog { file("/var/log/console.log"); };
destination all { file("/var/log/all.log"); };
destination newscrit { file("/var/log/news/news.crit"); };
destination newserr { file("/var/log/news/news.err"); };
destination newsnotice { file("/var/log/news/news.notice"); };
destination slip { file("/var/log/slip.log"); };
destination ppp { file("/var/log/ppp.log"); };
destination console { file("/dev/console"); };
destination allusers { usertty("*"); };
destination cvsupd { file("/var/log/cvsupd.log"); };

destination d_switchs {
file("/var/log/switchs/$HOST");
};
destination d_workstations {
file("/var/log/workstations/$HOST");
};

destination r_messages { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/messages"); };
destination r_security { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/security"); };
destination r_authlog { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/auth.log"); };
destination r_maillog { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/maillog"); };
destination r_lpd-errs { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/lpd-errs"); };
destination r_xferlog { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/xferlog"); };
destination r_cron { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/cron"); };
destination r_debuglog { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/debug.log"); };
destination r_consolelog { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/console.log"); };
destination r_all { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/all.log"); };
destination r_newscrit { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/news/news.crit"); };
destination r_newserr { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/news/news.err"); };
destination r_newsnotice { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/news/news.notice"); };
destination r_slip { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/slip.log"); };
destination r_ppp { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/ppp.log"); };
destination r_console { file("/dev/console"); };
destination r_allusers { usertty("*"); };




#destination loghost { udp("loghost" port(514)); };

#
# log facility filters
#
filter f_auth { facility(auth); };
filter f_authpriv { facility(authpriv); };
filter f_not_authpriv { not facility(authpriv); };
filter f_console { facility(console); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_ftp { facility(ftp); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_security { facility(security); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };
filter f_local0 { facility(local0); };
filter f_local1 { facility(local1); };
filter f_local2 { facility(local2); };
filter f_local3 { facility(local3); };
filter f_local4 { facility(local4); };
filter f_local5 { facility(local5); };
filter f_local6 { facility(local6); };
filter f_local7 { facility(local7); };

#
# log level filters
#
filter f_emerg { level(emerg); };
filter f_alert { level(alert..emerg); };
filter f_crit { level(crit..emerg); };
filter f_err { level(err..emerg); };
filter f_warning { level(warning..emerg); };
filter f_notice { level(notice..emerg); };
filter f_info { level(info..emerg); };
filter f_debug { level(debug..emerg); };
filter f_is_debug { level(debug); };

#
# program filters
#
filter f_ppp { program("ppp"); };
filter f_slip { program("startslip"); };
filter f_cvsup { program("cvsupd"); };
#
# *.err;kern.warning;auth.notice;mail.crit /dev/console
#
log { source(src); filter(f_err); destination(console); };
log { source(src); filter(f_kern); filter(f_warning); destination(console); };
log { source(src); filter(f_auth); filter(f_notice); destination(console); };
log { source(src); filter(f_mail); filter(f_crit); destination(console); };

#
# *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages
#
log { source(src); filter(f_notice); filter(f_not_authpriv); destination(messages); };
log { source(src); filter(f_kern); filter(f_debug); destination(messages); };
log { source(src); filter(f_lpr); filter(f_info); destination(messages); };
log { source(src); filter(f_mail); filter(f_crit); destination(messages); };
log { source(src); filter(f_news); filter(f_err); destination(messages); };

#
# security.* /var/log/security
#
log { source(src); filter(f_security); destination(security); };

#
# auth.info;authpriv.info /var/log/auth.log
log { source(src); filter(f_auth); filter(f_info); destination(authlog); };
log { source(src); filter(f_authpriv); filter(f_info); destination(authlog); };

#
# mail.info /var/log/maillog
#
log { source(src); filter(f_mail); filter(f_info); destination(maillog); };

#
# lpr.info /var/log/lpd-errs
#
log { source(src); filter(f_lpr); filter(f_info); destination(lpd-errs); };

#
# ftp.info /var/log/xferlog
#
log { source(src); filter(f_ftp); filter(f_info); destination(xferlog); };

#
# cron.* /var/log/cron
#
log { source(src); filter(f_cron); destination(cron); };

#
# *.=debug /var/log/debug.log
#
log { source(src); filter(f_is_debug); destination(debuglog); };

#
# *.emerg *
#
log { source(src); filter(f_emerg); destination(allusers); };


log { source(src); filter(f_cvsup); filter(f_daemon); filter(f_info); destination (cvsupd); };
#
# uncomment this to log all writes to /dev/console to /var/log/console.log
# console.info /var/log/console.log
#
#log { source(src); filter(f_console); filter(f_info); destination(consolelog); };

#
# uncomment this to enable logging of all log messages to /var/log/all.log
# touch /var/log/all.log and chmod it to mode 600 before it will work
# *.* /var/log/all.log
#
#log { source(src); destination(all); };

#
# uncomment this to enable logging to a remote loghost named loghost
# *.* @loghost
#
#log { source(src); destination(loghost); };

#
# uncomment these if you're running inn
# news.crit /var/log/news/news.crit
# news.err /var/log/news/news.err
# news.notice /var/log/news/news.notice
#
#log { source(src); filter(f_news); filter(f_crit); destination(newscrit); };
#log { source(src); filter(f_news); filter(f_err); destination(newserr); };
#log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); };

#
# !startslip
# *.* /var/log/slip.log
#
log { source(src); filter(f_slip); destination(slip); };

#
# !ppp
# *.* /var/log/ppp.log
#
log { source(src); filter(f_ppp); destination(ppp); };

log { source(switchs); destination(d_switchs); };
log { source(r_workstations); destination(d_workstations); };
##########################################################################################################
#
# REMOTE STUFF
#
#
# *.err;kern.warning;auth.notice;mail.crit /dev/console
#
log { source(r_hosts); filter(f_err); destination(r_console); };
log { source(r_hosts); filter(f_kern); filter(f_warning); destination(r_console); };
log { source(r_hosts); filter(f_auth); filter(f_notice); destination(r_console); };
log { source(r_hosts); filter(f_mail); filter(f_crit); destination(r_console); };

#
# *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages
#
log { source(r_hosts); filter(f_notice); filter(f_not_authpriv); destination(r_messages); };
log { source(r_hosts); filter(f_kern); filter(f_debug); destination(r_messages); };
log { source(r_hosts); filter(f_lpr); filter(f_info); destination(r_messages); };
log { source(r_hosts); filter(f_mail); filter(f_crit); destination(r_messages); };
log { source(r_hosts); filter(f_news); filter(f_err); destination(r_messages); };

#
# security.* /var/log/security
#
log { source(r_hosts); filter(f_security); destination(r_security); };

#
# auth.info;authpriv.info /var/log/auth.log
log { source(r_hosts); filter(f_auth); filter(f_info); destination(r_authlog); };
log { source(r_hosts); filter(f_authpriv); filter(f_info); destination(r_authlog); };

#
# mail.info /var/log/maillog
#
log { source(r_hosts); filter(f_mail); filter(f_info); destination(r_maillog); };

#
# lpr.info /var/log/lpd-errs
#
log { source(r_hosts); filter(f_lpr); filter(f_info); destination(r_lpd-errs); };

#
# ftp.info /var/log/xferlog
#
log { source(r_hosts); filter(f_ftp); filter(f_info); destination(r_xferlog); };

#
# cron.* /var/log/cron
#
log { source(r_hosts); filter(f_cron); destination(r_cron); };

#
# *.=debug /var/log/debug.log
#
log { source(r_hosts); filter(f_is_debug); destination(r_debuglog); };

#
# *.emerg *
#
log { source(r_hosts); filter(f_emerg); destination(r_allusers); };


# uncomment this to log all writes to /dev/console to /var/log/console.log
# console.info /var/log/console.log
#
#log { source(src); filter(f_console); filter(f_info); destination(consolelog); };

#
# uncomment this to enable logging of all log messages to /var/log/all.log
# touch /var/log/all.log and chmod it to mode 600 before it will work
# *.* /var/log/all.log
#
#log { source(src); destination(all); };

#
# uncomment this to enable logging to a remote loghost named loghost
# *.* @loghost
#
#log { source(src); destination(loghost); };

#
# uncomment these if you're running inn
# news.crit /var/log/news/news.crit
# news.err /var/log/news/news.err
# news.notice /var/log/news/news.notice
#
#log { source(src); filter(f_news); filter(f_crit); destination(newscrit); };
#log { source(src); filter(f_news); filter(f_err); destination(newserr); };
#log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); };

#
# !startslip
# *.* /var/log/slip.log
#
log { source(r_hosts); filter(f_slip); destination(r_slip); };

#
# !ppp
# *.* /var/log/ppp.log
#
log { source(r_hosts); filter(f_ppp); destination(r_ppp); };
_______________________________________________
freebsd-isp@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: March 29, 2006 total eclipse - IT admins WORST NIGHTMARE
    ... and NewsProxy is the answer for that. ... > Comcast news server. ... simply filters out what I dont want on the network. ... NewsProxy - Network level killfile and content filter for Usenet. ...
    (comp.security.firewalls)
  • Re: Getting rid of SMTP Q emails
    ... DWord for contentfilterstate to 1. ... That's too fun about restarting the SMTP service. ... I am going to whip this server in shape and get the SPAMMING under control. ... After you added connection filter provider you need to ...
    (microsoft.public.windows.server.sbs)
  • Re: Security Logs are hard to read
    ... But there are thousands of events in the security logs during this time. ... If I filter by the user name, ... Is the server box running on the SBS 2003 server? ... server status report for you to monitor the server to ensure the server ...
    (microsoft.public.windows.server.sbs)
  • Re: SMTP Connectors with massive queues .. How do i stop this?
    ... >report) to the sender. ... entitled: "Clean Up the Exchange ... >Drop connection if address matches filter ... >c) Default SMTP Server ...
    (microsoft.public.windows.server.sbs)
  • Re: SMTP Connectors with massive queues .. How do i stop this?
    ... Your server will accept email for non_existent_user@yourdomain.com. ... check out the section entitled: "Clean Up the Exchange ... Server's SMTP Queues" for the steps you can use to clean up the queue. ... Drop connection if address matches filter ...
    (microsoft.public.windows.server.sbs)