Trifork Blog

Axon Framework, DDD, Microservices

Logging to the syslog from a java application

January 14th, 2010 by
|

Every application needs logging, it can help you during development and when debugging those annoying things that do not work in production. One question is where to put the logging events. All linux servers use a system log to log events that take place on the operating system level. You can find logs for the kernel, deamons, user actions and a lot of other items. The nice part about system logging is that maintenance people will always know where to look and that it is possible to use one server for logging.

At the moment I am on a project that uses a fair amount of servers. We have more than 20 servers for the different environments and a lot of components to investigate when trying to find problems. Think about squid logs, apache httpd logs, tomcat logs and more. To make this doable, we have a syslog server.

An application running in Tomcat does not log to the system log by default. In our situation, we want our components to log to different files. Syslog has a special facility that makes it easy to do just that.

This blog post discusses the different parts of configuring system logging from a java application using the well known log4j.

syslog-ng introduction

This very short introduction is based on this article.

With syslog you can log to the local server but also to a remote server. It is also possible to do them both using a little bit of configuration. The following image gives an idea about how a logging environment can be used. (took it from the mentioned article)

network-overview.png

Syslog consists of a few elements.

  • Sources – Where can logs come from, the facilities.
  • Filters – Used to for instance log events based on their severity. I used this to filter on Facility
  • Destinations – Where to actually send the logs to, a file a remote host.
  • Logs – Combine the sources, filters and destinations

Before we can talk about the configuration I need to tell you something about Facilities. A facility defines the source of the log. You can think about kernel messages, user-level messages, mail system and many more. The javadoc of the appender shows the possible values for these Facilities. There are also a few special facilities, these are defined as local0, local1, etc. These are the ones you can use for your own application. We use one of these for every different logfile we want.

Configure syslog-ng

The mentioned components are easily identified in the configuration of the syslog. The first thing to configure is udp access to the syslog. To enable udp logging, open the file “/etc/syslog-ng/syslog-ng.conf” and make sure to configure the sources part like the following block:

######
# sources

# all known message sources
source s_all {
# message generated by Syslog-NG
internal();
# standard Linux log source (this is the default place for the syslog()
# function to send logs to)
unix-stream("/dev/log");
# messages from the kernel
file("/proc/kmsg" log_prefix("kernel: "));
# use the following line if you want to receive remote UDP logging messages
# (this is equivalent to the "-r" syslogd flag)
udp();
};

Next up is configure the log files to which the syslog events will be send. First we configure the destination:

# destinations
destination df_local0 { file ("/var/log/cms.log"); };
destination df_local1 { file ("/var/log/site.log"); };
destination df_local2 { file ("/var/log/importer.log"); };

This code block configures three destinations with the names df_local0/1/2 and points them to the mentioned log files. Now we have to filter the incoming log events. We use the facilities that we discussed to filter the incoming events. There are lots of other options, but this is very easy to use. The following lines show the configuration of the filters.

# filters
filter f_local0 { facility(local0); };
filter f_local1 { facility(local1); };
filter f_local2 { facility(local2); };

Finally we configure the logs, these are combinations of a source, a destination and a filter.

log {
source(s_all);
filter(f_local0);
destination(df_local0);
};
log {
source(s_all);
filter(f_local1);
destination(df_local1);
};
log {
source(s_all);
filter(f_local2);
destination(df_local2);
};

Do not forget to restart the syslog service after configuration changes

Testing to see that it works

Before you start messing around with log4j, you can use a few tools to send log events to the syslog server. There are command line utilities available on all linux systems. Check this post for more information. The following command sends a message:

nc 192.168.1.1 514 <<< "<14>User Info msg from remote through TCP."

Be sure to check the port of the server (514 is the default).

Configure log4j

The final step is to send syslog events from your java application. Log4j comes out of the box with a SyslogAppender. There is one requirement to be able to use log4j with sys logging. You need to have udp enabled like we discussed in the configuration section. Most of this comes from the mentioned blog post.

An example of log configuration for log4j :

log4j.rootLogger=INFO, SYSLOG

log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=127.0.0.1
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2} %x - %m%n
log4j.appender.SYSLOG.Facility=LOCAL1

That is it, now you have a java application logging to the syslog. Feedback about improvements is welcome.

6 Responses

  1. January 21, 2010 at 01:58 by Rusty Wright

    If you prefer something platform independent and written in java there are also Lilith and Chainsaw. Chainsaw looks a bit moribund.

    http://lilith.huxhorn.de/
    http://logging.apache.org/chainsaw/

  2. July 11, 2010 at 20:52 by Shantanu

    * It seems like facilities like local0, local1 etc. are reserved for local use only. Can we define our custom facilities? http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SyslogAppender.html#LOG_LOCAL0 Or am I missing something here?

    * Right now I have a filename-appender in log4j which writes to different files based on webapp and package name, e.g.: webapp1-axis.log, webapp1-hibernate.log etc.. I am not sure how to do that using Syslog appender. I was wondering if I could use facilities and filters for this e.g. write to different files based on webapp name and package name. Any thoughts?

  3. December 22, 2010 at 05:32 by Steve Taylor

    I notice that each of the titles on this page is actually a separate Flash. They seem to have no function other than to show text in your chosen font. I’m all for Flash as a tool for making games and interactive toys, but surely that’s overkill just to get the font you want?

  4. February 14, 2012 at 16:49 by Andrew

    I found it strange that your example said “User Info msg from remote through TCP” since you are clearly using UDP for this. As of this writing, log4j doesn’t appear to support TCP unless you manually patch it.

  5. February 22, 2012 at 01:16 by Tim Pozar

    One comment… With testing you may try UDP as that is the default. Also I used the command…

    echo “User Info msg from remote through UDP.” | nc -u 192.168.1.1 514

  6. February 25, 2015 at 09:00 by Gaurav

    Hi … Please help on the problem, I have a working syslog from client to server using logback.xml but I want to make it work on HA, as in when the target IP of logging at syslogappender in logback.xml switched from one server to other, how can i make syslog to log to the new server ???
    Please let me know if anyone has some insight on it.