refcodes-logger: Fancy runtime-logs and highly scalable cloud logging

refcodes-logger: Fancy runtime-logs and highly scalable cloud logging

README

The REFCODES.ORG codes represent a group of artifacts consolidating parts of my work in the past years. Several topics are covered which I consider useful for you, programmers, developers and software engineers.

What is this repository for?

… An introduction to the refcodes-logger framework; let’s start with giving your logs some color and tidy them up! Then let’s take over spring-boot’s log output …” [Logging like the nerds log, 04/04/2015]

The refcodes-logger artifact provides the refcodes-logging framework for flexible logging of any data to any data sink (including files, databases or the console provided as alternate implementations).

The refcodes-logger artifact supports straight forward, composite (clustering) or partitioning functionality as implementations of the Logger type. The REFCODES.ORG runtime logger RuntimeLogger integrates with SLF4J seamlessly and also acts as an alternative to to SLF4J.

The RuntimeLogger implementations are being configured with a Logger implementation. In your code, you use the RuntimeLogger type which, depending on you configuring it, logs to a SimpleDB cluster, the console (with ANSI escape sequence support) or any I/O device as well as to an SLF4J logger.

Being an alternative to SLF4J, the RuntimeLogger’s architecture settles upon the much more generic Logger; which actually can be used to log high volume logs of any data type and not being restricted to runtime logs. The RuntimeLogger implementations add functionality not found in other logging frameworks (logging out the class- and method-names of the logging methods without any configuration or additional lines of code)

How do I get set up?

To get up and running, include the following dependency (without the three dots “…”) in your pom.xml:

1 <dependencies>
2   ...
3   <dependency>
4     <artifactId>refcodes-logger</artifactId>
5     <groupId>org.refcodes</groupId>
6     <version>1.1.7</version>
7   </dependency>
8   ...
9 </dependencies>

The artifact is hosted directly at Maven Central. Jump straight to the source codes at Bitbucket. Read the artifact’s javadoc at javadoc.io.

How do I get started?

You configure your RuntimeLoggerFactoryImpl (and the RuntimeLoggerFactorySingleton sub-class) by providing a runtimelogger-config.xml file in one of those locations relative to your main class’s location:

  • .
  • ./config
  • ./etc
  • ./settings
  • ./.config
  • ./.settings
  • ../config
  • ../etc
  • ../settings
  • ../.config
  • ../.settings

Given your main class’s JAR file resides in the folder /opt/app/lib, then the valid locations for the runtimelogger-config.xml file are:

  • /opt/app/lib
  • /opt/app/lib/config
  • /opt/app/lib/etc
  • /opt/app/lib/settings
  • /opt/app/lib/.config
  • /opt/app/lib/.settings
  • /opt/app/config
  • /opt/app/etc
  • /opt/app/settings
  • /opt/app/.config
  • /opt/app/.settings

(see RuntimeUtility.toAppConfigDirs())

In case you pass a JVM argument via -Dconfig.dir=path_to_your_config_dir (where path_to_your_config_dir stands for the path to the directory where you placed configuration files such as the runtimelogger-config.xml file), then your path_to_your_config_dir is placed first in the list of configuration directories to look at (in case the directory exists). See RuntimeConsts.SYS_PROP_CONFIG_DIR and RuntimeUtility.toAppConfigDirs().

Use the JVM argument -Dconfig.dir=path_to_your_config_dir for your custom runtimelogger-config.xml configuration folder.

The runtimelogger-config.xml configuration is deadly simple:

 1 <?xml version="1.0" encoding="ISO-8859-1" ?>
 2 <config>
 3   <root config-class="org.refcodes.logger.impls.RuntimeLoggerImpl" logPriorityName="INFO" name="*">
 4     <logger config-class="org.refcodes.logger.impls.SystemLoggerImpl" />
 5   </root>
 6   <com>
 7     <acme config-class="org.refcodes.logger.impls.RuntimeLoggerImpl" logPriorityName="INFO" name="AcmeLogger">
 8       <logger config-class="org.refcodes.logger.impls.SystemLoggerImpl" />
 9     </acme>
10   </com>
11 </config>

The XML element nesting represents the Java package for which the therein configured RuntimeLogger is responsible; e.g. a log issued from inside a class located in the package com.acme (or in one of its sub-packages) will be handled by the AcmeLogger. In case no logger is found for a given package, then the root logger found in the <root> ... </root> element is used. Useful to know that a logger for a package namespace is only created once.

If you like logs colored nicely with ANSI escape codes, then you will love the ConsoleLoggerSingleton:

1 <?xml version="1.0" encoding="ISO-8859-1" ?>
2 <config>
3   <root config-class="org.refcodes.logger.impls.RuntimeLoggerImpl" logPriorityName="INFO" name="*">
4     <logger config-class="org.refcodes.logger.alt.console.impls.ConsoleLoggerSingleton" />
5   </root>
6 </config>

Make sure to include the refcodces-logger-alt-console dependency in your build setup to include the ConsoleLoggerSingleton logger.

Runtime logging

Using the refcodes-logging framework in your code to do runtime logging (such as LOGGER.debug("Starting ...") or LOGGER.error("Earth control, we have a problem ...") is also deadly simple. Just declare a static member variable to your class where you want to do your logging:

private static RuntimeLogger LOGGER = RuntimeLoggerFactorySingleton.getInstance().createInstance();

The usage is very similar to a logger such as Log4j or SLF4J:

1 LOGGER.info( "Starting the application ..." );
2 LOGGER.error( "Failed to read file as of an IO exception!", e );
3 LOGGER.trace( "Start time is {0}, duration is {1} ms, end time is {2}", theStartTime, theDurationMs, theEndTime );

By default, the refcodes-logger logs out the additional following information:

  • The overall line number of the current logged line
  • The class and the method from which you produced a log line

Console logging

The artifact refcodes-logger-alt-console provides a good starting point playing around with the refcodes-logger tool-box. With the refcodes-logger-alt-console you can print out colorful console logs into your terminal. It auto-detects the features of your terminal such as the number of chars per row, whether you use a shell on a Unix alike operating system or CMD.EXE on Windows and determines whether your terminal supports coloring of the output (ANSI escape codes).

Add the following dependency to your pom.xml:

1 <dependencies>
2   ...
3   <dependency>
4     <artifactId>refcodes-logger-alt-console</artifactId>
5     <groupId>org.refcodes</groupId>
6     <version>1.1.7</version>
7   </dependency>
8   ...
9 </dependencies>

It transitively pulls the refcodes-logger artifact. Then create a runtimelogger-config.xml similar to the one above:

 1 <?xml version="1.0" encoding="ISO-8859-1" ?>
 2 <config>
 3   <root config-class="org.refcodes.logger.impls.RuntimeLoggerImpl" logPriorityName="INFO" name="*">
 4     <logger config-class="org.refcodes.logger.alt.console.impls.ConsoleLoggerSingleton" tableStyleName="ASCII" />
 5   </root>
 6   <com>
 7     <acme config-class="org.refcodes.logger.impls.RuntimeLoggerImpl" logPriorityName="DEBUG" name="*" >
 8       <logger config-class="org.refcodes.logger.alt.console.impls.ConsoleLoggerSingleton" tableStyleName="ASCII" />
 9     </acme>
10   </com>
11 </config>

There are some element attributes which can be (optionally) configured when declaring the ConsoleLoggerSingleton in your runtimelogger-config.xml:

  • tableStyleName: Valid values for the tableStyleName attribute are as of the TableStyle enumeration:
  • SINGLE
  • DOUBLE
  • DOUBLE_SINGLE
  • SINGLE_DOUBLE
  • SINGLE_DASHED
  • ASCII
  • BLANK
  • SINGLE_BLANK
  • ASCII_BLANK

  • rowWidth: In case the row width cannot be determined automatically, you can either set the system property -Dconsole.width=n (see below) or you can provide the number of chars you desire per row with the rowWidth attribute, e.g. rowWidth=160.

The ConsoleLoggerImpl (and in turn as being its sub-class the ConsoleLoggerSingleton) by default uses the SystemUtility.getConsoleWidth() method, which determines the width in characters of the system’s console in use. In case you pass a -Dconsole.width=n JVM argument (where n stands for the number of chars you desire for your console row width), then your width is taken, else the actual console’s width is being tried to be determined. See SystemConsts.SYS_PROP_CONSOLE_WIDTH.

A fancy output could look as follows:

such

The output as a console copy’n’paste dump (note that the tilde character ~ is used to truncate long lines; all above WARNING is not truncated):

steiner@proteus:~/Workspaces/org.refcodes/refcodes-runtime-ext/refcodes-runtime-ext-console$ java -jar target/refcodes-runtime-ext-console-0.1.1-SNAPSHOT.jar
───────┬───────────────────┬───────┬───────────────┬──────────────────────────────┬──────────────────────┬────────────────────────────────────────────────────
0000001│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │~12 12:19:35 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
0000002│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │Console width    := 158                             
0000003│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │Console height   := 24                              
0000004│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │ANSI support     := true                            
0000005│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │Operating system := UNIX                            
0000006│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │CLI              := SHELL                           
0000007│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │Encoding         := UTF8                            
0000008│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │Process (PID)    := 13082                           
0000009│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │~odes.runtime.ext.console.impls.SystemInfoConsoleApp
0000010│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │stty size        :=                                 
0000011│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │tput cols        := 80                              
0000012│2015-04-04T11:21:19│INFO   │main           │~le.impls.SystemInfoConsoleApp│printInfo()           │tput lines       := 24                              
steiner@proteus:~/Workspaces/org.refcodes/refcodes-runtime-ext/refcodes-runtime-ext-console$ 

SLF4J support

Instead of logging to the console via refcodes-logger-alt-console, you can use the existing SLF4J binding refcodes-logger-alt-slf4j and make your refcodes-logger log via SLF4J just by including the below dependency (instead of the above one) in your pom.xml:

1 <dependencies>
2   ...
3   <dependency>
4     <artifactId>refcodes-logger-alt-slf4j</artifactId>
5     <groupId>org.refcodes</groupId>
6     <version>1.1.7</version>
7   </dependency>
8   ...
9 </dependencies>

This will cause your …

private static RuntimeLogger LOGGER = RuntimeLoggerFactorySingleton.getInstance().createInstance();

… to do logging with your SLF4J binding, which could be a Log4J binding.

SLF4J binding

Via the refcodes-logger-ext-slf4j artifact, you can use the refcodes-logger framework itself as an SLF4J binding - in order to make, for example, your Tomcat or spring-boot log to the fancy refcodes-logger-alt-console or a NoSQL DB cluster (as of refcodes-logger-alt-simpledb):

1 <dependencies>
2   ...
3   <dependency>
4     <artifactId>refcodes-logger-ext-slf4j</artifactId>
5     <groupId>org.refcodes</groupId>
6     <version>1.1.7</version>
7   </dependency>
8   ...
9 </dependencies>

In addition you have to add a dependency of one of the refcodes-logger-alt-* artifacts to your pom.xml and an according runtimelogger-config.xml pulling that logger implementation you want your SLF4J logs to log to …

Attention: Never use the SLF4J binding refcodes-logger-ext-slf4j together with the refcodes-logger-alt-slf4j logger implementation, else the one will look for the other, the other for the one … until you get an StackOverflowException :D

With the SLF4J binding, you can log all your SLF4J logs (also the ones of the libs you included in your app) to any of the refcodes-logger-alt-* implementations:

Make your microservices log into a NoSQL database cluster with the refcodes-logger-alt-simpledb artifact providing support for Amazon’s SimpleDB.

Scalability

Just some keywords on scalability: The refcodes-logger framework implements the composite pattern and various partitioning and composition strategies for high scalability and therewith high throughput.

  • The CompositeLoggerImpl is an asynchronous logger with a log lines queue. Depending on the availability of the encapsulated loggers, the one or the other encapsulated logger takes the next waiting log line in the queue and processes it. The encapsulated refcodes-loggers can be any instance implementing the Logger interface (e.g. one of the loggers found in the refcodes-logger artifact and the refcodes-logger-alt-* artifacts).

  • The PartedLoggerImpl implements partitioning functionality. You define a partitioning criteria for your log lines, e.g. a tenant’s ID (or the date or in case of the refcodes-logger a package, class name or method) and the parted-logger takes care to address the encapsulated responsible logger (which in turn can be another parted-logger or a composite-logger or any other refcodes-logger). The encapsulated refcodes-loggers can be any instance implementing the Logger interface (e.g. one of the loggers found in the refcodes-logger artifact and the refcodes-logger-alt-* artifacts).

Querying

As the data sink to which a refcodes-logger writes its output can be write only (e.g. a terminal window) or write, read and delete (e.g. a NoSQL database such as the SimpleDB logger), there are three flavors of the Logger interface. For example the composite-logger and the parted-logger provides them three flavors:

Contribution guidelines

  • Writing tests and logger implementations for other data sinks.
  • Implement an observable implementation of the log line queue based Logger implementations (e.g. in a refcodes-logger-ext multi module project.
  • Extending the refcodes-logger to contain a correlation ID (useful when working in distributed cloud environments or with microservices)
  • Code review

Who do I talk to?

  • Siegfried Steiner (steiner@refcodes.org)

Terms and conditions

The REFCODES.ORG group of artifacts is published under some open source licenses; covered by the refcodes-licensing (org.refcodes group) artifact - evident in each artifact in question as of the pom.xml dependency included in such artifact.

comments powered by Disqus