Multiple Log Pattern Support

A single application can write different module logs in two or more formats. For example, an application may write access log information in one format and error log information in another. In that case, instead of creating individual log types for each log format for the same application, you can create multiple log patterns under a single log type.

Site24x7's multiple log pattern support enables you to centralize logs written by an application in a different log format and helps you visualize the data on a single dashboard. Learn more about Site24x7 AppLogs.

Getting started

Log in to your Site24x7 account.
Download and install the Site24x7 server monitoring agent (Windows | Linux).
Go to Admin > AppLogs > Log Types > Add Log Type.

You can refer to this Adding a log type help doc for more information.

Example: Two log files with different log formats

Here is an example of an application writing two different log files: access.log and error.log.

File 1: A sample access log and its log pattern

127.0.0.1 - - [21/Jul/2022:12:41:20 +0530] "GET / HTTP/1.1" 200 44 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"

$RemoteAddress$ - $RemoteUser$ [$Datetime:date$] "$Method$ $RequestURI$ $Protocol$" $Status:number$ $BytesSent:number$ "$Referer$" "$UserAgent$"! "$ForwardedFor$"!

File 2: A sample error log and its log pattern

[2022-05-29 16:29:38 +0530] [212777] [INFO] Starting gunicorn 20.1.0

[$Datetime:date$] [$ProcessID:number$] [$LogLevel$] $Message$

Using multiple log pattern support, you can add different log patterns written in an access.log file and an error.log file under a single log type.

This helps you visualize the access and error log data on a single dashboard.

You can also specify a log pattern name in the query language to filter out the data specific to the log pattern. For example, you can query the Access log pattern using the query below:

logtype="Gunicorn" and logpattern_name="Access log"

Example: A single file with JSON and normal logs

Here is an example of the same file having JSON and single-line logs.

File 1: A sample JSON log and its log pattern

{"log":"Starting Application Server","stream":"stdout","time":"2022-06-22T20:19:58.980395832Z"}

json $log$ $stream$ $time as Datetime:date:yyyy-MM-dd'T'HH:mm:ss.SSS'Z'$

File 1: A sample single-line log and its log pattern

2021/08/05 20:26:49:000 portAvailability 9090 127.0.0.1

$Datetime:date:yyy/MM/dd HH:mm:ss:SSS$ $Message$

Using multiple log pattern support, you can create the above two log patterns under a single log type.

While parsing, the log file follows the order cited in the Log Pattern. For example, in the screenshot above, the log file will parse the JSON pattern logs first, then the single-line pattern logs.
If a log line matches more than one pattern, then the agent will consider the first matched pattern.
You can shuffle the Log Pattern using the up/down arrow icons.
The order of the Sample Logs and Log Pattern may not match because rearranging the Log Pattern will not change the Sample Logs order.

Example: Using a combination of specific and generic log patterns

Here is an example of an application writing similar logs with a slight change in the log pattern.

A sample log with a specific log pattern

[05-Jun-2020 12:16:43] [212778] [ERROR] Exception occurred while starting.

[$Datetime:date:dd-MMM-yyyy HH:mm:ss$] [$ProcessID:number$] [$LogLevel$] $Message$

A sample log with a generic log pattern

[05-Jun-2020 12:16:43] [INFO] Starting gunicorn 20.1.0

[$Datetime:date:dd-MMM-yyyy HH:mm:ss$] $Message$

In the examples above, the two sample log patterns are almost identical, with a slight change in the message field format. The first log pattern is more specific to the ProcessID and LogLevel fields than the second one. In such a case, always mention the specific log pattern first, then the generic pattern second.

Limitations

The following are the limitations of using the multiple log pattern support:

The data types for the same field name should be the same across the log pattern. The pattern below is not supported because the field name ProcessID cannot take two different data types (string and number).
- Log pattern 1: [$Datetime:date$] [$ProcessID:number$] [$LogLevel$] $Message$
- Log pattern 2: [$Datetime:date$] [$ProcessID$] [$LogLevel$] $Message$
You can add a maximum of five log patterns.
The field name for the date time field should be the same across the log pattern.
The Log Pattern names must be unique. The naming convention below is not allowed.

Do not combine multiline JSON with any other log pattern. Treat it as a separate log type.
Do not combine two different JSON formats in the same log type.
Do not combine two multiline log patterns in the same log type.