Server Log Integration

The platform provides SmartAgent-based log collection capabilities, supporting mainstream host devices such as Linux, Windows, and ARM, to meet the log collection requirements of different host types.

Prerequisites

If SmartAgent is not installed, you need to install it as prompted.

Getting Started

Navigate to the Log Configuration > Add Log page and switch to the Server-Side Logs tab.

Once the SmartAgent Log Collection toggle is enabled, the SmartAgent gains the ability to automatically discover logs. Paths not discovered automatically can be added via Custom Source Paths . Paths that do not require collection can be specified using the Blacklist .

In the Select Collection Paths section, select the host paths from which logs need to be collected. The system automatically updates the list of available paths every 60 seconds. You can set a collection strategy (e.g., "Collect upon selection") and choose the viewing perspective (by Host, Process Group, or Pod). Upon completion, logs will be collected based on the configuration for the selected hosts or process groups.

Viewing Modes

When viewing by Host , expanding a host entry reveals the names of its process groups. Expanding a process group then displays the list of associated log paths.

When viewing by Process Group , expanding a process group shows its log paths. Expanding a path further reveals the specific hosts associated with it.

When viewing by Pod , expanding a pod entry displays Kubernetes path information, along with data about the pod's cluster and namespace.

For log types that are disabled or for hosts where logging is disabled in the configuration, the recognized paths or hosts underneath them should be grayed out and unavailable for selection. Paths or hosts that have never been recognized are not displayed. The specific logic is as follows:

1. When logging for a specific type is turned off, the agent probes will no longer scan for directories of that type.
2. For directories that were previously detected and recorded, if their recorded time window expires, they will no longer be displayed.

Custom Source Paths (Optional)

The platform provides both pre-configured log monitoring types and custom log source paths, with support for enabling or disabling them. Once enabled, the agent will identify log directories of the corresponding type. Note that the platform does not immediately start collecting these logs; it only automatically discovers the paths for these log types. Custom source paths support search, deletion, and editing.

Automatic discovery requires that log files meet the following criteria:

1. The log file must belong to a critical process and be in an open state to be discoverable.
2. The log file must have existed for at least longer than 1 minute.
3. The log must have a supported character encoding. By default, the supported encoding is UTF-8. Other supported types include UTF-8 with BOM, and if the file contains a Byte Order Mark (BOM), UTF-16LE and UTF-16BE.
4. The log file must be at least larger than 0.5 KB in size.
5. The log file must have been updated within the last 7 days.
6. The log file must be located in an actual log folder or its subfolder.

Valid path examples:
c:\log\log_file.txt
c:\logs\NewFolder\log_file.txt

Invalid path example:
c:\log\NewFolder\NewFolder\log_file.txt

Or, the log filename must contain the string log, which must be immediately preceded or followed by a period (.) or underscore (_) character.

Valid filename examples:
c:\NewFolder\abc.log
c:\NewFolder\0865842.log.txt

Invalid filename example: (No _ or . immediately before or after log)
c:\NewFolder\logfile.txt

The custom log source configuration provides the capability to add log sources manually that were not auto-detected.

Source Definition (Custom log source configuration, supporting global configuration based on hosts or environments.)

• Allows defining log sources without a Process Group context (adding a Process Group is optional). A maximum of three Process Groups can be added to each log source. If no Process Group is needed, it must be placed in the virtual group: default process group name. If configured this way, this virtual group will exist on every host.
• Multiple path rules can be added within a single Process Group. Custom paths must be absolute paths; relative paths are not supported.
• Paths must be unique within the same Process Group but can be duplicated across different Process Groups.
• Paths support configurable tags, which are predefined attributes within the platform's definition.
• Wildcards are supported in directories. Custom log sources can include wildcards: # replaces a single digit character, and * replaces any string of characters except a forward slash (/) or backslash (). Note: # can only be used within filenames.

Security Restrictions

• The log path must not be located in any of the following: /etc, /boot, /dev, /bin, /sbin, /usr.
• The log path cannot be /usr, with the exception of /usr/local.
• The log path must not contain .ssh.
• The log path must not have a .pem extension.
• The log path must not be within a directory whose name begins with a . (e.g., /.hidden).
• The filename or path must meet one of the following requirements:
  • The log filename must have a log extension, separated by a ., -, or _ (which may be followed by another extension using the same set of separators).
  • The log path must reside at the first or second level within a log directory.
  • The log path must be at any level within the /var/log directory.
  • The log path must have the filename catalina.out.

Advanced Settings (Optional)

This interface provides advanced configuration for log collection and management, allowing fine-tuned control over the following core parameters:

• Data Reporting Policy : Configure the SmartAgent log reporting frequency and the file size threshold for reporting, balancing data real-time requirements with transfer efficiency.
• File Detection Rules : Define the minimum file size threshold for detection, ensuring only log files meeting the specification are effectively recognized.
• Path Aggregation Mechanism : Set rules for aggregating character and numeric fields in paths when they exceed a specified length, optimizing the storage and retrieval efficiency for path-based data.

All configuration changes take effect after clicking Save , enabling precise control over the entire log lifecycle (collection, transmission, querying, storage).

Custom Time Format & Multi-line Segmentation Guide

By defining custom time formats and multi-line segmentation rules, multi-line logs can be split into individual entries for analysis in log queries.

• Rule Name: (Required) Set a rule name, not exceeding twenty characters. Rule names must be unique.

• Applicable Scope: (Required) Define the conditions under which the rule applies to logs. Multiple conditions can be combined using AND or OR logic.

• Time Format: (Optional) Define the timestamp format. It must specify at least month, day, hour, minute, and second. For details, refer to Time Format Specification.

• Time Zone: The time zone in which the original logs were generated.

• Multi-line Delimiter: (Optional) Define a string pattern to serve as the delimiter for splitting log entries.

• Search Limit: Set the byte range (default: 64 bytes) for searching time information and the delimiter. When set to 0, it indicates plain text logs where every line not starting with a space is considered the start of a new log record and is automatically assigned a timestamp (the time the probe read the log entry).

• Description: A description is automatically generated based on the rule configuration.

• Validation Logic for Saving:

  • In addition to required fields, at least one of Time Format or Multi-line Delimiter must be configured to save the rule.

• Logic for Logs Matching Multiple Rules:

  • Multi-line delimiter evaluation takes precedence over timestamp evaluation.
  • When a log entry contains one or more timestamps, timestamps are evaluated based on rule priority, starting from the first rule in the list. If a timestamp in the entry matches a rule, the probe uses that timestamp as the start of the log entry and stops evaluating other conditions.
  • Logs that do not satisfy any custom rule are all split according to the default rule. The default rule only supports modifying the time zone and search limit configuration.
  • 

Time Format Specification

• The time format dropdown supports the following common formats:

Time Format
%Y-%m-%d %H:%M:%S
%Y-%m-%d %H:%M:%S.%f
%d-%b-%y %H:%M:%S
%d %b %Y %H:%M:%S
%Y%m-%dT%H:%M:%S
%Y-%m-%dT%H:%M:%S

• In addition to the provided options, you can also define a custom time format based on the time information in your logs. Supported format specifiers include:

Format Specifier	Description	Example
%b	Abbreviated month name.	Jan
%B	Full month name.	January
%d	Day of the month as a zero-padded decimal number (01-31).	07, 31
%f	Fractional seconds (milliseconds, microseconds, or nanoseconds).	123
%h	Abbreviated month name (same as %b).	Jan
%H	Hour (24-hour clock) as a zero-padded decimal number (00-23).	22
%I	Hour (12-hour clock) as a zero-padded decimal number (01-12).	11
%m	Month as a zero-padded decimal number (01-12).	08
%M	Minute as a zero-padded decimal number (00-59).	59
%p	Locale’s equivalent of either AM or PM.	AM, PM
%r	12-hour clock time, equivalent to %I:%M:%S %p.	11:59:59 AM
%R	Hour and minute, equivalent to %H:%M.	23:59
%S	Second as a zero-padded decimal number (00-59).	59
%t	Tab character.	N/A
%y	Year without century as a zero-padded decimal number (00-99).	04, 98
%Y	Year with century as a decimal number.	2004, 1998
%C	Century as a zero-padded decimal number (00-99).	16
%e	Day of the month as a space-padded decimal number ( 1-31).	7, 31
%j	Day of the year as a zero-padded decimal number (001-366).	365
%c	Standard date and time representation. Equivalent to %b %e %H:%M:%S %Y.	Nov 20 14:12:58 2020
%x	Standard date representation (without time). Equivalent to %b %e %Y.	Nov 20 2020
%X	Standard time representation (without date).	11:59:59

• Time formats must satisfy the following validation conditions:
  • Mutual Exclusion Rules:
    • Using the same time format specifier multiple times (e.g., %Y%Y or %S%S) is invalid.
    • Month specifiers are mutually exclusive: Only one of %m, %b/%h, or %B can appear.
    • Day-of-month specifiers are mutually exclusive: Only one of %d or %e can appear.
    • Hour specifiers are mutually exclusive: Only one of %H or %I can appear.
    • Year specifiers are mutually exclusive:
      • %Y and %C cannot appear together.
      • %Y and %y cannot appear together.
    • Day-of-year (%j) is mutually exclusive with month and day-of-month:
      • %j and month specifiers (%m, %b/%h, %B) are mutually exclusive.
      • %j and day-of-month specifiers (%d, %e) are mutually exclusive.
  • Mandatory Components: The format must include "month, day, hour, minute, and second".
    • Date (Month & Day). Satisfy one of the following:
      • Condition A: Include month (%m/%b/%h/%B) + day-of-month (%d/%e).
      • Condition B: Include day-of-year (%j).
    • Time (Hour, Minute, Second).
      • Must include: %H or %I (Hour).
      • Must include: %M (Minute).
      • Must include: %S (Second).
  • Other Rules:
    • If %I is used, %p must also be present. If %p is used, %I must also be present.
    • The % character is special:
      • A % not followed by a valid format specifier is invalid.
      • %% is invalid.