3.8 KiB
Log Collecting And Analysis
Collecting
There are various ways to collect logs from application.
Log files collector
You can use Filebeat, Fluentd and FluentBit to collect logs, and then transport the logs to SkyWalking OAP through Kafka or HTTP protocol, with the formats Kafka JSON or HTTP JSON array.
Filebeat
Filebeat supports using Kafka to transport logs, you need to
open kafka-fetcher and enable configs enableNativeJsonLog.
Take the following filebeat config yaml as an example to set up Filebeat
Fluentd
Fluentd supports using Kafka to transport logs, you need to
open kafka-fetcher and enable configs enableNativeJsonLog.
Take the following fluentd config file as an example to set up Fluentd
Fluent-bit
Fluent-bit sends logs to OAP through HTTP(rest port) directly.
Point the output address to restHost:restPort of receiver-sharing-server or core(if receiver-sharing-server inactivated)
Take the following fluent-bit config files as an example to set up Fluent-bit
Java agent's toolkits
Java agent provides toolkit for log4j, log4j2, logback to report logs through gRPC with automatic injected trace context.
SkyWalking Satellite sidecar is a recommended proxy/side to
forward logs including to use Kafka MQ to transport logs. When use this, need to open kafka-fetcher
and enable configs enableNativeProtoLog.
Java agent provides toolkit for log4j, log4j2, logback to report logs through files with automatic injected trace context.
Log framework config examples:
Python agent log reporter
SkyWalking Python Agent implements a log reporter for the Logging module with functionalities aligning with the Java toolkits.
To explore how to enable the reporting features for your use cases, please refer to the Log Reporter Doc for a detailed guide.
Log Analyzer
Log analyzer of OAP server supports native log data. OAP could use Log Analysis Language to structurize log content through parse, extract, and save logs. Also the analyzer leverages Meter Analysis Language Engine for further metrics calculation.
log-analyzer:
selector: ${SW_LOG_ANALYZER:default}
default:
lalFiles: ${SW_LOG_LAL_FILES:default}
malFiles: ${SW_LOG_MAL_FILES:""}
Read Log Analysis Language documentation to learn log structurize and metrics analysis.