* Make the log relative docs more clear, and easier for further more formats support. (#10749)
This commit is contained in:
吴晟 Wu Sheng
GitHub
parent
8276328436
commit
df7486c700
|
|
@ -62,5 +62,6 @@
|
|||
|
||||
* Add Profiling related documentations.
|
||||
* Add `SUM_PER_MIN` to MAL documentation.
|
||||
* Make the log relative docs more clear, and easier for further more formats support.
|
||||
|
||||
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/169?closed=1)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,40 @@
|
|||
# Collecting File Log
|
||||
|
||||
Application's logs are important data for troubleshooting, usually they are persistent through local or network file
|
||||
system.
|
||||
SkyWalking provides ways to collect logs from those files by leveraging popular open-source tools.
|
||||
|
||||
## Log files collector
|
||||
|
||||
You can use [Filebeat](https://www.elastic.co/cn/beats/filebeat), [Fluentd](https://www.fluentd.org/)
|
||||
and [FluentBit](http://fluentbit.io) to collect logs, and then transport the logs to SkyWalking OAP through Kafka or
|
||||
HTTP protocol, with the formats [Kafka JSON](../../api/log-data-protocol.md#native-kafka-protocol)
|
||||
or [HTTP JSON array](../../api/log-data-protocol.md#http-api).
|
||||
|
||||
### Filebeat
|
||||
|
||||
Filebeat supports using Kafka to transport logs. Open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher) and enable
|
||||
configs `enableNativeJsonLog`.
|
||||
|
||||
Take the following Filebeat config YAML as an example to set up Filebeat:
|
||||
|
||||
- [filebeat.yml](../../../../test/e2e-v2/cases/kafka/log/filebeat.yml)
|
||||
|
||||
### Fluentd
|
||||
|
||||
Fluentd supports using Kafka to transport logs. Open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher) and enable
|
||||
configs `enableNativeJsonLog`.
|
||||
|
||||
Take the following fluentd config file as an example to set up Fluentd:
|
||||
|
||||
- [fluentd.conf](../../../../test/e2e-v2/cases/kafka/log/fluentd.conf)
|
||||
|
||||
### Fluent-bit
|
||||
|
||||
Fluent-bit sends logs to OAP directly through HTTP(rest port).
|
||||
Point the output address to `restHost`:`restPort` of `receiver-sharing-server` or `core`(if `receiver-sharing-server` is
|
||||
inactivated)
|
||||
|
||||
Take the following fluent-bit config files as an example to set up Fluent-bit:
|
||||
|
||||
- [fluent-bit.conf](../../../../test/e2e-v2/cases/log/fluent-bit/fluent-bit.conf)
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
# Collecting Logs by Agents
|
||||
|
||||
Some of SkyWalking native agents support collecting logs and sending them to OAP server without local files and/or file
|
||||
agents, which are listed in [here](filelog-native.md).
|
||||
|
||||
## Java agent's toolkits
|
||||
|
||||
Java agent provides toolkits for
|
||||
[log4j](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/application-toolkit-log4j-1.x/),
|
||||
[log4j2](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/application-toolkit-log4j-2.x/),
|
||||
and
|
||||
[logback](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/application-toolkit-logback-1.x/)
|
||||
to report logs through gRPC with automatically injected trace context.
|
||||
|
||||
[SkyWalking Satellite sidecar](https://github.com/apache/skywalking-satellite) is a recommended proxy/side that
|
||||
forwards logs (including the use of Kafka MQ to transport logs). When using this,
|
||||
open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher)
|
||||
and enable configs `enableNativeProtoLog`.
|
||||
|
||||
Java agent provides toolkits for
|
||||
[log4j](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/application-toolkit-log4j-1.x/#print-skywalking-context-in-your-logs),
|
||||
[log4j2](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/application-toolkit-log4j-2.x/#print-skywalking-context-in-your-logs),
|
||||
and
|
||||
[logback](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/application-toolkit-logback-1.x/#print-skywalking-context-in-your-logs)
|
||||
to report logs through files with automatically injected trace context.
|
||||
|
||||
Log framework config examples:
|
||||
|
||||
- [log4j1.x fileAppender](../../../../test/e2e-v2/java-test-service/e2e-service-provider/src/main/resources/log4j.properties)
|
||||
- [log4j2.x fileAppender](../../../../test/e2e-v2/java-test-service/e2e-service-provider/src/main/resources/log4j2.xml)
|
||||
- [logback fileAppender](../../../../test/e2e-v2/java-test-service/e2e-service-provider/src/main/resources/logback.xml)
|
||||
|
||||
## Python agent log reporter
|
||||
|
||||
[SkyWalking Python Agent](https://github.com/apache/skywalking-python) implements a log reporter for the [logging
|
||||
module](https://docs.python.org/3/library/logging.html) with functionalities aligning with the Java toolkits.
|
||||
|
||||
To explore how to enable the reporting features for your use cases, please refer to the
|
||||
[Log Reporter Doc](https://skywalking.apache.org/docs/skywalking-python/next/en/setup/advanced/logreporter/) for a
|
||||
detailed guide.
|
||||
|
|
@ -1,69 +1,4 @@
|
|||
# Log Collection and Analysis
|
||||
|
||||
## Collection
|
||||
There are various ways to collect logs from applications.
|
||||
|
||||
### Log files collector
|
||||
|
||||
You can use [Filebeat](https://www.elastic.co/cn/beats/filebeat), [Fluentd](https://www.fluentd.org/)
|
||||
and [FluentBit](http://fluentbit.io) to collect logs, and then transport the logs to SkyWalking OAP through Kafka or
|
||||
HTTP protocol, with the formats [Kafka JSON](../../api/log-data-protocol.md#native-kafka-protocol)
|
||||
or [HTTP JSON array](../../api/log-data-protocol.md#http-api).
|
||||
|
||||
#### Filebeat
|
||||
Filebeat supports using Kafka to transport logs. Open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
|
||||
|
||||
Take the following Filebeat config YAML as an example to set up Filebeat:
|
||||
- [filebeat.yml](../../../../test/e2e-v2/cases/kafka/log/filebeat.yml)
|
||||
|
||||
#### Fluentd
|
||||
Fluentd supports using Kafka to transport logs. Open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
|
||||
|
||||
Take the following fluentd config file as an example to set up Fluentd:
|
||||
- [fluentd.conf](../../../../test/e2e-v2/cases/kafka/log/fluentd.conf)
|
||||
|
||||
#### Fluent-bit
|
||||
Fluent-bit sends logs to OAP directly through HTTP(rest port).
|
||||
Point the output address to `restHost`:`restPort` of `receiver-sharing-server` or `core`(if `receiver-sharing-server` is inactivated)
|
||||
|
||||
Take the following fluent-bit config files as an example to set up Fluent-bit:
|
||||
- [fluent-bit.conf](../../../../test/e2e-v2/cases/log/fluent-bit/fluent-bit.conf)
|
||||
|
||||
#### OpenTelemetry
|
||||
You can use OpenTelemetry Collector to transport the logs to SkyWalking OAP.
|
||||
Read the doc on [Skywalking Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/skywalkingexporter) for a detailed guide.
|
||||
|
||||
### Java agent's toolkits
|
||||
Java agent provides toolkits for
|
||||
[log4j](https://github.com/apache/skywalking-java/blob/main/docs/en/setup/service-agent/java-agent/Application-toolkit-log4j-1.x.md),
|
||||
[log4j2](https://github.com/apache/skywalking-java/blob/main/docs/en/setup/service-agent/java-agent/Application-toolkit-log4j-2.x.md), and
|
||||
[logback](https://github.com/apache/skywalking-java/blob/main/docs/en/setup/service-agent/java-agent/Application-toolkit-logback-1.x.md)
|
||||
to report logs through gRPC with automatically injected trace context.
|
||||
|
||||
[SkyWalking Satellite sidecar](https://github.com/apache/skywalking-satellite) is a recommended proxy/side that
|
||||
forwards logs (including the use of Kafka MQ to transport logs). When using this, open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher)
|
||||
and enable configs `enableNativeProtoLog`.
|
||||
|
||||
Java agent provides toolkits for
|
||||
[log4j](https://github.com/apache/skywalking-java/blob/main/docs/en/setup/service-agent/java-agent/Application-toolkit-log4j-1.x.md#print-skywalking-context-in-your-logs),
|
||||
[log4j2](https://github.com/apache/skywalking-java/blob/main/docs/en/setup/service-agent/java-agent/Application-toolkit-log4j-2.x.md#print-skywalking-context-in-your-logs), and
|
||||
[logback](https://github.com/apache/skywalking-java/blob/main/docs/en/setup/service-agent/java-agent/Application-toolkit-logback-1.x.md#print-skywalking-context-in-your-logs)
|
||||
to report logs through files with automatically injected trace context.
|
||||
|
||||
Log framework config examples:
|
||||
- [log4j1.x fileAppender](../../../../test/e2e-v2/java-test-service/e2e-service-provider/src/main/resources/log4j.properties)
|
||||
- [log4j2.x fileAppender](../../../../test/e2e-v2/java-test-service/e2e-service-provider/src/main/resources/log4j2.xml)
|
||||
- [logback fileAppender](../../../../test/e2e-v2/java-test-service/e2e-service-provider/src/main/resources/logback.xml)
|
||||
|
||||
### Python agent log reporter
|
||||
|
||||
[SkyWalking Python Agent](https://github.com/apache/skywalking-python) implements a log reporter for the [logging
|
||||
module](https://docs.python.org/3/library/logging.html) with functionalities aligning with the Java toolkits.
|
||||
|
||||
To explore how to enable the reporting features for your use cases, please refer to the
|
||||
[Log Reporter Doc](https://github.com/apache/skywalking-python/blob/master/docs/en/setup/advanced/LogReporter.md) for a detailed guide.
|
||||
|
||||
## Log Analyzer
|
||||
# Log Analysis
|
||||
|
||||
Log analyzer of OAP server supports native log data. OAP could use Log Analysis Language to
|
||||
structure log content through parsing, extracting and saving logs.
|
||||
|
|
@ -77,4 +12,6 @@ log-analyzer:
|
|||
malFiles: ${SW_LOG_MAL_FILES:""}
|
||||
```
|
||||
|
||||
Read the doc on [Log Analysis Language](../../concepts-and-designs/lal.md) for more on log structuring and metrics analysis.
|
||||
Read the doc on [Log Analysis Language(LAL)](../../concepts-and-designs/lal.md) for more on log structuring and metrics analysis.
|
||||
The [LAL's `metrics` extracts](../../concepts-and-designs/lal.md#extractor) provide the capabilities to generate new metrics
|
||||
from the raw log text for further calculation.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,4 @@
|
|||
# OpenTelemetry Logging Format
|
||||
|
||||
OpenTelemetry Collector supports to transport the logs to SkyWalking OAP.
|
||||
Read the doc on [Skywalking Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/skywalkingexporter) for a detailed guide.
|
||||
|
|
@ -131,8 +131,18 @@ catalog:
|
|||
path: "/en/setup/backend/backend-alarm"
|
||||
- name: "Logging"
|
||||
catalog:
|
||||
- name: "Log Collecting And Analysis"
|
||||
path: "/en/setup/backend/log-analyzer"
|
||||
- name: "Persistent Logging"
|
||||
catalog:
|
||||
- name: "Native Logging"
|
||||
catalog:
|
||||
- name: "Collecting File Log"
|
||||
path: "/en/setup/backend/filelog-native"
|
||||
- name: "Collecting Logs by Agents"
|
||||
path: "/en/setup/backend/log-agent-native"
|
||||
- name: "OpenTelemetry Logging Exporter"
|
||||
path: "/en/setup/backend/log-otlp"
|
||||
- name: "Log Analysis"
|
||||
path: "/en/setup/backend/log-analyzer"
|
||||
- name: "On Demand Pod Logs"
|
||||
path: "/en/setup/backend/on-demand-pod-log"
|
||||
- name: "Profiling"
|
||||
|
|
|
|||
Loading…
Reference in New Issue