refine backend doc (#7559)
This commit is contained in:
Wing
GitHub
parent
db2bc93573
commit
c778007c02
|
|
@ -1,26 +1,26 @@
|
|||
# Support gRPC SSL transportation for OAP server
|
||||
# gRPC SSL transportation support for OAP server
|
||||
|
||||
For OAP communication we are currently using gRPC, a multi-platform RPC framework that uses protocol buffers for
|
||||
For OAP communication, we are currently using gRPC, a multi-platform RPC framework that uses protocol buffers for
|
||||
message serialization. The nice part about gRPC is that it promotes the use of SSL/TLS to authenticate and encrypt
|
||||
exchanges. Now OAP supports to enable SSL transportation for gRPC receivers.
|
||||
exchanges. Now OAP supports enabling SSL transportation for gRPC receivers.
|
||||
|
||||
You can follow below steps to enable this feature
|
||||
To enable this feature, follow the steps below.
|
||||
|
||||
## Creating SSL/TLS Certificates
|
||||
|
||||
It seems like step one is to generate certificates and key files for encrypting communication. I thought this would be
|
||||
fairly straightforward using `openssl` from the command line.
|
||||
The first step is to generate certificates and key files for encrypting communication. This is
|
||||
fairly straightforward: use `openssl` from the command line.
|
||||
|
||||
Use this [script](../../../../tools/TLS/tls_key_generate.sh) if you are not familiar with how to generate key files.
|
||||
|
||||
We need below files:
|
||||
- `server.pem` a private RSA key to sign and authenticate the public key. It's either a PKCS#8(PEM) or PKCS#1(DER).
|
||||
- `server.crt` self-signed X.509 public keys for distribution.
|
||||
- `ca.crt` a certificate authority public key for a client to validate the server's certificate.
|
||||
We need the following files:
|
||||
- `server.pem`: A private RSA key to sign and authenticate the public key. It's either a PKCS#8(PEM) or PKCS#1(DER).
|
||||
- `server.crt`: Self-signed X.509 public keys for distribution.
|
||||
- `ca.crt`: A certificate authority public key for a client to validate the server's certificate.
|
||||
|
||||
## Config OAP server
|
||||
|
||||
You can enable gRPC SSL by add following lines to `application.yml/core/default`.
|
||||
You can enable gRPC SSL by adding the following lines to `application.yml/core/default`.
|
||||
```json
|
||||
gRPCSslEnabled: true
|
||||
gRPCSslKeyPath: /path/to/server.pem
|
||||
|
|
@ -28,12 +28,12 @@ gRPCSslCertChainPath: /path/to/server.crt
|
|||
gRPCSslTrustedCAPath: /path/to/ca.crt
|
||||
```
|
||||
|
||||
`gRPCSslKeyPath` and `gRPCSslCertChainPath` are loaded by OAP server to encrypt the communication. `gRPCSslTrustedCAPath`
|
||||
helps gRPC client to verify server certificates in cluster mode.
|
||||
`gRPCSslKeyPath` and `gRPCSslCertChainPath` are loaded by the OAP server to encrypt communication. `gRPCSslTrustedCAPath`
|
||||
helps the gRPC client to verify server certificates in cluster mode.
|
||||
|
||||
When new files are in place, they can be load dynamically instead of restarting OAP instance.
|
||||
When new files are in place, they can be loaded dynamically, and you won't have to restart an OAP instance.
|
||||
|
||||
If you enable `sharding-server` to ingest data from external, add following lines to `application.yml/receiver-sharing-server/default`:
|
||||
If you enable `sharding-server` to ingest data from an external source, add the following lines to `application.yml/receiver-sharing-server/default`:
|
||||
|
||||
```json
|
||||
gRPCSslEnabled: true
|
||||
|
|
@ -41,6 +41,6 @@ gRPCSslKeyPath: /path/to/server.pem
|
|||
gRPCSslCertChainPath: /path/to/server.crt
|
||||
```
|
||||
|
||||
Because `sharding-server` only receives data from external, so it doesn't need CA at all.
|
||||
Since `sharding-server` only receives data from an external source, it doesn't need a CA at all.
|
||||
|
||||
If you port to java agent, refer to [TLS.md](../service-agent/java-agent/TLS.md) to config java agent to enable TLS.
|
||||
If you port to java agent, refer to [TLS.md](../service-agent/java-agent/TLS.md) to config java agent and enable TLS.
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# Log Collecting And Analysis
|
||||
# Log Collection and Analysis
|
||||
|
||||
## Collecting
|
||||
There are various ways to collect logs from application.
|
||||
## Collection
|
||||
There are various ways to collect logs from applications.
|
||||
|
||||
### Log files collector
|
||||
|
||||
|
|
@ -11,42 +11,40 @@ HTTP protocol, with the formats [Kafka JSON](../../protocols/Log-Data-Protocol.m
|
|||
or [HTTP JSON array](../../protocols/Log-Data-Protocol.md#http-api).
|
||||
|
||||
#### Filebeat
|
||||
Filebeat supports using Kafka to transport logs, you need to
|
||||
open [kafka-fetcher](backend-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
|
||||
Filebeat supports using Kafka to transport logs. Open [kafka-fetcher](backend-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
|
||||
|
||||
Take the following filebeat config yaml as an example to set up Filebeat
|
||||
Take the following filebeat config yaml as an example to set up Filebeat:
|
||||
- [filebeat.yml](../../../../test/e2e/e2e-test/docker/kafka/filebeat.yml)
|
||||
|
||||
#### Fluentd
|
||||
Fluentd supports using Kafka to transport logs, you need to
|
||||
open [kafka-fetcher](backend-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
|
||||
Fluentd supports using Kafka to transport logs. Open [kafka-fetcher](backend-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
|
||||
|
||||
Take the following fluentd config file as an example to set up Fluentd
|
||||
Take the following fluentd config file as an example to set up Fluentd:
|
||||
- [fluentd.conf](../../../../test/e2e/e2e-test/docker/kafka/fluentd.conf)
|
||||
|
||||
#### Fluent-bit
|
||||
Fluent-bit sends logs to OAP through HTTP(rest port) directly.
|
||||
Point the output address to `restHost`:`restPort` of `receiver-sharing-server` or `core`(if `receiver-sharing-server` inactivated)
|
||||
Fluent-bit sends logs to OAP directly through HTTP(rest port).
|
||||
Point the output address to `restHost`:`restPort` of `receiver-sharing-server` or `core`(if `receiver-sharing-server` is inactivated)
|
||||
|
||||
Take the following fluent-bit config files as an example to set up Fluent-bit
|
||||
Take the following fluent-bit config files as an example to set up Fluent-bit:
|
||||
- [fluent-bit.conf](../../../../test/e2e/e2e-test/docker/log/fluent-bit)
|
||||
|
||||
### Java agent's toolkits
|
||||
Java agent provides toolkit for
|
||||
Java agent provides toolkits for
|
||||
[log4j](../service-agent/java-agent/Application-toolkit-log4j-1.x.md),
|
||||
[log4j2](../service-agent/java-agent/Application-toolkit-log4j-2.x.md),
|
||||
[log4j2](../service-agent/java-agent/Application-toolkit-log4j-2.x.md), and
|
||||
[logback](../service-agent/java-agent/Application-toolkit-logback-1.x.md)
|
||||
to report logs through gRPC with automatic injected trace context.
|
||||
to report logs through gRPC with automatically injected trace context.
|
||||
|
||||
[SkyWalking Satellite sidecar](https://github.com/apache/skywalking-satellite) is a recommended proxy/side to
|
||||
forward logs including to use Kafka MQ to transport logs. When use this, need to open [kafka-fetcher](backend-fetcher.md#kafka-fetcher)
|
||||
[SkyWalking Satellite sidecar](https://github.com/apache/skywalking-satellite) is a recommended proxy/side that
|
||||
forwards logs (including the use of Kafka MQ to transport logs). When using this, open [kafka-fetcher](backend-fetcher.md#kafka-fetcher)
|
||||
and enable configs `enableNativeProtoLog`.
|
||||
|
||||
Java agent provides toolkit for
|
||||
Java agent provides toolkits for
|
||||
[log4j](../service-agent/java-agent/Application-toolkit-log4j-1.x.md#print-skywalking-context-in-your-logs),
|
||||
[log4j2](../service-agent/java-agent/Application-toolkit-log4j-2.x.md#print-skywalking-context-in-your-logs),
|
||||
[log4j2](../service-agent/java-agent/Application-toolkit-log4j-2.x.md#print-skywalking-context-in-your-logs), and
|
||||
[logback](../service-agent/java-agent/Application-toolkit-logback-1.x.md#print-skywalking-context-in-your-logs)
|
||||
to report logs through files with automatic injected trace context.
|
||||
to report logs through files with automatically injected trace context.
|
||||
|
||||
Log framework config examples:
|
||||
- [log4j1.x fileAppender](../../../../test/e2e/e2e-service-provider/src/main/resources/log4j.properties)
|
||||
|
|
@ -54,7 +52,7 @@ Log framework config examples:
|
|||
- [logback fileAppender](../../../../test/e2e/e2e-service-provider/src/main/resources/logback.xml)
|
||||
|
||||
### Python agent log reporter
|
||||
[SkyWalking Python Agent](https://github.com/apache/skywalking-python) implements a log reporter for the [Logging
|
||||
[SkyWalking Python Agent](https://github.com/apache/skywalking-python) implements a log reporter for the [logging
|
||||
module](https://docs.python.org/3/library/logging.html) with functionalities aligning with the Java toolkits.
|
||||
|
||||
To explore how to enable the reporting features for your use cases, please refer to the
|
||||
|
|
@ -63,8 +61,8 @@ To explore how to enable the reporting features for your use cases, please refer
|
|||
## Log Analyzer
|
||||
|
||||
Log analyzer of OAP server supports native log data. OAP could use Log Analysis Language to
|
||||
structurize log content through parse, extract, and save logs.
|
||||
Also the analyzer leverages Meter Analysis Language Engine for further metrics calculation.
|
||||
structure log content through parsing, extracting, and saving logs.
|
||||
The analyzer also uses Meter Analysis Language Engine for further metrics calculation.
|
||||
|
||||
```yaml
|
||||
log-analyzer:
|
||||
|
|
@ -74,4 +72,4 @@ log-analyzer:
|
|||
malFiles: ${SW_LOG_MAL_FILES:""}
|
||||
```
|
||||
|
||||
Read [Log Analysis Language](../../concepts-and-designs/lal.md) documentation to learn log structurize and metrics analysis.
|
||||
Read the doc on [Log Analysis Language](../../concepts-and-designs/lal.md) for more on log structuring and metrics analysis.
|
||||
|
|
|
|||
|
|
@ -1,15 +1,15 @@
|
|||
# Metrics Exporter
|
||||
SkyWalking provides basic and most important metrics aggregation, alarm and analysis.
|
||||
In real world, people may want to forward the data to their 3rd party system, for deeper analysis or anything else.
|
||||
**Metrics Exporter** makes that possible.
|
||||
SkyWalking provides the essential functions of metrics aggregation, alarm, and analysis.
|
||||
In the real world, many may want to forward their data to a 3rd party system for an in-depth analysis or otherwise.
|
||||
**Metrics Exporter** has made that possible.
|
||||
|
||||
Metrics exporter is an independent module, you need manually active it.
|
||||
Metrics exporter is an independent module that has to be manually activated.
|
||||
|
||||
Right now, we provide the following exporters
|
||||
Right now, we provide the following exporters:
|
||||
1. gRPC exporter
|
||||
|
||||
## gRPC exporter
|
||||
gRPC exporter uses SkyWalking native exporter service definition. Here is proto definition.
|
||||
gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
|
||||
```proto
|
||||
service MetricExportService {
|
||||
rpc export (stream ExportMetricValue) returns (ExportResponse) {
|
||||
|
|
@ -60,7 +60,7 @@ message ExportResponse {
|
|||
}
|
||||
```
|
||||
|
||||
To active the exporter, you should add this into your `application.yml`
|
||||
To activate the exporter, you should add this into your `application.yml`
|
||||
```yaml
|
||||
exporter:
|
||||
grpc:
|
||||
|
|
@ -69,13 +69,13 @@ exporter:
|
|||
```
|
||||
|
||||
- `targetHost`:`targetPort` is the expected target service address. You could set any gRPC server to receive the data.
|
||||
- Target gRPC service needs to be standby, otherwise, the OAP starts up failure.
|
||||
- Target gRPC service needs to go on standby; otherwise, the OAP startup may fail.
|
||||
|
||||
## For target exporter service
|
||||
### subscription implementation
|
||||
Return the expected metrics name list with event type(increment or total), all the names must match the OAL/MAL script definition.
|
||||
Return empty list, if you want to export all metrics in increment event type.
|
||||
## Target exporter service
|
||||
### Subscription implementation
|
||||
Return the expected metrics name list with event type (incremental or total). All names must match the OAL/MAL script definition.
|
||||
Return empty list, if you want to export all metrics in the incremental event type.
|
||||
|
||||
### export implementation
|
||||
Stream service, all subscribed metrics will be sent to here, based on OAP core schedule. Also, if the OAP deployed as cluster,
|
||||
then this method will be called concurrently. For metrics value, you need follow `#type` to choose `#longValue` or `#doubleValue`.
|
||||
### Export implementation
|
||||
Stream service. All subscribed metrics will be sent here based on the OAP core schedule. Also, if the OAP is deployed as cluster,
|
||||
this method will be called concurrently. For metrics value, you need to follow `#type` to choose `#longValue` or `#doubleValue`.
|
||||
|
|
|
|||
Loading…
Reference in New Issue