kazusa
/
skywalking-java
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Refine lal.md (#6626)

This commit is contained in:
Wing 2021-03-26 12:04:55 +08:00 committed by GitHub
parent f3ca0d0ea5
commit f94bf971fd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 28 additions and 30 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
docs/en/concepts-and-designs/lal.md
View File

@ -1,32 +1,31 @@
# Log Analysis Language # Log Analysis Language
Log Analysis Language (LAL) in SkyWalking is essentially a Domain-Specific Language (DSL) to analyze logs. You can use Log Analysis Language (LAL) in SkyWalking is essentially a Domain-Specific Language (DSL) to analyze logs. You can use
LAL to parse, extract, and save the logs, as well as collaborate the logs with traces (by extracting the trace id, LAL to parse, extract, and save the logs, as well as collaborate the logs with traces (by extracting the trace ID,
segment id and span id) and metrics (by generating metrics from the logs and send them to the meter system). segment ID and span ID) and metrics (by generating metrics from the logs and sending them to the meter system).
The LAL config files are in YAML format, and are located under directory `lal`, you can The LAL config files are in YAML format, and are located under directory `lal`. You can
set `log-analyzer/default/lalFiles` in the `application.yml` file or set environment variable `SW_LOG_LAL_FILES` to set `log-analyzer/default/lalFiles` in the `application.yml` file or set environment variable `SW_LOG_LAL_FILES` to
activate specific LAL config files. activate specific LAL config files.
## Filter ## Filter
A filter is a group of [parser](#parser), [extractor](#extractor) and [sink](#sink). Users can use one or more filters A filter is a group of [parser](#parser), [extractor](#extractor) and [sink](#sink). Users can use one or more filters
to organize their processing logics. Every piece of log will be sent to all filters in an LAL rule. The piece of log to organize their processing logic. Every piece of log will be sent to all filters in an LAL rule. A piece of log
sent into the filter is available as property `log` in the LAL, therefore you can access the log service name sent to the filter is available as property `log` in the LAL, therefore you can access the log service name
via `log.service`, for all available fields of `log`, please refer to [the protocol definition](https://github.com/apache/skywalking-data-collect-protocol/blob/master/logging/Logging.proto#L41). via `log.service`. For all available fields of `log`, please refer to [the protocol definition](https://github.com/apache/skywalking-data-collect-protocol/blob/master/logging/Logging.proto#L41).
All components are executed sequentially in the orders they are declared. All components are executed sequentially in the orders they are declared.
### Global Functions ### Global Functions
There are functions globally available that you can use them in all components (i.e. parsers, extractors, and sinks) Globally available functions may be used them in all components (i.e. parsers, extractors, and sinks) where necessary.
when needed.
- `abort` - `abort`
By default, all components declared are executed no matter what flags (`dropped`, `saved`, etc.) have been set. There By default, all components declared are executed no matter what flags (`dropped`, `saved`, etc.) have been set. There
are cases where you may want the filter chain to stop earlier when specified conditions are met. `abort` function aborts are cases where you may want the filter chain to stop earlier when specified conditions are met. `abort` function aborts
the remaining filter chain from where it's declared, all the remaining components won't be executed at all. the remaining filter chain from where it's declared, and all the remaining components won't be executed at all.
`abort` function serves as a fast-fail mechanism in LAL. `abort` function serves as a fast-fail mechanism in LAL.
```groovy ```groovy
@ -47,8 +46,8 @@ Parsers are responsible for parsing the raw logs into structured data in SkyWalk
types of parsers at the moment, namely `json`, `yaml`, and `text`. types of parsers at the moment, namely `json`, `yaml`, and `text`.
When a piece of log is parsed, there is a corresponding property available, called `parsed`, injected by LAL. When a piece of log is parsed, there is a corresponding property available, called `parsed`, injected by LAL.
Property `parsed` is typically a map, containing all the fields parsed from the raw logs, for example, if the parser Property `parsed` is typically a map, containing all the fields parsed from the raw logs. For example, if the parser
is `json` / `yaml`, `parsed` is a map containing all the key-values in the `json` / `yaml`, if the parser is `text` is `json` / `yaml`, `parsed` is a map containing all the key-values in the `json` / `yaml`; if the parser is `text`
, `parsed` is a map containing all the captured groups and their values (for `regexp` and `grok`). , `parsed` is a map containing all the captured groups and their values (for `regexp` and `grok`).
All parsers share the following options: All parsers share the following options:
@ -108,7 +107,7 @@ filter {
- `grok` (TODO) - `grok` (TODO)
Because grok Java library has performance issue, we need some investigations and benchmark on it. Contributions are We're aware of certains performance issues in the grok Java library, and so we're currently conducting investigations and benchmarking. Contributions are
welcome. welcome.
### Extractor ### Extractor
@ -155,8 +154,7 @@ The unit of `timestamp` is millisecond.
- `tag` - `tag`
`tag` extracts the tags from the `parsed` result, and set them into the `LogData`. The form of this extractor is `tag` extracts the tags from the `parsed` result, and set them into the `LogData`. The form of this extractor should look something like this: `tag key1: value, key2: value2`. You may use the properties of `parsed` as both keys and values.
something like `tag key1: value, key2: value2`, you can use the properties of `parsed` as both keys and values.
```groovy ```groovy
filter { filter {
@ -171,9 +169,9 @@ filter {
- `metrics` - `metrics`
`metrics` extracts / generates metrics from the logs, and sends the generated metrics to the meter system, you can `metrics` extracts / generates metrics from the logs, and sends the generated metrics to the meter system. You may
configure [MAL](mal.md) for further analysis of these metrics. The dedicated MAL config files are under configure [MAL](mal.md) for further analysis of these metrics. The dedicated MAL config files are under
directory `log-mal-rules`, you can set `log-analyzer/default/malFiles` to enable configured files. directory `log-mal-rules`, and you can set `log-analyzer/default/malFiles` to enable configured files.
```yaml ```yaml
# application.yml # application.yml
@ -209,7 +207,7 @@ filter {
} }
``` ```
The extractor above generates a metrics named `log_count`, with tag key `level` and value `1`, after this, you can The extractor above generates a metrics named `log_count`, with tag key `level` and value `1`. After that, you can
configure MAL rules to calculate the log count grouping by logging level like this: configure MAL rules to calculate the log count grouping by logging level like this:
```yaml ```yaml
@ -223,7 +221,7 @@ metrics:
``` ```
The other metrics generated is `http_response_time`, so that you can configure MAL rules to generate more useful metrics The other metrics generated is `http_response_time`, so you can configure MAL rules to generate more useful metrics
like percentiles. like percentiles.
```yaml ```yaml
@ -237,17 +235,17 @@ metrics:
### Sink ### Sink
Sinks are the persistent layer of the LAL. By default, all the logs of each filter are persisted into the storage. Sinks are the persistent layer of the LAL. By default, all the logs of each filter are persisted into the storage.
However, there are some mechanisms that allow you to selectively save some logs, or even drop all the logs after you've However, some mechanisms allow you to selectively save some logs, or even drop all the logs after you've
extracted useful information, such as metrics. extracted useful information, such as metrics.
#### Sampler #### Sampler
Sampler allows you to save the logs in a sampling manner. Currently, sampling strategy `rateLimit` is supported, welcome Sampler allows you to save the logs in a sampling manner. Currently, the sampling strategy `rateLimit` is supported. We welcome
to contribute more sampling strategies. If multiple samplers are specified, the last one determines the final sampling contributions on more sampling strategies. If multiple samplers are specified, the last one determines the final sampling
result, see examples in [Enforcer](#enforcer). result. See examples in [Enforcer](#enforcer).
`rateLimit` samples `n` logs at most in 1 second. `rateLimit("SamplerID")` requires an ID for the sampler, sampler `rateLimit` samples `n` logs at a maximum rate of 1 second. `rateLimit("SamplerID")` requires an ID for the sampler. Sampler
declarations with the same ID share the same sampler instance, and thus share the same `qps`, resetting logics. declarations with the same ID share the same sampler instance, thus sharing the same `qps` and resetting logic.
Examples: Examples:
@ -273,8 +271,8 @@ filter {
#### Dropper #### Dropper
Dropper is a special sink, meaning that all the logs are dropped without any exception. This is useful when you want to Dropper is a special sink, meaning that all logs are dropped without any exception. This is useful when you want to
drop debugging logs, drop debugging logs.
```groovy ```groovy
filter { filter {
@ -292,7 +290,7 @@ filter {
} }
``` ```
or you have multiple filters, some of which are for extracting metrics, only one of them needs to be persisted. Or if you have multiple filters, some of which are for extracting metrics, only one of them has to be persisted.
```groovy ```groovy
filter { // filter A: this is for persistence filter { // filter A: this is for persistence
@ -319,9 +317,9 @@ filter { // filter B:
#### Enforcer #### Enforcer
Enforcer is another special sink that forcibly samples the log, a typical use case of enforcer is when you have Enforcer is another special sink that forcibly samples the log. A typical use case of enforcer is when you have
configured a sampler and want to save some logs forcibly, for example, to save error logs even if the sampling mechanism configured a sampler and want to save some logs forcibly, such as to save error logs even if the sampling mechanism
is configured. has been configured.
```groovy ```groovy
filter { filter {