kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[Feature] Redis slow log collection (#10825)

This commit is contained in:
yswdqz 2023-05-23 00:31:00 +08:00 committed by GitHub
parent c5512a839c
commit 81ec3af923
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
19 changed files with 452 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/skywalking.yaml vendored
View File

@ -677,7 +677,7 @@ jobs:
config: test/e2e-v2/cases/promql/e2e.yaml
- name: AWS API Gateway
config: test/e2e-v2/cases/aws/api-gateway/e2e.yaml
- name: Redis Prometheus
- name: Redis Prometheus and Log Collecting
config: test/e2e-v2/cases/redis/redis-exporter/e2e.yaml
- name: Elasticsearch
config: test/e2e-v2/cases/elasticsearch/e2e.yaml

1
docs/en/changes/changes.md
View File

@ -45,6 +45,7 @@
* Fix potential NPE in Zipkin receiver when the `Span` is missing some fields.
* Filter out unknown_cluster metric data.
* Support RabbitMQ Monitoring.
* Support Redis slow logs collection.
#### UI
* Revert: cpm5d function. This feature is cancelled from backend.

33
docs/en/setup/backend/backend-redis-monitoring.md
View File

@ -36,3 +36,36 @@ You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in `/config/otel-rules/redis`.
The Redis dashboard panel configurations are found in `/config/ui-initialized-templates/redis`.
## Collect sampled slow commands
SkyWalking leverages [fluentbit](https://fluentbit.io/) or other log agents for collecting slow commands from Redis.
### Data flow
1. Execute [commands](../../../../test/e2e-v2/cases/redis/redis-exporter/scripts/slowlog.sh) periodically to collect slow logs from Redis and save the result locally.
2. Fluent-bit agent collects slow logs from local file.
3. fluent-bit agent sends data to SkyWalking OAP Server using native meter APIs via HTTP.
4. The SkyWalking OAP Server parses the expression with [LAL](../../concepts-and-designs/lal.md) to parse/extract and store the results.
### Set up
1. Set up [fluentbit](https://docs.fluentbit.io/manual/installation/docker).
2. Config fluentbit from [here](../../../../test/e2e-v2/cases/redis/redis-exporter/fluent-bit.conf) for Redis.
3. Config slow log from [here](../../../../test/e2e-v2/cases/redis/redis-exporter/redis.conf) for Redis.
4. Periodically execute the [commands](../../../../test/e2e-v2/cases/redis/redis-exporter/scripts/slowlog.sh).
**Notice:**
1.The `slowlog-log-slower-than` and `slowlog-max-len` configuration items in the configuration file are for the slow log, the former indicating that execution time longer than the specified time (in milliseconds) will be logged to the slowlog, and the latter indicating the maximum number of slow logs that will be stored in the slow log file.
2.In the e2e test, SkyWalking uses cron to periodically execute the redis command to fetch the slow logs and write them to a local file, which is then collected by fluent-bit to send the data to the OAP. You can see the relevant configuration files [here](../../../../test/e2e-v2/cases/redis/redis-exporter).You can also get slow logs periodically and send them to OAP in other ways than using cron and fluent-bit.
### Slow Commands Monitoring
Slow SQL monitoring provides monitoring of the slow commands of the Redis servers. Redis servers are cataloged as a `Layer: REDIS` `Service` in OAP.
#### Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|-----|------|-----|--------------------------------------------------|-----|
|Slow Statements | ms | top_n_database_statement | The latency and statement of Redis slow commands | fluentbit|
### Customizations
You can customize your own metrics/expression/dashboard panel.
The slowsql expression rules are found in `/config/lal/redis-slowsql.yaml`
The Redis dashboard panel configurations are found in `/config/ui-initialized-templates/redis`.
`

2
oap-server/server-starter/src/main/resources/application.yml
View File

@ -238,7 +238,7 @@ agent-analyzer:
log-analyzer:
selector: ${SW_LOG_ANALYZER:default}
default:
lalFiles: ${SW_LOG_LAL_FILES:envoy-als,mesh-dp,mysql-slowsql,pgsql-slowsql,k8s-service,default}
lalFiles: ${SW_LOG_LAL_FILES:envoy-als,mesh-dp,mysql-slowsql,pgsql-slowsql,redis-slowsql,k8s-service,default}
malFiles: ${SW_LOG_MAL_FILES:""}
event-analyzer:

35
oap-server/server-starter/src/main/resources/lal/redis-slowsql.yaml Normal file
View File

@ -0,0 +1,35 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
rules:
- name: redis-slowsql
layer: REDIS
dsl: |
filter {
json{
}
extractor{
layer parsed.layer as String
service parsed.service as String
timestamp parsed.time as String
if (tag("LOG_KIND") == "SLOW_SQL") {
slowSql {
id parsed.id as String
statement parsed.statement as String
latency parsed.query_time as Long
}
}
}
}

31
oap-server/server-starter/src/main/resources/ui-initialized-templates/redis/redis-service.json
View File

@ -311,6 +311,37 @@
}
}
]
},
{
"name":"Slow Commands",
"children":[
{
"x":0,
"y":0,
"w":24,
"h":31,
"i":"0",
"type":"Widget",
"metricTypes":[
"readRecords"
],
"metrics":[
"top_n_database_statement"
],
"graph":{
"type":"TopList",
"color":"purple"
},
"widget":{
"title":"Slow Commands (ms)"
},
"metricConfig":[
{
"calculation":"msTos"
}
]
}
]
}
]
}

29
test/e2e-v2/cases/redis/expected/db-has-value.yml Normal file
View File

@ -0,0 +1,29 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{{- contains . }}
- key: 0
value:
{{- contains .value }}
- key: name
value: {{ notEmpty .value }}
- key: id
value: ""
- key: value
value: {{ notEmpty .value }}
- key: refid
value: ""
{{- end }}
{{- end }}

29
test/e2e-v2/cases/redis/expected/metrics-has-value-label.yml Normal file
View File

@ -0,0 +1,29 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{{- contains . }}
- key: {{ notEmpty .key }}
value:
{{- contains .value }}
- key: {{ notEmpty .key }}
value:
value: 0
isemptyvalue: true
- key: {{ notEmpty .key }}
value:
value: {{ ge .value.value 1 }}
isemptyvalue: false
{{- end }}
{{- end }}

5
test/e2e-v2/cases/redis/redis-cases.yaml
View File

@ -63,3 +63,8 @@ cases:
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_redis_instance_expired_keys_total --service-name=redis::root[root] --instance-name=redis_exporter_1:9121 |yq e 'to_entries' -
expected: expected/metrics-has-value.yml
# slow sql
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql records list --name=top_n_database_statement --service-name=redis::root[root] |yq e 'to_entries | with(.[] ; .value=(.value | to_entries))' -
expected: expected/db-has-value.yml

23
test/e2e-v2/cases/redis/redis-exporter/docker-compose.yml
View File

@ -24,6 +24,9 @@ services:
- 12800
networks:
- e2e
environment:
- TZ=Asia/Shanghai
- SW_CORE_TOPN_REPORT_PERIOD=2
redis_1: &redis
image: redis:6.0
networks:
@ -88,6 +91,26 @@ services:
depends_on:
oap:
condition: service_healthy
fluentbit:
image: fluent/fluent-bit:1.9
environment:
- TZ=Asia/Shanghai
volumes:
- ./scripts:/scripts
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
- ./fluent-bit-parser.conf:/fluent-bit/etc/fluent-bit-parser.conf
- ./fluent-bit-script.lua:/fluent-bit/etc/fluent-bit-script.lua
networks:
- e2e
depends_on:
- slowlog
slowlog:
image: redis:6.0
volumes:
- ./scripts:/scripts
networks:
- e2e
command: sh -c "chmod 777 /scripts/start.sh /scripts/slowlog.sh /scripts/slowlog.log && /scripts/start.sh"
networks:
e2e:

19
test/e2e-v2/cases/redis/redis-exporter/fluent-bit-parser.conf Normal file
View File

@ -0,0 +1,19 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[PARSER]
name my-log-format
format regex
regex ^\S+\s+\S+\s+\S+\s+([\S\s]+)\s+\S+:\S+$

132
test/e2e-v2/cases/redis/redis-exporter/fluent-bit-script.lua Normal file
View File

@ -0,0 +1,132 @@
--
-- Licensed to the Apache Software Foundation (ASF) under one or more
-- contributor license agreements. See the NOTICE file distributed with
-- this work for additional information regarding copyright ownership.
-- The ASF licenses this file to You under the Apache License, Version 2.0
-- (the "License"); you may not use this file except in compliance with
-- the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-- The following is an example of a simple Redis slow log
-- 102 1684379526 2691 flushall 192.168.150.29:42904
function rewrite_body(tag, timestamp, record)
local log = record.log
if isStringOnlyWhitespace(log) then
return -1, timestamp, record
end
if log:sub(1, 1) == "#" then
return -1, timestamp, record
end
record.log = nil
record.tags = { data = { { key = "LOG_KIND", value = "SLOW_SQL" } } }
local trimmed = trim(log)
local splitResult = split(trimmed, " ")
inner_record = {}
inner_record.time = os.time() * 1000
inner_record.layer = "REDIS"
record.layer = "REDIS"
-- Here simply set the service name to `root[root] for e2e testing, you can also just set ip:port to service name
record.service = "redis::" .. "root[root]"
inner_record.service = "redis::" .. "root[root]"
local query_time = splitResult[3]
local qt = math.floor(query_time)
inner_record.query_time = qt
inner_record.statement = ""
inner_record.id = splitResult[1]
for i = 4, #splitResult - 1 do
inner_record.statement = inner_record.statement .. splitResult[i]
inner_record.statement = inner_record.statement .. " "
end
inner_record.statement = trim(inner_record.statement)
inner_record.statement = string.gsub(inner_record.statement, '"', '\\"')
local jsonstr = table2json(inner_record)
record.body = { json = { json = jsonstr } }
return 1, timestamp, record
end
function trim(str)
return str:gsub("^%s*(.-)%s*$", "%1")
end
function split(str, delimiter)
local result = {}
for match in (str..delimiter):gmatch("(.-)"..delimiter) do
table.insert(result, match)
end
return result
end
function isStringOnlyWhitespace(str)
return str:match("^%s*$") ~= nil
end
function table2json(t)
local function serialize(tbl, structure)
local tmp = {}
for k, v in pairs(tbl) do
local k_type = type(k)
local v_type = type(v)
local key
if k_type == "string" then
key = '"' .. k .. '":'
elseif k_type == "number" then
key = ""
if structure == "array" then
key = ""
else
key = '"' .. k .. '":'
end
else
key = '"' .. tostring(k) .. '":'
end
local value
if v_type == "table" then
if next(v) == nil then
value = "null"
else
value = serialize(v, structure)
end
elseif v_type == "boolean" then
value = tostring(v)
elseif v_type == "string" then
value = '"' .. v .. '"'
else
value = v
end
tmp[#tmp + 1] = key and value and tostring(key) .. tostring(value) or nil
end
if structure == "array" then
return "[" .. table.concat(tmp, ",") .. "]"
else
return "{" .. table.concat(tmp, ",") .. "}"
end
end
assert(type(t) == "table")
if next(t) == nil then
return "null"
elseif next(t, next(t)) == nil and type(next(t)) == "number" then
return serialize(t, "array")
else
return serialize(t, "table")
end
end

40
test/e2e-v2/cases/redis/redis-exporter/fluent-bit.conf Normal file
View File

@ -0,0 +1,40 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[SERVICE]
flush 1
log_level info
parsers_File fluent-bit-parser.conf
[INPUT]
name tail
path /scripts/slowlog.log
read_from_head true
parser my-log-format
[FILTER]
name lua
match *
script fluent-bit-script.lua
call rewrite_body
[OUTPUT]
name stdout
match *
format json
[OUTPUT]
name http
match *
host oap
port 12800
uri /v3/logs
format json

1
test/e2e-v2/cases/redis/redis-exporter/mock.txt
View File

@ -3,6 +3,7 @@ set c d
get a
set e f
debug sleep 10
debug sleep 3
get a
get b
get c

4
test/e2e-v2/cases/redis/redis-exporter/redis.conf
View File

@ -54,8 +54,8 @@ auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
slowlog-log-slower-than 1000
slowlog-max-len 1200
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512

2
test/e2e-v2/cases/redis/redis-exporter/scripts/crontable.txt Normal file
View File

@ -0,0 +1,2 @@
* * * * * /scripts/slowlog.sh

16
test/e2e-v2/cases/redis/redis-exporter/scripts/slowlog.log Normal file
View File

@ -0,0 +1,16 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

28
test/e2e-v2/cases/redis/redis-exporter/scripts/slowlog.sh Normal file
View File

@ -0,0 +1,28 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
len=$(/usr/local/bin/redis-cli -h redis_1 slowlog len)
if [[ $len -gt 0 ]]; then
result=$(/usr/local/bin/redis-cli -h redis_1 slowlog get $len)
single_line_log="$(echo "$result" | tr '\n' ' ')"
processed_result=$(echo "$single_line_log" | sed 's/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}:[0-9]\{1,5\}/&\n/g')
echo "$processed_result" >> /scripts/slowlog.log
fi
/usr/local/bin/redis-cli -h redis_1 slowlog reset

24
test/e2e-v2/cases/redis/redis-exporter/scripts/start.sh Normal file
View File

@ -0,0 +1,24 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apt-get update && apt-get install -y cron
crontab /scripts/crontable.txt
service cron start
tail -f /dev/null