kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Support adding additional attr[0-5] for endpoint level metrics. (#12753)

This commit is contained in:
Wan Kai 2024-11-06 12:51:26 +08:00 committed by GitHub
parent df7450b4b4
commit 2ed6b58073
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
17 changed files with 313 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/en/api/metrics-query-expression.md
View File

@ -240,7 +240,7 @@ top_n(<metric_name>, <top_number>, <order>, <attrs>)
The attrs filter also supports not-equal filter `!=`, the format is `attr0 != 'value'`.
**Notice**:
- The `attrs` only support Service metrics for now and should be added in the metrics first, see [Metrics Additional Attributes](../concepts-and-designs/metrics-additional-attributes.md).
- The `attrs` should be added in the metrics first, see [Metrics Additional Attributes](../concepts-and-designs/metrics-additional-attributes.md).
- When use not-equal filter, for example `attr1 != 'value'`, if the storage is using `MySQL` or other JDBC storage and `attr1 value is NULL` in the metrics,
the result of `attr1 != 'value'` will always `false` and would NOT include this metric in the result due to SQL can't compare `NULL` with the `value`.

5
docs/en/changes/changes.md
View File

@ -11,7 +11,7 @@
* Change the endpoint_traffic to updatable for the additional column `last_ping`.
* Add Component ID(5023) for the GoZero framework.
* Support Kong monitoring.
* Support adding additional attr[0-4] for service level metrics.
* Support adding additional attr[0-5] for service/endpoint level metrics.
* Support async-profiler feature for performance analysis.
* Add metrics value owner for metrics topN query result.
* Add naming control for `EndpointDependencyBuilder`.
@ -28,6 +28,9 @@
* Add service global topN widget on `General-Root`, `Mesh-Root`, `K8S-Root` dashboard.
* Fix initialization dashboards.
* Update the Kubernetes metrics for reduce multiple metrics calculate in MQE.
* Support view data value related dashboards in TopList widgets.
* Add endpoint global topN widget on `General-Root`, `Mesh-Root`.
#### Documentation
* Update release document to adopt newly added revision-based process.

22
docs/en/concepts-and-designs/metrics-additional-attributes.md
View File

@ -18,7 +18,10 @@ and the Java Class must follow the following rules:
- The Class must implement the [ISourceDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/ISourceDecorator.java) interface.
- The Class package must be under the `org.apache.skywalking.*`.
SkyWalking provides a default implementation [ServiceDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ServiceDecorator.java) which set the `attr0` to the service `Layer`.
### Default Decorator
SkyWalking provides some default implementation of decorator:
- [ServiceDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ServiceDecorator.java) which set the `attr0` to the service `Layer`.
The following OAL metrics had been decorated by the `ServiceDecorator` by default:
```text
// Service scope metrics
@ -28,6 +31,23 @@ service_cpm = from(Service.*).cpm().decorator("ServiceDecorator");
service_apdex = from(Service.latency).apdex(name, status).decorator("ServiceDecorator");
```
- [EndpointDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/EndpointDecorator.java) which set the `attr0` to the endpoint `Layer`.
The following OAL metrics had been decorated by the `EndpointDecorator` by default:
```text
endpoint_cpm = from(Endpoint.*).cpm().decorator("EndpointDecorator");
endpoint_resp_time = from(Endpoint.latency).longAvg().decorator("EndpointDecorator");
endpoint_sla = from(Endpoint.*).percent(status == true).decorator("EndpointDecorator");
```
- [K8SServiceDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/K8SServiceDecorator.java) which set the `attr0` to the k8s service `Layer`.
The following OAL metrics had been decorated by the `K8SServiceDecorator` by default:
```text
kubernetes_service_http_call_cpm = from(K8SService.*).filter(detectPoint == DetectPoint.SERVER).filter(type == "protocol").filter(protocol.type == "http").cpm().decorator("K8SServiceDecorator");
kubernetes_service_http_call_time = from(K8SService.protocol.http.latency).filter(detectPoint == DetectPoint.SERVER).filter(type == "protocol").filter(protocol.type == "http").longAvg().decorator("K8SServiceDecorator");
kubernetes_service_http_call_successful_rate = from(K8SService.*).filter(detectPoint == DetectPoint.SERVER).filter(type == "protocol").filter(protocol.type == "http").percent(protocol.success == true).decorator("K8SServiceDecorator");
kubernetes_service_apdex = from(K8SService.protocol.http.latency).filter(detectPoint == DetectPoint.SERVER).filter(type == "protocol").filter(protocol.type == "http").apdex(name, protocol.success).decorator("K8SServiceDecorator");
```
## MAL Source Decorate
In the MAL script, you can use the [decorate](mal.md#decorate-function) function to decorate the source, and must follow the following rules:
- The decorate function must after service() function.

6
oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java
View File

@ -125,8 +125,10 @@ public class OALListener extends OALParserBaseListener {
current.setSourceDecorator(decoratorNameTrim);
Map<String, ISourceDecorator<ISource>> map = SourceDecoratorManager.DECORATOR_MAP;
int currentScopeId = current.getFrom().getSourceScopeId();
if (currentScopeId != DefaultScopeDefine.SERVICE && currentScopeId != DefaultScopeDefine.K8S_SERVICE) {
throw new IllegalArgumentException("OAL metric: " + current.getMetricsName() + ", decorate source only support service scope.");
if (currentScopeId != DefaultScopeDefine.SERVICE
&& currentScopeId != DefaultScopeDefine.K8S_SERVICE
&& currentScopeId != DefaultScopeDefine.ENDPOINT) {
throw new IllegalArgumentException("OAL metric: " + current.getMetricsName() + ", decorate source not support scope: " + DefaultScopeDefine.nameOf(currentScopeId));
}
ISourceDecorator<ISource> decorator = map.get(decoratorNameTrim);
if (decorator == null) {

4
oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/DeepAnalysisTest.java
View File

@ -89,7 +89,7 @@ public class DeepAnalysisTest {
Assertions.assertEquals("(long)(1)", method.getArgsExpressions().get(1));
List<SourceColumn> source = result.getFieldsFromSource();
Assertions.assertEquals(2, source.size());
Assertions.assertEquals(8, source.size());
List<DataColumn> persistentFields = result.getPersistentFields();
Assertions.assertEquals(4, persistentFields.size());
@ -117,7 +117,7 @@ public class DeepAnalysisTest {
Assertions.assertEquals("(long)(1)", method.getArgsExpressions().get(1));
List<SourceColumn> source = result.getFieldsFromSource();
Assertions.assertEquals(2, source.size());
Assertions.assertEquals(8, source.size());
List<DataColumn> persistentFields = result.getPersistentFields();
Assertions.assertEquals(4, persistentFields.size());

35
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/Endpoint.java
View File

@ -23,7 +23,9 @@ import java.util.Map;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.analysis.ISourceDecorator;
import org.apache.skywalking.oap.server.core.analysis.Layer;
import org.apache.skywalking.oap.server.core.analysis.SourceDecoratorManager;
import org.apache.skywalking.oap.server.library.util.StringUtil;
import org.apache.skywalking.oap.server.core.analysis.IDManager;
@ -89,6 +91,39 @@ public class Endpoint extends Source {
@Getter
@Setter
private Layer serviceLayer;
@Getter
@Setter
@ScopeDefaultColumn.DefinedByField(columnName = "attr0", isAttribute = true)
private String attr0;
@Getter
@Setter
@ScopeDefaultColumn.DefinedByField(columnName = "attr1", isAttribute = true)
private String attr1;
@Getter
@Setter
@ScopeDefaultColumn.DefinedByField(columnName = "attr2", isAttribute = true)
private String attr2;
@Getter
@Setter
@ScopeDefaultColumn.DefinedByField(columnName = "attr3", isAttribute = true)
private String attr3;
@Getter
@Setter
@ScopeDefaultColumn.DefinedByField(columnName = "attr4", isAttribute = true)
private String attr4;
@Getter
@Setter
@ScopeDefaultColumn.DefinedByField(columnName = "attr5", isAttribute = true)
private String attr5;
/**
* Get the decorator through given name and invoke.
* @param decorator The decorator class simpleName.
*/
public void decorate(String decorator) {
ISourceDecorator<ISource> sourceDecorator = SourceDecoratorManager.DECORATOR_MAP.get(decorator);
sourceDecorator.decorate(this);
}
@Override
public void prepare() {

37
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/EndpointDecorator.java Normal file
View File

@ -0,0 +1,37 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.source;
import org.apache.skywalking.oap.server.core.analysis.ISourceDecorator;
public class EndpointDecorator implements ISourceDecorator<Endpoint> {
@Override
public int getSourceScope() {
return DefaultScopeDefine.ENDPOINT;
}
/**
* Set the Layer name to attr0
* @param source The source instance to be decorated
*/
@Override
public void decorate(final Endpoint source) {
source.setAttr0(source.getServiceLayer().name());
}
}

5
oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/AsyncQueryUtils.java
View File

@ -22,6 +22,7 @@ import java.util.concurrent.Callable;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Executor;
import java.util.concurrent.ForkJoinPool;
import lombok.extern.slf4j.Slf4j;
import static java.util.concurrent.ForkJoinPool.defaultForkJoinWorkerThreadFactory;
@ -29,6 +30,7 @@ import static java.util.concurrent.ForkJoinPool.defaultForkJoinWorkerThreadFacto
* The utility class for async GraphQL query.
* All the async GraphQL query should be wrapped by this class and shared the same executor.
*/
@Slf4j
public class AsyncQueryUtils {
private static final Executor EXECUTOR = new ForkJoinPool(
Runtime.getRuntime().availableProcessors(), defaultForkJoinWorkerThreadFactory, null, true);
@ -37,7 +39,8 @@ public class AsyncQueryUtils {
return CompletableFuture.supplyAsync(() -> {
try {
return caller.call();
} catch (Exception e) {
} catch (Throwable e) {
log.error(e.getMessage(), e);
throw new RuntimeException(e);
}
}, EXECUTOR);

6
oap-server/server-starter/src/main/resources/oal/core.oal
View File

@ -51,9 +51,9 @@ service_instance_resp_time = from(ServiceInstance.latency).longAvg();
service_instance_cpm = from(ServiceInstance.*).cpm();
// Endpoint scope metrics
endpoint_cpm = from(Endpoint.*).cpm();
endpoint_resp_time = from(Endpoint.latency).longAvg();
endpoint_sla = from(Endpoint.*).percent(status == true);
endpoint_cpm = from(Endpoint.*).cpm().decorator("EndpointDecorator");
endpoint_resp_time = from(Endpoint.latency).longAvg().decorator("EndpointDecorator");
endpoint_sla = from(Endpoint.*).percent(status == true).decorator("EndpointDecorator");
endpoint_percentile = from(Endpoint.latency).percentile2(10); // Multiple values including p50, p75, p90, p95, p99
endpoint_mq_consume_latency = from((str->long)Endpoint.tag["transmission.latency"]).filter(type == RequestType.MQ).filter(tag["transmission.latency"] != null).longAvg();

74
oap-server/server-starter/src/main/resources/ui-initialized-templates/general/general-root.json
View File

@ -5,7 +5,7 @@
"children": [
{
"x": 0,
"y": 13,
"y": 24,
"w": 24,
"h": 52,
"i": "1",
@ -211,7 +211,8 @@
},
"widget": {
"title": "Service Apdex"
}
},
"valueRelatedDashboard": "General-Service"
},
{
"x": 12,
@ -229,7 +230,8 @@
},
"widget": {
"title": "Service Avg Response Time (ms)"
}
},
"valueRelatedDashboard": "General-Service"
},
{
"x": 6,
@ -242,12 +244,13 @@
"top_n(service_sla,10,asc,attr0='GENERAL')/100"
],
"widget": {
"title": "Success Rate"
"title": "Service Success Rate"
},
"graph": {
"type": "TopList",
"color": "purple"
}
},
"valueRelatedDashboard": "General-Service"
},
{
"x": 18,
@ -266,7 +269,66 @@
"graph": {
"type": "TopList",
"color": "purple"
}
},
"valueRelatedDashboard": "General-Service"
},
{
"x": 0,
"y": 13,
"w": 8,
"h": 11,
"i": "105",
"type": "Widget",
"expressions": [
"top_n(endpoint_sla,10,asc,attr0='GENERAL')/100"
],
"graph": {
"type": "TopList",
"color": "purple"
},
"widget": {
"title": "Endpoint Success Rate"
},
"valueRelatedDashboard": "General-Endpoint"
},
{
"x": 8,
"y": 13,
"w": 8,
"h": 11,
"i": "106",
"type": "Widget",
"expressions": [
"top_n(endpoint_resp_time,10,des,attr0='GENERAL')"
],
"graph": {
"type": "TopList",
"color": "purple"
},
"widget": {
"title": "Endpoint Avg Response Time (ms)"
},
"valueRelatedDashboard": "General-Endpoint"
},
{
"x": 16,
"y": 13,
"w": 8,
"h": 11,
"i": "107",
"type": "Widget",
"expressions": [
"top_n(endpoint_cpm,10,des,attr0='GENERAL')"
],
"graph": {
"type": "TopList",
"color": "purple"
},
"widget": {
"title": "Endpoint Load (calls / min)",
"tips": "For HTTP 1/2, gRPC, RPC services, this means Calls Per Minute (calls / min), for TCP services"
},
"valueRelatedDashboard": "General-Endpoint"
}
],
"layer": "GENERAL",

40
oap-server/server-starter/src/main/resources/ui-initialized-templates/general/general-service.json
View File

@ -451,7 +451,8 @@
},
"expressions": [
"top_n(service_instance_cpm,10,des)"
]
],
"valueRelatedDashboard": "General-Instance"
},
{
"x": 8,
@ -469,7 +470,8 @@
},
"expressions": [
"top_n(service_instance_resp_time,10,des)"
]
],
"valueRelatedDashboard": "General-Instance"
},
{
"x": 16,
@ -487,7 +489,8 @@
},
"expressions": [
"top_n(service_instance_sla,10,asc)/100"
]
],
"valueRelatedDashboard": "General-Instance"
},
{
"x": 16,
@ -505,7 +508,8 @@
},
"expressions": [
"top_n(endpoint_sla,10,asc)/100"
]
],
"valueRelatedDashboard": "General-Endpoint"
},
{
"x": 8,
@ -523,7 +527,8 @@
},
"expressions": [
"top_n(endpoint_resp_time,10,des)"
]
],
"valueRelatedDashboard": "General-Endpoint"
},
{
"x": 0,
@ -542,7 +547,8 @@
},
"expressions": [
"top_n(endpoint_cpm,10,des)"
]
],
"valueRelatedDashboard": "General-Endpoint"
}
]
},
@ -820,29 +826,9 @@
"eventAssociate": true
}
],
"name": "General-Service",
"layer": "GENERAL",
"entity": "Service",
"isDefault": true,
"expressions": [
"avg(service_cpm)",
"avg(service_sla/100)",
"avg(service_resp_time)"
],
"expressionsConfig": [
{
"unit": "calls / min",
"label": "Load"
},
{
"unit": "%",
"label": "Success Rate"
},
{
"unit": "ms",
"label": "Latency"
}
]
"name": "General-Service"
}
}
]

71
oap-server/server-starter/src/main/resources/ui-initialized-templates/mesh/mesh-root.json
View File

@ -5,7 +5,7 @@
"children": [
{
"x": 0,
"y": 13,
"y": 25,
"w": 24,
"h": 52,
"i": "1",
@ -197,7 +197,8 @@
},
"widget": {
"title": "Service Avg Response Time (ms)"
}
},
"valueRelatedDashboard": "Mesh-Service"
},
{
"x": 18,
@ -215,7 +216,8 @@
"graph": {
"type": "TopList",
"color": "purple"
}
},
"valueRelatedDashboard": "Mesh-Service"
},
{
"x": 6,
@ -228,12 +230,13 @@
"top_n(service_sla,10,asc,attr0='MESH')/100"
],
"widget": {
"title": "Success Rate"
"title": "Service Success Rate"
},
"graph": {
"type": "TopList",
"color": "purple"
}
},
"valueRelatedDashboard": "Mesh-Service"
},
{
"x": 0,
@ -251,6 +254,64 @@
},
"widget": {
"title": "Service Apdex"
},
"valueRelatedDashboard": "Mesh-Service"
},
{
"x": 0,
"y": 13,
"w": 8,
"h": 12,
"i": "105",
"type": "Widget",
"expressions": [
"top_n(endpoint_sla,10,asc,attr0='MESH')/100"
],
"graph": {
"type": "TopList",
"color": "purple"
},
"valueRelatedDashboard": "Mesh-Endpoint",
"widget": {
"title": "Endpoint Success Rate"
}
},
{
"x": 8,
"y": 13,
"w": 8,
"h": 12,
"i": "106",
"type": "Widget",
"expressions": [
"top_n(endpoint_resp_time,10,des,attr0='MESH')"
],
"graph": {
"type": "TopList",
"color": "purple"
},
"valueRelatedDashboard": "Mesh-Endpoint",
"widget": {
"title": "Endpoint Avg Response Time (ms)"
}
},
{
"x": 16,
"y": 13,
"w": 8,
"h": 12,
"i": "107",
"type": "Widget",
"expressions": [
"top_n(endpoint_cpm,10,des,attr0='MESH')"
],
"graph": {
"type": "TopList",
"color": "purple"
},
"valueRelatedDashboard": "Mesh-Endpoint",
"widget": {
"title": "Endpoint Load (calls or packets / min)"
}
}
],

40
oap-server/server-starter/src/main/resources/ui-initialized-templates/mesh/mesh-service.json
View File

@ -330,7 +330,8 @@
},
"expressions": [
"top_n(service_instance_cpm,10,des)"
]
],
"valueRelatedDashboard": "Mesh-Instance"
},
{
"x": 8,
@ -348,7 +349,8 @@
},
"expressions": [
"top_n(service_instance_resp_time,10,des)"
]
],
"valueRelatedDashboard": "Mesh-Instance"
},
{
"x": 16,
@ -366,7 +368,8 @@
},
"expressions": [
"top_n(service_instance_sla,10,asc)/100"
]
],
"valueRelatedDashboard": "Mesh-Instance"
},
{
"x": 16,
@ -384,7 +387,8 @@
},
"expressions": [
"top_n(endpoint_sla,10,asc)/100"
]
],
"valueRelatedDashboard": "Mesh-Endpoint"
},
{
"x": 8,
@ -402,7 +406,8 @@
},
"expressions": [
"top_n(endpoint_resp_time,10,des)"
]
],
"valueRelatedDashboard": "Mesh-Endpoint"
},
{
"x": 0,
@ -420,7 +425,8 @@
},
"expressions": [
"top_n(endpoint_cpm,10,des)"
]
],
"valueRelatedDashboard": "Mesh-Endpoint"
},
{
"x": 0,
@ -718,27 +724,7 @@
],
"layer": "MESH",
"entity": "Service",
"name": "Mesh-Service",
"isDefault": true,
"expressions": [
"avg(service_cpm)",
"avg(service_sla/100)",
"avg(service_resp_time)"
],
"expressionsConfig": [
{
"unit": "calls / min",
"label": "Load"
},
{
"unit": "%",
"label": "Success Rate"
},
{
"unit": "ms",
"label": "Latency"
}
]
"name": "Mesh-Service"
}
}
]

2
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBAggregationQueryDAO.java
View File

@ -65,7 +65,7 @@ public class BanyanDBAggregationQueryDAO extends AbstractBanyanDBDAO implements
}
// BanyanDB server-side TopN support for metrics pre-aggregation.
if (schema.getTopNSpec() != null) {
if (schema.getTopNSpec() != null && CollectionUtils.isEmpty(condition.getAttributes())) {
// 1) no additional conditions
// 2) additional conditions are all group by tags
if (CollectionUtils.isEmpty(additionalConditions) ||

2
skywalking-ui

@ -1 +1 @@
Subproject commit 224e761d70a005a21aa2f4fc4effde57dcfc4a5d
Subproject commit 14fa5d65b60570a77052793062eb5225de9132b5

32
test/e2e-v2/cases/storage/expected/topN-OP-endpoint.yml Normal file
View File

@ -0,0 +1,32 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
debuggingtrace: null
type: SORTED_LIST
results:
{{- contains .results }}
- metric:
labels: []
values:
{{- contains .values }}
- id: {{ notEmpty .id }}
value: "100"
traceid: null
- id: {{ notEmpty .id }}
value: "100"
traceid: null
{{- end }}
{{- end }}
error: null

8
test/e2e-v2/cases/storage/storage-cases.yaml
View File

@ -168,3 +168,11 @@ cases:
expected: expected/topN-OP-service.yml
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression="top_n(service_sla,3,des,attr0!='Not_GENERAL')/100"
expected: expected/topN-OP-service.yml
# topN-OP-endpoint Global
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression="top_n(endpoint_sla,3,des)/100"
expected: expected/topN-OP-endpoint.yml
# topN-OP-endpoint Global with attrs
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression="top_n(endpoint_sla,3,des,attr0='GENERAL')/100"
expected: expected/topN-OP-endpoint.yml
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression="top_n(endpoint_sla,3,des,attr0!='Not_GENERAL')/100"
expected: expected/topN-OP-endpoint.yml