kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

PromQL Service: traffic query support `limit` and regex match. (#13162)

This commit is contained in:
Wan Kai 2025-04-09 14:51:19 +08:00 committed by GitHub
parent dc953802bf
commit 2e8d6e3a5c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
16 changed files with 477 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

81
docs/en/api/promql-service.md
View File

@ -268,10 +268,31 @@ GET|POST /api/v1/series
| match[] | series selector | yes | no |
| start | start, format: RFC3399 or unix_timestamp in seconds | yes | no |
| end | end, format: RFC3399 or unix_timestamp in seconds | yes | no |
| limit | integer, maximum number of returned series | yes | yes |
**For metadata metrics**:
**Note: SkyWalking's metadata exists in the following metrics(traffics):**
| Name | Require Labels | Optional Labels | Support Label Match |
|------------------|----------------|--------------------------|-----------------------------------------------------|
| service_traffic | layer | service, limit | =, (only service label support !=, =~, !~) |
| instance_traffic | layer, service | service_instance, limit | =, (only service_instance label support !=, =~, !~) |
| endpoint_traffic | layer, service | endpoint, keyword, limit | =, (only endpoint label support !=, =~, !~) |
- **=**: Label value equals the provided string.
- **!=**: Label value does not equal the provided string.
- **=~**: Label value regex-match the provided string.
- **!~**: Label value does not regex-match the provided string
**If the `limit` is not set by parameter or label, the default value is 100. If the `limit ' is also set in the query parameter, it returns the minimum of the two.**
For example:
```text
/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636
/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636&limit=5
```
or
```text
/api/v1/series?match[]=service_traffic{layer='GENERAL', limit='5'}&start=1677479336&end=1677479636
```
Result:
@ -313,10 +334,23 @@ Result:
}
```
**Note: SkyWalking's metadata exists in the following metrics(traffics):**
- service_traffic
- instance_traffic
- endpoint_traffic
- You can use the `service` label to filter the service_traffic result.
```text
/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs'}&start=1677479336&end=1677479636
```
use regex:
```text
/api/v1/series?match[]=service_traffic{layer='GENERAL', service=~'agent::songs|agent::recommendation'}&start=1677479336&end=1677479636
```
- You can use the `service_instance` label to filter the instance_traffic result.
```text
/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs', service_instance=~'instance1|instance2'}&start=1677479336&end=1677479636
```
- You can use the `endpoint` label to filter the endpoint_traffic result.
```text
/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs', endpoint=~'endpoint1|endpoint2'}&start=1677479336&end=1677479636
```
#### Getting label names
[Prometheus Docs Reference](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names)
@ -330,6 +364,7 @@ GET|POST /api/v1/labels
| match[] | series selector | yes | yes |
| start | start, format: RFC3399 or unix_timestamp in seconds | **no** | yes |
| end | end timestamp, if end time is not present, use current time as default end time | yes | yes |
| limit | integer, maximum number of returned labels, default 100 | yes | yes |
For example:
```text
@ -363,11 +398,12 @@ GET /api/v1/label/<label_name>/values
| match[] | series selector | yes | yes |
| start | start, format: RFC3399 or unix_timestamp in seconds | **no** | yes |
| end | end, format: RFC3399 or unix_timestamp in seconds, if end time is not present, use current time as default end time | yes | yes |
| limit | integer, maximum number of returned label values, default 100 | yes | yes |
For example:
```text
/api/v1/label/__name__/values
```
```
Result:
```json
@ -391,6 +427,39 @@ Result:
}
```
**For metadata metrics**:
| Name | Require Labels | Optional Labels | Support Label Match |
|------------------|----------------|--------------------------|-----------------------------------------------------|
| service_traffic | layer | service, limit | =, (only service label support !=, =~, !~) |
| instance_traffic | layer, service | service_instance, limit | =, (only service_instance label support !=, =~, !~) |
| endpoint_traffic | layer, service | endpoint, keyword, limit | =, (only endpoint label support !=, =~, !~) |
- **=**: Label value equal to the provided string.
- **!=**: Label value not equal to the provided string.
- **=~**: Label value regex-match the provided string.
- **!~**: Label value do not regex-match the provided string
**If the `limit` is not set by parameter or label, the default value is 100. And If the `limit` also set in the query parameter, will return the min number of the two.**
For example:
- If you want to query the label values of the `service` label in the `service_traffic` metric:
```text
/api/v1/label/service/values?match[]=service_traffic{layer='GENERAL', service='agent::songs|agent::recommendation'}&limit=1
```
or
```text
/api/v1/label/service/values?match[]=service_traffic{layer='GENERAL', service='agent::songs|agent::recommendation',limit='1'}
```
- If you want to query the label values of the `service_instance` label in the `instance_traffic` metric:
```text
/api/v1/label/service_instance/values?match[]=instance_traffic{layer='GENERAL', service='agent::songs', service_instance='instance1|instance2'}
```
- If you want to query the label values of the `endpoint` label in the `endpoint_traffic` metric:
```text
/api/v1/label/endpoint/values?match[]=endpoint_traffic{layer='GENERAL', service='agent::songs', endpoint='endpoint1|endpoint2'}
```
#### Querying metric metadata
[Prometheus Docs Reference](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata)

1
docs/en/changes/changes.md
View File

@ -7,6 +7,7 @@
* BanyanDB: Support `hot/warm/cold` stages configuration.
* Fix query continues profiling policies error when the policy is already in the cache.
* Support `hot/warm/cold` stages TTL query in the status API.
* PromQL Service: traffic query support `limit` and regex match.
#### UI

5
oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLLexer.g4
View File

@ -32,6 +32,11 @@ L_BRACE: '{';
R_BRACE: '}';
EQ: '=';
//regex-match
RM: '=~';
//regex-not-match
NRM: '!~';
// Scalar Binary operators
SUB: '-';
ADD: '+';

4
oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLParser.g4
View File

@ -50,10 +50,12 @@ metricRange: metricInstant L_BRACKET DURATION R_BRACKET;
labelName: NAME_STRING;
labelValue: VALUE_STRING;
label: labelName EQ labelValue;
label: labelName matchOp labelValue;
labelList: label (COMMA label)*;
labelNameList: labelName (COMMA labelName)*;
matchOp: EQ | NEQ | RM | NRM;
numberLiteral: NUMBER;
badRange: NUMBER L_BRACKET DURATION R_BRACKET;

2
oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/LabelsQueryRsp.java
View File

@ -26,5 +26,5 @@ import lombok.EqualsAndHashCode;
@EqualsAndHashCode(callSuper = true)
@Data
public class LabelsQueryRsp extends QueryResponse {
private final List<String> data = new ArrayList<>();
private List<String> data = new ArrayList<>();
}

28
oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/QueryFormatRsp.java Normal file
View File

@ -0,0 +1,28 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.query.promql.entity.response;
import lombok.Data;
import lombok.EqualsAndHashCode;
@EqualsAndHashCode(callSuper = true)
@Data
public class QueryFormatRsp extends QueryResponse {
private String data;
}

265
oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/handler/PromQLApiHandler.java
View File

@ -60,12 +60,14 @@ import org.apache.skywalking.oap.query.promql.entity.ResultStatus;
import org.apache.skywalking.oap.query.promql.entity.response.MetricRspData;
import org.apache.skywalking.oap.query.promql.entity.response.MetadataQueryRsp;
import org.apache.skywalking.oap.query.promql.entity.response.MetricType;
import org.apache.skywalking.oap.query.promql.entity.response.QueryFormatRsp;
import org.apache.skywalking.oap.query.promql.entity.response.QueryResponse;
import org.apache.skywalking.oap.query.promql.entity.response.ResultType;
import org.apache.skywalking.oap.query.promql.entity.response.ScalarRspData;
import org.apache.skywalking.oap.query.promql.entity.response.SeriesQueryRsp;
import org.apache.skywalking.oap.query.promql.rt.PromOpUtils;
import org.apache.skywalking.oap.query.promql.rt.PromQLMatchVisitor;
import org.apache.skywalking.oap.query.promql.rt.exception.IllegalExpressionException;
import org.apache.skywalking.oap.query.promql.rt.exception.ParseErrorListener;
import org.apache.skywalking.oap.query.promql.rt.result.MatcherSetResult;
import org.apache.skywalking.oap.query.promql.rt.result.MetricsRangeResult;
@ -188,12 +190,14 @@ public class PromQLApiHandler {
public HttpResponse labels(
@Param("match[]") Optional<String> match,
@Param("start") Optional<String> start,
@Param("end") Optional<String> end) throws IOException {
@Param("end") Optional<String> end,
@Param("limit") Optional<Integer> limit) throws IOException {
LabelsQueryRsp response = new LabelsQueryRsp();
long endTS = System.currentTimeMillis();
if (end.isPresent()) {
endTS = formatTimestamp2Millis(end.get());
}
int limitNum = limit.orElse(100);
Duration duration = getDayDurationFromTimestamp(endTS);
if (match.isPresent()) {
MatcherSetResult parseResult;
@ -219,12 +223,17 @@ public class PromQLApiHandler {
if (metaData.isMultiIntValues()) {
response.getData().remove(DataLabel.GENERAL_LABEL_NAME);
}
} else if (ServiceTraffic.INDEX_NAME.equals(metricName) || InstanceTraffic.INDEX_NAME.equals(metricName)
|| EndpointTraffic.INDEX_NAME.equals(metricName)) {
response.getData().addAll(buildLabelNamesForTraffic(metricName));
}
} else {
Arrays.stream(LabelName.values()).forEach(label -> {
response.getData().add(label.getLabel());
});
}
List<String> result = response.getData().stream().limit(limitNum).collect(Collectors.toList());
response.setData(result);
response.setStatus(ResultStatus.SUCCESS);
return jsonResponse(response);
}
@ -241,7 +250,8 @@ public class PromQLApiHandler {
@Param("label_name") String labelName,
@Param("match[]") Optional<String> match,
@Param("start") Optional<String> start,
@Param("end") Optional<String> end) throws IOException {
@Param("end") Optional<String> end,
@Param("limit") Optional<Integer> limit) throws IOException {
LabelValuesQueryRsp response = new LabelValuesQueryRsp();
response.setStatus(ResultStatus.SUCCESS);
long endTS = System.currentTimeMillis();
@ -249,20 +259,21 @@ public class PromQLApiHandler {
endTS = formatTimestamp2Millis(end.get());
}
Duration duration = getDayDurationFromTimestamp(endTS);
int limitNum = limit.orElse(100);
//general labels
if (LabelName.NAME.getLabel().equals(labelName)) {
getMetricsMetadataQueryService().listMetrics("").forEach(definition -> {
getMetricsMetadataQueryService().listMetrics("").stream().limit(limitNum).forEach(definition -> {
response.getData().add(definition.getName());
});
return jsonResponse(response);
} else if (LabelName.LAYER.getLabel().equals(labelName)) {
for (Layer layer : Layer.values()) {
for (Layer layer : Arrays.stream(Layer.values()).limit(limitNum).collect(Collectors.toList())) {
response.getData().add(layer.name());
}
return jsonResponse(response);
} else if (LabelName.SCOPE.getLabel().equals(labelName)) {
for (Scope scope : Scope.values()) {
for (Scope scope : Arrays.stream(Scope.values()).limit(limitNum).collect(Collectors.toList())) {
response.getData().add(scope.name());
}
return jsonResponse(response);
@ -283,43 +294,35 @@ public class PromQLApiHandler {
metricName);
if (valueColumn.isPresent() && Column.ValueDataType.LABELED_VALUE == valueColumn.get().getDataType()) {
List<MetricsValues> matchedMetrics = getMatcherMetricsValues(parseResult, duration);
response.getData().addAll(buildLabelValuesFromQuery(matchedMetrics, labelName));
response.getData().addAll(buildLabelValuesFromQuery(matchedMetrics, labelName).stream().limit(limitNum).collect(Collectors.toList()));
} else {
// Make compatible with Grafana 11 when use old config variables
// e.g. query service list config: `label_values(service_traffic{layer='$layer'}, service)`
// Grafana 11 will query this API by default rather than `/api/v1/series`(< 11)
if (Objects.equals(metricName, ServiceTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
if (StringUtil.isNotBlank(serviceName)) {
Service service = metadataQuery.findService(serviceName).join();
response.getData().add(service.getName());
} else {
List<Service> services = metadataQuery.listServices(
parseResult.getLabelMap().get(LabelName.LAYER.getLabel())).join();
services.forEach(service -> {
try {
// Make compatible with Grafana 11 when use old config variables
// e.g. query service list config: `label_values(service_traffic{layer='$layer'}, service)`
// Grafana 11 will query this API by default rather than `/api/v1/series`(< 11)
limitNum = getLimitNum(limit, parseResult);
String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel());
if (Objects.equals(metricName, ServiceTraffic.INDEX_NAME)) {
queryServiceTraffic(parseResult, layer, limitNum).forEach(service -> {
response.getData().add(service.getName());
});
} else if (Objects.equals(metricName, InstanceTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
queryInstanceTraffic(parseResult, duration, layer, serviceName, limitNum).forEach(instance -> {
response.getData().add(instance.getName());
});
} else if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
String keyword = parseResult.getLabelMap().getOrDefault(LabelName.KEYWORD.getLabel(), "");
queryEndpointTraffic(parseResult, duration, layer, serviceName, keyword, limitNum).forEach(
endpoint -> {
response.getData().add(endpoint.getName());
});
}
} else if (Objects.equals(metricName, InstanceTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel());
List<ServiceInstance> instances = metadataQuery.listInstances(
duration, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal())).join();
instances.forEach(instance -> {
response.getData().add(instance.getName());
});
} else if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel());
String keyword = parseResult.getLabelMap().getOrDefault(LabelName.KEYWORD.getLabel(), "");
String limit = parseResult.getLabelMap().getOrDefault(LabelName.LIMIT.getLabel(), "100");
List<Endpoint> endpoints = metadataQuery.findEndpoint(
keyword, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal()),
Integer.parseInt(limit), duration
).join();
endpoints.forEach(endpoint -> {
response.getData().add(endpoint.getName());
});
} catch (IllegalExpressionException e) {
response.setStatus(ResultStatus.ERROR);
response.setErrorType(ErrorType.BAD_DATA);
response.setError(e.getMessage());
}
}
}
@ -333,7 +336,8 @@ public class PromQLApiHandler {
public HttpResponse series(
@Param("match[]") String match,
@Param("start") String start,
@Param("end") String end) throws IOException {
@Param("end") String end,
@Param("limit") Optional<Integer> limit) throws IOException {
long startTS = formatTimestamp2Millis(start);
long endTS = formatTimestamp2Millis(end);
Duration duration = DurationUtils.timestamp2Duration(startTS, endTS);
@ -350,43 +354,37 @@ public class PromQLApiHandler {
String metricName = parseResult.getMetricName();
Optional<ValueColumnMetadata.ValueColumn> valueColumn = ValueColumnMetadata.INSTANCE.readValueColumnDefinition(
metricName);
int limitNum = getLimitNum(limit, parseResult);
if (valueColumn.isPresent()) {
ValueColumnMetadata.ValueColumn metaData = valueColumn.get();
Scope scope = Scope.Finder.valueOf(metaData.getScopeId());
Column.ValueDataType dataType = metaData.getDataType();
response.getData().add(buildMetaMetricInfo(metricName, scope, dataType));
} else if (Objects.equals(metricName, ServiceTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
if (StringUtil.isNotBlank(serviceName)) {
Service service = metadataQuery.findService(serviceName).join();
response.getData().add(buildMetricInfoFromTraffic(metricName, service));
} else {
List<Service> services = metadataQuery.listServices(
parseResult.getLabelMap().get(LabelName.LAYER.getLabel())).join();
services.forEach(service -> {
response.getData().add(buildMetricInfoFromTraffic(metricName, service));
});
} else {
try {
String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel());
if (Objects.equals(metricName, ServiceTraffic.INDEX_NAME)) {
queryServiceTraffic(parseResult, layer, limitNum).forEach(service -> {
response.getData().add(buildMetricInfoFromTraffic(metricName, service));
});
} else if (Objects.equals(metricName, InstanceTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
queryInstanceTraffic(parseResult, duration, layer, serviceName, limitNum).forEach(instance -> {
response.getData().add(buildMetricInfoFromTraffic(metricName, instance));
});
} else if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
String keyword = parseResult.getLabelMap().getOrDefault(LabelName.KEYWORD.getLabel(), "");
queryEndpointTraffic(parseResult, duration, layer, serviceName, keyword, limitNum).forEach(
endpoint -> {
response.getData().add(buildMetricInfoFromTraffic(metricName, endpoint));
});
}
} catch (IllegalExpressionException e) {
response.setStatus(ResultStatus.ERROR);
response.setErrorType(ErrorType.BAD_DATA);
response.setError(e.getMessage());
}
} else if (Objects.equals(metricName, InstanceTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel());
List<ServiceInstance> instances = metadataQuery.listInstances(
duration, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal())).join();
instances.forEach(instance -> {
response.getData().add(buildMetricInfoFromTraffic(metricName, instance));
});
} else if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) {
String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel());
String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel());
String keyword = parseResult.getLabelMap().getOrDefault(LabelName.KEYWORD.getLabel(), "");
String limit = parseResult.getLabelMap().getOrDefault(LabelName.LIMIT.getLabel(), "100");
List<Endpoint> endpoints = metadataQuery.findEndpoint(
keyword, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal()),
Integer.parseInt(limit), duration
).join();
endpoints.forEach(endpoint -> {
response.getData().add(buildMetricInfoFromTraffic(metricName, endpoint));
});
}
response.setStatus(ResultStatus.SUCCESS);
@ -467,6 +465,7 @@ public class PromQLApiHandler {
@Param("timeout") Optional<String> timeout) throws IOException {
long startTS = formatTimestamp2Millis(start);
long endTS = formatTimestamp2Millis(end);
//OAP downsample data by min/hour/day step, and generate step query condition automatically by the time range.
Duration duration = DurationUtils.timestamp2Duration(startTS, endTS);
ExprQueryRsp response = new ExprQueryRsp();
PromQLLexer lexer = new PromQLLexer(
@ -509,6 +508,15 @@ public class PromQLApiHandler {
return jsonResponse(response);
}
@Get
@Post
@Path("/api/v1/format_query")
public HttpResponse query_range(@Param("query") String query) throws IOException {
QueryFormatRsp rsp = new QueryFormatRsp();
rsp.setData(query.replaceAll("\\s", ""));
return jsonResponse(rsp);
}
@Get
@Post
@Path("/api/v1/status/buildinfo")
@ -617,6 +625,17 @@ public class PromQLApiHandler {
return labelNames;
}
private List<String> buildLabelNamesForTraffic(String metricName) {
List<String> labelNames = new ArrayList<>();
labelNames.add(LabelName.LAYER.getLabel());
labelNames.add(LabelName.LIMIT.getLabel());
labelNames.add(LabelName.SERVICE.getLabel());
if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) {
labelNames.add(LabelName.KEYWORD.getLabel());
}
return labelNames;
}
private List<String> buildLabelNamesFromQuery(List<MetricsValues> metricsValues) {
Set<String> labelNames = new LinkedHashSet<>();
metricsValues.forEach(metricsValue -> {
@ -727,6 +746,112 @@ public class PromQLApiHandler {
return getMetricsQueryService().readLabeledMetricsValuesWithoutEntity(metricName, matchLabels, duration);
}
private int getLimitNum(Optional<Integer> limitParam, MatcherSetResult parseResult) {
String limitLabel = parseResult.getLabelMap().getOrDefault(LabelName.LIMIT.getLabel(), "100");
int limitNum = Integer.parseInt(limitLabel);
if (limitParam.isPresent()) {
limitNum = Integer.min(limitParam.get(), Integer.parseInt(limitLabel));
}
return limitNum;
}
private List<Service> queryServiceTraffic(MatcherSetResult parseResult, String layer, int limitNum) throws IllegalExpressionException {
if (StringUtil.isBlank(layer)) {
throw new IllegalExpressionException("label {layer} should not be empty.");
}
List<Service> result = new ArrayList<>();
MatcherSetResult.NameMatcher matcher = parseResult.getNameMatcher();
if (matcher != null) {
String serviceName = matcher.getMatchString();
if (matcher.getMatchOp() == PromQLParser.EQ) {
Service service = metadataQuery.findService(serviceName).join();
if (service != null) {
result.add(service);
}
} else if (matcher.getMatchOp() == PromQLParser.NEQ) {
List<Service> services = metadataQuery.listServices(layer).join();
services.stream().filter(s -> !s.getName().equals(serviceName)).limit(limitNum).forEach(result::add);
} else if (matcher.getMatchOp() == PromQLParser.RM) {
List<Service> services = metadataQuery.listServices(layer).join();
services.stream().filter(s -> s.getName().matches(serviceName)).limit(limitNum).forEach(result::add);
} else if (matcher.getMatchOp() == PromQLParser.NRM) {
List<Service> services = metadataQuery.listServices(layer).join();
services.stream().filter(s -> !s.getName().matches(serviceName)).limit(limitNum).forEach(result::add);
}
} else {
List<Service> services = metadataQuery.listServices(
parseResult.getLabelMap().get(LabelName.LAYER.getLabel())).join();
services.stream().limit(limitNum).forEach(result::add);
}
return result;
}
private List<ServiceInstance> queryInstanceTraffic(MatcherSetResult parseResult,
Duration duration,
String layer,
String serviceName,
int limitNum) throws IllegalExpressionException {
if (StringUtil.isBlank(layer)) {
throw new IllegalExpressionException("label {layer} should not be empty.");
}
if (StringUtil.isBlank(serviceName)) {
throw new IllegalExpressionException("label {service} should not be empty.");
}
List<ServiceInstance> result = new ArrayList<>();
MatcherSetResult.NameMatcher matcher = parseResult.getNameMatcher();
List<ServiceInstance> instances = metadataQuery.listInstances(
duration, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal())).join();
if (matcher != null) {
String instanceName = matcher.getMatchString();
if (matcher.getMatchOp() == PromQLParser.EQ) {
instances.stream().filter(n -> n.getName().equals(instanceName)).findFirst().ifPresent(result::add);
} else if (matcher.getMatchOp() == PromQLParser.NEQ) {
instances.stream().filter(n -> !n.getName().equals(instanceName)).limit(limitNum).forEach(result::add);
} else if (matcher.getMatchOp() == PromQLParser.RM) {
instances.stream().filter(n -> n.getName().matches(instanceName)).limit(limitNum).forEach(result::add);
} else if (matcher.getMatchOp() == PromQLParser.NRM) {
instances.stream().filter(n -> !n.getName().matches(instanceName)).limit(limitNum).forEach(result::add);
}
} else {
instances.stream().limit(limitNum).forEach(result::add);
}
return result;
}
private List<Endpoint> queryEndpointTraffic(MatcherSetResult parseResult,
Duration duration,
String layer,
String serviceName,
String keyword,
int limitNum) throws IllegalExpressionException {
if (StringUtil.isBlank(layer)) {
throw new IllegalExpressionException("label {layer} should not be empty.");
}
if (StringUtil.isBlank(serviceName)) {
throw new IllegalExpressionException("label {service} should not be empty.");
}
List<Endpoint> result = new ArrayList<>();
List<Endpoint> endpoints = metadataQuery.findEndpoint(
keyword, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal()), limitNum, duration
).join();
MatcherSetResult.NameMatcher matcher = parseResult.getNameMatcher();
if (matcher != null) {
String endpointName = matcher.getMatchString();
if (matcher.getMatchOp() == PromQLParser.EQ) {
endpoints.stream().filter(e -> e.getName().equals(endpointName)).findFirst().ifPresent(result::add);
} else if (matcher.getMatchOp() == PromQLParser.NEQ) {
endpoints.stream().filter(e -> !e.getName().equals(endpointName)).limit(limitNum).forEach(result::add);
} else if (matcher.getMatchOp() == PromQLParser.RM) {
endpoints.stream().filter(e -> e.getName().matches(endpointName)).limit(limitNum).forEach(result::add);
} else if (matcher.getMatchOp() == PromQLParser.NRM) {
endpoints.stream().filter(e -> !e.getName().matches(endpointName)).limit(limitNum).forEach(result::add);
}
} else {
endpoints.stream().limit(limitNum).forEach(result::add);
}
return result;
}
public enum QueryType {
INSTANT,
RANGE,

16
oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/PromQLMatchVisitor.java
View File

@ -19,8 +19,12 @@
package org.apache.skywalking.oap.query.promql.rt;
import java.util.Map;
import org.apache.skywalking.oap.query.promql.entity.LabelName;
import org.apache.skywalking.oap.query.promql.rt.result.MatcherSetResult;
import org.apache.skywalking.oap.query.promql.rt.result.ParseResultType;
import org.apache.skywalking.oap.server.core.analysis.manual.endpoint.EndpointTraffic;
import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
import org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTraffic;
import org.apache.skywalking.promql.rt.grammar.PromQLParser;
import org.apache.skywalking.promql.rt.grammar.PromQLParserBaseVisitor;
@ -38,7 +42,17 @@ public class PromQLMatchVisitor extends PromQLParserBaseVisitor<MatcherSetResult
String labelName = labelCtx.labelName().getText();
String labelValue = labelCtx.labelValue().getText();
String labelValueTrim = labelValue.substring(1, labelValue.length() - 1);
labelMap.put(labelName, labelValueTrim);
if ((metricName.equals(ServiceTraffic.INDEX_NAME) && LabelName.SERVICE.getLabel().equals(labelName))
|| ((metricName.equals(InstanceTraffic.INDEX_NAME)) && LabelName.SERVICE_INSTANCE.getLabel()
.equals(labelName))
|| ((metricName.equals(EndpointTraffic.INDEX_NAME)) && LabelName.ENDPOINT.getLabel()
.equals(labelName))) {
result.setNameMatcher(
new MatcherSetResult.NameMatcher(
metricName, labelValueTrim, labelCtx.matchOp().getStart().getType()));
} else {
labelMap.put(labelName, labelValueTrim);
}
}
}
return result;

8
oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/result/MatcherSetResult.java
View File

@ -27,5 +27,13 @@ import lombok.EqualsAndHashCode;
@Data
public class MatcherSetResult extends ParseResult {
private String metricName;
private NameMatcher nameMatcher;
private Map<String, String> labelMap = new HashMap<>();
@Data
public static class NameMatcher {
public final String trafficName;
public final String matchString;
public final int matchOp;
}
}

24
test/e2e-v2/cases/promql/docker-compose.yml
View File

@ -20,15 +20,12 @@ services:
extends:
file: ../../script/docker-compose/base-compose.yml
service: oap
environment:
SW_STORAGE: elasticsearch
SW_STORAGE_ES_CLUSTER_NODES: es:9200
volumes:
- ./oal/core.oal:/skywalking/config/oal/core.oal
ports:
- 9090
depends_on:
es:
banyandb:
condition: service_healthy
networks:
- e2e
@ -57,21 +54,12 @@ services:
provider:
condition: service_healthy
es:
image: elastic/elasticsearch:7.10.2
banyandb:
extends:
file: ../../script/docker-compose/base-compose.yml
service: banyandb
ports:
- 9200:9200
networks:
- e2e
environment:
- discovery.type=single-node
- cluster.routing.allocation.disk.threshold_enabled=false
- xpack.security.enabled=false
healthcheck:
test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9200"]
interval: 5s
timeout: 60s
retries: 120
- 17912
networks:

21
test/e2e-v2/cases/promql/expected/endpoint-names.yml Normal file
View File

@ -0,0 +1,21 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{
"status": "success",
"data": [
"POST:/users"
]
}

21
test/e2e-v2/cases/promql/expected/instance-names.yml Normal file
View File

@ -0,0 +1,21 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{
"status": "success",
"data": [
"provider1"
]
}

21
test/e2e-v2/cases/promql/expected/service-names-limit.yml Normal file
View File

@ -0,0 +1,21 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{
"status": "success",
"data": [
"e2e-service-provider"
]
}

22
test/e2e-v2/cases/promql/expected/service-names.yml Normal file
View File

@ -0,0 +1,22 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{
"status": "success",
"data": [
"e2e-service-provider",
"e2e-service-consumer"
]
}

26
test/e2e-v2/cases/promql/expected/service-traffic-limit.yml Normal file
View File

@ -0,0 +1,26 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{
"status": "success",
"data": [
{
"__name__": "service_traffic",
"service": "e2e-service-provider",
"scope": "Service",
"layer": "GENERAL"
}
]
}

31
test/e2e-v2/cases/promql/promql-cases.yaml
View File

@ -17,12 +17,22 @@ cases:
# traffics query
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL"}&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/service-traffic.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL"}&limit=1&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/service-traffic-limit.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL",limit="1"}&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/service-traffic-limit.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL",service=~".*-provider"}&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/service-traffic-limit.yml
#RFC3399
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL"}&start='$(($(date +%s)-1800))'&end='$(date -u +%Y-%m-%dT%H:%M:%S.111Z)
expected: expected/service-traffic.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider"}&start='$(($(date +%s)-1800))'&end='$(date +%s)
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider"}&limit=1&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/instance-traffic.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", keyword="POST:/users"}&start='$(($(date +%s)-1800))'&end='$(date +%s)
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider", service_instance=~"provider1|provider2"}&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/instance-traffic.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", keyword="POST:/users"}&limit=1&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/endpoint-traffic.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", endpoint=~"^POST:.*"}&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/endpoint-traffic.yml
# metrics series
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_cpm&start='$(($(date +%s)-1800))'&end='$(date +%s)
@ -31,6 +41,23 @@ cases:
expected: expected/instance-metric-series.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=endpoint_cpm&start='$(($(date +%s)-1800))'&end='$(date +%s)
expected: expected/endpoint-metric-series.yml
# traffic names query
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service/values -d 'match[]=service_traffic{layer="GENERAL"}&start='$(($(date +%s)-1800))'&end='$(date -u +%Y-%m-%dT%H:%M:%S.111Z)
expected: expected/service-names.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service/values -d 'match[]=service_traffic{layer="GENERAL",limit="1"}&start='$(($(date +%s)-1800))'&end='$(date -u +%Y-%m-%dT%H:%M:%S.111Z)
expected: expected/service-names-limit.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service/values -d 'match[]=service_traffic{layer="GENERAL"}&limit=1'
expected: expected/service-names-limit.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service/values -d 'match[]=service_traffic{layer="GENERAL",service=~".*-provider"}'
expected: expected/service-names-limit.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service_instance/values -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider"}&limit=1'
expected: expected/instance-names.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service_instance/values -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider", service_instance=~"provider1|provider2"}&limit=1'
expected: expected/instance-names.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/endpoint/values -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", keyword="POST:/users"}&limit=1'
expected: expected/endpoint-names.yml
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/endpoint/values -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", endpoint=~"^POST:.*"}'
expected: expected/endpoint-names.yml
# metrics names query
- query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/__name__/values
expected: expected/metrics-names.yml