kazusa
/
skywalking-java
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

support oal list includes and excludes & add tags to some source (#5739) 

* add oal excludes and includes to support tags
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
This commit is contained in:
Evan 2020-11-01 09:48:48 +08:00 committed by GitHub
parent 9668672205
commit 9b6386f31c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 222 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
docs/en/concepts-and-designs/oal.md
View File

@ -34,7 +34,7 @@ Read [Scope Definitions](scope-definitions.md), you can find all existing Scopes
Use filter to build the conditions for the value of fields, by using field name and expression.
The expressions support to link by `and`, `or` and `(...)`.
The OPs support `==`, `!=`, `>`, `<`, `>=`, `<=`, `in [...]` ,`like %...`, `like ...%` and `like %...%`, with type detection based of field type. Trigger compile
The OPs support `==`, `!=`, `>`, `<`, `>=`, `<=`, `in [...]` ,`like %...`, `like ...%` , `like %...%` , `contain` and `not contain`, with type detection based of field type. Trigger compile
or code generation error if incompatible.
## Aggregation Function
@ -105,33 +105,39 @@ In default, no one is being disable.
## Examples
```
// Caculate p99 of both Endpoint1 and Endpoint2
// Calculate p99 of both Endpoint1 and Endpoint2
endpoint_p99 = from(Endpoint.latency).filter(name in ("Endpoint1", "Endpoint2")).summary(0.99)
// Caculate p99 of Endpoint name started with `serv`
// Calculate p99 of Endpoint name started with `serv`
serv_Endpoint_p99 = from(Endpoint.latency).filter(name like "serv%").summary(0.99)
// Caculate the avg response time of each Endpoint
// Calculate the avg response time of each Endpoint
endpoint_avg = from(Endpoint.latency).avg()
// Caculate the p50, p75, p90, p95 and p99 of each Endpoint by 50 ms steps.
// Calculate the p50, p75, p90, p95 and p99 of each Endpoint by 50 ms steps.
endpoint_percentile = from(Endpoint.latency).percentile(10)
// Caculate the percent of response status is true, for each service.
// Calculate the percent of response status is true, for each service.
endpoint_success = from(Endpoint.*).filter(status == true).percent()
// Caculate the sum of response code in [404, 500, 503], for each service.
// Calculate the sum of response code in [404, 500, 503], for each service.
endpoint_abnormal = from(Endpoint.*).filter(responseCode in [404, 500, 503]).sum()
// Caculate the sum of request type in [RequestType.PRC, RequestType.gRPC], for each service.
// Calculate the sum of request type in [RequestType.PRC, RequestType.gRPC], for each service.
endpoint_rpc_calls_sum = from(Endpoint.*).filter(type in [RequestType.PRC, RequestType.gRPC]).sum()
// Caculate the sum of endpoint name in ["/v1", "/v2"], for each service.
// Calculate the sum of endpoint name in ["/v1", "/v2"], for each service.
endpoint_url_sum = from(Endpoint.*).filter(endpointName in ["/v1", "/v2"]).sum()
// Caculate the sum of calls for each service.
// Calculate the sum of calls for each service.
endpoint_calls = from(Endpoint.*).sum()
// Calculate the CPM with the GET method for each service.The value is made up with `tagKey:tagValue`.
service_cpm_http_get = from(Service.*).filter(tags contain "http.method:GET").cpm()
// Calculate the CPM with the HTTP method except for the GET method for each service.The value is made up with `tagKey:tagValue`.
service_cpm_http_other = from(Service.*).filter(tags not contain "http.method:GET").cpm()
disable(segment);
disable(endpoint_relation_server_side);
disable(top_n_database_statement);

4
docs/en/concepts-and-designs/scope-definitions.md
View File

@ -12,6 +12,7 @@ By using Aggregation Function, the requests will group by time and **Group Key(s
| status | Represent whether success or fail of the request. | | bool(true for success) |
| responseCode | Represent the response code of HTTP response, if this request is the HTTP call. e.g. 200, 404, 302| | int |
| type | Represent the type of each request. Such as: Database, HTTP, RPC, gRPC. | | enum |
| tags | Represent the labels of each request and each value is made up with the `TagKey:TagValue` in the segment. | | `List<String>` |
### SCOPE `Service`
@ -27,6 +28,7 @@ Calculate the metrics data from each request of the service.
| status | Represent whether success or fail of the request. | | bool(true for success) |
| responseCode | Represent the response code of HTTP response, if this request is the HTTP call | | int|
| type | Represent the type of each request. Such as: Database, HTTP, RPC, gRPC. | | enum |
| tags | Represent the labels of each request and each value is made up with the `TagKey:TagValue` in the segment. | | `List<String>` |
### SCOPE `ServiceInstance`
@ -42,6 +44,7 @@ Calculate the metrics data from each request of the service instance.
| status | Represent whether success or fail of the request. | | bool(true for success) |
| responseCode | Represent the response code of HTTP response, if this request is the HTTP call. | | int |
| type | Represent the type of each request. Such as: Database, HTTP, RPC, gRPC. | | enum |
| tags | Represent the labels of each request and each value is made up with the `TagKey:TagValue` in the segment. | | `List<String>` |
#### Secondary scopes of `ServiceInstance`
@ -113,6 +116,7 @@ Calculate the metrics data from each request of the endpoint in the service.
| status | Represent whether success or fail of the request.| | bool(true for success) |
| responseCode | Represent the response code of HTTP response, if this request is the HTTP call. | | int |
| type | Represent the type of each request. Such as: Database, HTTP, RPC, gRPC. | | enum |
| tags | Represent the labels of each request and each value is made up with the `TagKey:TagValue` in the segment. | | `List<String>` |
### SCOPE `ServiceRelation`

1
oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/MultiScopesAnalysisListener.java
View File

@ -245,6 +245,7 @@ public class MultiScopesAnalysisListener implements EntryAnalysisListener, ExitA
log.warn("span {} has illegal status code {}", span, tag.getValue());
}
}
sourceBuilder.setTag(tag);
});
sourceBuilder.setStatus(!span.getIsError());

13
oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SourceBuilder.java
View File

@ -18,9 +18,12 @@
package org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener;
import java.util.ArrayList;
import java.util.List;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import org.apache.skywalking.apm.network.common.v3.KeyStringValuePair;
import org.apache.skywalking.apm.util.StringUtil;
import org.apache.skywalking.oap.server.core.analysis.NodeType;
import org.apache.skywalking.oap.server.core.config.NamingControl;
@ -120,6 +123,8 @@ class SourceBuilder {
@Getter
@Setter
private long timeBucket;
@Getter
private final List<String> tags = new ArrayList<>();
/**
* The global level metrics source
@ -134,6 +139,7 @@ class SourceBuilder {
all.setResponseCode(responseCode);
all.setType(type);
all.setTimeBucket(timeBucket);
all.setTags(tags);
return all;
}
@ -150,6 +156,7 @@ class SourceBuilder {
service.setStatus(status);
service.setResponseCode(responseCode);
service.setType(type);
service.setTags(tags);
service.setTimeBucket(timeBucket);
return service;
}
@ -190,6 +197,7 @@ class SourceBuilder {
serviceInstance.setStatus(status);
serviceInstance.setResponseCode(responseCode);
serviceInstance.setType(type);
serviceInstance.setTags(tags);
serviceInstance.setTimeBucket(timeBucket);
return serviceInstance;
}
@ -232,6 +240,7 @@ class SourceBuilder {
endpoint.setStatus(status);
endpoint.setResponseCode(responseCode);
endpoint.setType(type);
endpoint.setTags(tags);
endpoint.setTimeBucket(timeBucket);
return endpoint;
}
@ -294,4 +303,8 @@ class SourceBuilder {
databaseAccess.setTimeBucket(timeBucket);
return databaseAccess;
}
public void setTag(KeyStringValuePair tag) {
tags.add(tag.getKey().trim() + ":" + tag.getValue().trim());
}
}

2
oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4
View File

@ -90,6 +90,8 @@ LESS_EQUAL: '<=';
NOT_EQUAL: '!=';
LIKE: 'like';
IN: 'in';
CONTAIN: 'contain';
NOT_CONTAIN: 'not contain';
// Literals

10
oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4
View File

@ -90,7 +90,15 @@ literalExpression
;
expression
: booleanMatch | stringMatch | greaterMatch | lessMatch | greaterEqualMatch | lessEqualMatch | notEqualMatch | booleanNotEqualMatch | likeMatch | inMatch
: booleanMatch | stringMatch | greaterMatch | lessMatch | greaterEqualMatch | lessEqualMatch | notEqualMatch | booleanNotEqualMatch | likeMatch | inMatch | containMatch | notContainMatch
;
containMatch
: conditionAttribute CONTAIN stringConditionValue
;
notContainMatch
: conditionAttribute NOT_CONTAIN stringConditionValue
;
booleanMatch

10
oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java
View File

@ -153,6 +153,16 @@ public class OALListener extends OALParserBaseListener {
conditionExpression.setExpressionType("likeMatch");
}
@Override
public void enterContainMatch(final OALParser.ContainMatchContext ctx) {
conditionExpression.setExpressionType("containMatch");
}
@Override
public void enterNotContainMatch(final OALParser.NotContainMatchContext ctx) {
conditionExpression.setExpressionType("notContainMatch");
}
@Override
public void enterInMatch(final OALParser.InMatchContext ctx) {
conditionExpression.setExpressionType("inMatch");

36
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/expression/ContainMatch.java Normal file
View File

@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.analysis.metrics.expression;
import java.util.List;
import java.util.Objects;
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.FilterMatcher;
@FilterMatcher
public class ContainMatch {
public boolean match(List<String> left, String right) {
if (Objects.isNull(left)) {
return false;
}
if (right.startsWith("\"") && right.endsWith("\"")) {
right = right.substring(1, right.length() - 1);
}
return left.contains(right);
}
}

36
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/expression/NotContainMatch.java Normal file
View File

@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.analysis.metrics.expression;
import java.util.List;
import java.util.Objects;
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.FilterMatcher;
@FilterMatcher
public class NotContainMatch {
public boolean match(List<String> left, String right) {
if (Objects.isNull(left)) {
return false;
}
if (right.startsWith("\"") && right.endsWith("\"")) {
right = right.substring(1, right.length() - 1);
}
return !left.contains(right);
}
}

4
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/All.java
View File

@ -18,6 +18,7 @@
package org.apache.skywalking.oap.server.core.source;
import java.util.List;
import lombok.Getter;
import lombok.Setter;
@ -56,4 +57,7 @@ public class All extends Source {
@Getter
@Setter
private RequestType type;
@Getter
@Setter
private List<String> tags;
}

4
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/Endpoint.java
View File

@ -18,6 +18,7 @@
package org.apache.skywalking.oap.server.core.source;
import java.util.List;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
@ -75,6 +76,9 @@ public class Endpoint extends Source {
@Getter
@Setter
private RequestType type;
@Getter
@Setter
private List<String> tags;
@Override
public void prepare() {

5
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/Service.java
View File

@ -18,6 +18,7 @@
package org.apache.skywalking.oap.server.core.source;
import java.util.List;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.analysis.IDManager;
@ -64,4 +65,8 @@ public class Service extends Source {
@Getter
@Setter
private RequestType type;
@Getter
@Setter
private List<String> tags;
}

4
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ServiceInstance.java
View File

@ -18,6 +18,7 @@
package org.apache.skywalking.oap.server.core.source;
import java.util.List;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.analysis.IDManager;
@ -67,6 +68,9 @@ public class ServiceInstance extends Source {
@Getter
@Setter
private RequestType type;
@Getter
@Setter
private List<String> tags;
@Override
public void prepare() {

34
oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/metrics/expression/ContainMatchTest.java Normal file
View File

@ -0,0 +1,34 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.analysis.metrics.expression;
import java.util.Arrays;
import org.junit.Assert;
import org.junit.Test;
public class ContainMatchTest {
@Test
public void match() {
ContainMatch containMatch = new ContainMatch();
Assert.assertFalse(containMatch.match(null, "http.method:GET"));
Assert.assertTrue(containMatch.match(Arrays.asList("http.method:GET", "http.method:POST"), "http.method:GET"));
Assert.assertFalse(
containMatch.match(Arrays.asList("http.method:GET", "http.method:POST"), "http.method:PUT"));
}
}

35
oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/metrics/expression/NotContainMatchTest.java Normal file
View File

@ -0,0 +1,35 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.analysis.metrics.expression;
import java.util.Arrays;
import org.junit.Assert;
import org.junit.Test;
public class NotContainMatchTest {
@Test
public void match() {
NotContainMatch notContainMatch = new NotContainMatch();
Assert.assertFalse(notContainMatch.match(null, "http.method:GET"));
Assert.assertFalse(
notContainMatch.match(Arrays.asList("http.method:GET", "http.method:POST"), "http.method:GET"));
Assert.assertTrue(notContainMatch.match(Arrays.asList("http.method:GET", "http.method:POST"), "http.method:PUT"));
}
}

9
oap-server/server-receiver-plugin/skywalking-trace-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/trace/provider/parser/listener/MultiScopesAnalysisListenerTest.java
View File

@ -190,6 +190,10 @@ public class MultiScopesAnalysisListenerTest {
.setIsError(true)
.setSpanType(SpanType.Entry)
.setSpanLayer(SpanLayer.RPCFramework)
.addTags(KeyStringValuePair.newBuilder()
.setKey("http.method")
.setValue("GET")
.build())
.addRefs(
SegmentReference.newBuilder()
.setRefType(RefType.CrossProcess)
@ -226,6 +230,11 @@ public class MultiScopesAnalysisListenerTest {
Assert.assertEquals(serviceInstance.getName(), serviceInstanceRelation.getDestServiceInstanceName());
Assert.assertEquals("downstream-endpoint", endpointRelation.getEndpoint());
Assert.assertEquals(endpoint.getName(), endpointRelation.getChildEndpoint());
// tags test
Assert.assertEquals("http.method:GET", all.getTags().get(0));
Assert.assertEquals("http.method:GET", service.getTags().get(0));
Assert.assertEquals("http.method:GET", serviceInstance.getTags().get(0));
Assert.assertEquals("http.method:GET", endpoint.getTags().get(0));
}
/**