Add `view_as_seq` function in MQE for listing metrics in the given prioritized sequence (#11194)

This commit is contained in:
mrproliu 2023-08-09 11:59:49 +08:00 committed by GitHub
parent 64b24f7ddb
commit 12744bba02
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
12 changed files with 242 additions and 42 deletions

View File

@ -48,7 +48,7 @@ If we want to rename the label values to `P50,P75,P90,P95,P99`, see [Relabel Ope
The `ExpressionResultType` of the expression is `TIME_SERIES_VALUES` and with labels.
## Binary Operation
Binary Operation is an operation that takes two expressions and performs a calculation on their results.
The binary Operation is an operation that takes two expressions and performs a calculation on their results.
The following table lists the binary operations supported by MQE.
Expression:
@ -71,11 +71,11 @@ service_sla / 100
```
### Result Type
The result type of the expression please refer to the following table.
For the result type of the expression, please refer to the following table.
### Binary Operation Rules
The following table listed if the difference result types of the input expressions could do this operation and the result type after the operation.
The expression could on the left or right side of the operator.
The following table lists if the different result types of the input expressions could do this operation and the result type after the operation.
The expression could be on the left or right side of the operator.
**Note**: If the expressions on both sides of the operator are the `TIME_SERIES_VALUES with labels`, they should have the same labels for calculation.
| Expression | Expression | Yes/No | ExpressionResultType |
@ -88,7 +88,7 @@ The expression could on the left or right side of the operator.
| SORTED_LIST/RECORD_LIST | SORTED_LIST/RECORD_LIST | no | |
## Aggregation Operation
Aggregation Operation takes an expression and performs aggregate calculation on its results.
Aggregation Operation takes an expression and performs aggregate calculations on its results.
Expression:
```text
@ -112,14 +112,14 @@ avg(service_cpm)
```
### Result Type
The different operator could impact the `ExpressionResultType`, please refer to the above table.
The different operators could impact the `ExpressionResultType`, please refer to the above table.
## Function Operation
Function Operation takes an expression and performs function calculation on its results.
## Mathematical Operation
Mathematical Operation takes an expression and performs mathematical calculations on its results.
Expression:
```text
<Function-Operator>(Expression, parameters)
<Mathematical-Operator>(Expression, parameters)
```
| Operator | Definition | parameters | ExpressionResultType |
@ -138,7 +138,7 @@ round(service_cpm / 60 , 2)
```
### Result Type
The different operator could impact the `ExpressionResultType`, please refer to the above table.
The different operators could impact the `ExpressionResultType`, please refer to the above table.
## TopN Operation
TopN Operation takes an expression and performs TopN calculation on its results.
@ -163,14 +163,14 @@ top_n(service_instance_cpm, 10, des)
According to the type of the metric, the `ExpressionResultType` of the expression will be `SORTED_LIST` or `RECORD_LIST`.
## Relabel Operation
Relabel Operation takes an expression and replace the label values to new label values on its results.
Relabel Operation takes an expression and replaces the label values with new label values on its results.
Expression:
```text
relabel(Expression, _='<new_label_value_1>,...')
```
`_` is the new label valuess of the metric after the label is relabeled, the order of the new label values should be the same as the order of the label values in the input expression result.
`_` is the new label of the metric after the label is relabeled, the order of the new label values should be the same as the order of the label values in the input expression result.
For example:
If we want to query the `service_percentile` metric with the label values `0,1,2,3,4`, and rename the label values to `P50,P75,P90,P95,P99`, we can use the following expression:
@ -183,7 +183,7 @@ relabel(service_percentile{_='0,1,2,3,4'}, _='P50,P75,P90,P95,P99')
Follow the input expression.
## AggregateLabels Operation
AggregateLabels Operation takes an expression and performs aggregate calculation on its `Labeled Value Metrics` results. It aggregates a group of `TIME_SERIES_VALUES` into a single `TIME_SERIES_VALUES`.
AggregateLabels Operation takes an expression and performs an aggregate calculation on its `Labeled Value Metrics` results. It aggregates a group of `TIME_SERIES_VALUES` into a single `TIME_SERIES_VALUES`.
Expression:
```text
@ -194,11 +194,11 @@ aggregate_labels(Expression, parameter)
|-----------|-----------------------------------------------------|----------------------|
| avg | calculate avg value of a `Labeled Value Metrics` | TIME_SERIES_VALUES |
| sum | calculate sum value of a `Labeled Value Metrics` | TIME_SERIES_VALUES |
| max | select maximum value from a `Labeled Value Metrics` | TIME_SERIES_VALUES |
| min | select minimum value from a `Labeled Value Metrics` | TIME_SERIES_VALUES |
| max | select the maximum value from a `Labeled Value Metrics` | TIME_SERIES_VALUES |
| min | select the minimum value from a `Labeled Value Metrics` | TIME_SERIES_VALUES |
For example:
If we want to query all redis command total rate, we can use the following expression(`total_commands_rate` is a metric which recorded every command rate in labeled value):
If we want to query all Redis command total rates, we can use the following expression(`total_commands_rate` is a metric which recorded every command rate in labeled value):
```text
aggregate_labels(total_commands_rate, SUM)
@ -206,3 +206,23 @@ aggregate_labels(total_commands_rate, SUM)
### Result Type
The ExpressionResultType of the aggregateLabels operation is TIME_SERIES_VALUES.
## Logical Operation
### ViewAsSequence Operation
ViewAsSequence operation represents the first not-null metric from the listing metrics in the given prioritized sequence(left to right). It could also be considered as a `short-circuit` of given metrics for the first value existing metric.
Expression:
```text
view_as_seq([<expression_1>, <expression_2>, ...])
```
For example:
if the first expression value is empty but the second one is not empty, it would return the result from the second expression.
The following example would return the content of the **service_cpm** metric.
```text
view_as_seq(not_existing, service_cpm)
```
#### Result Type
The result type is determined by the type of selected not-null metric expression.

View File

@ -67,7 +67,7 @@
* Add OpenSearch 2.8.0 to our client lib tests.
* Apply MQE on RabbitMQ Dashboards.
* Use listening mode for apollo implementation of dynamic configuration.
* Add `view_as_seq` function in MQE for listing metrics in the given prioritized sequence.
#### UI

View File

@ -63,6 +63,9 @@ ROUND: 'round';
// TopN
TOP_N: 'top_n';
// ViewAsSeq
VIEW_AS_SEQ: 'view_as_seq';
// Relabels
RELABELS: 'relabels';

View File

@ -28,13 +28,18 @@ expression
| expression addSub expression # addSubOp
| expression compare expression # compareOp
| aggregation L_PAREN expression R_PAREN # aggregationOp
| function0 L_PAREN expression R_PAREN #function0OP
| function1 L_PAREN expression COMMA parameter R_PAREN #function1OP
| mathematical_operator0 L_PAREN expression R_PAREN #mathematicalOperator0OP
| mathematical_operator1 L_PAREN expression COMMA parameter R_PAREN #mathematicalOperator1OP
| logical_operator L_PAREN expressionList R_PAREN #logicalOperatorOP
| topN L_PAREN metric COMMA parameter COMMA order R_PAREN #topNOP
| relabels L_PAREN expression COMMA label R_PAREN #relablesOP
| aggregateLabels L_PAREN expression COMMA aggregateLabelsFunc R_PAREN #aggregateLabelsOp
;
expressionList
: expression (COMMA expression)*
;
expressionNode: metric| scalar;
addSub: ADD | SUB ;
@ -57,14 +62,17 @@ aggregation:
AVG | COUNT | LATEST | SUM | MAX | MIN | ;
// 0 parameter function
function0:
mathematical_operator0:
ABS | CEIL | FLOOR;
// 1 parameter function
function1:
mathematical_operator1:
ROUND;
topN: TOP_N;
logical_operator:
VIEW_AS_SEQ;
relabels: RELABELS;
parameter: INTEGER;

View File

@ -27,7 +27,8 @@ import org.apache.skywalking.mqe.rt.grammar.MQEParserBaseVisitor;
import org.apache.skywalking.mqe.rt.operation.AggregateLabelsOp;
import org.apache.skywalking.mqe.rt.operation.AggregationOp;
import org.apache.skywalking.mqe.rt.operation.BinaryOp;
import org.apache.skywalking.mqe.rt.operation.FunctionOp;
import org.apache.skywalking.mqe.rt.operation.LogicalFunctionOp;
import org.apache.skywalking.mqe.rt.operation.MathematicalFunctionOp;
import org.apache.skywalking.mqe.rt.type.ExpressionResult;
import org.apache.skywalking.mqe.rt.exception.IllegalExpressionException;
import org.apache.skywalking.mqe.rt.type.ExpressionResultType;
@ -136,14 +137,14 @@ public abstract class MQEVisitorBase extends MQEParserBaseVisitor<ExpressionResu
}
@Override
public ExpressionResult visitFunction0OP(MQEParser.Function0OPContext ctx) {
int opType = ctx.function0().getStart().getType();
public ExpressionResult visitMathematicalOperator0OP(MQEParser.MathematicalOperator0OPContext ctx) {
int opType = ctx.mathematical_operator0().getStart().getType();
ExpressionResult expResult = visit(ctx.expression());
if (StringUtil.isNotEmpty(expResult.getError())) {
return expResult;
}
try {
return FunctionOp.doFunction0Op(expResult, opType);
return MathematicalFunctionOp.doFunction0Op(expResult, opType);
} catch (IllegalExpressionException e) {
ExpressionResult result = new ExpressionResult();
result.setType(ExpressionResultType.UNKNOWN);
@ -153,14 +154,14 @@ public abstract class MQEVisitorBase extends MQEParserBaseVisitor<ExpressionResu
}
@Override
public ExpressionResult visitFunction1OP(MQEParser.Function1OPContext ctx) {
int opType = ctx.function1().getStart().getType();
public ExpressionResult visitMathematicalOperator1OP(MQEParser.MathematicalOperator1OPContext ctx) {
int opType = ctx.mathematical_operator1().getStart().getType();
ExpressionResult expResult = visit(ctx.expression());
if (StringUtil.isNotEmpty(expResult.getError())) {
return expResult;
}
try {
return FunctionOp.doFunction1Op(expResult, opType, Integer.parseInt(ctx.parameter().INTEGER().getText()));
return MathematicalFunctionOp.doFunction1Op(expResult, opType, Integer.parseInt(ctx.parameter().INTEGER().getText()));
} catch (IllegalExpressionException e) {
ExpressionResult result = new ExpressionResult();
result.setType(ExpressionResultType.UNKNOWN);
@ -204,6 +205,19 @@ public abstract class MQEVisitorBase extends MQEParserBaseVisitor<ExpressionResu
return result;
}
@Override
public ExpressionResult visitLogicalOperatorOP(MQEParser.LogicalOperatorOPContext ctx) {
int opType = ctx.logical_operator().getStart().getType();
try {
return LogicalFunctionOp.doOP(opType, ctx.expressionList(), this);
} catch (IllegalExpressionException e) {
ExpressionResult result = new ExpressionResult();
result.setType(ExpressionResultType.UNKNOWN);
result.setError(e.getMessage());
return result;
}
}
@Override
public abstract ExpressionResult visitMetric(MQEParser.MetricContext ctx);
}

View File

@ -0,0 +1,60 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.mqe.rt.operation;
import org.apache.skywalking.mqe.rt.exception.IllegalExpressionException;
import org.apache.skywalking.mqe.rt.grammar.MQEParser;
import org.apache.skywalking.mqe.rt.grammar.MQEParserBaseVisitor;
import org.apache.skywalking.mqe.rt.type.ExpressionResult;
import org.apache.skywalking.oap.server.library.util.CollectionUtils;
public class LogicalFunctionOp {
public static ExpressionResult doOP(int opType, MQEParser.ExpressionListContext expressionListContext,
MQEParserBaseVisitor<ExpressionResult> visitor) throws IllegalExpressionException {
switch (opType) {
case MQEParser.VIEW_AS_SEQ:
return viewAsSeq(expressionListContext, visitor);
}
throw new IllegalExpressionException("Unsupported function.");
}
private static ExpressionResult viewAsSeq(MQEParser.ExpressionListContext expressionListContext,
MQEParserBaseVisitor<ExpressionResult> visitor) {
ExpressionResult firstResult = null;
for (MQEParser.ExpressionContext expContext : expressionListContext.expression()) {
final ExpressionResult result = visitor.visit(expContext);
if (firstResult == null) {
firstResult = result;
}
if (result == null || CollectionUtils.isEmpty(result.getResults())) {
continue;
}
final boolean isNotEmptyValue = result.getResults().stream()
.filter(s -> s != null && CollectionUtils.isNotEmpty(s.getValues()))
.flatMap(s -> s.getValues().stream()).anyMatch(s -> !s.isEmptyValue());
if (isNotEmptyValue) {
return result;
}
}
return firstResult;
}
}

View File

@ -25,16 +25,16 @@ import org.apache.skywalking.mqe.rt.exception.IllegalExpressionException;
import org.apache.skywalking.mqe.rt.grammar.MQEParser;
import org.apache.skywalking.mqe.rt.type.ExpressionResult;
public class FunctionOp {
public class MathematicalFunctionOp {
public static ExpressionResult doFunction0Op(ExpressionResult expResult,
int opType) throws IllegalExpressionException {
switch (opType) {
case MQEParser.ABS:
return FunctionOp.transResult(expResult, Math::abs);
return MathematicalFunctionOp.transResult(expResult, Math::abs);
case MQEParser.CEIL:
return FunctionOp.transResult(expResult, Math::ceil);
return MathematicalFunctionOp.transResult(expResult, Math::ceil);
case MQEParser.FLOOR:
return FunctionOp.transResult(expResult, Math::floor);
return MathematicalFunctionOp.transResult(expResult, Math::floor);
}
throw new IllegalExpressionException("Unsupported function.");
@ -45,7 +45,7 @@ public class FunctionOp {
int scale) throws IllegalExpressionException {
switch (opType) {
case MQEParser.ROUND:
return FunctionOp.transResult(expResult, aDouble -> {
return MathematicalFunctionOp.transResult(expResult, aDouble -> {
BigDecimal bd = BigDecimal.valueOf(aDouble);
return bd.setScale(scale, RoundingMode.HALF_UP).doubleValue();
});

View File

@ -0,0 +1,59 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.library.mqe.rt;
import org.apache.skywalking.mqe.rt.grammar.MQEParser;
import org.apache.skywalking.mqe.rt.grammar.MQEParserBaseVisitor;
import org.apache.skywalking.mqe.rt.operation.LogicalFunctionOp;
import org.apache.skywalking.mqe.rt.type.ExpressionResult;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
import java.util.Arrays;
import static org.junit.jupiter.api.Assertions.assertEquals;
@ExtendWith(MockitoExtension.class)
public class LogicalFunctionOpTest {
private final MockData mockData = new MockData();
@Test
public void viewAsSeqTest() throws Exception {
MQEParser.ExpressionListContext expressions = Mockito.mock(MQEParser.ExpressionListContext.class);
MQEParser.ExpressionContext emptyExpression = Mockito.mock(MQEParser.ExpressionContext.class);
MQEParser.ExpressionContext notEmptyExpression = Mockito.mock(MQEParser.ExpressionContext.class);
Mockito.when(expressions.expression()).thenReturn(Arrays.asList(emptyExpression, notEmptyExpression));
final MQEParserBaseVisitor<ExpressionResult> visitor = Mockito.mock(MQEParserBaseVisitor.class);
final ExpressionResult emptyResult = mockData.newSeriesNoLabeledResult(0, 0);
Mockito.when(visitor.visit(emptyExpression)).thenReturn(emptyResult);
final ExpressionResult notEmptyResult = mockData.newSeriesNoLabeledResult(100, 200);
Mockito.when(visitor.visit(notEmptyExpression)).thenReturn(notEmptyResult);
ExpressionResult result;
result = LogicalFunctionOp.doOP(MQEParser.VIEW_AS_SEQ, expressions, visitor);
assertEquals(notEmptyResult, result);
Mockito.when(expressions.expression()).thenReturn(Arrays.asList(emptyExpression, emptyExpression));
result = LogicalFunctionOp.doOP(MQEParser.VIEW_AS_SEQ, expressions, visitor);
assertEquals(emptyResult, result);
}
}

View File

@ -19,20 +19,20 @@
package org.apache.skywalking.library.mqe.rt;
import org.apache.skywalking.mqe.rt.grammar.MQEParser;
import org.apache.skywalking.mqe.rt.operation.FunctionOp;
import org.apache.skywalking.mqe.rt.operation.MathematicalFunctionOp;
import org.apache.skywalking.mqe.rt.type.ExpressionResult;
import org.apache.skywalking.mqe.rt.type.ExpressionResultType;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class FunctionOpTest {
public class MathematicalFunctionOpTest {
private final MockData mockData = new MockData();
//ABS/CEIL/FLOOR/ROUND... are the same logic and tested in here, the others only test ABS is enough.
@Test
public void seriesNoLabeledTest() throws Exception {
ExpressionResult abs = FunctionOp.doFunction0Op(
ExpressionResult abs = MathematicalFunctionOp.doFunction0Op(
mockData.newSeriesNoLabeledResult(-100.111, -300), MQEParser.ABS);
assertEquals(ExpressionResultType.TIME_SERIES_VALUES, abs.getType());
assertEquals("100", abs.getResults().get(0).getValues().get(0).getId());
@ -40,7 +40,7 @@ public class FunctionOpTest {
assertEquals("300", abs.getResults().get(0).getValues().get(1).getId());
assertEquals(300, abs.getResults().get(0).getValues().get(1).getDoubleValue());
ExpressionResult ceil = FunctionOp.doFunction0Op(
ExpressionResult ceil = MathematicalFunctionOp.doFunction0Op(
mockData.newSeriesNoLabeledResult(100.111, 300.2), MQEParser.CEIL);
assertEquals(ExpressionResultType.TIME_SERIES_VALUES, ceil.getType());
assertEquals("100", ceil.getResults().get(0).getValues().get(0).getId());
@ -48,7 +48,7 @@ public class FunctionOpTest {
assertEquals("300", ceil.getResults().get(0).getValues().get(1).getId());
assertEquals(301, ceil.getResults().get(0).getValues().get(1).getDoubleValue());
ExpressionResult floor = FunctionOp.doFunction0Op(
ExpressionResult floor = MathematicalFunctionOp.doFunction0Op(
mockData.newSeriesNoLabeledResult(100.111, 300.2), MQEParser.FLOOR);
assertEquals(ExpressionResultType.TIME_SERIES_VALUES, ceil.getType());
assertEquals("100", floor.getResults().get(0).getValues().get(0).getId());
@ -57,7 +57,7 @@ public class FunctionOpTest {
assertEquals(300, floor.getResults().get(0).getValues().get(1).getDoubleValue());
MQEParser.ParameterContext parameterContext = new MQEParser.ParameterContext(null, 0);
ExpressionResult round = FunctionOp.doFunction1Op(
ExpressionResult round = MathematicalFunctionOp.doFunction1Op(
mockData.newSeriesNoLabeledResult(100.111, 300.222), MQEParser.ROUND, 2);
assertEquals(ExpressionResultType.TIME_SERIES_VALUES, ceil.getType());
assertEquals("100", round.getResults().get(0).getValues().get(0).getId());
@ -68,7 +68,7 @@ public class FunctionOpTest {
@Test
public void seriesLabeledTest() throws Exception {
ExpressionResult abs = FunctionOp.doFunction0Op(
ExpressionResult abs = MathematicalFunctionOp.doFunction0Op(
mockData.newSeriesLabeledResult(-100.111, -300, -101.333, -301.666), MQEParser.ABS);
assertEquals(ExpressionResultType.TIME_SERIES_VALUES, abs.getType());
//label=1
@ -87,7 +87,7 @@ public class FunctionOpTest {
@Test
public void listTest() throws Exception {
ExpressionResult abs = FunctionOp.doFunction0Op(mockData.newListResult(-100.111, -300), MQEParser.ABS);
ExpressionResult abs = MathematicalFunctionOp.doFunction0Op(mockData.newListResult(-100.111, -300), MQEParser.ABS);
assertEquals(ExpressionResultType.SORTED_LIST, abs.getType());
assertEquals("service_A", abs.getResults().get(0).getValues().get(0).getId());
assertEquals(100.111, abs.getResults().get(0).getValues().get(0).getDoubleValue());

View File

@ -113,6 +113,7 @@ public class MockData {
MQEValue mqeValue = new MQEValue();
mqeValue.setId(id);
mqeValue.setDoubleValue(value);
mqeValue.setEmptyValue(value == 0);
return mqeValue;
}

View File

@ -0,0 +1,31 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
type: TIME_SERIES_VALUES
results:
{{- contains .results }}
- metric:
labels: []
values:
{{- contains .values }}
- id: {{ notEmpty .id }}
value: {{ .value }}
traceid: null
- id: {{ notEmpty .id }}
value: null
traceid: null
{{- end}}
{{- end}}
error: null

View File

@ -58,4 +58,8 @@ cases:
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression="aggregate_labels(service_percentile,max)" --service-name=e2e-service-provider
expected: expected/aggregateLabels-OP.yml
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression="aggregate_labels(service_percentile,min)" --service-name=e2e-service-provider
expected: expected/aggregateLabels-OP.yml
expected: expected/aggregateLabels-OP.yml
# viewAsSeq-OP
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression="view_as_seq(mq_service_consume_sla,service_sla)" --service-name=e2e-service-provider
expected: expected/viewAsSeq-OP.yml