kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Support `Get Effective TTL Configurations` API. (#12847) 

* [Breaking Change] Rename `debugging-query` module to `status-query` module. Relative exposed APIs are **UNCHANGED**. 
* Support `Get Effective TTL Configurations` API.
This commit is contained in:
吴晟 Wu Sheng 2024-12-12 09:51:47 +08:00 committed by GitHub
parent 1c9c685db8
commit 841a92b621
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
45 changed files with 527 additions and 58 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/en/changes/changes.md
View File

@ -6,6 +6,7 @@
* This change would not impact the existing deployment and its feature for our official release users.
* **Warning** If there are custom query plugins for our Elasticsearch indices, this change could break them as
sort queries and aggregation queries which used the unexpected fields are being blocked.
* [Breaking Change] Rename `debugging-query` module to `status-query` module. Relative exposed APIs are **UNCHANGED**.
#### OAP Server
@ -34,6 +35,7 @@
* BanyanDB: Support update the Schema when OAP starting.
* BanyanDB: Speed up OAP booting while initializing BanyanDB.
* BanyanDB: Support `@EnableSort` on the column to enable sorting for `IndexRule` and set the default to false.
* Support `Get Effective TTL Configurations` API.
#### UI
@ -61,5 +63,6 @@
* Improve BanyanDB documentation.
* Improve component-libraries documentation.
* Improve configuration-vocabulary documentation.
* Add `Get Effective TTL Configurations` API documentation.
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/224?closed=1)

13
docs/en/debugging/config_dump.md
View File

@ -1,14 +1,15 @@
# Scratch The OAP Config Dump
# Dump Effective Initial Configurations
SkyWalking OAP behaviors could be controlled through hundreds of configurations. It is hard to know what is the final
configuration as all the configurations could be overrided by system environments.
configuration as all the configurations could be override by system environments.
The core config file [application.yml](../../../oap-server/server-starter/src/main/resources/application.yml) lists all
the configurations
The core config file [application.yml](../../../oap-server/server-starter/src/main/resources/application.yml) lists all the configurations
and their default values. However, it is still hard to know the runtime value.
Scratch is a tool to dump the final configuration. It is provided within OAP rest server, which could be accessed
through HTTP GET `http://{core restHost}:{core restPort}/debugging/config/dump`.
Dump Effective Initial Configurations API is designed to help users to understand the effective configurations, no matter
they are initialized in the `application.yml`, or override through system environments.
- URL, `http://{core restHost}:{core restPort}/debugging/config/dump`
- HTTP GET method.
```shell
> curl http://127.0.0.1:12800/debugging/config/dump

2
docs/en/setup/backend/configuration-vocabulary.md
View File

@ -344,7 +344,7 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | kafkaTopicLog | Kafka topic name for log. | SW_EXPORTER_KAFKA_TOPIC_LOG | skywalking-export-log |
| - | - | exportErrorStatusTraceOnly | Export error status trace segments through the Kafka channel. | SW_EXPORTER_KAFKA_TRACE_FILTER_ERROR | false |
| health-checker | default | checkIntervalSeconds | The period of checking OAP internal health status (in seconds). | SW_HEALTH_CHECKER_INTERVAL_SECONDS | 5 |
| debugging-query | default | | | | |
| status-query | default | | | | |
| - | - | keywords4MaskingSecretsOfConfig | Include the list of keywords to filter configurations including secrets. Separate keywords by a comma. | SW_DEBUGGING_QUERY_KEYWORDS_FOR_MASKING_SECRETS | user,password,token,accessKey,secretKey,authentication |
| configuration-discovery | default | disableMessageDigest | If true, agent receives the latest configuration every time, even without making any changes. By default, OAP uses the SHA512 message digest mechanism to detect changes in configuration. | SW_DISABLE_MESSAGE_DIGEST | false |
| receiver-event | default | gRPC services that handle events data. | - | - | |

11
docs/en/setup/backend/ttl.md
View File

@ -1,4 +1,6 @@
# TTL
# Data Lifecycle. Time To Live (TTL)
Time To Live (TTL) is a mechanism to automatically delete data that is older than the specified time.
In SkyWalking, there are two types of observability data:
1. Records include traces, logs, topN sampled statements and alarm. `recordDataTTL` applies to **record** data.
1. Metrics include all metrics for service, instance, endpoint, and topology map. Metadata(lists of services, instances, or endpoints) also belongs to metrics. `metricsDataTTL` applies to **Metrics** data.
@ -12,6 +14,9 @@ These are the settings for the different types:
## BanyanDB TTL
BanyanDB has a TTL mechanism to automatically delete data that is older than the specified time. When you use BanyanDB as the storage backend, `recordDataTTL` and `metricsDataTTL` are not used. Instead, you should configure the TTL settings in `storage.banyandb`.
BanyanDB has a TTL mechanism to automatically delete data that is older than the specified time. When you use BanyanDB
as the storage backend, `recordDataTTL` and `metricsDataTTL` are not used. Instead, you should configure the TTL
settings in `storage.banyandb`.
Please refer to the [Storage BanyanDB](storages/banyandb.md) document for more information.
Please refer to the [Storage BanyanDB](storages/banyandb.md) and [BanyanDB's Progressive TTL](../../banyandb/ttl.md)
documents for more information.

48
docs/en/status/query_ttl_setup.md Normal file
View File

@ -0,0 +1,48 @@
# Get Effective TTL Configurations
Time To Live (TTL) mechanism has different behaviors according to different storage implementations. By default, the
core module provides two TTL configurations: [`recordDataTTL` and `metricsDataTTL`](../setup/backend/ttl.md).
But some storage implementations could override these settings and provide its own TTL configurations, for example,
BanyanDB provides its native TTL mechanism to support [progressive TTL](../banyandb/ttl.md) feature.
This API is used to get the unified and effective TTL configurations.
- URL, `http://{core restHost}:{core restPort}/status/config/ttl`
- HTTP GET method.
```shell
> curl -X GET "http://127.0.0.1:12800/status/config/ttl"
# Metrics TTL includes the definition of the TTL of the metrics-ish data in the storage,
# e.g.
# 1. The metadata of the service, instance, endpoint, topology map, etc.
# 2. Generated metrics data from OAL and MAL engines.
#
# TTLs for each granularity metrics are listed separately.
metrics.minute=7
metrics.hour=7
metrics.day=7
# Records TTL includes the definition of the TTL of the records data in the storage,
# Records include traces, logs, sampled slow SQL statements, HTTP requests(by Rover), alarms, etc.
# Super dataset of records are traces and logs, which volume should be much larger.
records.default=3
records.superDataset=3
```
This API also provides the response in JSON format, which is more friendly for programmatic usage.
```shell
> curl -X GET "http://127.0.0.1:12800/status/config/ttl" \
-H "Accept: application/json"
{
"metrics": {
"minute": 7,
"hour": 7,
"day": 7
},
"records": {
"default": 3,
"superDataset": 3
}
}
```

6
docs/menu.yml
View File

@ -352,12 +352,14 @@ catalog:
path: "/en/concepts-and-designs/metrics-attributes"
- name: "Security Notice"
path: "/en/security/readme"
- name: "Debugging"
- name: "Status APIs"
catalog:
- name: "Scratch The OAP Config Dump"
- name: "Dump Effective Initial Configurations"
path: "/en/debugging/config_dump"
- name: "Tracing Query Execution"
path: "/en/debugging/query-tracing"
- name: "Get Effective TTL Configurations"
path: "/en/status/query_ttl_setup"
- name: "Academy"
catalog:
- name: "Scaling SkyWalking server automatically in kubernetes"

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModule.java
View File

@ -52,6 +52,7 @@ import org.apache.skywalking.oap.server.core.query.MetadataQueryService;
import org.apache.skywalking.oap.server.core.query.MetricsMetadataQueryService;
import org.apache.skywalking.oap.server.core.query.MetricsQueryService;
import org.apache.skywalking.oap.server.core.query.RecordQueryService;
import org.apache.skywalking.oap.server.core.query.TTLStatusQuery;
import org.apache.skywalking.oap.server.core.query.TagAutoCompleteQueryService;
import org.apache.skywalking.oap.server.core.query.TopNRecordsQueryService;
import org.apache.skywalking.oap.server.core.query.TopologyQueryService;
@ -96,7 +97,7 @@ public class CoreModule extends ModuleDefine {
addServerInterface(classes);
addReceiverInterface(classes);
addInsideService(classes);
addInternalServices(classes);
addCacheService(classes);
addQueryService(classes);
addProfileService(classes);
@ -153,6 +154,7 @@ public class CoreModule extends ModuleDefine {
classes.add(TagAutoCompleteQueryService.class);
classes.add(RecordQueryService.class);
classes.add(HierarchyQueryService.class);
classes.add(TTLStatusQuery.class);
}
private void addServerInterface(List<Class> classes) {
@ -160,7 +162,7 @@ public class CoreModule extends ModuleDefine {
classes.add(HTTPHandlerRegister.class);
}
private void addInsideService(List<Class> classes) {
private void addInternalServices(List<Class> classes) {
classes.add(ModelCreator.class);
classes.add(IModelManager.class);
classes.add(ModelManipulator.class);

8
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
View File

@ -77,6 +77,7 @@ import org.apache.skywalking.oap.server.core.query.MetadataQueryService;
import org.apache.skywalking.oap.server.core.query.MetricsMetadataQueryService;
import org.apache.skywalking.oap.server.core.query.MetricsQueryService;
import org.apache.skywalking.oap.server.core.query.RecordQueryService;
import org.apache.skywalking.oap.server.core.query.TTLStatusQuery;
import org.apache.skywalking.oap.server.core.query.TagAutoCompleteQueryService;
import org.apache.skywalking.oap.server.core.query.TopNRecordsQueryService;
import org.apache.skywalking.oap.server.core.query.TopologyQueryService;
@ -304,6 +305,13 @@ public class CoreModuleProvider extends ModuleProvider {
TagAutoCompleteQueryService.class, new TagAutoCompleteQueryService(getManager(), moduleConfig));
this.registerServiceImplementation(RecordQueryService.class, new RecordQueryService(getManager()));
this.registerServiceImplementation(HierarchyQueryService.class, new HierarchyQueryService(getManager(), moduleConfig));
this.registerServiceImplementation(
TTLStatusQuery.class, new TTLStatusQuery(
getManager(),
moduleConfig.getMetricsDataTTL(),
moduleConfig.getRecordDataTTL()
)
);
// add profile service implementations
this.registerServiceImplementation(

60
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/TTLStatusQuery.java Normal file
View File

@ -0,0 +1,60 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.query;
import lombok.RequiredArgsConstructor;
import org.apache.skywalking.oap.server.core.storage.StorageModule;
import org.apache.skywalking.oap.server.core.storage.ttl.MetricsTTL;
import org.apache.skywalking.oap.server.core.storage.ttl.RecordsTTL;
import org.apache.skywalking.oap.server.core.storage.ttl.StorageTTLStatusQuery;
import org.apache.skywalking.oap.server.core.storage.ttl.TTLDefinition;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.library.module.Service;
@RequiredArgsConstructor
public class TTLStatusQuery implements Service {
private final ModuleManager moduleManager;
private final int coreMetricsDataTTL;
private final int coreRecordDataTTL;
private StorageTTLStatusQuery storageTTLStatusQuery;
private StorageTTLStatusQuery getStorageTTLStatusQuery() {
if (storageTTLStatusQuery == null) {
storageTTLStatusQuery = moduleManager.find(StorageModule.NAME)
.provider()
.getService(StorageTTLStatusQuery.class);
}
return storageTTLStatusQuery;
}
/**
* @return effective TTL configuration values.
*/
public TTLDefinition getTTL() {
TTLDefinition ttlDefinition = getStorageTTLStatusQuery().getTTL();
if (ttlDefinition == null) {
ttlDefinition = new TTLDefinition(
new MetricsTTL(coreMetricsDataTTL, coreMetricsDataTTL, coreMetricsDataTTL),
new RecordsTTL(coreRecordDataTTL, coreRecordDataTTL)
);
}
return ttlDefinition;
}
}

2
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/StorageModule.java
View File

@ -46,6 +46,7 @@ import org.apache.skywalking.oap.server.core.storage.query.IRecordsQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IZipkinQueryDAO;
import org.apache.skywalking.oap.server.core.storage.ttl.StorageTTLStatusQuery;
import org.apache.skywalking.oap.server.library.module.ModuleDefine;
/**
@ -95,6 +96,7 @@ public class StorageModule extends ModuleDefine {
IAsyncProfilerTaskQueryDAO.class,
IAsyncProfilerTaskLogQueryDAO.class,
IJFRDataQueryDAO.class,
StorageTTLStatusQuery.class
};
}
}

29
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/ttl/DefaultStorageTTLStatusQuery.java Normal file
View File

@ -0,0 +1,29 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.storage.ttl;
/**
* The default implementation of {@link StorageTTLStatusQuery}.
* This is used when the storage doesn't provide the TTL customization.
* Typically, BanyanDB provides advanced progressive TTL configurations.
* For more details, visit <a href="https://skywalking.apache.org/docs/main/next/en/banyandb/ttl">Progressive TTL
* Documentation</a>.
*/
public class DefaultStorageTTLStatusQuery implements StorageTTLStatusQuery {
}

36
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/ttl/MetricsTTL.java Normal file
View File

@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.storage.ttl;
import lombok.Data;
/**
* Metrics TTL includes the definition of the TTL of the metrics-ish data in the storage,
* e.g.
* 1. The metadata of the service, instance, endpoint, topology map, etc.
* 2. Generated metrics data from OAL and MAL engines.
*
* TTLs for ach granularity metrics are listed separately.
*/
@Data
public class MetricsTTL {
private final int minute;
private final int hour;
private final int day;
}

34
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/ttl/RecordsTTL.java Normal file
View File

@ -0,0 +1,34 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.storage.ttl;
import com.google.gson.annotations.SerializedName;
import lombok.Data;
/**
* RecordsTTL includes the definition of the TTL of the records data in the storage,
* Records include traces, logs, sampled slow SQL statements, HTTP requests(by Rover), alarms, etc.
* Super dataset of records are traces and logs, which volume should be much larger.
*/
@Data
public class RecordsTTL {
@SerializedName("default")
private final int value;
private final int superDataset;
}

33
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/ttl/StorageTTLStatusQuery.java Normal file
View File

@ -0,0 +1,33 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.storage.ttl;
import org.apache.skywalking.oap.server.library.module.Service;
public interface StorageTTLStatusQuery extends Service {
/**
* Get the TTL of the metrics and records data from the selected storage.
*
* @return null if the storage doesn't support TTL customization. Or return the TTL definition from specific storage
* implementation.
*/
default TTLDefinition getTTL() {
return null;
}
}

56
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/ttl/TTLDefinition.java Normal file
View File

@ -0,0 +1,56 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.storage.ttl;
import com.google.gson.Gson;
import lombok.Data;
/**
* TTLDefinition defines the TTL of the data in the storage.
*/
@Data
public class TTLDefinition {
private final static Gson GSON = new Gson();
private final MetricsTTL metrics;
private final RecordsTTL records;
public String generateTTLDefinition() {
StringBuilder ttlDefinition = new StringBuilder();
ttlDefinition.append("# Metrics TTL includes the definition of the TTL of the metrics-ish data in the storage,\n");
ttlDefinition.append("# e.g.\n");
ttlDefinition.append("# 1. The metadata of the service, instance, endpoint, topology map, etc.\n");
ttlDefinition.append("# 2. Generated metrics data from OAL and MAL engines.\n");
ttlDefinition.append("#\n");
ttlDefinition.append("# TTLs for each granularity metrics are listed separately.\n");
ttlDefinition.append("metrics.minute=").append(metrics.getMinute()).append("\n");
ttlDefinition.append("metrics.hour=").append(metrics.getHour()).append("\n");
ttlDefinition.append("metrics.day=").append(metrics.getDay()).append("\n");
ttlDefinition.append("\n");
ttlDefinition.append("# Records TTL includes the definition of the TTL of the records data in the storage,\n");
ttlDefinition.append("# Records include traces, logs, sampled slow SQL statements, HTTP requests(by Rover), alarms, etc.\n");
ttlDefinition.append("# Super dataset of records are traces and logs, which volume should be much larger.\n");
ttlDefinition.append("records.default=").append(records.getValue()).append("\n");
ttlDefinition.append("records.superDataset=").append(records.getSuperDataset()).append("\n");
return ttlDefinition.toString();
}
public String generateTTLDefinitionAsJSONStr() {
return GSON.toJson(this);
}
}

2
oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/CoreModuleTest.java
View File

@ -26,6 +26,6 @@ public class CoreModuleTest {
public void testOpenServiceList() {
CoreModule coreModule = new CoreModule();
Assertions.assertEquals(47, coreModule.services().length);
Assertions.assertEquals(48, coreModule.services().length);
}
}

2
oap-server/server-query-plugin/pom.xml
View File

@ -46,6 +46,6 @@
<module>zipkin-query-plugin</module>
<module>promql-plugin</module>
<module>logql-plugin</module>
<module>debugging-query-plugin</module>
<module>status-query-plugin</module>
</modules>
</project>

2
oap-server/server-query-plugin/debugging-query-plugin/pom.xml → oap-server/server-query-plugin/status-query-plugin/pom.xml
View File

@ -25,7 +25,7 @@
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>debugging-query-plugin</artifactId>
<artifactId>status-query-plugin</artifactId>
<packaging>jar</packaging>
<dependencies>

6
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingHTTPHandler.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingHTTPHandler.java
View File

@ -85,7 +85,7 @@ import org.apache.skywalking.oap.server.library.module.ModuleManager;
import zipkin2.Span;
@Slf4j
@ExceptionHandler(DebuggingQueryExceptionHandler.class)
@ExceptionHandler(StatusQueryExceptionHandler.class)
public class DebuggingHTTPHandler {
private final ServerStatusService serverStatusService;
private final MetricsExpressionQuery mqeQuery;
@ -93,9 +93,9 @@ public class DebuggingHTTPHandler {
private final ZipkinQueryHandler zipkinQueryHandler;
private final TopologyQuery topologyQuery;
private final LogQuery logQuery;
final DebuggingQueryConfig config;
final StatusQueryConfig config;
public DebuggingHTTPHandler(final ModuleManager manager, final DebuggingQueryConfig config) {
public DebuggingHTTPHandler(final ModuleManager manager, final StatusQueryConfig config) {
serverStatusService = manager.find(CoreModule.NAME)
.provider()
.getService(ServerStatusService.class);

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingSpanRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingSpanRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingTraceRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingTraceRsp.java
View File

2
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingQueryConfig.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/StatusQueryConfig.java
View File

@ -22,7 +22,7 @@ import lombok.Getter;
import org.apache.skywalking.oap.server.library.module.ModuleConfig;
@Getter
public class DebuggingQueryConfig extends ModuleConfig {
public class StatusQueryConfig extends ModuleConfig {
/**
* Include the list of keywords to filter configurations including secrets. Separate keywords by a comma.
*

2
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingQueryExceptionHandler.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/StatusQueryExceptionHandler.java
View File

@ -29,7 +29,7 @@ import static com.linecorp.armeria.common.HttpStatus.INTERNAL_SERVER_ERROR;
import static com.linecorp.armeria.common.MediaType.ANY_TEXT_TYPE;
@Slf4j
public class DebuggingQueryExceptionHandler implements ExceptionHandlerFunction {
public class StatusQueryExceptionHandler implements ExceptionHandlerFunction {
@Override
public HttpResponse handleException(final ServiceRequestContext ctx, final HttpRequest req, final Throwable cause) {
String rspMsg = cause.getMessage() != null ? cause.getMessage() : cause.getClass().getSimpleName();

6
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingQueryModule.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/StatusQueryModule.java
View File

@ -20,10 +20,10 @@ package org.apache.skywalking.oap.query.debug;
import org.apache.skywalking.oap.server.library.module.ModuleDefine;
public class DebuggingQueryModule extends ModuleDefine {
public static final String NAME = "debugging-query";
public class StatusQueryModule extends ModuleDefine {
public static final String NAME = "status-query";
public DebuggingQueryModule() {
public StatusQueryModule() {
super(NAME);
}

18
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/DebuggingQueryProvider.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/StatusQueryProvider.java
View File

@ -28,28 +28,28 @@ import org.apache.skywalking.oap.server.library.module.ModuleProvider;
import org.apache.skywalking.oap.server.library.module.ModuleStartException;
import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
public class DebuggingQueryProvider extends ModuleProvider {
public class StatusQueryProvider extends ModuleProvider {
public static final String NAME = "default";
private DebuggingQueryConfig config;
private StatusQueryConfig config;
public String name() {
return NAME;
}
public Class<? extends ModuleDefine> module() {
return DebuggingQueryModule.class;
return StatusQueryModule.class;
}
public ConfigCreator<? extends ModuleConfig> newConfigCreator() {
return new ConfigCreator<DebuggingQueryConfig>() {
return new ConfigCreator<StatusQueryConfig>() {
@Override
public Class<DebuggingQueryConfig> type() {
return DebuggingQueryConfig.class;
public Class<StatusQueryConfig> type() {
return StatusQueryConfig.class;
}
@Override
public void onInitialized(final DebuggingQueryConfig initialized) {
public void onInitialized(final StatusQueryConfig initialized) {
config = initialized;
}
};
@ -67,6 +67,10 @@ public class DebuggingQueryProvider extends ModuleProvider {
new DebuggingHTTPHandler(getManager(), config),
Collections.singletonList(HttpMethod.GET)
);
service.addHandler(
new TTLConfigQueryHandler(getManager()),
Collections.singletonList(HttpMethod.GET)
);
}
public void notifyAfterCompleted() throws ServiceNotProvidedException, ModuleStartException {

58
oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/TTLConfigQueryHandler.java Normal file
View File

@ -0,0 +1,58 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.query.debug;
import com.linecorp.armeria.common.HttpHeaderNames;
import com.linecorp.armeria.common.HttpRequest;
import com.linecorp.armeria.common.HttpResponse;
import com.linecorp.armeria.common.MediaType;
import com.linecorp.armeria.server.annotation.ExceptionHandler;
import com.linecorp.armeria.server.annotation.Get;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.CoreModule;
import org.apache.skywalking.oap.server.core.query.TTLStatusQuery;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
@Slf4j
@ExceptionHandler(StatusQueryExceptionHandler.class)
public class TTLConfigQueryHandler {
private final ModuleManager moduleManager;
private TTLStatusQuery ttlStatusQuery;
public TTLConfigQueryHandler(final ModuleManager manager) {
this.moduleManager = manager;
}
private TTLStatusQuery getTTLStatusQuery() {
if (ttlStatusQuery == null) {
ttlStatusQuery = moduleManager.find(CoreModule.NAME)
.provider()
.getService(TTLStatusQuery.class);
}
return ttlStatusQuery;
}
@Get("/status/config/ttl")
public HttpResponse affectedTTLConfigurations(HttpRequest request) {
if ("application/json".equalsIgnoreCase(request.headers().get(HttpHeaderNames.ACCEPT))) {
return HttpResponse.of(MediaType.JSON_UTF_8, getTTLStatusQuery().getTTL().generateTTLDefinitionAsJSONStr());
}
return HttpResponse.of(MediaType.PLAIN_TEXT_UTF_8, getTTLStatusQuery().getTTL().generateTTLDefinition());
}
}

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/log/DebuggingQueryLogsRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/log/DebuggingQueryLogsRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/mqe/DebuggingMQERsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/mqe/DebuggingMQERsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/topology/DebuggingQueryEndpointTopologyRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/topology/DebuggingQueryEndpointTopologyRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/topology/DebuggingQueryInstanceTopologyRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/topology/DebuggingQueryInstanceTopologyRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/topology/DebuggingQueryProcessTopologyRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/topology/DebuggingQueryProcessTopologyRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/topology/DebuggingQueryServiceTopologyRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/topology/DebuggingQueryServiceTopologyRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/trace/DebuggingQueryTraceBriefRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/trace/DebuggingQueryTraceBriefRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/trace/DebuggingQueryTraceRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/trace/DebuggingQueryTraceRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/trace/zipkin/DebuggingZipkinQueryTraceRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/trace/zipkin/DebuggingZipkinQueryTraceRsp.java
View File

0
oap-server/server-query-plugin/debugging-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/trace/zipkin/DebuggingZipkinQueryTracesRsp.java → oap-server/server-query-plugin/status-query-plugin/src/main/java/org/apache/skywalking/oap/query/debug/trace/zipkin/DebuggingZipkinQueryTracesRsp.java
View File

2
oap-server/server-query-plugin/debugging-query-plugin/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine → oap-server/server-query-plugin/status-query-plugin/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine
View File

@ -16,4 +16,4 @@
#
#
org.apache.skywalking.oap.query.debug.DebuggingQueryModule
org.apache.skywalking.oap.query.debug.StatusQueryModule

2
oap-server/server-query-plugin/debugging-query-plugin/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider → oap-server/server-query-plugin/status-query-plugin/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider
View File

@ -16,4 +16,4 @@
#
#
org.apache.skywalking.oap.query.debug.DebuggingQueryProvider
org.apache.skywalking.oap.query.debug.StatusQueryProvider

2
oap-server/server-starter/pom.xml
View File

@ -113,7 +113,7 @@
</dependency>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>debugging-query-plugin</artifactId>
<artifactId>status-query-plugin</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>

4
oap-server/server-starter/src/main/resources/application.yml
View File

@ -641,8 +641,8 @@ health-checker:
default:
checkIntervalSeconds: ${SW_HEALTH_CHECKER_INTERVAL_SECONDS:5}
debugging-query:
selector: ${SW_DEBUGGING_QUERY:default}
status-query:
selector: ${SW_STATUS_QUERY:default}
default:
# Include the list of keywords to filter configurations including secrets. Separate keywords by a comma.
keywords4MaskingSecretsOfConfig: ${SW_DEBUGGING_QUERY_KEYWORDS_FOR_MASKING_SECRETS:user,password,token,accessKey,secretKey,authentication}

24
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageConfig.java
View File

@ -70,10 +70,26 @@ public class BanyanDBStorageConfig extends ModuleConfig {
private int segmentQueryMaxSize = 200;
private int profileDataQueryBatchSize = 100;
/**
* The configuration of the groups.
* @since 10.2.0
*/
// ----------------------------------------
// The configuration of the groups.
// since 10.2.0
// ----------------------------------------
// The group settings of record.
// `gr` is the short name of the group settings of record.
//
// The "normal"(`gr...`) section defines settings for datasets not specified in "super".
// Each dataset will be grouped under a single group named "normal".
// "super"(`grSuper...`) is a special dataset designed to store trace or log data that is too large for normal datasets.
// # Each super dataset will be a separate group in BanyanDB, following the settings defined in the "super" section.
// ----------------------------------------
// The group settings of metrics.
// `gm` is the short name of the group settings of metrics.
//
// OAP stores metrics based its granularity.
// Valid values are "day", "hour", and "minute". That means metrics will be stored in the three separate groups.
// Non-"minute" are governed by the "core.downsampling" setting.
// For example, if "core.downsampling" is set to "hour", the "hour" will be used, while "day" are ignored.
private int grNormalShardNum = 1;
private int grNormalSIDays = 1;
private int grNormalTTLDays = 3;

6
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageProvider.java
View File

@ -55,6 +55,7 @@ import org.apache.skywalking.oap.server.core.storage.query.ITagAutoCompleteQuery
import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IZipkinQueryDAO;
import org.apache.skywalking.oap.server.core.storage.ttl.StorageTTLStatusQuery;
import org.apache.skywalking.oap.server.library.module.ModuleDefine;
import org.apache.skywalking.oap.server.library.module.ModuleProvider;
import org.apache.skywalking.oap.server.library.module.ModuleStartException;
@ -186,8 +187,11 @@ public class BanyanDBStorageProvider extends ModuleProvider {
IAsyncProfilerTaskLogQueryDAO.class, new BanyanDBAsyncProfilerTaskLogQueryDAO(client,
this.config.getAsyncProfilerTaskQueryMaxSize()
));
this.registerServiceImplementation(IJFRDataQueryDAO.class, new BanyanDBJFRDataQueryDAO(client));
this.registerServiceImplementation(
IJFRDataQueryDAO.class, new BanyanDBJFRDataQueryDAO(client));
StorageTTLStatusQuery.class,
new BanyanDBTTLStatusQuery(config)
);
}
@Override

43
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBTTLStatusQuery.java Normal file
View File

@ -0,0 +1,43 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.banyandb;
import org.apache.skywalking.oap.server.core.storage.ttl.StorageTTLStatusQuery;
import org.apache.skywalking.oap.server.core.storage.ttl.TTLDefinition;
public class BanyanDBTTLStatusQuery implements StorageTTLStatusQuery {
private final int grNormalTTLDays;
private final int grSuperTTLDays;
private final int gmMinuteTTLDays;
private final int gmHourTTLDays;
private final int gmDayTTLDays;
public BanyanDBTTLStatusQuery(BanyanDBStorageConfig config) {
grNormalTTLDays = config.getGrNormalTTLDays();
grSuperTTLDays = config.getGrSuperTTLDays();
gmMinuteTTLDays = config.getGmMinuteTTLDays();
gmHourTTLDays = config.getGmHourTTLDays();
gmDayTTLDays = config.getGmDayTTLDays();
}
@Override
public TTLDefinition getTTL() {
return null;
}
}

6
oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/StorageModuleElasticsearchProvider.java
View File

@ -61,6 +61,8 @@ import org.apache.skywalking.oap.server.core.storage.query.ITagAutoCompleteQuery
import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IZipkinQueryDAO;
import org.apache.skywalking.oap.server.core.storage.ttl.DefaultStorageTTLStatusQuery;
import org.apache.skywalking.oap.server.core.storage.ttl.StorageTTLStatusQuery;
import org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient;
import org.apache.skywalking.oap.server.library.module.ModuleDefine;
import org.apache.skywalking.oap.server.library.module.ModuleProvider;
@ -273,6 +275,10 @@ public class StorageModuleElasticsearchProvider extends ModuleProvider {
IJFRDataQueryDAO.class,
new JFRDataQueryEsDAO(elasticSearchClient)
);
this.registerServiceImplementation(
StorageTTLStatusQuery.class,
new DefaultStorageTTLStatusQuery()
);
IndexController.INSTANCE.setLogicSharding(config.isLogicSharding());
IndexController.INSTANCE.setEnableCustomRouting(config.isEnableCustomRouting());
this.registerServiceImplementation(IHierarchyQueryDAO.class, new HierarchyQueryEsDAO(elasticSearchClient, config));

51
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/JDBCStorageProvider.java
View File

@ -55,6 +55,8 @@ import org.apache.skywalking.oap.server.core.storage.query.ITagAutoCompleteQuery
import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IZipkinQueryDAO;
import org.apache.skywalking.oap.server.core.storage.ttl.DefaultStorageTTLStatusQuery;
import org.apache.skywalking.oap.server.core.storage.ttl.StorageTTLStatusQuery;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCClient;
import org.apache.skywalking.oap.server.library.module.ModuleDefine;
import org.apache.skywalking.oap.server.library.module.ModuleProvider;
@ -207,39 +209,56 @@ public abstract class JDBCStorageProvider extends ModuleProvider {
this.registerServiceImplementation(
IEBPFProfilingTaskDAO.class,
new JDBCEBPFProfilingTaskDAO(jdbcClient, tableHelper));
new JDBCEBPFProfilingTaskDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
IEBPFProfilingScheduleDAO.class,
new JDBCEBPFProfilingScheduleDAO(jdbcClient, tableHelper));
new JDBCEBPFProfilingScheduleDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
IEBPFProfilingDataDAO.class,
new JDBCEBPFProfilingDataDAO(jdbcClient, tableHelper));
new JDBCEBPFProfilingDataDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
IContinuousProfilingPolicyDAO.class,
new JDBCContinuousProfilingPolicyDAO(jdbcClient, tableHelper));
new JDBCContinuousProfilingPolicyDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
IServiceLabelDAO.class,
new JDBCServiceLabelQueryDAO(jdbcClient, tableHelper));
new JDBCServiceLabelQueryDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
ITagAutoCompleteQueryDAO.class,
new JDBCTagAutoCompleteQueryDAO(jdbcClient, tableHelper));
new JDBCTagAutoCompleteQueryDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
IZipkinQueryDAO.class,
new JDBCZipkinQueryDAO(jdbcClient, tableHelper));
new JDBCZipkinQueryDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
ISpanAttachedEventQueryDAO.class,
new JDBCSpanAttachedEventQueryDAO(jdbcClient, tableHelper));
this.registerServiceImplementation(IHierarchyQueryDAO.class,
new JDBCHierarchyQueryDAO(jdbcClient, config.getMetadataQueryMaxSize(), tableHelper));
new JDBCSpanAttachedEventQueryDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
IAsyncProfilerTaskQueryDAO.class,
new JDBCAsyncProfilerTaskQueryDAO(jdbcClient, tableHelper));
IHierarchyQueryDAO.class,
new JDBCHierarchyQueryDAO(jdbcClient, config.getMetadataQueryMaxSize(), tableHelper)
);
this.registerServiceImplementation(
IAsyncProfilerTaskLogQueryDAO.class,
new JDBCAsyncProfilerTaskLogQueryDAO(jdbcClient, tableHelper));
IAsyncProfilerTaskQueryDAO.class,
new JDBCAsyncProfilerTaskQueryDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
IJFRDataQueryDAO.class,
new JDBCJFRDataQueryDAO(jdbcClient, tableHelper));
IAsyncProfilerTaskLogQueryDAO.class,
new JDBCAsyncProfilerTaskLogQueryDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
IJFRDataQueryDAO.class,
new JDBCJFRDataQueryDAO(jdbcClient, tableHelper)
);
this.registerServiceImplementation(
StorageTTLStatusQuery.class,
new DefaultStorageTTLStatusQuery()
);
}
@Override