kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Support sharding MySQL database instances and tables. (#9689) 

* [**Breaking Change**] Support sharding MySQL database instances and tables by [Shardingsphere-Proxy](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-proxy).
  SQL-Database requires removing tables `log_tag/segment_tag/zipkin_query` before OAP starts, if bump up from previous releases.
This commit is contained in:
Wan Kai 2022-09-30 08:39:14 +08:00 committed by GitHub
parent fdc24ed822
commit d7ee9a01d6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
101 changed files with 3208 additions and 64 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
.github/workflows/skywalking.yaml vendored
View File

@ -573,6 +573,15 @@ jobs:
config: test/e2e-v2/cases/zipkin/postgres/e2e.yaml
- name: Zipkin Kafka
config: test/e2e-v2/cases/zipkin/kafka/e2e.yaml
- name: MySQL-Sharding storage
config: test/e2e-v2/cases/storage/mysql/sharding/e2e.yaml
- name: MySQL-Sharding log
config: test/e2e-v2/cases/log/mysql/sharding/e2e.yaml
- name: MySQL-Sharding browser
config: test/e2e-v2/cases/browser/mysql/sharding/e2e.yaml
- name: MySQL-Sharding zipkin
config: test/e2e-v2/cases/zipkin/mysql/sharding/e2e.yaml
steps:
- uses: actions/checkout@v3
with:

2
README.md
View File

@ -50,7 +50,7 @@ and multiple formats, including
6. Browser agent.
7. Service Mesh Observability. Control plane and data plane.
8. Metrics system, including Prometheus, OpenTelemetry, Micrometer(Spring Sleuth), Zabbix.
9. Logs， including OpenTelemetry.
9. Logs, including OpenTelemetry.
10. Zipkin v1/v2 trace.(No Analysis).
# Documentation

2
docs/en/changes/changes.md
View File

@ -20,6 +20,8 @@
in `no-init` mode.
* Make sure instance list ordered in TTL processor to avoid TTL timer never runs.
* Support monitoring PostgreSQL slow SQLs.
* [**Breaking Change**] Support sharding MySQL database instances and tables by [Shardingsphere-Proxy](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-proxy).
SQL-Database requires removing tables `log_tag/segment_tag/zipkin_query` before OAP starts, if bump up from previous releases.
* Fix meter functions `avgHistogram`, `avgHistogramPercentile`, `avgLabeled`, `sumHistogram` having data conflict when
downsampling.
* Do sorting `readLabeledMetricsValues` result forcedly in case the storage(database) doesn't return data consistent

29
docs/en/setup/backend/backend-storage.md
View File

@ -12,6 +12,7 @@ Natively supported storage:
- OpenSearch
- ElasticSearch 6, 7, 8
- MySQL
- MySQL-Sharding(Shardingsphere-Proxy 5.1.2)
- TiDB
- PostgreSQL
- BanyanDB
@ -211,6 +212,34 @@ All connection-related settings, including URL link, username, and password, are
Only part of the settings is listed here. See the [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
To understand the function of the parameter `rewriteBatchedStatements=true` in MySQL, see the [MySQL official document](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-performance-extensions.html#cj-conn-prop_rewriteBatchedStatements).
## MySQL-Sharding
MySQL-Sharding plugin provides the MySQL database sharding and table sharding, this feature
leverage [Shardingsphere-Proxy](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-proxy)
to manage the JDBC between OAP and multi-database instances, and according to the sharding rules do routing to the database and table sharding.
Tested Shardingsphere-Proxy 5.1.2 version, and MySQL Client driver 8.0.13 version is currently available.
Activate MySQL and Shardingsphere-Proxy as storage, and set storage provider to **mysql-sharding**.
**NOTE:** MySQL driver is NOT allowed in Apache official distribution and source codes.
Please download the MySQL driver on your own. Copy the connection driver jar to `oap-libs`.
```yaml
storage:
selector: ${SW_STORAGE:mysql-sharding}
mysql-sharding:
properties:
jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://localhost:13307/swtest?rewriteBatchedStatements=true"}
dataSource.user: ${SW_DATA_SOURCE_USER:root}
dataSource.password: ${SW_DATA_SOURCE_PASSWORD:root}
metadataQueryMaxSize: ${SW_STORAGE_MYSQL_QUERY_MAX_SIZE:5000}
maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:2000}
asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:4}
# The dataSources are configured in ShardingSphere-Proxy config-sharding.yaml
# The dataSource name should include the prefix "ds_" and separated by ","
dataSources: ${SW_JDBC_SHARDING_DATA_SOURCES:ds_0,ds_1}
```
## TiDB
Tested TiDB Server 4.0.8 version, and MySQL Client driver 8.0.13 version is currently available.
Activate TiDB as storage, and set storage provider to **tidb**.

10
docs/en/setup/backend/configuration-vocabulary.md
View File

@ -126,15 +126,17 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | mysql | - | MySQL Storage. The MySQL JDBC Driver is not in the dist. Please copy it into the oap-lib folder manually. | - | - |
| - | - | properties | Hikari connection pool configurations. | - | Listed in the `application.yaml`. |
| - | - | metadataQueryMaxSize | The maximum size of metadata per query. | SW_STORAGE_MYSQL_QUERY_MAX_SIZE | 5000 |
| - | - | maxSizeOfArrayColumn | Some entities (e.g. trace segments) include the logic column with multiple values. In MySQL, we use multiple physical columns to host the values, e.g. change column_a with values [1,2,3,4,5] to `column_a_0 = 1, column_a_1 = 2, column_a_2 = 3 , column_a_3 = 4, column_a_4 = 5`. | SW_STORAGE_MAX_SIZE_OF_ARRAY_COLUMN | 20 |
| - | - | numOfSearchableValuesPerTag | In a trace segment, this includes multiple spans with multiple tags. Different spans may have same tag key, e.g. multiple HTTP exit spans all have their own `http.method` tags. This configuration sets the limit on the maximum number of values for the same tag key. | SW_STORAGE_NUM_OF_SEARCHABLE_VALUES_PER_TAG | 2 |
| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution | SW_STORAGE_MAX_SIZE_OF_BATCH_SQL | 2000 |
| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size | SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE | 4 |
| - | mysql-sharding | - | Sharding-Proxy for MySQL properties. The MySQL JDBC Driver is not in the dist. Please copy it into the oap-lib folder manually. | - | - |
| - | - | properties | Hikari connection pool configurations. | - | Listed in the `application.yaml`. |
| - | - | metadataQueryMaxSize | The maximum size of metadata per query. | SW_STORAGE_MYSQL_QUERY_MAX_SIZE | 5000 |
| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution | SW_STORAGE_MAX_SIZE_OF_BATCH_SQL | 2000 |
| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size | SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE | 4 |
| - | - | dataSources | The dataSources are configured in ShardingSphere-Proxy config-sharding.yaml.The dataSource name should include the prefix "ds_" and separated by "," and start from ds_0 | SW_JDBC_SHARDING_DATA_SOURCES | ds_0,ds_1 |
| - | postgresql | - | PostgreSQL storage. | - | - |
| - | - | properties | Hikari connection pool configurations. | - | Listed in the `application.yaml`. |
| - | - | metadataQueryMaxSize | The maximum size of metadata per query. | SW_STORAGE_MYSQL_QUERY_MAX_SIZE | 5000 |
| - | - | maxSizeOfArrayColumn | Some entities (e.g. trace segments) include the logic column with multiple values. In PostgreSQL, we use multiple physical columns to host the values, e.g. change column_a with values [1,2,3,4,5] to `column_a_0 = 1, column_a_1 = 2, column_a_2 = 3 , column_a_3 = 4, column_a_4 = 5` | SW_STORAGE_MAX_SIZE_OF_ARRAY_COLUMN | 20 |
| - | - | numOfSearchableValuesPerTag | In a trace segment, this includes multiple spans with multiple tags. Different spans may have same tag key, e.g. multiple HTTP exit spans all have their own `http.method` tags. This configuration sets the limit on the maximum number of values for the same tag key. | SW_STORAGE_NUM_OF_SEARCHABLE_VALUES_PER_TAG | 2 |
| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution | SW_STORAGE_MAX_SIZE_OF_BATCH_SQL | 2000 |
| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size | SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE | 4 |
| - | banyandb | - | BanyanDB storage. | - | - |

2
oap-server/analyzer/event-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/event/listener/EventRecordAnalyzerListener.java
View File

@ -25,7 +25,7 @@ import org.apache.skywalking.oap.server.core.analysis.Layer;
import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.config.NamingControl;
import org.apache.skywalking.oap.server.core.source.Event;
import org.apache.skywalking.oap.server.core.analysis.metrics.Event;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import lombok.RequiredArgsConstructor;

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/endpoint/EndpointTraffic.java
View File

@ -30,16 +30,22 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.ElasticSearch;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ID;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
@Stream(name = EndpointTraffic.INDEX_NAME, scopeId = DefaultScopeDefine.ENDPOINT,
builder = EndpointTraffic.Builder.class, processor = MetricsStreamProcessor.class)
@MetricsExtension(supportDownSampling = false, supportUpdate = false)
@EqualsAndHashCode
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_MIN_RANGE_SHARDING_ALGORITHM, tableShardingColumn = TIME_BUCKET, dataSourceShardingColumn = ID)
public class EndpointTraffic extends Metrics {
public static final String INDEX_NAME = "endpoint_traffic";

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java
View File

@ -30,8 +30,10 @@ import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.ElasticSearch;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -46,6 +48,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.SE
"serviceId",
"name"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class InstanceTraffic extends Metrics {
public static final String INDEX_NAME = "instance_traffic";
public static final String SERVICE_ID = "service_id";

1
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/AbstractLogRecord.java
View File

@ -50,6 +50,7 @@ public abstract class AbstractLogRecord extends Record {
@Getter
@Column(columnName = SERVICE_ID)
@BanyanDB.ShardingKey(index = 0)
@SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_TAG_TABLE}, reserveOriginalColumns = true)
private String serviceId;
@Setter
@Getter

4
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/LogRecord.java
View File

@ -22,16 +22,20 @@ import lombok.Setter;
import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.annotation.SuperDataset;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import static org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord.SERVICE_ID;
import static org.apache.skywalking.oap.server.core.analysis.record.Record.TIME_BUCKET;
@SuperDataset
@Stream(name = LogRecord.INDEX_NAME, scopeId = DefaultScopeDefine.LOG, builder = LogRecord.Builder.class, processor = RecordStreamProcessor.class)
@SQLDatabase.ExtraColumn4AdditionalEntity(additionalTable = AbstractLogRecord.ADDITIONAL_TAG_TABLE, parentColumn = TIME_BUCKET)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM, dataSourceShardingColumn = SERVICE_ID, tableShardingColumn = TIME_BUCKET)
public class LogRecord extends AbstractLogRecord {
public static final String INDEX_NAME = "log";

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/networkalias/NetworkAddressAlias.java
View File

@ -28,8 +28,10 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.ScopeDeclaration;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -43,6 +45,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.NE
@EqualsAndHashCode(of = {
"address"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class NetworkAddressAlias extends Metrics {
public static final String INDEX_NAME = "network_address_alias";
public static final String ADDRESS = "address";

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/process/ProcessTraffic.java
View File

@ -31,7 +31,9 @@ import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -48,6 +50,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.PR
"instanceId",
"name",
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class ProcessTraffic extends Metrics {
public static final String INDEX_NAME = "process_traffic";
public static final String SERVICE_ID = "service_id";

5
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/process/ServiceLabelRecord.java
View File

@ -26,7 +26,9 @@ import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -50,6 +52,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.SE
"serviceId",
"label"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class ServiceLabelRecord extends Metrics {
public static final String INDEX_NAME = "service_label";
@ -125,4 +128,4 @@ public class ServiceLabelRecord extends Metrics {
converter.accept(TIME_BUCKET, entity.getTimeBucket());
}
}
}
}

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/endpoint/EndpointRelationServerSideMetrics.java
View File

@ -27,17 +27,23 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
@Stream(name = EndpointRelationServerSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.ENDPOINT_RELATION,
builder = EndpointRelationServerSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_BUCKET_SHARDING_ALGORITHM, tableShardingColumn = TIME_BUCKET, dataSourceShardingColumn = ENTITY_ID)
public class EndpointRelationServerSideMetrics extends Metrics {
public static final String INDEX_NAME = "endpoint_relation_server_side";

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationClientSideMetrics.java
View File

@ -27,16 +27,22 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
@Stream(name = ServiceInstanceRelationClientSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.SERVICE_INSTANCE_RELATION,
builder = ServiceInstanceRelationClientSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_BUCKET_SHARDING_ALGORITHM, tableShardingColumn = TIME_BUCKET, dataSourceShardingColumn = ENTITY_ID)
public class ServiceInstanceRelationClientSideMetrics extends Metrics {
public static final String INDEX_NAME = "service_instance_relation_client_side";

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationServerSideMetrics.java
View File

@ -27,17 +27,23 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
@Stream(name = ServiceInstanceRelationServerSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.SERVICE_INSTANCE_RELATION,
builder = ServiceInstanceRelationServerSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_BUCKET_SHARDING_ALGORITHM, tableShardingColumn = TIME_BUCKET, dataSourceShardingColumn = ENTITY_ID)
public class ServiceInstanceRelationServerSideMetrics extends Metrics {
public static final String INDEX_NAME = "service_instance_relation_server_side";

5
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationClientSideMetrics.java
View File

@ -28,7 +28,9 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -39,6 +41,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class ProcessRelationClientSideMetrics extends Metrics {
public static final String INDEX_NAME = "process_relation_client_side";
@ -146,4 +149,4 @@ public class ProcessRelationClientSideMetrics extends Metrics {
converter.accept(COMPONENT_ID, storageData.getComponentId());
}
}
}
}

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationServerSideMetrics.java
View File

@ -28,7 +28,9 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -39,6 +41,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class ProcessRelationServerSideMetrics extends Metrics {
public static final String INDEX_NAME = "process_relation_server_side";

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationClientSideMetrics.java
View File

@ -27,16 +27,22 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
@Stream(name = ServiceRelationClientSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.SERVICE_RELATION,
builder = ServiceRelationClientSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_BUCKET_SHARDING_ALGORITHM, tableShardingColumn = TIME_BUCKET, dataSourceShardingColumn = ENTITY_ID)
public class ServiceRelationClientSideMetrics extends Metrics {
public static final String INDEX_NAME = "service_relation_client_side";

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationServerSideMetrics.java
View File

@ -27,17 +27,23 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
@Stream(name = ServiceRelationServerSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.SERVICE_RELATION,
builder = ServiceRelationServerSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_BUCKET_SHARDING_ALGORITHM, tableShardingColumn = TIME_BUCKET, dataSourceShardingColumn = ENTITY_ID)
public class ServiceRelationServerSideMetrics extends Metrics {
public static final String INDEX_NAME = "service_relation_server_side";

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java
View File

@ -27,7 +27,9 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -40,6 +42,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
"tagValue",
"tagType"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class TagAutocompleteData extends Metrics {
public static final String INDEX_NAME = "tag_autocomplete";
public static final String TAG_KEY = "tag_key";

5
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/segment/SegmentRecord.java
View File

@ -25,6 +25,7 @@ import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.record.Record;
import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
@ -32,11 +33,14 @@ import org.apache.skywalking.oap.server.core.storage.annotation.SuperDataset;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import static org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord.SERVICE_ID;
import static org.apache.skywalking.oap.server.core.analysis.record.Record.TIME_BUCKET;
@SuperDataset
@Stream(name = SegmentRecord.INDEX_NAME, scopeId = DefaultScopeDefine.SEGMENT, builder = SegmentRecord.Builder.class, processor = RecordStreamProcessor.class)
@SQLDatabase.ExtraColumn4AdditionalEntity(additionalTable = SegmentRecord.ADDITIONAL_TAG_TABLE, parentColumn = TIME_BUCKET)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM, dataSourceShardingColumn = SERVICE_ID, tableShardingColumn = TIME_BUCKET)
public class SegmentRecord extends Record {
public static final String INDEX_NAME = "segment";
@ -65,6 +69,7 @@ public class SegmentRecord extends Record {
@Getter
@Column(columnName = SERVICE_ID)
@BanyanDB.ShardingKey(index = 0)
@SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_TAG_TABLE}, reserveOriginalColumns = true)
private String serviceId;
@Setter
@Getter

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java
View File

@ -30,8 +30,10 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.ElasticSearch;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -46,6 +48,7 @@ import static org.apache.skywalking.oap.server.core.Const.DOUBLE_COLONS_SPLIT;
"name",
"layer"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class ServiceTraffic extends Metrics {
public static final String INDEX_NAME = "service_traffic";

2
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/sumpermin/SumPerMinFunction.java
View File

@ -193,4 +193,4 @@ public abstract class SumPerMinFunction extends Meter implements AcceptableValue
public int hashCode() {
return Objects.hash(getEntityId(), getTimeBucket());
}
}
}

7
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/Event.java → oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Event.java
View File

@ -16,7 +16,7 @@
*
*/
package org.apache.skywalking.oap.server.core.source;
package org.apache.skywalking.oap.server.core.analysis.metrics;
import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.EVENT;
import static org.apache.skywalking.oap.server.library.util.StringUtil.isNotBlank;
@ -25,10 +25,12 @@ import org.apache.skywalking.oap.server.core.analysis.Layer;
import org.apache.skywalking.oap.server.core.analysis.MetricsExtension;
import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.ScopeDeclaration;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -45,6 +47,7 @@ import lombok.Setter;
callSuper = false,
of = "uuid"
)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class Event extends Metrics {
public static final String INDEX_NAME = "events";

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxLongMetrics.java
View File

@ -25,9 +25,6 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Metrics
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.SourceFrom;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
/**
*
**/
@MetricsFunction(functionName = "max")
public abstract class MaxLongMetrics extends Metrics implements LongValueHolder {

7
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Metrics.java
View File

@ -23,8 +23,13 @@ import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
import org.apache.skywalking.oap.server.core.remote.data.StreamData;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.StorageData;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ID;
/**
* Metrics represents the statistic data, which analysis by OAL script or hard code. It has the lifecycle controlled by
@ -33,10 +38,12 @@ import org.apache.skywalking.oap.server.core.storage.annotation.Column;
@EqualsAndHashCode(of = {
"timeBucket"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_RELATIVE_ID_SHARDING_ALGORITHM, tableShardingColumn = ID, dataSourceShardingColumn = ENTITY_ID)
public abstract class Metrics extends StreamData implements StorageData {
public static final String TIME_BUCKET = "time_bucket";
public static final String ENTITY_ID = "entity_id";
public static final String ID = "id";
/**
* Time attribute

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/browser/manual/errorlog/BrowserErrorLogRecord.java
View File

@ -23,15 +23,21 @@ import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.record.Record;
import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.annotation.SuperDataset;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import static org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord.SERVICE_ID;
import static org.apache.skywalking.oap.server.core.analysis.record.Record.TIME_BUCKET;
@SuperDataset
@Stream(name = BrowserErrorLogRecord.INDEX_NAME, scopeId = DefaultScopeDefine.BROWSER_ERROR_LOG, builder = BrowserErrorLogRecord.Builder.class, processor = RecordStreamProcessor.class)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM, dataSourceShardingColumn = SERVICE_ID, tableShardingColumn = TIME_BUCKET)
public class BrowserErrorLogRecord extends Record {
public static final String INDEX_NAME = "browser_error_log";
public static final String UNIQUE_ID = "unique_id";

4
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/ConfigService.java
View File

@ -29,6 +29,8 @@ public class ConfigService implements Service {
private final String searchableTracesTags;
private final String searchableLogsTags;
private final String searchableAlarmTags;
private final int metricsDataTTL;
private final int recordDataTTL;
public ConfigService(CoreModuleConfig moduleConfig) {
this.gRPCHost = moduleConfig.getGRPCHost();
@ -36,5 +38,7 @@ public class ConfigService implements Service {
this.searchableTracesTags = moduleConfig.getSearchableTracesTags();
this.searchableLogsTags = moduleConfig.getSearchableLogsTags();
this.searchableAlarmTags = moduleConfig.getSearchableAlarmTags();
this.metricsDataTTL = moduleConfig.getMetricsDataTTL();
this.recordDataTTL = moduleConfig.getRecordDataTTL();
}
}

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/ebpf/storage/EBPFProfilingScheduleRecord.java
View File

@ -26,7 +26,9 @@ import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -48,6 +50,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.EB
"processId",
"startTime",
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class EBPFProfilingScheduleRecord extends Metrics {
public static final String INDEX_NAME = "ebpf_profiling_schedule";

189
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/ShardingAlgorithm.java Normal file
View File

@ -0,0 +1,189 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.storage;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
/**
* The following algorithms represent different Table Sharding strategies.
* Cooperate with tableShardingColumn and dataSourceShardingColumn which are defined in {@link SQLDatabase.Sharding}
*
* The DataSource sharding strategy are same:
* If we have {dataSourceList = ds_0...ds_n} and {dataSourceShardingColumn.hashcode() % dataSourceList.size() = 2}
* then the route target is ds_2
*
* The sharding number of the tables is according to TTL, One table per day:
* {tableName = table_timeSeries}, {timeSeries = currentDate - TTL +1 ... currentDate + 1}
* For example: if TTL=3, currentDate = 20220907, the sharding tables are:
* Table_20220905
* Table_20220906
* Table_20220907
* Table_20220908
*/
public enum ShardingAlgorithm {
/**
* Wouldn't sharding Table nor DataSource, keep single Table.
*/
NO_SHARDING,
/**
* Use the time_bucket inside the ID column to sharding by day.
* The precision of time_bucket could be second, minute, hour and day in the same table.
*
* For example, the single table `service_metrics`:
* ┌────────────────────────┬────────────┐
* │ id │ value │
* ├────────────────────────┼────────────┤
* │ 20220905_Service_A │ 300 │
* ├────────────────────────┼────────────┤
* │ 2022090512_Service_A │ 200 │
* ├────────────────────────┼────────────┤
* │ 202209051211_Service_A │ 100 │
* ├────────────────────────┼────────────┤
* │ 20220906_Service_A │ 500 │
* ├────────────────────────┼────────────┤
* │ 2022090612_Service_A │ 300 │
* ├────────────────────────┼────────────┤
* │ 202209061211_Service_A │ 100 │
* └────────────────────────┴────────────┘
*
* The sharding tables will be:
* `service_metrics_20220905`
* ┌────────────────────────┬────────────┐
* │ id │ value │
* ├────────────────────────┼────────────┤
* │ 20220905_Service_A │ 300 │
* ├────────────────────────┼────────────┤
* │ 2022090512_Service_A │ 200 │
* ├────────────────────────┼────────────┤
* │ 202209051211_Service_A │ 100 │
* └────────────────────────┴────────────┘
* and `service_metrics_20220906`
* ┌────────────────────────┬────────────┐
* │ id │ value │
* ├────────────────────────┼────────────┤
* │ 20220906_Service_A │ 500 │
* ├────────────────────────┼────────────┤
* │ 2022090612_Service_A │ 300 │
* ├────────────────────────┼────────────┤
* │ 202209061211_Service_A │ 100 │
* └────────────────────────┴────────────┘
*
*/
TIME_RELATIVE_ID_SHARDING_ALGORITHM,
/**
* Use the time_bucket column to sharding by day.
* The precision of time_bucket should be `second`.
*
* For example, the single table `service_records`:
* ┌──────────────┬───────────────┬─────────┐
* │ Service │ time_bucket │ value │
* ├──────────────┼───────────────┼─────────┤
* │ Service_A │20220905121130 │ 300 │
* ├──────────────┼───────────────┼─────────┤
* │ Service_A │20220906181233 │ 200 │
* └──────────────┴───────────────┴─────────┘
* The sharding tables will be:
* `service_records_20220905`
* ┌──────────────┬───────────────┬─────────┐
* │ Service │ time_bucket │ value │
* ├──────────────┼───────────────┼─────────┤
* │ Service_A │20220905121130 │ 300 │
* └──────────────┴───────────────┴─────────┘
* and `service_records_20220906`
* ┌──────────────┬───────────────┬─────────┐
* │ Service │ time_bucket │ value │
* ├──────────────┼───────────────┼─────────┤
* │ Service_A │20220906181233 │ 200 │
* └──────────────┴───────────────┴─────────┘
*/
TIME_SEC_RANGE_SHARDING_ALGORITHM,
/**
* Use the time_bucket column to sharding by day.
* The precision of time_bucket should be `minute`.
*
* For example, the single table `endpoint_traffic`:
* ┌──────────────┬───────────────┐
* │ Endpoint │ time_bucket │
* ├──────────────┼───────────────┤
* │ Endpoint_A │202209051211 │
* ├──────────────┼───────────────┤
* │ Endpoint_B │202209061812 │
* └──────────────┴───────────────┘
* The sharding tables will be:
* `endpoint_traffic_20220905`
* ┌──────────────┬───────────────┐
* │ Endpoint │ time_bucket │
* ├──────────────┼───────────────┤
* │ Endpoint_A │202209051211 │
* └──────────────┴───────────────┘
* and `endpoint_traffic_20220906`
* ┌──────────────┬───────────────┐
* │ Endpoint │ time_bucket │
* ├──────────────┼───────────────┤
* │ Endpoint_B │202209061812 │
* └──────────────┴───────────────┘
*/
TIME_MIN_RANGE_SHARDING_ALGORITHM,
/**
* Use the time_bucket column to sharding by day.
* The precision of time_bucket could be `second, minute, hour and day` in the same table.
* For example, the single table `service_relation`:
* ┌────────────────┬────────────────┐
* │ relation │ time_bucket │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 20220905 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 2022090512 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 202209051211 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 20220906 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 2022090612 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 202209061211 │
* └────────────────┴────────────────┘
* The sharding tables will be:
* `service_relation_20220905`
* ┌────────────────┬────────────────┐
* │ relation │ time_bucket │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 20220905 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 2022090512 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 202209051211 │
* └────────────────┴────────────────┘
* and `endpoint_traffic_20220906`
* ┌────────────────┬────────────────┐
* │ relation │ time_bucket │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 20220906 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 2022090612 │
* ├────────────────┼────────────────┤
* │ Service_A_B │ 202209061211 │
* └────────────────┴────────────────┘
*/
TIME_BUCKET_SHARDING_ALGORITHM
}

13
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/SQLDatabase.java
View File

@ -23,6 +23,7 @@ import java.lang.annotation.Repeatable;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
/**
* SQLDatabase annotation is a holder including all annotations for SQL-based RDBMS storage
@ -135,4 +136,16 @@ public @interface SQLDatabase {
@interface MultipleExtraColumn4AdditionalEntity {
ExtraColumn4AdditionalEntity[] value();
}
/**
* Support add sharding policy for a table model. Can be declared on the superclass and overridden by subclass..
* @since 9.3.0
*/
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@interface Sharding {
ShardingAlgorithm shardingAlgorithm();
String dataSourceShardingColumn() default "";
String tableShardingColumn() default "";
}
}

18
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/SQLDatabaseModelExtension.java
View File

@ -22,10 +22,13 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import lombok.AccessLevel;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
/**
* @since 9.1.0
@ -37,6 +40,9 @@ public class SQLDatabaseModelExtension {
//exclude the columns from the main table
private final List<ModelColumn> excludeColumns = new ArrayList<>(5);
@Setter
private Optional<Sharding> sharding = Optional.empty();
public void appendAdditionalTable(String tableName, ModelColumn column) {
additionalTables.computeIfAbsent(tableName, AdditionalTable::new)
.appendColumn(column);
@ -46,6 +52,10 @@ public class SQLDatabaseModelExtension {
excludeColumns.add(column);
}
public boolean isShardingTable() {
return this.sharding.isPresent() && !this.sharding.get().getShardingAlgorithm().equals(ShardingAlgorithm.NO_SHARDING);
}
@Getter
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public static class AdditionalTable {
@ -63,4 +73,12 @@ public class SQLDatabaseModelExtension {
columns.add(column);
}
}
@Getter
@RequiredArgsConstructor
public static class Sharding {
private final ShardingAlgorithm shardingAlgorithm;
private final String dataSourceShardingColumn;
private final String tableShardingColumn;
}
}

11
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java
View File

@ -24,6 +24,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.StorageException;
@ -250,6 +251,16 @@ public class StorageModels implements IModelManager, ModelCreator, ModelManipula
}
}
// For the annotation need to be declared on the superclass, the other annotation should be declared on the subclass.
if (!sqlDBModelExtension.getSharding().isPresent() && clazz.isAnnotationPresent(SQLDatabase.Sharding.class)) {
SQLDatabase.Sharding sharding = clazz.getAnnotation(SQLDatabase.Sharding.class);
sqlDBModelExtension.setSharding(
Optional.of(new SQLDatabaseModelExtension.Sharding(sharding.shardingAlgorithm(),
sharding.dataSourceShardingColumn(),
sharding.tableShardingColumn()
)));
}
if (Objects.nonNull(clazz.getSuperclass())) {
retrieval(clazz.getSuperclass(), modelName, modelColumns, scopeId, checker, sqlDBModelExtension, record);
}

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceRelationTraffic.java
View File

@ -28,7 +28,9 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -40,6 +42,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
"serviceName",
"remoteServiceName"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class ZipkinServiceRelationTraffic extends Metrics {
public static final String INDEX_NAME = "zipkin_service_relation_traffic";

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceSpanTraffic.java
View File

@ -28,7 +28,9 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -40,6 +42,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
"serviceName",
"spanName"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class ZipkinServiceSpanTraffic extends Metrics {
public static final String INDEX_NAME = "zipkin_service_span_traffic";

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceTraffic.java
View File

@ -28,7 +28,9 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@ -39,6 +41,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"serviceName"
})
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.NO_SHARDING)
public class ZipkinServiceTraffic extends Metrics {
public static final String INDEX_NAME = "zipkin_service_traffic";

6
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinSpanRecord.java
View File

@ -28,6 +28,7 @@ import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.analysis.record.Record;
import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
@ -37,11 +38,14 @@ import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
import org.apache.skywalking.oap.server.library.util.BooleanUtils;
import org.apache.skywalking.oap.server.library.util.StringUtil;
import static org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord.TRACE_ID;
import static org.apache.skywalking.oap.server.core.analysis.record.Record.TIME_BUCKET;
@SuperDataset
@Stream(name = ZipkinSpanRecord.INDEX_NAME, scopeId = DefaultScopeDefine.ZIPKIN_SPAN, builder = ZipkinSpanRecord.Builder.class, processor = RecordStreamProcessor.class)
@SQLDatabase.ExtraColumn4AdditionalEntity(additionalTable = ZipkinSpanRecord.ADDITIONAL_QUERY_TABLE, parentColumn = TIME_BUCKET)
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM, dataSourceShardingColumn = TRACE_ID, tableShardingColumn = TIME_BUCKET)
public class ZipkinSpanRecord extends Record {
private static final Gson GSON = new Gson();
public static final String INDEX_NAME = "zipkin_span";
@ -71,6 +75,7 @@ public class ZipkinSpanRecord extends Record {
@Setter
@Getter
@Column(columnName = TRACE_ID)
@SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_QUERY_TABLE}, reserveOriginalColumns = true)
private String traceId;
@Setter
@Getter
@ -210,6 +215,7 @@ public class ZipkinSpanRecord extends Record {
converter.accept(KIND, storageData.getKind());
converter.accept(TIMESTAMP, storageData.getTimestamp());
converter.accept(TIMESTAMP_MILLIS, storageData.getTimestampMillis());
converter.accept(TIME_BUCKET, storageData.getTimeBucket());
converter.accept(DURATION, storageData.getDuration());
converter.accept(LOCAL_ENDPOINT_SERVICE_NAME, storageData.getLocalEndpointServiceName());
converter.accept(LOCAL_ENDPOINT_IPV4, storageData.getLocalEndpointIPV4());

10
oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/query/DurationTest.java
View File

@ -18,6 +18,9 @@
package org.apache.skywalking.oap.server.core.query;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import org.apache.skywalking.oap.server.core.query.enumeration.Step;
import org.junit.Assert;
import org.junit.Test;
@ -80,4 +83,11 @@ public class DurationTest {
Assert.assertTrue(true);
}
}
@Test
public void testGetDurationPoints() {
List<PointOfTime> pointOfTimes = DurationUtils.INSTANCE.getDurationPoints(Step.DAY, 20220910, 20220912);
Assert.assertTrue(Arrays.asList(20220910L, 20220911L, 20220912L)
.equals(pointOfTimes.stream().map(PointOfTime::getPoint).collect(Collectors.toList())));
}
}

1
oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/resolver/MetricQuery.java
View File

@ -84,7 +84,6 @@ public class MetricQuery implements GraphQLQueryResolver {
public IntValues getLinearIntValues(final MetricCondition metrics,
final Duration duration) throws IOException {
MetricsCondition condition = new MetricsCondition();
condition.setName(metrics.getName());
condition.setEntity(new MockEntity(metrics.getId()));

5
oap-server/server-starter/pom.xml
View File

@ -197,6 +197,11 @@
<artifactId>storage-banyandb-plugin</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>storage-shardingsphere-plugin</artifactId>
<version>${project.version}</version>
</dependency>
<!-- storage module -->
<!-- queryBuild module -->

11
oap-server/server-starter/src/main/resources/application.yml
View File

@ -187,6 +187,17 @@ storage:
metadataQueryMaxSize: ${SW_STORAGE_MYSQL_QUERY_MAX_SIZE:5000}
maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:2000}
asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:4}
mysql-sharding:
properties:
jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://localhost:13307/swtest?rewriteBatchedStatements=true"}
dataSource.user: ${SW_DATA_SOURCE_USER:root}
dataSource.password: ${SW_DATA_SOURCE_PASSWORD:root}
metadataQueryMaxSize: ${SW_STORAGE_MYSQL_QUERY_MAX_SIZE:5000}
maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:2000}
asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:4}
# The dataSources are configured in ShardingSphere-Proxy config-sharding.yaml.
# The dataSource name should include the prefix "ds_" and separated by "," and start from ds_0
dataSources: ${SW_JDBC_SHARDING_DATA_SOURCES:ds_0,ds_1}
tidb:
properties:
jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://localhost:4000/tidbswtest?rewriteBatchedStatements=true"}

1
oap-server/server-storage-plugin/pom.xml
View File

@ -32,5 +32,6 @@
<module>storage-elasticsearch-plugin</module>
<module>storage-tidb-plugin</module>
<module>storage-banyandb-plugin</module>
<module>storage-shardingsphere-plugin</module>
</modules>
</project>

2
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBEventQueryDAO.java
View File

@ -36,7 +36,7 @@ import org.apache.skywalking.oap.server.core.query.type.event.EventQueryConditio
import org.apache.skywalking.oap.server.core.query.type.event.EventType;
import org.apache.skywalking.oap.server.core.query.type.event.Events;
import org.apache.skywalking.oap.server.core.query.type.event.Source;
import org.apache.skywalking.oap.server.core.source.Event;
import org.apache.skywalking.oap.server.core.analysis.metrics.Event;
import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageClient;
import org.apache.skywalking.oap.server.storage.plugin.banyandb.stream.AbstractBanyanDBDAO;

2
oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/ESEventQueryDAO.java
View File

@ -37,7 +37,7 @@ import org.apache.skywalking.oap.server.core.query.type.event.EventQueryConditio
import org.apache.skywalking.oap.server.core.query.type.event.EventType;
import org.apache.skywalking.oap.server.core.query.type.event.Events;
import org.apache.skywalking.oap.server.core.query.type.event.Source;
import org.apache.skywalking.oap.server.core.source.Event;
import org.apache.skywalking.oap.server.core.analysis.metrics.Event;
import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
import org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient;
import org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.EsDAO;

16
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2AggregationQueryDAO.java
View File

@ -83,13 +83,15 @@ public class H2AggregationQueryDAO implements IAggregationQueryDAO {
protected StringBuilder buildMetricsValueSql(String valueColumnName, String metricsName) {
StringBuilder sql = new StringBuilder();
sql.append("select * from (select avg(")
.append(valueColumnName)
.append(") result,")
.append(Metrics.ENTITY_ID)
.append(" from ")
.append(metricsName)
.append(" where ");
sql.append("select result,")
.append(Metrics.ENTITY_ID)
.append(" from (select avg(")
.append(valueColumnName)
.append(") result,")
.append(Metrics.ENTITY_ID)
.append(" from ")
.append(metricsName)
.append(" where ");
return sql;
}
}

2
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2EventQueryDAO.java
View File

@ -36,7 +36,7 @@ import org.apache.skywalking.oap.server.core.query.type.event.EventQueryConditio
import org.apache.skywalking.oap.server.core.query.type.event.EventType;
import org.apache.skywalking.oap.server.core.query.type.event.Events;
import org.apache.skywalking.oap.server.core.query.type.event.Source;
import org.apache.skywalking.oap.server.core.source.Event;
import org.apache.skywalking.oap.server.core.analysis.metrics.Event;
import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;

2
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2LogQueryDAO.java
View File

@ -118,7 +118,7 @@ public class H2LogQueryDAO implements ILogQueryDAO {
}
if (StringUtil.isNotEmpty(serviceId)) {
sql.append(" and ").append(SERVICE_ID).append(" = ?");
sql.append(" and ").append(LogRecord.INDEX_NAME).append(".").append(SERVICE_ID).append(" = ?");
parameters.add(serviceId);
}
if (StringUtil.isNotEmpty(serviceInstanceId)) {

39
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2MetricsQueryDAO.java
View File

@ -52,6 +52,12 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
public long readMetricsValue(final MetricsCondition condition,
String valueColumnName,
final Duration duration) throws IOException {
final List<PointOfTime> pointOfTimes = duration.assembleDurationPoints();
List<String> ids = new ArrayList<>(pointOfTimes.size());
final String entityId = condition.getEntity().buildId();
pointOfTimes.forEach(pointOfTime -> {
ids.add(pointOfTime.id(entityId));
});
int defaultValue = ValueColumnMetadata.INSTANCE.getDefaultValue(condition.getName());
final Function function = ValueColumnMetadata.INSTANCE.getValueFunction(condition.getName());
if (function == Function.Latest) {
@ -66,15 +72,22 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
op = "sum";
}
StringBuilder sql = buildMetricsValueSql(op, valueColumnName, condition.getName());
final String entityId = condition.getEntity().buildId();
List<Object> parameters = new ArrayList();
if (entityId != null) {
sql.append(Metrics.ENTITY_ID + " = ? and ");
parameters.add(entityId);
}
sql.append(Metrics.TIME_BUCKET + ">= ? and " + Metrics.TIME_BUCKET + "<=?" + " group by " + Metrics.ENTITY_ID);
parameters.add(duration.getStartTimeBucket());
parameters.add(duration.getEndTimeBucket());
sql.append("id in (");
for (int i = 0; i < ids.size(); i++) {
if (i == 0) {
sql.append("?");
} else {
sql.append(",?");
}
parameters.add(ids.get(i));
}
sql.append(")");
sql.append(" group by " + Metrics.ENTITY_ID);
try (Connection connection = h2Client.getConnection()) {
try (ResultSet resultSet = h2Client.executeQuery(
@ -103,8 +116,9 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
final Duration duration) throws IOException {
final List<PointOfTime> pointOfTimes = duration.assembleDurationPoints();
List<String> ids = new ArrayList<>(pointOfTimes.size());
final String entityId = condition.getEntity().buildId();
pointOfTimes.forEach(pointOfTime -> {
ids.add(pointOfTime.id(condition.getEntity().buildId()));
ids.add(pointOfTime.id(entityId));
});
StringBuilder sql = new StringBuilder(
@ -120,6 +134,8 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
}
sql.append(")");
buildShardingCondition(sql, parameters, entityId);
MetricsValues metricsValues = new MetricsValues();
// Label is null, because in readMetricsValues, no label parameter.
final IntValues intValues = metricsValues.getValues();
@ -152,8 +168,9 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
final Duration duration) throws IOException {
final List<PointOfTime> pointOfTimes = duration.assembleDurationPoints();
List<String> ids = new ArrayList<>(pointOfTimes.size());
final String entityId = condition.getEntity().buildId();
pointOfTimes.forEach(pointOfTime -> {
ids.add(pointOfTime.id(condition.getEntity().buildId()));
ids.add(pointOfTime.id(entityId));
});
StringBuilder sql = new StringBuilder(
@ -170,6 +187,8 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
}
sql.append(")");
buildShardingCondition(sql, parameters, entityId);
Map<String, DataTable> idMap = new HashMap<>();
try (Connection connection = h2Client.getConnection()) {
try (ResultSet resultSet = h2Client.executeQuery(
@ -199,8 +218,9 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
final Duration duration) throws IOException {
final List<PointOfTime> pointOfTimes = duration.assembleDurationPoints();
List<String> ids = new ArrayList<>(pointOfTimes.size());
final String entityId = condition.getEntity().buildId();
pointOfTimes.forEach(pointOfTime -> {
ids.add(pointOfTime.id(condition.getEntity().buildId()));
ids.add(pointOfTime.id(entityId));
});
StringBuilder sql = new StringBuilder(
@ -216,6 +236,8 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
}
sql.append(")");
buildShardingCondition(sql, parameters, entityId);
final int defaultValue = ValueColumnMetadata.INSTANCE.getDefaultValue(condition.getName());
try (Connection connection = h2Client.getConnection()) {
@ -236,4 +258,7 @@ public class H2MetricsQueryDAO extends H2SQLExecutor implements IMetricsQueryDAO
throw new IOException(e);
}
}
protected void buildShardingCondition(StringBuilder sql, List<Object> parameters, String entityId) {
}
}

20
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2SQLExecutor.java
View File

@ -27,6 +27,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.UnexpectedException;
import org.apache.skywalking.oap.server.core.storage.StorageData;
import org.apache.skywalking.oap.server.core.storage.model.Model;
import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
@ -225,23 +226,30 @@ public class H2SQLExecutor {
storageBuilder.entity2Storage(metrics, toStorage);
Map<String, Object> objectMap = toStorage.obtain();
SQLBuilder sqlBuilder = new SQLBuilder("UPDATE " + modelName + " SET ");
List<ModelColumn> columns = TableMetaInfo.get(modelName).getColumns();
StringBuilder sqlBuilder = new StringBuilder("UPDATE " + modelName + " SET ");
Model model = TableMetaInfo.get(modelName);
List<ModelColumn> columns = model.getColumns();
List<Object> param = new ArrayList<>();
for (int i = 0; i < columns.size(); i++) {
ModelColumn column = columns.get(i);
sqlBuilder.append(column.getColumnName().getStorageName() + "= ?");
if (i != columns.size() - 1) {
sqlBuilder.append(",");
String columnName = column.getColumnName().getName();
if (model.getSqlDBModelExtension().isShardingTable()) {
SQLDatabaseModelExtension.Sharding sharding = model.getSqlDBModelExtension().getSharding().orElseThrow(
() -> new UnexpectedException("Sharding should not be empty."));
if (columnName.equals(sharding.getDataSourceShardingColumn()) || columnName.equals(sharding.getTableShardingColumn())) {
continue;
}
}
sqlBuilder.append(column.getColumnName().getStorageName()).append("= ?,");
Object value = objectMap.get(column.getColumnName().getName());
Object value = objectMap.get(columnName);
if (value instanceof StorageDataComplexObject) {
param.add(((StorageDataComplexObject) value).toStorageData());
} else {
param.add(value);
}
}
sqlBuilder.replace(sqlBuilder.length() - 1, sqlBuilder.length(), "");
sqlBuilder.append(" WHERE id = ?");
param.add(metrics.id());

5
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2TableInstaller.java
View File

@ -27,6 +27,7 @@ import java.util.List;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.analysis.Layer;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.storage.StorageException;
import org.apache.skywalking.oap.server.core.storage.model.Model;
import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
@ -46,7 +47,7 @@ import org.apache.skywalking.oap.server.storage.plugin.jdbc.TableMetaInfo;
*/
@Slf4j
public class H2TableInstaller extends ModelInstaller {
public static final String ID_COLUMN = "id";
public static final String ID_COLUMN = Metrics.ID;
public H2TableInstaller(Client client, ModuleManager moduleManager) {
super(client, moduleManager);
@ -128,7 +129,7 @@ public class H2TableInstaller extends ModelInstaller {
}
}
private void createTable(JDBCHikariCPClient client,
protected void createTable(JDBCHikariCPClient client,
Connection connection,
String tableName, List<ModelColumn> columns, boolean additionalTable) throws JDBCClientException {
SQLBuilder tableCreateSQL = new SQLBuilder("CREATE TABLE IF NOT EXISTS " + tableName + " (");

2
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2TraceQueryDAO.java
View File

@ -121,7 +121,7 @@ public class H2TraceQueryDAO implements ITraceQueryDAO {
parameters.add(maxDuration);
}
if (StringUtil.isNotEmpty(serviceId)) {
sql.append(" and ").append(SegmentRecord.SERVICE_ID).append(" = ?");
sql.append(" and ").append(SegmentRecord.INDEX_NAME).append(".").append(SegmentRecord.SERVICE_ID).append(" = ?");
parameters.add(serviceId);
}
if (StringUtil.isNotEmpty(serviceInstanceId)) {

6
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2ZipkinQueryDAO.java
View File

@ -144,7 +144,7 @@ public class H2ZipkinQueryDAO implements IZipkinQueryDAO {
StringBuilder sql = new StringBuilder();
List<Object> condition = new ArrayList<>(5);
List<Map.Entry<String, String>> annotations = new ArrayList<>(request.annotationQuery().entrySet());
sql.append("select ").append(ZipkinSpanRecord.TRACE_ID).append(", ")
sql.append("select ").append(ZipkinSpanRecord.INDEX_NAME).append(".").append(ZipkinSpanRecord.TRACE_ID).append(", ")
.append("max(").append(ZipkinSpanRecord.TIMESTAMP_MILLIS).append(")").append(" from ");
sql.append(ZipkinSpanRecord.INDEX_NAME);
/**
@ -168,6 +168,7 @@ public class H2ZipkinQueryDAO implements IZipkinQueryDAO {
sql.append(" and ");
sql.append(ZipkinSpanRecord.TIMESTAMP_MILLIS).append(" <= ?");
condition.add(endTimeMillis);
buildShardingCondition(sql, condition, startTimeMillis, endTimeMillis);
}
if (request.minDuration() != null) {
sql.append(" and ");
@ -319,4 +320,7 @@ public class H2ZipkinQueryDAO implements IZipkinQueryDAO {
}
return span.build();
}
protected void buildShardingCondition(StringBuilder sql, List<Object> parameters, long startTimeMillis, long endTimeMillis) {
}
}

49
oap-server/server-storage-plugin/storage-shardingsphere-plugin/pom.xml Normal file
View File

@ -0,0 +1,49 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
~
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>server-storage-plugin</artifactId>
<groupId>org.apache.skywalking</groupId>
<version>9.3.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>storage-shardingsphere-plugin</artifactId>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>server-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>library-client</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>storage-jdbc-hikaricp-plugin</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>

140
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/DurationWithinTTL.java Normal file
View File

@ -0,0 +1,140 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.UnexpectedException;
import org.apache.skywalking.oap.server.core.config.ConfigService;
import org.apache.skywalking.oap.server.core.query.DurationUtils;
import org.apache.skywalking.oap.server.core.query.enumeration.Step;
import org.apache.skywalking.oap.server.core.query.input.Duration;
import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
public enum DurationWithinTTL {
INSTANCE;
private static final DateTimeFormatter YYYYMMDD = DateTimeFormat.forPattern("yyyyMMdd");
private static final DateTimeFormatter YYYYMMDDHH = DateTimeFormat.forPattern("yyyyMMddHH");
private static final DateTimeFormatter YYYYMMDDHHMM = DateTimeFormat.forPattern("yyyyMMddHHmm");
private static final DateTimeFormatter YYYYMMDDHHMMSS = DateTimeFormat.forPattern("yyyyMMddHHmmss");
@Setter
private ConfigService configService;
/**
* @param duration Origin Duration instance
* @return new Duration instance and the time range within the Metrics TTL, if origin duration is null then return null.
*/
public Duration getMetricDurationWithinTTL(final Duration duration) {
if (duration == null) {
return null;
}
Duration durationWithinTTL = new Duration();
durationWithinTTL.setStart(DurationWithinTTL.INSTANCE.getTrimmedMetricStartTime(duration.getStep(), duration.getStart()));
durationWithinTTL.setEnd(DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(duration.getStep(), duration.getEnd()));
durationWithinTTL.setStep(duration.getStep());
return durationWithinTTL;
}
/**
* @param duration Origin Duration instance
* @return new Duration instance and the time range within the Record TTL, if origin duration is null then return null.
*/
public Duration getRecordDurationWithinTTL(final Duration duration) {
if (duration == null) {
return null;
}
Duration durationWithinTTL = new Duration();
durationWithinTTL.setStart(DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(duration.getStep(), duration.getStart()));
durationWithinTTL.setEnd(DurationWithinTTL.INSTANCE.getTrimmedRecordEndTime(duration.getStep(), duration.getEnd()));
durationWithinTTL.setStep(duration.getStep());
return durationWithinTTL;
}
// Trim the startStr according to the record TTL, for query
public String getTrimmedRecordStartTime(Step step, String startStr) {
if (configService == null) {
throw new UnexpectedException("ConfigService can not be null, should set ConfigService first.");
}
return trimStartTime(step, startStr, configService.getRecordDataTTL());
}
// Trim the startStr according to the metric TTL, for query
public String getTrimmedMetricStartTime(Step step, String startStr) {
if (configService == null) {
throw new UnexpectedException("ConfigService can not be null, should set ConfigService first.");
}
return trimStartTime(step, startStr, configService.getMetricsDataTTL());
}
// Trim the endStr according to the record TTL, for query
public String getTrimmedRecordEndTime(Step step, String endStr) {
return trimEndTime(step, endStr);
}
// Trim the endStr according to the metric TTL, for query
public String getTrimmedMetricEndTime(Step step, String endStr) {
return trimEndTime(step, endStr);
}
private String trimStartTime(Step step, String startStr, int ttl) {
long startDate = DurationUtils.INSTANCE.convertToTimeBucket(step, startStr);
long timeFloor = Long.parseLong(DateTime.now().plusDays(1 - ttl).toString("yyyyMMdd"));
switch (step) {
case DAY:
return YYYYMMDD.parseDateTime(String.valueOf(Math.max(timeFloor, startDate))).toString("yyyy-MM-dd");
case HOUR:
timeFloor = timeFloor * 100;
return YYYYMMDDHH.parseDateTime(String.valueOf(Math.max(timeFloor, startDate))).toString("yyyy-MM-dd HH");
case MINUTE:
timeFloor = timeFloor * 100 * 100;
return YYYYMMDDHHMM.parseDateTime(String.valueOf(Math.max(timeFloor, startDate))).toString("yyyy-MM-dd HHmm");
case SECOND:
timeFloor = timeFloor * 100 * 100 * 100;
return YYYYMMDDHHMMSS.parseDateTime(String.valueOf(Math.max(timeFloor, startDate))).toString("yyyy-MM-dd HHmmss");
}
throw new UnexpectedException("Unsupported step " + step.name());
}
// Trim the endStr according to the real date, for query
public String trimEndTime(Step step, String endStr) {
long endDate = DurationUtils.INSTANCE.convertToTimeBucket(step, endStr);
long timeCeiling = Long.parseLong(DateTime.now().toString("yyyyMMdd"));
switch (step) {
case DAY:
return YYYYMMDD.parseDateTime(String.valueOf(Math.min(endDate, timeCeiling))).toString("yyyy-MM-dd");
case HOUR:
timeCeiling = (timeCeiling * 100) + 23;
return YYYYMMDDHH.parseDateTime(String.valueOf(Math.min(endDate, timeCeiling))).toString("yyyy-MM-dd HH");
case MINUTE:
timeCeiling = ((timeCeiling * 100) + 23) * 100 + 59;
return YYYYMMDDHHMM.parseDateTime(String.valueOf(Math.min(endDate, timeCeiling))).toString("yyyy-MM-dd HHmm");
case SECOND:
timeCeiling = (((timeCeiling * 100) + 23) * 100 + 59) * 100 + 59;
return YYYYMMDDHHMMSS.parseDateTime(String.valueOf(Math.min(endDate, timeCeiling))).toString("yyyy-MM-dd HHmmss");
}
throw new UnexpectedException("Unsupported step " + step.name());
}
}

64
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/ShardingRule.java Normal file
View File

@ -0,0 +1,64 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere;
import lombok.Builder;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
@Getter
@Setter
@Builder(toBuilder = true)
public class ShardingRule {
private String operation;
private String table;
private String actualDataNodes;
private String actualDataSources;
private String databaseStrategyType;
private String databaseShardingColumn;
private String databaseShardingAlgorithmType;
private String databaseShardingAlgorithmProps;
private String tableStrategyType;
private String tableShardingColumn;
private String tableShardingAlgorithmType;
private String tableShardingAlgorithmProps;
private String keyGenerateColumn;
private String keyGeneratorType;
private String keyGeneratorProps;
public String toShardingRuleSQL() {
SQLBuilder ruleSQL = new SQLBuilder();
ruleSQL.append(operation);
ruleSQL.append(" SHARDING TABLE RULE ");
ruleSQL.append(table).appendLine("(");
ruleSQL.append("DATANODES(").append(actualDataNodes).appendLine("),");
ruleSQL.append("DATABASE_STRATEGY(TYPE=").append(databaseStrategyType).appendLine(",");
ruleSQL.append("SHARDING_COLUMN=").append(databaseShardingColumn).appendLine(",");
ruleSQL.append("SHARDING_ALGORITHM(TYPE(NAME=").append(databaseShardingAlgorithmType).appendLine(",");
ruleSQL.append("PROPERTIES(").append(databaseShardingAlgorithmProps);
ruleSQL.appendLine(")))),");
ruleSQL.append("TABLE_STRATEGY(TYPE=").append(tableStrategyType).appendLine(",");
ruleSQL.append("SHARDING_COLUMN=").append(tableShardingColumn).appendLine(",");
ruleSQL.append("SHARDING_ALGORITHM(TYPE(NAME=").append(tableShardingAlgorithmType).appendLine(",");
ruleSQL.append("PROPERTIES(").append(tableShardingAlgorithmProps);
ruleSQL.append(")))))");
return ruleSQL.toString();
}
}

50
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/dao/ShardingAggregationQueryDAO.java Normal file
View File

@ -0,0 +1,50 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao;
import java.io.IOException;
import java.util.List;
import org.apache.skywalking.oap.server.core.query.input.Duration;
import org.apache.skywalking.oap.server.core.query.input.TopNCondition;
import org.apache.skywalking.oap.server.core.query.type.KeyValue;
import org.apache.skywalking.oap.server.core.query.type.SelectedRecord;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2AggregationQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
public class ShardingAggregationQueryDAO extends H2AggregationQueryDAO {
public ShardingAggregationQueryDAO(JDBCHikariCPClient h2Client) {
super(h2Client);
}
@Override
public List<SelectedRecord> sortMetrics(final TopNCondition metrics,
final String valueColumnName,
final Duration duration,
List<KeyValue> additionalConditions) throws IOException {
return super.sortMetrics(
metrics,
valueColumnName,
DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration),
additionalConditions
);
}
}

52
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/dao/ShardingBrowserLogQueryDAO.java Normal file
View File

@ -0,0 +1,52 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao;
import java.io.IOException;
import org.apache.skywalking.oap.server.core.browser.source.BrowserErrorCategory;
import org.apache.skywalking.oap.server.core.query.input.Duration;
import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLogs;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BrowserLogQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
public class ShardingBrowserLogQueryDAO extends H2BrowserLogQueryDAO {
public ShardingBrowserLogQueryDAO(JDBCHikariCPClient h2Client) {
super(h2Client);
}
@Override
public BrowserErrorLogs queryBrowserErrorLogs(String serviceId,
String serviceVersionId,
String pagePathId,
BrowserErrorCategory category,
Duration duration,
int limit,
int from) throws IOException {
return super.queryBrowserErrorLogs(
serviceId,
serviceVersionId,
pagePathId,
category,
DurationWithinTTL.INSTANCE.getRecordDurationWithinTTL(duration),
limit,
from
);
}
}

66
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/dao/ShardingLogQueryDAO.java Normal file
View File

@ -0,0 +1,66 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao;
import java.io.IOException;
import java.util.List;
import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
import org.apache.skywalking.oap.server.core.query.enumeration.Order;
import org.apache.skywalking.oap.server.core.query.input.Duration;
import org.apache.skywalking.oap.server.core.query.input.TraceScopeCondition;
import org.apache.skywalking.oap.server.core.query.type.Logs;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2LogQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
public class ShardingLogQueryDAO extends H2LogQueryDAO {
public ShardingLogQueryDAO(final JDBCHikariCPClient h2Client,
final ModuleManager manager) {
super(h2Client, manager);
}
@Override
public Logs queryLogs(String serviceId,
String serviceInstanceId,
String endpointId,
TraceScopeCondition relatedTrace,
Order queryOrder,
int from,
int limit,
final Duration duration,
final List<Tag> tags,
final List<String> keywordsOfContent,
final List<String> excludingKeywordsOfContent) throws IOException {
return super.queryLogs(
serviceId,
serviceInstanceId,
endpointId,
relatedTrace,
queryOrder,
from,
limit,
DurationWithinTTL.INSTANCE.getRecordDurationWithinTTL(duration),
tags,
keywordsOfContent,
excludingKeywordsOfContent
);
}
}

102
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/dao/ShardingMetricsQueryDAO.java Normal file
View File

@ -0,0 +1,102 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.query.PointOfTime;
import org.apache.skywalking.oap.server.core.query.input.Duration;
import org.apache.skywalking.oap.server.core.query.input.MetricsCondition;
import org.apache.skywalking.oap.server.core.query.type.HeatMap;
import org.apache.skywalking.oap.server.core.query.type.MetricsValues;
import org.apache.skywalking.oap.server.core.storage.annotation.ValueColumnMetadata;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2MetricsQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
public class ShardingMetricsQueryDAO extends H2MetricsQueryDAO {
public ShardingMetricsQueryDAO(JDBCHikariCPClient h2Client) {
super(h2Client);
}
@Override
public long readMetricsValue(final MetricsCondition condition,
String valueColumnName,
final Duration duration) throws IOException {
return super.readMetricsValue(condition, valueColumnName, DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
}
@Override
public MetricsValues readMetricsValues(final MetricsCondition condition,
final String valueColumnName,
final Duration duration) throws IOException {
MetricsValues result = super.readMetricsValues(condition, valueColumnName,
DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
MetricsValues values = new MetricsValues();
values.setValues(Util.sortValues(result.getValues(), getOriginIds(condition, duration),
ValueColumnMetadata.INSTANCE.getDefaultValue(condition.getName())));
return values;
}
@Override
public List<MetricsValues> readLabeledMetricsValues(final MetricsCondition condition,
final String valueColumnName,
final List<String> labels,
final Duration duration) throws IOException {
List<MetricsValues> result = super.readLabeledMetricsValues(condition, valueColumnName, labels,
DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
return Util.sortValues(result, getOriginIds(condition, duration),
ValueColumnMetadata.INSTANCE.getDefaultValue(condition.getName()));
}
@Override
public HeatMap readHeatMap(final MetricsCondition condition,
final String valueColumnName,
final Duration duration) throws IOException {
HeatMap result = super.readHeatMap(condition, valueColumnName,
DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
result.fixMissingColumns(getOriginIds(condition, duration),
ValueColumnMetadata.INSTANCE.getDefaultValue(condition.getName()));
return result;
}
@Override
protected void buildShardingCondition(StringBuilder sql, List<Object> parameters, String entityId) {
sql.append(" and ");
sql.append(Metrics.ENTITY_ID + " = ?");
parameters.add(entityId);
}
private List<String> getOriginIds(final MetricsCondition condition, final Duration duration) {
final List<PointOfTime> pointOfTimes = duration.assembleDurationPoints();
List<String> ids = new ArrayList<>(pointOfTimes.size());
final String entityId = condition.getEntity().buildId();
pointOfTimes.forEach(pointOfTime -> {
ids.add(pointOfTime.id(entityId));
});
return ids;
}
}

91
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/dao/ShardingTopologyQueryDAO.java Normal file
View File

@ -0,0 +1,91 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao;
import java.io.IOException;
import java.util.List;
import org.apache.skywalking.oap.server.core.query.input.Duration;
import org.apache.skywalking.oap.server.core.query.type.Call;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2TopologyQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
public class ShardingTopologyQueryDAO extends H2TopologyQueryDAO {
public ShardingTopologyQueryDAO(JDBCHikariCPClient h2Client) {
super(h2Client);
}
@Override
public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(Duration duration,
List<String> serviceIds) throws IOException {
return super.loadServiceRelationsDetectedAtServerSide(DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
}
@Override
public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(Duration duration,
List<String> serviceIds) throws IOException {
return super.loadServiceRelationDetectedAtClientSide(DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration), serviceIds);
}
@Override
public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(Duration duration) throws IOException {
return super.loadServiceRelationsDetectedAtServerSide(DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
}
@Override
public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(Duration duration) throws IOException {
return super.loadServiceRelationDetectedAtClientSide(DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
}
@Override
public List<Call.CallDetail> loadInstanceRelationDetectedAtServerSide(String clientServiceId,
String serverServiceId,
Duration duration) throws IOException {
return super.loadInstanceRelationDetectedAtServerSide(clientServiceId, serverServiceId,
DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
}
@Override
public List<Call.CallDetail> loadInstanceRelationDetectedAtClientSide(String clientServiceId,
String serverServiceId,
Duration duration) throws IOException {
return super.loadInstanceRelationDetectedAtClientSide(clientServiceId, serverServiceId,
DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
}
@Override
public List<Call.CallDetail> loadEndpointRelation(Duration duration,
String destEndpointId) throws IOException {
return super.loadEndpointRelation(DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration),
destEndpointId);
}
@Override
public List<Call.CallDetail> loadProcessRelationDetectedAtClientSide(String serviceInstanceId, Duration duration) throws IOException {
return super.loadProcessRelationDetectedAtClientSide(serviceInstanceId,
DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
}
@Override
public List<Call.CallDetail> loadProcessRelationDetectedAtServerSide(String serviceInstanceId, Duration duration) throws IOException {
return super.loadProcessRelationDetectedAtServerSide(serviceInstanceId,
DurationWithinTTL.INSTANCE.getMetricDurationWithinTTL(duration));
}
}

68
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/dao/ShardingTraceQueryDAO.java Normal file
View File

@ -0,0 +1,68 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao;
import java.io.IOException;
import java.util.List;
import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
import org.apache.skywalking.oap.server.core.query.input.Duration;
import org.apache.skywalking.oap.server.core.query.type.QueryOrder;
import org.apache.skywalking.oap.server.core.query.type.TraceBrief;
import org.apache.skywalking.oap.server.core.query.type.TraceState;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2TraceQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
public class ShardingTraceQueryDAO extends H2TraceQueryDAO {
public ShardingTraceQueryDAO(ModuleManager manager,
JDBCHikariCPClient h2Client) {
super(manager, h2Client);
}
@Override
public TraceBrief queryBasicTraces(Duration duration,
long minDuration,
long maxDuration,
String serviceId,
String serviceInstanceId,
String endpointId,
String traceId,
int limit,
int from,
TraceState traceState,
QueryOrder queryOrder,
final List<Tag> tags) throws IOException {
return super.queryBasicTraces(
DurationWithinTTL.INSTANCE.getRecordDurationWithinTTL(duration),
minDuration,
maxDuration,
serviceId,
serviceInstanceId,
endpointId,
traceId,
limit,
from,
traceState,
queryOrder,
tags
);
}
}

53
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/dao/ShardingZipkinQueryDAO.java Normal file
View File

@ -0,0 +1,53 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao;
import java.io.IOException;
import java.util.List;
import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
import org.apache.skywalking.oap.server.core.query.input.Duration;
import org.apache.skywalking.oap.server.core.zipkin.ZipkinSpanRecord;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ZipkinQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
import zipkin2.Span;
import zipkin2.storage.QueryRequest;
public class ShardingZipkinQueryDAO extends H2ZipkinQueryDAO {
public ShardingZipkinQueryDAO(final JDBCHikariCPClient h2Client) {
super(h2Client);
}
@Override
public List<List<Span>> getTraces(final QueryRequest request, Duration duration) throws IOException {
return super.getTraces(request,
DurationWithinTTL.INSTANCE.getRecordDurationWithinTTL(duration));
}
@Override
protected void buildShardingCondition(StringBuilder sql, List<Object> parameters, long startTimeMillis, long endTimeMillis) {
sql.append(" and ");
sql.append(ZipkinSpanRecord.INDEX_NAME).append(".").append(ZipkinSpanRecord.TIME_BUCKET + " >= ?");
parameters.add(TimeBucket.getRecordTimeBucket(startTimeMillis));
sql.append(" and ");
sql.append(ZipkinSpanRecord.INDEX_NAME).append(".").append(ZipkinSpanRecord.TIME_BUCKET + " <= ?");
parameters.add(TimeBucket.getRecordTimeBucket(endTimeMillis));
}
}

147
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingHistoryDeleteDAO.java Normal file
View File

@ -0,0 +1,147 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
import java.io.IOException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.analysis.DownSampling;
import org.apache.skywalking.oap.server.core.storage.StorageException;
import org.apache.skywalking.oap.server.core.storage.model.Model;
import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2HistoryDeleteDAO;
import org.joda.time.DateTime;
@Slf4j
public class MySQLShardingHistoryDeleteDAO extends H2HistoryDeleteDAO {
private final JDBCHikariCPClient client;
private final MySQLShardingStorageConfig config;
private final ModuleManager manager;
private final Set<String> dataSources;
private final Map<String, Long> tableLatestSuccess;
public MySQLShardingHistoryDeleteDAO(JDBCHikariCPClient client, MySQLShardingStorageConfig config,
ModuleManager manager) {
super(client);
this.client = client;
this.config = config;
this.manager = manager;
this.dataSources = config.getDataSources();
this.tableLatestSuccess = new HashMap<>();
}
@Override
public void deleteHistory(Model model, String timeBucketColumnName, int ttl) throws IOException {
if (!model.isRecord()) {
if (!DownSampling.Minute.equals(model.getDownsampling())) {
return;
}
}
long deadline = Long.parseLong(DateTime.now().plusDays(-ttl).toString("yyyyMMdd"));
//If it's a sharding table drop expired tables
if (model.getSqlDBModelExtension().isShardingTable()) {
boolean isRuleExecuted = false;
Long latestSuccessDeadline = this.tableLatestSuccess.get(model.getName());
if (latestSuccessDeadline != null && deadline <= latestSuccessDeadline) {
if (log.isDebugEnabled()) {
log.debug("Table = {} already deleted, skip, deadline = {}, ttl = {}", model.getName(), deadline, ttl);
}
return;
}
try {
//refresh sharding rules
isRuleExecuted = ShardingRulesOperator.INSTANCE.createOrUpdateShardingRule(client, model, this.dataSources, ttl);
if (isRuleExecuted) {
MySQLShardingTableInstaller installer = new MySQLShardingTableInstaller(client, manager, config);
installer.createTable(model);
}
} catch (StorageException e) {
throw new IOException(e.getMessage(), e);
}
List<String> realTables = new ArrayList<>();
try (Connection connection = client.getConnection()) {
ResultSet resultSet = connection.getMetaData()
.getTables(connection.getCatalog(), null, model.getName() + "_20%", null);
while (resultSet.next()) {
realTables.add(resultSet.getString("TABLE_NAME"));
}
//delete additional tables
for (String additionalTable : model.getSqlDBModelExtension().getAdditionalTables().keySet()) {
ResultSet additionalTableRS = connection.getMetaData()
.getTables(connection.getCatalog(), null,
additionalTable + "_20%", null);
while (additionalTableRS.next()) {
realTables.add(additionalTableRS.getString("TABLE_NAME"));
}
}
} catch (JDBCClientException | SQLException e) {
throw new IOException(e.getMessage(), e);
}
List<String> prepareDeleteTables = new ArrayList<>();
for (String table : realTables) {
long timeSeries = isolateTimeFromTableName(table);
if (deadline >= timeSeries) {
prepareDeleteTables.add(table);
}
}
if (log.isDebugEnabled()) {
log.debug("Tables to be dropped: {}", prepareDeleteTables);
}
try (Connection connection = client.getConnection()) {
Set<String> dsList = config.getDataSources();
for (String prepareDeleteTable : prepareDeleteTables) {
for (String ds : dsList) {
client.execute(connection, getDropSQL(ds, prepareDeleteTable));
}
}
} catch (JDBCClientException | SQLException e) {
throw new IOException(e.getMessage(), e);
}
this.tableLatestSuccess.put(model.getName(), deadline);
} else {
//Delete rows as previous
super.deleteHistory(model, timeBucketColumnName, ttl);
}
}
private long isolateTimeFromTableName(String tableName) {
return Long.parseLong(tableName.substring(tableName.lastIndexOf("_") + 1));
}
private String getDropSQL(String dataSource, String table) {
String dropSQL = "/* ShardingSphere hint: dataSourceName=" + dataSource + "*/\n" +
"drop table if exists " + table;
return dropSQL;
}
}

39
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingStorageConfig.java Normal file
View File

@ -0,0 +1,39 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
import com.google.common.base.Splitter;
import java.util.HashSet;
import java.util.Set;
import lombok.Setter;
import org.apache.commons.lang3.StringUtils;
import org.apache.skywalking.oap.server.core.Const;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLStorageConfig;
@Setter
public class MySQLShardingStorageConfig extends MySQLStorageConfig {
private String dataSources = Const.EMPTY_STRING;
public Set<String> getDataSources() {
if (StringUtils.isEmpty(this.dataSources)) {
return null;
}
return new HashSet<>(Splitter.on(",").splitToList(this.dataSources));
}
}

185
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingStorageProvider.java Normal file
View File

@ -0,0 +1,185 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
import java.io.IOException;
import java.sql.SQLException;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.CoreModule;
import org.apache.skywalking.oap.server.core.config.ConfigService;
import org.apache.skywalking.oap.server.core.storage.IBatchDAO;
import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
import org.apache.skywalking.oap.server.core.storage.StorageDAO;
import org.apache.skywalking.oap.server.core.storage.StorageException;
import org.apache.skywalking.oap.server.core.storage.StorageModule;
import org.apache.skywalking.oap.server.core.storage.cache.INetworkAddressAliasDAO;
import org.apache.skywalking.oap.server.core.storage.management.UITemplateManagementDAO;
import org.apache.skywalking.oap.server.core.storage.model.ModelCreator;
import org.apache.skywalking.oap.server.core.storage.profiling.ebpf.IEBPFProfilingDataDAO;
import org.apache.skywalking.oap.server.core.storage.profiling.ebpf.IEBPFProfilingScheduleDAO;
import org.apache.skywalking.oap.server.core.storage.profiling.ebpf.IEBPFProfilingTaskDAO;
import org.apache.skywalking.oap.server.core.storage.profiling.ebpf.IServiceLabelDAO;
import org.apache.skywalking.oap.server.core.storage.profiling.trace.IProfileTaskLogQueryDAO;
import org.apache.skywalking.oap.server.core.storage.profiling.trace.IProfileTaskQueryDAO;
import org.apache.skywalking.oap.server.core.storage.profiling.trace.IProfileThreadSnapshotQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITagAutoCompleteQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITopNRecordsQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IZipkinQueryDAO;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.library.module.ModuleConfig;
import org.apache.skywalking.oap.server.library.module.ModuleDefine;
import org.apache.skywalking.oap.server.library.module.ModuleProvider;
import org.apache.skywalking.oap.server.library.module.ModuleStartException;
import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2EBPFProfilingDataDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2EBPFProfilingScheduleDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2EBPFProfilingTaskDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2EventQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2HistoryDeleteDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2MetadataQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2NetworkAddressAliasDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ServiceLabelQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ProfileTaskLogQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ProfileTaskQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ProfileThreadSnapshotQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2StorageDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2TagAutoCompleteQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2TopNRecordsQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2UITemplateManagementDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLAlarmQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao.ShardingAggregationQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao.ShardingMetricsQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao.ShardingTopologyQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao.ShardingZipkinQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.dao.MySQLShardingLogQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.dao.MySQLShardingTraceQueryDAO;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.dao.MysqlShardingBrowserLogQueryDAO;
@Slf4j
public class MySQLShardingStorageProvider extends ModuleProvider {
private MySQLShardingStorageConfig config;
private JDBCHikariCPClient mysqlClient;
public MySQLShardingStorageProvider() {
config = new MySQLShardingStorageConfig();
}
@Override
public String name() {
return "mysql-sharding";
}
@Override
public Class<? extends ModuleDefine> module() {
return StorageModule.class;
}
@Override
public ModuleConfig createConfigBeanIfAbsent() {
return config;
}
@Override
public void prepare() throws ServiceNotProvidedException {
this.registerServiceImplementation(StorageBuilderFactory.class, new StorageBuilderFactory.Default());
mysqlClient = new JDBCHikariCPClient(config.getProperties());
this.registerServiceImplementation(IBatchDAO.class, new H2BatchDAO(mysqlClient, config.getMaxSizeOfBatchSql(), config.getAsyncBatchPersistentPoolSize()));
this.registerServiceImplementation(
StorageDAO.class,
new H2StorageDAO(mysqlClient));
this.registerServiceImplementation(
INetworkAddressAliasDAO.class, new H2NetworkAddressAliasDAO(mysqlClient));
this.registerServiceImplementation(ITopologyQueryDAO.class, new ShardingTopologyQueryDAO(mysqlClient));
this.registerServiceImplementation(IMetricsQueryDAO.class, new ShardingMetricsQueryDAO(mysqlClient));
this.registerServiceImplementation(
ITraceQueryDAO.class,
new MySQLShardingTraceQueryDAO(getManager(), mysqlClient)
);
this.registerServiceImplementation(IBrowserLogQueryDAO.class, new MysqlShardingBrowserLogQueryDAO(mysqlClient));
this.registerServiceImplementation(
IMetadataQueryDAO.class, new H2MetadataQueryDAO(mysqlClient, config.getMetadataQueryMaxSize()));
this.registerServiceImplementation(IAggregationQueryDAO.class, new ShardingAggregationQueryDAO(mysqlClient));
this.registerServiceImplementation(IAlarmQueryDAO.class, new MySQLAlarmQueryDAO(mysqlClient, getManager()));
this.registerServiceImplementation(
IHistoryDeleteDAO.class, new H2HistoryDeleteDAO(mysqlClient));
this.registerServiceImplementation(ITopNRecordsQueryDAO.class, new H2TopNRecordsQueryDAO(mysqlClient));
this.registerServiceImplementation(
ILogQueryDAO.class,
new MySQLShardingLogQueryDAO(mysqlClient, getManager()));
this.registerServiceImplementation(IProfileTaskQueryDAO.class, new H2ProfileTaskQueryDAO(mysqlClient));
this.registerServiceImplementation(IProfileTaskLogQueryDAO.class, new H2ProfileTaskLogQueryDAO(mysqlClient));
this.registerServiceImplementation(
IProfileThreadSnapshotQueryDAO.class, new H2ProfileThreadSnapshotQueryDAO(mysqlClient));
this.registerServiceImplementation(UITemplateManagementDAO.class, new H2UITemplateManagementDAO(mysqlClient));
this.registerServiceImplementation(
IHistoryDeleteDAO.class, new MySQLShardingHistoryDeleteDAO(mysqlClient, config, getManager()));
this.registerServiceImplementation(IEventQueryDAO.class, new H2EventQueryDAO(mysqlClient));
this.registerServiceImplementation(IEBPFProfilingTaskDAO.class, new H2EBPFProfilingTaskDAO(mysqlClient));
this.registerServiceImplementation(IEBPFProfilingScheduleDAO.class, new H2EBPFProfilingScheduleDAO(mysqlClient));
this.registerServiceImplementation(IEBPFProfilingDataDAO.class, new H2EBPFProfilingDataDAO(mysqlClient));
this.registerServiceImplementation(IServiceLabelDAO.class, new H2ServiceLabelQueryDAO(mysqlClient));
this.registerServiceImplementation(ITagAutoCompleteQueryDAO.class, new H2TagAutoCompleteQueryDAO(mysqlClient));
this.registerServiceImplementation(IZipkinQueryDAO.class, new ShardingZipkinQueryDAO(mysqlClient));
}
@Override
public void start() throws ServiceNotProvidedException, ModuleStartException {
try {
mysqlClient.connect();
MySQLShardingTableInstaller installer = new MySQLShardingTableInstaller(mysqlClient, getManager(), config);
ShardingRulesOperator.INSTANCE.start(mysqlClient);
getManager().find(CoreModule.NAME).provider().getService(ModelCreator.class).addModelListener(installer);
DurationWithinTTL.INSTANCE.setConfigService(getManager().find(CoreModule.NAME)
.provider()
.getService(ConfigService.class));
} catch (StorageException | SQLException | IOException e) {
throw new ModuleStartException(e.getMessage(), e);
}
}
@Override
public void notifyAfterCompleted() throws ServiceNotProvidedException, ModuleStartException {
}
@Override
public String[] requiredModules() {
return new String[] {CoreModule.NAME};
}
}

157
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingTableInstaller.java Normal file
View File

@ -0,0 +1,157 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Arrays;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.CoreModule;
import org.apache.skywalking.oap.server.core.UnexpectedException;
import org.apache.skywalking.oap.server.core.config.ConfigService;
import org.apache.skywalking.oap.server.core.storage.StorageException;
import org.apache.skywalking.oap.server.core.storage.model.Model;
import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseExtension;
import org.apache.skywalking.oap.server.library.client.Client;
import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.TableMetaInfo;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLTableInstaller;
@Slf4j
public class MySQLShardingTableInstaller extends MySQLTableInstaller {
private final ConfigService configService;
private final Set<String> dataSources;
public MySQLShardingTableInstaller(Client client,
ModuleManager moduleManager,
MySQLShardingStorageConfig config) {
super(client, moduleManager);
this.configService = moduleManager.find(CoreModule.NAME).provider().getService(ConfigService.class);
this.dataSources = config.getDataSources();
}
@Override
protected boolean isExists(Model model) throws StorageException {
boolean isRuleExecuted = false;
boolean isTableExist = false;
if (configService == null) {
throw new UnexpectedException("ConfigService can not be null, should set ConfigService first.");
}
int ttl = model.isRecord() ? configService.getRecordDataTTL() : configService.getMetricsDataTTL();
TableMetaInfo.addModel(model);
JDBCHikariCPClient h2Client = (JDBCHikariCPClient) client;
try (Connection conn = h2Client.getConnection()) {
try (ResultSet rset = conn.getMetaData().getTables(conn.getCatalog(), null, model.getName(), null)) {
if (rset.next()) {
isTableExist = true;
}
}
} catch (SQLException | JDBCClientException e) {
throw new StorageException(e.getMessage(), e);
}
if (model.getSqlDBModelExtension().isShardingTable()) {
isRuleExecuted = ShardingRulesOperator.INSTANCE.createOrUpdateShardingRule(h2Client, model, this.dataSources, ttl);
}
return isTableExist && !isRuleExecuted;
}
@Override
protected void createTable(Model model) throws StorageException {
//todo: init
super.createTable(model);
}
@Override
protected void createTable(JDBCHikariCPClient client,
Connection connection,
String tableName,
List<ModelColumn> columns,
boolean additionalTable) throws JDBCClientException {
SQLBuilder tableCreateSQL = new SQLBuilder("CREATE TABLE IF NOT EXISTS " + tableName + " (");
tableCreateSQL.append(ID_COLUMN).append(" VARCHAR(512) ");
if (!additionalTable) {
tableCreateSQL.appendLine("PRIMARY KEY, ");
} else {
tableCreateSQL.appendLine(", ");
}
for (int i = 0; i < columns.size(); i++) {
ModelColumn column = columns.get(i);
tableCreateSQL.appendLine(
getColumn(column) + (i != columns.size() ? "," : ""));
}
int indexSeq = 0;
//Add indexes
List<String> columnList = columns.stream().map(column -> column.getColumnName().getStorageName()).collect(
Collectors.toList());
for (int i = 0; i < columns.size(); i++) {
ModelColumn column = columns.get(i);
for (final SQLDatabaseExtension.MultiColumnsIndex index : column.getSqlDatabaseExtension()
.getIndices()) {
final String[] multiColumns = index.getColumns();
//Create MultiColumnsIndex on the additional table only when it contains all need columns.
if (additionalTable && !columnList.containsAll(Arrays.asList(multiColumns))) {
continue;
}
tableCreateSQL.append("KEY K")
.append(String.valueOf(indexSeq++));
tableCreateSQL.append(" (");
for (int j = 0; j < multiColumns.length; j++) {
tableCreateSQL.append(multiColumns[j]);
if (j < multiColumns.length - 1) {
tableCreateSQL.append(",");
}
}
tableCreateSQL.appendLine("),");
}
if (column.shouldIndex() && column.getLength() < 513) {
tableCreateSQL.append("KEY K")
.append(String.valueOf(indexSeq++));
tableCreateSQL.append(" (").append(column.getColumnName().getStorageName()).append(")");
tableCreateSQL.appendLine(i != columns.size() - 1 ? "," : "");
}
}
tableCreateSQL.appendLine(")");
if (log.isDebugEnabled()) {
log.debug("creating table: " + tableCreateSQL.toStringInNewLine());
}
client.execute(connection, tableCreateSQL.toString());
}
@Override
protected void createTableIndexes(JDBCHikariCPClient client,
Connection connection,
String tableName,
List<ModelColumn> columns,
boolean additionalTable) throws JDBCClientException {
// Do nothing, indexes have been created inside create tables.
}
}

284
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/ShardingRulesOperator.java Normal file
View File

@ -0,0 +1,284 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
import java.io.IOException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.skywalking.oap.server.core.UnexpectedException;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.StorageException;
import org.apache.skywalking.oap.server.core.storage.model.Model;
import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseModelExtension;
import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.ShardingRule;
import org.joda.time.DateTime;
import org.joda.time.Days;
public enum ShardingRulesOperator {
INSTANCE;
private final Map<String, ShardingRule> modelShardingRules = new HashMap<>();
private static final String TIME_RELATIVE_ID_SHARDING_EXPRESSION =
"${long time_bucket = Long.parseLong(id.substring(0,id.indexOf('_')));" +
"if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;};" +
"if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);};" +
"if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);};" +
"if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);};" +
"}";
private static final String TIME_SEC_RANGE_SHARDING_EXPRESSION =
"\"datetime-pattern\"=\"yyyyMMddHHmmss\"," +
"\"datetime-interval-unit\"=\"days\"," +
"\"datetime-interval-amount\"=\"1\"," +
"\"sharding-suffix-pattern\"=\"yyyyMMdd\"," +
"\"datetime-lower\"=\"20220101000000\"," +
"\"datetime-upper\"=\"20991201000000\"";
private static final String TIME_MIN_RANGE_SHARDING_EXPRESSION =
"\"datetime-pattern\"=\"yyyyMMddHHmm\"," +
"\"datetime-interval-unit\"=\"days\"," +
"\"datetime-interval-amount\"=\"1\"," +
"\"sharding-suffix-pattern\"=\"yyyyMMdd\"," +
"\"datetime-lower\"=\"202201010000\"," +
"\"datetime-upper\"=\"209912010000\"";
private static final String TIME_BUCKET_SHARDING_EXPRESSION =
"${" +
"if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;};" +
"if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);};" +
"if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);};" +
"if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);};" +
"}";
public void start(JDBCHikariCPClient client) throws IOException, SQLException, StorageException {
initShardingRules(client);
}
boolean createOrUpdateShardingRule(JDBCHikariCPClient client, Model model, Set<String> dataSources, int ttl) throws StorageException {
boolean isExecuted;
ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
builder.table(model.getName());
String tableName = model.getName();
SQLDatabaseModelExtension.Sharding sharding = model.getSqlDBModelExtension().getSharding().orElseThrow(
() -> new UnexpectedException("Sharding should not be empty."));
isExecuted = executeShardingRule(
buildShardingRule(builder, tableName, dataSources, sharding.getShardingAlgorithm(),
sharding.getTableShardingColumn(),
sharding.getDataSourceShardingColumn(),
ttl,
DateTime.now()
),
client,
tableName
);
// additional tables
for (String additionalTable : model.getSqlDBModelExtension().getAdditionalTables().keySet()) {
ShardingRule.ShardingRuleBuilder additionalBuilder = ShardingRule.builder();
additionalBuilder.table(additionalTable);
isExecuted = executeShardingRule(
buildShardingRule(additionalBuilder, additionalTable, dataSources, sharding.getShardingAlgorithm(),
sharding.getTableShardingColumn(),
sharding.getDataSourceShardingColumn(), ttl, DateTime.now()
), client, additionalTable
);
}
return isExecuted;
}
private void initShardingRules(JDBCHikariCPClient client) throws IOException {
SQLBuilder sql = new SQLBuilder("SHOW SHARDING TABLE RULES");
try (Connection connection = client.getConnection()) {
ResultSet resultSet = client.executeQuery(connection, sql.toString());
while (resultSet.next()) {
ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
builder.table(resultSet.getString("TABLE"));
builder.actualDataNodes(resultSet.getString("ACTUAL_DATA_NODES"));
builder.actualDataSources(resultSet.getString("ACTUAL_DATA_SOURCES"));
// todo: shardingsphere-5.1.2 response DATABASE_STRATEGY and TABLE_STRATEGY type "inline" but should "standard"
builder.databaseStrategyType("standard");
builder.databaseShardingColumn(resultSet.getString("DATABASE_SHARDING_COLUMN"));
builder.databaseShardingAlgorithmType(resultSet.getString("DATABASE_SHARDING_ALGORITHM_TYPE"));
builder.databaseShardingAlgorithmProps(resultSet.getString("DATABASE_SHARDING_ALGORITHM_PROPS"));
// todo: shardingsphere-5.1.2 response DATABASE_STRATEGY and TABLE_STRATEGY type "inline" but should "standard"
builder.tableStrategyType("standard");
builder.tableShardingColumn(resultSet.getString("TABLE_SHARDING_COLUMN"));
builder.tableShardingAlgorithmType(resultSet.getString("TABLE_SHARDING_ALGORITHM_TYPE"));
builder.tableShardingAlgorithmProps(resultSet.getString("TABLE_SHARDING_ALGORITHM_PROPS"));
builder.keyGenerateColumn(resultSet.getString("KEY_GENERATE_COLUMN"));
builder.keyGeneratorType(resultSet.getString("KEY_GENERATOR_TYPE"));
builder.keyGeneratorProps(resultSet.getString("KEY_GENERATOR_PROPS"));
ShardingRule shardingRule = builder.build();
modelShardingRules.put(shardingRule.getTable(), shardingRule);
}
} catch (JDBCClientException | SQLException e) {
throw new IOException(e.getMessage(), e);
}
}
private void registerShardingRule(String tableName, ShardingRule rule) {
modelShardingRules.put(tableName, rule);
}
private ShardingRule.ShardingRuleBuilder buildShardingRule(ShardingRule.ShardingRuleBuilder builder,
String tableName,
Set<String> dataSources,
ShardingAlgorithm shardingAlgorithm,
String tableShardingColumn,
String dsShardingColumn,
int ttl,
DateTime currentDate) {
buildDataNodes(builder, tableName, dataSources, ttl, currentDate);
buildDatabaseStrategy(builder, dsShardingColumn, dataSources.size());
switch (shardingAlgorithm) {
case TIME_SEC_RANGE_SHARDING_ALGORITHM:
buildTimeRangeTableStrategy(builder, tableShardingColumn, TIME_SEC_RANGE_SHARDING_EXPRESSION);
break;
case TIME_MIN_RANGE_SHARDING_ALGORITHM:
buildTimeRangeTableStrategy(builder, tableShardingColumn, TIME_MIN_RANGE_SHARDING_EXPRESSION);
break;
case TIME_RELATIVE_ID_SHARDING_ALGORITHM:
buildExpressionTableStrategy(builder, tableName, tableShardingColumn,
TIME_RELATIVE_ID_SHARDING_EXPRESSION);
break;
case TIME_BUCKET_SHARDING_ALGORITHM:
buildExpressionTableStrategy(builder, tableName, tableShardingColumn, TIME_BUCKET_SHARDING_EXPRESSION);
break;
default:
throw new UnexpectedException("Unsupported sharding algorithm " + shardingAlgorithm);
}
return builder;
}
private void buildDataNodes(ShardingRule.ShardingRuleBuilder builder,
String tableName,
Set<String> dataSources,
int ttl,
DateTime currentDate) {
DateTime startDateTime = currentDate.plusDays(1 - ttl);
DateTime endDate = currentDate.plusDays(1);
int days = Days.daysBetween(startDateTime, endDate).getDays();
List<DateTime> timeRanges = new ArrayList<>();
StringBuilder nodesBuilder = new StringBuilder();
for (int i = 0; i <= days; i++) {
timeRanges.add(startDateTime.plusDays(i));
}
dataSources.forEach(dataSource -> {
timeRanges.forEach(dateTime -> {
nodesBuilder.append("\"")
.append(dataSource)
.append(".")
.append(tableName)
.append("_")
.append(dateTime.toString("yyyyMMdd"))
.append("\"")
.append(",");
});
});
builder.actualDataNodes(nodesBuilder.substring(0, nodesBuilder.length() - 1));
}
private void buildDatabaseStrategy(ShardingRule.ShardingRuleBuilder builder,
String shardingColumn,
int dsCount) {
StringBuilder propsBuilder = new StringBuilder();
propsBuilder.append("\"algorithm-expression\"=\"ds_${")
.append(shardingColumn)
.append(".hashCode()&Integer.MAX_VALUE%")
.append(dsCount)
.append("}\"");
builder.databaseStrategyType("\"standard\"")
.databaseShardingColumn(shardingColumn)
.databaseShardingAlgorithmType("\"inline\"")
.databaseShardingAlgorithmProps(propsBuilder.toString());
}
private void buildTimeRangeTableStrategy(ShardingRule.ShardingRuleBuilder builder,
String tableShardingColumn, String algorithmExpression) {
builder.tableStrategyType("\"standard\"")
.tableShardingColumn(tableShardingColumn)
.tableShardingAlgorithmType("\"interval\"")
.tableShardingAlgorithmProps(algorithmExpression);
}
private void buildExpressionTableStrategy(ShardingRule.ShardingRuleBuilder builder,
String tableName,
String tableShardingColumn,
String algorithmExpression
) {
StringBuilder propsBuilder = new StringBuilder();
propsBuilder.append("\"allow-range-query-with-inline-sharding\"=\"true\",")
.append("\"algorithm-expression\"=\"")
.append(tableName)
.append("_")
.append(algorithmExpression)
.append("\"");
builder.tableStrategyType("\"standard\"")
.tableShardingColumn(tableShardingColumn)
.tableShardingAlgorithmType("\"inline\"")
.tableShardingAlgorithmProps(propsBuilder.toString());
}
private boolean executeShardingRule(ShardingRule.ShardingRuleBuilder builder,
JDBCHikariCPClient client,
String tableName) throws StorageException {
ShardingRule existRule = modelShardingRules.get(tableName);
ShardingRule shardingRule;
String shardingRuleSQL;
if (existRule == null) {
builder.operation("CREATE");
shardingRule = builder.build();
shardingRuleSQL = shardingRule.toShardingRuleSQL();
} else {
builder.operation("ALTER");
existRule.setOperation("ALTER");
shardingRule = builder.build();
shardingRuleSQL = shardingRule.toShardingRuleSQL();
if (existRule.toShardingRuleSQL().equals(shardingRuleSQL.replaceAll("\"", ""))) {
return false;
}
}
try (Connection connection = client.getConnection()) {
SQLBuilder ruleSQL = new SQLBuilder(shardingRuleSQL);
client.execute(connection, ruleSQL.toString());
registerShardingRule(tableName, shardingRule);
} catch (JDBCClientException | SQLException e) {
throw new StorageException(e.getMessage(), e);
}
return true;
}
}

35
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/dao/MySQLShardingLogQueryDAO.java Normal file
View File

@ -0,0 +1,35 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.dao;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao.ShardingLogQueryDAO;
public class MySQLShardingLogQueryDAO extends ShardingLogQueryDAO {
public MySQLShardingLogQueryDAO(final JDBCHikariCPClient h2Client, final ModuleManager manager) {
super(h2Client, manager);
}
@Override
protected void buildLimit(StringBuilder sql, int from, int limit) {
sql.append(" LIMIT ").append(from).append(", ").append(limit);
}
}

36
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/dao/MySQLShardingTraceQueryDAO.java Normal file
View File

@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.dao;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao.ShardingTraceQueryDAO;
public class MySQLShardingTraceQueryDAO extends ShardingTraceQueryDAO {
public MySQLShardingTraceQueryDAO(ModuleManager manager,
JDBCHikariCPClient h2Client) {
super(manager, h2Client);
}
@Override
protected void buildLimit(StringBuilder sql, int from, int limit) {
sql.append(" LIMIT ").append(from).append(", ").append(limit);
}
}

32
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/dao/MysqlShardingBrowserLogQueryDAO.java Normal file
View File

@ -0,0 +1,32 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.dao;
import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao.ShardingBrowserLogQueryDAO;
public class MysqlShardingBrowserLogQueryDAO extends ShardingBrowserLogQueryDAO {
public MysqlShardingBrowserLogQueryDAO(final JDBCHikariCPClient h2Client) {
super(h2Client);
}
@Override
protected void buildLimit(StringBuilder sql, int from, int limit) {
sql.append(" limit ").append(from).append(", ").append(limit);
}
}

19
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider Normal file
View File

@ -0,0 +1,19 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#
org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.MySQLShardingStorageProvider

129
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/DurationWithinTTLTest.java Normal file
View File

@ -0,0 +1,129 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
import org.apache.skywalking.oap.server.core.config.ConfigService;
import org.apache.skywalking.oap.server.core.query.enumeration.Step;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
import org.joda.time.DateTime;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.powermock.api.mockito.PowerMockito;
import static org.powermock.api.mockito.PowerMockito.when;
public class DurationWithinTTLTest {
private final int recordDataTTL = 3;
@Before
public void init() {
ConfigService configService = PowerMockito.mock(ConfigService.class);
when(configService.getRecordDataTTL()).thenReturn(recordDataTTL);
DurationWithinTTL.INSTANCE.setConfigService(configService);
}
@Test
public void testGetTrimmedRecordStartTime() {
String expectedDay = new DateTime().toString("yyyy-MM-dd");
String inputDay = new DateTime().toString("yyyy-MM-dd");
Assert.assertEquals(expectedDay, DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.DAY, inputDay));
String expectedDayOutTTL = new DateTime().plusDays(1 - recordDataTTL).toString("yyyy-MM-dd");
String inputDayOutTTL = new DateTime().plusDays(-recordDataTTL).toString("yyyy-MM-dd");
Assert.assertEquals(
expectedDayOutTTL, DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.DAY, inputDayOutTTL));
String expectedHour = new DateTime().toString("yyyy-MM-dd HH");
String inputHour = new DateTime().toString("yyyy-MM-dd HH");
Assert.assertEquals(expectedHour, DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.HOUR, inputHour));
String inputHourOutTTL = new DateTime().plusDays(-recordDataTTL).toString("yyyy-MM-dd HH");
Assert.assertEquals(
expectedDayOutTTL + " 00",
DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.HOUR, inputHourOutTTL)
);
String expectedMin = new DateTime().toString("yyyy-MM-dd HHmm");
String inputMin = new DateTime().toString("yyyy-MM-dd HHmm");
Assert.assertEquals(expectedMin, DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.MINUTE, inputMin));
String inputMinOutTTL = new DateTime().plusDays(-recordDataTTL).toString("yyyy-MM-dd HHmm");
Assert.assertEquals(
expectedDayOutTTL + " 0000",
DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.MINUTE, inputMinOutTTL)
);
String inputSec = new DateTime().toString("yyyy-MM-dd HHmmss");
String expectedSec = new DateTime().toString("yyyy-MM-dd HHmmss");
Assert.assertEquals(expectedSec, DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.SECOND, inputSec));
String inputSecOutTTL = new DateTime().plusDays(-recordDataTTL).toString("yyyy-MM-dd HHmmss");
Assert.assertEquals(
expectedDayOutTTL + " 000000",
DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.SECOND, inputSecOutTTL)
);
try {
DurationWithinTTL.INSTANCE.getTrimmedRecordStartTime(Step.HOUR, "2022-09-08 30");
Assert.fail("Should throw IllegalArgumentException");
} catch (IllegalArgumentException e) {
Assert.assertTrue(true);
}
}
@Test
public void testGetTrimmedMetricEndTime() {
String expectedDay = new DateTime().toString("yyyy-MM-dd");
String inputDay = new DateTime().toString("yyyy-MM-dd");
Assert.assertEquals(expectedDay, DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.DAY, inputDay));
String expectedDayOutTTL = new DateTime().toString("yyyy-MM-dd");
String inputDayOutTTL = new DateTime().plusDays(1).toString("yyyy-MM-dd");
Assert.assertEquals(
expectedDayOutTTL, DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.DAY, inputDayOutTTL));
String expectedHour = new DateTime().toString("yyyy-MM-dd HH");
String inputHour = new DateTime().toString("yyyy-MM-dd HH");
Assert.assertEquals(expectedHour, DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.HOUR, inputHour));
String inputHourOutTTL = new DateTime().plusDays(1).toString("yyyy-MM-dd HH");
Assert.assertEquals(
expectedDayOutTTL + " 23",
DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.HOUR, inputHourOutTTL)
);
String expectedMin = new DateTime().toString("yyyy-MM-dd HHmm");
String inputMin = new DateTime().toString("yyyy-MM-dd HHmm");
Assert.assertEquals(expectedMin, DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.MINUTE, inputMin));
String inputMinOutTTL = new DateTime().plusDays(1).toString("yyyy-MM-dd HHmm");
Assert.assertEquals(
expectedDayOutTTL + " 2359",
DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.MINUTE, inputMinOutTTL)
);
String inputSec = new DateTime().toString("yyyy-MM-dd HHmmss");
String expectedSec = new DateTime().toString("yyyy-MM-dd HHmmss");
Assert.assertEquals(expectedSec, DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.SECOND, inputSec));
String inputSecOutTTL = new DateTime().plusDays(1).toString("yyyy-MM-dd HHmmss");
Assert.assertEquals(
expectedDayOutTTL + " 235959",
DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.SECOND, inputSecOutTTL)
);
try {
DurationWithinTTL.INSTANCE.getTrimmedMetricEndTime(Step.HOUR, "2022-09-08 30");
Assert.fail("Should throw IllegalArgumentException");
} catch (IllegalArgumentException e) {
Assert.assertTrue(true);
}
}
}

102
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/ShardingRulesTest.java Normal file
View File

@ -0,0 +1,102 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
import java.util.Arrays;
import java.util.HashSet;
import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.ShardingRule;
import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.powermock.reflect.Whitebox;
public class ShardingRulesTest {
private DateTime currentDate;
@Before
public void init() {
currentDate = DateTimeFormat.forPattern("yyyyMMdd").parseDateTime("20220919");
}
@Test
public void buildTimeRangeShardingRuleTest() throws Exception {
ShardingRulesOperator rulesOperator = ShardingRulesOperator.INSTANCE;
ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
builder.table("Test_table");
builder.operation("CREATE");
Whitebox.invokeMethod(rulesOperator, "buildShardingRule",
builder,
"Test_table",
new HashSet<>(Arrays.asList("ds_0", "ds_1")),
ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM,
"time_bucket",
"service_id",
3,
currentDate);
String rule = builder.build().toShardingRuleSQL();
String expectedRule = "CREATE SHARDING TABLE RULE Test_table(" + System.lineSeparator() +
"DATANODES(\"ds_1.Test_table_20220917\",\"ds_1.Test_table_20220918\",\"ds_1.Test_table_20220919\",\"ds_1.Test_table_20220920\",\"ds_0.Test_table_20220917\",\"ds_0.Test_table_20220918\",\"ds_0.Test_table_20220919\",\"ds_0.Test_table_20220920\")," + System.lineSeparator() +
"DATABASE_STRATEGY(TYPE=\"standard\"," + System.lineSeparator() +
"SHARDING_COLUMN=service_id," + System.lineSeparator() +
"SHARDING_ALGORITHM(TYPE(NAME=\"inline\"," + System.lineSeparator() +
"PROPERTIES(\"algorithm-expression\"=\"ds_${service_id.hashCode()&Integer.MAX_VALUE%2}\"))))," + System.lineSeparator() +
"TABLE_STRATEGY(TYPE=\"standard\"," + System.lineSeparator() +
"SHARDING_COLUMN=time_bucket," + System.lineSeparator() +
"SHARDING_ALGORITHM(TYPE(NAME=\"interval\"," + System.lineSeparator() +
"PROPERTIES(\"datetime-pattern\"=\"yyyyMMddHHmmss\",\"datetime-interval-unit\"=\"days\",\"datetime-interval-amount\"=\"1\",\"sharding-suffix-pattern\"=\"yyyyMMdd\",\"datetime-lower\"=\"20220101000000\",\"datetime-upper\"=\"20991201000000\")))))";
Assert.assertEquals(expectedRule, rule);
}
@Test
public void buildTimeRelativeIDShardingRuleTest() throws Exception {
ShardingRulesOperator rulesOperator = ShardingRulesOperator.INSTANCE;
ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
builder.table("Test_table");
builder.operation("CREATE");
Whitebox.invokeMethod(rulesOperator, "buildShardingRule",
builder,
"Test_table",
new HashSet<>(Arrays.asList("ds_0", "ds_1")),
ShardingAlgorithm.TIME_RELATIVE_ID_SHARDING_ALGORITHM,
"id",
"entity_id",
3,
currentDate);
String rule = builder.build().toShardingRuleSQL();
String expectedRule = "CREATE SHARDING TABLE RULE Test_table(" + System.lineSeparator() +
"DATANODES(\"ds_1.Test_table_20220917\",\"ds_1.Test_table_20220918\",\"ds_1.Test_table_20220919\",\"ds_1.Test_table_20220920\",\"ds_0.Test_table_20220917\",\"ds_0.Test_table_20220918\",\"ds_0.Test_table_20220919\",\"ds_0.Test_table_20220920\")," + System.lineSeparator() +
"DATABASE_STRATEGY(TYPE=\"standard\"," + System.lineSeparator() +
"SHARDING_COLUMN=entity_id," + System.lineSeparator() +
"SHARDING_ALGORITHM(TYPE(NAME=\"inline\"," + System.lineSeparator() +
"PROPERTIES(\"algorithm-expression\"=\"ds_${entity_id.hashCode()&Integer.MAX_VALUE%2}\"))))," + System.lineSeparator() +
"TABLE_STRATEGY(TYPE=\"standard\"," + System.lineSeparator() +
"SHARDING_COLUMN=id," + System.lineSeparator() +
"SHARDING_ALGORITHM(TYPE(NAME=\"inline\"," + System.lineSeparator() +
"PROPERTIES(\"allow-range-query-with-inline-sharding\"=\"true\",\"algorithm-expression\"=\"Test_table_${long time_bucket = Long.parseLong(id.substring(0,id.indexOf('_')));" +
"if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;};" +
"if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);};" +
"if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);};" +
"if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);};" +
"}\")))))";
Assert.assertEquals(expectedRule, rule);
}
}

2
test/e2e-v2/cases/alarm/mysql/docker-compose.yml
View File

@ -39,7 +39,7 @@ services:
environment:
SW_STORAGE: mysql
SW_SEARCHABLE_ALARM_TAG_KEYS: level,receivers
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
ports:
- 12800
depends_on:

2
test/e2e-v2/cases/browser/docker-compose.yml
View File

@ -30,7 +30,7 @@ services:
context: ./
dockerfile: docker/Dockerfile.provider
args:
- SW_AGENT_PYTHON_COMMIT=c8479000eb729cc86509222fd48b942edcaaca74
- SW_AGENT_PYTHON_COMMIT=${SW_AGENT_PYTHON_COMMIT}
command: [ 'python3', '/entrypoint.py' ]
networks:
- e2e

4
test/e2e-v2/cases/browser/es/docker-compose.yml
View File

@ -49,7 +49,7 @@ services:
context: ./../
dockerfile: docker/Dockerfile.provider
args:
- SW_AGENT_PYTHON_COMMIT=c8479000eb729cc86509222fd48b942edcaaca74
- SW_AGENT_PYTHON_COMMIT=${SW_AGENT_PYTHON_COMMIT}
command: [ 'python3', '/entrypoint.py' ]
networks:
- e2e
@ -133,4 +133,4 @@ services:
condition: service_started
networks:
e2e:
e2e:

2
test/e2e-v2/cases/browser/es/es-sharding/docker-compose.yml
View File

@ -50,7 +50,7 @@ services:
context: ./../../
dockerfile: docker/Dockerfile.provider
args:
- SW_AGENT_PYTHON_COMMIT=c8479000eb729cc86509222fd48b942edcaaca74
- SW_AGENT_PYTHON_COMMIT=${SW_AGENT_PYTHON_COMMIT}
command: [ 'python3', '/entrypoint.py' ]
networks:
- e2e

163
test/e2e-v2/cases/browser/mysql/sharding/docker-compose.yml Normal file
View File

@ -0,0 +1,163 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: '2.1'
services:
sharding-proxy:
extends:
file: ../../../../script/shardingsphere-proxy/base-compose.yml
service: sharding-proxy
networks:
- e2e
expose:
- 3307
depends_on:
mysql-1:
condition: service_healthy
mysql-2:
condition: service_healthy
mysql-1: &mysql
image: mysql/mysql-server:8.0.13
networks:
- e2e
expose:
- 3306
environment:
MYSQL_ROOT_PASSWORD: "root@1234"
MYSQL_DATABASE: "swtest"
MYSQL_ROOT_HOST: "%"
healthcheck:
test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
interval: 5s
timeout: 60s
retries: 120
mysql-2:
<<: *mysql
oap:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: oap
environment:
SW_STORAGE: mysql-sharding
SW_JDBC_URL: jdbc:mysql://sharding-proxy:3307/swtest
SW_JDBC_SHARDING_DATA_SOURCES: ds_0,ds_1
SW_CORE_METRICS_DATA_TTL: 2
SW_CORE_RECORD_DATA_TTL: 2
SW_OTEL_RECEIVER_ENABLED_OTEL_RULES: ""
SW_METER_ANALYZER_ACTIVE_FILES: ""
ports:
- 12800
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
networks:
- e2e
depends_on:
sharding-proxy:
condition: service_healthy
provider:
build:
context: ./../../
dockerfile: docker/Dockerfile.provider
args:
- SW_AGENT_PYTHON_COMMIT=${SW_AGENT_PYTHON_COMMIT}
command: [ 'python3', '/entrypoint.py' ]
networks:
- e2e
expose:
- 9091
environment:
SW_AGENT_COLLECTOR_BACKEND_SERVICES: oap:11800
SW_AGENT_LOGGING_LEVEL: DEBUG
volumes:
- ./../../docker/provider.py:/entrypoint.py
depends_on:
oap:
condition: service_healthy
healthcheck:
test: [ "CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9091" ]
interval: 5s
timeout: 60s
retries: 120
testui:
build:
context: ./../../
dockerfile: docker/Dockerfile.test-ui
args:
- SW_AGENT_CLIENT_JS_TEST_COMMIT=${SW_AGENT_CLIENT_JS_TEST_COMMIT}
- SW_AGENT_CLIENT_JS_COMMIT=${SW_AGENT_CLIENT_JS_COMMIT}
networks:
- e2e
ports:
- 80
depends_on:
provider:
condition: service_healthy
oap:
condition: service_healthy
selenium-hub:
image: selenium/hub:4.0.0-alpha-7-prerelease-20201009
networks:
- e2e
expose:
- 4444
depends_on:
testui:
condition: service_started
provider:
condition: service_healthy
chrome:
image: selenium/node-chrome:4.0.0-alpha-7-prerelease-20201009
networks:
- e2e
volumes:
- /dev/shm:/dev/shm
depends_on:
selenium-hub:
condition: service_started
testui:
condition: service_started
provider:
condition: service_healthy
environment:
- SE_EVENT_BUS_HOST=selenium-hub
- SE_EVENT_BUS_PUBLISH_PORT=4442
- SE_EVENT_BUS_SUBSCRIBE_PORT=4443
generate-traffic:
build:
context: ./../../
dockerfile: docker/Dockerfile.generate-traffic
networks:
- e2e
restart: always
depends_on:
selenium-hub:
condition: service_started
chrome:
condition: service_started
provider:
condition: service_healthy
testui:
condition: service_started
networks:
e2e:

40
test/e2e-v2/cases/browser/mysql/sharding/e2e.yaml Normal file
View File

@ -0,0 +1,40 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file is used to show how to write configuration files and can be used to test.
setup:
env: compose
file: docker-compose.yml
timeout: 20m
init-system-environment: ../../../../script/env
steps:
- name: set PATH
command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- name: install yq
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- name: install swctl
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
verify:
# verify with retry strategy
retry:
# max retry count
count: 20
# the interval between two retries, in millisecond.
interval: 3s
cases:
- includes:
- ../../browser-cases.yaml

2
test/e2e-v2/cases/event/mysql/docker-compose.yml
View File

@ -41,7 +41,7 @@ services:
depends_on:
mysql:
condition: service_healthy
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
ports:
- 11800
- 12800

2
test/e2e-v2/cases/log/mysql/docker-compose.yml
View File

@ -41,7 +41,7 @@ services:
SW_JDBC_URL: jdbc:mysql://mysql:3306/swtest
ports:
- 12800
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
networks:
- e2e
depends_on:

88
test/e2e-v2/cases/log/mysql/sharding/docker-compose.yml Normal file
View File

@ -0,0 +1,88 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: '2.1'
services:
sharding-proxy:
extends:
file: ../../../../script/shardingsphere-proxy/base-compose.yml
service: sharding-proxy
networks:
- e2e
expose:
- 3307
depends_on:
mysql-1:
condition: service_healthy
mysql-2:
condition: service_healthy
mysql-1: &mysql
image: mysql/mysql-server:8.0.13
networks:
- e2e
expose:
- 3306
environment:
MYSQL_ROOT_PASSWORD: "root@1234"
MYSQL_DATABASE: "swtest"
MYSQL_ROOT_HOST: "%"
healthcheck:
test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
interval: 5s
timeout: 60s
retries: 120
mysql-2:
<<: *mysql
oap:
extends:
file: ../../log-base-compose.yml
service: oap
environment:
SW_STORAGE: mysql-sharding
SW_JDBC_URL: jdbc:mysql://sharding-proxy:3307/swtest
SW_JDBC_SHARDING_DATA_SOURCES: ds_0,ds_1
SW_CORE_METRICS_DATA_TTL: 2
SW_CORE_RECORD_DATA_TTL: 2
SW_OTEL_RECEIVER_ENABLED_OTEL_RULES: ""
SW_METER_ANALYZER_ACTIVE_FILES: ""
SW_ENVOY_METRIC: "-"
ports:
- 12800
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
networks:
- e2e
depends_on:
sharding-proxy:
condition: service_healthy
provider:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: provider
depends_on:
oap:
condition: service_healthy
ports:
- 9090
networks:
- e2e
networks:
e2e:

47
test/e2e-v2/cases/log/mysql/sharding/e2e.yaml Normal file
View File

@ -0,0 +1,47 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file is used to show how to write configuration files and can be used to test.
setup:
env: compose
file: docker-compose.yml
timeout: 20m
init-system-environment: ../../../../script/env
steps:
- name: set PATH
command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- name: install yq
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- name: install swctl
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
trigger:
action: http
interval: 3s
times: 10
url: http://${provider_host}:${provider_9090}/logs/trigger
method: GET
verify:
# verify with retry strategy
retry:
# max retry count
count: 20
# the interval between two retries, in millisecond.
interval: 10s
cases:
- includes:
- ../../log-cases.yaml

4
test/e2e-v2/cases/mysql/mysql-slowsql/docker-compose.yaml
View File

@ -22,7 +22,7 @@ services:
service: oap
ports:
- 12800
entrypoint: ['sh', '-c', '/download-mysql.sh && chmod 777 /skywalking/docker-entrypoint.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && chmod 777 /skywalking/docker-entrypoint.sh && /skywalking/docker-entrypoint.sh']
networks:
- e2e
environment:
@ -103,4 +103,4 @@ services:
oap:
condition: service_healthy
networks:
e2e:
e2e:

4
test/e2e-v2/cases/profiling/ebpf/oncpu/mysql/docker-compose.yml
View File

@ -40,7 +40,7 @@ services:
- e2e
environment:
SW_STORAGE: mysql
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
ports:
- 12800
depends_on:
@ -65,4 +65,4 @@ services:
condition: service_healthy
networks:
e2e:
e2e:

4
test/e2e-v2/cases/profiling/ebpf/oncpu/tidb/docker-compose.yml
View File

@ -41,7 +41,7 @@ services:
SW_STORAGE: tidb
SW_JDBC_URL: "jdbc:mysql://tidb:4000/test"
SW_DATA_SOURCE_PASSWORD: ""
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
depends_on:
tidb:
condition: service_healthy
@ -66,4 +66,4 @@ services:
condition: service_healthy
networks:
e2e:
e2e:

2
test/e2e-v2/cases/profiling/trace/mysql/docker-compose.yml
View File

@ -51,7 +51,7 @@ services:
depends_on:
mysql:
condition: service_healthy
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
ports:
- 12800

2
test/e2e-v2/cases/profiling/trace/tidb/docker-compose.yml
View File

@ -49,7 +49,7 @@ services:
SW_STORAGE: tidb
SW_JDBC_URL: "jdbc:mysql://tidb:4000/test"
SW_DATA_SOURCE_PASSWORD: ""
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
depends_on:
tidb:
condition: service_healthy

2
test/e2e-v2/cases/storage/mysql/docker-compose.yml
View File

@ -41,7 +41,7 @@ services:
SW_JDBC_URL: jdbc:mysql://mysql:3306/swtest
ports:
- 12800
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
networks:
- e2e
depends_on:

101
test/e2e-v2/cases/storage/mysql/sharding/docker-compose.yml Normal file
View File

@ -0,0 +1,101 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: '2.1'
services:
sharding-proxy:
extends:
file: ../../../../script/shardingsphere-proxy/base-compose.yml
service: sharding-proxy
networks:
- e2e
expose:
- 3307
depends_on:
mysql-1:
condition: service_healthy
mysql-2:
condition: service_healthy
mysql-1: &mysql
image: mysql/mysql-server:8.0.13
networks:
- e2e
expose:
- 3306
environment:
MYSQL_ROOT_PASSWORD: "root@1234"
MYSQL_DATABASE: "swtest"
MYSQL_ROOT_HOST: "%"
healthcheck:
test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
interval: 5s
timeout: 60s
retries: 120
mysql-2:
<<: *mysql
oap:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: oap
environment:
SW_STORAGE: mysql-sharding
SW_JDBC_URL: jdbc:mysql://sharding-proxy:3307/swtest
SW_JDBC_SHARDING_DATA_SOURCES: ds_0,ds_1
SW_CORE_METRICS_DATA_TTL: 2
SW_CORE_RECORD_DATA_TTL: 2
SW_OTEL_RECEIVER_ENABLED_OTEL_RULES: ""
SW_METER_ANALYZER_ACTIVE_FILES: ""
SW_LOG_LAL_FILES: ""
SW_ENVOY_METRIC: "-"
ports:
- 12800
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
networks:
- e2e
depends_on:
sharding-proxy:
condition: service_healthy
provider:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: provider
depends_on:
oap:
condition: service_healthy
ports:
- 9090
networks:
- e2e
consumer:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: consumer
ports:
- 9092
depends_on:
oap:
condition: service_healthy
provider:
condition: service_healthy
networks:
e2e:

50
test/e2e-v2/cases/storage/mysql/sharding/e2e.yaml Normal file
View File

@ -0,0 +1,50 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file is used to show how to write configuration files and can be used to test.
setup:
env: compose
file: docker-compose.yml
timeout: 20m
init-system-environment: ../../../../script/env
steps:
- name: set PATH
command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- name: install yq
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- name: install swctl
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
trigger:
action: http
interval: 3s
times: 10
url: http://${consumer_host}:${consumer_9092}/users
method: POST
body: '{"id":"123","name":"skywalking"}'
headers:
"Content-Type": "application/json"
verify:
# verify with retry strategy
retry:
# max retry count
count: 20
# the interval between two retries, in millisecond.
interval: 10s
cases:
- includes:
- ../../storage-cases.yaml

2
test/e2e-v2/cases/storage/tidb/docker-compose.yml
View File

@ -41,7 +41,7 @@ services:
SW_DATA_SOURCE_PASSWORD: ""
ports:
- 12800
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
networks:
- e2e
depends_on:

2
test/e2e-v2/cases/ttl/tidb/docker-compose.yml
View File

@ -41,7 +41,7 @@ services:
SW_CORE_METRICS_DATA_TTL: 7
SW_JDBC_URL: "jdbc:mysql://tidb:4000/test"
SW_DATA_SOURCE_PASSWORD: ""
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
depends_on:
tidb:
condition: service_healthy

2
test/e2e-v2/cases/zipkin/mysql/docker-compose.yml
View File

@ -45,7 +45,7 @@ services:
- 9411
ports:
- 9412
entrypoint: ['sh', '-c', '/download-mysql.sh && /skywalking/docker-entrypoint.sh']
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
networks:
- e2e
depends_on:

100
test/e2e-v2/cases/zipkin/mysql/sharding/docker-compose.yml Normal file
View File

@ -0,0 +1,100 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: '2.1'
services:
sharding-proxy:
extends:
file: ../../../../script/shardingsphere-proxy/base-compose.yml
service: sharding-proxy
networks:
- e2e
expose:
- 3307
depends_on:
mysql-1:
condition: service_healthy
mysql-2:
condition: service_healthy
mysql-1: &mysql
image: mysql/mysql-server:8.0.13
networks:
- e2e
expose:
- 3306
environment:
MYSQL_ROOT_PASSWORD: "root@1234"
MYSQL_DATABASE: "swtest"
MYSQL_ROOT_HOST: "%"
healthcheck:
test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
interval: 5s
timeout: 60s
retries: 120
mysql-2:
<<: *mysql
oap:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: oap
environment:
SW_STORAGE: mysql-sharding
SW_JDBC_URL: jdbc:mysql://sharding-proxy:3307/swtest
SW_JDBC_SHARDING_DATA_SOURCES: ds_0,ds_1
SW_CORE_METRICS_DATA_TTL: 2
SW_CORE_RECORD_DATA_TTL: 2
SW_OTEL_RECEIVER_ENABLED_OTEL_RULES: ""
SW_METER_ANALYZER_ACTIVE_FILES: ""
SW_LOG_LAL_FILES: ""
SW_ENVOY_METRIC: "-"
SW_QUERY_ZIPKIN: default
SW_RECEIVER_ZIPKIN: default
expose:
- 9411
ports:
- 9412
entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && /skywalking/docker-entrypoint.sh']
networks:
- e2e
depends_on:
sharding-proxy:
condition: service_healthy
frontend:
extends:
file: ../../docker-compose-brave.yml
service: frontend
depends_on:
backend:
condition: service_healthy
oap:
condition: service_healthy
ports:
- 8081
backend:
extends:
file: ../../docker-compose-brave.yml
service: backend
depends_on:
oap:
condition: service_healthy
networks:
e2e:

47
test/e2e-v2/cases/zipkin/mysql/sharding/e2e.yaml Normal file
View File

@ -0,0 +1,47 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file is used to show how to write configuration files and can be used to test.
setup:
env: compose
file: docker-compose.yml
timeout: 20m
init-system-environment: ../../../../script/env
steps:
- name: set PATH
command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- name: install yq
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- name: install swctl
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
trigger:
action: http
interval: 3s
times: 10
url: http://${frontend_host}:${frontend_8081}
method: POST
verify:
# verify with retry strategy
retry:
# max retry count
count: 20
# the interval between two retries, in millisecond.
interval: 10s
cases:
- includes:
- ../../zipkin-cases.yaml

4
test/e2e-v2/script/prepare/setup-oap/download-mysql.sh
View File

@ -17,7 +17,7 @@
set -ex
SW_HOME=/skywalking
LIB_HOME=$1
MYSQL_URL="https://repo.maven.apache.org/maven2/mysql/mysql-connector-java/8.0.13/mysql-connector-java-8.0.13.jar"
MYSQL_DRIVER="mysql-connector-java-8.0.13.jar"
@ -26,7 +26,7 @@ if ! command -v curl &> /dev/null; then
apt update -y && apt install -y curl
fi
if ! curl -Lo "${SW_HOME}/oap-libs/${MYSQL_DRIVER}" ${MYSQL_URL}; then
if ! curl -Lo "${LIB_HOME}/${MYSQL_DRIVER}" ${MYSQL_URL}; then
echo "Fail to download ${MYSQL_DRIVER}."
exit 1
fi

36
test/e2e-v2/script/shardingsphere-proxy/base-compose.yml Normal file
View File

@ -0,0 +1,36 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: '2.1'
services:
sharding-proxy:
image: apache/shardingsphere-proxy:5.1.2
volumes:
- ./../prepare/setup-oap/download-mysql.sh:/opt/shardingsphere-proxy/download-mysql.sh
- ./conf:/opt/shardingsphere-proxy/conf
networks:
- e2e
expose:
- 3307
entrypoint: ['sh', '-c', '/opt/shardingsphere-proxy/download-mysql.sh /opt/shardingsphere-proxy/ext-lib && /opt/shardingsphere-proxy/bin/start.sh 3307 && tail -f /opt/shardingsphere-proxy/logs/stdout.log']
healthcheck:
test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3307"]
interval: 5s
timeout: 60s
retries: 120
networks:
e2e:

45
test/e2e-v2/script/shardingsphere-proxy/conf/config-sharding.yaml Normal file
View File

@ -0,0 +1,45 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######################################################################################################
#
# Here you can configure the rules for the proxy.
# This example is configuration of sharding rule.
#
######################################################################################################
#
databaseName: swtest
dataSources:
ds_0:
url: jdbc:mysql://mysql-1:3306/swtest?allowPublicKeyRetrieval=true
username: root
password: root@1234
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_1:
url: jdbc:mysql://mysql-2:3306/swtest?allowPublicKeyRetrieval=true
username: root
password: root@1234
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1

Some files were not shown because too many files have changed in this diff Show More