kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

BanyanDB: support TLS connection and configuration. (#12639)

This commit is contained in:
Wan Kai 2024-09-24 16:24:17 +08:00 committed by GitHub
parent 15ad71d633
commit 7d1b65c639
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
14 changed files with 215 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/skywalking.yaml vendored
View File

@ -359,6 +359,8 @@ jobs:
- name: BanyanDB
config: test/e2e-v2/cases/storage/banyandb/e2e.yaml
- name: BanyanDB TLS
config: test/e2e-v2/cases/storage/banyandb/tls/e2e.yaml
- name: Storage H2
config: test/e2e-v2/cases/storage/h2/e2e.yaml
- name: Storage MySQL

1
.licenserc.yaml
View File

@ -42,6 +42,7 @@ header:
- '**/*.ini'
- '**/*.crt'
- '**/*.pem'
- '**/*.key'
- '**/*.txt'
- 'LICENSE'
- 'NOTICE'

1
docs/en/changes/changes.md
View File

@ -70,6 +70,7 @@
* Add Component ID(5022) for the GoFrame framework.
* Bump up protobuf java dependencies to 3.25.5.
* BanyanDB: support using native term searching for `keyword` in query `findEndpoint` and `getAlarm`.
* BanyanDB: support TLS connection and configuration.
#### UI

4
docs/en/setup/backend/configuration-vocabulary.md
View File

@ -149,6 +149,10 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | superDatasetShardsFactor | Shards Factor for a super dataset record, i.e. Shard number of a super dataset is recordShardsNumber*superDatasetShardsFactor. | SW_STORAGE_BANYANDB_SUPERDATASET_SHARDS_FACTOR | 2 |
| - | - | concurrentWriteThreads | Concurrent consumer threads for batch writing. | SW_STORAGE_BANYANDB_CONCURRENT_WRITE_THREADS | 15 |
| - | - | profileTaskQueryMaxSize | Max size of ProfileTask to be fetched. | SW_STORAGE_BANYANDB_PROFILE_TASK_QUERY_MAX_SIZE | 200 |
| - | - | segmentIntervalDays | Data is stored in BanyanDB in segments. A segment is a time range of data.The segment interval is the time range of a segment.The value should be less or equal to data TTL relevant settings. | SW_STORAGE_BANYANDB_SEGMENT_INTERVAL_DAYS | 1 |
| - | - | superDatasetSegmentIntervalDays | The super dataset segment interval is the time range of a segment in the super dataset. | SW_STORAGE_BANYANDB_SUPER_DATASET_SEGMENT_INTERVAL_DAYS | 1 |
| - | - | specificGroupSettings | Specific groups settings. | SW_STORAGE_BANYANDB_SPECIFIC_GROUP_SETTINGS | - |
| - | - | sslTrustCAPath | If the BanyanDB server is configured with TLS, config the TLS cert file path and open tls connection. | SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH | - |
| agent-analyzer | default | Agent Analyzer. | SW_AGENT_ANALYZER | default | |
| - | - | traceSamplingPolicySettingsFile | The sampling policy including `sampling rate` and `the threshold of trace segment latency` can be configured by the `traceSamplingPolicySettingsFile` file. | SW_TRACE_SAMPLING_POLICY_SETTINGS_FILE | `trace-sampling-policy-settings.yml` |
| - | - | slowDBAccessThreshold | The slow database access threshold (in milliseconds). | SW_SLOW_DB_THRESHOLD | default:200,mongodb:100 |

2
docs/en/setup/backend/storages/banyandb.md
View File

@ -49,6 +49,8 @@ storage:
# Please refer to https://github.com/apache/skywalking-banyandb/blob/${BANYANDB_RELEASE}/docs/crud/group.md#create-operation
# for group setting details.
specificGroupSettings: ${SW_STORAGE_BANYANDB_SPECIFIC_GROUP_SETTINGS:""}
# If the BanyanDB server is configured with TLS, config the TLS cert file path and open tls connection.
sslTrustCAPath: ${SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH:""}
```
BanyanDB Server supports two installation modes: standalone and cluster. The standalone mode is suitable for small-scale deployments, while the cluster mode is suitable for large-scale deployments.

6
oap-server/server-starter/src/main/resources/application.yml
View File

@ -220,7 +220,7 @@ storage:
# Targets is the list of BanyanDB servers, separated by commas.
# Each target is a BanyanDB server in the format of `host:port`
# The host is the IP address or domain name of the BanyanDB server, and the port is the port number of the BanyanDB server.
targets: ${SW_STORAGE_BANYANDB_TARGETS:127.0.0.1:17912}
targets: ${SW_STORAGE_BANYANDB_TARGETS:localhost:17912}
# The max number of records in a bulk write request.
# Bigger value can improve the write performance, but also increase the OAP and BanyanDB Server memory usage.
maxBulkSize: ${SW_STORAGE_BANYANDB_MAX_BULK_SIZE:10000}
@ -254,9 +254,11 @@ storage:
superDatasetSegmentIntervalDays: ${SW_STORAGE_BANYANDB_SUPER_DATASET_SEGMENT_INTERVAL_DAYS:1}
# Specific groups settings.
# For example, {"group1": {"blockIntervalHours": 4, "segmentIntervalDays": 1}}
# Please refer to https://github.com/apache/skywalking-banyandb/blob/${BANYANDB_RELEASE}/docs/crud/group.md#create-operation
# Please refer to https://github.com/apache/skywalking-banyandb/blob/${BANYANDB_RELEASE}/docs/interacting/bydbctl/schema/group.md#create-operation
# for group setting details.
specificGroupSettings: ${SW_STORAGE_BANYANDB_SPECIFIC_GROUP_SETTINGS:""}
# If the BanyanDB server is configured with TLS, config the TLS cert file path and open tls connection.
sslTrustCAPath: ${SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH:""}
agent-analyzer:
selector: ${SW_AGENT_ANALYZER:default}

9
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageClient.java
View File

@ -25,6 +25,7 @@ import org.apache.skywalking.banyandb.v1.client.MeasureBulkWriteProcessor;
import org.apache.skywalking.banyandb.v1.client.MeasureQuery;
import org.apache.skywalking.banyandb.v1.client.MeasureQueryResponse;
import org.apache.skywalking.banyandb.v1.client.MeasureWrite;
import org.apache.skywalking.banyandb.v1.client.Options;
import org.apache.skywalking.banyandb.v1.client.StreamBulkWriteProcessor;
import org.apache.skywalking.banyandb.v1.client.StreamQuery;
import org.apache.skywalking.banyandb.v1.client.StreamQueryResponse;
@ -58,9 +59,11 @@ public class BanyanDBStorageClient implements Client, HealthCheckable {
private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
private final int flushTimeout;
public BanyanDBStorageClient(int flushTimeout, String... targets) {
this.client = new BanyanDBClient(targets);
this.flushTimeout = flushTimeout;
public BanyanDBStorageClient(BanyanDBStorageConfig config) {
Options options = new Options();
options.setSslTrustCAPath(config.getSslTrustCAPath());
this.client = new BanyanDBClient(config.getTargetArray(), options);
this.flushTimeout = config.getFlushTimeout();
}
@Override

5
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageConfig.java
View File

@ -93,6 +93,11 @@ public class BanyanDBStorageConfig extends ModuleConfig {
*/
private String specificGroupSettings;
/**
* If the BanyanDB server is configured with TLS, config the TLS cert file path and open tls connection.
*/
private String sslTrustCAPath = "";
public String[] getTargetArray() {
return Iterables.toArray(Splitter.on(",").omitEmptyStrings().trimResults().split(this.targets), String.class);
}

2
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageProvider.java
View File

@ -117,7 +117,7 @@ public class BanyanDBStorageProvider extends ModuleProvider {
public void prepare() throws ServiceNotProvidedException, ModuleStartException {
this.registerServiceImplementation(StorageBuilderFactory.class, new StorageBuilderFactory.Default());
this.client = new BanyanDBStorageClient(config.getFlushTimeout(), config.getTargetArray());
this.client = new BanyanDBStorageClient(config);
this.modelInstaller = new BanyanDBIndexInstaller(client, getManager(), this.config);
// Stream

69
test/e2e-v2/cases/storage/banyandb/tls/docker-compose.yml Normal file
View File

@ -0,0 +1,69 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: '2.1'
services:
banyandb:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: banyandb
volumes:
- ./tls:/tls
command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data --tls=true --key-file=/tls/cert.key --cert-file=/tls/cert.crt --http-grpc-cert-file=/tls/ca.crt
networks:
- e2e
oap:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: oap
volumes:
- ./tls/cert.crt:/tls/cert.crt
environment:
SW_STORAGE: banyandb
SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH: /tls/cert.crt
ports:
- 12800
depends_on:
banyandb:
condition: service_healthy
provider:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: provider
ports:
- 9090
networks:
- e2e
depends_on:
oap:
condition: service_healthy
consumer:
extends:
file: ../../../../script/docker-compose/base-compose.yml
service: consumer
ports:
- 9092
depends_on:
oap:
condition: service_healthy
provider:
condition: service_healthy
networks:
e2e:

50
test/e2e-v2/cases/storage/banyandb/tls/e2e.yaml Normal file
View File

@ -0,0 +1,50 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file is used to show how to write configuration files and can be used to test.
setup:
env: compose
file: docker-compose.yml
timeout: 20m
init-system-environment: ../../../../script/env
steps:
- name: set PATH
command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- name: install yq
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- name: install swctl
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
trigger:
action: http
interval: 5s
times: 40
url: http://${consumer_host}:${consumer_9092}/users
method: POST
body: '{"id":"123","name":"skywalking"}'
headers:
"Content-Type": "application/json"
verify:
# verify with retry strategy
retry:
# max retry count
count: 20
# the interval between two retries, in millisecond.
interval: 10s
cases:
- includes:
- ../../storage-cases.yaml

21
test/e2e-v2/cases/storage/banyandb/tls/tls/ca.crt Normal file
View File

@ -0,0 +1,21 @@
-----BEGIN CERTIFICATE-----
MIIDYzCCAkugAwIBAgIUAJ+VBTfduocXTnjwAAt5/xLcuhcwDQYJKoZIhvcNAQEL
BQAwQDELMAkGA1UEBhMCQ04xDTALBgNVBAgMBFRlc3QxDTALBgNVBAcMBFRlc3Qx
EzARBgNVBAoMClNreVdhbGtpbmcwIBcNMjQwOTE5MDIxNjEzWhgPMjEyNDA4MjYw
MjE2MTNaMEAxCzAJBgNVBAYTAkNOMQ0wCwYDVQQIDARUZXN0MQ0wCwYDVQQHDARU
ZXN0MRMwEQYDVQQKDApTa3lXYWxraW5nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEAy1DBH5bTwGj+hGewLPJZ/wFAlsi2oWH01vlJSIXHLoFm/vNPh1/5
psCSr08KJT+h445b5TEphOgK/83NhRKlWPjhUL+MfqnQ4VES+iFIpxvUNcwxWosS
Ky2U2it000kiuX9dPC2eR5tC7AmU0/TrON9ZBLzzsAlDPgTA63BSdH3NUqgy/w/7
r6gpiXwsjMKTdB/S2PQ+ojwR0xgqmiDI2GiA7nTTo9AE+gkMx3EUS6BF6Dn8Vrxf
F0UNjwVGzL15S5VaD3nn1ky5RO++D3ie+8dfUTjwnmEOJPf6jLdZPI/bsI+mGZOQ
kmcp67ojMsmbI8mca4fh9KrLMmFaFVekDwIDAQABo1MwUTAdBgNVHQ4EFgQUQrK4
dKKuvLwTSfoWE/fmgX25nj0wHwYDVR0jBBgwFoAUQrK4dKKuvLwTSfoWE/fmgX25
nj0wDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAqOVwBk53gZfM
SndGoQxZgMM6jTlsKSCkxvdzjuNKYqa5SGEWwevlEXkC+cNgt2biAzVSUQhYPSle
I6LrGhTvWQia0lEcjto5flAzg+2oqKVb+i5RULpFvmiGSvB5+CDitthAIRDx3z2Y
UhRlSL8Fx371jn9/wHPjsYbR0wgfYWZkFa+uF7YEKmidzcAdvXgFVX6I2/rztfMZ
3SClJqL5/fEfQM6V4lBpQI63XVZlIEJuSu9ce5SwdIfNT0Hj2JbZilpuICRuScDw
O19AERTKp2Y+fGZNmBHdPdG3yIXPRiCilZLd3wRBaV3uTUoAMldZt53txWUk5PQu
P6rokRsE1g==
-----END CERTIFICATE-----

21
test/e2e-v2/cases/storage/banyandb/tls/tls/cert.crt Normal file
View File

@ -0,0 +1,21 @@
-----BEGIN CERTIFICATE-----
MIIDejCCAmKgAwIBAgIUX+aT7K4RC+yPZZgycUYzVjGjym0wDQYJKoZIhvcNAQEL
BQAwQDELMAkGA1UEBhMCQ04xDTALBgNVBAgMBFRlc3QxDTALBgNVBAcMBFRlc3Qx
EzARBgNVBAoMClNreVdhbGtpbmcwIBcNMjQwOTE5MDIxNjEzWhgPMjEyNDA4MjYw
MjE2MTNaMFMxCzAJBgNVBAYTAkNOMQ0wCwYDVQQIDARUZXN0MQ0wCwYDVQQHDARU
ZXN0MRMwEQYDVQQKDApTa3lXYWxraW5nMREwDwYDVQQDDAhiYW55YW5kYjCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJT3AVIUrW5hHbvK5UHNUhudBM2P
BaDDKmhzdnG0jNRbfbzRnSc5/E733bAtOHp+q1yCmeLlOiIuN7Kzw7JRGrps3UwA
4a5dC8T64QabGJec0uKSQ8OSTxxlX5cKwMvbS3oukgBztwrXZdgFUHmLhT7E274I
w8qpoqGSU003DQxyLQi4NxKuzXvUUlGLLfOjvf3DUN6HcoY2pmiRg+yRmJ0CcDYI
iNTLUnZ++Ev+DLYoz2fn/POhUzb3QaqXd9og07GjR1g55O7TuhDQNGYZ1ugrU3wG
8LwQfRBGyLVgF9sCmOoJwwxAePifqDhK+qjG2nlCdaS+EyNODTauJBy7GoUCAwEA
AaNXMFUwEwYDVR0RBAwwCoIIYmFueWFuZGIwHQYDVR0OBBYEFBfAaFUvCKWlx/av
SfdJztGCGendMB8GA1UdIwQYMBaAFEKyuHSirry8E0n6FhP35oF9uZ49MA0GCSqG
SIb3DQEBCwUAA4IBAQBbY7Q8TVTRnhr+mx8eeaKtubz8IqPoKxURTuYRiDWXmIq0
tYSE/ZAuaXXA1yCDAa4QVqu2JnMl4V/RN4ekdGjOo3baPUAeINhB/gBPVNs7Ek5B
RmRLs+quPfl7YraTGMAC2ykr6GTB2v7eW9OEoriND8ien14SdqKuf87lFRAHmuAG
3lFQqMdlGPS3PbxcDlcfcgsXi6FevINUBXl7Q2cMcFiXiyG0C1w2oA0Q611O6rZv
7yov42kAWnQiuN3HtLQKhG296V66zla5j9oE2NmTqM4b3/wyBNCEedIPGumcrdMU
ICdzwef06UIRHpaV39+P4dJZWE8whj5p3GGMGw1a
-----END CERTIFICATE-----

28
test/e2e-v2/cases/storage/banyandb/tls/tls/cert.key Normal file
View File

@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCU9wFSFK1uYR27
yuVBzVIbnQTNjwWgwypoc3ZxtIzUW3280Z0nOfxO992wLTh6fqtcgpni5ToiLjey
s8OyURq6bN1MAOGuXQvE+uEGmxiXnNLikkPDkk8cZV+XCsDL20t6LpIAc7cK12XY
BVB5i4U+xNu+CMPKqaKhklNNNw0Mci0IuDcSrs171FJRiy3zo739w1Deh3KGNqZo
kYPskZidAnA2CIjUy1J2fvhL/gy2KM9n5/zzoVM290Gql3faINOxo0dYOeTu07oQ
0DRmGdboK1N8BvC8EH0QRsi1YBfbApjqCcMMQHj4n6g4Svqoxtp5QnWkvhMjTg02
riQcuxqFAgMBAAECggEAAK+QJEd0Fzu2p1RjOc7l5CheCwHv1vtYLX5almJHjNdM
s7+McIZc9UlBIJem/TKFzkvLoG2ABLqNK6oDoTMZ0PjBnlf0+yrN7DWpLvoH0Krf
BIdw/0fIlXRPvFkinF/vOQVZS0vPXjREOp7wNsObYpMqAhqIOe/E2QvcjSwqkO08
+3Pd6LQy8VnF50nLsagPDbTrvdaSZZOxEwK0RtojHp3TqYt9wzDhFo3IBeQOp2FW
MCesOHgDQ5lSMUo4FazQmC2MaGf1hlCpIoZ0TyIlavpyqKE4aRNtUrCZFk0BLSbQ
DjIVkSCpNEuTy9D8PpafXQ4M15+Qs/2mKmQRhq21AQKBgQDG58Wg8a9+Nr+4e0fK
B/qdPjvoSgVMceFF/zQ8ABeK0IEfmaywNAedDUc5X8Rav+l1cbNHdywZbB5nqZL4
wWeZTcsC4lh9LDC7a/IIubBipOgWwOSPE1CQh05iUgFXWzfxWdhLptZGf1G75BD3
Ow4ye/5QinLeOovUh3GRCw8IDwKBgQC/uXB2ISXpl2HW/z7rqHn64vHQ5+D6zTpm
XHYgyPlZw/HOPBt21kfV5FvcxtLuE1iv+AVcJ2lUyd2idIwOs5jBykTcf8BAf1Py
Pm20ghYTyyJVRWsyIpWlZ9NBgRxJVhqZneSTs4ZHexDBNQSg4Zd8JE8kKuruA15z
zVdPxL9AKwKBgQCWEB6GAlFIg725TSkoP0TQhTdlVRMSTYiL6rJZ4ifgztXcJXCn
r1lfH7lwi9P0M33K4XFEpaBbvbpiswjzK9chThOvpVzlcwWBITuFYbK1eXywTe8i
EoxhvfNhMkVYxrU6HXKWANUAcuYTP0mdfKyFYg7IbOukxb7hJR6FMzav+QKBgGst
4qXXQBeG7kY/qNAa/h34EtEDX+01H6mrlModL/AzrxDo3z2e6sezAnu4lqBLJnAx
CeYLjkG9G+6+krGBSytCxR4zOKmVxLsOlUus0WS9UXMz/IopdVjjYuEiqUqCHRq1
ftvnJ3D8ZlBsMCs0EfhvX+LLL0xhi8LXDzFjvrrDAoGAWSW09H4jgiFGzTfZwUEF
Ra/375b9ajI9Uy27Do+jyU0jfVGsC+K46w+U989Told2adFj89dTcjochRcv0U2w
M+TN3od1Sb3XzJF2+bOkhQqyoq1VzG5AcbxROu6a+LFg3KQgEfHNJWQbCs4pfa1n
tI1ZOwddeyXKvOPnAj30kyo=
-----END PRIVATE KEY-----