kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Bump up iotdb-session to 0.12.4 and fix IoTDB storage option bug (#8484) 

* Fix IoTDB Storage Option insert null index value.

* Set the default value of SW_STORAGE_IOTDB_SESSIONPOOL_SIZE to 8

* Bump up iotdb-session to 0.12.4

* Update doc
This commit is contained in:
刘威 2022-01-30 09:02:39 +08:00 committed by GitHub
parent 9033030788
commit b0a875fe3f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
17 changed files with 36 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGES.md
View File

@ -47,6 +47,9 @@ Release Notes.
* Doc update: `configuration-vocabulary`,`backend-cluster` about env variables `SW_CLUSTER_INTERNAL_COM_HOST` and `SW_CLUSTER_INTERNAL_COM_PORT`.
* Add Python MysqlClient component ID(7013) with mapping information.
* Support Java thread pool metrics analysis.
* Fix IoTDB Storage Option insert null index value.
* Set the default value of SW_STORAGE_IOTDB_SESSIONPOOL_SIZE to 8.
* Bump up iotdb-session to 0.12.4.
#### UI

8
dist-material/release-docs/LICENSE
View File

@ -330,10 +330,10 @@ The text of each license is the standard Apache 2.0 license.
Armeria 1.13.4, http://github.com/line/armeria, Apache 2.0
Brotli4j 1.6.0, https://github.com/hyperxpro/Brotli4j, Apache 2.0
micrometer 1.7.6, https://github.com/micrometer-metrics/micrometer, Apache 2.0
iotdb-session 0.12.3: https://github.com/apache/iotdb, Apache 2.0
iotdb-thrift 0.12.3: https://github.com/apache/iotdb, Apache 2.0
service-rpc 0.12.3: https://github.com/apache/iotdb, Apache 2.0
tsfile 0.12.3 https://github.com/apache/iotdb Apache 2.0
iotdb-session 0.12.4: https://github.com/apache/iotdb, Apache 2.0
iotdb-thrift 0.12.4: https://github.com/apache/iotdb, Apache 2.0
service-rpc 0.12.4: https://github.com/apache/iotdb, Apache 2.0
tsfile 0.12.4 https://github.com/apache/iotdb Apache 2.0
libthrift 0.14.1: https://github.com/apache/thrift Apache 2.0
j2objc 1.3: https://github.com/google/j2objc Apache 2.0

2
docs/en/setup/backend/backend-storage.md
View File

@ -294,7 +294,7 @@ storage:
username: ${SW_STORAGE_IOTDB_USERNAME:root}
password: ${SW_STORAGE_IOTDB_PASSWORD:root}
storageGroup: ${SW_STORAGE_IOTDB_STORAGE_GROUP:root.skywalking}
sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:16}
sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:8} # If it's zero, the SessionPool size will be 2*CPU_Cores
fetchTaskLogMaxSize: ${SW_STORAGE_IOTDB_FETCH_TASK_LOG_MAX_SIZE:1000} # the max number of fetch task log in a request
```
All connection related settings, including host, rpcPort, username, and password are found in `application.yml`. Please ensure the IoTDB version >= 0.12.3.

8
docs/en/setup/backend/configuration-vocabulary.md
View File

@ -143,6 +143,14 @@ core|default|role|Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode
| - | - | batchEnabled | If true, write points with batch API. | SW_STORAGE_INFLUXDB_BATCH_ENABLED | true|
| - | - | fetchTaskLogMaxSize | The maximum number of fetch task log in a request. | SW_STORAGE_INFLUXDB_FETCH_TASK_LOG_MAX_SIZE | 5000|
| - | - | connectionResponseFormat | The response format of connection to influxDB. It can only be MSGPACK or JSON. | SW_STORAGE_INFLUXDB_CONNECTION_RESPONSE_FORMAT | MSGPACK |
| - | iotdb | - | IoTDB storage. | - | - |
| - | - | host | The host of IoTDB server. | SW_STORAGE_IOTDB_HOST | 127.0.0.1 |
| - | - | rpcPort | The port listened by IoTDB server. | SW_STORAGE_IOTDB_RPC_PORT | 6667 |
| - | - | username | The username of IoTDB | SW_STORAGE_IOTDB_USERNAME | root |
| - | - | password | The password of IoTDB | SW_STORAGE_IOTDB_PASSWORD | root |
| - | - | storageGroup | The path of Storage Group and it must start with `root.` | SW_STORAGE_IOTDB_STORAGE_GROUP | root.skywalking |
| - | - | sessionPoolSize | The connection pool size for IoTDB. If the value is 0, the size of SessionPool will be 2 * CPU_Cores | SW_STORAGE_IOTDB_SESSIONPOOL_SIZE | 8 |
| - | - | fetchTaskLogMaxSize | the max number of fetch task log in a request | SW_STORAGE_IOTDB_FETCH_TASK_LOG_MAX_SIZE | 1000 |
| agent-analyzer | default | Agent Analyzer. | SW_AGENT_ANALYZER | default |
| - | - | traceSamplingPolicySettingsFile | The sampling policy including `sampling rate` and `the threshold of trace segment latency` can be configured by the `traceSamplingPolicySettingsFile` file. | SW_TRACE_SAMPLING_POLICY_SETTINGS_FILE | `trace-sampling-policy-settings.yml` |
| - | - |slowDBAccessThreshold| The slow database access threshold (in milliseconds). |SW_SLOW_DB_THRESHOLD|default:200,mongodb:100|

2
oap-server-bom/pom.xml
View File

@ -78,7 +78,7 @@
<awaitility.version>3.0.0</awaitility.version>
<httpcore.version>4.4.13</httpcore.version>
<commons-compress.version>1.21</commons-compress.version>
<iotdb-session.version>0.12.3</iotdb-session.version>
<iotdb-session.version>0.12.4</iotdb-session.version>
<lz4-java.version>1.6.0</lz4-java.version>
</properties>

2
oap-server/server-starter/src/main/resources/application.yml
View File

@ -258,7 +258,7 @@ storage:
username: ${SW_STORAGE_IOTDB_USERNAME:root}
password: ${SW_STORAGE_IOTDB_PASSWORD:root}
storageGroup: ${SW_STORAGE_IOTDB_STORAGE_GROUP:root.skywalking}
sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:16}
sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:8} # If it's zero, the SessionPool size will be 2*CPU_Cores
fetchTaskLogMaxSize: ${SW_STORAGE_IOTDB_FETCH_TASK_LOG_MAX_SIZE:1000} # the max number of fetch task log in a request
agent-analyzer:

7
oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
View File

@ -67,8 +67,11 @@ public class IoTDBClient implements Client, HealthCheckable {
@Override
public void connect() throws IoTDBConnectionException, StatementExecutionException {
try {
final int sessionPoolSize = config.getSessionPoolSize() == 0 ?
Runtime.getRuntime().availableProcessors() * 2 : config.getSessionPoolSize();
log.info("SessionPool Size: {}", sessionPoolSize);
sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
config.getPassword(), config.getSessionPoolSize());
config.getPassword(), sessionPoolSize, false, false);
sessionPool.setStorageGroup(storageGroup);
healthChecker.health();
@ -254,7 +257,7 @@ public class IoTDBClient implements Client, HealthCheckable {
/**
* Query with aggregation function: count, sum, avg, last_value, first_value, min_time, max_time, min_value, max_value
*
* @param querySQL the SQL for query which should contain aggregation function
* @param querySQL the SQL for query which should contain aggregation function
* @return the result of aggregation function
* @throws IOException IoTDBConnectionException or StatementExecutionException
*/

4
oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
View File

@ -61,8 +61,8 @@ public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
if (index.equals(IoTDBIndexes.ID_IDX)) {
indexValues.add(storageData.id());
} else if (storageMap.containsKey(index)) {
// avoid `service_group` be "null" when inserting
if (index.equals(IoTDBIndexes.GROUP_IDX) && storageMap.get(index) == null) {
// avoid indexValue be "null" when inserting
if (storageMap.get(index) == null) {
indexValues.add("");
} else {
indexValues.add(String.valueOf(storageMap.get(index)));

3
oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBRecordDAO.java
View File

@ -74,9 +74,6 @@ public class IoTDBRecordDAO implements IRecordDAO {
measurementValues.add(rawTag.getValue());
});
}
request.setMeasurements(measurements);
request.setMeasurementTypes(measurementTypes);
request.setMeasurementValues(measurementValues);
return request;
}
}

3
test/e2e-v2/cases/alarm/iotdb/docker-compose.yml
View File

@ -17,7 +17,7 @@ version: '2.1'
services:
iotdb:
image: apache/iotdb:0.12.3-node
image: apache/iotdb:0.12.4-node
expose:
- 6667
networks:
@ -33,7 +33,6 @@ services:
file: ../../../script/docker-compose/base-compose.yml
service: oap
environment:
SW_STORAGE: iotdb
SW_SEARCHABLE_ALARM_TAG_KEYS: level,receivers
ports:
- 12800

2
test/e2e-v2/cases/event/iotdb/docker-compose.yml
View File

@ -17,7 +17,7 @@ version: '3.8'
services:
iotdb:
image: apache/iotdb:0.12.3-node
image: apache/iotdb:0.12.4-node
expose:
- 6667
networks:

2
test/e2e-v2/cases/log/iotdb/docker-compose.yml
View File

@ -17,7 +17,7 @@ version: '2.1'
services:
iotdb:
image: apache/iotdb:0.12.3-node
image: apache/iotdb:0.12.4-node
expose:
- 6667
networks:

2
test/e2e-v2/cases/profile/iotdb/docker-compose.yml
View File

@ -17,7 +17,7 @@ version: '2.1'
services:
iotdb:
image: apache/iotdb:0.12.3-node
image: apache/iotdb:0.12.4-node
expose:
- 6667
networks:

2
test/e2e-v2/cases/storage/iotdb/docker-compose.yml
View File

@ -17,7 +17,7 @@ version: '2.1'
services:
iotdb:
image: apache/iotdb:0.12.3-node
image: apache/iotdb:0.12.4-node
expose:
- 6667
networks:

2
test/e2e-v2/cases/ttl/iotdb/docker-compose.yml
View File

@ -17,7 +17,7 @@ version: '2.1'
services:
iotdb:
image: apache/iotdb:0.12.3-node
image: apache/iotdb:0.12.4-node
expose:
- 6667
networks:

1
test/e2e-v2/script/docker-compose/base-compose.yml
View File

@ -34,6 +34,7 @@ services:
SW_JDBC_URL: jdbc:mysql://mysql:3306/swtest
SW_STORAGE_INFLUXDB_URL: http://influxdb:8086
SW_STORAGE_IOTDB_HOST: iotdb
SW_STORAGE_IOTDB_SESSIONPOOL_SIZE: 2
SW_CONFIG_ETCD_PERIOD: 1
SW_CONFIG_ETCD_ENDPOINTS: http://etcd:2379
SW_CLUSTER_ETCD_ENDPOINTS: http://etcd:2379

8
tools/dependencies/known-oap-backend-dependencies.txt
View File

@ -63,8 +63,8 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
iotdb-session-0.12.3.jar
iotdb-thrift-0.12.3.jar
iotdb-session-0.12.4.jar
iotdb-thrift-0.12.4.jar
j2objc-annotations-1.3.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
@ -144,7 +144,7 @@ protobuf-java-3.19.2.jar
protobuf-java-util-3.19.2.jar
reactive-streams-1.0.2.jar
retrofit-2.5.0.jar
service-rpc-0.12.3.jar
service-rpc-0.12.4.jar
simpleclient-0.6.0.jar
simpleclient_common-0.6.0.jar
simpleclient_hotspot-0.6.0.jar
@ -153,7 +153,7 @@ slf4j-api-1.7.30.jar
snakeyaml-1.28.jar
snappy-java-1.1.7.3.jar
swagger-annotations-1.6.3.jar
tsfile-0.12.3.jar
tsfile-0.12.4.jar
vavr-0.10.3.jar
vavr-match-0.10.3.jar
zipkin-2.9.1.jar