kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

BanyanDB: Support `hot/warm/cold` stages configuration. (#13144)

This commit is contained in:
Wan Kai 2025-03-28 11:41:05 +08:00 committed by GitHub
parent 1795110742
commit 27c37d3bf8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
12 changed files with 529 additions and 132 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

82
docs/en/banyandb/stages.md Normal file
View File

@ -0,0 +1,82 @@
# Data Lifecycle Stages(Hot/Warm/Cold)
Lifecycle Stages provide a mechanism to optimize storage costs and query performance based on the time granularity of records/metrics,
specially if you require keep mass of data for a long time.
The data lifecycle includes hot, warm, and cold stages. Each stage has different TTL settings and Segment Creation Policies.
Each group of records/metrics can be automatically migrated and stored in different stages according to the configuration.
## Stages Definition
- **hot**: The default first stage of data storage. The data is the newest, can be updated(metrics), and is most frequently queried.
- **warm**: Optional, the second stage of data storage. The data is less frequently queried than the hot stage, can't be updated, and still performs well.
- **cold**: Optional, the third stage of data storage. The data is rarely queried and is stored for a long time. The query performance is significantly lower than the hot/warm stages data.
If necessary, you also can jump the warm stage, and only use hot and cold stages. Then the data will be moved to the cold stage after the TTL of the hot stage.
## Configuration Guidelines
The lifecycle stages configuration is under each group settings of the `bydb.yml` file, for example, the `metricsMin` group:
```yaml
metricsMin:
# The settings for the default `hot` stage.
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS:7}
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SI_DAYS:3}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_TTL_DAYS:15}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SI_DAYS:5}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_TTL_DAYS:60}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_NODE_SELECTOR:"type=cold"}
```
1. **shardNum**: The number of shards for the group.
2. **segmentInterval**: The time interval in days for creating a new data segment.
- According to the freshness of the data, the `segmentInterval` days should: `hot` < `warm` < `cold`.
3. **ttl**: The time-to-live for data within the group, in days.
4. **enableWarmStage/enableColdStage**: Enable the warm/cold stage for the group.
- The `hot` stage is always enabled by default.
- If the `warm` stage is enabled, the data will be moved to the `warm` stage after the TTL of the `hot` stage.
- If the `cold` stage is enabled and `warm` stage is disabled, the data will be moved to the `cold` stage after the TTL of the `hot` stage.
- If both `warm` and `cold` stages are enabled, the data will be moved to the `warm` stage after the TTL of the `hot` stage, and then to the `cold` stage after the TTL of the `warm` stage.
- OAP will query the data from the `hot and warm` stage by default if the `warm` stage is enabled.
5. **nodeSelector**: Specifying target nodes for this stage.
For more details on configuring `segmentIntervalDays` and `ttlDays`, refer to the [BanyanDB Rotation](https://skywalking.apache.org/docs/skywalking-banyandb/latest/concept/rotation/) documentation.
## Procedure and The TTL for Stages
About the TTL can refer to [Progressive TTL](ttl.md).
The following diagram illustrates the lifecycle stages, assuming the TTL settings for hot, warm and cold stages are `TTL1, TTL2 and TTL3` days respectively:
```mermaid
sequenceDiagram
Data(T0) ->> Hot Data(TTL1): Input
Hot Data(TTL1) -->+ Hot Data(TTL1): TTL1
Hot Data(TTL1) ->>- Warm Data(TTL2): Migrate
Warm Data(TTL2) -->+ Warm Data(TTL2): TTL2 - TTL1
Warm Data(TTL2) ->>- Cold Data(TTL3): Migrate
Cold Data(TTL3) -->+ Cold Data(TTL3): TTL3 - TTL2
Cold Data(TTL3) ->>- Deleted: Delete
Data(T0) --> Hot Data(TTL1): Live TTL1 Days
Data(T0) --> Warm Data(TTL2): Live TTL2 Days
Data(T0) --> Cold Data(TTL3): Live TTL3 Days
```
- When the data is input, it will be stored in the hot stage and live for `TTL1` days.
- After `TTL1` days, the data will be migrated to the warm stage and live until `TTL2` days. **It means data will in this stage for (TTL2 - TTL1) days**.
- After `TTL2` days, the data will be migrated to the cold stage and live until `TTL3` days. **It means data will in this stage for (TTL3 - TTL2) days**.
- After `TTL3` days, the data will be deleted.
- The data will live for `TTL3` days in total.
## Querying
- According to the lifecycle stages configuration, OAP will query the data from the `hot and warm` stage by default if the `warm` stage is enabled.
Otherwise, OAP will query the data from the `hot` stage only.
- If the `cold` stage is enabled, for better query performance, you should specify the stage in the query and OAP will limit the query time range.

18
docs/en/banyandb/ttl.md
View File

@ -36,17 +36,17 @@ For both standard and super datasets:
Configure `SIDays` and `TTLDays` based on data retention and query requirements. Recommended settings include:
| Group | `SIDays` | `TTLDays` |
|----------------|----------|-----------|
| Minute (`gmMinute`) | 1 | 7 |
| Hour (`gmHour`) | 5 | 15 |
| Day (`gmDay`) | 15 | 15 |
| Index (`gmIndex`) | 15 | 15 |
|------------------------|----------|-----------|
| Minute (`metricsMin`) | 1 | 7 |
| Hour (`metricsHour`) | 5 | 15 |
| Day (`metricsDay`) | 15 | 15 |
| Index (`metadata`) | 15 | 15 |
**Group Descriptions:**
- **Minute (`gmMinute`)**: Stores metrics with a 1-minute granularity. Suitable for recent data queries requiring minute-level detail. Consequently, it has shorter `SIDays` and `TTLDays` compared to other groups.
- **Hour (`gmHour`)**: Stores metrics with a 1-hour granularity. Designed for queries that need hour-level detail over a longer period than minute-level data.
- **Day (`gmDay`)**: Stores metrics with a 1-day granularity. This group handles the longest segment intervals and TTLs among all granularity groups.
- **Index (`gmIndex`)**: Stores metrics used solely for indexing without value columns. Since queries often scan all segments in the `index` group, it shares the same `SIDays` and `TTLDays` as the `day` group to optimize performance. This group's `TTL` must be set to the **max** value of all groups.
- **Minute (`metricsMin`)**: Stores metrics with a 1-minute granularity. Suitable for recent data queries requiring minute-level detail. Consequently, it has shorter `SIDays` and `TTLDays` compared to other groups.
- **Hour (`metricsHour`)**: Stores metrics with a 1-hour granularity. Designed for queries that need hour-level detail over a longer period than minute-level data.
- **Day (`metricsDay`)**: Stores metrics with a 1-day granularity. This group handles the longest segment intervals and TTLs among all granularity groups.
- **Index (`metadata`)**: Stores metrics used solely for indexing without value columns. Since queries often scan all segments in the `index` group, it shares the same `SIDays` and `TTLDays` as the `day` group to optimize performance. This group's `TTL` must be set to the **max** value of all groups.
For more details on configuring `segmentIntervalDays` and `ttlDays`, refer to the [BanyanDB Rotation](https://skywalking.apache.org/docs/skywalking-banyandb/latest/concept/rotation/) documentation.

3
docs/en/changes/changes.md
View File

@ -4,10 +4,13 @@
#### OAP Server
* BanyanDB: Support `hot/warm/cold` stages configuration.
#### UI
#### Documentation
* BanyanDB: Add `Data Lifecycle Stages(Hot/Warm/Cold)` documentation.
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/230?closed=1)

158
docs/en/setup/backend/configuration-vocabulary.md
View File

@ -354,50 +354,122 @@ It divided into several modules, each of which has its own settings. The followi
Since 10.2.0, the banyandb configuration is separated to an independent configuration file: `bydb.yaml`.
The following table lists the configuration items:
| Part | Group | Settings | Value(s) and Explanation | System Environment Variable¹ | Default |
|--------|---------------|---------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|-----------------|
| global | - | - | The global settings for the whole BanyanDB. | - | - |
| - | - | targets | Hosts with ports of the BanyanDB. | SW_STORAGE_BANYANDB_TARGETS | 127.0.0.1:17912 |
| - | - | maxBulkSize | The maximum size of write entities in a single batch write call. | SW_STORAGE_BANYANDB_MAX_BULK_SIZE | 10000 |
| - | - | flushInterval | Period of flush interval. In the timeunit of seconds. | SW_STORAGE_BANYANDB_FLUSH_INTERVAL | 15 |
| - | - | flushTimeout | The timeout seconds of a bulk flush. | SW_STORAGE_BANYANDB_FLUSH_TIMEOUT | 10 |
| - | - | concurrentWriteThreads | Concurrent consumer threads for batch writing. | SW_STORAGE_BANYANDB_CONCURRENT_WRITE_THREADS | 15 |
| - | - | profileTaskQueryMaxSize | Max size of ProfileTask to be fetched. | SW_STORAGE_BANYANDB_PROFILE_TASK_QUERY_MAX_SIZE | 200 |
| - | - | resultWindowMaxSize | The maximum size of dataset when the OAP loads cache, such as network aliases. | SW_STORAGE_BANYAND_QUERY_MAX_WINDOW_SIZE | 10000 |
| - | - | metadataQueryMaxSize | The maximum size of metadata per query. | SW_STORAGE_BANYAND_QUERY_MAX_SIZE | 10000 |
| - | - | segmentQueryMaxSize | The maximum size of trace segments per query. | SW_STORAGE_BANYAND_QUERY_SEGMENT_SIZE | 200 |
| - | - | asyncProfilerTaskQueryMaxSize | Max size of AsyncProfilerTask to be fetched. | SW_STORAGE_BANYANDB_ASYNC_PROFILER_TASK_QUERY_MAX_SIZE | 200 |
| - | - | profileDataQueryScrollBatchSize | The batch size of query profiling data. | SW_STORAGE_BANYAND_QUERY_PROFILE_DATA_BATCH_SIZE | 100 |
| - | - | sslTrustCAPath | If the BanyanDB server is configured with TLS, config the TLS cert file path and open tls connection. | SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH | - |
| groups | - | - | The settings for each group. | - | - |
| - | recordsNormal | - | The group for records not specified in "super". Each dataset will be grouped under a single group named "normal". | - | - |
| - | - | shardNum | Shards Number for normal records group. | SW_STORAGE_BANYANDB_GR_NORMAL_SHARD_NUM | 1 |
| - | - | segmentInterval | Shards Number for normal records group. | SW_STORAGE_BANYANDB_GR_NORMAL_SI_DAYS | 1 |
| - | - | ttl | Shards Number for normal records group. | SW_STORAGE_BANYANDB_GR_NORMAL_TTL_DAYS | 3 |
| - | recordsSuper | - | `super` is a special dataset designed to store trace or log data that is too large for normal datasets.Each super dataset will be a separate group in BanyanDB, following the settings defined in the "super" section.| - | - |
| - | - | shardNum | Shards Number for super records group. | SW_STORAGE_BANYANDB_GR_SUPER_SHARD_NUM | 2 |
| - | - | segmentInterval | Shards Number for super records group. | SW_STORAGE_BANYANDB_GR_SUPER_SI_DAYS | 1 |
| - | - | ttl | Shards Number for super records group. | SW_STORAGE_BANYANDB_GR_SUPER_TTL_DAYS | 3 |
| - | metricsMin | - | The group for minute granularity metrics group. | - | - |
| - | - | shardNum | Shards Number for minute granularity metrics group. | SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM | 2 |
| - | - | segmentInterval | Shards Number for minute granularity metrics group. | SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS | 1 |
| - | - | ttl | Shards Number for minute granularity metrics group. | SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS | 7 |
| - | metricsHour | - | The group for hour granularity metrics. | - | - |
| - | - | shardNum | Shards Number for hour granularity metrics group. | SW_STORAGE_BANYANDB_GM_HOUR_SHARD_NUM | 1 |
| - | - | segmentInterval | Shards Number for hour granularity metrics group. | SW_STORAGE_BANYANDB_GM_HOUR_SI_DAYS | 5 |
| - | - | ttl | Shards Number for hour granularity metrics group. | SW_STORAGE_BANYANDB_GM_HOUR_TTL_DAYS | 15 |
| - | metricsDay | - | The group for day granularity metrics. | - | - |
| - | - | shardNum | Shards Number for day granularity metrics group. | SW_STORAGE_BANYANDB_GM_DAY_SHARD_NUM | 1 |
| - | - | segmentInterval | Shards Number for day granularity metrics group. | SW_STORAGE_BANYANDB_GM_DAY_SI_DAYS | 15 |
| - | - | ttl | Shards Number for day granularity metrics group. | SW_STORAGE_BANYANDB_GM_DAY_TTL_DAYS | 15 |
| - | metadata | - | The `index` group is designed to store metrics that are used for indexing without value columns. Such as `service_traffic`, `network_address_alias`, etc. Since BanyanDB *0.8.0*. | - | - |
| - | - | shardNum | Shards Number for metadata `index` group. | SW_STORAGE_BANYANDB_GM_INDEX_SHARD_NUM | 2 |
| - | - | segmentInterval | Shards Number for metadata `index` group. | SW_STORAGE_BANYANDB_GM_INDEX_SI_DAYS | 15 |
| - | - | ttl | Shards Number for metadata `index` group. | SW_STORAGE_BANYANDB_GM_INDEX_TTL_DAYS | 15 |
| - | property | - | The group settings of property, such as UI and profiling. | - | - |
| - | - | shardNum | Shards Number for property group. | SW_STORAGE_BANYANDB_GP_PROPERTY_SHARD_NUM | 1 |
### Global Configuration
The global settings for the whole BanyanDB:
### Note
| Settings | Value(s) and Explanation | System Environment Variable¹ | Default |
|---------------------------------|-------------------------------------------------------------------------------------------------------|---------------------------------------------------------|-----------------|
| targets | Hosts with ports of the BanyanDB. | SW_STORAGE_BANYANDB_TARGETS | 127.0.0.1:17912 |
| maxBulkSize | The maximum size of write entities in a single batch write call. | SW_STORAGE_BANYANDB_MAX_BULK_SIZE | 10000 |
| flushInterval | Period of flush interval. In the timeunit of seconds. | SW_STORAGE_BANYANDB_FLUSH_INTERVAL | 15 |
| flushTimeout | The timeout seconds of a bulk flush. | SW_STORAGE_BANYANDB_FLUSH_TIMEOUT | 10 |
| concurrentWriteThreads | Concurrent consumer threads for batch writing. | SW_STORAGE_BANYANDB_CONCURRENT_WRITE_THREADS | 15 |
| profileTaskQueryMaxSize | Max size of ProfileTask to be fetched. | SW_STORAGE_BANYANDB_PROFILE_TASK_QUERY_MAX_SIZE | 200 |
| resultWindowMaxSize | The maximum size of dataset when the OAP loads cache, such as network aliases. | SW_STORAGE_BANYAND_QUERY_MAX_WINDOW_SIZE | 10000 |
| metadataQueryMaxSize | The maximum size of metadata per query. | SW_STORAGE_BANYAND_QUERY_MAX_SIZE | 10000 |
| segmentQueryMaxSize | The maximum size of trace segments per query. | SW_STORAGE_BANYAND_QUERY_SEGMENT_SIZE | 200 |
| asyncProfilerTaskQueryMaxSize | Max size of AsyncProfilerTask to be fetched. | SW_STORAGE_BANYANDB_ASYNC_PROFILER_TASK_QUERY_MAX_SIZE | 200 |
| profileDataQueryScrollBatchSize | The batch size of query profiling data. | SW_STORAGE_BANYAND_QUERY_PROFILE_DATA_BATCH_SIZE | 100 |
| sslTrustCAPath | If the BanyanDB server is configured with TLS, config the TLS cert file path and open tls connection. | SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH | - |
### Group Configuration
The settings for each group:
The `hot` stage is enabled by default for all groups.
If the `warm` stage is enabled, the data will be moved to the `warm` stage after the TTL of the `hot` stage.
If the `cold` stage is enabled and `warm` stage is disabled, the data will be moved to the `cold` stage after the TTL of the `hot` stage.
If both `warm` and `cold` stages are enabled, the data will be moved to the `warm` stage after the TTL of the `hot` stage, and then to the `cold` stage after the TTL of the `warm` stage.
OAP will query the data from the "hot and warm" stage by default if the "warm" stage is enabled.
| Group | Settings | Stage Settings | Value(s) and Explanation | System Environment Variable¹ | Default |
|---------------|-----------------|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------|-----------|
| recordsNormal | - | - | The group for records not specified in `super`. Each dataset will be grouped under a single group named `normal`. | - | - |
| - | shardNum | - | Shards Number for normal records group. | SW_STORAGE_BANYANDB_GR_NORMAL_SHARD_NUM | 1 |
| - | segmentInterval | - | Segment Interval Days for normal records group. | SW_STORAGE_BANYANDB_GR_NORMAL_SI_DAYS | 1 |
| - | ttl | - | TTL Days for normal records group. | SW_STORAGE_BANYANDB_GR_NORMAL_TTL_DAYS | 3 |
| - | enableWarmStage | - | Activate warm stage for normal records group. | SW_STORAGE_BANYANDB_GR_NORMAL_ENABLE_WARM_STAGE | false |
| - | enableColdStage | - | Activate cold stage for normal records group. | SW_STORAGE_BANYANDB_GR_NORMAL_ENABLE_COLD_STAGE | false |
| - | warm | - | The warm stage settings. | - | - |
| - | - | shardNum | Shards Number for this stage. | SW_STORAGE_BANYANDB_GR_NORMAL_WARM_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GR_NORMAL_WARM_SI_DAYS | 2 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GR_NORMAL_WARM_TTL_DAYS | 7 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GR_NORMAL_WARM_NODE_SELECTOR | type=warm |
| - | cold | - | The cold stage settings. | - | - |
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_GR_NORMAL_COLD_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GR_NORMAL_COLD_SI_DAYS | 3 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GR_NORMAL_COLD_TTL_DAYS | 30 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GR_NORMAL_COLD_NODE_SELECTOR | type=cold |
| recordsSuper | - | - | `super` is a special dataset designed to store trace or log data that is too large for normal datasets.Each super dataset will be a separate group in BanyanDB, following the settings defined in the "super" section. | - | - |
| - | shardNum | - | Shards Number for super records group. | SW_STORAGE_BANYANDB_GR_SUPER_SHARD_NUM | 2 |
| - | segmentInterval | - | Segment Interval Days for super records group. | SW_STORAGE_BANYANDB_GR_SUPER_SI_DAYS | 1 |
| - | ttl | - | TTL Days for super records group. | SW_STORAGE_BANYANDB_GR_SUPER_TTL_DAYS | 3 |
| - | enableWarmStage | - | Activate warm stage for super records group. | SW_STORAGE_BANYANDB_GR_SUPER_ENABLE_WARM_STAGE | false |
| - | enableColdStage | - | Activate cold stage for super records group. | SW_STORAGE_BANYANDB_GR_SUPER_ENABLE_COLD_STAGE | false |
| - | warm | - | The warm stage settings. | - | - |
| - | - | shardNum | Shards Number for this stage. | SW_STORAGE_BANYANDB_GR_SUPER_WARM_SHARD_NUM | 2 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GR_SUPER_WARM_SI_DAYS | 1 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GR_SUPER_WARM_TTL_DAYS | 7 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GR_SUPER_WARM_NODE_SELECTOR | type=warm |
| - | cold | - | The cold stage settings. | - | - |
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_GR_SUPER_COLD_SHARD_NUM | 2 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GR_SUPER_COLD_SI_DAYS | 1 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GR_SUPER_COLD_TTL_DAYS | 30 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GR_SUPER_COLD_NODE_SELECTOR | type=cold |
| metricsMin | - | - | The group for minute granularity metrics group. | - | - |
| - | shardNum | - | Shards Number for minute granularity metrics group. | SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM | 2 |
| - | segmentInterval | - | Segment Interval Days for minute granularity metrics group. | SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS | 1 |
| - | ttl | - | TTL Days for minute granularity metrics group. | SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS | 7 |
| - | enableWarmStage | - | Activate warm stage for minute granularity metrics group. | SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_WARM_STAGE | false |
| - | enableColdStage | - | Activate cold stage for minute granularity metrics group. | SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_COLD_STAGE | false |
| - | warm | - | The warm stage settings. | - | - |
| - | - | shardNum | Shards Number for this stage. | SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SHARD_NUM | 2 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SI_DAYS | 3 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GM_MINUTE_WARM_TTL_DAYS | 15 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GM_MINUTE_WARM_NODE_SELECTOR | type=warm |
| - | cold | - | The cold stage settings. | - | - |
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SHARD_NUM | 2 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SI_DAYS | 5 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GM_MINUTE_COLD_TTL_DAYS | 60 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GM_MINUTE_COLD_NODE_SELECTOR | type=cold |
| metricsHour | - | - | The group for hour granularity metrics. | - | - |
| - | shardNum | - | Shards Number for hour granularity metrics group. | SW_STORAGE_BANYANDB_GM_HOUR_SHARD_NUM | 1 |
| - | segmentInterval | - | Segment Interval Days for hour granularity metrics group. | SW_STORAGE_BANYANDB_GM_HOUR_SI_DAYS | 5 |
| - | ttl | - | TTL Days for hour granularity metrics group. | SW_STORAGE_BANYANDB_GM_HOUR_TTL_DAYS | 15 |
| - | enableWarmStage | - | Activate warm stage for hour granularity metrics group. | SW_STORAGE_BANYANDB_GM_HOUR_ENABLE_WARM_STAGE | false |
| - | enableColdStage | - | Activate cold stage for hour granularity metrics group. | SW_STORAGE_BANYANDB_GM_HOUR_ENABLE_COLD_STAGE | false |
| - | warm | - | The warm stage settings. | - | - |
| - | - | shardNum | Shards Number for this stage. | SW_STORAGE_BANYANDB_GM_HOUR_WARM_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GM_HOUR_WARM_SI_DAYS | 7 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GM_HOUR_WARM_TTL_DAYS | 30 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GM_HOUR_WARM_NODE_SELECTOR | type=warm |
| - | cold | - | The cold stage settings. | - | - |
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_GM_HOUR_COLD_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GM_HOUR_COLD_SI_DAYS | 15 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GM_HOUR_COLD_TTL_DAYS | 120 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GM_HOUR_COLD_NODE_SELECTOR | type=cold |
| metricsDay | - | - | The group for day granularity metrics. | - | - |
| - | shardNum | - | Shards Number for day granularity metrics group. | SW_STORAGE_BANYANDB_GM_DAY_SHARD_NUM | 1 |
| - | segmentInterval | - | Segment Interval Days for day granularity metrics group. | SW_STORAGE_BANYANDB_GM_DAY_SI_DAYS | 15 |
| - | ttl | - | TTL Days for day granularity metrics group. | SW_STORAGE_BANYANDB_GM_DAY_TTL_DAYS | 15 |
| - | enableWarmStage | - | Activate warm stage for day granularity metrics group. | SW_STORAGE_BANYANDB_GM_DAY_ENABLE_WARM_STAGE | false |
| - | enableColdStage | - | Activate cold stage for day granularity metrics group. | SW_STORAGE_BANYANDB_GM_DAY_ENABLE_COLD_STAGE | false |
| - | warm | - | The warm stage settings. | - | - |
| - | - | shardNum | Shards Number for this stage. | SW_STORAGE_BANYANDB_GM_DAY_WARM_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GM_DAY_WARM_SI_DAYS | 15 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GM_DAY_WARM_TTL_DAYS | 30 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GM_DAY_WARM_NODE_SELECTOR | type=warm |
| - | cold | - | The cold stage settings. | - | - |
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_GM_DAY_COLD_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_GM_DAY_COLD_SI_DAYS | 15 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_GM_DAY_COLD_TTL_DAYS | 120 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_GM_DAY_COLD_NODE_SELECTOR | type=cold |
| metadata | - | - | The `index` group is designed to store metrics that are used for indexing without value columns. Such as `service_traffic`, `network_address_alias`, etc. Since BanyanDB *0.8.0*. | - | - |
| - | shardNum | - | Shards Number for metadata `index` group. | SW_STORAGE_BANYANDB_GM_INDEX_SHARD_NUM | 2 |
| - | segmentInterval | - | Segment Interval Days for metadata `index` group. | SW_STORAGE_BANYANDB_GM_INDEX_SI_DAYS | 15 |
| - | ttl | - | TTL Days for metadata `index` group. | SW_STORAGE_BANYANDB_GM_INDEX_TTL_DAYS | 15 |
| property | - | - | The group settings of property, such as UI and profiling. | - | - |
| - | shardNum | - | Shards Number for property group. | SW_STORAGE_BANYANDB_GP_PROPERTY_SHARD_NUM | 1 |
## Note
¹ System Environment Variable name could be declared and changed in `application.yml/bydb.yaml`. The names listed here are simply
provided in the default `application.yml/bydb.yaml` file.

86
docs/en/setup/backend/storages/banyandb.md
View File

@ -32,6 +32,21 @@ storage:
Since 10.2.0, the banyandb configuration is separated to an independent configuration file: `bydb.yaml`:
```yaml
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
global:
# Targets is the list of BanyanDB servers, separated by commas.
# Each target is a BanyanDB server in the format of `host:port`.
@ -71,15 +86,46 @@ groups:
# The "normal" section defines settings for datasets not specified in "super".
# Each dataset will be grouped under a single group named "normal".
recordsNormal:
# The settings for the default "hot" stage.
shardNum: ${SW_STORAGE_BANYANDB_GR_NORMAL_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_NORMAL_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GR_NORMAL_TTL_DAYS:3}
# If the "warm" stage is enabled, the data will be moved to the "warm" stage after the TTL of the "hot" stage.
# If the "cold" stage is enabled and "warm" stage is disabled, the data will be moved to the "cold" stage after the TTL of the "hot" stage.
# If both "warm" and "cold" stages are enabled, the data will be moved to the "warm" stage after the TTL of the "hot" stage, and then to the "cold" stage after the TTL of the "warm" stage.
# OAP will query the data from the "hot and warm" stage by default if the "warm" stage is enabled.
enableWarmStage: ${SW_STORAGE_BANYANDB_GR_NORMAL_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GR_NORMAL_ENABLE_COLD_STAGE:false}
# The settings for the "warm" stage.
warm:
shardNum: ${SW_STORAGE_BANYANDB_GR_NORMAL_WARM_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_NORMAL_WARM_SI_DAYS:2}
ttl: ${SW_STORAGE_BANYANDB_GR_NORMAL_WARM_TTL_DAYS:7}
nodeSelector: ${SW_STORAGE_BANYANDB_GR_NORMAL_WARM_NODE_SELECTOR:"type=warm"}
# The settings for the "cold" stage.
cold:
shardNum: ${SW_STORAGE_BANYANDB_GR_NORMAL_COLD_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_NORMAL_COLD_SI_DAYS:3}
ttl: ${SW_STORAGE_BANYANDB_GR_NORMAL_COLD_TTL_DAYS:30}
nodeSelector: ${SW_STORAGE_BANYANDB_GR_NORMAL_COLD_NODE_SELECTOR:"type=cold"}
# "super" is a special dataset designed to store trace or log data that is too large for normal datasets.
# Each super dataset will be a separate group in BanyanDB, following the settings defined in the "super" section.
recordsSuper:
shardNum: ${SW_STORAGE_BANYANDB_GR_SUPER_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_SUPER_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GR_SUPER_TTL_DAYS:3}
enableWarmStage: ${SW_STORAGE_BANYANDB_GR_SUPER_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GR_SUPER_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GR_SUPER_WARM_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_SUPER_WARM_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GR_SUPER_WARM_TTL_DAYS:7}
nodeSelector: ${SW_STORAGE_BANYANDB_GR_SUPER_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GR_SUPER_COLD_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_SUPER_COLD_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GR_SUPER_COLD_TTL_DAYS:30}
nodeSelector: ${SW_STORAGE_BANYANDB_GR_SUPER_COLD_NODE_SELECTOR:"type=cold"}
# The group settings of metrics.
#
# OAP stores metrics based its granularity.
@ -90,14 +136,50 @@ groups:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS:7}
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SI_DAYS:3}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_TTL_DAYS:15}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SI_DAYS:5}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_TTL_DAYS:60}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_NODE_SELECTOR:"type=cold"}
metricsHour:
shardNum: ${SW_STORAGE_BANYANDB_GM_HOUR_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_HOUR_SI_DAYS:5}
ttl: ${SW_STORAGE_BANYANDB_GM_HOUR_TTL_DAYS:15}
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_HOUR_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GM_HOUR_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GM_HOUR_WARM_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_HOUR_WARM_SI_DAYS:7}
ttl: ${SW_STORAGE_BANYANDB_GM_HOUR_WARM_TTL_DAYS:30}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_HOUR_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GM_HOUR_COLD_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_HOUR_COLD_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_HOUR_COLD_TTL_DAYS:120}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_HOUR_COLD_NODE_SELECTOR:"type=cold"}
metricsDay:
shardNum: ${SW_STORAGE_BANYANDB_GM_DAY_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_DAY_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_DAY_TTL_DAYS:15}
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_DAY_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GM_DAY_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GM_DAY_WARM_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_DAY_WARM_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_DAY_WARM_TTL_DAYS:30}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_DAY_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GM_DAY_COLD_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_DAY_COLD_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_DAY_COLD_TTL_DAYS:120}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_DAY_COLD_NODE_SELECTOR:"type=cold"}
# If the metrics is marked as "index_mode", the metrics will be stored in the "index" group.
# The "index" group is designed to store metrics that are used for indexing without value columns.
# Such as `service_traffic`, `network_address_alias`, etc.
@ -106,9 +188,11 @@ groups:
shardNum: ${SW_STORAGE_BANYANDB_GM_INDEX_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_INDEX_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_INDEX_TTL_DAYS:15}
# The group settings of property such as UI and profiling.
# The group settings of property, such as UI and profiling.
property:
shardNum: ${SW_STORAGE_BANYANDB_GP_PROPERTY_SHARD_NUM:1}
```
### Installation Modes

2
docs/menu.yml
View File

@ -204,6 +204,8 @@ catalog:
catalog:
- name: "Progressive TTL"
path: "/en/banyandb/ttl"
- name: "Data Lifecycle Stages(Hot/Warm/Cold)"
path: "/en/banyandb/lifecycle-stages"
- name: "Tracing"
catalog:
- name: "Trace Sampling"

67
oap-server/server-starter/src/main/resources/bydb.yml
View File

@ -52,15 +52,46 @@ groups:
# The "normal" section defines settings for datasets not specified in "super".
# Each dataset will be grouped under a single group named "normal".
recordsNormal:
# The settings for the default "hot" stage.
shardNum: ${SW_STORAGE_BANYANDB_GR_NORMAL_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_NORMAL_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GR_NORMAL_TTL_DAYS:3}
# If the "warm" stage is enabled, the data will be moved to the "warm" stage after the TTL of the "hot" stage.
# If the "cold" stage is enabled and "warm" stage is disabled, the data will be moved to the "cold" stage after the TTL of the "hot" stage.
# If both "warm" and "cold" stages are enabled, the data will be moved to the "warm" stage after the TTL of the "hot" stage, and then to the "cold" stage after the TTL of the "warm" stage.
# OAP will query the data from the "hot and warm" stage by default if the "warm" stage is enabled.
enableWarmStage: ${SW_STORAGE_BANYANDB_GR_NORMAL_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GR_NORMAL_ENABLE_COLD_STAGE:false}
# The settings for the "warm" stage.
warm:
shardNum: ${SW_STORAGE_BANYANDB_GR_NORMAL_WARM_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_NORMAL_WARM_SI_DAYS:2}
ttl: ${SW_STORAGE_BANYANDB_GR_NORMAL_WARM_TTL_DAYS:7}
nodeSelector: ${SW_STORAGE_BANYANDB_GR_NORMAL_WARM_NODE_SELECTOR:"type=warm"}
# The settings for the "cold" stage.
cold:
shardNum: ${SW_STORAGE_BANYANDB_GR_NORMAL_COLD_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_NORMAL_COLD_SI_DAYS:3}
ttl: ${SW_STORAGE_BANYANDB_GR_NORMAL_COLD_TTL_DAYS:30}
nodeSelector: ${SW_STORAGE_BANYANDB_GR_NORMAL_COLD_NODE_SELECTOR:"type=cold"}
# "super" is a special dataset designed to store trace or log data that is too large for normal datasets.
# Each super dataset will be a separate group in BanyanDB, following the settings defined in the "super" section.
recordsSuper:
shardNum: ${SW_STORAGE_BANYANDB_GR_SUPER_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_SUPER_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GR_SUPER_TTL_DAYS:3}
enableWarmStage: ${SW_STORAGE_BANYANDB_GR_SUPER_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GR_SUPER_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GR_SUPER_WARM_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_SUPER_WARM_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GR_SUPER_WARM_TTL_DAYS:7}
nodeSelector: ${SW_STORAGE_BANYANDB_GR_SUPER_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GR_SUPER_COLD_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GR_SUPER_COLD_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GR_SUPER_COLD_TTL_DAYS:30}
nodeSelector: ${SW_STORAGE_BANYANDB_GR_SUPER_COLD_NODE_SELECTOR:"type=cold"}
# The group settings of metrics.
#
# OAP stores metrics based its granularity.
@ -71,14 +102,50 @@ groups:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS:7}
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SI_DAYS:3}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_TTL_DAYS:15}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SI_DAYS:5}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_TTL_DAYS:60}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_NODE_SELECTOR:"type=cold"}
metricsHour:
shardNum: ${SW_STORAGE_BANYANDB_GM_HOUR_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_HOUR_SI_DAYS:5}
ttl: ${SW_STORAGE_BANYANDB_GM_HOUR_TTL_DAYS:15}
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_HOUR_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GM_HOUR_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GM_HOUR_WARM_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_HOUR_WARM_SI_DAYS:7}
ttl: ${SW_STORAGE_BANYANDB_GM_HOUR_WARM_TTL_DAYS:30}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_HOUR_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GM_HOUR_COLD_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_HOUR_COLD_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_HOUR_COLD_TTL_DAYS:120}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_HOUR_COLD_NODE_SELECTOR:"type=cold"}
metricsDay:
shardNum: ${SW_STORAGE_BANYANDB_GM_DAY_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_DAY_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_DAY_TTL_DAYS:15}
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_DAY_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GM_DAY_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GM_DAY_WARM_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_DAY_WARM_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_DAY_WARM_TTL_DAYS:30}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_DAY_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GM_DAY_COLD_SHARD_NUM:1}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_DAY_COLD_SI_DAYS:15}
ttl: ${SW_STORAGE_BANYANDB_GM_DAY_COLD_TTL_DAYS:120}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_DAY_COLD_NODE_SELECTOR:"type=cold"}
# If the metrics is marked as "index_mode", the metrics will be stored in the "index" group.
# The "index" group is designed to store metrics that are used for indexing without value columns.
# Such as `service_traffic`, `network_address_alias`, etc.

87
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBConfigLoader.java
View File

@ -57,26 +57,8 @@ public class BanyanDBConfigLoader {
Map<String, Properties> configProperties = new HashMap<>();
configMap.forEach((part, c) -> {
final Properties properties = new Properties();
if (c != null) {
for (Map.Entry<String, ?> entry : c.entrySet()) {
String propertyName = entry.getKey();
Object propertyValue = entry.getValue();
if (propertyValue instanceof Map) {
Properties subProperties = new Properties();
for (Map.Entry<String, ?> e : ((Map<String, ?>) propertyValue).entrySet()) {
String key = e.getKey();
Object value = e.getValue();
subProperties.put(key, value);
replacePropertyAndLog(key, value, subProperties, this.moduleProvider.name(), yaml);
}
properties.put(propertyName, subProperties);
} else {
properties.put(propertyName, propertyValue);
replacePropertyAndLog(
propertyName, propertyValue, properties, this.moduleProvider.name(), yaml);
}
}
final Properties properties = parseConfig(c);
configProperties.put(part, properties);
}
});
@ -87,26 +69,41 @@ public class BanyanDBConfigLoader {
moduleProvider.name()
);
Properties groups = configProperties.get("groups");
Properties recordsNormal = (Properties) groups.get("recordsNormal");
copyProperties(
config.getRecordsNormal(), (Properties) groups.get("recordsNormal"),
config.getRecordsNormal(), recordsNormal,
moduleProvider.getModule().name(), moduleProvider.name()
);
copyStages(recordsNormal, config.getRecordsNormal());
Properties recordsSupper = (Properties) groups.get("recordsSuper");
copyProperties(
config.getRecordsSuper(), (Properties) groups.get("recordsSuper"),
config.getRecordsSuper(), recordsSupper,
moduleProvider.getModule().name(), moduleProvider.name()
);
copyStages(recordsSupper, config.getRecordsSuper());
Properties metricsMin = (Properties) groups.get("metricsMin");
copyProperties(
config.getMetricsMin(), (Properties) groups.get("metricsMin"),
config.getMetricsMin(), metricsMin,
moduleProvider.getModule().name(), moduleProvider.name()
);
copyStages(metricsMin, config.getMetricsMin());
Properties metricsHour = (Properties) groups.get("metricsHour");
copyProperties(
config.getMetricsHour(), (Properties) groups.get("metricsHour"),
config.getMetricsHour(), metricsHour,
moduleProvider.getModule().name(), moduleProvider.name()
);
copyStages(metricsHour, config.getMetricsHour());
Properties metricsDay = (Properties) groups.get("metricsDay");
copyProperties(
config.getMetricsDay(), (Properties) groups.get("metricsDay"),
config.getMetricsDay(), metricsDay,
moduleProvider.getModule().name(), moduleProvider.name()
);
copyStages(metricsDay, config.getMetricsDay());
copyProperties(
config.getMetadata(), (Properties) groups.get("metadata"),
moduleProvider.getModule().name(), moduleProvider.name()
@ -120,4 +117,46 @@ public class BanyanDBConfigLoader {
}
return config;
}
private Properties parseConfig(final Map<String, ?> config) {
final Properties properties = new Properties();
for (Map.Entry<String, ?> entry : config.entrySet()) {
String propertyName = entry.getKey();
Object propertyValue = entry.getValue();
if (propertyValue instanceof Map) {
Properties subProperties = parseConfig((Map<String, ?>) propertyValue);
properties.put(propertyName, subProperties);
} else {
properties.put(propertyName, propertyValue);
replacePropertyAndLog(
propertyName, propertyValue, properties, this.moduleProvider.name(), yaml);
}
}
return properties;
}
private void copyStages(final Properties group,
final BanyanDBStorageConfig.GroupResource groupResource) throws IllegalAccessException {
if (groupResource.isEnableWarmStage()) {
BanyanDBStorageConfig.Stage warm = new BanyanDBStorageConfig.Stage();
warm.setName(BanyanDBStorageConfig.StageName.warm);
copyProperties(
warm, (Properties) group.get(BanyanDBStorageConfig.StageName.warm.name()),
moduleProvider.getModule().name(), moduleProvider.name()
);
groupResource.getAdditionalLifecycleStages().add(warm);
groupResource.getDefaultQueryStages().add(BanyanDBStorageConfig.StageName.warm.name());
}
if (groupResource.isEnableColdStage()) {
BanyanDBStorageConfig.Stage cold = new BanyanDBStorageConfig.Stage();
cold.setName(BanyanDBStorageConfig.StageName.cold);
cold.setClose(true);
copyProperties(
cold, (Properties) group.get(BanyanDBStorageConfig.StageName.cold.name()),
moduleProvider.getModule().name(), moduleProvider.name()
);
groupResource.getAdditionalLifecycleStages().add(cold);
}
}
}

57
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java
View File

@ -234,9 +234,29 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
*/
private boolean checkGroup(MetadataRegistry.SchemaMetadata metadata, BanyanDBClient client) throws BanyanDBException {
Group g = client.findGroup(metadata.getGroup());
return g.getResourceOpts().getShardNum() != metadata.getShard()
|| g.getResourceOpts().getSegmentInterval().getNum() != metadata.getSegmentIntervalDays()
|| g.getResourceOpts().getTtl().getNum() != metadata.getTtlDays();
if (g.getResourceOpts().getShardNum() != metadata.getResource().getShardNum()
|| g.getResourceOpts().getSegmentInterval().getNum() != metadata.getResource().getSegmentInterval()
|| g.getResourceOpts().getTtl().getNum() != metadata.getResource().getTtl()) {
return true;
}
if (g.getResourceOpts().getStagesCount() != metadata.getResource().getAdditionalLifecycleStages().size()) {
return true;
}
for (int i = 0; i < g.getResourceOpts().getStagesCount(); i++) {
BanyandbCommon.LifecycleStage stage = g.getResourceOpts().getStages(i);
BanyanDBStorageConfig.Stage stageConfig = metadata.getResource().getAdditionalLifecycleStages().get(i);
if (!stage.getName().equals(stageConfig.getName().name())
|| stage.getShardNum() != stageConfig.getShardNum()
|| stage.getSegmentInterval().getNum() != stageConfig.getSegmentInterval()
|| stage.getTtl().getNum() != stageConfig.getTtl()
|| !stage.getNodeSelector().equals(stageConfig.getNodeSelector())
|| stage.getClose() != stageConfig.isClose()) {
return true;
}
}
return false;
}
private ResourceExist checkResourceExistence(MetadataRegistry.SchemaMetadata metadata,
@ -245,7 +265,7 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
Group.Builder gBuilder
= Group.newBuilder()
.setMetadata(BanyandbCommon.Metadata.newBuilder().setName(metadata.getGroup()));
BanyandbCommon.ResourceOpts.Builder optsBuilder = BanyandbCommon.ResourceOpts.newBuilder().setShardNum(metadata.getShard());
BanyandbCommon.ResourceOpts.Builder optsBuilder = BanyandbCommon.ResourceOpts.newBuilder().setShardNum(metadata.getResource().getShardNum());
switch (metadata.getKind()) {
case STREAM:
@ -254,13 +274,13 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.setUnit(
IntervalRule.Unit.UNIT_DAY)
.setNum(
metadata.getSegmentIntervalDays()))
metadata.getResource().getSegmentInterval()))
.setTtl(
IntervalRule.newBuilder()
.setUnit(
IntervalRule.Unit.UNIT_DAY)
.setNum(
metadata.getTtlDays()));
metadata.getResource().getTtl()));
resourceExist = client.existStream(metadata.getGroup(), metadata.name());
gBuilder.setCatalog(BanyandbCommon.Catalog.CATALOG_STREAM).build();
break;
@ -270,13 +290,13 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.setUnit(
IntervalRule.Unit.UNIT_DAY)
.setNum(
metadata.getSegmentIntervalDays()))
metadata.getResource().getSegmentInterval()))
.setTtl(
IntervalRule.newBuilder()
.setUnit(
IntervalRule.Unit.UNIT_DAY)
.setNum(
metadata.getTtlDays()));
metadata.getResource().getTtl()));
resourceExist = client.existMeasure(metadata.getGroup(), metadata.name());
gBuilder.setCatalog(BanyandbCommon.Catalog.CATALOG_MEASURE).build();
break;
@ -287,6 +307,27 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
default:
throw new IllegalStateException("unknown metadata kind: " + metadata.getKind());
}
if (CollectionUtils.isNotEmpty(metadata.getResource().getAdditionalLifecycleStages())) {
for (BanyanDBStorageConfig.Stage stage : metadata.getResource().getAdditionalLifecycleStages()) {
optsBuilder.addStages(
BanyandbCommon.LifecycleStage.newBuilder()
.setName(stage.getName().name())
.setShardNum(stage.getShardNum())
.setSegmentInterval(
IntervalRule.newBuilder().setUnit(IntervalRule.Unit.UNIT_DAY)
.setNum(stage.getSegmentInterval()))
.setTtl(
IntervalRule.newBuilder()
.setUnit(
IntervalRule.Unit.UNIT_DAY)
.setNum(
stage.getTtl()))
.setNodeSelector(stage.getNodeSelector())
.setClose(stage.isClose())
//todo: set the default query stages
);
}
}
gBuilder.setResourceOpts(optsBuilder.build());
if (!RunningMode.isNoInitMode()) {
if (!groupAligned.contains(metadata.getGroup())) {

71
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageConfig.java
View File

@ -20,6 +20,8 @@ package org.apache.skywalking.oap.server.storage.plugin.banyandb;
import com.google.common.base.Splitter;
import com.google.common.collect.Iterables;
import java.util.ArrayList;
import java.util.List;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.library.module.ModuleConfig;
@ -94,6 +96,42 @@ public class BanyanDBStorageConfig extends ModuleConfig {
// The configuration of the groups.
// since 10.2.0
@Getter
@Setter
public static class Stage {
private StageName name;
// Node selector specifying target nodes for this stage.
// Optional; if provided, it must be a non-empty string.
private String nodeSelector;
private int shardNum;
private int segmentInterval;
private int ttl;
// Indicates whether segments that are no longer live should be closed.
private boolean close = false;
}
public enum StageName {
hot,
warm,
cold;
}
@Getter
@Setter
public static class GroupResource {
private int shardNum;
private int segmentInterval;
private int ttl;
private boolean enableWarmStage = false;
private boolean enableColdStage = false;
private List<String> defaultQueryStages = new ArrayList<>(2);
private List<Stage> additionalLifecycleStages = new ArrayList<>(2);
public GroupResource() {
defaultQueryStages.add(StageName.hot.name());
}
}
//The group settings of records.
/**
* The RecordsNormal defines settings for datasets not specified in "super".
@ -101,10 +139,7 @@ public class BanyanDBStorageConfig extends ModuleConfig {
*/
@Getter
@Setter
public static class RecordsNormal {
private int shardNum = 1;
private int segmentInterval = 1;
private int ttl = 3;
public static class RecordsNormal extends BanyanDBStorageConfig.GroupResource {
}
/**
@ -113,10 +148,7 @@ public class BanyanDBStorageConfig extends ModuleConfig {
*/
@Getter
@Setter
public static class RecordsSuper {
private int shardNum = 2;
private int segmentInterval = 1;
private int ttl = 3;
public static class RecordsSuper extends BanyanDBStorageConfig.GroupResource {
}
// The group settings of metrics.
@ -131,10 +163,7 @@ public class BanyanDBStorageConfig extends ModuleConfig {
*/
@Getter
@Setter
public static class MetricsMin {
private int shardNum = 2;
private int segmentInterval = 1;
private int ttl = 7;
public static class MetricsMin extends BanyanDBStorageConfig.GroupResource {
}
/**
@ -142,10 +171,7 @@ public class BanyanDBStorageConfig extends ModuleConfig {
*/
@Getter
@Setter
public static class MetricsHour {
private int shardNum = 1;
private int segmentInterval = 5;
private int ttl = 15;
public static class MetricsHour extends BanyanDBStorageConfig.GroupResource {
}
/**
@ -153,10 +179,7 @@ public class BanyanDBStorageConfig extends ModuleConfig {
*/
@Getter
@Setter
public static class MetricsDay {
private int shardNum = 1;
private int segmentInterval = 15;
private int ttl = 15;
public static class MetricsDay extends BanyanDBStorageConfig.GroupResource {
}
/**
@ -167,10 +190,7 @@ public class BanyanDBStorageConfig extends ModuleConfig {
*/
@Getter
@Setter
public static class Metadata {
private int shardNum = 2;
private int segmentInterval = 15;
private int ttl = 15;
public static class Metadata extends BanyanDBStorageConfig.GroupResource {
}
/**
@ -178,7 +198,6 @@ public class BanyanDBStorageConfig extends ModuleConfig {
*/
@Getter
@Setter
public static class Property {
private int shardNum = 1;
public static class Property extends BanyanDBStorageConfig.GroupResource {
}
}

26
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java
View File

@ -477,24 +477,20 @@ public enum MetadataRegistry {
public SchemaMetadata parseMetadata(Model model, BanyanDBStorageConfig config, DownSamplingConfigService configService) {
if (!model.isTimeSeries()) {
return new SchemaMetadata(BanyanDBStorageConfig.PROPERTY_GROUP_NAME, model.getName(), Kind.PROPERTY, DownSampling.None, config.getProperty().getShardNum(), 0, 0);
return new SchemaMetadata(BanyanDBStorageConfig.PROPERTY_GROUP_NAME, model.getName(), Kind.PROPERTY, DownSampling.None, config.getProperty());
}
if (model.isRecord()) { // stream
return new SchemaMetadata(model.isSuperDataset() ? model.getName() : "normal",
model.getName(),
Kind.STREAM,
model.getDownsampling(),
model.isSuperDataset() ? config.getRecordsSuper().getShardNum() : config.getRecordsNormal().getShardNum(),
model.isSuperDataset() ? config.getRecordsSuper().getSegmentInterval() : config.getRecordsNormal().getSegmentInterval(),
model.isSuperDataset() ? config.getRecordsSuper().getTtl() : config.getRecordsNormal().getTtl());
model.isSuperDataset() ? config.getRecordsSuper() : config.getRecordsNormal());
}
if (model.getBanyanDBModelExtension().isIndexMode()) {
return new SchemaMetadata("index", model.getName(), Kind.MEASURE,
model.getDownsampling(),
config.getMetadata().getShardNum(),
config.getMetadata().getSegmentInterval(),
config.getMetadata().getTtl());
config.getMetadata());
}
switch (model.getDownsampling()) {
@ -503,9 +499,7 @@ public enum MetadataRegistry {
model.getName(),
Kind.MEASURE,
model.getDownsampling(),
config.getMetricsMin().getShardNum(),
config.getMetricsMin().getSegmentInterval(),
config.getMetricsMin().getTtl());
config.getMetricsMin());
case Hour:
if (!configService.shouldToHour()) {
throw new UnsupportedOperationException("downsampling to hour is not supported");
@ -514,9 +508,7 @@ public enum MetadataRegistry {
model.getName(),
Kind.MEASURE,
model.getDownsampling(),
config.getMetricsHour().getShardNum(),
config.getMetricsHour().getSegmentInterval(),
config.getMetricsHour().getTtl());
config.getMetricsHour());
case Day:
if (!configService.shouldToDay()) {
throw new UnsupportedOperationException("downsampling to day is not supported");
@ -525,9 +517,7 @@ public enum MetadataRegistry {
model.getName(),
Kind.MEASURE,
model.getDownsampling(),
config.getMetricsDay().getShardNum(),
config.getMetricsDay().getSegmentInterval(),
config.getMetricsDay().getTtl());
config.getMetricsDay());
default:
throw new UnsupportedOperationException("unsupported downSampling interval:" + model.getDownsampling());
}
@ -547,9 +537,7 @@ public enum MetadataRegistry {
* down-sampling of the {@link Model}
*/
private final DownSampling downSampling;
private final int shard;
private final int segmentIntervalDays;
private final int ttlDays;
private final BanyanDBStorageConfig.GroupResource resource;
/**
* Format the entity name for BanyanDB

2
test/e2e-v2/cases/kafka/log/Dockerfile.fluentd
View File

@ -16,4 +16,4 @@
FROM bitnami/fluentd:1.12.1
## Install output kafka plugins
RUN fluent-gem install 'fluent-plugin-kafka'
RUN fluent-gem install fluent-plugin-kafka -v 0.19.3