Commit Graph

395 Commits

Author SHA1 Message Date
Wan Kai d7ee9a01d6
Support sharding MySQL database instances and tables. (#9689)
* [**Breaking Change**] Support sharding MySQL database instances and tables by [Shardingsphere-Proxy](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-proxy).
  SQL-Database requires removing tables `log_tag/segment_tag/zipkin_query` before OAP starts, if bump up from previous releases.
2022-09-30 08:39:14 +08:00
Wan Kai 4a7e8191dd
Unify query services and DAOs codes time range condition to `Duration`. (#9702) 2022-09-29 08:08:15 +08:00
吴晟 Wu Sheng 2c265db6e8
Fix two kernel level bugs. (#9690)
* Fix meter functions `avgHistogram`, `avgHistogramPercentile`, `avgLabeled`, `sumHistogram` having data conflict when downsampling. FYI @jinho-son thanks for reporting.
* Do sorting `readLabeledMetricsValues` result forcedly in case the storage(database) doesn't return data consistent with the parameter list.
2022-09-27 16:05:17 +08:00
吴晟 Wu Sheng c3e0b12648
[Critical] Remove physical index existing check and fix TTL activation bug (#9632)
* Remove physical index existing check

* Make sure instance list ordered in TTL processor to avoid TTL timer never runs.
2022-09-19 19:37:03 +08:00
Wan Kai b045924179
clean unused codes (#9606) 2022-09-15 13:00:58 +08:00
吴晟 Wu Sheng 5ad3782231
Use prepareStatement in H2SQLExecutor#getByIDs (#9561) 2022-09-05 09:25:27 +08:00
吴晟 Wu Sheng 608a084350
Begin 9.3.0 iteration, 9.2.0 release is on vote (#9539)
* [maven-release-plugin] prepare release v9.2.0

* [maven-release-plugin] prepare for next development iteration

* Create changes-9.2.0.md

* Update changes.md

* Update LICENSE
2022-08-31 00:31:04 +08:00
mrproliu 922f1b958d
Add Network Profiling for Service Mesh DP instance and K8s pod panels. (#9536) 2022-08-30 14:38:55 +08:00
Wan Kai 7df469a6e8
Fix JDBC TTL to delete additional tables data. (#9508) 2022-08-24 18:41:21 +08:00
Wan Kai aa8ddd2587
Optimize TTL mechanism for Elasticsearch storage, skip executed indices in one TTL rotation. (#9473)
* Optimize TTL mechanism for Elasticsearch storage, skip executed indices in one TTL rotation

* remove inspect
2022-08-16 12:41:43 +08:00
mrproliu 9e07ad7e9f
Fix query history process data failure (#9471) 2022-08-15 18:17:52 +08:00
Wan Kai 6ab00031a0
Fix query services by serviceId error when Elasticsearch storage `SW_STORAGE_ES_QUERY_MAX_SIZE` > 10000 (#9468) 2022-08-12 20:05:54 +08:00
Wan Kai 8a8070025a
Support Elasticsearch column alias for the compatibility between storage logicSharding model and no-logicSharding model. (#9442)
## New ElasticSearch storage option explanation in 9.2.0
Since v9.2.0, SkyWalking OAP provides 2 storage options for metrics/meter and records, 
system environment variable is (`SW_STORAGE_ES_LOGIC_SHARDING`):

### No-Sharding Model (OAP default setting, `SW_STORAGE_ES_LOGIC_SHARDING = false`)
1. OAP merges all metrics/meter and records(without super datasets, such as segments) indices into one physical 
index template `metrics-all` and `records-all`.
2. The logic index name would be present in columns `metric_table` and `record_table`.
3. If the logic column name has an alias (configured by `@ElasticSearch.Column()`), the alias would be the real physical column name.

### No-Sharding Model (`SW_STORAGE_ES_LOGIC_SHARDING = true `)
1. OAP shard metrics/meter indices into multi-physical indices as in the previous versions(one index template per metric/meter aggregation function).
2. Records and metrics without configuring aggregation function in `@MetricsFunction` and `@MeterFunction` would not be sharded.
3. The shard template name would be `metrics-aggregation function name` or `meter-aggregation function name` such as `metrics-count`,
and the logic index name would be present in column `metric_table`.
4. The OAP **would not** use the column alias, the logic column name would be the real physical column name.

**Notice**: 
Users still could choose to adjust ElasticSearch's shard number(`SW_STORAGE_ES_INDEX_SHARDS_NUMBER`) to scale out no matter the option.
2022-08-09 22:24:45 +08:00
Brandon Fergerson caa6a0de5a
refactor: make more consistent (#9439) 2022-08-09 10:27:20 +08:00
吴晟 Wu Sheng 174b8d6ef0
Enhance #9437 fix to avoid too many small flushing. (#9440)
Move the forcedly flush out of each worker's #flush, which would only make the left data flushing into the storage ASAP. In my previous fix, it would increase the number of flushing as the number of workers, which is unnecessary too.
2022-08-09 07:28:17 +08:00
吴晟 Wu Sheng 7f371ed565
[Critical] Fix a low performance issue of metrics persistent in ES (#9437)
[Critical] Fix a low-performance issue of metrics persistent in the ElasticSearch storage implementation. One single metric could have to wait for an unnecessary 7~10s(System Env Variable SW_STORAGE_ES_FLUSH_INTERVAL).

This is a solid and critical performance issue, from the self-observability metric, we could find one metric that would need 7-10s to execute the data flush, which is not expected.
2022-08-08 19:38:58 +08:00
mrproliu 0c2bf2c9ff
Support analyze the component in the eBPF Network profiling (#9430) 2022-08-06 23:32:14 +08:00
mrproliu 4bd4231b60
Elasticsearch storage support merge all records(without super datasets) indices into one physical index (#9424) 2022-08-05 10:40:33 +08:00
Wan Kai 85c09153b5
Fix Zipkin trace query the max size of spans (#9423) 2022-08-04 13:32:42 +08:00
Wan Kai 0a032ced18
Elasticsearch storage support merge all metrics/meter indices into one physical index (#9416) 2022-08-03 21:18:30 +08:00
mrproliu b8cf95d165
Add `SUM_PER_MIN`(sumHistogramPercentile) downsampling in MAL. Add eBPF Network Profiling E2E (#9377) 2022-07-28 10:00:36 +08:00
Brandon Fergerson fd6d0ee743
Fixes Banynadb race condition (#9360) 2022-07-20 15:23:23 +08:00
Wan Kai 14800cbc57
Optimize elasticsearch query performance by using `_mGet` and physical index name (#9339)
Optimize ElasticSearch query performance by using `_mGet` and physical index name rather than alias in these scenarios:
(a) Metrics aggregation 
(b) Zipkin query
(c) Metrics query
(d) Log query
2022-07-14 10:40:21 +08:00
mrproliu 5ea7c0d778
Support eBPF Network Profiling (#9337) 2022-07-13 22:41:22 +08:00
mrproliu 46ae1cfe1e
Add `VIRTUAL` detect type to Process for Network Profiling (#9280) 2022-06-28 17:07:57 +08:00
Wan Kai 43d65775f0
Add more entities for Zipkin to improve performance (#9198) 2022-06-10 20:04:11 +08:00
kezhenxu94 9ccb04e6cf
ElasticSearch: scroll id should be updated when scrolling as it may change (#9199) 2022-06-10 18:13:59 +08:00
吴晟 Wu Sheng 8584ada7cc
Begin 9.2.0 iteration (#9191)
* [maven-release-plugin] prepare release v9.1.0

* [maven-release-plugin] prepare for next development iteration
2022-06-08 17:24:15 +08:00
Wan Kai f0aef53d49
Support Zipkin traces collect and zipkin traces query API. (#9177) 2022-06-08 07:40:36 +08:00
mrproliu 0447cb2b71
Fix write record data to the wrong index name in ES storage (#9148) 2022-05-31 13:30:33 +08:00
mrproliu 31624a3c09
Simplify the format index name logical in ES storage (#9145) 2022-05-30 13:38:09 +08:00
Wan Kai d69213ec6b
Add limit config for query autocomplete tags (#9140) 2022-05-27 20:34:49 +08:00
mrproliu 1934f308a7
Fix scroll query in Elastic Search storage (#9128) 2022-05-25 19:19:27 +08:00
Jiajing LU dc39ce9bb4
Add shards factor for BanyanDB (#9116) 2022-05-22 17:34:25 +08:00
Jiajing LU 25ebd6c242
Add Banyandb storage e2e (#9112) 2022-05-21 19:48:56 +08:00
mrproliu 47509715f5
Support OFF CPU eBPF Profiling (#9107) 2022-05-20 14:46:16 +08:00
Wan Kai 051431fde7
SQL Database: provide `@SQLDatabase.AdditionalEntity` to support create additional tables from a model. (#9098)
* SQL Database: provide `@SQLDatabase.AdditionalEntity` to support create additional tables from a model.
* [Breaking Change] SQL Database: remove SQL Database config `maxSizeOfArrayColumn` and `numOfSearchableValuesPerTag`.
* [Breaking Change] SQL Database: move `Tags list` from `Segment`,`Logs`,`Alarms` to their additional table.
* Upgrade doc
2022-05-20 08:29:32 +08:00
mrproliu e451052357
Remove `total` field in Alarm, Browser Logs, and Alarm query (#9102) 2022-05-19 23:04:44 +08:00
mrproliu 7c295738a8
Remove `total` field in Log and Trace query (#9094) 2022-05-19 06:50:37 +08:00
Jiajing LU 439f843d89
Banyandb OAP integration (#9064) 2022-05-18 16:05:41 +08:00
吴晟 Wu Sheng b60855e1e2
Refactor the storage annotations. (#9075) 2022-05-17 12:44:26 +08:00
吴晟 Wu Sheng fcc6a16215
Remove InfluxDB 1.x and Apache IoTDB 0.X as storage options (#9060)
* Remove InfluxDB 1.x and Apache IoTDB as storage options.

* Update dependencies.
2022-05-13 21:30:24 +08:00
Wan Kai c4ebe831af
fix: set tag autocomplete query time range unit to day (#9053) 2022-05-12 23:36:50 +08:00
mrproliu 9494fb2545
Remove layer from process and remove duration from eBPF Profiling scheduler query (#9034) 2022-05-11 17:15:09 +08:00
Wan Kai 2e7cf5db10
Remove `Layer` concept from `ServiceInstance`. (#9029) 2022-05-11 09:59:33 +08:00
吴晟 Wu Sheng 9eae2a53c4
Fix typo and comment links. (#9001) 2022-05-06 23:42:45 +08:00
Wan Kai 6c6ef24664
Support autocomplete tags in logs query. (#8999) 2022-05-06 20:18:45 +08:00
kezhenxu94 369949b29a
Clean up scroll context after going through the results (#8967) 2022-05-06 09:35:27 +08:00
Hper c5bde6d37f
Improve the performance of TermsAggregation (#8959) 2022-05-05 08:05:09 +08:00
吴晟 Wu Sheng 32b1596e5f
Enhance Converter mechanism in kernel level (#8986)
* Enhance Converter mechanism in kernel level to make BanyanDB native feature more effective.
2022-05-04 18:55:18 +08:00