The key logic behinds all these is, metrics persistence is fully asynchronous.
* The core/maxSyncOperationNum setting(added in 8.5.0) is removed due to metrics persistence is fully asynchronous.
* The core/syncThreads setting(added in 8.5.0) is removed due to metrics persistence is fully asynchronous.
* Optimization: Concurrency mode of execution stage for metrics is removed(added in 8.5.0). The only concurrency of prepare stage is meaningful and kept.
* Remove the outside preparedRequest list initialization, worker instance could always build a suitable size list in the first place (Reduce Array.copy and GC load a little).
* Performance: remove the synchronous persistence mechanism from batch ElasticSearch DAO. Because the current enhanced persistent session mechanism, don't require the data queryable immediately after the insert and update anymore.
* Performance: share `flushInterval` setting for both metrics and record data, due to `synchronous persistence mechanism` removed. Record flush interval used to be hardcoded as 10s.
* Remove `syncBulkActions` in ElasticSearch storage option.
* Increase the default bulkActions(env, SW_STORAGE_ES_BULK_ACTIONS) to 5000(from 1000).
* Increase the flush interval of ElasticSearch indices to 15s(from 10s)
Add these 2 references. According to these, **(same _index, _type and _id) in same bulk will be in order**
1. https://github.com/elastic/elasticsearch/issues/50199
2. https://discuss.elastic.co/t/order-of--bulk-request-operations/98124
Notice, the order of different bulks is not guaranteed by the ElasticSearch cluster. We are going to have the risk of dirty write. But consider we set over 20s period between flush, and index flush period is 10, we should be safe.
Recommend 5000 bulk size and 15s flush interval only. The persistent period has been set to 25s.
There are a lot of calls to `Metrics.id()` and `ISource.getEntityId`, which calculates the id by manipulating strings in every single call, producing many garbage objects.
In this patch, I cache the id and only calculate the id if it's requested for the first time.
* Add Mariadb JDBC plugin and provide tests
* Refactor component name
* Reformat plugin description and remove healthcheck span from the test
Co-authored-by: Efe Kaptan <Efe.Kaptan@webtrekk.com>
Co-authored-by: zhang-wei <pknfe@outlook.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
吴晟 Wu Sheng
kezhenxu94
Alvin
wankai123
史精文
mrproliu
Evan
zhang-wei
zifeihan
Xin,Zhang
xbkaishui
Gao Hongtao
Daming
hailin0
dagmom
于玉桔
Efe Kaptan