Fix bug, the enhanced session could cache the metadata metrics(hot entity) forever. A new timeout mechanism is designed for avoiding this specific case.
Optimize this timeout mechanism, make it different for ES(one index per day) and non-ES storage implementation.
Remove the following plugins due to they are legacy and unstable releases.
1. sharding-jdbc-1.5.x-plugin
2. sharding-sphere-4.x-rc3-plugin
3. sharding-sphere-4.x-plugin (This is for RC1 and RC2)
To people who want to use these removing versions, you still could copy the plugins from SkyWalking's 8.6.0 release to the '/plugins' folder to activate, unless SkyWalking has some core level APIs upgrade. This is very rare to happen.
When there is a fallback in webflux (like a hystrix fallback or a webflux fallback method that involves visiting a new URL). This interceptor would be invoked twice, one for origin request and another for fallback request. The ServerWebExchange for these two aren't exactly the same one, but the former one is a delegate inside of the latter one, thus, they share the same attributes.
The problem of the old code is that get the span from attributes within the execution of Mono would get span for the latter request twice. so just move this part of code out could solve the problem.
It seems impossible to make the mono chain work like expected. So add a check about whether the best matching pattern really matches the URL. This check code is how webflux generating the best match pattern,
The key logic behinds all these is, metrics persistence is fully asynchronous.
* The core/maxSyncOperationNum setting(added in 8.5.0) is removed due to metrics persistence is fully asynchronous.
* The core/syncThreads setting(added in 8.5.0) is removed due to metrics persistence is fully asynchronous.
* Optimization: Concurrency mode of execution stage for metrics is removed(added in 8.5.0). The only concurrency of prepare stage is meaningful and kept.
* Remove the outside preparedRequest list initialization, worker instance could always build a suitable size list in the first place (Reduce Array.copy and GC load a little).
Adjust index refresh period as INT(flushInterval * 2/3), it used to be as same as bulk flush period. At the edge case, in low traffic(traffic < bulkActions in the whole period), there is a possible case, 2 period bulks are included in one index refresh rebuild operation, which could cause version conflicts. And this case can't be fixed through core/persistentPeriod as the bulk fresh is not controlled by the persistent timer anymore.
This PR should avoid the following exception in the low load case, especially when bulkActions is set larger than the number of a metric type.
* Performance: remove the synchronous persistence mechanism from batch ElasticSearch DAO. Because the current enhanced persistent session mechanism, don't require the data queryable immediately after the insert and update anymore.
* Performance: share `flushInterval` setting for both metrics and record data, due to `synchronous persistence mechanism` removed. Record flush interval used to be hardcoded as 10s.
* Remove `syncBulkActions` in ElasticSearch storage option.
* Increase the default bulkActions(env, SW_STORAGE_ES_BULK_ACTIONS) to 5000(from 1000).
* Increase the flush interval of ElasticSearch indices to 15s(from 10s)
Add these 2 references. According to these, **(same _index, _type and _id) in same bulk will be in order**
1. https://github.com/elastic/elasticsearch/issues/50199
2. https://discuss.elastic.co/t/order-of--bulk-request-operations/98124
Notice, the order of different bulks is not guaranteed by the ElasticSearch cluster. We are going to have the risk of dirty write. But consider we set over 20s period between flush, and index flush period is 10, we should be safe.
Recommend 5000 bulk size and 15s flush interval only. The persistent period has been set to 25s.
Groovy naturally supports many dynamic features that we don't benefit for now but cost performance loss, in this patch we compile our Groovy-based DSL scripts statically to optimize performance.
hostInfo.getDatabaseUrl() result in version 8.0.19 or above:
jdbc:mysql:loadbalance://**internally_generated**1620805591623**
hostInfo.getDatabaseUrl() result in version 8.0.18 or below:
jdbc:mysql:replication://127.0.0.1:3306,127.0.0.1:3306,127.0.0.1:3306/mer_goods_admin_sit?useSSL=true&useUnicode=true&autoReconnect=true&rewriteBatchedStatements=TRUE&allowMultiQueries=true&serverTimezone=GMT%2B8
* Allow configuring max request header size
This allows configuring the HTTP max request header size from the jetty
server.
By default it uses 8192, the same jetty default.
1.0Performance: Add L1 aggregation flush period, which reduces the CPU load and helps young GC.
2. Replace do not direct send after the first aggregation to reduce the network #6400.
3. Enhance the DataCarrier to notify the consumer in no enqueue event in short term.
4. L1 aggregation flush period still works even no further metrics generated, powered by <3>
5. Fix gRPC remote client OOM. The concurrency control mechanism failed.
There are a lot of calls to `Metrics.id()` and `ISource.getEntityId`, which calculates the id by manipulating strings in every single call, producing many garbage objects.
In this patch, I cache the id and only calculate the id if it's requested for the first time.
The possible metrics names are relatively fixed in an OAP run, we previously always escape the metrics names, building a new regex Pattern from scratch, which is cpu-consuming.
In this patch, I cache the escaped metrics names, and if it missed, use a pre-compiled regex Pattern to escape the metrics name.
This patch also replace string concatenation `+` with `StringBuilder` in generated classes, which won't get optimized by Java compiler
This may reduce 1~1.5 vCPU under 20K RPS environment.
Jared Tan
kezhenxu94
Cool-Coding
吴晟 Wu Sheng
吴伟杰
yujiaxinlong
Lu Jiajing
wankai123
Fine0830
Elliot Duan
Ax1an
whl12345
Switch
Sergi Castro
Daming
mrproliu
liqiangz
Lin1997
AngryMills
Jing
CharliePu
hn
zhyyu