Commit Graph

54 Commits

Author SHA1 Message Date
kezhenxu94 18c802c587
Remove string limitation in Jackson deserializer for ES Client (#13283) 2025-06-02 23:13:27 +08:00
kezhenxu94 b83213705b
Add `doc_values` for fields that need to be sorted or aggregated in ElasticSearch, and disable all others. (#12782) 2024-11-24 12:17:16 +08:00
kezhenxu94 a5d8777ee7
Use CI-friendly revision to set the project version (#12640) 2024-09-24 17:53:14 +08:00
Wan Kai 05373d69d4
Support tracing trace query(SkyWalking and Zipkin) for debugging. (#12378) 2024-06-27 10:12:44 +08:00
吴晟 Wu Sheng 054d0e4ff9
Release 10.0.0 and move forward to 10.1.0 (#12203) 2024-05-10 09:19:56 +08:00
吴晟 Wu Sheng 497a7bdb02
Release 9.7 and start 10.0 iteration (#11607) 2023-11-28 23:49:44 -08:00
dogblues ce707a4d52
Specific only necessary fields when queryBasicTraces in ElasticSearch (#11419) 2023-10-18 01:21:58 -05:00
Wan Kai 8e529ee956
ElasticSearchClient: Add `deleteById` API. (#11305) 2023-09-05 17:20:43 +08:00
吴晟 Wu Sheng a336b3cff4
Release 9.6.0 and begin 9.7.0 iteration (#11289) 2023-09-01 22:57:18 +08:00
吴晟 Wu Sheng abc7aa88f1
Polish ElasticSearch relative docs and bump up tested versions. (#11179) 2023-08-05 22:56:22 +08:00
吴晟 Wu Sheng 1f6ea1d744
Begin 9.6.0 iteration (#10935)
* [maven-release-plugin] prepare release v9.5.0

* [maven-release-plugin] prepare for next development iteration

* Create changes-9.5.0.md

* Set changes.md for 9.6.0
2023-06-14 18:26:50 +08:00
吴晟 Wu Sheng 4874374ace
Tag 9.4.0 release, prepare for 9.5.0 iteration. (#10512) 2023-03-08 23:30:37 +08:00
kezhenxu94 49594c4db1
Bump up Java version to 11 and use junit tag to group IT and slow IT (#10390)
- `./mvnw test ...` by its nature only runs unit tests, whose name is pattern of `*Test`, and it does never runs integration tests, whose name is patter of `IT*` or `*IT`, so in this PR we use `./mvnw clean test ...` to only run unit tests.
- `./mvnw integration-test ...` will run integration tests and unit tests, so we have `skipUTs` to control whether to skip the UT when running ITs, we already had this before.
  - As for ITs, we have two groups, one is normal tests without any `@Tag`s, the other one is slow integration tests, which is annotated with `@Tag("slow")`, so we divided the integration tests into two workflow jobs:
  - `./mvnw -DskipUTs=true clean integration-test -DexcludedGroups=slow ...` run the ITs but don't run the slow ITs and UTs, `-DexcludedGroups=slow` excludes tests annotated with `@Tag("slow")`
  - `./mvnw -DskipUTs=true clean integration-test -Dcheckstyle.skip -Dtest=${{ matrix.test.class }}` run the slow ITs (because `excludedGroups` is not set) one case at a time, by setting `-Dtest={{ class }}`, not run UTs,
2023-02-14 23:19:32 +08:00
kezhenxu94 383a5bbf7b
Migrate tests from junit 4 to junit 5 (#10382) 2023-02-13 18:54:32 +08:00
Hper e56308995a
[Elasticsearch] Support batchOfBytes configuration to limit size of bulk flush. (#10287) 2023-02-06 12:46:24 +08:00
Shichao b97dfb1a65
[Optional] Support trace/span on ElasticSearch to use traceIDs routing. (#10282) 2023-01-18 16:44:15 +08:00
吴晟 Wu Sheng c5cbd3a14c
Optimize `flushInterval` of ElasticSearch BulkProcessor (#10135)
Optimize `flushInterval` of ElasticSearch BulkProcessor to avoid extra periodical flush in the continuous bulk streams.
2022-12-08 21:57:32 +08:00
吴晟 Wu Sheng 18008aa6b2
Release 9.3.0. Begin 9.4.0 iteration (#10077) 2022-12-02 21:19:13 +08:00
kezhenxu94 a7210f6633
Add OpenSearch 1.3.6 and 2.4.0 to test matrix (#10040) 2022-11-29 11:13:01 +08:00
kezhenxu94 1da72b6af9
Internal API: remove undocumented ElasticSearch API usage and use documented one (#10031) 2022-11-25 21:15:25 +08:00
kezhenxu94 9ba31b63a2
Do not log error logs when failed to create ElasticSearch index because the index is created already (#9856) 2022-10-27 23:49:13 +08:00
kezhenxu94 04c4fedf43
Adapt ElasticSearch 8.1+, migrate from removed APIs to recommended APIs (#9603) 2022-09-14 12:22:40 +08:00
吴晟 Wu Sheng 608a084350
Begin 9.3.0 iteration, 9.2.0 release is on vote (#9539)
* [maven-release-plugin] prepare release v9.2.0

* [maven-release-plugin] prepare for next development iteration

* Create changes-9.2.0.md

* Update changes.md

* Update LICENSE
2022-08-31 00:31:04 +08:00
吴晟 Wu Sheng 7f371ed565
[Critical] Fix a low performance issue of metrics persistent in ES (#9437)
[Critical] Fix a low-performance issue of metrics persistent in the ElasticSearch storage implementation. One single metric could have to wait for an unnecessary 7~10s(System Env Variable SW_STORAGE_ES_FLUSH_INTERVAL).

This is a solid and critical performance issue, from the self-observability metric, we could find one metric that would need 7-10s to execute the data flush, which is not expected.
2022-08-08 19:38:58 +08:00
Wan Kai 14800cbc57
Optimize elasticsearch query performance by using `_mGet` and physical index name (#9339)
Optimize ElasticSearch query performance by using `_mGet` and physical index name rather than alias in these scenarios:
(a) Metrics aggregation 
(b) Zipkin query
(c) Metrics query
(d) Log query
2022-07-14 10:40:21 +08:00
吴晟 Wu Sheng 8584ada7cc
Begin 9.2.0 iteration (#9191)
* [maven-release-plugin] prepare release v9.1.0

* [maven-release-plugin] prepare for next development iteration
2022-06-08 17:24:15 +08:00
yangyiweigege 8b40d2269f
Fix SumAggregationBuilder#build should use the SumAggregation rather than MaxAggregation.(#9111) (#9114)
Co-authored-by: xishui <xishui@wacai.com>
2022-05-21 19:01:03 +08:00
kezhenxu94 369949b29a
Clean up scroll context after going through the results (#8967) 2022-05-06 09:35:27 +08:00
Hper c5bde6d37f
Improve the performance of TermsAggregation (#8959) 2022-05-05 08:05:09 +08:00
yangyiweigege 00b58f6210
fix ES flush thread will stop work when flush schedule task have ES connection exception (#8909)
Co-authored-by: xishui <xishui@wacai.com>
2022-04-19 20:39:55 +08:00
吴晟 Wu Sheng 91413a753c
Start 9.1.0-dev iteration (#8822) 2022-04-06 18:56:49 +08:00
吴晟 Wu Sheng d96eed9145
Fix event type of export data, reduce redundancy ThreadLocal in MAL core. (#8754)
* Fix event type of export data is incorrect, it was `EventType.TOTAL` always.
* Reduce redundancy ThreadLocal in MAL core. Improve MAL performance.
* ExpressionDelegate public->private
2022-03-27 16:53:15 +08:00
Kai 2bf7f0f4e7
Support ElasticSearch template mappings `properties parameters` and `_source` update (#8706)
Previous logic can't update the mappings `properties parameters` and `_source` only support adding a new property.

### Now
1. As the OAP core merges the index of metrics, properties would not be deleted still.
2. Would not update the current index to avoid conflict, a new index follows the new template to create.

**Warning**, if there are properties conflicts, same key with different values, they would override each other according to booting sequence. Unless we change booting process as merging modules before merging with existing templates and indices, we can't add check mechanism and warnings.
2022-03-19 00:55:47 +08:00
kezhenxu94 bf03d9f67a
Improve CI with the GHA new feature (#8691) 2022-03-17 10:06:53 +08:00
Kai 91a82e056a
Add storage column attribute `indexOnly` (#8679)
* Add storage column attribute `indexOnly`, support ElasticSearch only index and not store some fields.
* Add `indexOnly=true` to `SegmentRecord.tags`, to reduce unnecessary storage.
2022-03-16 19:19:29 +08:00
kezhenxu94 b1d3f3b637
Expose more ElasticSearch health check related logs (#8652) 2022-03-10 19:19:44 +08:00
kezhenxu94 eb7529adb6
Support large service list query and replace deprecated API of Armeria (#8628) 2022-03-08 13:18:18 +08:00
kezhenxu94 68e40eb9fc
Support ElasticSearch 8 and add it into E2E tests (#8564) 2022-02-21 08:11:27 +08:00
kezhenxu94 c49a82a6f5
Expose responseTimeout of ES client (#8531) 2022-02-10 14:46:40 +08:00
吴晟 Wu Sheng 1fdf3a8acf
Set version to 9.0.0-snapshot (#8242) 2021-12-06 10:04:20 +08:00
吴晟 Wu Sheng e7d6f89021
Set the version to 8.10.0, 8.9.0 release is in process. (#8233) 2021-12-02 18:54:21 +08:00
kezhenxu94 a2897db471
Fix the so11y latency of persistence execution latency not correct in ElasticSearch storage (#8161)
The ES persistence execution is now asynchronous and the execution
latency only counts the time to insert the requests into the bulk
processor, instead of the time after the requests are flushed into the
storage, this patch fixes that issue.

There is the same issue in prepare latency but that needs more changes
so I'll leave it to another pull request.
2021-11-20 16:51:54 +08:00
Kirs f7bcb76784
move the codes in apm-commons into server-library (#8003) 2021-10-26 20:31:36 +08:00
kezhenxu94 508945f552
Upgrade Armeria to 1.12, upgrade OpenSearch test version to 1.1.0 (#7866) 2021-10-08 19:45:33 +08:00
kezhenxu94 a4c68d30d8
test: fix that E2E is not run, also add ElasticSearch 7.15 to test matrix (#7826) 2021-09-28 21:57:19 +08:00
吴晟 Wu Sheng 8b681053df
Begin 8.9.0 iteration (#7801)
* [maven-release-plugin] prepare release v8.8.0

* [maven-release-plugin] prepare for next development iteration

* Create changes-8.8.0.md

* Reset changelog for 8.9.0
2021-09-25 23:46:18 +08:00
kezhenxu94 ad9d616297
Add `socketTimeout` back to the new implementation (#7798) 2021-09-24 22:35:56 +08:00
吴晟 Wu Sheng c4b8c66174
Only report `bug warning` in precise conditions. (#7772)
* Only report `bug warning` in precise conditions.

* Add a test check for template existing.
2021-09-23 10:46:08 +08:00
kezhenxu94 ad842013f4
Replace `mget` with ids' query as `mget` doesn't apply on aliases (#7731) 2021-09-17 17:54:26 +08:00
kezhenxu94 73b6c67344
Fix ser/deserializers of ES 6/7-7.8/7.8+, search query, aggregation, order, terms 2021-09-15 17:49:41 +08:00