Commit Graph

50 Commits

Author SHA1 Message Date
吴晟 Wu Sheng de975c7e2b
Remove synchronous persistence mechanism in API level and ElasticSearch implementation (#7293)
* Performance: remove the synchronous persistence mechanism from batch ElasticSearch DAO. Because the current enhanced persistent session mechanism, don't require the data queryable immediately after the insert and update anymore.
* Performance: share `flushInterval` setting for both metrics and record data, due to `synchronous persistence mechanism` removed. Record flush interval used to be hardcoded as 10s.
* Remove `syncBulkActions` in ElasticSearch storage option.
* Increase the default bulkActions(env, SW_STORAGE_ES_BULK_ACTIONS) to 5000(from 1000).
* Increase the flush interval of ElasticSearch indices to 15s(from 10s)

Add these 2 references. According to these, **(same _index, _type and _id) in same bulk will be in order**
1. https://github.com/elastic/elasticsearch/issues/50199
2. https://discuss.elastic.co/t/order-of--bulk-request-operations/98124

Notice, the order of different bulks is not guaranteed by the ElasticSearch cluster. We are going to have the risk of dirty write. But consider we set over 20s period between flush, and index flush period is 10, we should be safe.

Recommend 5000 bulk size and 15s flush interval only. The persistent period has been set to 25s.
2021-07-14 17:19:49 +08:00
吴晟 Wu Sheng 536b7d2328
Support connectTimeout and socketTimeout settings (#7214) 2021-07-01 08:20:59 +08:00
Gao Hongtao b8a040ea74
Append the root slash(/) to getIndex and getTemplate requests in ES client (#6663)
* Append the root slash(/) to getIndex and getTemplate requests

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Document this bug

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>
2021-04-01 14:26:57 +08:00
Jingguo Yao bfef6d9120
Fix typos (#6568) 2021-03-18 09:58:47 +08:00
Evan fd1d21021f
[Incompatible Enhancement]New index policy of ElasticSearch storage option (#6499) 2021-03-11 15:11:37 +08:00
Jared Tan 8d164d3dd8
Add Tidb provider module. (#5844)
* fix transaction too large error when use TiDB as storage

* add CHANGES.md

* fix typo

* add tidb storage provider module.

* add e2e.

* revert ui submodule.

* fix license.

* Fix wrong health check test

* add tidb ttl e2e.

Co-authored-by: moonsphere <moonsphere@m.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-11-22 23:28:54 +08:00
xbkaishui 56fe8a4b41
Cluster manager health v2 (#5784) 2020-11-04 10:04:24 +08:00
xbkaishui d86570a88e
Health check for InfluxDB (#5637) 2020-10-13 16:59:42 +08:00
Lu Jiajing 5bfac1b50f
Remove style-check exception for logger (#5363) 2020-08-21 18:26:25 +08:00
Evan 197ef1e836
Improve Es trace basic query performance (#5132) 2020-08-15 08:24:37 +08:00
Gao Hongtao 3fdfdf3b83
Storage elasticsearch health check (#5099) 2020-07-19 23:37:13 +08:00
Gao Hongtao b6661b5164
Add health checker module (#5046) 2020-07-10 00:30:43 +08:00
dagmom deccc262f3
Bulk executed with failures,then log detail message (#4845)
Co-authored-by: guhao <dev_guhao@163.com>
2020-05-31 15:19:40 +08:00
Gao Hongtao 78d14ebbd6
Enable OAP gRPC SSL transportation (#4470)
* Enable OAP gRPC SSL transportation

Porting to OpenSSL to enable SSL transportation. The server private
key is in the format of PCKS#8, certificates is x509 though.

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add netty-tcnative-boringssl-static 2.0.26 licnese

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update docs/en/setup/backend/grpc-ssl.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* Update docs/en/setup/backend/grpc-ssl.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* Add SSL to grpc remote client and sharing server

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update documents

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Fix nits

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Fix test case

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add e2e-ssl and remove other test cases temporarily

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add TLS relevant files

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Decrease timeout

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Test config works

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* triger CI on oap-ssl branch

* Disable agent ca

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Switch to sharing server

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add agent log to console

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Upload log files

* Change backend

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Revert "Change backend"

This reverts commit 7a085711e3653d3bf642aa84f73650e22595e383.

* Update backend

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Revert e2e

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Ignore TSL relevant files

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Polish codes

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update certificates expires year to 2120

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-03-15 08:15:28 +08:00
吴晟 Wu Sheng e69391a207
Support Secrets Management File in the ElasticSearch 6/7 storage (#4493)
* Temp commit

* Support secretsManagementFile file.

* Update doc.

* 1. Support JKS/pass runtime change too.
2. Follow review.

* Fix format.

* Fix username/password/trustPass haven't been updated in the es client.

* Fix doc issue.

Co-authored-by: Jared Tan <jian.tan@daocloud.io>
2020-03-13 20:50:14 +08:00
吴晟 Wu Sheng 329f7e150b
Support Downsampling Data Packing feature in ES storage implementation (#4364)
* Support day/hour/minute metrics merging into one index. Reduce the number of index 50%.
2020-02-16 09:03:30 +08:00
kezhenxu94 5b255ba3e3
Set up more strict code styles and fix existing issues (#4337)
Motivation:

Review code styles with some bots automatically.

Modifications:

Set up ReviewDog in GitHub Action to review code style.

Add more check rules to checkstyle plugin.

Result:

Obvious code styles can be reviewed and commented automatically.
2020-02-11 17:10:49 +08:00
Weiyi Liu 9415f68127 Fix the wrong comments. (#4139) 2019-12-27 12:10:37 +08:00
吴晟 Wu Sheng b570996006
Update JDK compiling level to 8, and remove zip dist. (#4119)
* Update JDK compiling level to 8, and remove zip dist.

* Exclude annotation and shade API.

* Fix more exclusion.

* 1. Update grava and gson 2. Fix compile issue. 3. Keep protoc and maven plugin in old version. Some incompatible of 3rd party grpc proto.

* Fix dependency conflicts, license and notice issue, and dependency check script issue

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2019-12-25 15:55:24 +08:00
kezhenxu94 9b408e82d5 [Feature] Support ElasticSearch 7 as backend storage (#3870)
* [Feature] Officially support ElasticSearch 7 as backend storage

* Extract bootstrap module to be shared by multiple starters

* Add missing configuration file and distinguish different version in test

* Update known-oap-backend-dependencies-es7.txt

Update the dependency lib

* Fix missing adaptation to ElasticSearch 7

* Rename ElasticSearch 7 specific class with Es7 infix

* Fix miss-adapted ElasticSearch DAO API

* Add ES7 specific configuration and polish documentations

* Polish documentations

* Polish documentations
2019-11-21 09:26:23 +08:00
aderm 7d3b285b6b 提取公共代码 (#3566) 2019-10-07 16:49:13 +08:00
Jared Tan dbbf22a177 support es https ssl config. (#3370)
* support es https ssl config.

* update

* update docs.

* refactoring var name.

* fix

* update docs.

* add config in es section.

* fix logical.
2019-09-02 11:15:43 +08:00
Jared Tan 02affffa50 support elasticsearch address configuration with http scheme. (#3236)
* support elasticsearch address configuration with http scheme.

* update logical.
2019-08-09 19:15:04 +08:00
彭勇升 pengys 594994ffae 1. Provide InsertRequest and UpdateRequest interface for prepare persistence. (#3131)
2. Implement the ids query for H2 metrics DAO.
2019-07-21 13:56:12 +08:00
彭勇升 pengys 6338de392f Improve OAP server performance. (#3127)
* Improve OAP server performance.
2019-07-20 22:59:52 +08:00
彭勇升 pengys ed507b78ef
Dirty-Read in concurrency (#3117)
* Fixed the problem of dirty reading of the storage implementation of elasticsearch.

* Fixed a mistake.
2019-07-19 14:22:01 +08:00
zhangwei deb2d90029 fix deleteIndex repeat append namespace (#3017)
* fix deleteIndex repeat append namespace

* fix deleteIndex repeat append namespace

* fix checkStyle

* fix

* fix test assert

* update test method name

* fix test

* fix

* del elastic.search.namespace

* fix check style

* update ITElasticSearchNamespaceClient to ITElasticSearchClientOfNamespace

* move undoFormatIndexName to ITElasticSearchClien

* move #getIndex to ITElasticSearchClient

* add deleteByModelName and deleteByIndexName

* add deleteByModelName and deleteByIndexName

* revert ITElasticSearchClient#getIndex

* Use whitebox get from powermock to get this.

* undo namespacePrefix
2019-07-12 07:53:11 +09:00
吴晟 Wu Sheng 4deae17fb3 Fix lose query data bug. (#2941) 2019-06-25 15:06:31 +08:00
彭勇升 pengys 129d2918ac Use ids query method instead of get method for metrics persistence. (#2934)
* Use ids query method instead of get method for metrics persistence.

#2933

* Query by alias name.
2019-06-23 16:53:45 +08:00
彭勇升 pengys 70064f47cb Use the ids query method instead of get method. (#2887)
#2886
2019-06-17 12:05:21 +08:00
kezhenxu94 f5db90b8bd Integrate Jenkins CI (#2864)
* Test

* Sync

* Update ITNacosConfigurationTest.java

* Trigger CI

* Trigger CI

* Trigger CI

* Trigger CI

* Trigger CI

* Remove unused imports to fix checkstyle

* Remove unused imports to fix checkstyle

* Increase timeout since nacos starts slowlly

* Bump up Oracle JDK version

* Remove OracleJDK due to license

* Add CI-with-IT profile and openjdk9

* Add missing profiles

* Lift failsafe plugin to root

* Update pom.xml

* Remove OPENJDK9

* Try parallel builds and add agent

* Try parallel build and windows

* Try parallel build and windows

* Try parallel build and windows

* Try parallel build

* Try parallel build

* Try parallel build

* Fix indentation

* Trigger CI

* Trigger CI

* Remove Windows stage

* Update mvnw.cmd
2019-06-13 17:06:21 +08:00
彭勇升 pengys 4e63331d35 #2860 (#2861)
Get the document by index name.
2019-06-12 18:56:11 +08:00
彭勇升 pengys 5036e83343 Good news of an important feature: time series implementation in Elasticsearch storage. (#2808)
* The new feature of Elasticsearch storage with the time series implementation.

* Time bucket util improve.

* Fixed some Gramma error.

* Update ttl.md

Polish English a little.

* Update ttl.md

Add more explicit description of TTL

* Update ttl.md

* Update StorageModuleElasticsearchConfig.java

Set otherMetricsDataTTL = 0 as default, make user more clear.

* Rename the timeSeriesAble to capableOfTimeSeries

* Fixed a test case failure.
2019-06-03 18:46:42 +08:00
彭勇升 pengys 73e8853fa5 Provide elasticsearch template operation method. (#2474)
* Provide elasticsearch template operation method.

* Fixed compile failure issue.
2019-04-11 20:08:51 +08:00
彭勇升 pengys 4beaac001b Integration test for elastic search client. (#2464)
* Integration test for elastic search client.

* Just test verify command.

* CI command order change.

* Remove maven clean.

* no message

* Create Jenkinsfile to try the apache build for our CI.
2019-04-10 10:15:54 +08:00
吴晟 Wu Sheng e59a488005
Setup Jaeger receiver and revert #2425 (#2431)
* Fix missing licenses in dist.

* Move all CoreRegisterLinker together into SharingReceiver.

* Setup receiver endpoint

* Revert "Provide create, delete, isExists template method by elastic client. (#2425)"

This reverts commit e6a7720f03.
2019-04-01 10:52:29 -07:00
彭勇升 pengys e6a7720f03 Provide create, delete, isExists template method by elastic client. (#2425) 2019-03-30 13:19:10 -07:00
kl c69eafefad Added elasticsearch warehouse certification support (#2384)
* Added elasticsearch warehouse certification support

* Add the elasticsearch service for Http Basic instructions

* Update backend-storage.md

Change document.
2019-03-20 21:09:29 -07:00
a198720 b1b5ee5103 fixed the index name to format name in logs (#2165)
* fixed  the index name to format name in logs

* modify the log level from info to debug
2019-01-16 16:34:59 +08:00
彭勇升 pengys 27c475f74c 1. Make storage and cluster using same name space. (#2014)
2. Zookeeper implementation of the cluster module using name space to support using same zookeeper cluster by different OAP clusters.
2018-12-06 21:03:00 +08:00
吴晟 Wu Sheng 215eab7428
Mysql style storage implementation (#1964)
Also fixed several bugs in this merge.
2018-11-28 22:59:02 +08:00
a198720 578fd70395 fixed bug: multiple formatIndexName in forceUpdate method (#1960) 2018-11-25 22:14:08 +08:00
彭勇升 pengys 7c8a683c2f
gRPC client usage improve (#1946)
* Close the clients which are unreachable.
Remote client manager test case and comments.

* Test the GRPCRemote client.

* 1. Catch the throwable for onComplete method cause of it will throw exception when connection lost.
2. Check the gRPC channel state, send message when state is ready, wait 5 second when state is not ready.

Notice: gRPC channel getState with true parameter will trigger reconnect operation.

* gRPC client will reconnect to the server when network recorvered.

* Recovery application.yml

* Recovery proto module commit id.

* no message

* Fixed compile error.
2018-11-22 20:50:27 +08:00
吴晟 Wu Sheng a9489b7067
H2 storage (#1712)
* H2 storage code skeleton.

* Add batch dao for H2.

* Fix wrong refactor.

* Add codes of other dao. No metric dao, wait for @peng-yongsheng new PR.

* Change license of distribution

* Add topology query in H2.

* Add more daos.

* Add some daos.

* Commit all H2 daos. Wait @peng-yongsheng to submit new IAggregationQueryDAO interface.

* Fix some CIs

* Fix wrong refactor, which causes h2 plugin failure.

* Fix all codes of H2 storage. :) and CI should pass. No test.

* Fix CI and typo

* Fix connection closed.

* Fix register bug

* no message

* Fixed service inventory register bug.

* Fix a compile issue.

* Sync change

* Reset es as default storage.

* Fix rat

* Fix H2 compile issue.

* Fix bugs for H2 storage.

* Fix bug in mesh.

* 1. Fix Thermodynamic query bug
2. Fix service inventory query bug
3. Fix missing traceid in trace query bug

* Fix query bug.

* Submit reverse caused by prev PR.

* Fix duplicate spans.

* Use H2 as default storage implementor.

* Fix bug in topology query.

* Fix CI

* Fix rat

* Fix two query in H2 storage.

* Fix H2 query

* Make release running in H2 mode and develop mode running is ES mode in default
2018-10-28 10:31:58 +08:00
彭勇升 pengys 7bb577a4e9 Provide auto delete history data service. (#1797) 2018-10-22 20:20:29 +08:00
彭勇升 pengys 7e149ab3f6 Metric query. (#1677) 2018-09-15 15:53:28 +08:00
彭勇升 pengys 6c88659c0c [OAP Server] Register lock implementation. (#1528)
* Use elasticsearch's document version to implement a lock for register source id.

* Return false when could not lock the document.
2018-08-07 17:34:36 +08:00
彭勇升 pengys 06165a0359 Feature/oap/storage (#1516)
* Storage and Persistence.

* Storage config.

* Fixed the CI failure.
2018-08-03 12:01:53 +08:00
彭勇升 pengys efe5299d03 Feature/oap/remote (#1505)
* Sample operator code.

* Indicator aggregator framework.

* Provide some annotation for OAL.

* Remote module.

* Register service.

* Add apache license header.

* Ignore comments when load definition files.
2018-07-30 09:54:36 +08:00
彭勇升 pengys de3bc220e0 [OAP Server] Core framework (#1472)
* Receiver, aggregate, server finished.
2018-07-19 11:24:01 +08:00