Commit Graph

46 Commits

Author SHA1 Message Date
Zhenxu Ke c5e0c01f06
Add OpenSearch as a storage option (#6916) 2021-05-09 10:09:15 +08:00
Evan 2af77d4d03
make sync metrics concurrency (#6642)
* make sync metrics concurrency

* add changelog

* add changelog

* polish codes

* change default value

* remove unnecessary codes

Co-authored-by: Evan <evanljp@outlook.com>
Co-authored-by: Zhenxu Ke <kezhenxu94@apache.org>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2021-04-02 11:33:46 +08:00
吴晟 Wu Sheng 0e1d5c3767
Make Zipkin trace query available (#6454) 2021-02-27 14:50:54 +08:00
haoyann c45bb361ee
Storage plugin supports PostgreSQL (#6345) 2021-02-20 20:59:51 +08:00
Evan e138d29778
add text analyzer for es (#6249) 2021-01-25 17:01:43 +08:00
Jared Tan 8d164d3dd8
Add Tidb provider module. (#5844)
* fix transaction too large error when use TiDB as storage

* add CHANGES.md

* fix typo

* add tidb storage provider module.

* add e2e.

* revert ui submodule.

* fix license.

* Fix wrong health check test

* add tidb ttl e2e.

Co-authored-by: moonsphere <moonsphere@m.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-11-22 23:28:54 +08:00
吴晟 Wu Sheng d1c8831d31
Add FAQ about VNode on the UI. (#5768)
* Add FAQ about VNode on the UI.

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-11-02 15:51:21 +08:00
Ax1an 287b8b9be5
Add syncBulkActions config to avoid the large amount of metrics data written ES in a single sync bulk request. (#5699) 2020-10-21 12:23:32 +08:00
Gallardot b24e3f3737
Fix typo in backend-storage.md (#5662) 2020-10-14 17:07:17 +08:00
Evan 15b0068fdf
Add superdataset replicas number config (#5424)
* distinct the number of metrics or common record index with super dataset index.
2020-09-01 23:06:21 +08:00
Evan 2df3c683ba
Super Size Dataset record index es rolling step (#5282) 2020-08-10 14:49:11 +08:00
吴晟 Wu Sheng 2c676ead24
Add OAP configuration vocabulary documentation. (#5149) 2020-07-23 08:24:40 +08:00
吴晟 Wu Sheng 6fe2041b47
Make 8.0.0 Core available. New protocol and register removed. (#4599)
Here are the list of changes we made in this pull request. Top 3 are the principle changes, others are following these principles.
1. New agent and mesh report protocol. 
2. New agent header protocol. 
3. Service register, instance register and network address register have been removed permanently.
4. Service traffic, instance traffic and network alias metrics are added to replace the service, instance and network address inventory.
5. Register process has been removed.
6. Metrics stream process supports insert only mode, especially for traffic entities.
7. Metrics stream process supports no-downsampling mode for traffic entities and network alias.
8. Remove all register mechanism and cache in the java agent.
9. Remove MONTH step in GraphQL query.
10. Update UI to remove MONTH step query, the max query range is 60 days now.
11. Simplify the TTL to metrics and record. And the unit has been formatted in Day unit. No specific TTL for ElasticSearch storage.
12. Buffer mechanism of trace receiver and mesh receiver has been removed due to no register.
13. New service id, instance id and endpoint id rules, including service relation, instance relation and endpoint relation id rules.
14. Java agent support `keep tracing` mode, meaning, agent generating tracing context even the backend is unconnected/unavailable.
15. Plugin test tool up to date, in order to support new protocol.
16. Plugin tests expected data files updated.
17. E2E tests updated.
18. Telemetry of Grafana config has been merged into one.
19. Documentation updates.
20. [TBD] InfluxDB storage implementation is not available, need @dmsolr to fix later, in order to reduce the master change block by this PR.
2020-04-10 11:03:24 +08:00
kezhenxu94 69e3d80e24
Add selector property in application.yml (#4514)
### Motivation

 Ease the efforts of modifying configurations in automatic workflow and scripts, like Docker and helm chart.

 ### Modification

 Add a `selector` property into `application.yml` to enable switching configuration simply by environment variables or system properties, and filter the unselected options, leaving only selected one.

 ### Result

 - We can switch a configuration by environment variables or system property

 - Closes #4511
2020-03-15 23:28:14 +08:00
吴晟 Wu Sheng e69391a207
Support Secrets Management File in the ElasticSearch 6/7 storage (#4493)
* Temp commit

* Support secretsManagementFile file.

* Update doc.

* 1. Support JKS/pass runtime change too.
2. Follow review.

* Fix format.

* Fix username/password/trustPass haven't been updated in the es client.

* Fix doc issue.

Co-authored-by: Jared Tan <jian.tan@daocloud.io>
2020-03-13 20:50:14 +08:00
kezhenxu94 9c974b80b4
Fix unclear doc and add missing variable (#4484)
* Revert "[BugFix] Docker InfluxDB cannot be enabled (#4483)"

This reverts commit 73d0802cd6.

* Fix unclear doc and add missing variable
2020-03-11 07:46:59 +08:00
Daming 29da5738bc
Provide influxdb as a new storage plugin (#4239)
* provide influxdb as a new storage plugin

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-19 18:00:57 +08:00
吴晟 Wu Sheng 334239388a
Support Daily Index Step feature in the OAP (#4368)
* Support Daily Index Step.

* Fix format.

* Fix mischanged ES.
2020-02-16 20:48:04 +08:00
吴晟 Wu Sheng 329f7e150b
Support Downsampling Data Packing feature in ES storage implementation (#4364)
* Support day/hour/minute metrics merging into one index. Reduce the number of index 50%.
2020-02-16 09:03:30 +08:00
mrproliu d38a8eaaa7 Provide thread monitor create task feature (#4145)
* provide thread monitor task service to create

* provide thread monitor task GraphQL implements

* 1. change thread monitor field names
2. fix `getThreadMonitorTaskList` query time bucket error

* provide config stream processor

* update submodule `query-protocol`

* resolve e2e-mysql error

* remove useless storage interface method.

* rename `ThreadMonitorTask` to `ProfileTask`

* change e2e-profile to the top level

* fix profile test analyze error

* remove StringUtil#isBlank.

* 1. remove create profile task duration unit
2. remove GraphQL getTask list duration field
3. add `profileTaskQueryMaxSize` in `storage` -> `elasticsearch(7)` configuration, default get 200 profile task

* provide e2e different storage tests

* 1. fix rat check
2. remove DurationUtils.java#toSecond
3. remove ProfileTaskQueryEs7DAO

* fix e2e code format error

* provide es6 and es7 storage e2e tests

* change e2e profile es module artifactId

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2019-12-31 20:49:45 +08:00
kezhenxu94 aa022ffa00 Remove unused ES7 config (#3998) 2019-12-04 00:20:10 +08:00
Jared Tan d581f2294f Support custom elasticsearch settings in index level. (#3913) 2019-11-21 22:30:59 +08:00
kezhenxu94 9b408e82d5 [Feature] Support ElasticSearch 7 as backend storage (#3870)
* [Feature] Officially support ElasticSearch 7 as backend storage

* Extract bootstrap module to be shared by multiple starters

* Add missing configuration file and distinguish different version in test

* Update known-oap-backend-dependencies-es7.txt

Update the dependency lib

* Fix missing adaptation to ElasticSearch 7

* Rename ElasticSearch 7 specific class with Es7 infix

* Fix miss-adapted ElasticSearch DAO API

* Add ES7 specific configuration and polish documentations

* Polish documentations

* Polish documentations
2019-11-21 09:26:23 +08:00
野狼Bang 464d7c1962 Typo fix in document (#3764) 2019-11-01 12:11:58 +05:30
panjuan fb001867e5 Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564)
* Is it a typo or not?

* delete datasource-settings.properties

* handle 'metadataQueryMaxSize'

* delete DynamicModuleConfig.java

* modify mysql config

* parsing yaml to support the configuration for multiple properties

* delete datasource-settings.properties

* modify doc

* add properties for docker-entrypoint.sh

* add testLoadConfig()
2019-10-12 20:47:38 +08:00
Jared Tan dbbf22a177 support es https ssl config. (#3370)
* support es https ssl config.

* update

* update docs.

* refactoring var name.

* fix

* update docs.

* add config in es section.

* fix logical.
2019-09-02 11:15:43 +08:00
Jared Tan 02affffa50 support elasticsearch address configuration with http scheme. (#3236)
* support elasticsearch address configuration with http scheme.

* update logical.
2019-08-09 19:15:04 +08:00
吴晟 Wu Sheng eff9faa8c9
Update backend-storage.md (#3222)
* Update backend-storage.md

* Update backend-storage.md

* Update ES-Server-FAQ.md
2019-08-06 21:19:00 +08:00
吴晟 Wu Sheng 0922548069
Update backend-storage.md (#2950) 2019-06-27 00:14:37 +08:00
吴晟 Wu Sheng fe66feccb7
Update backend-storage.md (#2915) 2019-06-20 12:47:48 +08:00
彭勇升 pengys 5036e83343 Good news of an important feature: time series implementation in Elasticsearch storage. (#2808)
* The new feature of Elasticsearch storage with the time series implementation.

* Time bucket util improve.

* Fixed some Gramma error.

* Update ttl.md

Polish English a little.

* Update ttl.md

Add more explicit description of TTL

* Update ttl.md

* Update StorageModuleElasticsearchConfig.java

Set otherMetricsDataTTL = 0 as default, make user more clear.

* Rename the timeSeriesAble to capableOfTimeSeries

* Fixed a test case failure.
2019-06-03 18:46:42 +08:00
吴晟 Wu Sheng ff6fb90d38
Add jaeger traces support (#2434)
* Make Jaeger receiver works in 12450 as default.

* Finish compile and codebase.

* Make codes runnable.

* Finish Jaeger query tests.

* Add document for jaeger receiver.

* Add readme.

* Fix a break link.

* Finish doc.
2019-04-01 20:52:59 -07:00
吴晟 Wu Sheng 601b472ef6 Support Backend acts as pure Zipkin collector (#2424)
* Codebase for zipkin span persistence.

* Fix missing fields in storage.

* Miss the latency field.

* Finish some tests.

* Fix wrong latency.

* Finish doc and reset application.yml

* Make the description more clear.
2019-03-31 01:36:36 +08:00
kl c69eafefad Added elasticsearch warehouse certification support (#2384)
* Added elasticsearch warehouse certification support

* Add the elasticsearch service for Http Basic instructions

* Update backend-storage.md

Change document.
2019-03-20 21:09:29 -07:00
彭勇升 pengys 27c475f74c 1. Make storage and cluster using same name space. (#2014)
2. Zookeeper implementation of the cluster module using name space to support using same zookeeper cluster by different OAP clusters.
2018-12-06 21:03:00 +08:00
吴晟 Wu Sheng e2f2356a6f
Update user wall and document (#1982)
* Update document and user wall.

* Change format.

* user wall updated.

* Fix typo
2018-11-30 16:07:56 +08:00
Jared.Tan 728033ce85 update application.yml for namespace. (#1975)
* Correct namespace description
2018-11-30 13:57:32 +08:00
吴晟 Wu Sheng 5577f67de7 Support backend sampling (#1977)
* Support sampling trace at server side and keep metric right.

* Add a trace sampling document

* Fix wrong default value.

* Fix document issue.

* Fix assemble issue.

* Fix wrong settings and doc.
2018-11-29 23:33:07 +08:00
吴晟 Wu Sheng f5545a87a5 Update backend-storage.md (#1979) 2018-11-29 20:46:27 +08:00
Jared.Tan 4392ec3ee9 TiDB storage provided and features tested (#1978)
* add TiDB configuration description.

* add tested mysql client driver version
2018-11-29 20:10:01 +08:00
吴晟 Wu Sheng 215eab7428
Mysql style storage implementation (#1964)
Also fixed several bugs in this merge.
2018-11-28 22:59:02 +08:00
Lemon c36ffb015f
Update backend-storage docs (#1861) 2018-11-01 20:01:32 +08:00
吴晟 Wu Sheng 2dd4f54957
Add some missing documents for v6 (#1830)
* Add more documents for v6.

* Make default setting consistent .
2018-10-28 21:45:58 +08:00
吴晟 Wu Sheng 7ad619ac91
Prepare new doc for v6 (#1695)
* Replace v5 changes by a link to 5.x branch. New v6 changelog.

* Finish query doc and fix a missing field in alarm doc.

* Add link to official oal in alarm doc.

* Fix words.

* Fix some doc

* Update README.md

* Update README.md

* Update README.md

* Update README.md
2018-10-07 09:51:29 +08:00
吴晟 Wu Sheng 64d886f197
Add document of contributions and some guides (#1672)
* Change oal doc.

* Remove typo

* Remove document WIP tag.

* Fix few words.

* no message

* Adjust contribution guides.
2018-09-13 21:01:36 +08:00
Wu Sheng f80b031250 Add many new chapters of v6 document. Make the document more clear. 2018-09-02 23:11:56 +08:00