Commit Graph

104 Commits

Author SHA1 Message Date
吴晟 Wu Sheng 6fe2041b47
Make 8.0.0 Core available. New protocol and register removed. (#4599)
Here are the list of changes we made in this pull request. Top 3 are the principle changes, others are following these principles.
1. New agent and mesh report protocol. 
2. New agent header protocol. 
3. Service register, instance register and network address register have been removed permanently.
4. Service traffic, instance traffic and network alias metrics are added to replace the service, instance and network address inventory.
5. Register process has been removed.
6. Metrics stream process supports insert only mode, especially for traffic entities.
7. Metrics stream process supports no-downsampling mode for traffic entities and network alias.
8. Remove all register mechanism and cache in the java agent.
9. Remove MONTH step in GraphQL query.
10. Update UI to remove MONTH step query, the max query range is 60 days now.
11. Simplify the TTL to metrics and record. And the unit has been formatted in Day unit. No specific TTL for ElasticSearch storage.
12. Buffer mechanism of trace receiver and mesh receiver has been removed due to no register.
13. New service id, instance id and endpoint id rules, including service relation, instance relation and endpoint relation id rules.
14. Java agent support `keep tracing` mode, meaning, agent generating tracing context even the backend is unconnected/unavailable.
15. Plugin test tool up to date, in order to support new protocol.
16. Plugin tests expected data files updated.
17. E2E tests updated.
18. Telemetry of Grafana config has been merged into one.
19. Documentation updates.
20. [TBD] InfluxDB storage implementation is not available, need @dmsolr to fix later, in order to reduce the master change block by this PR.
2020-04-10 11:03:24 +08:00
吴晟 Wu Sheng fa9a44b750
Version number changed including a little documentation. (#4597) 2020-03-31 12:57:20 +08:00
吴晟 Wu Sheng 61f98c81f0
Move the main branch to 7.1.0 iteration. (#4537)
* [maven-release-plugin] prepare release v7.0.0

* [maven-release-plugin] prepare for next development iteration
2020-03-19 10:38:41 +08:00
kezhenxu94 69e3d80e24
Add selector property in application.yml (#4514)
### Motivation

 Ease the efforts of modifying configurations in automatic workflow and scripts, like Docker and helm chart.

 ### Modification

 Add a `selector` property into `application.yml` to enable switching configuration simply by environment variables or system properties, and filter the unselected options, leaving only selected one.

 ### Result

 - We can switch a configuration by environment variables or system property

 - Closes #4511
2020-03-15 23:28:14 +08:00
Gao Hongtao 78d14ebbd6
Enable OAP gRPC SSL transportation (#4470)
* Enable OAP gRPC SSL transportation

Porting to OpenSSL to enable SSL transportation. The server private
key is in the format of PCKS#8, certificates is x509 though.

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add netty-tcnative-boringssl-static 2.0.26 licnese

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update docs/en/setup/backend/grpc-ssl.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* Update docs/en/setup/backend/grpc-ssl.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* Add SSL to grpc remote client and sharing server

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update documents

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Fix nits

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Fix test case

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add e2e-ssl and remove other test cases temporarily

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add TLS relevant files

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Decrease timeout

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Test config works

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* triger CI on oap-ssl branch

* Disable agent ca

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Switch to sharing server

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add agent log to console

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Upload log files

* Change backend

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Revert "Change backend"

This reverts commit 7a085711e3653d3bf642aa84f73650e22595e383.

* Update backend

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Revert e2e

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Ignore TSL relevant files

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Polish codes

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update certificates expires year to 2120

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-03-15 08:15:28 +08:00
吴晟 Wu Sheng e69391a207
Support Secrets Management File in the ElasticSearch 6/7 storage (#4493)
* Temp commit

* Support secretsManagementFile file.

* Update doc.

* 1. Support JKS/pass runtime change too.
2. Follow review.

* Fix format.

* Fix username/password/trustPass haven't been updated in the es client.

* Fix doc issue.

Co-authored-by: Jared Tan <jian.tan@daocloud.io>
2020-03-13 20:50:14 +08:00
吴晟 Wu Sheng 069b99bdbc
Add file change detection mechanism (#4475)
* Add file change detection mechanism

* Fix format.

* Make file content changed monitor easier.

* Add secondary round change and check the result.

* Update to support multiple files change monitoring.
2020-03-11 10:29:44 +08:00
Jared Tan aa21765628
support http api for upstream trace. (#4399)
* support http api.

* e2e

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-03-06 21:07:39 +08:00
吴晟 Wu Sheng 329f7e150b
Support Downsampling Data Packing feature in ES storage implementation (#4364)
* Support day/hour/minute metrics merging into one index. Reduce the number of index 50%.
2020-02-16 09:03:30 +08:00
吴晟 Wu Sheng ae442e364f
More comments for important classes (#4361) 2020-02-14 17:03:50 +08:00
Jared Tan 7ff9a0cfd9
supprt gRPC alarm hook. (#4344)
* support gRPC hook.

* revert submodule commit.

* revert submodule commitid.

* finish logical.

* remove author anonatation.

* fix ci.

* revert submodule.

* update logical.

* update logical.

* update logical.

* fix

* fix

* fix empty settings.

* fix.

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-12 16:20:20 +08:00
kezhenxu94 5b255ba3e3
Set up more strict code styles and fix existing issues (#4337)
Motivation:

Review code styles with some bots automatically.

Modifications:

Set up ReviewDog in GitHub Action to review code style.

Add more check rules to checkstyle plugin.

Result:

Obvious code styles can be reviewed and commented automatically.
2020-02-11 17:10:49 +08:00
kezhenxu94 019af3e345
Drop support of protocol v1 (sw3) (#4244)
* Drop support of protocol v1 (sw3)

* Remove versioning logic further

* Remove more version5-only classes

* Commit missing files

* Update submodule and address review comment

* Fix failed tests
2020-01-17 20:33:52 +08:00
John Niang 738e4975cb Fix typo: guideline and code document (#4248) 2020-01-16 19:38:03 +08:00
Jared Tan 1310719645
add token authentication between agent and oap receiver. (#4197)
* add token authentication between agent and oap receiver.

* revert ui.

* fix code style.

* update docs.

* Update docs/en/setup/backend/backend-token-auth.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* fix hits.

* filter

* revert yml.

* update logical.

* fix codestyle check.

* fix ci and add e2e test case.

* revert mistake commit.

* fix ci

* remove duplicate maven dependency.

* revert pom

* revert unchanged.

* update javadocs.

* update logical.

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-12 23:08:39 +08:00
mrproliu 4cf0600f35 Provide profile task downstream to sniffer (#4172)
* Provide profile task downstream to sniffer

* fix agent unit testcase issue

* add profile switch config on sniffer

* fix es error

* 1. add @DefaultImplementor on the sniffer profile task service
2. change ProfileTaskExecutionService#PROFILE_TASK_READY_SCHEDULE to final and remove volatile
2. fix style error

* change timeFromStartMills use `<` to compare

* 1. add `maxSamplingCount` to profile task
2. make profile task limit to the common package

* 1. change `agent.active_profile` to `profile.active` and make true on default
2. add `maxSamplingCount` in profile task
3. use `createTime` to check has new command list
4. add task re-check before process profile task

* 1. add `profile-receiver` document
2. change `ProfileTaskExecutionService` use single schedule thread pool
3. cache dont need fetch data when no data, use auto-fresh mechanism only

* remove navigate time judge, fix comment wrong meaning
2020-01-09 09:35:39 +08:00
Weiyi Liu 9415f68127 Fix the wrong comments. (#4139) 2019-12-27 12:10:37 +08:00
吴晟 Wu Sheng b570996006
Update JDK compiling level to 8, and remove zip dist. (#4119)
* Update JDK compiling level to 8, and remove zip dist.

* Exclude annotation and shade API.

* Fix more exclusion.

* 1. Update grava and gson 2. Fix compile issue. 3. Keep protoc and maven plugin in old version. Some incompatible of 3rd party grpc proto.

* Fix dependency conflicts, license and notice issue, and dependency check script issue

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2019-12-25 15:55:24 +08:00
吴晟 Wu Sheng d7f7799c80
Set the main branch to 7.0.0-SNAPSHOT (#4115)
* [maven-release-plugin] prepare release v6.6.0

* [maven-release-plugin] prepare for next development iteration
2019-12-24 16:37:13 +08:00
kezhenxu94 9b408e82d5 [Feature] Support ElasticSearch 7 as backend storage (#3870)
* [Feature] Officially support ElasticSearch 7 as backend storage

* Extract bootstrap module to be shared by multiple starters

* Add missing configuration file and distinguish different version in test

* Update known-oap-backend-dependencies-es7.txt

Update the dependency lib

* Fix missing adaptation to ElasticSearch 7

* Rename ElasticSearch 7 specific class with Es7 infix

* Fix miss-adapted ElasticSearch DAO API

* Add ES7 specific configuration and polish documentations

* Polish documentations

* Polish documentations
2019-11-21 09:26:23 +08:00
吴晟 Wu Sheng 3fbaec42dc
Move to 6.6.0 SNAPSHOT (#3839)
* [maven-release-plugin] prepare release v6.5.0

* [maven-release-plugin] prepare for next development iteration
2019-11-12 22:33:39 +08:00
panjuan fb001867e5 Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564)
* Is it a typo or not?

* delete datasource-settings.properties

* handle 'metadataQueryMaxSize'

* delete DynamicModuleConfig.java

* modify mysql config

* parsing yaml to support the configuration for multiple properties

* delete datasource-settings.properties

* modify doc

* add properties for docker-entrypoint.sh

* add testLoadConfig()
2019-10-12 20:47:38 +08:00
aderm 7d3b285b6b 提取公共代码 (#3566) 2019-10-07 16:49:13 +08:00
kezhenxu94 81f4c087b0
Provide Consul dynamic configuration center implementation (#3560)
* Provide Consul dynamic configuration center implementation

* Add more unit tests

* Fix unit test

* Add more unit tests and remove unused file

* Remove redundant doc
2019-10-05 10:49:53 +08:00
吴晟 Wu Sheng ce20fe107d
Move to 6.5.0 (#3439)
* [maven-release-plugin] prepare release v6.4.0

* [maven-release-plugin] prepare for next development iteration
2019-09-08 16:24:21 -07:00
Jared Tan dbbf22a177 support es https ssl config. (#3370)
* support es https ssl config.

* update

* update docs.

* refactoring var name.

* fix

* update docs.

* add config in es section.

* fix logical.
2019-09-02 11:15:43 +08:00
kezhenxu94 54614a0594 Support gateway without agent (#3308)
* Support Gateway without agent
2019-08-30 16:17:41 +08:00
kezhenxu94 db9759fcc2 Check license header strictly (#3274)
* Check license header strictly

* Ignore header template

* Include more files to check list
2019-08-16 14:37:44 +08:00
Jared Tan 02affffa50 support elasticsearch address configuration with http scheme. (#3236)
* support elasticsearch address configuration with http scheme.

* update logical.
2019-08-09 19:15:04 +08:00
yantaowu 1bfda1dd65 Modify the thread pool of grpc server. (#3225)
* Modify the thread pool of grpc server.

* add license header

* make param confingable

* Add sharing and Jaeger config.

* Remove grpc thread config from yml
2019-08-09 09:14:55 +08:00
吴晟 Wu Sheng b247dc979e
Move project master branch to 6.4.0-SNAPSHOT (#3220)
* [maven-release-plugin] prepare release v6.3.0

* [maven-release-plugin] prepare for next development iteration
2019-08-05 21:19:17 +08:00
彭勇升 pengys 594994ffae 1. Provide InsertRequest and UpdateRequest interface for prepare persistence. (#3131)
2. Implement the ids query for H2 metrics DAO.
2019-07-21 13:56:12 +08:00
彭勇升 pengys 6338de392f Improve OAP server performance. (#3127)
* Improve OAP server performance.
2019-07-20 22:59:52 +08:00
彭勇升 pengys 8d70533921 Fixed buffer lost. (#3118) 2019-07-19 22:07:53 +08:00
彭勇升 pengys ed507b78ef
Dirty-Read in concurrency (#3117)
* Fixed the problem of dirty reading of the storage implementation of elasticsearch.

* Fixed a mistake.
2019-07-19 14:22:01 +08:00
kezhenxu94 4db1773d3e Fix wrong logger names (#3075)
* Fix wrong logger name

* Fix more wrong logger names in agent side
2019-07-15 00:35:15 +08:00
zhangwei deb2d90029 fix deleteIndex repeat append namespace (#3017)
* fix deleteIndex repeat append namespace

* fix deleteIndex repeat append namespace

* fix checkStyle

* fix

* fix test assert

* update test method name

* fix test

* fix

* del elastic.search.namespace

* fix check style

* update ITElasticSearchNamespaceClient to ITElasticSearchClientOfNamespace

* move undoFormatIndexName to ITElasticSearchClien

* move #getIndex to ITElasticSearchClient

* add deleteByModelName and deleteByIndexName

* add deleteByModelName and deleteByIndexName

* revert ITElasticSearchClient#getIndex

* Use whitebox get from powermock to get this.

* undo namespacePrefix
2019-07-12 07:53:11 +09:00
吴晟 Wu Sheng 0225711e26
Check duplicate providers in bootstrap (#2976)
* Enhance module manager
2019-07-01 15:54:12 +08:00
吴晟 Wu Sheng 4faa950c14
Move master to 6.3.0 (#2968)
* [maven-release-plugin] prepare release v6.2.0

* [maven-release-plugin] prepare for next development iteration
2019-06-29 13:49:36 +08:00
吴晟 Wu Sheng 4deae17fb3 Fix lose query data bug. (#2941) 2019-06-25 15:06:31 +08:00
彭勇升 pengys 129d2918ac Use ids query method instead of get method for metrics persistence. (#2934)
* Use ids query method instead of get method for metrics persistence.

#2933

* Query by alias name.
2019-06-23 16:53:45 +08:00
daming fa7a1983f9 Fix ModuleProvider is repeated loading (#2923) 2019-06-21 21:01:20 +08:00
彭勇升 pengys 70064f47cb Use the ids query method instead of get method. (#2887)
#2886
2019-06-17 12:05:21 +08:00
kezhenxu94 f5db90b8bd Integrate Jenkins CI (#2864)
* Test

* Sync

* Update ITNacosConfigurationTest.java

* Trigger CI

* Trigger CI

* Trigger CI

* Trigger CI

* Trigger CI

* Remove unused imports to fix checkstyle

* Remove unused imports to fix checkstyle

* Increase timeout since nacos starts slowlly

* Bump up Oracle JDK version

* Remove OracleJDK due to license

* Add CI-with-IT profile and openjdk9

* Add missing profiles

* Lift failsafe plugin to root

* Update pom.xml

* Remove OPENJDK9

* Try parallel builds and add agent

* Try parallel build and windows

* Try parallel build and windows

* Try parallel build and windows

* Try parallel build

* Try parallel build

* Try parallel build

* Fix indentation

* Trigger CI

* Trigger CI

* Remove Windows stage

* Update mvnw.cmd
2019-06-13 17:06:21 +08:00
彭勇升 pengys 4e63331d35 #2860 (#2861)
Get the document by index name.
2019-06-12 18:56:11 +08:00
吴晟 Wu Sheng 2e846c6078 Dynamic Configuration and Dynamic Configuration Service (#2810)
* Implement the base of config API module.

* Finish configuration module codebases w/o tests, revert some changes.

* Finish mock tests.

* Add new gRPC service for configuration.

* Make gRPC configuration sync provider works.

* Add database slow statement dynamic setting.

* Finish the dynamic configuration service.

* Fix version.
2019-06-03 23:11:52 +08:00
彭勇升 pengys 5036e83343 Good news of an important feature: time series implementation in Elasticsearch storage. (#2808)
* The new feature of Elasticsearch storage with the time series implementation.

* Time bucket util improve.

* Fixed some Gramma error.

* Update ttl.md

Polish English a little.

* Update ttl.md

Add more explicit description of TTL

* Update ttl.md

* Update StorageModuleElasticsearchConfig.java

Set otherMetricsDataTTL = 0 as default, make user more clear.

* Rename the timeSeriesAble to capableOfTimeSeries

* Fixed a test case failure.
2019-06-03 18:46:42 +08:00
彭勇升 pengys 0a696813c8 Time bucket util improve. (#2795) 2019-05-31 15:02:49 +08:00
彭勇升 pengys ed78dabe8b Rename metric and indicator to metrics. (#2643)
* Rename metric to metrics.

* Fixed test case execute failure issues.
2019-05-10 08:05:37 +08:00
Michael Hausegger ac2cc7339d Added Unit Tests to increase code coverage (#2571) 2019-05-03 08:29:24 +08:00