Commit Graph

150 Commits

Author SHA1 Message Date
吴晟 Wu Sheng 9149de2a6e Support multiple linear values and merging p50/75/90/95/99 into percentile (#4214)
* Support new percentile func with new alarm and exporter for this new func.

* Fix e2e and OAL script

* Fix wrong column.

* Fix percentile bug and oal engine bug.

* Update query protocol and add percentile test case

* Support new query

* Adopt GraphQL requirement

* Fix wrong type cast.

* Fix query in H2 and ES.

* Fix docs and comments.

* Fix an e2e compile issue

* Fix javadoc issue and e2e test issue.

* Change CPM to Apdex in TTL test.

* Fix OAL for TTL e2e

* Add metrics query for service percentile.

* Fix OAL engine bug. Method deserialize is not working when more than two field types are IntKeyLongValueHashMap

* Support multiple IntKeyLongValueHashMap fields in remote. About serialize/deserialize methods.

* Fix graphql statement error in e2e.

* Fix serialize not working and add generated serialize/deserialize of percentile into test cases.

* Fix test case format

* Remove generated code test.

* Fix failed e2e test

* Use avg resp time to apdex in the TTL test.

* ADD multiple linear metrics check for endpoint in e2e cluster.

* Support `-` to represent no threshold and doc of alarm about this.

* Move break to right place.

* Fix wrong break(s)

* Fix break and add a test case for multiple values alarm.

* Fix format.

* Add more doc for this new feature and GraphQL query protocol.

Co-authored-by: Jared Tan <jian.tan@daocloud.io>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-12 22:10:50 +08:00
mrproliu 4cf0600f35 Provide profile task downstream to sniffer (#4172)
* Provide profile task downstream to sniffer

* fix agent unit testcase issue

* add profile switch config on sniffer

* fix es error

* 1. add @DefaultImplementor on the sniffer profile task service
2. change ProfileTaskExecutionService#PROFILE_TASK_READY_SCHEDULE to final and remove volatile
2. fix style error

* change timeFromStartMills use `<` to compare

* 1. add `maxSamplingCount` to profile task
2. make profile task limit to the common package

* 1. change `agent.active_profile` to `profile.active` and make true on default
2. add `maxSamplingCount` in profile task
3. use `createTime` to check has new command list
4. add task re-check before process profile task

* 1. add `profile-receiver` document
2. change `ProfileTaskExecutionService` use single schedule thread pool
3. cache dont need fetch data when no data, use auto-fresh mechanism only

* remove navigate time judge, fix comment wrong meaning
2020-01-09 09:35:39 +08:00
吴晟 Wu Sheng 6c20f4a41b
Add service type from register and support browser as service type (#4157)
* Add service type from register and support browser as service type

* Add service type at agent side.

* Fix format.

* Fix a test.

* Fix formats.

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-01 18:01:41 +08:00
mrproliu d38a8eaaa7 Provide thread monitor create task feature (#4145)
* provide thread monitor task service to create

* provide thread monitor task GraphQL implements

* 1. change thread monitor field names
2. fix `getThreadMonitorTaskList` query time bucket error

* provide config stream processor

* update submodule `query-protocol`

* resolve e2e-mysql error

* remove useless storage interface method.

* rename `ThreadMonitorTask` to `ProfileTask`

* change e2e-profile to the top level

* fix profile test analyze error

* remove StringUtil#isBlank.

* 1. remove create profile task duration unit
2. remove GraphQL getTask list duration field
3. add `profileTaskQueryMaxSize` in `storage` -> `elasticsearch(7)` configuration, default get 200 profile task

* provide e2e different storage tests

* 1. fix rat check
2. remove DurationUtils.java#toSecond
3. remove ProfileTaskQueryEs7DAO

* fix e2e code format error

* provide es6 and es7 storage e2e tests

* change e2e profile es module artifactId

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2019-12-31 20:49:45 +08:00
mrproliu 2fcf7c4649 provide config stream processor (#4133)
* provide config stream processor

* 1. rename config to none stream
2. add comments on none stream relate classes

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2019-12-27 14:58:15 +08:00
aderm 8476316e96 refactor code. (#4120) 2019-12-25 23:35:35 +08:00
吴晟 Wu Sheng d7f7799c80
Set the main branch to 7.0.0-SNAPSHOT (#4115)
* [maven-release-plugin] prepare release v6.6.0

* [maven-release-plugin] prepare for next development iteration
2019-12-24 16:37:13 +08:00
Juan Pan(Trista) 0fd634092b Modify query SQL for MySQLDAO (#4047)
* Modify MySQLDAO and H2DAO
2019-12-17 23:36:22 +08:00
zhang-wei 3741f4e386 [Feature] Service instance dependency (#3978)
* dispatcher

* getServiceInstanceTopology implementation

* CacheUpdateTimer

* fix compilation failure

* fix compilation failure

* fix esDao

* Update dao list

* dispatcher

* getServiceInstanceTopology implementation

* CacheUpdateTimer

* fix compilation failure

* fix compilation failure

* fix esDao

* Update dao list

* test e2e

* add official_analysis.oal & update getServiceInstanceTopology

* test instance topo for single service

* fix gql

* fix verifyServiceInstances

* verifyServiceInstanceTopo

* fix verifyServiceInstanceTopo

* fix ServiceInstanceInventory#name

* fix ServiceInstanceInventory#name

* e2e test

* fix action

* fix provider port

* fix Instance Node Type

* re run

* fix expected-data

* add e2e mysql

* set @Test timeout

* fix ci

* increase timeout

* re check

* test ttl es7

* test ttl

* fix action

* merge group

* test ttl es7

* replace official_analysis.oal

* replace official_analysis.oal

* comments

* add TODO

* add TODO

* fix filed name
2019-12-09 08:23:21 +08:00
kezhenxu94 aa022ffa00 Remove unused ES7 config (#3998) 2019-12-04 00:20:10 +08:00
Jared Tan d581f2294f Support custom elasticsearch settings in index level. (#3913) 2019-11-21 22:30:59 +08:00
kezhenxu94 9b408e82d5 [Feature] Support ElasticSearch 7 as backend storage (#3870)
* [Feature] Officially support ElasticSearch 7 as backend storage

* Extract bootstrap module to be shared by multiple starters

* Add missing configuration file and distinguish different version in test

* Update known-oap-backend-dependencies-es7.txt

Update the dependency lib

* Fix missing adaptation to ElasticSearch 7

* Rename ElasticSearch 7 specific class with Es7 infix

* Fix miss-adapted ElasticSearch DAO API

* Add ES7 specific configuration and polish documentations

* Polish documentations

* Polish documentations
2019-11-21 09:26:23 +08:00
吴晟 Wu Sheng 3fbaec42dc
Move to 6.6.0 SNAPSHOT (#3839)
* [maven-release-plugin] prepare release v6.5.0

* [maven-release-plugin] prepare for next development iteration
2019-11-12 22:33:39 +08:00
Jared Tan fbabcb1782 make query max window size configurable. (#3765)
* fix elasticsearch query data window size too large error.

* make query max window size configurable.
2019-11-01 16:02:06 +05:30
吴晟 Wu Sheng e9b404f1fa
Remove two max size 500 limit. (#3748) 2019-10-30 18:34:33 +05:30
Jared Tan 852f03d1db
remove unused time range query parameter. (#3730) 2019-10-28 22:21:54 +08:00
Kdump c0be757f43 add graphql getServiceInstance instanceUUID field. (#3595)
* add graphql getServiceInstance instanceUUID field.

* add graphql getServiceInstance instanceUUID field.

* add graphql getServiceInstance instanceUUID field.

* sync query-protocol repo

* add instanceUUID field e2e

* add instanceUUID field e2e

* add instanceUUID field e2e

* add instanceUUID field e2e
2019-10-17 06:38:00 +03:00
panjuan fb001867e5 Instead of datasource-settings.properties, use application.yml for MySQLStorageProvider (#3564)
* Is it a typo or not?

* delete datasource-settings.properties

* handle 'metadataQueryMaxSize'

* delete DynamicModuleConfig.java

* modify mysql config

* parsing yaml to support the configuration for multiple properties

* delete datasource-settings.properties

* modify doc

* add properties for docker-entrypoint.sh

* add testLoadConfig()
2019-10-12 20:47:38 +08:00
吴晟 Wu Sheng 6942ec8fcf
Language of instance could be NULL (#3485)
* Language of instance could be NULL

* Fix style check.

* Fix duplicated test cases.
2019-09-19 00:23:11 +08:00
Jared Tan 6c347a1169 agent support custom properties. (#3367)
* agent support custom properties.
2019-09-17 14:44:42 +08:00
吴晟 Wu Sheng ce20fe107d
Move to 6.5.0 (#3439)
* [maven-release-plugin] prepare release v6.4.0

* [maven-release-plugin] prepare for next development iteration
2019-09-08 16:24:21 -07:00
Jared Tan dbbf22a177 support es https ssl config. (#3370)
* support es https ssl config.

* update

* update docs.

* refactoring var name.

* fix

* update docs.

* add config in es section.

* fix logical.
2019-09-02 11:15:43 +08:00
Jared Tan cadaab03ae
set alarm record ordered by start time desc. (#3353) 2019-08-28 09:56:55 +08:00
彭勇升 pengys 3264fd4e31 Fixed #3328, keyword data type is not long enough for saving PXX metrics. (#3332) 2019-08-26 16:33:55 +08:00
webb2019 429c9b3c9c Data binary limit (#3321)
*  fix "DATA_BINARY" limit VARCHAR(20000)
2019-08-26 09:40:37 +08:00
吴晟 Wu Sheng dc0f6530cb
Fix TTL issue. (#3292)
* Fix TTL issue.

* Fix issue.

* Rename `getSecondTimeBucket` to `getRecordTimeBucket`.
2019-08-20 15:36:31 +08:00
吴晟 Wu Sheng 8888029e9a
Module config set override is not working (#3283)
* set Override is not working.

* Submodule update

* Add header.
2019-08-17 10:10:30 +08:00
kezhenxu94 db9759fcc2 Check license header strictly (#3274)
* Check license header strictly

* Ignore header template

* Include more files to check list
2019-08-16 14:37:44 +08:00
Jared Tan 02affffa50 support elasticsearch address configuration with http scheme. (#3236)
* support elasticsearch address configuration with http scheme.

* update logical.
2019-08-09 19:15:04 +08:00
吴晟 Wu Sheng 0df76e4e38
Remove the scope-meta file (#3226)
* Remove the meta file
2019-08-07 17:35:12 +08:00
吴晟 Wu Sheng b247dc979e
Move project master branch to 6.4.0-SNAPSHOT (#3220)
* [maven-release-plugin] prepare release v6.3.0

* [maven-release-plugin] prepare for next development iteration
2019-08-05 21:19:17 +08:00
kezhenxu94 e8f94d55c3 Fix bug: NetworkAddressInventory uses wrong builder to convert data (#3183) 2019-07-29 14:07:37 +08:00
彭勇升 pengys 16b8236506 1. All the inventories add last_update_time attribute, delete the mapping_last_update_time attribute in service inventory. (#3168)
2. Fixed the service inventory and net address inventory update bugs.
2019-07-26 14:02:27 +08:00
彭勇升 pengys f658d9eb4d Simplify the PxxMetrics and ThermodynamicMetrics to improve performance (#3162)
* Feature of database session

* Make it configurable.

* Change the metrics process flow.
before: metrics entrance -> aggregate worker -> remote worker -> trans worker -> minute, hour, day, month persistence worker -> storage
after: metrics entrance -> aggregate worker -> remote worker -> minute persistence worker ->  trans worker -> hour, day, month persistence worker -> storage

* IntKeyLongValueHashMap instead of IntKeyLongValueArray.

* Make the OAP server can't startup.

* Finish

* Rename the method and fixed some test case issues.

* Rename field.

* no message
2019-07-25 19:08:36 +08:00
彭勇升 pengys 8d234a91e5 Feature of database session in OAP server. (#3147)
* Feature of database session

* Make it configurable.

* Make the OAP server can't startup.
2019-07-24 13:43:19 +08:00
彭勇升 pengys 8306ad4531 Reduce the number of threads to improve performance. (#3133)
* Refactor Persistence worker.

* 1. Provide InsertRequest and UpdateRequest interface for prepare persistence.
2. Implement the ids query for H2 metrics DAO.

* Refactor worker framework

* Use queue to receive asynchronous batch request.

* Rename the Datacarrier thread name.

* Fixed some mistake.

* New mistake.
2019-07-21 15:51:24 +08:00
吴晟 Wu Sheng ab9bc9226b Fix H2 metrics read bug, can't read expected existing metrics. (#3132)
Make agent mock works as usual.
2019-07-21 15:07:35 +08:00
彭勇升 pengys 594994ffae 1. Provide InsertRequest and UpdateRequest interface for prepare persistence. (#3131)
2. Implement the ids query for H2 metrics DAO.
2019-07-21 13:56:12 +08:00
彭勇升 pengys 6338de392f Improve OAP server performance. (#3127)
* Improve OAP server performance.
2019-07-20 22:59:52 +08:00
彭勇升 pengys ed507b78ef
Dirty-Read in concurrency (#3117)
* Fixed the problem of dirty reading of the storage implementation of elasticsearch.

* Fixed a mistake.
2019-07-19 14:22:01 +08:00
kezhenxu94 4db1773d3e Fix wrong logger names (#3075)
* Fix wrong logger name

* Fix more wrong logger names in agent side
2019-07-15 00:35:15 +08:00
zhangwei deb2d90029 fix deleteIndex repeat append namespace (#3017)
* fix deleteIndex repeat append namespace

* fix deleteIndex repeat append namespace

* fix checkStyle

* fix

* fix test assert

* update test method name

* fix test

* fix

* del elastic.search.namespace

* fix check style

* update ITElasticSearchNamespaceClient to ITElasticSearchClientOfNamespace

* move undoFormatIndexName to ITElasticSearchClien

* move #getIndex to ITElasticSearchClient

* add deleteByModelName and deleteByIndexName

* add deleteByModelName and deleteByIndexName

* revert ITElasticSearchClient#getIndex

* Use whitebox get from powermock to get this.

* undo namespacePrefix
2019-07-12 07:53:11 +09:00
吴晟 Wu Sheng 4faa950c14
Move master to 6.3.0 (#2968)
* [maven-release-plugin] prepare release v6.2.0

* [maven-release-plugin] prepare for next development iteration
2019-06-29 13:49:36 +08:00
daming 6827839c21 typo (#2960) 2019-06-27 19:34:17 +08:00
daming 1e584ddf50 fix missed condition in H2LogQueryDAO (#2947)
* fix missed condition

* fix missed condition

* fix missed condition
2019-06-26 15:23:02 +08:00
彭勇升 pengys dea6cfb010 Change the statement column type from Keyword to Text. (#2944)
* Use ids query method instead of get method for metrics persistence.

#2933

* Query by alias name.

* #2943

Change the statement column type from Keyword to Text.
2019-06-25 17:55:12 +08:00
彭勇升 pengys 129d2918ac Use ids query method instead of get method for metrics persistence. (#2934)
* Use ids query method instead of get method for metrics persistence.

#2933

* Query by alias name.
2019-06-23 16:53:45 +08:00
彭勇升 pengys 70064f47cb Use the ids query method instead of get method. (#2887)
#2886
2019-06-17 12:05:21 +08:00
彭勇升 pengys 4e63331d35 #2860 (#2861)
Get the document by index name.
2019-06-12 18:56:11 +08:00
吴晟 Wu Sheng dbb8bd8a7f Fix wrong query condition (#2858) 2019-06-12 17:17:15 +08:00