Commit Graph

536 Commits

Author SHA1 Message Date
zhang-wei 7a302d4ff3
Ignore no upstream real address agent (#4454)
* ignore no upstream real address agent

* fix bug

* follow review

* fix checkstyle

* follow review

* follow review

* follow review

* follow review

* follow review

* fix language trans

* lower case
2020-03-07 16:11:16 +08:00
Jared Tan aa21765628
support http api for upstream trace. (#4399)
* support http api.

* e2e

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-03-06 21:07:39 +08:00
mrproliu 0df2d0a5e9
Provide profile exporter tool (#4416) 2020-03-04 20:28:14 +08:00
吴晟 Wu Sheng b6f7dce8c7
forward compatbility for DCS (#4442) 2020-03-04 17:28:53 +08:00
吴晟 Wu Sheng d32450aa7d
Support name column(s) in the storage for 3rd party integration, like Kibana. Default OFF (#4439)
* Support dynamic column in the source.

* Support activeExtraModelColumns as a default OFF option.

* Fix words

* Update backend-setup.md

Fix format issue.
2020-03-04 14:44:54 +08:00
吴晟 Wu Sheng be048d4236
Support change detection mechanism in DCS (#4438)
* Add UUID to DCS
2020-03-04 10:17:04 +08:00
aderm 036aee8d2c
refactor Metrics class. (#4418)
* refactor Metrics class.
2020-02-28 08:49:12 +08:00
吴晟 Wu Sheng 58a787f9e1
Support ref endpoint name register (#4422) 2020-02-27 11:06:13 +08:00
mrproliu b28c7d728e
Add profile multiple time ranges unit test (#4420)
* adding unit for multiple time ranges analyze

* remove un-use import
2020-02-26 21:43:21 +08:00
mrproliu fa0b3df369
provide multiple time ranges to query profile analyze (#4417)
* profide multi time range to query profile analyze

* use method to avoid direct reference field
2020-02-26 10:51:38 +08:00
mrproliu 2ea341b890
Add missed profiled segment query (#4413)
* add profiled segment query

* fix query field error
2020-02-25 10:12:16 +08:00
吴晟 Wu Sheng d63f2c9427
Support topology with multiple selected services (#4408)
* 1. Format codes

2. Add comments for storage interface of topology

3. Narrow the scope of secondary query to get the component name. Only query the nodes as client out of the given service Ids.

* Support topology with multiple selected services

* Sync the typo fix in the protocol
2020-02-24 23:01:13 +08:00
kezhenxu94 8dd0ce18a5
Add missing test for component-libraries.yml (#4398)
### Motivation:

Add test to guarantee that the components defined in class ComponentsDefine are in sync with those in file component-libraries.yml.

### Modifications:

- Add missing component in component-libraries.yml.

- Add test to verify the class and the yml are in sync.

- Fix wrong component id of Undertow.

### Result:

Components definitions in class ComponentsDefine and yaml are in sync, and will be verified in every pull request.
2020-02-23 09:28:47 +08:00
吴晟 Wu Sheng bc5bb84bbc
Add Nginx component ID to main repo (#4397)
* Add Nginx component ID to main repo

* Update LUA as a separated part.

* Update doc.
2020-02-23 00:12:04 +08:00
Daming 29da5738bc
Provide influxdb as a new storage plugin (#4239)
* provide influxdb as a new storage plugin

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-19 18:00:57 +08:00
mrproliu 6185351688
fix profile analyze missed data (#4378) 2020-02-17 22:33:55 +08:00
吴晟 Wu Sheng 334239388a
Support Daily Index Step feature in the OAP (#4368)
* Support Daily Index Step.

* Fix format.

* Fix mischanged ES.
2020-02-16 20:48:04 +08:00
吴晟 Wu Sheng 329f7e150b
Support Downsampling Data Packing feature in ES storage implementation (#4364)
* Support day/hour/minute metrics merging into one index. Reduce the number of index 50%.
2020-02-16 09:03:30 +08:00
吴晟 Wu Sheng ae442e364f
More comments for important classes (#4361) 2020-02-14 17:03:50 +08:00
mrproliu 5f4b566c48
Fix missing mysql index for profile entities (#4360)
* fix profile entities missed mysql index

* add data length config
2020-02-14 16:03:17 +08:00
吴晟 Wu Sheng 6be3e9909a
Help people to read source codes of core module. (#4357)
* Add more comments in the core module, should help the source code readers a lot.
2020-02-13 21:11:05 +08:00
吴晟 Wu Sheng 0300048d92
Fill some comments for core. (#4356)
* Fill some comments for core.

* Fix a comment issue.

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-02-13 11:27:17 +08:00
cui-liqiang c761f30e3b
Some code style and refactor (#4355)
* extract duplicated method

* change while loop to if since once rs.next() return false, it will never return true for the following calls

* simplify if() return true; return false pattern

* fix compilation

Signed-off-by: liqiang.clq <liqiang.clq@alibaba-inc.com>
2020-02-13 10:34:32 +08:00
吴晟 Wu Sheng 7c6016b18f
Make Profile entities could be disable by OAL (#4354)
* Make Profile entities could be disable by OAL
2020-02-12 23:28:27 +08:00
Jared Tan 7ff9a0cfd9
supprt gRPC alarm hook. (#4344)
* support gRPC hook.

* revert submodule commit.

* revert submodule commitid.

* finish logical.

* remove author anonatation.

* fix ci.

* revert submodule.

* update logical.

* update logical.

* update logical.

* fix

* fix

* fix empty settings.

* fix.

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-12 16:20:20 +08:00
kezhenxu94 5b255ba3e3
Set up more strict code styles and fix existing issues (#4337)
Motivation:

Review code styles with some bots automatically.

Modifications:

Set up ReviewDog in GitHub Action to review code style.

Add more check rules to checkstyle plugin.

Result:

Obvious code styles can be reviewed and commented automatically.
2020-02-11 17:10:49 +08:00
mrproliu fa526e5227
Provide profile analyze query (#4335)
* provide profile analyze query

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-11 10:11:08 +08:00
Gao Hongtao fc4402f5a4 Add trace log (#4339)
* Add trace log

 * Trace alarm rule reader
 * Trace the process of running alarm rule, especailly the moving of window
 * Add toString method to relevant class for tracing

* Remove unused importsC

* Revert "Remove unused importsC"

This reverts commit 395e1678

* Update query

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-10 15:53:43 +08:00
mrproliu e01829aa57
Change profile stack element data structure (#4332)
* Change profile stack element to single level, not using tree
2020-02-09 16:37:45 +08:00
mrproliu e44bc36281
provide profiled segment list query (#4322)
* provide profiled segment query

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: AirTrioa <44222967+AirTrioa@users.noreply.github.com>
2020-02-06 21:02:08 +08:00
mrproliu 30ebe1caf9
provide thread stack analyzes (#4308)
* provide thread analyzes

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-04 16:24:05 +08:00
John Niang 7cf79b1127
Move lombok dependency into root dependency manager for easier version management (#4309)
* Add lombok dependency into root pom.xml

* Update all depending lombok pom.xml(s)

* Change lombok version into latest(1.18.10)

* Revert lombok dependency config of test projects

* Remove redundant lombok dependency of submodule

* Move Lombok into dependencies node for applying globally
2020-02-03 10:44:50 +08:00
mrproliu 3252b6ed08
provide profile analyze empty interfaces (#4303)
* provide profile analyze empty interfaces

* fix comment error

* change comments

* update submodule

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-01-31 21:11:43 +08:00
Daming 9005d1e82c
Provide AVRO plugin (#4276)
* Provide Avro plugin
* provide testcase for avro-plugin

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-30 17:49:43 +08:00
吴晟 Wu Sheng 17e3fff83a
Refactor the storage to support `getSubsetOfMultipleLinearIntValues` (#4289) 2020-01-29 17:03:39 +08:00
kezhenxu94 75aefd34de Fix potential concurrent problems and tune performance (#4274) 2020-01-20 14:59:41 +08:00
Liu Hao 72e6888ae3 Fix thread unsafe problem in server-alarm-plugin (#4230) (#4247)
* fix thread unsafe problem in server-alarm-plugin (#4230)

* remove redundant #moveTo

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-01-20 09:57:35 +08:00
mrproliu 90f3c4de55 Sniffer processing profile task and report status and snapshot (#4220)
* sniffer processing profile task and report status and snapshot

* resolve testServiceDependencies test case error, use same register with `TraceSegmentServiceClient`

* resolve names

* change profile to single one thread run.

* 1. change to the ArrayList, because known the max size
2. rename issue resolved

* add profiling status enum

* change sniffer use full name issue

* 1. remove `prepareProfiling` method, build profiling status when construct `TracingContext`
2. add `TracingThreadListenerManager`, notify when tracing main thread finish
3. change ProfileThread start when process new profile task

* remove unnecessary getter

* add test assert error message

* adding `AgentServiceRule`

* revert original assert

* remove unnecessary getter

* resolve issues

* reduce findService invoke

* resolve style error

* recheck profiling when change first span operatin name

* resolve issues
1. remove `ContextManager#profilingRecheck`, only check on `TracingContext`
2. rename comments
3. resolve volatile array setting

* remove article link

* add `ProfileTask#maxSamplingCount` check

* resolve conflict (Downstream -> Commands)

* 1. change profilingSegmentSlots init on construct
2. if is profiling, recheck dont need to stop
3. total profiling count increment on first dump

* remove unused return val

* remove some `@param` and `@return`

* add profile task check result data bean

* change profiler slot to `AtomicReferenceArray`

* resolved java doc error

* fix doc error, remove meaningless descriptions

* resolve missed profile receiver on oap starter

* resolve method invoke error

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-18 18:38:03 +08:00
kezhenxu94 019af3e345
Drop support of protocol v1 (sw3) (#4244)
* Drop support of protocol v1 (sw3)

* Remove versioning logic further

* Remove more version5-only classes

* Commit missing files

* Update submodule and address review comment

* Fix failed tests
2020-01-17 20:33:52 +08:00
John Niang 738e4975cb Fix typo: guideline and code document (#4248) 2020-01-16 19:38:03 +08:00
kezhenxu94 094eeb870f
Add alarm metrics OP >= and <=, and refactor the operator logics (#4221)
* Add alarm metrics OP >= and <=, and refactor the operator logics

* Add unit test for OP

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-01-13 00:50:37 +08:00
Jared Tan 1310719645
add token authentication between agent and oap receiver. (#4197)
* add token authentication between agent and oap receiver.

* revert ui.

* fix code style.

* update docs.

* Update docs/en/setup/backend/backend-token-auth.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* fix hits.

* filter

* revert yml.

* update logical.

* fix codestyle check.

* fix ci and add e2e test case.

* revert mistake commit.

* fix ci

* remove duplicate maven dependency.

* revert pom

* revert unchanged.

* update javadocs.

* update logical.

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-12 23:08:39 +08:00
吴晟 Wu Sheng 9149de2a6e Support multiple linear values and merging p50/75/90/95/99 into percentile (#4214)
* Support new percentile func with new alarm and exporter for this new func.

* Fix e2e and OAL script

* Fix wrong column.

* Fix percentile bug and oal engine bug.

* Update query protocol and add percentile test case

* Support new query

* Adopt GraphQL requirement

* Fix wrong type cast.

* Fix query in H2 and ES.

* Fix docs and comments.

* Fix an e2e compile issue

* Fix javadoc issue and e2e test issue.

* Change CPM to Apdex in TTL test.

* Fix OAL for TTL e2e

* Add metrics query for service percentile.

* Fix OAL engine bug. Method deserialize is not working when more than two field types are IntKeyLongValueHashMap

* Support multiple IntKeyLongValueHashMap fields in remote. About serialize/deserialize methods.

* Fix graphql statement error in e2e.

* Fix serialize not working and add generated serialize/deserialize of percentile into test cases.

* Fix test case format

* Remove generated code test.

* Fix failed e2e test

* Use avg resp time to apdex in the TTL test.

* ADD multiple linear metrics check for endpoint in e2e cluster.

* Support `-` to represent no threshold and doc of alarm about this.

* Move break to right place.

* Fix wrong break(s)

* Fix break and add a test case for multiple values alarm.

* Fix format.

* Add more doc for this new feature and GraphQL query protocol.

Co-authored-by: Jared Tan <jian.tan@daocloud.io>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-12 22:10:50 +08:00
Jared Tan 7e9812e857
remove duplicate maven dependency. (#4215) 2020-01-11 11:01:18 +08:00
mrproliu 4cf0600f35 Provide profile task downstream to sniffer (#4172)
* Provide profile task downstream to sniffer

* fix agent unit testcase issue

* add profile switch config on sniffer

* fix es error

* 1. add @DefaultImplementor on the sniffer profile task service
2. change ProfileTaskExecutionService#PROFILE_TASK_READY_SCHEDULE to final and remove volatile
2. fix style error

* change timeFromStartMills use `<` to compare

* 1. add `maxSamplingCount` to profile task
2. make profile task limit to the common package

* 1. change `agent.active_profile` to `profile.active` and make true on default
2. add `maxSamplingCount` in profile task
3. use `createTime` to check has new command list
4. add task re-check before process profile task

* 1. add `profile-receiver` document
2. change `ProfileTaskExecutionService` use single schedule thread pool
3. cache dont need fetch data when no data, use auto-fresh mechanism only

* remove navigate time judge, fix comment wrong meaning
2020-01-09 09:35:39 +08:00
aderm f181f7b483 delete unused code. (#4178) 2020-01-06 13:15:48 +08:00
web-xiaxia 5fba91901f add Dynamic Configuration Consul AclToken (#4161)
* Update ConsulConfigurationCenterSettings.java

add aclTokne

* Update ConsulConfigurationWatcherRegister.java

add aclToken

* Update application.yml

add aclToken

* Update ConsulConfigurationCenterSettings.java

add aclToken

* Update application.yml

add aclToken

* add aclToken

* Replace with util

* Replace with util

* add consul aclToken
2020-01-02 20:05:16 +08:00
吴晟 Wu Sheng 6c20f4a41b
Add service type from register and support browser as service type (#4157)
* Add service type from register and support browser as service type

* Add service type at agent side.

* Fix format.

* Fix a test.

* Fix formats.

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-01 18:01:41 +08:00
mrproliu d38a8eaaa7 Provide thread monitor create task feature (#4145)
* provide thread monitor task service to create

* provide thread monitor task GraphQL implements

* 1. change thread monitor field names
2. fix `getThreadMonitorTaskList` query time bucket error

* provide config stream processor

* update submodule `query-protocol`

* resolve e2e-mysql error

* remove useless storage interface method.

* rename `ThreadMonitorTask` to `ProfileTask`

* change e2e-profile to the top level

* fix profile test analyze error

* remove StringUtil#isBlank.

* 1. remove create profile task duration unit
2. remove GraphQL getTask list duration field
3. add `profileTaskQueryMaxSize` in `storage` -> `elasticsearch(7)` configuration, default get 200 profile task

* provide e2e different storage tests

* 1. fix rat check
2. remove DurationUtils.java#toSecond
3. remove ProfileTaskQueryEs7DAO

* fix e2e code format error

* provide es6 and es7 storage e2e tests

* change e2e profile es module artifactId

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2019-12-31 20:49:45 +08:00
吴晟 Wu Sheng 1ee14fbbc8
Support customized instance name by a new UUID naming policy. (#4148) 2019-12-30 14:59:29 +08:00