Commit Graph

35 Commits

Author SHA1 Message Date
Daming 29da5738bc
Provide influxdb as a new storage plugin (#4239)
* provide influxdb as a new storage plugin

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-19 18:00:57 +08:00
kezhenxu94 5b255ba3e3
Set up more strict code styles and fix existing issues (#4337)
Motivation:

Review code styles with some bots automatically.

Modifications:

Set up ReviewDog in GitHub Action to review code style.

Add more check rules to checkstyle plugin.

Result:

Obvious code styles can be reviewed and commented automatically.
2020-02-11 17:10:49 +08:00
mrproliu fa526e5227
Provide profile analyze query (#4335)
* provide profile analyze query

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-11 10:11:08 +08:00
吴晟 Wu Sheng f29fc4bcb3
Set a shared submodule for e2e test project. (#4329) 2020-02-08 21:56:23 +08:00
mrproliu e44bc36281
provide profiled segment list query (#4322)
* provide profiled segment query

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: AirTrioa <44222967+AirTrioa@users.noreply.github.com>
2020-02-06 21:02:08 +08:00
John Niang 7cf79b1127
Move lombok dependency into root dependency manager for easier version management (#4309)
* Add lombok dependency into root pom.xml

* Update all depending lombok pom.xml(s)

* Change lombok version into latest(1.18.10)

* Revert lombok dependency config of test projects

* Remove redundant lombok dependency of submodule

* Move Lombok into dependencies node for applying globally
2020-02-03 10:44:50 +08:00
kezhenxu94 408a777d9d
Make traces verification loosely (#4306)
### Motivation:

Make E2E tests more stable.

### Modifications:

Verify the traces loosely as other tests, and remove the `verify` method which verifies traces strictly, as we have preparation that also generates traces.

### Result:

E2E tests are more stable.
2020-02-02 14:11:47 +08:00
mrproliu 8e4c1e95d6 Merge e2e profile test project (#4268)
* merge e2e profile test project, Use `env` to distinguish different storage types

* remove maven child module

* change `env` -> `storage`

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-20 19:34:53 +08:00
mrproliu 90f3c4de55 Sniffer processing profile task and report status and snapshot (#4220)
* sniffer processing profile task and report status and snapshot

* resolve testServiceDependencies test case error, use same register with `TraceSegmentServiceClient`

* resolve names

* change profile to single one thread run.

* 1. change to the ArrayList, because known the max size
2. rename issue resolved

* add profiling status enum

* change sniffer use full name issue

* 1. remove `prepareProfiling` method, build profiling status when construct `TracingContext`
2. add `TracingThreadListenerManager`, notify when tracing main thread finish
3. change ProfileThread start when process new profile task

* remove unnecessary getter

* add test assert error message

* adding `AgentServiceRule`

* revert original assert

* remove unnecessary getter

* resolve issues

* reduce findService invoke

* resolve style error

* recheck profiling when change first span operatin name

* resolve issues
1. remove `ContextManager#profilingRecheck`, only check on `TracingContext`
2. rename comments
3. resolve volatile array setting

* remove article link

* add `ProfileTask#maxSamplingCount` check

* resolve conflict (Downstream -> Commands)

* 1. change profilingSegmentSlots init on construct
2. if is profiling, recheck dont need to stop
3. total profiling count increment on first dump

* remove unused return val

* remove some `@param` and `@return`

* add profile task check result data bean

* change profiler slot to `AtomicReferenceArray`

* resolved java doc error

* fix doc error, remove meaningless descriptions

* resolve missed profile receiver on oap starter

* resolve method invoke error

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-18 18:38:03 +08:00
kezhenxu94 8e5e35f096
Fix failed E2E MySQL case (#4246) 2020-01-16 16:34:27 +08:00
Jared Tan 1310719645
add token authentication between agent and oap receiver. (#4197)
* add token authentication between agent and oap receiver.

* revert ui.

* fix code style.

* update docs.

* Update docs/en/setup/backend/backend-token-auth.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* fix hits.

* filter

* revert yml.

* update logical.

* fix codestyle check.

* fix ci and add e2e test case.

* revert mistake commit.

* fix ci

* remove duplicate maven dependency.

* revert pom

* revert unchanged.

* update javadocs.

* update logical.

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-12 23:08:39 +08:00
吴晟 Wu Sheng 9149de2a6e Support multiple linear values and merging p50/75/90/95/99 into percentile (#4214)
* Support new percentile func with new alarm and exporter for this new func.

* Fix e2e and OAL script

* Fix wrong column.

* Fix percentile bug and oal engine bug.

* Update query protocol and add percentile test case

* Support new query

* Adopt GraphQL requirement

* Fix wrong type cast.

* Fix query in H2 and ES.

* Fix docs and comments.

* Fix an e2e compile issue

* Fix javadoc issue and e2e test issue.

* Change CPM to Apdex in TTL test.

* Fix OAL for TTL e2e

* Add metrics query for service percentile.

* Fix OAL engine bug. Method deserialize is not working when more than two field types are IntKeyLongValueHashMap

* Support multiple IntKeyLongValueHashMap fields in remote. About serialize/deserialize methods.

* Fix graphql statement error in e2e.

* Fix serialize not working and add generated serialize/deserialize of percentile into test cases.

* Fix test case format

* Remove generated code test.

* Fix failed e2e test

* Use avg resp time to apdex in the TTL test.

* ADD multiple linear metrics check for endpoint in e2e cluster.

* Support `-` to represent no threshold and doc of alarm about this.

* Move break to right place.

* Fix wrong break(s)

* Fix break and add a test case for multiple values alarm.

* Fix format.

* Add more doc for this new feature and GraphQL query protocol.

Co-authored-by: Jared Tan <jian.tan@daocloud.io>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-12 22:10:50 +08:00
mrproliu 4cf0600f35 Provide profile task downstream to sniffer (#4172)
* Provide profile task downstream to sniffer

* fix agent unit testcase issue

* add profile switch config on sniffer

* fix es error

* 1. add @DefaultImplementor on the sniffer profile task service
2. change ProfileTaskExecutionService#PROFILE_TASK_READY_SCHEDULE to final and remove volatile
2. fix style error

* change timeFromStartMills use `<` to compare

* 1. add `maxSamplingCount` to profile task
2. make profile task limit to the common package

* 1. change `agent.active_profile` to `profile.active` and make true on default
2. add `maxSamplingCount` in profile task
3. use `createTime` to check has new command list
4. add task re-check before process profile task

* 1. add `profile-receiver` document
2. change `ProfileTaskExecutionService` use single schedule thread pool
3. cache dont need fetch data when no data, use auto-fresh mechanism only

* remove navigate time judge, fix comment wrong meaning
2020-01-09 09:35:39 +08:00
mrproliu d38a8eaaa7 Provide thread monitor create task feature (#4145)
* provide thread monitor task service to create

* provide thread monitor task GraphQL implements

* 1. change thread monitor field names
2. fix `getThreadMonitorTaskList` query time bucket error

* provide config stream processor

* update submodule `query-protocol`

* resolve e2e-mysql error

* remove useless storage interface method.

* rename `ThreadMonitorTask` to `ProfileTask`

* change e2e-profile to the top level

* fix profile test analyze error

* remove StringUtil#isBlank.

* 1. remove create profile task duration unit
2. remove GraphQL getTask list duration field
3. add `profileTaskQueryMaxSize` in `storage` -> `elasticsearch(7)` configuration, default get 200 profile task

* provide e2e different storage tests

* 1. fix rat check
2. remove DurationUtils.java#toSecond
3. remove ProfileTaskQueryEs7DAO

* fix e2e code format error

* provide es6 and es7 storage e2e tests

* change e2e profile es module artifactId

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2019-12-31 20:49:45 +08:00
Gao Hongtao 4211eff558 Fix os maven plugin issues for Eclipse m2e (#4144)
* Fix mave-os plugin issues for Eclipse m2e

* Add os-msven-plugin.version property
2019-12-29 13:49:27 +08:00
kezhenxu94 ea9acc5c24
Add E2E test case to verify compatibility between 6.x agents and 7.x … (#4132) 2019-12-26 15:26:08 +08:00
zhang-wei 4c99480f70 Add relation metrics verification in e2e test (#4080) 2019-12-19 13:26:27 +08:00
zhang-wei 3741f4e386 [Feature] Service instance dependency (#3978)
* dispatcher

* getServiceInstanceTopology implementation

* CacheUpdateTimer

* fix compilation failure

* fix compilation failure

* fix esDao

* Update dao list

* dispatcher

* getServiceInstanceTopology implementation

* CacheUpdateTimer

* fix compilation failure

* fix compilation failure

* fix esDao

* Update dao list

* test e2e

* add official_analysis.oal & update getServiceInstanceTopology

* test instance topo for single service

* fix gql

* fix verifyServiceInstances

* verifyServiceInstanceTopo

* fix verifyServiceInstanceTopo

* fix ServiceInstanceInventory#name

* fix ServiceInstanceInventory#name

* e2e test

* fix action

* fix provider port

* fix Instance Node Type

* re run

* fix expected-data

* add e2e mysql

* set @Test timeout

* fix ci

* increase timeout

* re check

* test ttl es7

* test ttl

* fix action

* merge group

* test ttl es7

* replace official_analysis.oal

* replace official_analysis.oal

* comments

* add TODO

* add TODO

* fix filed name
2019-12-09 08:23:21 +08:00
zhang-wei e67d7b85ef [test/e2e] Enhancement TopoMatcher (#3990)
* Enhancement TopoMatcher

* optimize parse

* Update test/e2e/e2e-base/src/main/java/org/apache/skywalking/e2e/assertor/VariableExpressParser.java

Co-Authored-By: kezhenxu94 <kezhenxu94@163.com>
2019-12-02 16:20:00 +08:00
Gao Hongtao 6475ab3d1d Add apdex function to OAL (#3855)
* Add apdex function to OAL

* Add empty line

* Setup config watcher

* Add identifier type to function parameter

* Add config test

* Update score algorithm

* Replace responseCode with status

* Add comments about apdex score algorithm

* Add docs

* Add e2e test case

* Update test case

* Fix disptch class generating error

* Update value name of apdex metric

* Tuning threshold

* Fix single tolerated point bug
2019-11-29 12:54:33 +08:00
kezhenxu94 9b408e82d5 [Feature] Support ElasticSearch 7 as backend storage (#3870)
* [Feature] Officially support ElasticSearch 7 as backend storage

* Extract bootstrap module to be shared by multiple starters

* Add missing configuration file and distinguish different version in test

* Update known-oap-backend-dependencies-es7.txt

Update the dependency lib

* Fix missing adaptation to ElasticSearch 7

* Rename ElasticSearch 7 specific class with Es7 infix

* Fix miss-adapted ElasticSearch DAO API

* Add ES7 specific configuration and polish documentations

* Polish documentations

* Polish documentations
2019-11-21 09:26:23 +08:00
panjuan f44b09483b Modify application.yml to set MySQL as storage provider (#3733)
*  Modify application.yml to set MySQL as storage provider

* use volume
2019-10-29 16:18:36 +08:00
panjuan be3db6794b Add E2E test for MySQL storage. (#3648) 2019-10-27 23:29:06 +08:00
kezhenxu94 50f398a6d2
[CI] Make E2E tests run in multiple JDK versions (#3657)
* [CI] Make E2E tests run in multiple JDK versions

* Polish

* Use new e2e-container version number

* Install socat in reboot case

* Fix reboot case

* Polish

* Reorganize jobs
2019-10-19 23:09:56 +08:00
Kdump c0be757f43 add graphql getServiceInstance instanceUUID field. (#3595)
* add graphql getServiceInstance instanceUUID field.

* add graphql getServiceInstance instanceUUID field.

* add graphql getServiceInstance instanceUUID field.

* sync query-protocol repo

* add instanceUUID field e2e

* add instanceUUID field e2e

* add instanceUUID field e2e

* add instanceUUID field e2e
2019-10-17 06:38:00 +03:00
kezhenxu94 ff0e67e595 [CI] Make e2e more friendly for local test (#3606)
And group the test cases according to their consuming time
2019-10-13 14:11:30 +08:00
吴晟 Wu Sheng 1046944d69 Fix variable name and log in e2e tests. (#3487) 2019-09-19 12:32:45 +08:00
kezhenxu94 721f86485c Add TTL E2E test (#3437)
* Add TTL E2E test

* Add to Jenkins stage and minor bugfix

* Upgrade e2e container version

* Polish and minor fix

* Polish
2019-09-09 22:06:57 -07:00
kezhenxu94 54614a0594 Support gateway without agent (#3308)
* Support Gateway without agent
2019-08-30 16:17:41 +08:00
kezhenxu94 3549d074b0 Increase test service latency to help it to be counted (#3211)
The current latency of the test service is too slow, causing it's not counted and
the query result is 0, this patch increases the latency intentionally by sleeping
1 second
2019-08-03 18:18:32 +08:00
kezhenxu94 49f50a24a4 Support upgrade backend w/o rebooting agents (#3170)
* Support upgrade backend w/o rebooting agents
2019-08-02 09:25:17 +08:00
kezhenxu94 758b085c4f Fix that e2e fails occasionally (#3125)
* Override ES settings in FAQ

* Resend traffic data when invalid

* Remove unsupported ES setting

* Use seperated buffer directory and remove unnecessary web app 2
2019-07-20 13:22:10 +08:00
kezhenxu94 673b0bbcff Fix bug conflicts when containers start simultaneously (#3092)
The e2e containers' names of single mode and cluster mode
would conflict when they start up simultaneously in Jenkins job.
2019-07-16 10:29:19 +08:00
kezhenxu94 37efd1f95a Add e2e cluster tests (#3016)
* Add e2e cluster tests
2019-07-14 23:25:41 +08:00
kezhenxu94 d1a521b705 Add end to end tests (#2935)
e2e test setup. cc @hanahmily @peng-yongsheng @IanCao @zhaoyuguang @JaredTan95 

# e2e test case
One single standalone Spring service, accessing H2 database. This service has been installed the agent. 
OAP runs in single mode with H2 storage too.

The e2e test will verify the results through GraphQL query
1. Traces exist.
2. Service/Service Instance/Endpoint exist and have expected metrics.
3. Topology exists and match expected.
2019-06-30 20:44:24 +08:00