Commit Graph

33 Commits

Author SHA1 Message Date
mrproliu 925c0d74c7
Provide server-side meter (#4972) 2020-07-25 21:17:32 +08:00
Jared Tan db9f8e6e00
Support Jetty server advanced configuration. (#5147) 2020-07-25 17:41:49 +08:00
吴晟 Wu Sheng 2c676ead24
Add OAP configuration vocabulary documentation. (#5149) 2020-07-23 08:24:40 +08:00
songzhendong b363b0ea4f
Add Nacos back into SkyWalking as an option(nacos-1.3.1) (#5137) 2020-07-21 23:30:06 +08:00
Evan e0325a4407
Configmap configuration (#4959) 2020-07-20 12:35:35 +08:00
xin a6c3aad8fa
Fix config file and document typo for ttl (#5087) (#5088) 2020-07-12 17:36:35 +08:00
Gao Hongtao b6661b5164
Add health checker module (#5046) 2020-07-10 00:30:43 +08:00
Gao Hongtao 1654a0fbf0
Add opencensus reciever (#5029)
Signed-off-by: Gao Hongtao <hanahmily@gmail.com>
2020-07-05 20:53:16 +08:00
Evan ad47c9045b
update the kubernetes client verion to 8.0.0 (#4844) 2020-06-22 07:56:39 +08:00
Gao Hongtao 4f0f39ffcc
Ingest prometheus metrics through meter system (#4783) 2020-06-08 09:19:34 +08:00
songzhendong f3d907bf6d
Remove nacos (#4867)
* Add annotation(AlarmCore.start)

* Upgrade nacos version to 1.2.0

* Revert "Upgrade nacos version to 1.2.0"

This reverts commit 4d7b06f6d20ee09ad377ad6ef736aeab429314e0.

* Update nacos version to 1.2.0(Solve the security problem of the old version of fastsjon)

* Recovery annotation

* Recovery annotation

* Update nacos version to 1.2.0

* Update tls_key_generate.sh path

Update tls_key_generate.sh path

* Add annotation(AlarmCore.start)

* Revert "Upgrade nacos version to 1.2.0"

This reverts commit 4d7b06f6d20ee09ad377ad6ef736aeab429314e0.

* Update nacos version to 1.2.0(Solve the security problem of the old version of fastsjon)

* Recovery annotation

* Recovery annotation

* Update tls_key_generate.sh path

* Execution Authority

* Remove nacos

* remove nacos code

* remove nacos

remove nacos

* remove nacos

remove nacos

* remove nacos

* remove nacos

Co-authored-by: songzhendong <289505773@qq.com>
Co-authored-by: songzhendong <songzhendong@xiaomi.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-06-06 14:23:12 +08:00
Evan 1652e0c057
Add a SuperDataset tag for custom super size dataset shards config in es storage (#4793)
* add SuperDataset tag for add super size dataset shards config in es
2020-05-17 17:48:15 +08:00
吴晟 Wu Sheng 39adda09f3
Make the default application.yml more friendly in the configuration s… (#4777)
* Make the default application.yml more friendly in the configuration section.

* Fix an issue in python e2e.
2020-05-12 16:55:31 +08:00
吴晟 Wu Sheng 2880699860
Make OAL controlled by the receivers. (#4767)
* Split the official_analysis.oal into different parts

* Add Envoy OAL define.

* Fix license.
2020-05-09 21:39:19 +08:00
吴晟 Wu Sheng b635e254c4
Provide Meter(metrics) system (#4694)
Here are the basic and simple usages of the MeterSystem APIs.

## Meter Creation
New meter could be created based on metrics name, function, scope and data type. 
1. Metrics name is still the storage entity name.
2. Functions are every similar with the OAL function, just accepting different input.
```java
final MeterSystem meterSystem = MeterSystem.meterSystem(getManager());
meterSystem.create("test_long_metrics", "avg", ScopeType.SERVICE, Long.class);
```
NOTICE, this creation should only be called in the `module#prepare` stage, otherwise, `Can't create new metrics anymore` exception will be raised after the **CORE** module `start` stage finished. You may find out, there is a chance `create` could be executed successfully in your own `module#start` stage, but it it just because of the sequence of provider loaded by the class loader, **no guarantee, so please don't do that**.

## Runtime Calculation
`AcceptableValue` is the object created at the runtime to accept new metrics value.
```java
         final MeterSystem service = getManager().find(CoreModule.NAME).provider().getService(MeterSystem.class);
        Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                final AcceptableValue<Long> value = service.buildMetrics("test_long_metrics", Long.class);
                value.accept(MeterEntity.newService("abc"), 5L);
                value.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis()));
                service.doStreamingCalculation(value);
            }
        }, 2, 2, TimeUnit.SECONDS);
```

## Meter Functions
Right now, only `avg` function has been implemented. I submit this PR as soon as possible to get your feedback. I will add more functions.

## Notice
1. Make slow trace query available in the sampled record. Logically, they are the same thing. The UI doesn't need to concern about the trace as a special case.
2. Endpoint dependency will be removed from the dashboard. Because no query available for it. We will provide a new page for that in 8.1
3. Comparison page will be removed due to dashboard is powerful enough to replace it.
2020-04-24 23:33:13 +08:00
吴晟 Wu Sheng 31385159c2
Support to use empty string to override existing config. (#4635) 2020-04-11 13:05:57 +08:00
吴晟 Wu Sheng 44cae97c25
Set up restrict length rules for service, instance and endpoint (#4633)
* Set up restrict length rules for service, instance and endpoint

* Adjust two env names.
2020-04-10 17:59:21 +08:00
吴晟 Wu Sheng 6fe2041b47
Make 8.0.0 Core available. New protocol and register removed. (#4599)
Here are the list of changes we made in this pull request. Top 3 are the principle changes, others are following these principles.
1. New agent and mesh report protocol. 
2. New agent header protocol. 
3. Service register, instance register and network address register have been removed permanently.
4. Service traffic, instance traffic and network alias metrics are added to replace the service, instance and network address inventory.
5. Register process has been removed.
6. Metrics stream process supports insert only mode, especially for traffic entities.
7. Metrics stream process supports no-downsampling mode for traffic entities and network alias.
8. Remove all register mechanism and cache in the java agent.
9. Remove MONTH step in GraphQL query.
10. Update UI to remove MONTH step query, the max query range is 60 days now.
11. Simplify the TTL to metrics and record. And the unit has been formatted in Day unit. No specific TTL for ElasticSearch storage.
12. Buffer mechanism of trace receiver and mesh receiver has been removed due to no register.
13. New service id, instance id and endpoint id rules, including service relation, instance relation and endpoint relation id rules.
14. Java agent support `keep tracing` mode, meaning, agent generating tracing context even the backend is unconnected/unavailable.
15. Plugin test tool up to date, in order to support new protocol.
16. Plugin tests expected data files updated.
17. E2E tests updated.
18. Telemetry of Grafana config has been merged into one.
19. Documentation updates.
20. [TBD] InfluxDB storage implementation is not available, need @dmsolr to fix later, in order to reduce the master change block by this PR.
2020-04-10 11:03:24 +08:00
吴晟 Wu Sheng 54fb6e7285
Set the endpoint name length to 150, to keep safe and avoid over-length. (#4595)
* Set the endpoint name length to 150, to keep safe and avoid over-length.

* Set endpointNameMaxLength in the application.yml

* Fix a mischange.

* Correct the endpoint name length control logic

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-03-31 09:04:10 +08:00
kezhenxu94 69e3d80e24
Add selector property in application.yml (#4514)
### Motivation

 Ease the efforts of modifying configurations in automatic workflow and scripts, like Docker and helm chart.

 ### Modification

 Add a `selector` property into `application.yml` to enable switching configuration simply by environment variables or system properties, and filter the unselected options, leaving only selected one.

 ### Result

 - We can switch a configuration by environment variables or system property

 - Closes #4511
2020-03-15 23:28:14 +08:00
Gao Hongtao 78d14ebbd6
Enable OAP gRPC SSL transportation (#4470)
* Enable OAP gRPC SSL transportation

Porting to OpenSSL to enable SSL transportation. The server private
key is in the format of PCKS#8, certificates is x509 though.

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add netty-tcnative-boringssl-static 2.0.26 licnese

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update docs/en/setup/backend/grpc-ssl.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* Update docs/en/setup/backend/grpc-ssl.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* Add SSL to grpc remote client and sharing server

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update documents

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Fix nits

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Fix test case

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add e2e-ssl and remove other test cases temporarily

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add TLS relevant files

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Decrease timeout

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Test config works

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* triger CI on oap-ssl branch

* Disable agent ca

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Switch to sharing server

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add agent log to console

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Upload log files

* Change backend

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Revert "Change backend"

This reverts commit 7a085711e3653d3bf642aa84f73650e22595e383.

* Update backend

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Revert e2e

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Ignore TSL relevant files

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Polish codes

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update certificates expires year to 2120

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-03-15 08:15:28 +08:00
吴晟 Wu Sheng e69391a207
Support Secrets Management File in the ElasticSearch 6/7 storage (#4493)
* Temp commit

* Support secretsManagementFile file.

* Update doc.

* 1. Support JKS/pass runtime change too.
2. Follow review.

* Fix format.

* Fix username/password/trustPass haven't been updated in the es client.

* Fix doc issue.

Co-authored-by: Jared Tan <jian.tan@daocloud.io>
2020-03-13 20:50:14 +08:00
吴晟 Wu Sheng d32450aa7d
Support name column(s) in the storage for 3rd party integration, like Kibana. Default OFF (#4439)
* Support dynamic column in the source.

* Support activeExtraModelColumns as a default OFF option.

* Fix words

* Update backend-setup.md

Fix format issue.
2020-03-04 14:44:54 +08:00
Daming 29da5738bc
Provide influxdb as a new storage plugin (#4239)
* provide influxdb as a new storage plugin

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-19 18:00:57 +08:00
吴晟 Wu Sheng 334239388a
Support Daily Index Step feature in the OAP (#4368)
* Support Daily Index Step.

* Fix format.

* Fix mischanged ES.
2020-02-16 20:48:04 +08:00
吴晟 Wu Sheng 329f7e150b
Support Downsampling Data Packing feature in ES storage implementation (#4364)
* Support day/hour/minute metrics merging into one index. Reduce the number of index 50%.
2020-02-16 09:03:30 +08:00
mrproliu 90f3c4de55 Sniffer processing profile task and report status and snapshot (#4220)
* sniffer processing profile task and report status and snapshot

* resolve testServiceDependencies test case error, use same register with `TraceSegmentServiceClient`

* resolve names

* change profile to single one thread run.

* 1. change to the ArrayList, because known the max size
2. rename issue resolved

* add profiling status enum

* change sniffer use full name issue

* 1. remove `prepareProfiling` method, build profiling status when construct `TracingContext`
2. add `TracingThreadListenerManager`, notify when tracing main thread finish
3. change ProfileThread start when process new profile task

* remove unnecessary getter

* add test assert error message

* adding `AgentServiceRule`

* revert original assert

* remove unnecessary getter

* resolve issues

* reduce findService invoke

* resolve style error

* recheck profiling when change first span operatin name

* resolve issues
1. remove `ContextManager#profilingRecheck`, only check on `TracingContext`
2. rename comments
3. resolve volatile array setting

* remove article link

* add `ProfileTask#maxSamplingCount` check

* resolve conflict (Downstream -> Commands)

* 1. change profilingSegmentSlots init on construct
2. if is profiling, recheck dont need to stop
3. total profiling count increment on first dump

* remove unused return val

* remove some `@param` and `@return`

* add profile task check result data bean

* change profiler slot to `AtomicReferenceArray`

* resolved java doc error

* fix doc error, remove meaningless descriptions

* resolve missed profile receiver on oap starter

* resolve method invoke error

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-18 18:38:03 +08:00
Jared Tan 1310719645
add token authentication between agent and oap receiver. (#4197)
* add token authentication between agent and oap receiver.

* revert ui.

* fix code style.

* update docs.

* Update docs/en/setup/backend/backend-token-auth.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* fix hits.

* filter

* revert yml.

* update logical.

* fix codestyle check.

* fix ci and add e2e test case.

* revert mistake commit.

* fix ci

* remove duplicate maven dependency.

* revert pom

* revert unchanged.

* update javadocs.

* update logical.

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-01-12 23:08:39 +08:00
web-xiaxia 5fba91901f add Dynamic Configuration Consul AclToken (#4161)
* Update ConsulConfigurationCenterSettings.java

add aclTokne

* Update ConsulConfigurationWatcherRegister.java

add aclToken

* Update application.yml

add aclToken

* Update ConsulConfigurationCenterSettings.java

add aclToken

* Update application.yml

add aclToken

* add aclToken

* Replace with util

* Replace with util

* add consul aclToken
2020-01-02 20:05:16 +08:00
mrproliu d38a8eaaa7 Provide thread monitor create task feature (#4145)
* provide thread monitor task service to create

* provide thread monitor task GraphQL implements

* 1. change thread monitor field names
2. fix `getThreadMonitorTaskList` query time bucket error

* provide config stream processor

* update submodule `query-protocol`

* resolve e2e-mysql error

* remove useless storage interface method.

* rename `ThreadMonitorTask` to `ProfileTask`

* change e2e-profile to the top level

* fix profile test analyze error

* remove StringUtil#isBlank.

* 1. remove create profile task duration unit
2. remove GraphQL getTask list duration field
3. add `profileTaskQueryMaxSize` in `storage` -> `elasticsearch(7)` configuration, default get 200 profile task

* provide e2e different storage tests

* 1. fix rat check
2. remove DurationUtils.java#toSecond
3. remove ProfileTaskQueryEs7DAO

* fix e2e code format error

* provide es6 and es7 storage e2e tests

* change e2e profile es module artifactId

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2019-12-31 20:49:45 +08:00
kezhenxu94 aa022ffa00 Remove unused ES7 config (#3998) 2019-12-04 00:20:10 +08:00
Jared Tan d581f2294f Support custom elasticsearch settings in index level. (#3913) 2019-11-21 22:30:59 +08:00
kezhenxu94 9b408e82d5 [Feature] Support ElasticSearch 7 as backend storage (#3870)
* [Feature] Officially support ElasticSearch 7 as backend storage

* Extract bootstrap module to be shared by multiple starters

* Add missing configuration file and distinguish different version in test

* Update known-oap-backend-dependencies-es7.txt

Update the dependency lib

* Fix missing adaptation to ElasticSearch 7

* Rename ElasticSearch 7 specific class with Es7 infix

* Fix miss-adapted ElasticSearch DAO API

* Add ES7 specific configuration and polish documentations

* Polish documentations

* Polish documentations
2019-11-21 09:26:23 +08:00