Commit Graph

273 Commits

Author SHA1 Message Date
xiaoyu 164124fc22
provider shardingSphere 4.0.0 4.0.1 plugin. (#4927) 2020-06-15 17:03:17 +08:00
xiaoyu a24efcf5e9
ShardingSphere plugins conflict among themselves. (#4921)
* Fix shardingSphere plugin test fail if have some same ENHANCE_CLASS define.

* support 4.1.0 and 4.1.1

* support 4.1.0 and 4.1.1
2020-06-14 18:48:07 +08:00
songzhendong 86fd1900de
Update backend-cluster.md (#4910) 2020-06-12 07:50:40 +08:00
Xin,Zhang 7b99019c49
Refactory spring gateway plugin (#4878) 2020-06-12 00:25:41 +08:00
Gao Hongtao 5038f07094
Update prometheus fetcher for so11y (#4902) 2020-06-11 16:58:28 +08:00
xiaoyu 9a9401fdb6
provide shardingSphere(4.0.0,4.0.1, 4.1.0 ,4.1.1) plugin. (#4894) 2020-06-11 14:49:04 +08:00
吴晟 Wu Sheng df83490ae6
UI is ready for 8.0.0 release. (#4896) 2020-06-10 20:35:46 +08:00
吴晟 Wu Sheng eb1cc71b6e
Clean up removed configurations. (#4884) 2020-06-09 12:03:59 +08:00
Gao Hongtao 4f0f39ffcc
Ingest prometheus metrics through meter system (#4783) 2020-06-08 09:19:34 +08:00
songzhendong f3d907bf6d
Remove nacos (#4867)
* Add annotation(AlarmCore.start)

* Upgrade nacos version to 1.2.0

* Revert "Upgrade nacos version to 1.2.0"

This reverts commit 4d7b06f6d20ee09ad377ad6ef736aeab429314e0.

* Update nacos version to 1.2.0(Solve the security problem of the old version of fastsjon)

* Recovery annotation

* Recovery annotation

* Update nacos version to 1.2.0

* Update tls_key_generate.sh path

Update tls_key_generate.sh path

* Add annotation(AlarmCore.start)

* Revert "Upgrade nacos version to 1.2.0"

This reverts commit 4d7b06f6d20ee09ad377ad6ef736aeab429314e0.

* Update nacos version to 1.2.0(Solve the security problem of the old version of fastsjon)

* Recovery annotation

* Recovery annotation

* Update tls_key_generate.sh path

* Execution Authority

* Remove nacos

* remove nacos code

* remove nacos

remove nacos

* remove nacos

remove nacos

* remove nacos

* remove nacos

Co-authored-by: songzhendong <289505773@qq.com>
Co-authored-by: songzhendong <songzhendong@xiaomi.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-06-06 14:23:12 +08:00
吴晟 Wu Sheng d094a0f7c0
Support UI template management. (#4843) 2020-06-01 23:06:14 +08:00
Jared Tan 42d79bf08d
Fix too many mang ipv4 records reported by the agent when deployed in container. (#4817) 2020-05-25 16:15:10 +08:00
吴晟 Wu Sheng a10e044932
Group Parameterized Endpoints (#4802)
* Group Parameterized Endpoints. Configure the grouping rules for parameterized endpoints, to improve the meaning of the metrics.
2020-05-20 22:50:22 +08:00
吴晟 Wu Sheng 2dd9a3fd19
Add v8 upgrade document and fix some tiny doc issues. (#4787)
* Add v8 upgrade document and fix some tiny doc issues.

* Add link to STAM paper.
2020-05-14 18:43:32 +08:00
mrproliu 8af10bed11
Support regex to include or exclude the names in alarm module (#4775)
* Support regex to include or exclude the names, requested through mail list, https://lists.apache.org/x/thread.html/rb6d91093776cf61880425449d3d25f9d560cd32363c0075de9a85d13@%3Cdev.skywalking.apache.org%3E

Co-authored-by: Mrproliu <mrproliu@lagou.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-05-13 14:34:16 +08:00
吴晟 Wu Sheng 39adda09f3
Make the default application.yml more friendly in the configuration s… (#4777)
* Make the default application.yml more friendly in the configuration section.

* Fix an issue in python e2e.
2020-05-12 16:55:31 +08:00
mrproliu 749af3e04e
Alarm support multiple scope (#4769)
* Support service relation and database access alarm

* support service instance relation alarm

* support endpoint relation alarm

* document and format the name

* resolve issues

* add database access and endpoint relation rule example

* resolve code format

* remove unnecessary getter and setter package declare

* remove unnecessary fields

Co-authored-by: Mrproliu <mrproliu@lagou.com>
2020-05-11 00:28:10 +08:00
songzhendong d713076f01
Update the path of tls_key_generate.sh script (#4756) 2020-05-08 08:59:19 +08:00
Efe Kaptan d19ba41678
Mariadb JDBC plugin (#4740)
* Add Mariadb JDBC plugin and provide tests

* Refactor component name

* Reformat plugin description and remove healthcheck span from the test

Co-authored-by: Efe Kaptan <Efe.Kaptan@webtrekk.com>
Co-authored-by: zhang-wei <pknfe@outlook.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-05-07 20:54:30 +08:00
kezhenxu94 c9ac104120
Add python project into the project page (#4752)
* Add python project into the project page

* Update README.md

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-05-07 08:25:59 +08:00
Jared Tan d51c788113
support user to customize an endpoint (#4716)
* support user to customize an endpoint with toolkit.

* fix code style.

* add md.

* Update docs/en/setup/service-agent/java-agent/Application-toolkit-trace.md

Co-Authored-By: 吴晟 Wu Sheng <wu.sheng@foxmail.com>

* add test case.

* fix expecteddata.tml.

* fix data.

* fix expecteddata.tml.

* remove operationId in expectedData.yaml.

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-04-29 19:39:52 +08:00
innerpeacez 3bc6cb270c
k8s doc https://github.com/apache/skywalking-kubernetes (#4715)
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-04-26 12:38:20 +08:00
吴晟 Wu Sheng 071708a4a3
Support default value in labeled-value and heatmap query. (#4711)
* Support default value in labeled-value and heatmap query.

* Update a little document.
2020-04-25 21:24:15 +08:00
J-Cod3r 2b41aea639
log4j2 support AsyncAppender print tid (#4670)
* log4j2 support AsyncAppender print tid
2020-04-25 17:35:22 +08:00
吴晟 Wu Sheng b635e254c4
Provide Meter(metrics) system (#4694)
Here are the basic and simple usages of the MeterSystem APIs.

## Meter Creation
New meter could be created based on metrics name, function, scope and data type. 
1. Metrics name is still the storage entity name.
2. Functions are every similar with the OAL function, just accepting different input.
```java
final MeterSystem meterSystem = MeterSystem.meterSystem(getManager());
meterSystem.create("test_long_metrics", "avg", ScopeType.SERVICE, Long.class);
```
NOTICE, this creation should only be called in the `module#prepare` stage, otherwise, `Can't create new metrics anymore` exception will be raised after the **CORE** module `start` stage finished. You may find out, there is a chance `create` could be executed successfully in your own `module#start` stage, but it it just because of the sequence of provider loaded by the class loader, **no guarantee, so please don't do that**.

## Runtime Calculation
`AcceptableValue` is the object created at the runtime to accept new metrics value.
```java
         final MeterSystem service = getManager().find(CoreModule.NAME).provider().getService(MeterSystem.class);
        Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                final AcceptableValue<Long> value = service.buildMetrics("test_long_metrics", Long.class);
                value.accept(MeterEntity.newService("abc"), 5L);
                value.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis()));
                service.doStreamingCalculation(value);
            }
        }, 2, 2, TimeUnit.SECONDS);
```

## Meter Functions
Right now, only `avg` function has been implemented. I submit this PR as soon as possible to get your feedback. I will add more functions.

## Notice
1. Make slow trace query available in the sampled record. Logically, they are the same thing. The UI doesn't need to concern about the trace as a special case.
2. Endpoint dependency will be removed from the dashboard. Because no query available for it. We will provide a new page for that in 8.1
3. Comparison page will be removed due to dashboard is powerful enough to replace it.
2020-04-24 23:33:13 +08:00
songzhendong b3b8aaea6b
Add Tomcat 9 's Java agent deployment instructions(Linux) (#4645) 2020-04-13 18:12:54 +08:00
kezhenxu94 00e8f2d886
Fix bug that the agent heart beats too frequently (#4640)
* Fix bug that the agent heart beats too frequently

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-04-12 20:48:44 +08:00
吴晟 Wu Sheng 44cae97c25
Set up restrict length rules for service, instance and endpoint (#4633)
* Set up restrict length rules for service, instance and endpoint

* Adjust two env names.
2020-04-10 17:59:21 +08:00
吴晟 Wu Sheng 6fe2041b47
Make 8.0.0 Core available. New protocol and register removed. (#4599)
Here are the list of changes we made in this pull request. Top 3 are the principle changes, others are following these principles.
1. New agent and mesh report protocol. 
2. New agent header protocol. 
3. Service register, instance register and network address register have been removed permanently.
4. Service traffic, instance traffic and network alias metrics are added to replace the service, instance and network address inventory.
5. Register process has been removed.
6. Metrics stream process supports insert only mode, especially for traffic entities.
7. Metrics stream process supports no-downsampling mode for traffic entities and network alias.
8. Remove all register mechanism and cache in the java agent.
9. Remove MONTH step in GraphQL query.
10. Update UI to remove MONTH step query, the max query range is 60 days now.
11. Simplify the TTL to metrics and record. And the unit has been formatted in Day unit. No specific TTL for ElasticSearch storage.
12. Buffer mechanism of trace receiver and mesh receiver has been removed due to no register.
13. New service id, instance id and endpoint id rules, including service relation, instance relation and endpoint relation id rules.
14. Java agent support `keep tracing` mode, meaning, agent generating tracing context even the backend is unconnected/unavailable.
15. Plugin test tool up to date, in order to support new protocol.
16. Plugin tests expected data files updated.
17. E2E tests updated.
18. Telemetry of Grafana config has been merged into one.
19. Documentation updates.
20. [TBD] InfluxDB storage implementation is not available, need @dmsolr to fix later, in order to reduce the master change block by this PR.
2020-04-10 11:03:24 +08:00
吴晟 Wu Sheng fa9a44b750
Version number changed including a little documentation. (#4597) 2020-03-31 12:57:20 +08:00
吴晟 Wu Sheng 4474907fef
[Inventory Design Change] Remove endpoint register and endpoint inventory (#4570)
* Finish step one, source and entity changed.

* Step 2, finish the basic and core DAO changes.

* Step 3. Change all source codes of backend, and make project compiling successfully.

* Make startup successful and persistence works.

* Fix dead codes in the MySQLTableInstaller.

* Fix mischange.

* Fix MySQL storage bootstrap error.

* Rollback dependency change.

* Simply enable tailing logs from container no mater it's in CI or not

* Fix segment table issue.

* Fix MySQL Instanller.

* Support Influxdb implementation of new endpoint traffic.

* Set and modify timeout to 90mins

* Remove unnecessary entity in the EndpointTraffic.

* Fix wrong entity builder and make name more clear.

* Fix 2 equal methods bug.

* Fix InfluxDB identifier(name) conflict issue.

* Fix endpoint topology query, endpoint id should be changed as a string.

* Fix a missing change.

* Fix a wrong commit.

* Column name can't be changed, due to it affect the hybrid metadb(h2/mysql) query, change it explicitly with double quotation marks.

* Fix formats.

* Remove endpoint/operation register from the agent. and change the plugin tests(without local tests). operationId relation codes have been removed from agent too.

* Try to fix the hytrix case.

* Fix 3 cases.

* Try to fix cases.

* Fix 2.

* Fix one.

* Fix the serviceId type mis-change.

* Fix int query

* Fix hystrix case.

* Fix a case.

* Fix EndpointCallRelationDispatcher set the source/dest endpoint without id encoding.

* Update documentation and add 2 test cases.

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-03-28 22:51:41 +08:00
yi.liang 1bc4082821
fix elasticsearch-5.x-plugin when use es6.x TransportClient error (#4517)
* up ui

* up ui

* fix elasticsearch-5.x-plugin on es6.x TransportClient error, Found class org.elasticsearch.common.transport.TransportAddress, but interface was expected
up elasticsearch-6.x-plugin support TransportClient

* fix elasticsearch-5.x-plugin on es6.x TransportClient error, Found class org.elasticsearch.common.transport.TransportAddress, but interface was expected
up elasticsearch-6.x-plugin support TransportClient

* fix elasticsearch-5.x-plugin on es6.x TransportClient error, Found class org.elasticsearch.common.transport.TransportAddress, but interface was expected
up elasticsearch-6.x-plugin support TransportClient

* const string witnesses
add some unit test
recommended to change ; to ,

* const string witnesses
add some unit test
recommended to change ; to ,

* up ES6.X test-plugin

* up es6.x test-plugin

* up es6.x test-plugin

* up es6.x test-plugin

* add DeleteIndexRequest

* up es6.x test-plugin

* remove TransportCaseController.java

* fix TransportActionNodeProxyExecuteMethodsInterceptorTest

Co-authored-by: yi.liang <yi.liang@zhangmen.cn>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: 梁懿 <liangyi@xforceplus.com>
Co-authored-by: aderm <394102339@qq.com>
2020-03-27 13:50:27 +08:00
kezhenxu94 509b20e7ce
Add JDK13, JDK14 to build matrix (#4574)
Motivation:

Guarantee that the agent supports JDK13 and JDK14

Modifications:

Add JDK13 and JDK14 to build matrix

Result:

Build with JDK13 and JDK14
2020-03-26 22:13:28 +08:00
mrproliu 569baf3236
Correlation protocol implement (#4555)
Co-authored-by: Mrproliu <mrproliu@lagou.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-03-26 09:02:14 +08:00
吴晟 Wu Sheng 259a147682
Add miss doc about collecting parameters in the profiled traces. (#4552) 2020-03-23 09:41:02 +08:00
cngdkxw 2e9f001633
Add spring cloud feign 2.x plugin (#4526) 2020-03-20 20:28:30 +08:00
吴晟 Wu Sheng 1c7529a559
Add a miss link about apdex documentation. (#4543) 2020-03-20 08:24:35 +08:00
kezhenxu94 69e3d80e24
Add selector property in application.yml (#4514)
### Motivation

 Ease the efforts of modifying configurations in automatic workflow and scripts, like Docker and helm chart.

 ### Modification

 Add a `selector` property into `application.yml` to enable switching configuration simply by environment variables or system properties, and filter the unselected options, leaving only selected one.

 ### Result

 - We can switch a configuration by environment variables or system property

 - Closes #4511
2020-03-15 23:28:14 +08:00
Gao Hongtao 78d14ebbd6
Enable OAP gRPC SSL transportation (#4470)
* Enable OAP gRPC SSL transportation

Porting to OpenSSL to enable SSL transportation. The server private
key is in the format of PCKS#8, certificates is x509 though.

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add netty-tcnative-boringssl-static 2.0.26 licnese

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update docs/en/setup/backend/grpc-ssl.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* Update docs/en/setup/backend/grpc-ssl.md

Co-Authored-By: kezhenxu94 <kezhenxu94@apache.org>

* Add SSL to grpc remote client and sharing server

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update documents

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Fix nits

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Fix test case

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add e2e-ssl and remove other test cases temporarily

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add TLS relevant files

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Decrease timeout

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Test config works

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* triger CI on oap-ssl branch

* Disable agent ca

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Switch to sharing server

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Add agent log to console

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Upload log files

* Change backend

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Revert "Change backend"

This reverts commit 7a085711e3653d3bf642aa84f73650e22595e383.

* Update backend

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Revert e2e

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Ignore TSL relevant files

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Polish codes

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Update certificates expires year to 2120

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-03-15 08:15:28 +08:00
吴晟 Wu Sheng e69391a207
Support Secrets Management File in the ElasticSearch 6/7 storage (#4493)
* Temp commit

* Support secretsManagementFile file.

* Update doc.

* 1. Support JKS/pass runtime change too.
2. Follow review.

* Fix format.

* Fix username/password/trustPass haven't been updated in the es client.

* Fix doc issue.

Co-authored-by: Jared Tan <jian.tan@daocloud.io>
2020-03-13 20:50:14 +08:00
yoje ced5cb2828
Add finagle plugin (#4441)
* Add finagle plugin (#4433)

* fix failed checks for pr(#4441)
* Add licenses for some files
* Update Supported doc
* Add finagle to test component-libraries.yml

* Add finagle automatic tests into CI

* Add comments

* update comments

* Remove CompatibilityChecker

* Move query-protocol submodule to the right version

* Remove `enhancedInstance.setSkyWalkingDynamicField` in `ClientTracingFilterInterceptor`.

* Update expectedData.yaml of finagle tests

* Use lazy ContextCarrier injection.

* Add comments for `ContextCarrierHelper`

* Fix ContextHolderFactory to avoid context pollution

Co-authored-by: huangyongjie <huangyongjie@tigerbrokers>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-03-11 13:04:58 +08:00
kezhenxu94 9c974b80b4
Fix unclear doc and add missing variable (#4484)
* Revert "[BugFix] Docker InfluxDB cannot be enabled (#4483)"

This reverts commit 73d0802cd6.

* Fix unclear doc and add missing variable
2020-03-11 07:46:59 +08:00
吴晟 Wu Sheng d32450aa7d
Support name column(s) in the storage for 3rd party integration, like Kibana. Default OFF (#4439)
* Support dynamic column in the source.

* Support activeExtraModelColumns as a default OFF option.

* Fix words

* Update backend-setup.md

Fix format issue.
2020-03-04 14:44:54 +08:00
aderm 6d530c1d62
update doc. (#4403) 2020-02-24 00:30:13 +08:00
吴晟 Wu Sheng 4245334bac
Officially add LUA agent link in the main repo and document. (#4402) 2020-02-23 22:27:24 +08:00
Daming 29da5738bc
Provide influxdb as a new storage plugin (#4239)
* provide influxdb as a new storage plugin

Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-19 18:00:57 +08:00
吴晟 Wu Sheng 334239388a
Support Daily Index Step feature in the OAP (#4368)
* Support Daily Index Step.

* Fix format.

* Fix mischanged ES.
2020-02-16 20:48:04 +08:00
吴晟 Wu Sheng 329f7e150b
Support Downsampling Data Packing feature in ES storage implementation (#4364)
* Support day/hour/minute metrics merging into one index. Reduce the number of index 50%.
2020-02-16 09:03:30 +08:00
Trustin Lee 5fed153ee2
Add Armeria 0.98.0 to the supported version list (#4328)
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
2020-02-15 12:27:03 +08:00
Jared Tan 7ff9a0cfd9
supprt gRPC alarm hook. (#4344)
* support gRPC hook.

* revert submodule commit.

* revert submodule commitid.

* finish logical.

* remove author anonatation.

* fix ci.

* revert submodule.

* update logical.

* update logical.

* update logical.

* fix

* fix

* fix empty settings.

* fix.

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-02-12 16:20:20 +08:00