Commit Graph

785 Commits

Author SHA1 Message Date
mrproliu 306000761c
Add description and document link into the UI menu query protocol (#11070) 2023-07-11 10:18:15 +08:00
Wan Kai 1630f1afa7
[Breaking change] Change the default label name in MQE from `labe`l to `_`. (#11051) 2023-07-06 13:03:04 +08:00
weixiang1862 a8cf4fb4b6
Support reduce aggregate function in MQE.(#11031) (#11036) 2023-07-05 14:24:02 +02:00
Jiajing LU 8aab7d7ac7
Bump up BanyanDB and Java Client to 0.4.0 (#11039) 2023-07-04 18:41:23 +02:00
mrproliu 6131e4c6a3
Support dynamic UI menu query (#11005) 2023-06-29 14:09:32 +08:00
kezhenxu94 e2f0bee06b
Clean up tests for components that are too old (#10981) 2023-06-23 22:42:23 +08:00
吴晟 Wu Sheng 56cfa52d87
Remove `matchedCounter` from `HttpUriRecognitionService#feedRawData` (#10966)
* [Breaking change] Remove `matchedCounter` from `HttpUriRecognitionService#feedRawData`.

* * Remove patterns from `HttpUriRecognitionService#feedRawData` and add max 10 candidates of raw URIs for each pattern.

---------

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2023-06-20 13:17:05 -04:00
Wan Kai feb3ca2b6d
Add UT and E2E for MQE. (#10920) 2023-06-13 17:58:15 +08:00
mrproliu 43668b7655
Add cause message field when query continuous profiling task (#10919) 2023-06-13 11:51:56 +08:00
mrproliu de3e3d27ec
Update continuous profiling E2E for adapt new query protocol (#10901) 2023-06-09 09:44:07 +08:00
kezhenxu94 6587dd413a
Pin Selenium version to fix E2E (#10904) 2023-06-08 22:49:55 +08:00
sheiy 081a55f276
Fix a typo in e2e otel-collector-config about log level(#10863) 2023-05-27 03:14:31 -04:00
yswdqz 81ec3af923
[Feature] Redis slow log collection (#10825) 2023-05-22 20:31:00 +04:00
kezhenxu94 fc2eced32c
Add OpenTelemetry log protocol support (#10808) 2023-05-18 11:46:42 +08:00
innerpeacez 1230bf4d4b
Support RabbitMQ monitoring (#10814) 2023-05-17 11:56:44 +08:00
kezhenxu94 d874c211f6
Clean up OpenCensus related codes and docs (#10755) 2023-05-04 11:13:17 +08:00
Jiajing LU 06558fcf5a
Integrate BanyanDB server-side TopN (#10448) 2023-04-25 10:37:55 +08:00
Wan Kai 1ca9e0acfe
Fix PromQL HTTP API `/api/v1/series` response missing `service` label when matching metric (#10715) 2023-04-23 10:31:52 +08:00
吴晟 Wu Sheng af2cf9958f
Polish log of alert GRPCCallback (#10712) 2023-04-22 21:24:20 +08:00
innerpeacez cce87a9917
Support Elasticsearch monitoring (#10690) 2023-04-20 17:53:01 +08:00
yswdqz 2cbeb55359
Support Redis Mornitoring (#10655) 2023-04-08 19:03:31 +08:00
pg.yang b7f456d29f
Support Amazon API Gateway monitoring (#10645) 2023-04-05 12:19:46 +08:00
Wan Kai 6d432c31f2
PromQL: Remove empty values from the query result, fix `/api/v1/metadata` param `limit` could cause out of bound. (#10628) 2023-03-31 16:35:06 +08:00
Wan Kai ec2ca354ea
Support isEmptyValue flag for metrics query. (#10616) 2023-03-30 14:46:07 +08:00
mrproliu 0de087a100
[Breaking Change] Support cross-thread trace profiling. (#10575)
The data structure and query APIs are changed.
2023-03-22 23:50:26 +08:00
mrproliu c217ead555
Adapt the service level component in E2E (#10530) 2023-03-21 14:45:14 +08:00
mrproliu 7dcdb279ef
Support continuous profiling feature (#10473) 2023-03-20 20:26:42 +08:00
kezhenxu94 74a8589307
[Breaking Change] Enhance JDBC storage through merging table and managing day-based table rolling (#10544)
* [Breaking Change] Enhance JDBC storage through merging tables and managing day-based table rolling.

* [Breaking Change] Sharding-MySQL implementations and tests get removed due to we have the day-based rolling mechanism by default
2023-03-17 21:01:48 +08:00
yswdqz 7447999b4c
Fix wrong layer of metric `userError` in DynamoDB monitoring. (#10518) 2023-03-09 15:09:55 +08:00
yswdqz 6549dc0c0e
Support for double quotes in SlowSQL statements (#10489) 2023-03-06 20:27:07 +08:00
pg.yang cc7a2c9e97
Fix wrong time for OTEL mock service, invalid condition(ge 0) for e2e cases (#10437) 2023-02-23 23:50:25 +08:00
Wan Kai fb9117b8e7
Support prometheus HTTP APIs and promQL. (#10415)
Support HTTP APIs list:

- /api/v1/metadata
- /api/v1/labels
- /api/v1/label/{label_name}/values
- /api/v1/series
- /api/v1/query
-/api/v1/query_range

Support basic promQL for query traffic/metrics/labeled_metrics/topN/sampled_records from OAP
2023-02-22 12:59:45 +08:00
Gao Hongtao 0a35bdd1e2
Update BanyanDB to 0.3.1 release (#10426) 2023-02-22 08:09:27 +08:00
yswdqz f6ef55b5af
Support Monitoring DynamoDB. (#10418) 2023-02-21 09:24:17 +08:00
吴晟 Wu Sheng af3889b47e
Bump up BanyanDB 0.3.0 client. (#10406) 2023-02-20 09:18:41 +08:00
pg.yang cabe40eb47
Support Amazon Simple Storage Service (AWS S3) metrics monitoring (#10407) 2023-02-19 17:23:10 +08:00
Mike 574b83f095
feature: windows monitoring (#10298) 2023-02-15 23:14:04 +08:00
Raigor a87794fb6b
Update ShardingSphere version to 5.3.1 release. (#10259) 2023-01-11 00:24:43 +08:00
Gao Hongtao 6a78773743
Improve BanyanDB measure schema (#10233) 2023-01-08 18:50:00 +08:00
mrproliu 2b40270f71
Add the eBPF network profiling E2E Test in per storage (#10245) 2023-01-06 21:53:39 +08:00
Raigor 2756e0cfab
Update ShardingSphere version for compatibility verification (#10244)
* Integrate with ShardingSphere 5.3.1-SNAPSHOT.

* Update the version of ShardingSphere in e2e

* Remove providerType when the transaction type is LOCAL.

* Update backend-storage.md & changes.md
2023-01-06 17:46:23 +08:00
pg.yang 8de8c7c1e8
Support monitoring AWS Cloud EKS (#10199) 2022-12-31 17:34:30 +08:00
Jiajing LU 222bd4acdb
Refactor Topology query for BanyanDB (#10181) 2022-12-23 16:36:44 +08:00
mrproliu 0b838f9a98
Make the satellite E2E test more stable. (#10176) 2022-12-17 21:04:52 +08:00
kezhenxu94 4ee8389b5c
Add Zipkin UI to webapp (#10167) 2022-12-17 14:30:51 +08:00
吴晟 Wu Sheng 85be0bd568
Remove component_id from the instance topology (#10170) 2022-12-16 21:37:13 +08:00
Jiajing LU bee8a86bda
Optimize BanyanDB interval rules (#10128)
* resolve #10104 #10102
2022-12-13 10:27:41 +08:00
Mahmoud Anwer 3229a5c5ec
Support monitoring mariadb (#10115)
Co-authored-by: yswdqz <945407026@qq.com>
2022-12-11 21:10:23 +08:00
吴晟 Wu Sheng 6ec721e933
Use more aggressive persistent period to stable TTL testing. (#10141) 2022-12-10 22:47:06 +08:00
吴晟 Wu Sheng 4b7ec0bab5
Polish doc (#10114) 2022-12-06 21:59:33 +08:00
Wan Kai ce3196beff
Fix Elasticsearch storage: Query `sortMetrics` missing transform real index column name. (#10036) 2022-11-26 11:15:17 +08:00
Wan Kai ffa4990938
[Feature] Zipkin module support BanyanDB storage. (#10004) 2022-11-22 22:38:38 +08:00
kezhenxu94 87f3fb1890
Use ghcr.io Helm Chart registry in E2E tests (#10005) 2022-11-22 21:44:11 +08:00
yswdqz 6efa15c23f
Modify the format of timebucket in ExtractorSpec (#9988) 2022-11-19 11:33:11 +08:00
kezhenxu94 d48323b958
Adjust OAP log level in E2E to reduce log size (#9972) 2022-11-16 22:42:34 +08:00
Jiajing LU d538868332
Use release version of BanyanDB 0.2.0 and Java Client 0.2.0 (#9946) 2022-11-11 08:46:47 +08:00
yswdqz 109e6dea16
Support MySQL/PostgreSQL cluster mode (#9937) 2022-11-10 22:29:31 +08:00
kezhenxu94 2e3134952e
Refactor jdbc storage implementations to reuse logics (#9904) 2022-11-07 20:02:24 +08:00
soander 42f3396b64
Support the telegraf receiver plugin module (#9620)
* The telegraf receiver plugin module development.

* Refactored code of converting telegraf data and fixed some errors.

* Support Telegraf receiver plugin module.

* Add telegraf-receiver.md, update changes.md and vm-monitoring.md.

* Rename YAML file and change code style.

* Change receiver-telegraf of application.yml.

* The receiver-telegraf e2e test.

* The telegraf receiver e2e test.

* Fix an issue and delete redundant configs.

* Add Unit Test about converting Telegraf metrics.

* Adjust Unit Test about converting Telegraf metrics.

* Add License Header to Unit Test and change binary.xml.

* Change module provider's config initialization mechanism.

* Exclude the telegraf-rules in server-starter pom.xml.

* Fix telegraf e2e test issues.

* Change telegraf e2e test.

* Fix issues

* Fix issues.

* Add Sample convert Unit Test.

* Change vm.yaml, related documents and fix some issues.

* Change backend-vm-monitoring.md.

* Fix SampleConvertTest checkstyle issue.

* Change vm.yaml swap MAL.

* Update menu.yml and binary.xml.

* Update Telegraf Unit test.

* Delete telegraf config package, use meter.analyzer.prometheus package to load config file.

* Reorder telegraf metrics in menu.yml.

* Change e2e, vm, config, linux-service and vm.md.

* Update telegraf.conf file.

* Change url of telegraf.conf file.

* Update e2e.yaml, conf file, menu.yml and delete useless code of provider.

* Update grouping sampleFamily by timestamp and name, and add new UTs.

* Update vm-monitoring.md and .asf.yaml.

Co-authored-by: Superskyyy (ONLINE) <Superskyyy@outlook.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2022-11-05 08:08:20 +08:00
Jiajing LU d32a318637
Update BanyanDB storage module to adapt BanyanDB 0.2 (#9839)
* update to banyandb client 0.2

* simplify group check and creation

* support indexOnly

* partial apply configuration

* support CompleteableFuture

* fix BanyanDBProfileThreadSnapshotQueryDAO

* optimize BanyanDBProfileTaskQueryDAO

* optimize *LogQueryDAO

* support indexType

* add configuration

* use TTL from core config

Signed-off-by: Megrez Lu <lujiajing1126@gmail.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2022-11-04 11:33:21 +08:00
pg.yang 828e6e2f2b
Add virtual MQ analysis for native traces (#9855) 2022-11-02 08:05:35 +08:00
yswdqz e497efd245
Change the way of loading MAL rules (#9822) 2022-11-01 15:11:04 +08:00
kezhenxu94 fb2ac0e7f9
Split TCP related service mesh metrics protobuffer definition (#9851) 2022-10-28 09:14:38 +08:00
mrproliu 2e6e3faef0
Support new records query protocol (#9827) 2022-10-22 11:30:52 +08:00
Wan Kai d98775790a
Support export `Trace` and `Log` through Kafka. (#9817) 2022-10-21 08:04:18 +08:00
pg.yang 4aa516ad58
Polish up APISIX metrics to follow Service concept (#9775) 2022-10-13 15:49:57 +08:00
kezhenxu94 091b7810c1
Unify the service label in postgresql MAL (#9752) 2022-10-10 23:22:29 +08:00
pg.yang 8aa046b8d7
Add APISIX metrics monitoring (#9725) 2022-10-05 18:23:37 +08:00
kezhenxu94 dd6b7a1fc0
Remove Prometheuse fetcher, fix MAL Histogram type bug, and reorganize the doc (#9710)
* [**Breaking Change**]: Remove prometheus-fetcher plugin, please use OpenTelemetry to scrape Prometheus metrics and
   set up SkyWalking OpenTelemetry receiver instead.
 * BugFix: histogram metrics sent to MAL should be treated as OpenTelemetry style, not Prometheus style:
   ```
   (-infinity, explicit_bounds[i]] for i == 0
   (explicit_bounds[i-1], explicit_bounds[i]] for 0 < i < size(explicit_bounds)
   (explicit_bounds[i-1], +infinity) for i == size(explicit_bounds)
   ```
2022-09-30 13:13:03 +08:00
Wan Kai d7ee9a01d6
Support sharding MySQL database instances and tables. (#9689)
* [**Breaking Change**] Support sharding MySQL database instances and tables by [Shardingsphere-Proxy](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-proxy).
  SQL-Database requires removing tables `log_tag/segment_tag/zipkin_query` before OAP starts, if bump up from previous releases.
2022-09-30 08:39:14 +08:00
kezhenxu94 bf4cf9d8a0
Clean up some legacy codes (#9682) 2022-09-25 23:33:42 +08:00
yswdqz f9db9ba2dc
Support monitoring PostgreSQL slow SQLs. (#9673) 2022-09-23 20:31:17 +08:00
pg.yang bbe0a7c4c2
Support analyzing cache related spans to provide metrics and slow commands for cache services from client side. (#9622)
* Support analyzing cache related spans to provide metrics and slow commands for cache services from client side
* Optimize virtual database, fix dynamic config watcher NPE when default value is null 
* [UI] Add virtual cache dashboard
2022-09-19 22:10:32 +08:00
yswdqz d3727b7a9f
Enhance LAL to support convert LogData to DatabaseSlowStatement. (#9575) 2022-09-16 12:42:43 +08:00
kezhenxu94 e4ad2580e1
test: fix unstable log cases (#9601) 2022-09-14 07:13:31 +08:00
yswdqz 12a18e6977
Add layer property to LAL script (#9593) 2022-09-12 22:16:07 +08:00
John Duncan 4adf8b1933
Add old config format cause the otlp OC handler only supports OAP server 9.2.0 (#9562) 2022-09-05 15:18:45 +08:00
kezhenxu94 74cd559c47
Bump up infra e2e version and fix istio cases (#9529) 2022-08-27 11:45:34 +08:00
kezhenxu94 e1a5ee198f
Fix a duplicate config key and polish doc (#9521) 2022-08-25 20:00:08 +08:00
kezhenxu94 c8a78149a6
Enable otlp OC handler by default and polish doc (#9499) 2022-08-22 16:09:59 +08:00
yswdqz 828669ce2a
Support PostgreSQL monitoring. (#9495) 2022-08-22 14:29:26 +08:00
yswdqz aa6c5e9182
Support MySQL monitoring. (#9432) 2022-08-12 16:51:36 +08:00
mrproliu 4c3010c97c
Fix MySQL related E2E failures (#9466) 2022-08-12 14:02:29 +08:00
mrproliu 0c2bf2c9ff
Support analyze the component in the eBPF Network profiling (#9430) 2022-08-06 23:32:14 +08:00
mrproliu 4bd4231b60
Elasticsearch storage support merge all records(without super datasets) indices into one physical index (#9424) 2022-08-05 10:40:33 +08:00
Wan Kai 0a032ced18
Elasticsearch storage support merge all metrics/meter indices into one physical index (#9416) 2022-08-03 21:18:30 +08:00
mrproliu b8cf95d165
Add `SUM_PER_MIN`(sumHistogramPercentile) downsampling in MAL. Add eBPF Network Profiling E2E (#9377) 2022-07-28 10:00:36 +08:00
cui fliter 3fe3a11547
fix some typos (#9292)
Signed-off-by: cui fliter <imcusg@gmail.com>
2022-06-30 23:30:08 +08:00
Wan Kai e346b618ea
Support Zipkin kafka collector. (#9252) 2022-06-21 20:40:43 +08:00
吴晟 Wu Sheng cc282232e4
UI sync & Update Java agent and satellite commit ID for testing. (#9239)
* Sync UI

* Update Java agent and satellite commit ID for testing.

* Fix e2e

* Fix Satellite plugin names

Co-authored-by: wankai123 <wankai123@foxmail.com>
Co-authored-by: mrproliu <741550557@qq.com>
2022-06-16 12:57:57 +08:00
Wan Kai f0aef53d49
Support Zipkin traces collect and zipkin traces query API. (#9177) 2022-06-08 07:40:36 +08:00
kezhenxu94 8d34c17871
Add on demand logs support to service mesh (#9171) 2022-06-06 13:16:55 +08:00
Jiajing LU 0148615350
add negative tags check (#9168) 2022-06-02 16:38:39 +08:00
mrproliu b695983fc5
Add OFF CPU eBPF Profiling E2E Testing and Sync UI (#9122) 2022-05-23 23:09:37 +08:00
Jiajing LU 133b2e4ad9
Add event E2E for BanyanDB (#9119) 2022-05-23 09:31:29 +08:00
吴晟 Wu Sheng 86fb88a10a
Bump up BanyanDB server version (#9118) 2022-05-23 08:43:23 +08:00
mrproliu e8b24e7fb2
Add TiDB, OpenSearch, Postgres storage optional to Trace and eBPF Profiling E2E testing. (#9117) 2022-05-22 14:54:26 +08:00
Jiajing LU 25ebd6c242
Add Banyandb storage e2e (#9112) 2022-05-21 19:48:56 +08:00
Wan Kai 051431fde7
SQL Database: provide `@SQLDatabase.AdditionalEntity` to support create additional tables from a model. (#9098)
* SQL Database: provide `@SQLDatabase.AdditionalEntity` to support create additional tables from a model.
* [Breaking Change] SQL Database: remove SQL Database config `maxSizeOfArrayColumn` and `numOfSearchableValuesPerTag`.
* [Breaking Change] SQL Database: move `Tags list` from `Segment`,`Logs`,`Alarms` to their additional table.
* Upgrade doc
2022-05-20 08:29:32 +08:00
mrproliu e451052357
Remove `total` field in Alarm, Browser Logs, and Alarm query (#9102) 2022-05-19 23:04:44 +08:00
mrproliu 7c295738a8
Remove `total` field in Log and Trace query (#9094) 2022-05-19 06:50:37 +08:00
mrproliu 91b94cfc10
Adapt the process procotol update (#9069) 2022-05-16 16:30:56 +08:00
吴晟 Wu Sheng fcc6a16215
Remove InfluxDB 1.x and Apache IoTDB 0.X as storage options (#9060)
* Remove InfluxDB 1.x and Apache IoTDB as storage options.

* Update dependencies.
2022-05-13 21:30:24 +08:00
mrproliu 9494fb2545
Remove layer from process and remove duration from eBPF Profiling scheduler query (#9034) 2022-05-11 17:15:09 +08:00
Wan Kai 2e7cf5db10
Remove `Layer` concept from `ServiceInstance`. (#9029) 2022-05-11 09:59:33 +08:00
kezhenxu94 59c63d90e8
Add Envoy internal cost metrics (#9020) 2022-05-10 12:53:38 +08:00
Wan Kai 26d8c36e3a
VM monitoring adapt the latest version `0.50.0`of the `opentelemetry-collector` (#9013) 2022-05-09 16:54:05 +08:00
Wan Kai 054e01a6b1
E2E: fix log test miss verify LAL and metrics (#8981)
* E2E: fix log test miss verify LAL and metrics

* fix kafka log case, set rover test interval 10s
2022-05-03 17:08:25 +08:00
mrproliu 48a5b8cc59
Add the support eBPF profiling field into the process entity (#8976) 2022-04-30 17:31:31 +08:00
Superskyyy 211efc0562
Further remove jetty references and config (#8974) 2022-04-29 14:32:44 +08:00
mrproliu a93667184f
Support receive Kubernetes process from the Rover (#8960) 2022-04-27 19:20:50 +08:00
Hoshea Jiang 8247bc3c71
Add layer field to event (#8881) 2022-04-19 16:49:52 +08:00
mrproliu db3c1da10d
Update the eBPF Profiling task as service level (#8840) 2022-04-14 07:30:40 +08:00
kezhenxu94 118ba3b569
Fix wrong example in doc, sync Python E2E version and adjust E2E (#8751)
* Fix wrong example in doc, sync Python E2E version and adjust E2E
2022-04-07 13:02:21 +08:00
吴晟 Wu Sheng 34a4553e23
Bump up versions for Apache httpclient, commons-io, postgresql, log4j2(test codes) (#8811) 2022-04-05 20:26:18 +08:00
mrproliu 72e4b12dc3
Add the Self Observability template of SkyWalking Satellite (#8776) 2022-03-31 18:39:59 +08:00
Kai b57866eb67
Adjust MAL for V9 and update UI template (#8772)
* Add ui template custom folder.
Adjust MAL for V9, remove some group, add new Service func for custom delimiter .
Sync ui.

* changes.md

* fix e2e

* fix e2e

* Update CHANGES.md

* remove e2e oap istio-dp group

* remove  vm group

* fix

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2022-03-31 13:49:07 +08:00
刘威 1554c087f4
Refactor IoTDB storage plugin and bump up iotdb-session to 0.12.5 (#8755) 2022-03-30 19:19:49 +08:00
mrproliu 2a8bf4deff
Support eBPF Profiling E2E (#8761) 2022-03-30 12:11:30 +08:00
吴晟 Wu Sheng d90e2e6ef6
Enhance endpoint dependency analysis to support cross threads cases. (#8747)
* [**Breaking Change**] Remove **Deprecated responseCode** from sources, including Service, ServiceInstance, Endpoint
* Enhance endpoint dependency analysis to support cross threads cases. Refactor span analysis code structures.
* Remove `isNotNormal` service requirement when use alias to merge service topology from client side. All RPCs' peer
  services from client side are always normal services. This cause the topology is not merged correctly.

-----------

In our showcase and Java agent scenarios, there are several cross-thread tracing cases, we were not generating endpoint dependency if ref exists in local and exit span. In this PR, I am adding this missing part back.
Also, in case the downstream endpoint of the dependency is another local/exit span's operation name, which was not generating endpoint traffic before, I add traffic generation of both sides' endpoints of dependency.
Another important point that drives me to fix this is, in the new Booster UI, metrics of endpoint dependency are shown in a separate dashboard, which requires the endpoints of both sides existing in metadata, otherwise, nothing shows up.

For e2e, updated the Java agent to the latest.
2022-03-26 20:00:12 +08:00
kezhenxu94 80d3a4a1ba
Sync NodeJS version and adjust E2E (#8730) 2022-03-23 23:07:45 +08:00
Cool-Coding b9c5283468
Fix the configuration of `Aggregation` and `GC Count` metrics for oap self observability(#8707) (#8719) 2022-03-21 19:03:39 +08:00
kezhenxu94 bf03d9f67a
Improve CI with the GHA new feature (#8691) 2022-03-17 10:06:53 +08:00
kezhenxu94 178844e24e
Add istio 1.13.1 to test matrix for verification (#8585) 2022-02-26 13:25:33 +08:00
Jiajing LU b9de4c06cb
Publish e2e-service-* images to GitHub container registry (#8590) 2022-02-25 22:17:25 +08:00
kezhenxu94 68e40eb9fc
Support ElasticSearch 8 and add it into E2E tests (#8564) 2022-02-21 08:11:27 +08:00
Jared Tan c1765b5177
[OAL] Keep the endpoint avg resp time meter name the same with others scope. (#8534)
* Keep the endpoint avg resp time meter name the same with others scope.

* fix e2e.

* update changes.

* Update CHANGES.md

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>

* sync submodule.

* revert break changes

* fix UT

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2022-02-11 20:48:02 +08:00
刘威 b0a875fe3f
Bump up iotdb-session to 0.12.4 and fix IoTDB storage option bug (#8484)
* Fix IoTDB Storage Option insert null index value.

* Set the default value of SW_STORAGE_IOTDB_SESSIONPOOL_SIZE to 8

* Bump up iotdb-session to 0.12.4

* Update doc
2022-01-30 09:02:39 +08:00
wankai123 c0d7d466bf
E2E: Fix wrong health check command and java agent eclipse-temurin image in kafka scenario Process No issue (#8486) 2022-01-28 21:20:26 +08:00
吴晟 Wu Sheng 81f05c349d
Bump commit ID of Java agent for 8.9.0 release RC (#8485)
* Bump commit ID of Java agent to 8.9.0 release RC

Prepare for Java agent 8.9.0 release

* Fix wrong health check command

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-01-28 19:08:50 +08:00
wankai123 473debfee3
E2E verify OAP cluster model data aggregation and fix `SelfRemoteClient` self observing metrics. (#8481)
* E2E: verify OAP cluster model data aggregation.
* Fix `SelfRemoteClient` self observing metrics.
* Remove unnecessary storage cases in the cluster e2e.
* Add steps in cluster e2e to verify the whole cluster is up
* update the doc
2022-01-27 15:40:20 +08:00
kezhenxu94 f729c0c5a4
Add Docker images based on 17 and test it (#8476) 2022-01-25 18:28:09 +08:00
wankai123 d418610357
E2E: Add normal field to Service (#8465)
* e2e: Add normal field to Service
2022-01-24 11:53:15 +08:00
Gao Hongtao 4d803e387f
Fix CVE-2022-23221 (#8464)
* Fix CVE-2022-23221

H2 Console before 2.1.210 allows remote attackers to execute arbitrary code via a jdbc:h2:mem JDBC URL containing the IGNORE_UNKNOWN_SETTINGS=TRUE;FORBID_CREATION=FALSE;INIT=RUNSCRIPT substring

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>
2022-01-22 21:03:08 +08:00
wankai123 17e8fa8ae5
Adjust e2e case for V9 core (#8429)
* Adjust e2e case for V9 core

* fix swctl install scrip

* fix cases of browser, iotdb

* fix instance serialize
2022-01-17 12:35:25 +08:00
mrproliu e97b2d2165
Fix query sort metrics failure in H2 storage (#8393)
* Fix sort metrics failured in H2 storage

* Fix postgre DB
2022-01-06 15:11:05 +08:00
wankai123 8bcf22f3d8
Support OAP server v9 Core concept: Layer, backend side (#8379) 2022-01-04 16:16:24 +08:00
kezhenxu94 8fa4eb7d75
Upgrade H2 version to 2.0.202 to fix CVE-2021-23463 (#8332) 2021-12-23 21:37:00 +08:00
mrproliu 05991f34a6
Change using service lebel metrics (#8319) 2021-12-21 15:24:13 +08:00
mrproliu 28844632ec
Add the metrics analysis in Satellite MetricsService. (#8312) 2021-12-18 20:03:18 +08:00
刘威 842b5d927e
Support Apache IoTDB as a storage option (#7766)
Support Apache IoTDB as a storage option, mostly refer to previous InfluxDB storage option.
* The Design of Apache IoTDB Storage Option, https://skywalking.apache.org/blog/2021-11-23-design-of-iotdb-storage-option/
2021-11-27 09:04:23 +08:00
kezhenxu94 a3bdbeb576
Add OpenSearch 1.2.0 to test and verify it works (#8183) 2021-11-25 22:18:02 +08:00
liqiangz 8436135dc9
Add MeterReportService collectBatch method. (#8165) 2021-11-22 22:11:39 +08:00
kezhenxu94 973fba0805
Add filter mechanism in MAL core and fix some bugs (#8157) 2021-11-19 23:33:35 +08:00
kezhenxu94 ba630a3cb9
Add Docker images for arm64 architecture (#8141) 2021-11-17 21:26:25 +08:00
何延龙 7aa0ba7048
[CI] Update PHP Agent (#8054) 2021-11-02 19:59:16 +08:00
wankai123 32f9ab01f2
E2e immigrate to e2e-v2 finished, remove old e2e. (#8037) 2021-10-29 14:32:16 +08:00
wankai123 fea9ce366c
Replace e2e cases to e2e-v2: Alarm (#8036) 2021-10-29 09:25:48 +08:00
wankai123 c9b28e4d00
Replace e2e cases to e2e-v2: TTL (#8028) 2021-10-28 12:21:35 +08:00
mrproliu c9bd79e8bb
Replace e2e cases to e2e-v2: Istio (#8013) 2021-10-27 12:11:47 +08:00
zhang-wei 3470f7fb3b
Replace e2e cases to e2e-v2: Client-JS (#7999) 2021-10-25 23:44:40 +08:00
wankai123 f13502fc2c
e2e-v2: Simplified cluster e2e to reduce failure (#7989) 2021-10-22 12:58:52 +08:00
wankai123 d1dcd3f96c
Replace e2e cases to e2e-v2: Kafka: Base, Meter, Log, Profile (#7986)
* Replace e2e cases to e2e-v2: Kafka: Base, Meter, Log, Profile
* Set `SW_KAFKA_FETCHER_ENABLE_NATIVE_PROTO_LOG`, `SW_KAFKA_FETCHER_ENABLE_NATIVE_JSON_LOG` default `true`.
2021-10-22 10:54:49 +08:00
mrproliu 8d24ba2d9f
Add `getProfileTaskLogs` query method implement (#7984) 2021-10-22 08:20:10 +08:00
wankai123 5a6a1dbb26
Replace e2e cases to e2e-v2: Profile (#7967) 2021-10-20 10:19:10 +08:00
wankai123 8e59e55040
Replace e2e cases to e2e-v2: Event (#7951) 2021-10-18 22:46:56 +08:00
wankai123 bd91ca7c80
Replace e2e cases to e2e-v2 Cluster (#7948) 2021-10-18 11:29:47 +08:00
Jared Tan b5578b68fd
Replace e2e cases to e2e-v2: Storage (#7934)
* replace log e2e cases to e2e-v2.

* update change log

* fix change log.

* revert proto chenge

* fix e2e

* active h2

* contains

* revert storage e2e mis-change
2021-10-16 18:25:29 +08:00
wankai123 d73c51a4dd
Replace e2e cases to e2e-v2: Python (#7932) 2021-10-15 16:31:35 +08:00
Jared Tan 61d4749288
Replace log e2e cases to e2e-v2 (#7856)
* replace e2e with infra e2e.

* update e2e

* update e2e

* fix fluentbit & es6

* fix fluentbit & es6

* fix rules.

* fix docker compose in h2 case.

* update e2e

* fix e2e log

* fix env in e2e.

* fix case in e2e.

* fix case in e2e.

* Fix and refine matrix

* update fluent mount.

* fix missed provider service and wrong triggers in e2e.

* fix docker compose

* fix typo

* fix endpoints expected.

* fix mysql storage mount mysql driver.

* add ports for provider.

* add oap port for trigger.

* update e2e check

* fix mysql

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2021-10-15 13:26:17 +08:00
wankai123 bafc3cde9d
Replace e2e cases to e2e-v2: go2sky (#7929) 2021-10-14 21:50:37 +08:00
wankai123 b08cac9ad9
Replace VM e2e cases to e2e-v2: Prometheus Node Exporter, Zabbix (#7924) 2021-10-14 17:37:09 +08:00
wankai123 d6f943bbd5
Replace e2e cases to e2e-v2: PHP (#7919) 2021-10-13 23:25:19 +08:00
wankai123 f4c99db4f0
Replace e2e cases to e2e-v2: Nodejs (#7914) 2021-10-13 09:59:29 +08:00
wankai123 29dbcf461e
Replace e2e cases to e2e-v2: Meter (#7912) 2021-10-12 15:51:11 +08:00
wankai123 52fc942aa0
Replace e2e cases to e2e-v2: Gateway (#7908) 2021-10-12 10:26:35 +08:00
wankai123 8579572be2
Replace e2e cases to e2e-v2: Lua Nginx/SelfObservability. (#7847) 2021-10-11 10:19:37 +08:00
Fine0830 58ce0b4056
test: update commitID for client-js (#7874)
* test: update commitID for client-js

* fix: remove BROWSER_APP_PAGE_REDIRECT_AVG for e2e

* fix: remove BROWSER_APP_PAGE_RES_AVG for e2e

* fix: remove unused imports

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: zhang-wei <zhangwei24@apache.org>
2021-10-09 19:06:39 +08:00
kezhenxu94 508945f552
Upgrade Armeria to 1.12, upgrade OpenSearch test version to 1.1.0 (#7866) 2021-10-08 19:45:33 +08:00
kezhenxu94 f76ad7a01d
Fix wrong rules for checking lists in E2E (#7849) 2021-10-08 14:54:27 +08:00
wankai123 cf40b9de0d
Replace e2e simple cases to v2 (#7827) 2021-09-28 22:42:42 +08:00
kezhenxu94 a4c68d30d8
test: fix that E2E is not run, also add ElasticSearch 7.15 to test matrix (#7826) 2021-09-28 21:57:19 +08:00
kezhenxu94 4c95f07adb
Set the Docker image names the same as the ones in hub.docker.io (#7820) 2021-09-28 08:11:19 +08:00
mrproliu 986073f949
Add tag mangement to e2e-v2 (#7821) 2021-09-27 23:56:28 +08:00
mrproliu c467e4df06
Support E2E testing satellite native protocols (#7810) 2021-09-27 15:35:00 +08:00
kezhenxu94 f61dff9a9f
Merge `Dockerfile`s in apache/skywalking-docker into this codebase (#7681) 2021-09-09 12:37:50 +08:00
kezhenxu94 559b88a17b
Clean up checkstyle import control and unused licenses (#7660) 2021-09-06 17:44:18 +08:00
kezhenxu94 26b81a2e59
Rebuilt ElasticSearch client on top of their REST API (#7634) 2021-09-06 15:21:35 +08:00
Daming 0b67d29996
Support mTLS for gRPC channel (#7565) 2021-09-03 09:17:16 +08:00
kezhenxu94 444adc6be1
Lock Java agent version in E2E tests (#7625) 2021-09-01 09:37:50 +08:00
kezhenxu94 789ad4c0cf
Remove agent codes and clean up (#7588)
* Remove agent codes and clean up
2021-08-27 20:05:08 +08:00
kezhenxu94 6bb4d35ce8
Add all agent config to `agent.conf` file for convenient containerization use cases (#7470)
* Add all agent config to `agent.conf` file for convenient containerization use cases

* Fix name and default value
2021-08-18 10:25:28 +08:00
hailin0 ce45397c56
Fix kafka-reporter-plugin shade package conflict (#7463)
* Fix kafka-reporter-plugin shade package conflict

Don't shade lz4, snappy or zstd package because they call native library files.
2021-08-15 21:55:45 +08:00
wallezhang 4b66f1e0a7
fix: Fix MultiScopesAnalysisListener#setPublicAttrs log warn frequently (#7271)
* fix: Fix MultiScopesAnalysisListener#setPublicAttrs log warn frequently. #6562

1. Rename current STATUS_CODE to HTTP_RESPONSE_STATUS_CODE, real tag key should be http.status_code.
2. Add a new string tag RPC_RESPONSE_STATUS_CODE to hold all string type value tag. The key is `rpc.status_code`.
3. At the backend, keep responseCode for HTTP_RESPONSE_STATUS_CODE tag, but add @Deprecated. Also, add httpResponseStatusCode(a duplicate of responseCode) and rpcStatusCode(for RPC_RESPONSE_STATUS_CODE) fields for OAL.

* Modify `Tags.STATUS_CODE` field name to `Tags.HTTP_RESPONSE_STATUS_CODE` and type from `StringTag` to `IntegerTag`, add `Tags.RPC_RESPONSE_STATUS_CODE` field to hold rpc response code value.

* Add `rpcStatusCode` for `rpc.status_code` tag in the OAL objects. The `responseCode` field is marked as deprecated and replaced by `httpResponseStatusCode` field. 

* Tag `status_code` is still supported in the backend for forward compatbility. Will be removed one year later.

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2021-08-14 20:17:46 +08:00
吴晟 Wu Sheng 6a241aed75
[Break Change] Remove endpoint name in backend log and browser log query (#7444)
* Remove page path in the browser log query condition.

* Remove endpoint name in the backend log query condition.
2021-08-12 09:44:19 +08:00
kezhenxu94 85e22f5124
Fix issues in ES7.14 and add log cases to ES7.14 (#7429) 2021-08-10 15:36:53 +08:00
wankai123 4e7d80fb88
Support Metric level function `serviceRelation` in `MAL` and binding envoy metrics into the topology. (#7399)
* Support Metric level function `serviceRelation` in `MAL`.
* Support envoy metrics binding into the topology.
* add istio args
* remove istio 1.7

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: Jared Tan <jian.tan@daocloud.io>
Co-authored-by: Zhenxu <kezhenxu94@apache.org>
2021-08-06 17:09:13 +08:00
吴晟 Wu Sheng b903bf17a0
Add ElasticSearch 7.14 to test and bump up OpenSearch version to 1.0 GA (#7415) 2021-08-06 10:36:45 +08:00
ThisSeanZhang 0b418bf379
Support Multiple DNS period resolving mechanism (#7398) 2021-08-02 11:16:21 +08:00
Switch 68e699550d
Add Thread and ClassLoader Metrics for the self-observability and otel-oc-rules (#7367)
* Add Thread and ClassLoader Metrics for the self-observability

* update CHANGES.md

* update OC receiver MAL settings
2021-07-25 07:32:03 +08:00
吴伟杰 da9426dd6e
Upgrade agent plugin for ElasticJob 3.0.0 GA version (#7345)
* Upgrade agent plugin for ElasticJob 3.0.0 GA version.

* Remove 3.0.0 alpha support.
2021-07-22 15:30:00 +08:00
吴伟杰 06e589fab0
Remove plugins for ShardingSphere legacy and unstable versions (#7323)
Remove the following plugins due to they are legacy and unstable releases.
1. sharding-jdbc-1.5.x-plugin
2. sharding-sphere-4.x-rc3-plugin
3. sharding-sphere-4.x-plugin (This is for RC1 and RC2)

To people who want to use these removing versions, you still could copy the plugins from SkyWalking's 8.6.0 release to the '/plugins' folder to activate, unless SkyWalking has some core level APIs upgrade. This is very rare to happen.
2021-07-18 20:35:54 +08:00
wankai123 9f3ff517d1
Support envoy cluster manager metrics (#7294) 2021-07-16 15:35:48 +08:00
Fine0830 ff7c8fa87e
feat: update e2e for client-js (#7297) 2021-07-14 21:43:27 +08:00
吴晟 Wu Sheng de975c7e2b
Remove synchronous persistence mechanism in API level and ElasticSearch implementation (#7293)
* Performance: remove the synchronous persistence mechanism from batch ElasticSearch DAO. Because the current enhanced persistent session mechanism, don't require the data queryable immediately after the insert and update anymore.
* Performance: share `flushInterval` setting for both metrics and record data, due to `synchronous persistence mechanism` removed. Record flush interval used to be hardcoded as 10s.
* Remove `syncBulkActions` in ElasticSearch storage option.
* Increase the default bulkActions(env, SW_STORAGE_ES_BULK_ACTIONS) to 5000(from 1000).
* Increase the flush interval of ElasticSearch indices to 15s(from 10s)

Add these 2 references. According to these, **(same _index, _type and _id) in same bulk will be in order**
1. https://github.com/elastic/elasticsearch/issues/50199
2. https://discuss.elastic.co/t/order-of--bulk-request-operations/98124

Notice, the order of different bulks is not guaranteed by the ElasticSearch cluster. We are going to have the risk of dirty write. But consider we set over 20s period between flush, and index flush period is 10, we should be safe.

Recommend 5000 bulk size and 15s flush interval only. The persistent period has been set to 25s.
2021-07-14 17:19:49 +08:00
kezhenxu94 d79094c178
Add pagination to event query (#7274) 2021-07-09 23:04:50 +08:00
kezhenxu94 fd92f4678a
perf: optimize Groovy-based LAL DSL with static compilation (#7263)
Groovy naturally supports many dynamic features that we don't benefit for now but cost performance loss, in this patch we compile our Groovy-based DSL scripts statically to optimize performance.
2021-07-09 18:00:25 +08:00
Ax1an 58dce444fc
Enhance the compatibility of `mysql-8.x-plugin` plugin. (#7261)
hostInfo.getDatabaseUrl() result in version 8.0.19 or above:

jdbc:mysql:loadbalance://**internally_generated**1620805591623**

hostInfo.getDatabaseUrl() result in version 8.0.18 or below:

jdbc:mysql:replication://127.0.0.1:3306,127.0.0.1:3306,127.0.0.1:3306/mer_goods_admin_sit?useSSL=true&useUnicode=true&autoReconnect=true&rewriteBatchedStatements=TRUE&allowMultiQueries=true&serverTimezone=GMT%2B8
2021-07-08 17:55:00 +08:00
whl12345 9e022d1a73
Provide plugin for Guava Cache (#7235) 2021-07-08 11:58:09 +08:00
Fine0830 f9140b80fa
test: update commitID (#7255) 2021-07-07 17:46:29 +08:00