Commit Graph

80 Commits

Author SHA1 Message Date
kezhenxu94 09cc0f96a7
Harden the security of Groovy-based DSL, MAL and LAL (#7485) 2021-08-18 18:17:43 +08:00
kezhenxu94 1af2e46e33
Add a new API to test log analysis language (#7475) 2021-08-18 16:40:25 +08:00
CharliePu 39d23c1678
Remove the duplicated tags to reduce the storage payload. (#7469) 2021-08-17 18:01:56 +08:00
wallezhang 4b66f1e0a7
fix: Fix MultiScopesAnalysisListener#setPublicAttrs log warn frequently (#7271)
* fix: Fix MultiScopesAnalysisListener#setPublicAttrs log warn frequently. #6562

1. Rename current STATUS_CODE to HTTP_RESPONSE_STATUS_CODE, real tag key should be http.status_code.
2. Add a new string tag RPC_RESPONSE_STATUS_CODE to hold all string type value tag. The key is `rpc.status_code`.
3. At the backend, keep responseCode for HTTP_RESPONSE_STATUS_CODE tag, but add @Deprecated. Also, add httpResponseStatusCode(a duplicate of responseCode) and rpcStatusCode(for RPC_RESPONSE_STATUS_CODE) fields for OAL.

* Modify `Tags.STATUS_CODE` field name to `Tags.HTTP_RESPONSE_STATUS_CODE` and type from `StringTag` to `IntegerTag`, add `Tags.RPC_RESPONSE_STATUS_CODE` field to hold rpc response code value.

* Add `rpcStatusCode` for `rpc.status_code` tag in the OAL objects. The `responseCode` field is marked as deprecated and replaced by `httpResponseStatusCode` field. 

* Tag `status_code` is still supported in the backend for forward compatbility. Will be removed one year later.

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2021-08-14 20:17:46 +08:00
吴晟 Wu Sheng 6a241aed75
[Break Change] Remove endpoint name in backend log and browser log query (#7444)
* Remove page path in the browser log query condition.

* Remove endpoint name in the backend log query condition.
2021-08-12 09:44:19 +08:00
吴晟 Wu Sheng de6cae3429
[Break Change] Remove endpoint name in the trace query condition. (#7438)
* [Break Change] Remove endpoint name in the trace query condition.

* Fix a local UT error(MacOS only)
2021-08-11 16:27:10 +08:00
吴晟 Wu Sheng cc66254729
Remove statement, version, and endtime from segment entity. (#7427) 2021-08-10 00:05:06 +08:00
wankai123 4e7d80fb88
Support Metric level function `serviceRelation` in `MAL` and binding envoy metrics into the topology. (#7399)
* Support Metric level function `serviceRelation` in `MAL`.
* Support envoy metrics binding into the topology.
* add istio args
* remove istio 1.7

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: Jared Tan <jian.tan@daocloud.io>
Co-authored-by: Zhenxu <kezhenxu94@apache.org>
2021-08-06 17:09:13 +08:00
吴晟 Wu Sheng 1c5e22a7be
Begin 8.8.0 iteration (#7395)
* [maven-release-plugin] prepare release v8.7.0

* [maven-release-plugin] prepare for next development iteration
2021-07-30 23:01:09 +08:00
kezhenxu94 65e8ab3ecc
Log rate limit sampler rename `qps` to `rpm` (#7384) 2021-07-29 18:11:07 +08:00
Huang bf0e22d2c1
Fix IllegalArgumentException in PrometheusMetricConverter and NaN value samples in SampleFamily (#7349) 2021-07-21 23:14:49 +08:00
kezhenxu94 982001c63a
perf: optimize Envoy access log persistence performance (#7290) 2021-07-13 18:44:52 +08:00
kezhenxu94 fd92f4678a
perf: optimize Groovy-based LAL DSL with static compilation (#7263)
Groovy naturally supports many dynamic features that we don't benefit for now but cost performance loss, in this patch we compile our Groovy-based DSL scripts statically to optimize performance.
2021-07-09 18:00:25 +08:00
Switch 4067fba2ab
Add some new thread metric and class metric to JVMMetric (#7230) (#7243) 2021-07-07 16:30:55 +08:00
kezhenxu94 84a7fe30c8
Perf: cache regex pattern and result, optimize string concatenation (#7199)
The possible metrics names are relatively fixed in an OAP run, we previously always escape the metrics names, building a new regex Pattern from scratch, which is cpu-consuming.

In this patch, I cache the escaped metrics names, and if it missed, use a pre-compiled regex Pattern to escape the metrics name.

This patch also replace string concatenation `+` with `StringBuilder` in generated classes, which won't get optimized by Java compiler

This may reduce 1~1.5 vCPU under 20K RPS environment.
2021-06-29 17:46:42 +08:00
zhyyu b8fadec033
fix slow db dynamic config bug (#7192) 2021-06-29 16:41:01 +08:00
liqiangz d63f3ffcc1
revert #7091 (#7189) 2021-06-28 21:05:28 +08:00
Daming d8a5e8c15b
eliminate `now` error at a fixed rate schedule executor (#7145) 2021-06-24 09:24:48 +08:00
kezhenxu94 3536b3c1c4
Extract dependency management to a bom (#7136) 2021-06-20 20:57:26 +08:00
kezhenxu94 528ee6de40
OAL supports generating metrics from events (#7134)
* OAL supports generating metrics from events

* Add new dashboard for event metrics

* Add some docs for event metrics
2021-06-19 20:20:37 +08:00
liqiangz 7a3355f956
Use MAL to calculate JVM metrics, remove OAL dependency. (#7091) 2021-06-11 22:21:10 +08:00
吴晟 Wu Sheng 5eb1021639
Initialize 8.7.0 iteration (#7078) 2021-06-08 07:41:09 +08:00
Jared Tan 17abef4c59
polish debug logging avoid null value when segment be ignored. (#7070)
* polish debug logging avoid null value.

* update CHANGES.md.
2021-06-05 19:23:46 +08:00
Zhenxu 56f2e86a07
CVE: upgrade snakeyaml to prevent billion laughs attack in dynamic configuration. (#7071) 2021-06-05 14:41:46 +08:00
Leon Yang a1c5dc62a0
FIX: register Watcher (#6956) 2021-05-17 17:48:46 +08:00
Zhenxu Ke 94adbf1c1f
Bump Kubernetes client-java version (#6920) 2021-05-10 05:07:09 +08:00
wankai123 db1d655b47
Fix: MAL function would miss samples name after creating new samples. (#6864) 2021-04-28 23:09:57 +08:00
吴晟 Wu Sheng 17e820f156
Begin 8.6.0 iteration (#6721) 2021-04-09 23:01:00 +08:00
wankai123 f9096f5086
Support k8s monitoring (#6479) 2021-04-06 14:11:02 +08:00
wankai123 523944a59f
Add arg `namespace` to func `retagByK8sMeta`, rebuild the relationship between pod and service by labels (#6650)
* Add `namespace` to func `retagByK8sMeta`, rebuild the relationship between pod and service by labels

* add param check
2021-03-30 08:48:22 +08:00
wankai123 92c0cb856d
Add function `retagByK8sMeta` and opt type `K8sRetagType.Pod2Service` in MAL for k8s to relate pods and services. (#6608) 2021-03-25 20:50:58 +08:00
wankai123 c2c3830a80
Add functions in MAL to filter metrics according to the metric value (#6587) 2021-03-20 17:12:34 +08:00
wankai123 a8810b1ab0
Support metrics grouped by scope labelValue in MAL, no need global same labelValue as before. (#6495) 2021-03-17 09:02:38 +08:00
吴晟 Wu Sheng af6f7d2c34
Remove group concept from AvgHistogramFunction (#6552) 2021-03-16 10:26:33 +08:00
Zhenxu Ke cb16e524da
Enhance the LAL to allow easily skipping logs with malformed formats (#6477) 2021-03-03 17:18:50 +08:00
Zhenxu Ke de6a731d72
Save Envoy http access logs when error occurs (#6322) 2021-03-02 22:40:30 +08:00
haoyann 72546a75c3
fix `DatabaseSlowStatementBuilder` statement maybe null (#6470) 2021-02-28 17:51:03 +08:00
Zhenxu Ke 77082de77b
Add `sum` function in meter system (#6427)
* Add `sum` function in meter system

* Fix minor potential bug and reformat some codes

* Fix a bug

* Use Set to hold the {aggregate,scope} labels as same labels maybe added by multiple aggregation functions

* Separate downsampling function and revert unnecessary codes
2021-02-25 15:10:28 +08:00
Zhenxu Ke 96f1c12159
Introduce log analysis language (LAL) (#6388) 2021-02-23 20:06:48 +08:00
mrproliu 317a6580f3
Remove filename suffix in the meter active file config (#6413) 2021-02-21 08:35:38 +08:00
mrproliu 6d92bc66dd
Meter system enhancement (#6398) 2021-02-19 20:51:03 +08:00
mrproliu 75f03acba7
Adopt Zabbix protocol (#6176)
Happy Chinese New Year !!
2021-02-10 22:40:07 +08:00
吴晟 Wu Sheng 5e8f1eb552
Support multiple implementations of StorageBuilder in different storage implementations - stage 2 (#6336) 2021-02-07 10:37:14 +08:00
Zhenxu Ke b92eac2cd6
Add a new concept "Event" and its implementations to collect events (#6183) 2021-02-05 18:37:18 +08:00
吴晟 Wu Sheng cdfe6ab358
Begin the 8.5.0 iteration (#6302)
* [maven-release-plugin] prepare release v8.4.0

* [maven-release-plugin] prepare for next development iteration
2021-02-02 08:01:49 +08:00
wankai123 c52aa02341
Fix bug that the same sample name in one MAL expression caused `IllegalArgumentException` in `Analyzer.analyse`. (#6250) 2021-01-25 12:56:06 +08:00
Neal Huang 825a7ccc81
Make source builder set service name and endpoint name in right order (#6188) 2021-01-14 20:51:26 +08:00
zhang-wei 15452025c6
Log collecting and query implementation (#6120) 2021-01-14 09:49:11 +08:00
Zhenxu Ke f54f639c4d
Enhance Envoy metrics service analyzer by MAL (#6091) 2021-01-02 14:34:54 +08:00
吴晟 Wu Sheng f4b739c26b
Fix CVE of UninstrumentedGateways in Dynamic Configuration activation. (#6098) 2020-12-29 22:17:30 +08:00