Commit Graph

226 Commits

Author SHA1 Message Date
gerry 3a2706d104
Fix storage-influxdb-plugin serviceCode query (#5792)
* Fix storage-influxdb-plugin serviceCode query

* Update CHANGES.md

Add the changelog for the PR as I have time.

Co-authored-by: yuanguohua <yuanguohua@fin-shine.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-11-04 19:39:06 +08:00
Jared Tan 9042ce6fe2
fix storage-jdbc isExists not set dbname (#5786)
* fix null catalog

* update CHANGES.md
2020-11-04 11:37:54 +08:00
zifeihan a9d756cd53
Fix deadlock problem when using elasticsearch-client-7.0.0 (#5775)
* Update elasticsearch client 7.0.0 to elasticsearch client 7.5.0.
* Update CHANGES.md and LICENSE.
2020-11-03 22:01:54 +08:00
kezhenxu94 f92a7bf897
Start next iteration 8.3.0 (#5719)
* [maven-release-plugin] prepare release v8.2.0

* [maven-release-plugin] prepare for next development iteration
2020-10-24 19:30:29 +08:00
Ax1an 287b8b9be5
Add syncBulkActions config to avoid the large amount of metrics data written ES in a single sync bulk request. (#5699) 2020-10-21 12:23:32 +08:00
xbkaishui d86570a88e
Health check for InfluxDB (#5637) 2020-10-13 16:59:42 +08:00
Kdump a5f4052965
fix INEFFICIENT_KEYSET_ITERATOR (#5612)
* fix INEFFICIENT_KEYSET_ITERATOR
2020-10-06 21:44:23 +08:00
Evan 15b0068fdf
Add superdataset replicas number config (#5424)
* distinct the number of metrics or common record index with super dataset index.
2020-09-01 23:06:21 +08:00
zhang-wei 7ef8b3e476
Support Browser protocol at OAP (#4228) 2020-08-31 15:46:37 +08:00
Lu Jiajing 5bfac1b50f
Remove style-check exception for logger (#5363) 2020-08-21 18:26:25 +08:00
Gao Hongtao 68eba154c4
Add latest function (#5321)
Signed-off-by: Gao Hongtao <hanahmily@gmail.com>
2020-08-15 17:28:32 +08:00
Evan 197ef1e836
Improve Es trace basic query performance (#5132) 2020-08-15 08:24:37 +08:00
吴晟 Wu Sheng 7f7e96b088
[IMPORTANT] Query traces with tags as condition (#5270) 2020-08-11 07:24:40 +08:00
Evan 2df3c683ba
Super Size Dataset record index es rolling step (#5282) 2020-08-10 14:49:11 +08:00
Gao Hongtao b0bb7cf3be
Query all of labels once selected labels condition is absent. (#5202) 2020-08-08 22:26:06 +08:00
吴晟 Wu Sheng 2f2840cbf5
Begin 8.2.0 iteration. (#5211)
* [maven-release-plugin] prepare release v8.1.0

* [maven-release-plugin] prepare for next development iteration
2020-07-31 20:15:27 +08:00
Daming 966403c0e0
missing time unit in query (#5210) 2020-07-31 16:49:19 +08:00
吴晟 Wu Sheng 2c676ead24
Add OAP configuration vocabulary documentation. (#5149) 2020-07-23 08:24:40 +08:00
Gao Hongtao 3fdfdf3b83
Storage elasticsearch health check (#5099) 2020-07-19 23:37:13 +08:00
denis ce2469dbec
optimization MetricsEsDAO multiGet (#5096) 2020-07-14 22:47:37 +08:00
Gao Hongtao b6661b5164
Add health checker module (#5046) 2020-07-10 00:30:43 +08:00
吴晟 Wu Sheng e8f7ae055c
Fix searchService bug in ES implementation . (#5059) 2020-07-09 00:01:30 +08:00
Daming d264f914d5
Fix timebucket conversion issue (#5014) 2020-07-02 22:05:03 +08:00
yangy fb7912c6bd
fix fuzzy query sql injection (#4970) 2020-06-26 10:08:10 +08:00
吴晟 Wu Sheng d305efcb52
In no-init mode, should only check the index existence. (#4929) 2020-06-16 08:06:02 +08:00
吴晟 Wu Sheng 5de0d2c6f6
Set version to 8.1.0 for next iteration. (#4916)
* [maven-release-plugin] prepare release v8.0.0

* [maven-release-plugin] prepare for next development iteration

* Remove all shardingsphere cases.
2020-06-12 23:44:42 +08:00
吴晟 Wu Sheng b07b395691
Fix 4 high priority bugs (#4913) 2020-06-12 12:07:45 +08:00
吴晟 Wu Sheng 0c1b8e4967
Add default templates for dashboard and topology. (#4876) 2020-06-09 08:24:12 +08:00
Daming 3f176bf50f
[hotfix]Fix UITemplate storage Influxdb implamented (#4849) 2020-06-02 16:05:21 +08:00
吴晟 Wu Sheng d094a0f7c0
Support UI template management. (#4843) 2020-06-01 23:06:14 +08:00
吴晟 Wu Sheng 6ed40b3395
Keep ID as ref ID in the readSampledRecords (#4822)
* Keep ID as ref ID in the readSampledRecords, as there is no meaningful logic entity, but based on the graphql, the client could request this.
2020-05-26 08:00:03 +08:00
吴晟 Wu Sheng ec175f5655
Miss the node type condition in the es searchServices query. (#4821) 2020-05-25 22:17:10 +08:00
Evan 1652e0c057
Add a SuperDataset tag for custom super size dataset shards config in es storage (#4793)
* add SuperDataset tag for add super size dataset shards config in es
2020-05-17 17:48:15 +08:00
吴晟 Wu Sheng cffcc58f8f
The NoneStreamEsDAO implementation doesn't use real index name to insert. (#4796) 2020-05-16 22:54:28 +08:00
Daming 544d419fc0
the number type casts error (#4788)
Co-authored-by: zhuanghaochao <zhuanghaochao@yy.com>
2020-05-14 21:13:22 +08:00
zhang-wei d4f3218dea
Support define multiple OAL configuration (#4748) 2020-05-08 23:53:49 +08:00
吴晟 Wu Sheng 7edcd5aa33
Change the model installation into the reactive module (#4761) 2020-05-08 21:21:03 +08:00
Gao Hongtao 1d5a0b0f1a
Format index name with namespace (#4758)
* Format index name with namespace

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

* Use another variable to compare index name

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>

Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
2020-05-08 20:12:56 +08:00
吴晟 Wu Sheng fac4ddc9e8
Fix two serious bugs. (#4741) 2020-05-05 12:55:14 +08:00
kezhenxu94 7b071bd7f7
Fix nested query and type conversion in MySQL, remove unused codes (#4738) 2020-05-04 13:54:12 +08:00
吴晟 Wu Sheng 217bb624f6
Support percentile function in the meter system. (#4725)
* Support percentile function in the meter system.

* Update document.

* Fix formats.
2020-04-29 08:43:26 +08:00
吴晟 Wu Sheng 071708a4a3
Support default value in labeled-value and heatmap query. (#4711)
* Support default value in labeled-value and heatmap query.

* Update a little document.
2020-04-25 21:24:15 +08:00
吴晟 Wu Sheng b635e254c4
Provide Meter(metrics) system (#4694)
Here are the basic and simple usages of the MeterSystem APIs.

## Meter Creation
New meter could be created based on metrics name, function, scope and data type. 
1. Metrics name is still the storage entity name.
2. Functions are every similar with the OAL function, just accepting different input.
```java
final MeterSystem meterSystem = MeterSystem.meterSystem(getManager());
meterSystem.create("test_long_metrics", "avg", ScopeType.SERVICE, Long.class);
```
NOTICE, this creation should only be called in the `module#prepare` stage, otherwise, `Can't create new metrics anymore` exception will be raised after the **CORE** module `start` stage finished. You may find out, there is a chance `create` could be executed successfully in your own `module#start` stage, but it it just because of the sequence of provider loaded by the class loader, **no guarantee, so please don't do that**.

## Runtime Calculation
`AcceptableValue` is the object created at the runtime to accept new metrics value.
```java
         final MeterSystem service = getManager().find(CoreModule.NAME).provider().getService(MeterSystem.class);
        Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                final AcceptableValue<Long> value = service.buildMetrics("test_long_metrics", Long.class);
                value.accept(MeterEntity.newService("abc"), 5L);
                value.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis()));
                service.doStreamingCalculation(value);
            }
        }, 2, 2, TimeUnit.SECONDS);
```

## Meter Functions
Right now, only `avg` function has been implemented. I submit this PR as soon as possible to get your feedback. I will add more functions.

## Notice
1. Make slow trace query available in the sampled record. Logically, they are the same thing. The UI doesn't need to concern about the trace as a special case.
2. Endpoint dependency will be removed from the dashboard. Because no query available for it. We will provide a new page for that in 8.1
3. Comparison page will be removed due to dashboard is powerful enough to replace it.
2020-04-24 23:33:13 +08:00
吴晟 Wu Sheng 991d083378
Support typeOfMetrics query (#4686)
* Add data type to the column definition to replace the simple isValue definition. And top N query is also requiring value column now.

* Support typeOfMetrics query.
2020-04-22 13:14:59 +08:00
吴晟 Wu Sheng 418bdff277
New metrics query protocol v2 (#4679)
1. Support the new query protocol, and the v1 query protocol is still supported 
2. All `numOfxxx`/`GlobalBrief` is only a mock now, no real number.
3. typeOfMetrics service is a mock only too, follow up PR will implement this.
2020-04-21 15:39:18 +08:00
Daming 28530cd79d
Upgrade the InfluxDB storage-plugin to protocol V3 (#4641)
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2020-04-19 20:10:59 +08:00
Gao Hongtao 12703fa54e
Keep latest index always present. (#4655)
* Keep latest index always present.
  * Avoid TTL timer to remove latest index
  * Create a latest index even without any data.

Signed-off-by: Gao Hongtao <hanahmily@gmail.com>
2020-04-15 15:08:38 +08:00
吴晟 Wu Sheng 4ce2e9e873
Fix security issue of the metrics query (#4639) 2020-04-12 15:44:48 +08:00
吴晟 Wu Sheng 31385159c2
Support to use empty string to override existing config. (#4635) 2020-04-11 13:05:57 +08:00
吴晟 Wu Sheng 6fe2041b47
Make 8.0.0 Core available. New protocol and register removed. (#4599)
Here are the list of changes we made in this pull request. Top 3 are the principle changes, others are following these principles.
1. New agent and mesh report protocol. 
2. New agent header protocol. 
3. Service register, instance register and network address register have been removed permanently.
4. Service traffic, instance traffic and network alias metrics are added to replace the service, instance and network address inventory.
5. Register process has been removed.
6. Metrics stream process supports insert only mode, especially for traffic entities.
7. Metrics stream process supports no-downsampling mode for traffic entities and network alias.
8. Remove all register mechanism and cache in the java agent.
9. Remove MONTH step in GraphQL query.
10. Update UI to remove MONTH step query, the max query range is 60 days now.
11. Simplify the TTL to metrics and record. And the unit has been formatted in Day unit. No specific TTL for ElasticSearch storage.
12. Buffer mechanism of trace receiver and mesh receiver has been removed due to no register.
13. New service id, instance id and endpoint id rules, including service relation, instance relation and endpoint relation id rules.
14. Java agent support `keep tracing` mode, meaning, agent generating tracing context even the backend is unconnected/unavailable.
15. Plugin test tool up to date, in order to support new protocol.
16. Plugin tests expected data files updated.
17. E2E tests updated.
18. Telemetry of Grafana config has been merged into one.
19. Documentation updates.
20. [TBD] InfluxDB storage implementation is not available, need @dmsolr to fix later, in order to reduce the master change block by this PR.
2020-04-10 11:03:24 +08:00