* Format index name with namespace
Signed-off-by: Gao Hongtao <hanahmily@gmail.com>
* Use another variable to compare index name
Signed-off-by: Gao Hongtao <hanahmily@gmail.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
1. Service Traffic should serialize and deserialize the time bucket, otherwise, there is a chance service_traffic-0 generated.
2. Make top N results more clear about the owner. Previously, the same endpoint names of different services look the same.
3. Sync UI.
4. Fix service traffic equal and hashcode method. The time bucket is not a part of the entity attribute.
* Add Mariadb JDBC plugin and provide tests
* Refactor component name
* Reformat plugin description and remove healthcheck span from the test
Co-authored-by: Efe Kaptan <Efe.Kaptan@webtrekk.com>
Co-authored-by: zhang-wei <pknfe@outlook.com>
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
* Update nacos version to 1.2.0(Solve the security problem of the old version of fastsjon)
Co-authored-by: songzhendong <289505773@qq.com>
Co-authored-by: songzhendong <songzhendong@xiaomi.com>
Here are the basic and simple usages of the MeterSystem APIs.
## Meter Creation
New meter could be created based on metrics name, function, scope and data type.
1. Metrics name is still the storage entity name.
2. Functions are every similar with the OAL function, just accepting different input.
```java
final MeterSystem meterSystem = MeterSystem.meterSystem(getManager());
meterSystem.create("test_long_metrics", "avg", ScopeType.SERVICE, Long.class);
```
NOTICE, this creation should only be called in the `module#prepare` stage, otherwise, `Can't create new metrics anymore` exception will be raised after the **CORE** module `start` stage finished. You may find out, there is a chance `create` could be executed successfully in your own `module#start` stage, but it it just because of the sequence of provider loaded by the class loader, **no guarantee, so please don't do that**.
## Runtime Calculation
`AcceptableValue` is the object created at the runtime to accept new metrics value.
```java
final MeterSystem service = getManager().find(CoreModule.NAME).provider().getService(MeterSystem.class);
Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
final AcceptableValue<Long> value = service.buildMetrics("test_long_metrics", Long.class);
value.accept(MeterEntity.newService("abc"), 5L);
value.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis()));
service.doStreamingCalculation(value);
}
}, 2, 2, TimeUnit.SECONDS);
```
## Meter Functions
Right now, only `avg` function has been implemented. I submit this PR as soon as possible to get your feedback. I will add more functions.
## Notice
1. Make slow trace query available in the sampled record. Logically, they are the same thing. The UI doesn't need to concern about the trace as a special case.
2. Endpoint dependency will be removed from the dashboard. Because no query available for it. We will provide a new page for that in 8.1
3. Comparison page will be removed due to dashboard is powerful enough to replace it.
* Add data type to the column definition to replace the simple isValue definition. And top N query is also requiring value column now.
* Support typeOfMetrics query.
1. Support the new query protocol, and the v1 query protocol is still supported
2. All `numOfxxx`/`GlobalBrief` is only a mock now, no real number.
3. typeOfMetrics service is a mock only too, follow up PR will implement this.
* Keep latest index always present.
* Avoid TTL timer to remove latest index
* Create a latest index even without any data.
Signed-off-by: Gao Hongtao <hanahmily@gmail.com>
Here are the list of changes we made in this pull request. Top 3 are the principle changes, others are following these principles.
1. New agent and mesh report protocol.
2. New agent header protocol.
3. Service register, instance register and network address register have been removed permanently.
4. Service traffic, instance traffic and network alias metrics are added to replace the service, instance and network address inventory.
5. Register process has been removed.
6. Metrics stream process supports insert only mode, especially for traffic entities.
7. Metrics stream process supports no-downsampling mode for traffic entities and network alias.
8. Remove all register mechanism and cache in the java agent.
9. Remove MONTH step in GraphQL query.
10. Update UI to remove MONTH step query, the max query range is 60 days now.
11. Simplify the TTL to metrics and record. And the unit has been formatted in Day unit. No specific TTL for ElasticSearch storage.
12. Buffer mechanism of trace receiver and mesh receiver has been removed due to no register.
13. New service id, instance id and endpoint id rules, including service relation, instance relation and endpoint relation id rules.
14. Java agent support `keep tracing` mode, meaning, agent generating tracing context even the backend is unconnected/unavailable.
15. Plugin test tool up to date, in order to support new protocol.
16. Plugin tests expected data files updated.
17. E2E tests updated.
18. Telemetry of Grafana config has been merged into one.
19. Documentation updates.
20. [TBD] InfluxDB storage implementation is not available, need @dmsolr to fix later, in order to reduce the master change block by this PR.
* Set the endpoint name length to 150, to keep safe and avoid over-length.
* Set endpointNameMaxLength in the application.yml
* Fix a mischange.
* Correct the endpoint name length control logic
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
- Add length definition with reasonable default value.
- #content of @Column has been renamed to storageOnly . I add this to many fields as they are not being query in any case.
- Merge H2 and MySQL columntype mapping back to consistent.
- Remove @IDColumn.
- Support @QueryUnifiedIndex.
- Refactor the MySQL and H2 installers to use @Column and @QueryUnifiedIndex definitions to create indices automatically. But the index naming rule has been changed to entityseqIDX. seq is the Increment Interger for every entity.
- Support @MetricsExtension and insertOnly in the MetricsPersistentWorker worker.
- Optimize MetricsStreamProcessor
* Finish step one, source and entity changed.
* Step 2, finish the basic and core DAO changes.
* Step 3. Change all source codes of backend, and make project compiling successfully.
* Make startup successful and persistence works.
* Fix dead codes in the MySQLTableInstaller.
* Fix mischange.
* Fix MySQL storage bootstrap error.
* Rollback dependency change.
* Simply enable tailing logs from container no mater it's in CI or not
* Fix segment table issue.
* Fix MySQL Instanller.
* Support Influxdb implementation of new endpoint traffic.
* Set and modify timeout to 90mins
* Remove unnecessary entity in the EndpointTraffic.
* Fix wrong entity builder and make name more clear.
* Fix 2 equal methods bug.
* Fix InfluxDB identifier(name) conflict issue.
* Fix endpoint topology query, endpoint id should be changed as a string.
* Fix a missing change.
* Fix a wrong commit.
* Column name can't be changed, due to it affect the hybrid metadb(h2/mysql) query, change it explicitly with double quotation marks.
* Fix formats.
* Remove endpoint/operation register from the agent. and change the plugin tests(without local tests). operationId relation codes have been removed from agent too.
* Try to fix the hytrix case.
* Fix 3 cases.
* Try to fix cases.
* Fix 2.
* Fix one.
* Fix the serviceId type mis-change.
* Fix int query
* Fix hystrix case.
* Fix a case.
* Fix EndpointCallRelationDispatcher set the source/dest endpoint without id encoding.
* Update documentation and add 2 test cases.
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
### Motivation
Ease the efforts of modifying configurations in automatic workflow and scripts, like Docker and helm chart.
### Modification
Add a `selector` property into `application.yml` to enable switching configuration simply by environment variables or system properties, and filter the unselected options, leaving only selected one.
### Result
- We can switch a configuration by environment variables or system property
- Closes#4511
* Support dynamic column in the source.
* Support activeExtraModelColumns as a default OFF option.
* Fix words
* Update backend-setup.md
Fix format issue.
* 1. Format codes
2. Add comments for storage interface of topology
3. Narrow the scope of secondary query to get the component name. Only query the nodes as client out of the given service Ids.
* Support topology with multiple selected services
* Sync the typo fix in the protocol
### Motivation:
Add test to guarantee that the components defined in class ComponentsDefine are in sync with those in file component-libraries.yml.
### Modifications:
- Add missing component in component-libraries.yml.
- Add test to verify the class and the yml are in sync.
- Fix wrong component id of Undertow.
### Result:
Components definitions in class ComponentsDefine and yaml are in sync, and will be verified in every pull request.
* extract duplicated method
* change while loop to if since once rs.next() return false, it will never return true for the following calls
* simplify if() return true; return false pattern
* fix compilation
Signed-off-by: liqiang.clq <liqiang.clq@alibaba-inc.com>
Motivation:
Review code styles with some bots automatically.
Modifications:
Set up ReviewDog in GitHub Action to review code style.
Add more check rules to checkstyle plugin.
Result:
Obvious code styles can be reviewed and commented automatically.
* Add lombok dependency into root pom.xml
* Update all depending lombok pom.xml(s)
* Change lombok version into latest(1.18.10)
* Revert lombok dependency config of test projects
* Remove redundant lombok dependency of submodule
* Move Lombok into dependencies node for applying globally
* sniffer processing profile task and report status and snapshot
* resolve testServiceDependencies test case error, use same register with `TraceSegmentServiceClient`
* resolve names
* change profile to single one thread run.
* 1. change to the ArrayList, because known the max size
2. rename issue resolved
* add profiling status enum
* change sniffer use full name issue
* 1. remove `prepareProfiling` method, build profiling status when construct `TracingContext`
2. add `TracingThreadListenerManager`, notify when tracing main thread finish
3. change ProfileThread start when process new profile task
* remove unnecessary getter
* add test assert error message
* adding `AgentServiceRule`
* revert original assert
* remove unnecessary getter
* resolve issues
* reduce findService invoke
* resolve style error
* recheck profiling when change first span operatin name
* resolve issues
1. remove `ContextManager#profilingRecheck`, only check on `TracingContext`
2. rename comments
3. resolve volatile array setting
* remove article link
* add `ProfileTask#maxSamplingCount` check
* resolve conflict (Downstream -> Commands)
* 1. change profilingSegmentSlots init on construct
2. if is profiling, recheck dont need to stop
3. total profiling count increment on first dump
* remove unused return val
* remove some `@param` and `@return`
* add profile task check result data bean
* change profiler slot to `AtomicReferenceArray`
* resolved java doc error
* fix doc error, remove meaningless descriptions
* resolve missed profile receiver on oap starter
* resolve method invoke error
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
* Support new percentile func with new alarm and exporter for this new func.
* Fix e2e and OAL script
* Fix wrong column.
* Fix percentile bug and oal engine bug.
* Update query protocol and add percentile test case
* Support new query
* Adopt GraphQL requirement
* Fix wrong type cast.
* Fix query in H2 and ES.
* Fix docs and comments.
* Fix an e2e compile issue
* Fix javadoc issue and e2e test issue.
* Change CPM to Apdex in TTL test.
* Fix OAL for TTL e2e
* Add metrics query for service percentile.
* Fix OAL engine bug. Method deserialize is not working when more than two field types are IntKeyLongValueHashMap
* Support multiple IntKeyLongValueHashMap fields in remote. About serialize/deserialize methods.
* Fix graphql statement error in e2e.
* Fix serialize not working and add generated serialize/deserialize of percentile into test cases.
* Fix test case format
* Remove generated code test.
* Fix failed e2e test
* Use avg resp time to apdex in the TTL test.
* ADD multiple linear metrics check for endpoint in e2e cluster.
* Support `-` to represent no threshold and doc of alarm about this.
* Move break to right place.
* Fix wrong break(s)
* Fix break and add a test case for multiple values alarm.
* Fix format.
* Add more doc for this new feature and GraphQL query protocol.
Co-authored-by: Jared Tan <jian.tan@daocloud.io>
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
* Provide profile task downstream to sniffer
* fix agent unit testcase issue
* add profile switch config on sniffer
* fix es error
* 1. add @DefaultImplementor on the sniffer profile task service
2. change ProfileTaskExecutionService#PROFILE_TASK_READY_SCHEDULE to final and remove volatile
2. fix style error
* change timeFromStartMills use `<` to compare
* 1. add `maxSamplingCount` to profile task
2. make profile task limit to the common package
* 1. change `agent.active_profile` to `profile.active` and make true on default
2. add `maxSamplingCount` in profile task
3. use `createTime` to check has new command list
4. add task re-check before process profile task
* 1. add `profile-receiver` document
2. change `ProfileTaskExecutionService` use single schedule thread pool
3. cache dont need fetch data when no data, use auto-fresh mechanism only
* remove navigate time judge, fix comment wrong meaning
* Add service type from register and support browser as service type
* Add service type at agent side.
* Fix format.
* Fix a test.
* Fix formats.
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
* Update JDK compiling level to 8, and remove zip dist.
* Exclude annotation and shade API.
* Fix more exclusion.
* 1. Update grava and gson 2. Fix compile issue. 3. Keep protoc and maven plugin in old version. Some incompatible of 3rd party grpc proto.
* Fix dependency conflicts, license and notice issue, and dependency check script issue
Co-authored-by: kezhenxu94 <kezhenxu94@163.com>
* Remove the local span and exit span register mechanism in Java agent scenario.
* Fix config, plugin, and document, as exit span is not required register anymore.
* Try to fix the test case expected data due to no register for exit span.
* Fix toolkit case.
* Remove exit span from plugin test framework document.
* Update protocol, FAQ, and CHANGELOG documents to highlight this change.
* Fix NPE when no entry span in the whole segment. Set endpoint to null in segment entity.
* Fix the snapshot and add inexistence operation name id.
* Fix several async cases.
* Fix undertow test case.
* Fix wrong error log name.
* Keep name consistent with 6.x concepts.
* Try to fix 3 async scenarios.
* Fix okhttp case.
* Fix Spring Async case.
* Fix another NPE
* Fix NPE when first span has no endpoint id when it is local or exit span.
* Still fill first operation name when no endpoint id.
* Add apdex function to OAL
* Add empty line
* Setup config watcher
* Add identifier type to function parameter
* Add config test
* Update score algorithm
* Replace responseCode with status
* Add comments about apdex score algorithm
* Add docs
* Add e2e test case
* Update test case
* Fix disptch class generating error
* Update value name of apdex metric
* Tuning threshold
* Fix single tolerated point bug
* [Feature] Officially support ElasticSearch 7 as backend storage
* Extract bootstrap module to be shared by multiple starters
* Add missing configuration file and distinguish different version in test
* Update known-oap-backend-dependencies-es7.txt
Update the dependency lib
* Fix missing adaptation to ElasticSearch 7
* Rename ElasticSearch 7 specific class with Es7 infix
* Fix miss-adapted ElasticSearch DAO API
* Add ES7 specific configuration and polish documentations
* Polish documentations
* Polish documentations
* Update EtcdCoordinator.java
when EtcdCoordinator cannot connect to etcd cluster for a long time,
the information registered in etcd will expire and refresh() function will not work.
* Add register persistent worker latency metrics #3671
* remove never thrown exception on Time#close()
add panel to mesh-mode-grafana.json
* remove never thrown exception on Time#close()
add panel to mesh-mode-grafana.json
* add plugin netty-socketio https://github.com/mrniko/netty-socketio
* delete create exit span interceptor.
* delete create exit span interceptor testcase.
* delete create exit span interceptor testcase.
* add constructor test unit
* change bind on random port in test case
* reverse upstream master whitespace
* resolve build_id param
* delete health check segment test case
* add ContextManager.isActive check before use activeSpan invoke
* remove unnecessary code and blank line
* Fix#3402
* Fix unecessary blankline
* Optimize rebuild logic
* Fix checkstyle fail
* Add concurrency test
* Fix text mistakes
* Optimize refresh logic && new test for return clients never change by refresh logic
* Make unit test more clearness
* Add more unit test && fix a bug
* Rename variable && format by skywalking style
* Is it a typo or not?
* delete datasource-settings.properties
* handle 'metadataQueryMaxSize'
* delete DynamicModuleConfig.java
* modify mysql config
* parsing yaml to support the configuration for multiple properties
* delete datasource-settings.properties
* modify doc
* add properties for docker-entrypoint.sh
* add testLoadConfig()
* Provide Consul dynamic configuration center implementation
* Add more unit tests
* Fix unit test
* Add more unit tests and remove unused file
* Remove redundant doc
* Add pulsar apm plugin.
* Fix check style
* Fix pulsar consumer component define.
* Add pulsar to component-libraries.yml
* Fix error interceptor class.
* Add pulsar to agent support list.
* Add Pulsar to ComponentsDefine and component-libraries.yml
* Move create entry span log of consumer from after method to before method
* Fix send callback issue when exception cause.
* Fix test issues
* Move pulsar plugin to optional plugins
* Add none messages tests for interceptor of producer and consumer.
* Remove unused comments.
* Move pulsar plugin back to the apm-sdk-plugin
* Fix comments
* remove set startTime for entry span(default is set by System.currentTimeMillis)
* Fix comments
* Add Light4J 1.6.x plug-in
* Add unit test for Light4J plug-in
* Add missing id to server-starter
* Remove mentions of version 1.6.x
Some local testing showed that the plugin successfully generated traces for a light4j project built with light-codegen v2.x. Therefore, there is no need for a separate plugin for v2.x and no distinction needs to be made with light4j v1.6.x.
* Update supported light4j versions
* Use ContextSnapshot to maintain trace
In a typical Light4j project, the ExceptionHandler#handleRequest is the asynchronous execution point. The flow is ExceptionHandler#handleRequest -> HttpServerExchange#dispatch -> ... -> ExceptionHandler#handleRequest. This means that handleRequest is being intercepted twice and each time a LocalSpan is created. Without handling thread propagation, the trace segment gets split up.
By saving a ContextSnapshot into the enhanced object (ExceptionHandler), we can determine if it is necessary to continue the segment.
* Update logic for span creation in trace
By default, the Light4J plugin now overrides the Undertow plugin's entry span with its own. This new entry span is created just before execution of the ExceptionHandler#handleRequest in the I/O thread. The entry span is generated here because it is considered the starting point of the Light4J handler chain.
There is now also the option to enable detailed span creation when tracing requests through Light4J. These local spans mark which middleware and business handlers are involved with handling the HTTP request to a Light4J service.
* Update skywalking version
* Update light4j version
* Fix LightInstrumentation javadoc
* Remove import violating checkstyle
This import was added from the link reference in the javadoc. The javadoc has now been changed to just reference the method name.
* Move light4j agent config property
* Sync UI
* Modify the thread pool of grpc server.
* add license header
* make param confingable
* Add sharing and Jaeger config.
* Remove grpc thread config from yml
吴晟 Wu Sheng
Gao Hongtao
Efe Kaptan
songzhendong
kezhenxu94
Jared Tan
mrproliu
Daming
zhang-wei
何延龙
Mikael Schirén
yoje
Weiyi Liu
aderm
cui-liqiang
John Niang
Liu Hao
web-xiaxia
Juan Pan(Trista)
Stalary
于玉桔
surechen
YuXiaoFeng
qiyang
coki230
Kdump
Yanick.xia
echooymxq
aderm
lipenghui
Louis Tsui
stone.wlg
康智冬
Andy Ai
彭勇升 pengys
webb2019
yantaowu
li.can