Support Amazon Simple Storage Service (AWS S3) metrics monitoring (#10407)
This commit is contained in:
pg.yang
GitHub
parent
feaf49390c
commit
cabe40eb47
|
|
@ -670,6 +670,8 @@ jobs:
|
|||
config: test/e2e-v2/cases/aws/eks/e2e.yaml
|
||||
- name: Windows
|
||||
config: test/e2e-v2/cases/win/e2e.yaml
|
||||
- name: AWS Cloud S3
|
||||
config: test/e2e-v2/cases/aws/s3/e2e.yaml
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
|
|
|
|||
|
|
@ -95,6 +95,7 @@
|
|||
* Add Python HTTPX module component ID(7019).
|
||||
* Migrate tests from junit 4 to junit 5.
|
||||
* Refactor http-based alarm plugins and extract common logic to `HttpAlarmCallback`.
|
||||
* Support Amazon Simple Storage Service (Amazon S3) metrics monitoring
|
||||
|
||||
#### UI
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,12 @@ Usually, the [AWS CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWat
|
|||
CloudWatch metrics with S3 --> CloudWatch Metric Stream (OpenTelemetry formart) --> Kinesis Data Firehose Delivery Stream --> AWS Firehose receiver(OAP) --> OpenTelemetry receiver(OAP)
|
||||
```
|
||||
|
||||
## Supported metrics
|
||||
|
||||
| Description | Configuration File | Data Source |
|
||||
|-----------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| Metrics of AWS Cloud S3 | otel-rules/aws-s3/s3-service.yaml | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) |
|
||||
|
||||
## Notice
|
||||
|
||||
1. Only OpenTelemetry format is supported (refer to [Metric streams output formats](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats.html))
|
||||
|
|
|
|||
|
|
@ -0,0 +1,37 @@
|
|||
# AWS Cloud S3 monitoring
|
||||
Amazon Simple Storage Service (Amazon S3) is an object storage service. SkyWalking leverages [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md) to transfer the CloudWatch metrics of s3 to
|
||||
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
|
||||
|
||||
### Data flow
|
||||
1. AWS CloudWatch collect metrics for S3, refer to [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html)
|
||||
2. [CloudWatch metric streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html) stream CloudWatch metrics of S3 to AWS Kinesis Data Firehose
|
||||
3. AWS Kinesis Data Firehose delivery metrics to [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md) through the HTTP endpoint
|
||||
|
||||
### Set up
|
||||
1. Create CloudWatch metrics configuration for S3, refer to [S3 metrics configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/metrics-configurations.html)
|
||||
2. Create an Amazon Kinesis Data Firehose Delivery Stream, and set [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md)'s address as HTTP(s) Destination, refer to [Create Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html)
|
||||
3. Create CloudWatch metric stream, and select the Firehose Delivery Stream which has been created above, set `Select namespaces` to `AWS/S3`, `Select output format` to `OpenTelemetry 0.7`. refer to [CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html)
|
||||
|
||||
### S3 Monitoring
|
||||
|
||||
SkyWalking observes CloudWatch metrics of the S3 bucket, which is cataloged as a `LAYER: AWS_S3` `Service` in the OAP.
|
||||
|
||||
#### Supported Metrics
|
||||
|
||||
| Monitoring Panel | Unit | Metric Name | Catalog | Description | Data Source |
|
||||
|----------------------------|-------|-----------------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|
|
||||
| 4xx Errors | count | aws_s3_4xx | Service | The number of HTTP 4xx client error status code requests made to the S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| 5xx Errors | count | aws_s3_5xx | Service | The number of HTTP 5xx client error status code requests made to the S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| Downloaded | bytes | aws_s3_downloaded_bytes | Service | The number of bytes downloaded for requests made to an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| Uploaded | bytes | aws_s3_uploaded_bytes | Service | The number of bytes uploaded for requests made to an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| Request Average Latency | bytes | aws_s3_request_latency | Service | The average of elapsed per-request time from the first byte received to the last byte sent to an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| First Byte Average Latency | bytes | aws_s3_request_latency | Service | The average of per-request time from the complete request being received by an Amazon S3 bucket to when the response starts to be returned | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| All Requests | bytes | aws_s3_delete_requests | Service | The number of HTTP All requests made for objects in an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| Get Requests | bytes | aws_s3_delete_requests | Service | The number of HTTP Get requests made for objects in an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| Put Requests | bytes | aws_s3_delete_requests | Service | The number of HTTP PUT requests made for objects in an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
| Delete Requests | bytes | aws_s3_delete_requests | Service | The number of HTTP Delete requests made for objects in an Amazon S3 bucket | [S3 monitoring with CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) |
|
||||
|
||||
### Customizations
|
||||
You can customize your own metrics/expression/dashboard panel.
|
||||
The metrics definition and expression rules are found in `/config/otel-rules/aws-s3/`.
|
||||
The AWS Cloud EKS dashboard panel configurations are found in `/config/ui-initialized-templates/aws_s3`.
|
||||
|
|
@ -189,6 +189,8 @@ catalog:
|
|||
catalog:
|
||||
- name: "EKS Monitoring"
|
||||
path: "/en/setup/backend/backend-aws-eks-monitoring"
|
||||
- name: "S3 Monitoring"
|
||||
path: "/en/setup/backend/backend-aws-s3-monitoring"
|
||||
- name: "Browser Monitoring"
|
||||
path: "/en/setup/service-agent/browser-agent"
|
||||
- name: "Gateway Monitoring"
|
||||
|
|
|
|||
|
|
@ -146,8 +146,12 @@ public enum Layer {
|
|||
/**
|
||||
* Windows Machine
|
||||
*/
|
||||
OS_WINDOWS(23, true);
|
||||
OS_WINDOWS(23, true),
|
||||
|
||||
/**
|
||||
* Amazon Simple Storage Service (Amazon S3) is an object storage service provided by AWS Cloud
|
||||
*/
|
||||
AWS_S3(24, true);
|
||||
|
||||
private final int value;
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -62,6 +62,7 @@ public class UITemplateInitializer {
|
|||
Layer.VIRTUAL_MQ.name(),
|
||||
Layer.AWS_EKS.name(),
|
||||
Layer.OS_WINDOWS.name(),
|
||||
Layer.AWS_S3.name(),
|
||||
"custom"
|
||||
};
|
||||
private final UITemplateManagementService uiTemplateManagementService;
|
||||
|
|
|
|||
|
|
@ -356,7 +356,7 @@ receiver-otel:
|
|||
selector: ${SW_OTEL_RECEIVER:default}
|
||||
default:
|
||||
enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc,otlp"}
|
||||
enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows"}
|
||||
enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*"}
|
||||
|
||||
receiver-zipkin:
|
||||
selector: ${SW_RECEIVER_ZIPKIN:-}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,55 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# This will parse a textual representation of a duration. The formats
|
||||
# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
|
||||
# with days considered to be exactly 24 hours.
|
||||
# <p>
|
||||
# Examples:
|
||||
# <pre>
|
||||
# "PT20.345S" -- parses as "20.345 seconds"
|
||||
# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds)
|
||||
# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds)
|
||||
# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
|
||||
# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes"
|
||||
# "P-6H3M" -- parses as "-6 hours and +3 minutes"
|
||||
# "-P6H3M" -- parses as "-6 hours and -3 minutes"
|
||||
# "-P-6H+3M" -- parses as "+6 hours and -3 minutes"
|
||||
# </pre>
|
||||
|
||||
filter: "{ tags -> {tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/S3' } }"
|
||||
expSuffix: tag({tags -> tags.bucket = 'aws-s3::' + tags.BucketName}).service(['bucket'], Layer.AWS_S3)
|
||||
metricPrefix: aws_s3
|
||||
metricsRules:
|
||||
- name: 4xx
|
||||
exp: amazonaws_com_AWS_S3_4xxErrors_sum.downsampling(SUM)
|
||||
- name: 5xx
|
||||
exp: amazonaws_com_AWS_S3_5xxErrors_sum.downsampling(SUM)
|
||||
- name: downloaded_bytes
|
||||
exp: amazonaws_com_AWS_S3_BytesDownloaded_sum.downsampling(SUM)
|
||||
- name: uploaded_bytes
|
||||
exp: amazonaws_com_AWS_S3_BytesUploaded_sum.downsampling(SUM)
|
||||
- name: request_latency
|
||||
exp: amazonaws_com_AWS_S3_TotalRequestLatency_sum.div(amazonaws_com_AWS_S3_TotalRequestLatency_count)
|
||||
- name: first_latency_bytes
|
||||
exp: amazonaws_com_AWS_S3_FirstByteLatency_sum.div(amazonaws_com_AWS_S3_FirstByteLatency_count)
|
||||
- name: all_requests
|
||||
exp: amazonaws_com_AWS_S3_AllRequests_sum.downsampling(SUM)
|
||||
- name: put_requests
|
||||
exp: amazonaws_com_AWS_S3_PutRequests_sum.downsampling(SUM)
|
||||
- name: get_requests
|
||||
exp: amazonaws_com_AWS_S3_GetRequests_sum.downsampling(SUM)
|
||||
- name: delete_requests
|
||||
exp: amazonaws_com_AWS_S3_DeleteRequests_sum.downsampling(SUM)
|
||||
|
|
@ -50,7 +50,7 @@
|
|||
"graph": {
|
||||
"fontColor":"blue",
|
||||
"backgroundColor":"white",
|
||||
"content":"Provide AWS Could monitoring through AWS Container Insights Receiver",
|
||||
"content":"Provide AWS Cloud EKS monitoring through AWS Container Insights Receiver",
|
||||
"fontSize":14,
|
||||
"textAlign":"left",
|
||||
"url":"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-aws-eks-monitoring/"
|
||||
|
|
|
|||
|
|
@ -0,0 +1,68 @@
|
|||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
[
|
||||
{
|
||||
"id": "AWS_S3-Root",
|
||||
"configuration": {
|
||||
"children": [
|
||||
{
|
||||
"x": 0,
|
||||
"y": 2,
|
||||
"w": 24,
|
||||
"h": 51,
|
||||
"i": "0",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": ""
|
||||
},
|
||||
"graph": {
|
||||
"type": "ServiceList",
|
||||
"dashboardName": "AWS_S3-service",
|
||||
"fontSize": 12,
|
||||
"showXAxis": false,
|
||||
"showYAxis": false,
|
||||
"showGroup": true
|
||||
},
|
||||
"moved": false
|
||||
},
|
||||
{
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 24,
|
||||
"h": 2,
|
||||
"i": "100",
|
||||
"type": "Text",
|
||||
"graph": {
|
||||
"fontColor":"blue",
|
||||
"backgroundColor":"white",
|
||||
"content":"Provide AWS Cloud S3 monitoring through AWS FireHose Receiver",
|
||||
"fontSize":14,
|
||||
"textAlign":"left",
|
||||
"url":"https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-aws-s3-monitoring/"
|
||||
},
|
||||
"moved": false
|
||||
}
|
||||
],
|
||||
"id": "AWS_S3-Root",
|
||||
"layer": "AWS_S3",
|
||||
"entity": "All",
|
||||
"name": "AWS_S3-Root",
|
||||
"isRoot": true
|
||||
}
|
||||
}
|
||||
]
|
||||
|
|
@ -0,0 +1,399 @@
|
|||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
[
|
||||
{
|
||||
"id": "AWS_S3-service",
|
||||
"configuration": {
|
||||
"children": [
|
||||
{
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 6,
|
||||
"h": 5,
|
||||
"i": "1",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "4xx Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Card",
|
||||
"fontSize": 14,
|
||||
"textAlign": "center",
|
||||
"showUnit": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_4xx"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValue"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 6,
|
||||
"y": 0,
|
||||
"w": 6,
|
||||
"h": 5,
|
||||
"i": "2",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "5xx Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Card",
|
||||
"fontSize": 14,
|
||||
"textAlign": "center",
|
||||
"showUnit": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_4xx"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValue"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 12,
|
||||
"y": 0,
|
||||
"w": 6,
|
||||
"h": 5,
|
||||
"i": "3",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "All Request Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Card",
|
||||
"fontSize": 14,
|
||||
"textAlign": "center",
|
||||
"showUnit": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_all_requests"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValue"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 18,
|
||||
"y": 0,
|
||||
"w": 6,
|
||||
"h": 5,
|
||||
"i": "4",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "Request Average Latency (ms)"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Card",
|
||||
"fontSize": 14,
|
||||
"textAlign": "center",
|
||||
"showUnit": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_request_latency"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValue"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 0,
|
||||
"y": 5,
|
||||
"w": 6,
|
||||
"h": 13,
|
||||
"i": "5",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "4xx Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_4xx"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 6,
|
||||
"y": 5,
|
||||
"w": 6,
|
||||
"h": 13,
|
||||
"i": "6",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "5xx Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_4xx"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 12,
|
||||
"y": 5,
|
||||
"w": 6,
|
||||
"h": 13,
|
||||
"i": "7",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "Request Average Latency (ms)"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_request_latency"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 18,
|
||||
"y": 5,
|
||||
"w": 6,
|
||||
"h": 13,
|
||||
"i": "8",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "First Byte Average Latency (ms)"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_first_latency_bytes"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 0,
|
||||
"y": 18,
|
||||
"w": 6,
|
||||
"h": 13,
|
||||
"i": "9",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "All Request Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_all_requests"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 6,
|
||||
"y": 18,
|
||||
"w": 6,
|
||||
"h": 13,
|
||||
"i": "10",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "Get Request Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_get_requests"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 12,
|
||||
"y": 18,
|
||||
"w": 6,
|
||||
"h": 13,
|
||||
"i": "11",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "Put Request Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_put_requests"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 18,
|
||||
"y": 18,
|
||||
"w": 6,
|
||||
"h": 13,
|
||||
"i": "12",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "Delete Request Count"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_delete_requests"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 0,
|
||||
"y": 41,
|
||||
"w": 12,
|
||||
"h": 13,
|
||||
"i": "13",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "Downloaded (KB)"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true,
|
||||
"legend": {
|
||||
"asTable": true,
|
||||
"toTheRight": true,
|
||||
"min": true,
|
||||
"max": true,
|
||||
"mean": true,
|
||||
"total": true
|
||||
}
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_downloaded_bytes"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
],
|
||||
"metricConfig": [
|
||||
{
|
||||
"calculation": "byteToKB"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"x": 12,
|
||||
"y": 41,
|
||||
"w": 12,
|
||||
"h": 13,
|
||||
"i": "14",
|
||||
"type": "Widget",
|
||||
"widget": {
|
||||
"title": "Uploaded (KB)"
|
||||
},
|
||||
"graph": {
|
||||
"type": "Line",
|
||||
"step": false,
|
||||
"smooth": false,
|
||||
"showSymbol": false,
|
||||
"showXAxis": true,
|
||||
"showYAxis": true,
|
||||
"legend": {
|
||||
"asTable": true,
|
||||
"toTheRight": true,
|
||||
"min": true,
|
||||
"max": true,
|
||||
"mean": true,
|
||||
"total": true
|
||||
}
|
||||
},
|
||||
"metrics": [
|
||||
"aws_s3_uploaded_bytes"
|
||||
],
|
||||
"metricTypes": [
|
||||
"readMetricsValues"
|
||||
],
|
||||
"metricConfig": [
|
||||
{
|
||||
"calculation": "byteToKB"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"layer": "AWS_S3",
|
||||
"entity": "Service",
|
||||
"name": "AWS_S3-service",
|
||||
"id": "AWS_S3-service",
|
||||
"isRoot": false
|
||||
}
|
||||
}
|
||||
]
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
version: '2.1'
|
||||
|
||||
services:
|
||||
oap:
|
||||
extends:
|
||||
file: ../../../script/docker-compose/base-compose.yml
|
||||
service: oap
|
||||
ports:
|
||||
- 12800
|
||||
networks:
|
||||
e2e:
|
||||
|
||||
sender:
|
||||
image: "eclipse-temurin:8-jre"
|
||||
volumes:
|
||||
- ./../../../java-test-service/e2e-mock-sender/target/e2e-mock-sender-2.0.0.jar:/e2e-mock-sender-2.0.0.jar
|
||||
- ./mock-data:/data/otel-metrics
|
||||
command: [ "java", "-jar", "/e2e-mock-sender-2.0.0.jar" ]
|
||||
environment:
|
||||
OAP_HOST: oap
|
||||
OAP_GRPC_PORT: 11800
|
||||
OTEL_METRICS_DATA_PATH: /data/otel-metrics
|
||||
networks:
|
||||
- e2e
|
||||
ports:
|
||||
- 9093:9093
|
||||
healthcheck:
|
||||
test: ["CMD", "sh", "-c", "nc -nz 127.0.0.1 9093"]
|
||||
interval: 5s
|
||||
timeout: 60s
|
||||
retries: 120
|
||||
depends_on:
|
||||
oap:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
e2e:
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# This file is used to show how to write configuration files and can be used to test.
|
||||
|
||||
setup:
|
||||
env: compose
|
||||
file: docker-compose.yml
|
||||
timeout: 20m
|
||||
init-system-environment: ../../../script/env
|
||||
steps:
|
||||
- name: set PATH
|
||||
command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
|
||||
- name: install yq
|
||||
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
|
||||
- name: install swctl
|
||||
command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
|
||||
|
||||
trigger:
|
||||
action: http
|
||||
interval: 3s
|
||||
times: 10
|
||||
url: http://localhost:9093/otel-metrics/send
|
||||
method: GET
|
||||
|
||||
verify:
|
||||
retry:
|
||||
count: 10
|
||||
interval: 3s
|
||||
cases:
|
||||
- includes:
|
||||
- ./s3-cases.yaml
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
{{- contains . }}
|
||||
- key: {{ notEmpty .key }}
|
||||
value: {{ ge .value 0 }}
|
||||
{{- end }}
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
{{- contains . }}
|
||||
- id: {{ b64enc "aws-s3::skywalking" }}.1
|
||||
name: aws-s3::skywalking
|
||||
group: aws-s3
|
||||
shortname: skywalking
|
||||
normal: true
|
||||
layers:
|
||||
- AWS_S3
|
||||
{{- end }}
|
||||
|
|
@ -0,0 +1,132 @@
|
|||
{
|
||||
"resourceMetrics": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "cloud.provider",
|
||||
"value": {
|
||||
"stringValue": "aws"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "cloud.account.id",
|
||||
"value": {
|
||||
"stringValue": "xxxxxxxx"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "cloud.region",
|
||||
"value": {
|
||||
"stringValue": "ap-northeast-1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "aws.exporter.arn",
|
||||
"value": {
|
||||
"stringValue": "arn:aws:cloudwatch:ap-northeast-1:xxxxxxxx:metric-stream/CustomFull-CmHV2P"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeMetrics": [
|
||||
{
|
||||
"metrics": [
|
||||
{
|
||||
"name": "amazonaws.com/AWS/S3/4xxErrors",
|
||||
"unit": "{Count}",
|
||||
"summary": {
|
||||
"dataPoints": [
|
||||
{
|
||||
"startTimeUnixNano": "1674547500000000000",
|
||||
"timeUnixNano": "1674547560000000000",
|
||||
"count": "1",
|
||||
"quantileValues": [
|
||||
{
|
||||
},
|
||||
{
|
||||
"quantile": 1.0
|
||||
}
|
||||
],
|
||||
"attributes": [
|
||||
{
|
||||
"key": "Namespace",
|
||||
"value": {
|
||||
"stringValue": "AWS/S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "MetricName",
|
||||
"value": {
|
||||
"stringValue": "4xxErrors"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "BucketName",
|
||||
"value": {
|
||||
"stringValue": "skywalking"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "FilterId",
|
||||
"value": {
|
||||
"stringValue": "test"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "amazonaws.com/AWS/S3/5xxErrors",
|
||||
"unit": "{Count}",
|
||||
"summary": {
|
||||
"dataPoints": [
|
||||
{
|
||||
"startTimeUnixNano": "1674547500000000000",
|
||||
"timeUnixNano": "1674547560000000000",
|
||||
"count": "1",
|
||||
"quantileValues": [
|
||||
{
|
||||
},
|
||||
{
|
||||
"quantile": 1.0
|
||||
}
|
||||
],
|
||||
"attributes": [
|
||||
{
|
||||
"key": "Namespace",
|
||||
"value": {
|
||||
"stringValue": "AWS/S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "MetricName",
|
||||
"value": {
|
||||
"stringValue": "5xxErrors"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "BucketName",
|
||||
"value": {
|
||||
"stringValue": "skywalking"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "FilterId",
|
||||
"value": {
|
||||
"stringValue": "test"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# This file is used to show how to write configuration files and can be used to test.
|
||||
|
||||
cases:
|
||||
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls
|
||||
expected: expected/service.yml
|
||||
- query: |
|
||||
swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \
|
||||
--name=aws_s3_4xx --service-name aws-s3::skywalking |yq e 'to_entries' -
|
||||
expected: expected/metrics-has-value.yml
|
||||
- query: |
|
||||
swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear \
|
||||
--name=aws_s3_5xx --service-name aws-s3::skywalking |yq e 'to_entries' -
|
||||
expected: expected/metrics-has-value.yml
|
||||
Loading…
Reference in New Issue