diff --git a/.github/workflows/skywalking.yaml b/.github/workflows/skywalking.yaml index 989404e372..25613873e6 100644 --- a/.github/workflows/skywalking.yaml +++ b/.github/workflows/skywalking.yaml @@ -509,6 +509,8 @@ jobs: config: test/e2e-v2/cases/vm/prometheus-node-exporter/e2e.yaml - name: So11y config: test/e2e-v2/cases/so11y/e2e.yaml + - name: MySQL Prometheus + config: test/e2e-v2/cases/mysql/prometheus-mysql-exporter/e2e.yaml - name: Rover with Istio Process 1.13.1 config: test/e2e-v2/cases/rover/process/istio/e2e.yaml diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index bb5f385d29..9fba81ce4a 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -47,6 +47,7 @@ * Fix Zipkin trace query the max size of spans. * Add `tls` and `https` component IDs for Network Profiling. * Support Elasticsearch column alias for the compatibility between storage logicSharding model and no-logicSharding model. +* Support MySQL monitoring. #### UI diff --git a/docs/en/setup/backend/backend-mysql-monitoring.md b/docs/en/setup/backend/backend-mysql-monitoring.md new file mode 100644 index 0000000000..d4e5fb9e81 --- /dev/null +++ b/docs/en/setup/backend/backend-mysql-monitoring.md @@ -0,0 +1,39 @@ +# MySQL monitoring +SkyWalking leverages prometheus/mysqld_exporter for collecting metrics data from MySQL. It leverages OpenTelemetry Collector to transfer the metrics to +[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md). + +## Data flow +1. mysqld_exporter collect metrics data from MySQL. +2. OpenTelemetry Collector fetches metrics from mysqld_exporter via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via the OpenCensus GRPC Exporter. +3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results. + +## Set up +1. Set up [mysqld_exporter](https://github.com/prometheus/mysqld_exporter#using-docker). +2. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#docker). For details on Prometheus Receiver in OpenTelemetry Collector, refer to [here](../../../../test/e2e-v2/cases/mysql/prometheus-mysql-exporter/otel-collector-config.yaml). +3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md). + +## MySQL Monitoring +MySQL monitoring provides monitoring of the status and resources of the MySQL server. MySQL server as a `Service` in OAP, and land on the `Layer: MYSQL`. + +### MySQL Supported Metrics +| Monitoring Panel | Unit | Metric Name | Description | Data Source | +|-----|------|-----|-----|-----| +| MySQL Uptime | day | meter_mysql_uptime | The MySQL startup time | mysqld_exporter| +| Max Connections | | meter_mysql_max_connections | The max number of connections. | mysqld_exporter| +| Innodb Buffer Pool Size | MB | meter_mysql_innodb_buffer_pool_size | The buffer pool size in Innodb engine | mysqld_exporter| +| Thread Cache Size | | meter_mysql_thread_cache_size | The size of thread cache | mysqld_exporter| +| Current QPS| | meter_mysql_qps | Queries Per Second | mysqld_exporter| +| Current TPS | | meter_mysql_tps | Transactions Per Second | mysqld_exporter| +| Commands Rate | | meter_mysql_commands_insert_rate
meter_mysql_commands_select_rate
meter_mysql_commands_delete_rate
meter_mysql_commands_update_rate | The rate of total number of insert/select/delete/update executed by the current server | mysqld_exporter| +| Threads | | meter_mysql_threads_connected
meter_mysql_threads_created
meter_mysql_threads_cached
meter_mysql_threads_running | The number of currently open connections(threads_connected)
The number of threads created(threads_created)
The number of threads in the thread cache(threads_cached)
The number of threads that are not sleeping(threads_running) | mysqld_exporter| +| Connects | | meter_mysql_connects_available
meter_mysql_connects_aborted | The number of available connections(connects_available)
The number of MySQL instance connection rejections(connects_aborted)| mysqld_exporter| +| Connection Errors | | meter_mysql_connection_errors_internal
meter_mysql_connection_errors_max_connections | Errors due to exceeding the max_connections(connection_errors_max_connections)
Error caused by internal system(connection_errors_internal) | mysqld_exporter| +| Slow Queries Rate | | meter_mysql_slow_queries_rate | The rate of slow queries | mysqld_exporter| + + +## Customizations +You can customize your own metrics/expression/dashboard panel. +The metrics definition and expression rules are found in `/config/otel-oc-rules/mysql.yaml`. +The MySQL dashboard panel configurations are found in `/config/ui-initialized-templates/mysql`. + + diff --git a/docs/en/setup/backend/opentelemetry-receiver.md b/docs/en/setup/backend/opentelemetry-receiver.md index b6b3c58648..b92c7722fe 100644 --- a/docs/en/setup/backend/opentelemetry-receiver.md +++ b/docs/en/setup/backend/opentelemetry-receiver.md @@ -35,5 +35,6 @@ for identification of the metric data. |k8s-cluster| Metrics of K8s cluster | otel-oc-rules/k8s-cluster.yaml | K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server | |k8s-node| Metrics of K8s cluster | otel-oc-rules/k8s-node.yaml | cAdvisor & K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server | |k8s-service| Metrics of K8s cluster | otel-oc-rules/k8s-service.yaml | cAdvisor & K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server | +|mysql| Metrics of MYSQL| otel-oc-rules/mysql.yaml | prometheus/mysqld_exporter -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server | **Note**: You can also use OpenTelemetry exporter to transport the metrics to SkyWalking OAP directly. See [OpenTelemetry Exporter](./backend-meter.md#opentelemetry-exporter). diff --git a/docs/menu.yml b/docs/menu.yml index 33fc87253b..95119e75f7 100644 --- a/docs/menu.yml +++ b/docs/menu.yml @@ -153,6 +153,8 @@ catalog: path: "/en/setup/service-agent/server-agents" - name: "Compatibility" path: "/en/setup/service-agent/agent-compatibility" + - name: "Virtual Database" + path: "/en/setup/service-agent/virtual-database" - name: "Service Mesh" catalog: - name: "Observe Service Mesh" @@ -171,8 +173,8 @@ catalog: path: "/en/setup/service-agent/browser-agent" - name: "Database Monitoring" catalog: - - name: "Virtual Database" - path: "/en/setup/service-agent/virtual-database" + - name: "MySQL Monitoring" + path: "/en/setup/backend/backend-mysql-monitoring" - name: "Self Observability" catalog: - name: "OAP Telemetry" diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java index c0c7f5582f..624e18434c 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java @@ -48,6 +48,7 @@ public class UITemplateInitializer { Layer.OS_LINUX.name(), Layer.MESH_CP.name(), Layer.MESH_DP.name(), + Layer.MYSQL.name(), Layer.K8S.name(), Layer.BROWSER.name(), Layer.SO11Y_OAP.name(), diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml index 771df8afeb..52b34f747b 100755 --- a/oap-server/server-starter/src/main/resources/application.yml +++ b/oap-server/server-starter/src/main/resources/application.yml @@ -339,7 +339,7 @@ receiver-otel: selector: ${SW_OTEL_RECEIVER:default} default: enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc"} - enabledOcRules: ${SW_OTEL_RECEIVER_ENABLED_OC_RULES:"istio-controlplane,k8s-node,oap,vm"} + enabledOcRules: ${SW_OTEL_RECEIVER_ENABLED_OC_RULES:"istio-controlplane,k8s-node,oap,vm,mysql"} receiver-zipkin: selector: ${SW_RECEIVER_ZIPKIN:-} diff --git a/oap-server/server-starter/src/main/resources/otel-oc-rules/mysql.yaml b/oap-server/server-starter/src/main/resources/otel-oc-rules/mysql.yaml new file mode 100644 index 0000000000..b41aad70fe --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-oc-rules/mysql.yaml @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This will parse a textual representation of a duration. The formats +# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS} +# with days considered to be exactly 24 hours. +#

+# Examples: +#

+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# 
+filter: "{ tags -> tags.job_name == 'mysql-monitoring' }" # The OpenTelemetry job name +expSuffix: tag({tags -> tags.host_name = 'mysql::' + tags.host_name}).service(['host_name'] , Layer.MYSQL) +metricPrefix: meter_mysql +metricsRules: + # mysql configurations + - name: uptime + exp: mysql_global_status_uptime + - name: innodb_buffer_pool_size + exp: mysql_global_variables_innodb_buffer_pool_size + - name: max_connections + exp: mysql_global_variables_max_connections + - name: thread_cache_size + exp: mysql_global_variables_thread_cache_size + + # database throughput + - name: commands_insert_rate + exp: mysql_global_status_commands_total.tagEqual('command','insert').rate('PT1M') + - name: commands_select_rate + exp: mysql_global_status_commands_total.tagEqual('command','select').rate('PT1M') + - name: commands_delete_rate + exp: mysql_global_status_commands_total.tagEqual('command','delete').rate('PT1M') + - name: commands_update_rate + exp: mysql_global_status_commands_total.tagEqual('command','update').rate('PT1M') + - name: qps + exp: mysql_global_status_queries.rate('PT1M') + - name: tps + exp: mysql_global_status_commands_total.tagMatch('command','rollback|commit').rate('PT1M') + + # connections + ## threads + - name: threads_connected + exp: mysql_global_status_threads_connected + - name: threads_created + exp: mysql_global_status_threads_created + - name: threads_running + exp: mysql_global_status_threads_running + - name: threads_cached + exp: mysql_global_status_threads_cached + ## connect + - name: connects_aborted + exp: mysql_global_status_aborted_connects + - name: connects_available + exp: mysql_global_variables_max_connections.sum(['host_name']) - mysql_global_status_threads_connected.sum(['host_name']) + - name: connection_errors_max_connections + exp: mysql_global_status_connection_errors_total.tagEqual('error','max_connection') + - name: connection_errors_internal + exp: mysql_global_status_connection_errors_total.tagEqual('error','internal') + + # slow queries + - name: slow_queries_rate + exp: mysql_global_status_slow_queries.rate('PT1M') + diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/mysql/mysql-root.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/mysql/mysql-root.json new file mode 100644 index 0000000000..b28ae7f7fb --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/mysql/mysql-root.json @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +[ + { + "id":"b6e3763c-882d-417f-8556-3cf0187cb8db", + "configuration":{ + "children":[ + { + "x":0, + "y":3, + "w":24, + "h":43, + "i":"0", + "type":"Widget", + "graph":{ + "type":"ServiceList", + "dashboardName":"MySQL-Service", + "fontSize":12, + "showXAxis":false, + "showYAxis":false, + "showGroup":true + } + }, + { + "x":0, + "y":0, + "w":24, + "h":3, + "i":"1", + "type":"Text", + "graph":{ + "fontColor":"blue", + "backgroundColor":"white", + "content":"Provide MySQL Server monitoring through OpenTelemetry's Prometheus Receiver", + "fontSize":14, + "textAlign":"left", + "url":"https://skywalking.apache.org/docs/main/latest/en/setup/backend/backend-mysql-monitoring/" + } + } + ], + "id":"b6e3763c-882d-417f-8556-3cf0187cb8db", + "layer":"MYSQL", + "entity":"All", + "name":"MySQL-Root", + "isRoot":true + } + } +] \ No newline at end of file diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/mysql/mysql-service.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/mysql/mysql-service.json new file mode 100644 index 0000000000..7ad4b2a19a --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/mysql/mysql-service.json @@ -0,0 +1,333 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +[ + { + "id":"2981ef0e-1552-4cf0-a68c-da023b4a0fd8", + "configuration":{ + "children":[ + { + "x":0, + "y":0, + "w":6, + "h":11, + "i":"0", + "type":"Widget", + "metricTypes":[ + "readMetricsValue" + ], + "metrics":[ + "meter_mysql_uptime" + ], + "graph":{ + "type":"Card", + "fontSize":40, + "textAlign":"center", + "showUnit":true + }, + "metricConfig":[ + { + "calculation":"secondToDay" + } + ], + "widget":{ + "title":"MySQL Uptime (day)" + } + }, + { + "x":0, + "y":11, + "w":6, + "h":14, + "i":"1", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "meter_mysql_qps" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Current QPS" + }, + "value":"1", + "label":"1" + }, + { + "x":12, + "y":0, + "w":6, + "h":11, + "i":"2", + "type":"Widget", + "metricTypes":[ + "readMetricsValue" + ], + "metrics":[ + "meter_mysql_innodb_buffer_pool_size" + ], + "graph":{ + "type":"Card", + "fontSize":40, + "textAlign":"center", + "showUnit":true + }, + "metricConfig":[ + { + "calculation":"byteToMB" + } + ], + "widget":{ + "title":"Innodb Buffer Pool Size (MB)" + } + }, + { + "x":6, + "y":0, + "w":6, + "h":11, + "i":"3", + "type":"Widget", + "metricTypes":[ + "readMetricsValue" + ], + "metrics":[ + "meter_mysql_max_connections" + ], + "graph":{ + "type":"Card", + "fontSize":40, + "textAlign":"center", + "showUnit":true + }, + "widget":{ + "title":"Max Connections" + } + }, + { + "x":18, + "y":0, + "w":6, + "h":11, + "i":"4", + "type":"Widget", + "metricTypes":[ + "readMetricsValue" + ], + "metrics":[ + "meter_mysql_thread_cache_size" + ], + "graph":{ + "type":"Card", + "fontSize":40, + "textAlign":"center", + "showUnit":true + }, + "widget":{ + "title":"Thread Cache Size" + } + }, + { + "x":6, + "y":11, + "w":6, + "h":14, + "i":"5", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "meter_mysql_tps" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Current TPS" + }, + "value":"5", + "label":"5" + }, + { + "x":12, + "y":11, + "w":6, + "h":14, + "i":"6", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues", + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "meter_mysql_commands_insert_rate", + "meter_mysql_commands_select_rate", + "meter_mysql_commands_delete_rate", + "meter_mysql_commands_update_rate" + ], + "graph":{ + "type":"Area", + "opacity":0.4, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Commands Trend" + }, + "value":"6", + "label":"6" + }, + { + "x":6, + "y":25, + "w":6, + "h":13, + "i":"7", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues", + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "meter_mysql_threads_cached", + "meter_mysql_threads_running", + "meter_mysql_threads_created", + "meter_mysql_threads_connected" + ], + "graph":{ + "type":"Area", + "opacity":0.4, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Threads" + }, + "value":"7", + "label":"7" + }, + { + "x":0, + "y":25, + "w":6, + "h":13, + "i":"8", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "meter_mysql_connects_available", + "meter_mysql_connects_aborted" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Connects" + }, + "value":"8", + "label":"8" + }, + { + "x":18, + "y":11, + "w":6, + "h":14, + "i":"10", + "type":"Widget", + "metricTypes":[ + "readMetricsValues" + ], + "metrics":[ + "meter_mysql_slow_queries_rate" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Slow Quries Trend" + }, + "value":"10", + "label":"10" + }, + { + "x":12, + "y":25, + "w":6, + "h":13, + "i":"11", + "type":"Widget", + "metricTypes":[ + "readMetricsValues", + "readMetricsValues" + ], + "metrics":[ + "meter_mysql_connection_errors_internal", + "meter_mysql_connection_errors_max_connections" + ], + "graph":{ + "type":"Line", + "step":false, + "smooth":false, + "showSymbol":false, + "showXAxis":true, + "showYAxis":true + }, + "widget":{ + "title":"Connection Errors" + }, + "value":"11", + "label":"11" + } + ], + "layer":"MYSQL", + "entity":"Service", + "name":"MySQL-Service", + "id":"2981ef0e-1552-4cf0-a68c-da023b4a0fd8" + } + } +] diff --git a/test/e2e-v2/cases/mysql/expected/metrics-has-value.yml b/test/e2e-v2/cases/mysql/expected/metrics-has-value.yml new file mode 100644 index 0000000000..d9c49854c9 --- /dev/null +++ b/test/e2e-v2/cases/mysql/expected/metrics-has-value.yml @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- contains . }} +- key: {{ notEmpty .key }} + value: {{ ge .value 1 }} +{{- end }} \ No newline at end of file diff --git a/test/e2e-v2/cases/mysql/expected/service.yml b/test/e2e-v2/cases/mysql/expected/service.yml new file mode 100644 index 0000000000..f499c9d32e --- /dev/null +++ b/test/e2e-v2/cases/mysql/expected/service.yml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- contains . }} +- id: {{ b64enc "mysql::showcase" }}.1 + name: mysql::showcase + shortname: showcase + group: mysql + normal: true + layers: + - MYSQL +{{- end }} \ No newline at end of file diff --git a/test/e2e-v2/cases/mysql/mysql-cases.yaml b/test/e2e-v2/cases/mysql/mysql-cases.yaml new file mode 100644 index 0000000000..7f4a79d69d --- /dev/null +++ b/test/e2e-v2/cases/mysql/mysql-cases.yaml @@ -0,0 +1,55 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + + cases: + # service list + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls + expected: expected/service.yml + # metrics + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_uptime --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_innodb_buffer_pool_size --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_max_connections --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_thread_cache_size --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_commands_select_rate --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_commands_insert_rate --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_commands_update_rate --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_commands_delete_rate --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_qps --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_tps --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_threads_connected --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_threads_created --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_threads_running --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_connects_aborted --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics linear --name=meter_mysql_connects_available --service-name=mysql::showcase |yq e 'to_entries' - + expected: expected/metrics-has-value.yml \ No newline at end of file diff --git a/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/docker-compose.yml b/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/docker-compose.yml new file mode 100644 index 0000000000..1d6047753f --- /dev/null +++ b/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/docker-compose.yml @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: '2.1' + +services: + oap: + extends: + file: ../../../script/docker-compose/base-compose.yml + service: oap + environment: + SW_OTEL_RECEIVER: default + SW_OTEL_RECEIVER_ENABLED_OC_RULES: mysql + ports: + - 12800 + mysql: + image: mysql:8.0.30 + networks: + - e2e + ports: + - 3306 + environment: + - MYSQL_ROOT_PASSWORD=password + - MYSQL_DATABASE=test2 + + mysql-load: + image: mysql:8.0.30 + depends_on: + - otel-collector + networks: + - e2e + entrypoint: bash + volumes: + - ../prometheus-mysql-exporter:/docker + command: + - -c + - "mysql -h mysql -uroot -ppassword < /docker/mock.sql" + + mysql-service: + image: prom/mysqld-exporter:v0.14.0 + ports: + - 9104 + environment: + - DATA_SOURCE_NAME=root:password@(mysql:3306)/ + networks: + - e2e + depends_on: + - mysql + + + otel-collector: + image: otel/opentelemetry-collector:0.50.0 + networks: + - e2e + command: [ "--config=/etc/otel-collector-config.yaml" ] + volumes: + - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml + expose: + - 55678 + depends_on: + oap: + condition: service_healthy + +networks: + e2e: \ No newline at end of file diff --git a/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/e2e.yaml b/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/e2e.yaml new file mode 100644 index 0000000000..87b91a1c02 --- /dev/null +++ b/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/e2e.yaml @@ -0,0 +1,38 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: compose + file: docker-compose.yml + timeout: 20m + init-system-environment: ../../../script/env + steps: + - name: set PATH + command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH + - name: install yq + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq + - name: install swctl + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl + +verify: + retry: + count: 20 + interval: 3s + cases: + - includes: + - ../mysql-cases.yaml + \ No newline at end of file diff --git a/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/mock.sql b/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/mock.sql new file mode 100644 index 0000000000..a3d60701ed --- /dev/null +++ b/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/mock.sql @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +CREATE DATABASE IF NOT EXISTS test; +USE test; +CREATE TABLE IF NOT EXISTS `t1`( + `te1` VARCHAR(100) NOT NULL +)ENGINE=InnoDB DEFAULT CHARSET=utf8; +SET GLOBAL event_scheduler = 1; + +CREATE EVENT `event_1` +ON SCHEDULE EVERY 1 SECOND +DO INSERT INTO t1 values('test'); + +CREATE EVENT `event_2` +ON SCHEDULE EVERY 1 SECOND +DO UPDATE t1 SET `te1` = 1; + +CREATE EVENT `event_3` +ON SCHEDULE EVERY 1 SECOND +DO DELETE FROM t1; + +CREATE EVENT `event_4` +ON SCHEDULE EVERY 1 SECOND +DO COMMIT; diff --git a/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/otel-collector-config.yaml b/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/otel-collector-config.yaml new file mode 100644 index 0000000000..3ac29a81b0 --- /dev/null +++ b/test/e2e-v2/cases/mysql/prometheus-mysql-exporter/otel-collector-config.yaml @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +receivers: + prometheus: + config: + scrape_configs: + - job_name: 'mysql-monitoring' + scrape_interval: 5s + static_configs: + - targets: ['mysql-service:9104'] + labels: + host_name: showcase +processors: + batch: + +exporters: + opencensus: + endpoint: oap:11800 + tls: + insecure: true +service: + pipelines: + metrics: + receivers: + - prometheus + processors: + - batch + exporters: + - opencensus \ No newline at end of file