Add support Kong monitoring (#12696)

.github/workflows/skywalking.yaml

@@ -667,6 +667,8 @@ jobs:
             config: test/e2e-v2/cases/clickhouse/clickhouse-prometheus-endpoint/e2e.yaml
           - name: ActiveMQ
             config: test/e2e-v2/cases/activemq/e2e.yaml
+          - name: Kong
+            config: test/e2e-v2/cases/kong/e2e.yaml
 
           - name: UI Menu BanyanDB
             config: test/e2e-v2/cases/menu/banyandb/e2e.yaml

docs/en/changes/changes.md

@@ -10,10 +10,12 @@
 * Support query endpoint list with duration parameter(optional).
 * Change the endpoint_traffic to updatable for the additional column `last_ping`.
 * Add Component ID(5023) for the GoZero framework.
-
+* Support Kong monitoring.
 
 #### UI
 
+* Add support for case-insensitive search in the dashboard list.
+* Add content decorations to Table and Card widgets.
 
 #### Documentation
 * Update release document to adopt newly added revision-based process.

docs/en/concepts-and-designs/service-hierarchy.md

@@ -37,6 +37,7 @@ If you want to customize it according to your own needs, please refer to [Servic
 | KAFKA            | K8S_SERVICE | [KAFKA On K8S_SERVICE](#kafka-on-k8s_service)                     |
 | PULSAR           | K8S_SERVICE | [PULSAR On K8S_SERVICE](#pulsar-on-k8s_service)                   |
 | SO11Y_OAP        | K8S_SERVICE | [SO11Y_OAP On K8S_SERVICE](#so11y_oap-on-k8s_service)             |
+| KONG             | K8S_SERVICE | [KONG On K8S_SERVICE](#kong-on-k8s_service)                       |
 
 - The following sections will describe the **default matching rules** in detail and use the `upper-layer On lower-layer` format. 
 - The example service name are based on SkyWalking [Showcase](https://github.com/apache/skywalking-showcase) default deployment.
@@ -220,6 +221,14 @@ If you want to customize it according to your own needs, please refer to [Servic
   - SO11Y_OAP.service.name: `demo-oap.skywalking-showcase`
   - K8S_SERVICE.service.name: `skywalking-showcase::demo-oap.skywalking-showcase`
 
+#### KONG On K8S_SERVICE
+- Rule name: `short-name`
+- Groovy script: `{ (u, l) -> u.shortName == l.shortName }`
+- Description: KONG.service.shortName == K8S_SERVICE.service.shortName
+- Matched Example:
+  - KONG.service.name: `kong::kong.skywalking-showcase`
+  - K8S_SERVICE.service.name: `skywalking-showcase::kong.skywalking-showcase`
+
 ### Build Through Specific Agents
 Use agent tech involved(such as eBPF) and deployment tools(such as operator and agent injector) to detect the service hierarchy relations.
 

docs/en/setup/backend/backend-kong-monitoring.md

@@ -0,0 +1,77 @@
+# KONG monitoring
+
+## KONG performance from `kong prometheus plugin`
+The [kong-prometheus](https://github.com/Kong/kong/tree/master/kong/plugins/prometheus) is a lua library that can be used with Kong to collect metrics.
+It exposes metrics related to Kong and proxied upstream services in Prometheus exposition format, which can be scraped by a Prometheus Server.
+SkyWalking leverages OpenTelemetry Collector to transfer the metrics to[OpenTelemetry receiver](opentelemetry-receiver.md) 
+and into the [Meter System](./../../concepts-and-designs/meter.md).
+
+### Data flow
+1. [KONG Prometheus plugin](https://docs.konghq.com/hub/kong-inc/prometheus/) collects metrics data from KONG.
+2. OpenTelemetry Collector fetches metrics from [KONG Prometheus plugin](https://docs.konghq.com/hub/kong-inc/prometheus/) via 
+   Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
+3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
+
+### Set up
+1. Enable KONG [KONG Prometheus plugin](https://docs.konghq.com/hub/kong-inc/prometheus/). Note that if need to monitor per_consumer, 
+   status_code_metrics, ai_metrics, latency_metrics, bandwidth_metrics or upstream_health_metrics, **need to enable them manually as needed**, 
+   which can be enabled in the [konga](https://pantsel.github.io/konga/) dashboard or through the Admin API, such as the following command
+   ~~~bash
+   curl -i -X POST http://{KONG-HOST}:{KONG_ADMIN_PORT}/plugins \
+    --data name=prometheus \
+    --data config.per_consumer=true \
+    --data config.status_code_metrics=true \
+    --data config.ai_metrics=true \
+    --data config.latency_metrics=true \
+    --data config.bandwidth_metrics=true \
+    --data config.upstream_health_metrics=true
+   ~~~
+2. Set up [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/#docker). 
+   For details on Prometheus Receiver in OpenTelemetry Collector, refer to [here](../../../../test/e2e-v2/cases/kong/otel-collector-config.yaml).
+3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).
+
+### KONG Monitoring
+
+[KONG prometheus plugin](https://docs.konghq.com/hub/kong-inc/prometheus/) provide multiple dimensions metrics for KONG server, upstream, route etc.
+Accordingly, SkyWalking observes the status, requests, and latency of the KONG server, which is cataloged as a `LAYER: KONG` `Service` in the OAP.
+Each Kong server is cataloged as a `LAYER: KONG` `instance`, meanwhile, the route rules would be recognized as a `LAYER: KONG` `endpoint`.
+
+
+#### Kong Request Supported Metrics
+
+| Monitoring Panel | Unit  | Metric Name                                                                                                       | Description                                          | Data Source |
+|------------------|-------|-------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|-------------|
+| Bandwidth        | bytes | meter_kong_service_http_bandwidth<br />meter_kong_instance_http_bandwidth<br />meter_kong_endpoint_http_bandwidth | Total bandwidth (ingress/egress) throughput          | Kong        |
+| HTTP Status      | count | meter_kong_service_http_status<br />meter_kong_instance_http_status<br />meter_kong_endpoint_http_status          | HTTP status codes per consumer/service/route in Kong | Kong        |
+| HTTP Request     | count | meter_kong_service_http_requests<br />meter_kong_instance_http_requests                                           | Total number of requests                             | Kong        |
+
+#### Kong Database Supported Metrics
+
+| Monitoring Panel | Unit  | Metric Name                                                                         | Description                               | Data Source |
+|------------------|-------|-------------------------------------------------------------------------------------|-------------------------------------------|-------------|
+| DB               | count | meter_kong_service_datastore_reachable<br />meter_kong_instance_datastore_reachable | Datastore reachable from Kong             | Kong        |
+| DB               | bytes | meter_kong_instance_shared_dict_bytes                                               | Allocated slabs in bytes in a shared_dict | Kong        |
+| DB               | bytes | meter_kong_instance_shared_dict_total_bytes                                         | Total capacity in bytes of a shared_dict  | Kong        |
+| DB               | bytes | meter_kong_instance_memory_workers_lua_vms_bytes                                    | Allocated bytes in worker Lua VM          | Kong        |
+
+#### Kong Latencies Supported Metrics
+
+| Monitoring Panel | Unit | Metric Name                                                                                                             | Description                                                              | Data Source |
+|------------------|------|-------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|-------------|
+| Latency          | ms   | meter_kong_service_kong_latency<br />meter_kong_instance_kong_latency<br />meter_kong_endpoint_kong_latency             | Latency added by Kong and enabled plugins for each service/route in Kong | Kong        |
+| Latency          | ms   | meter_kong_service_request_latency<br />meter_kong_instance_request_latency<br />meter_kong_endpoint_request_latency    | Total latency incurred during requests for each service/route in Kong    | Kong        |
+| Latency          | ms   | meter_kong_service_upstream_latency<br />meter_kong_instance_upstream_latency<br />meter_kong_endpoint_upstream_latency | Latency added by upstream response for each service/route in Kong        | Kong        |
+
+
+#### Kong Nginx Supported Metrics
+
+| Monitoring Panel | Unit  | Metric Name                                                                                 | Description                           | Data Source |
+|------------------|-------|---------------------------------------------------------------------------------------------|---------------------------------------|-------------|
+| Nginx            | count | meter_kong_service_nginx_metric_errors_total                                                | Number of nginx-lua-prometheus errors | Kong        |
+| Nginx            | count | meter_kong_service_nginx_connections_total<br />meter_kong_instance_nginx_connections_total | Number of connections by subsystem    | Kong        |
+| Nginx            | count | meter_kong_service_nginx_timers<br />meter_kong_instance_nginx_timers                       | Number of Nginx timers                | Kong        |
+
+### Customizations
+You can customize your own metrics/expression/dashboard panel.
+The metrics definition and expression rules are found in `/config/otel-rules/kong`.
+The KONG dashboard panel configurations are found in `/config/ui-initialized-templates/kong`.

docs/en/swip/SWIP-8.md

@@ -0,0 +1,61 @@
+# Support Kong Monitoring
+
+## Motivation
+
+[**Kong**](https://github.com/Kong/kong) or **Kong API Gateway** is a cloud-native, platform-agnostic, scalable API Gateway 
+distinguished for its high performance and extensibility via plugins. Now I want to add Kong monitoring via the OpenTelemetry Collector, 
+which fetches metrics from it's own HTTP endpoint to expose metrics data for [Prometheus](https://prometheus.io/).
+
+## Architecture Graph
+
+There is no significant architecture-level change.
+
+## Proposed Changes
+
+1. Kong expose its own [metrics](https://docs.konghq.com/hub/kong-inc/prometheus/) via HTTP endpoint to opentelemetry collector, OpenTelemetry Collector fetches metrics from it and pushes metrics to SkyWalking OTEL Receiver via OpenTelemetry exporter.
+2. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.
+3. These metrics can be displayed via the SkyWalking UI, and the metrics can be customized for display on the UI dashboard.
+
+### Kong Request Supported Metrics
+
+| Monitoring Panel | Unit  | Metric Name                                                                                                       | Description                                          | Data Source |
+|------------------|-------|-------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|-------------|
+| Bandwidth        | bytes | meter_kong_service_http_bandwidth<br />meter_kong_instance_http_bandwidth<br />meter_kong_endpoint_http_bandwidth | Total bandwidth (ingress/egress) throughput          | Kong        |
+| HTTP Status      | count | meter_kong_service_http_status<br />meter_kong_instance_http_status<br />meter_kong_endpoint_http_status          | HTTP status codes per consumer/service/route in Kong | Kong        |
+| HTTP Request     | count | meter_kong_service_http_requests<br />meter_kong_instance_http_requests                                           | Total number of requests                             | Kong        |
+
+### Kong Database Supported Metrics
+
+| Monitoring Panel | Unit  | Metric Name                                                                         | Description                               | Data Source |
+|------------------|-------|-------------------------------------------------------------------------------------|-------------------------------------------|-------------|
+| DB               | count | meter_kong_service_datastore_reachable<br />meter_kong_instance_datastore_reachable | Datastore reachable from Kong             | Kong        |
+| DB               | bytes | meter_kong_instance_shared_dict_bytes                                               | Allocated slabs in bytes in a shared_dict | Kong        |
+| DB               | bytes | meter_kong_instance_shared_dict_total_bytes                                         | Total capacity in bytes of a shared_dict  | Kong        |
+| DB               | bytes | meter_kong_instance_memory_workers_lua_vms_bytes                                    | Allocated bytes in worker Lua VM          | Kong        |
+
+### Kong Latencies Supported Metrics
+
+| Monitoring Panel | Unit | Metric Name                                                                                                             | Description                                                              | Data Source |
+|------------------|------|-------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|-------------|
+| Latency          | ms   | meter_kong_service_kong_latency<br />meter_kong_instance_kong_latency<br />meter_kong_endpoint_kong_latency             | Latency added by Kong and enabled plugins for each service/route in Kong | Kong        |
+| Latency          | ms   | meter_kong_service_request_latency<br />meter_kong_instance_request_latency<br />meter_kong_endpoint_request_latency    | Total latency incurred during requests for each service/route in Kong    | Kong        |
+| Latency          | ms   | meter_kong_service_upstream_latency<br />meter_kong_instance_upstream_latency<br />meter_kong_endpoint_upstream_latency | Latency added by upstream response for each service/route in Kong        | Kong        |
+
+
+### Kong Nginx Supported Metrics
+
+| Monitoring Panel | Unit  | Metric Name                                                                                 | Description                           | Data Source |
+|------------------|-------|---------------------------------------------------------------------------------------------|---------------------------------------|-------------|
+| Nginx            | count | meter_kong_service_nginx_metric_errors_total                                                | Number of nginx-lua-prometheus errors | Kong        |
+| Nginx            | count | meter_kong_service_nginx_connections_total<br />meter_kong_instance_nginx_connections_total | Number of connections by subsystem    | Kong        |
+| Nginx            | count | meter_kong_service_nginx_timers<br />meter_kong_instance_nginx_timers                       | Number of Nginx timers                | Kong        |
+
+## Imported Dependencies libs and their licenses.
+
+No new dependency.
+
+## Compatibility
+
+no breaking changes.
+
+## General usage docs

docs/menu.yml

@@ -110,6 +110,8 @@ catalog:
                 path: "/en/setup/backend/backend-apisix-monitoring"
               - name: "AWS API Gateway"
                 path: "/en/setup/backend/backend-aws-api-gateway-monitoring"
+              - name: "Kong Monitoring"
+                path: "/en/setup/backend/backend-kong-monitoring"
           - name: "Database Monitoring"
             catalog:
               - name: "MySQL/MariaDB Server"

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java

@@ -240,7 +240,12 @@ public enum Layer {
      * The self observability of SkyWalking Java Agent,
      * which provides the abilities to measure the tracing performance and error statistics of plugins.
      */
-    SO11Y_JAVA_AGENT(39, true);
+    SO11Y_JAVA_AGENT(39, true),
+
+    /**
+     * Kong is Cloud-Native API Gateway and AI Gateway.
+     */
+    KONG(40, true);
 
     private final int value;
     /**

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java

@@ -77,6 +77,7 @@ public class UITemplateInitializer {
         Layer.ACTIVEMQ.name(),
         Layer.CILIUM_SERVICE.name(),
         Layer.SO11Y_JAVA_AGENT.name(),
+        Layer.KONG.name(),
         "custom"
     };
     private final UITemplateManagementService uiTemplateManagementService;

oap-server/server-starter/src/main/resources/application.yml

@@ -399,7 +399,7 @@ receiver-otel:
   selector: ${SW_OTEL_RECEIVER:default}
   default:
     enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"otlp-metrics,otlp-logs"}
-    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,nginx/*,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*,rocketmq/*,clickhouse/*,activemq/*"}
+    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,nginx/*,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*,rocketmq/*,clickhouse/*,activemq/*,kong/*"}
 
 receiver-zipkin:
   selector: ${SW_RECEIVER_ZIPKIN:-}

oap-server/server-starter/src/main/resources/hierarchy-definition.yml

@@ -29,6 +29,7 @@ hierarchy:
   GENERAL:
     APISIX: lower-short-name-remove-ns
     K8S_SERVICE: lower-short-name-remove-ns
+    KONG: lower-short-name-remove-ns
 
   MYSQL:
     K8S_SERVICE: short-name
@@ -62,6 +63,9 @@ hierarchy:
 
   ACTIVEMQ:
     K8S_SERVICE: short-name
+  
+  KONG:
+    K8S_SERVICE: short-name
 
   VIRTUAL_DATABASE:
     MYSQL: lower-short-name-with-fqdn
@@ -110,6 +114,7 @@ layer-levels:
   KAFKA: 2
   PULSAR: 2
   ACTIVEMQ: 2
+  KONG: 2
 
   MESH_DP: 1
   CILIUM_SERVICE: 1

oap-server/server-starter/src/main/resources/otel-rules/kong/kong-endpoint.yaml

@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+filter: "{ tags -> tags.job_name == 'kong-monitoring' }"
+expSuffix: tag({tags -> tags.host_name = 'kong::' + tags.host_name}).endpoint(['host_name'], ['route'], Layer.KONG)
+metricPrefix: meter_kong_endpoint
+metricsRules:
+  # counter
+    # Total bandwidth (ingress/egress) throughput in bytes
+  - name: http_bandwidth
+    exp: kong_bandwidth_bytes.sum(['host_name','direction','route']).rate('PT1M')
+    # HTTP status codes per consumer/service/route in Kong
+  - name: http_status
+    exp: kong_http_requests_total.sum(['host_name','route','code']).rate('PT1M')
+    
+  # histogram
+    # Latency added by Kong and enabled plugins for each service/route in Kong
+  - name: kong_latency
+    exp: kong_kong_latency_ms.tagNotEqual('route','').sum(['host_name','route','le']).histogram().histogram_percentile([50,70,90,99])
+    # Total latency incurred during requests for each service/route in Kong
+  - name: request_latency
+    exp: kong_request_latency_ms.tagNotEqual('route','').sum(['host_name','route','le']).histogram().histogram_percentile([50,70,90,99])
+    # Latency added by upstream response for each service/route in Kong
+  - name: upstream_latency
+    exp: kong_upstream_latency_ms.tagNotEqual('route','').sum(['host_name','route','le']).histogram().histogram_percentile([50,70,90,99])

oap-server/server-starter/src/main/resources/otel-rules/kong/kong-instance.yaml

@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+filter: "{ tags -> tags.job_name == 'kong-monitoring' }"
+expSuffix: tag({tags -> tags.host_name = 'kong::' + tags.host_name}).instance(['host_name'], ['service_instance_id'], Layer.KONG)
+metricPrefix: meter_kong_instance
+metricsRules:
+  # counter
+    # Total bandwidth (ingress/egress) throughput in bytes
+  - name: http_bandwidth
+    exp: kong_bandwidth_bytes.sum(['host_name','service_instance_id','direction','route']).rate('PT1M')
+    # HTTP status codes per consumer/service/route in Kong
+  - name: http_status
+    exp: kong_http_requests_total.sum(['host_name','service_instance_id','code']).rate('PT1M')
+
+  # gauge
+    # Datastore reachable from Kong
+  - name: datastore_reachable
+    exp: kong_datastore_reachable.sum(['host_name','service_instance_id'])
+    # Total number of requests
+  - name: http_requests
+    exp: kong_nginx_requests_total.sum(['host_name','service_instance_id']).rate('PT1M')
+    # Allocated slabs in bytes in a shared_dict
+  - name: shared_dict_bytes
+    exp: kong_memory_lua_shared_dict_bytes.sum(['host_name','service_instance_id','shared_dict'])
+    # Total capacity in bytes of a shared_dict
+  - name: shared_dict_total_bytes
+    exp: kong_memory_lua_shared_dict_total_bytes.sum(['host_name','service_instance_id','shared_dict'])
+    # Allocated bytes in worker Lua VM
+  - name: memory_workers_lua_vms_bytes
+    exp: kong_memory_workers_lua_vms_bytes.tagNotEqual('pid','').sum(['host_name','service_instance_id','pid'])
+    # Number of connections by subsystem
+  - name: nginx_connections_total
+    exp: kong_nginx_connections_total.tagNotEqual('state','').sum(['host_name','service_instance_id','state']).rate('PT1M')
+    # Number of Nginx timers
+  - name: nginx_timers
+    exp: kong_nginx_timers.sum(['host_name','service_instance_id','state'])
+    
+  # histogram
+    # Latency added by Kong and enabled plugins for each service/route in Kong
+  - name: kong_latency
+    exp: kong_kong_latency_ms.sum(['host_name','service_instance_id','le']).histogram().histogram_percentile([50,70,90,99])
+    # Total latency incurred during requests for each service/route in Kong
+  - name: request_latency
+    exp: kong_request_latency_ms.sum(['host_name','service_instance_id','le']).histogram().histogram_percentile([50,70,90,99])
+    # Latency added by upstream response for each service/route in Kong
+  - name: upstream_latency
+    exp: kong_upstream_latency_ms.sum(['host_name','service_instance_id','le']).histogram().histogram_percentile([50,70,90,99])

oap-server/server-starter/src/main/resources/otel-rules/kong/kong-service.yaml

@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+filter: "{ tags -> tags.job_name == 'kong-monitoring' }"
+expSuffix: tag({tags -> tags.host_name = 'kong::' + tags.host_name}).service(['host_name'], Layer.KONG)
+metricPrefix: meter_kong_service
+metricsRules:
+  # counter
+    # Total bandwidth (ingress/egress) throughput in bytes
+  - name: http_bandwidth
+    exp: kong_bandwidth_bytes.sum(['host_name','service_instance_id','direction','route']).rate('PT1M')
+    # HTTP status codes per consumer/service/route in Kong
+  - name: http_status
+    exp: kong_http_requests_total.sum(['host_name','service_instance_id','code']).rate('PT1M')
+    # Number of nginx-lua-prometheus errors
+  - name: nginx_metric_errors_total
+    exp: kong_nginx_metric_errors_total.sum(['host_name','service_instance_id'])
+    
+  # gauge
+    # Datastore reachable from Kong
+  - name: datastore_reachable
+    exp: kong_datastore_reachable.sum(['host_name','service_instance_id'])
+    # Total number of requests
+  - name: http_requests
+    exp: kong_nginx_requests_total.sum(['host_name','service_instance_id']).rate('PT1M')
+    # Number of connections by subsystem
+  - name: nginx_connections_total
+    exp: kong_nginx_connections_total.sum(['host_name','service_instance_id','state']).rate('PT1M')
+    # Number of Nginx timers
+  - name: nginx_timers
+    exp: kong_nginx_timers.sum(['host_name','service_instance_id','state'])
+    
+  # histogram
+    # Latency added by Kong and enabled plugins for each service/route in Kong
+  - name: kong_latency
+    exp: kong_kong_latency_ms.sum(['le','host_name','service_instance_id']).histogram().histogram_percentile([50,70,90,99])
+    # Total latency incurred during requests for each service/route in Kong
+  - name: request_latency
+    exp: kong_request_latency_ms.sum(['le','host_name','service_instance_id']).histogram().histogram_percentile([50,70,90,99])
+    # Latency added by upstream response for each service/route in Kong
+  - name: upstream_latency
+    exp: kong_upstream_latency_ms.sum(['le','host_name','service_instance_id']).histogram().histogram_percentile([50,70,90,99])

oap-server/server-starter/src/main/resources/ui-initialized-templates/kong/kong-endpoint.json

@@ -0,0 +1,157 @@
+[
+  {
+    "id": "Kong-Endpoint",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 0,
+          "w": 12,
+          "h": 13,
+          "i": "0",
+          "type": "Widget",
+          "widget": {
+            "title": "Total bandwidth Trend (ingress/egress) (KB/s)"
+          },
+          "metricConfig": [
+            {
+              "label": "HTTP bandwidth Trend (requests per second)",
+              "unit": "KB/s"
+            }
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_kong_endpoint_http_bandwidth/1024"
+          ]
+        },
+        {
+          "x": 12,
+          "y": 0,
+          "w": 12,
+          "h": 13,
+          "i": "1",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Status Trend (requests per second)"
+          },
+          "metricConfig": [
+            {
+              "label": "HTTP Status Trend",
+              "unit": "req/s"
+            }
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_kong_endpoint_http_status"
+          ]
+        },
+        {
+          "x": 0,
+          "y": 13,
+          "w": 8,
+          "h": 13,
+          "i": "2",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_endpoint_kong_latency"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Kong Latency",
+            "unit": "ms"
+          },
+          "metricConfig": [
+            {
+              "label": "Kong Latency",
+              "tips": "Latency added by Kong and enabled plugins for each service/route in Kong"
+            }
+          ]
+        },
+        {
+          "x": 8,
+          "y": 13,
+          "w": 8,
+          "h": 13,
+          "i": "3",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_endpoint_request_latency"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Request Latency",
+            "unit": "ms"
+          },
+          "metricConfig": [
+            {
+              "label": "Request Latency",
+              "tips": "Total latency incurred during requests for each service/route in Kong"
+            }
+          ]
+        },
+        {
+          "x": 16,
+          "y": 13,
+          "w": 8,
+          "h": 13,
+          "i": "4",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_endpoint_upstream_latency"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Upstream Latency",
+            "unit": "ms"
+          },
+          "metricConfig": [
+            {
+              "label": "Upstream Latency",
+              "tips": "Latency added by upstream response for each service/route in Kong"
+            }
+          ]
+        }
+      ],
+      "layer": "KONG",
+      "entity": "Endpoint",
+      "name": "Kong-Endpoint",
+      "isRoot": false,
+      "isDefault": true
+    }
+  }
+]

oap-server/server-starter/src/main/resources/ui-initialized-templates/kong/kong-instance.json

@@ -0,0 +1,324 @@
+[
+  {
+    "id": "Kong-Instance",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 0,
+          "w": 12,
+          "h": 13,
+          "i": "0",
+          "type": "Widget",
+          "expressions": [
+            "latest(meter_kong_instance_datastore_reachable)"
+          ],
+          "graph": {
+            "type": "Card",
+            "fontSize": 24,
+            "textAlign": "center",
+            "showUnit": true,
+            "decorations": {
+              "0": "unreachable",
+              "1": "reachable"
+            }
+          },
+          "metricConfig": [
+            {
+              "label": "Datastore reachable"
+            }
+          ],
+          "widget": {
+            "title": "Datastore reachable",
+            "tips": "Datastore reachable from Kong, 0 is unreachable"
+          }
+        },
+        {
+          "x": 12,
+          "y": 0,
+          "w": 12,
+          "h": 13,
+          "i": "1",
+          "type": "Widget",
+          "widget": {
+            "title": "Total bandwidth Trend (ingress/egress) (KB/s)"
+          },
+          "metricConfig": [
+            {
+              "label": "HTTP bandwidth Trend (requests per second)",
+              "unit": "KB/s"
+            }
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_kong_instance_http_bandwidth/1024"
+          ]
+        },
+        {
+          "x": 0,
+          "y": 13,
+          "w": 12,
+          "h": 13,
+          "i": "2",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Status Trend (requests per second)"
+          },
+          "metricConfig": [
+            {
+              "label": "HTTP Status Trend",
+              "unit": "req/s"
+            }
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_kong_instance_http_status"
+          ]
+        },
+        {
+          "x": 12,
+          "y": 13,
+          "w": 12,
+          "h": 13,
+          "i": "3",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Request Trend (requests per second)"
+          },
+          "metricConfig": [
+            {
+              "label": "Number of requests total"
+            }
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_kong_instance_http_requests"
+          ]
+        },
+        {
+          "x": 0,
+          "y": 26,
+          "w": 12,
+          "h": 13,
+          "i": "4",
+          "type": "Widget",
+          "expressions": [
+            "(meter_kong_instance_shared_dict_bytes*100)/meter_kong_instance_shared_dict_total_bytes"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": false,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Shared memory usage (%)",
+            "tips": "Kong shared memory usage by Node (%)"
+          },
+          "metricConfig": [
+            {
+              "unit": "%",
+              "label": "Kong shared memory usage by Node (%)"
+            }
+          ]
+        },
+        {
+          "x": 12,
+          "y": 26,
+          "w": 12,
+          "h": 13,
+          "i": "5",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_instance_memory_workers_lua_vms_bytes/1024/1024"
+          ],
+          "graph": {
+            "type": "Area",
+            "opacity": 0.4,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Worker Lua VM usage by Node(MB)",
+            "tips": "Kong worker Lua VM usage by Node(MB)"
+          },
+          "metricConfig": [
+            {
+              "unit": "MB",
+              "label": "Kong worker Lua VM usage by Node"
+            }
+          ]
+        },
+        {
+          "x": 0,
+          "y": 39,
+          "w": 12,
+          "h": 13,
+          "i": "6",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_instance_nginx_connections_total"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "HTTP Connections Trend (requests per second)"
+          },
+          "metricConfig": [
+            {
+              "label": "HTTP Connections"
+            }
+          ]
+        },
+        {
+          "x": 12,
+          "y": 39,
+          "w": 12,
+          "h": 13,
+          "i": "7",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_instance_nginx_timers"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Nginx timers"
+          },
+          "metricConfig": [
+            {
+              "label": "Nginx timers"
+            }
+          ]
+        },
+        {
+          "x": 0,
+          "y": 52,
+          "w": 8,
+          "h": 13,
+          "i": "8",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_instance_kong_latency"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Kong Latency",
+            "unit": "ms"
+          },
+          "metricConfig": [
+            {
+              "label": "Kong Latency",
+              "tips": "Latency added by Kong and enabled plugins for each service/route in Kong"
+            }
+          ]
+        },
+        {
+          "x": 8,
+          "y": 52,
+          "w": 8,
+          "h": 13,
+          "i": "9",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_instance_request_latency"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Request Latency",
+            "unit": "ms"
+          },
+          "metricConfig": [
+            {
+              "label": "Request Latency",
+              "tips": "Total latency incurred during requests for each service/route in Kong"
+            }
+          ]
+        },
+        {
+          "x": 16,
+          "y": 52,
+          "w": 8,
+          "h": 13,
+          "i": "10",
+          "type": "Widget",
+          "expressions": [
+            "meter_kong_instance_upstream_latency"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "Upstream Latency",
+            "unit": "ms"
+          },
+          "metricConfig": [
+            {
+              "label": "Upstream Latency",
+              "tips": "Latency added by upstream response for each service/route in Kong"
+            }
+          ]
+        }
+      ],
+      "layer": "KONG",
+      "entity": "ServiceInstance",
+      "name": "Kong-Instance",
+      "isRoot": false,
+      "isDefault": true
+    }
+  }
+]

oap-server/server-starter/src/main/resources/ui-initialized-templates/kong/kong-root.json

@@ -0,0 +1,45 @@
+[
+  {
+    "id": "Kong-Root",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 3,
+          "w": 24,
+          "h": 36,
+          "i": "0",
+          "type": "Widget",
+          "graph": {
+            "type": "ServiceList",
+            "dashboardName": "Kong-Service",
+            "fontSize": 12,
+            "showXAxis": false,
+            "showYAxis": false,
+            "showGroup": true
+          }
+        },
+        {
+          "x": 0,
+          "y": 0,
+          "w": 24,
+          "h": 3,
+          "i": "1",
+          "type": "Text",
+          "graph": {
+            "fontColor": "theme",
+            "backgroundColor": "theme",
+            "content": "Provide Kong monitoring through OpenTelemetry's Prometheus Receiver",
+            "fontSize": 14,
+            "textAlign": "left",
+            "url": "https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-kong-monitoring/"
+          }
+        }
+      ],
+      "layer": "KONG",
+      "entity": "All",
+      "name": "Kong-Root",
+      "isRoot": true
+    }
+  }
+]

oap-server/server-starter/src/main/resources/ui-initialized-templates/kong/kong-service.json

@@ -0,0 +1,351 @@
+[
+  {
+    "id": "Kong-Service",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 0,
+          "w": 24,
+          "h": 48,
+          "i": "0",
+          "type": "Tab",
+          "children": [
+            {
+              "name": "Inspections",
+              "children": [
+                {
+                  "x": 0,
+                  "y": 0,
+                  "w": 12,
+                  "h": 8,
+                  "i": "0",
+                  "type": "Widget",
+                  "expressions": [
+                    "latest(aggregate_labels(meter_kong_service_nginx_metric_errors_total,sum))"
+                  ],
+                  "graph": {
+                    "type": "Card",
+                    "fontSize": 24,
+                    "textAlign": "center",
+                    "showUnit": true
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "Nginx metric errors"
+                    }
+                  ],
+                  "widget": {
+                    "title": "Nginx metric errors"
+                  }
+                },
+                {
+                  "x": 12,
+                  "y": 0,
+                  "w": 12,
+                  "h": 8,
+                  "i": "1",
+                  "type": "Widget",
+                  "expressions": [
+                    "latest(aggregate_labels(meter_kong_service_datastore_reachable,sum(service_instance_id)))"
+                  ],
+                  "graph": {
+                    "type": "Table",
+                    "showTableValues": true,
+                    "tableHeaderCol1": "Instances",
+                    "tableHeaderCol2": "Reachable",
+                    "decorations": {
+                      "0": "unreachable",
+                      "1": "reachable"
+                    }
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "Datastore reachable"
+                    }
+                  ],
+                  "widget": {
+                    "title": "Datastore reachable",
+                    "tips": "Datastore reachable from Kong, 0 is unreachable"
+                  }
+                },
+                {
+                  "x": 0,
+                  "y": 8,
+                  "w": 12,
+                  "h": 13,
+                  "i": "2",
+                  "type": "Widget",
+                  "widget": {
+                    "title": "Total bandwidth Trend (ingress/egress) (KB/s)"
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "HTTP bandwidth Trend (requests per second)",
+                      "unit": "KB/s"
+                    }
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "expressions": [
+                    "aggregate_labels(meter_kong_service_http_bandwidth,sum(direction))/1024"
+                  ]
+                },
+                {
+                  "x": 12,
+                  "y": 8,
+                  "w": 12,
+                  "h": 13,
+                  "i": "3",
+                  "type": "Widget",
+                  "widget": {
+                    "title": "HTTP Status Trend (requests per second)"
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "HTTP Status Trend",
+                      "unit": "req/s"
+                    }
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "expressions": [
+                    "aggregate_labels(meter_kong_service_http_status,sum(code))"
+                  ]
+                },
+                {
+                  "x": 0,
+                  "y": 21,
+                  "w": 12,
+                  "h": 13,
+                  "i": "4",
+                  "type": "Widget",
+                  "widget": {
+                    "title": "HTTP Request Trend (requests per second)"
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "Number of requests total",
+                      "unit": "req/s"
+                    }
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "expressions": [
+                    "aggregate_labels(meter_kong_service_http_requests,sum)"
+                  ]
+                },
+                {
+                  "x": 12,
+                  "y": 21,
+                  "w": 12,
+                  "h": 13,
+                  "i": "5",
+                  "type": "Widget",
+                  "expressions": [
+                    "aggregate_labels(meter_kong_service_nginx_connections_total,sum(state))"
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "widget": {
+                    "title": "Nginx Connections Trend (requests per second)"
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "Nginx Connections",
+                      "unit": "req/s"
+                    }
+                  ]
+                },
+                {
+                  "x": 0,
+                  "y": 34,
+                  "w": 12,
+                  "h": 13,
+                  "i": "6",
+                  "type": "Widget",
+                  "expressions": [
+                    "aggregate_labels(meter_kong_service_nginx_timers,sum(state))"
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "widget": {
+                    "title": "Nginx timers"
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "Nginx timers"
+                    }
+                  ]
+                },
+                {
+                  "x": 12,
+                  "y": 34,
+                  "w": 12,
+                  "h": 13,
+                  "i": "7",
+                  "type": "Widget",
+                  "expressions": [
+                    "aggregate_labels(meter_kong_service_kong_latency,avg(p))"
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "widget": {
+                    "title": "Kong Latency",
+                    "unit": "ms"
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "Kong Latency",
+                      "tips": "Latency added by Kong and enabled plugins for each service/route in Kong"
+                    }
+                  ]
+                },
+                {
+                  "x": 0,
+                  "y": 47,
+                  "w": 12,
+                  "h": 13,
+                  "i": "8",
+                  "type": "Widget",
+                  "expressions": [
+                    "aggregate_labels(meter_kong_service_request_latency,avg(p))"
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "widget": {
+                    "title": "Request Latency",
+                    "unit": "ms"
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "Request Latency",
+                      "tips": "Total latency incurred during requests for each service/route in Kong"
+                    }
+                  ]
+                },
+                {
+                  "x": 12,
+                  "y": 47,
+                  "w": 12,
+                  "h": 13,
+                  "i": "9",
+                  "type": "Widget",
+                  "expressions": [
+                    "aggregate_labels(meter_kong_service_upstream_latency,avg(p))"
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "widget": {
+                    "title": "Upstream Latency",
+                    "unit": "ms"
+                  },
+                  "metricConfig": [
+                    {
+                      "label": "Upstream Latency",
+                      "tips": "Latency added by upstream response for each service/route in Kong"
+                    }
+                  ]
+                }
+              ]
+            },
+            {
+              "name": "Instances",
+              "children": [
+                {
+                  "x": 0,
+                  "y": 0,
+                  "w": 24,
+                  "h": 17,
+                  "type": "Widget",
+                  "graph": {
+                    "type": "InstanceList",
+                    "dashboardName": "Kong-Instance",
+                    "fontSize": 12
+                  }
+                }
+              ]
+            }
+          ]
+        }
+      ],
+      "layer": "KONG",
+      "entity": "Service",
+      "name": "Kong-Service",
+      "isRoot": false,
+      "isDefault": true,
+      "expressions": [
+        "avg(aggregate_labels(meter_kong_service_http_requests,sum))",
+        "avg(aggregate_labels(meter_kong_service_http_status{code='200'}, sum))",
+        "avg(aggregate_labels(meter_kong_service_http_status{code='404'}, sum))",
+        "avg(aggregate_labels(meter_kong_service_http_status{code='500'}, sum))"
+      ],
+      "expressionsConfig": [
+        {
+          "label": "Avg HTTP Requests",
+          "unit": "req / s"
+        },
+        {
+          "label": "Avg HTTP Status 200",
+          "unit": "req / s"
+        },
+        {
+          "label": "Avg HTTP Status 404",
+          "unit": "req / s"
+        },
+        {
+          "label": "Avg HTTP Status 500",
+          "unit": "req / s"
+        }
+      ]
+    }
+  }
+]

skywalking-ui

@@ -1 +1 @@
-Subproject commit d65c18bd380a56df8886c931cbe72be2919667b6
+Subproject commit 61449f4b172a143d96cca139bed50839f5f2a5d6

test/e2e-v2/cases/kong/docker-compose.yml

@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+  oap:
+    extends:
+      file: ../../script/docker-compose/base-compose.yml
+      service: oap
+    ports:
+      - "12800:12800"
+    networks:
+      - e2e
+  kong-db:
+    image: postgres:alpine
+    networks:
+      - e2e
+    expose:
+      - 5432
+    environment:
+      POSTGRES_DB: kong
+      POSTGRES_USER: kong
+      POSTGRES_PASSWORD: kong
+    healthcheck:
+      test: [ "CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/5432" ]
+      interval: 5s
+      timeout: 60s
+      retries: 120
+  kong-migrations:
+    image: kong:3.8.0
+    networks:
+      - e2e
+    environment:
+      - KONG_DATABASE=postgres
+      - KONG_PG_USER=kong
+      - KONG_PG_PASSWORD=kong
+      - KONG_PG_HOST=kong-db
+      - KONG_CASSANDRA_CONTACT_POINTS=kong-db
+    command: kong migrations bootstrap
+    depends_on:
+      kong-db:
+        condition: service_healthy
+    healthcheck:
+      test: [ "CMD", "kong", "health" ]
+      interval: 10s
+      timeout: 60s
+      retries: 120
+  kong-1:
+    image: kong:3.8.0
+    networks:
+      - e2e
+    depends_on:
+      kong-migrations:
+        condition: service_completed_successfully
+    environment:
+      KONG_ADMIN_ACCESS_LOG: /dev/stdout
+      KONG_ADMIN_ERROR_LOG: /dev/stderr
+      KONG_PROXY_LISTEN: "0.0.0.0:8000"
+      KONG_ADMIN_LISTEN: "0.0.0.0:8001"
+      KONG_CASSANDRA_CONTACT_POINTS: kong-db
+      KONG_DATABASE: postgres
+      KONG_PG_DATABASE: kong
+      KONG_PG_PASSWORD: kong
+      KONG_PG_HOST: kong-db
+      KONG_PG_USER: kong
+      KONG_PROXY_ACCESS_LOG: /dev/stdout
+      KONG_PROXY_ERROR_LOG: /dev/stderr
+    ports:
+      - "8001:8001"
+      - "8000:8000"
+    restart: on-failure
+    healthcheck:
+      test: [ "CMD", "kong", "health" ]
+      interval: 10s
+      timeout: 60s
+      retries: 120
+  otel-collector:
+    image: otel/opentelemetry-collector:${OTEL_COLLECTOR_VERSION}
+    networks:
+      - e2e
+    command: [ "--config=/etc/otel-collector-config.yaml" ]
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
+    expose:
+      - 55678
+    depends_on:
+      oap:
+        condition: service_healthy
+  kong-mock:
+    image: curlimages/curl:8.10.1
+    networks:
+      - e2e
+    depends_on:
+      kong-1:
+        condition: service_healthy
+      oap:
+        condition: service_healthy
+    command: "sh /etc/kong-e2e-data-mock.sh"
+    volumes:
+      - ./kong-e2e-data-mock.sh:/etc/kong-e2e-data-mock.sh
+
+networks:
+  e2e:

test/e2e-v2/cases/kong/e2e.yaml

@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is used to show how to write configuration files and can be used to test.
+
+setup:
+  env: compose
+  file: docker-compose.yml
+  timeout: 20m
+  init-system-environment: ../../script/env
+  steps:
+    - name: set PATH
+      command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+    - name: install yq
+      command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
+    - name: install swctl
+      command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
+
+verify:
+  retry:
+    count: 20
+    interval: 3s
+  cases:
+    - includes:
+        - ./kong-cases.yaml

test/e2e-v2/cases/kong/expected/metrics-has-value-label-bandwidth-endpoint.yml

@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: direction
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-has-value-label-bandwidth-service.yml

@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: direction
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-has-value-label-bandwidth.yml

@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: direction
+          value: {{ .value }}
+        - key: route
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-has-value-label-percentile.yml

@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: p
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-has-value-label-pid.yml

@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: pid
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-has-value-label-shared-dict.yml

@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: shared_dict
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-has-value-label-state.yml

@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: state
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-has-value-label-status.yml

@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: code
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-has-value.yml

@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels: []
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-single-value-label.yml

@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: SINGLE_VALUE
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/metrics-single-value.yml

@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: SINGLE_VALUE
+results:
+  {{- contains .results }}
+  - metric:
+      labels: []
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null

test/e2e-v2/cases/kong/expected/service.yml

@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- contains . }}
+- id: {{ b64enc "kong::kong-cluster" }}.1
+  name: kong::kong-cluster
+  group: kong
+  shortname: kong-cluster
+  layers:
+    - KONG
+  normal: true
+{{- end }}

test/e2e-v2/cases/kong/kong-cases.yaml

@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is used to show how to write configuration files and can be used to test.
+
+cases:
+  # service list
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls
+    expected: expected/service.yml
+  # service metrics
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='latest(aggregate_labels(meter_kong_service_nginx_metric_errors_total,sum))' --service-name=kong::kong-cluster
+    expected: expected/metrics-single-value.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='latest(aggregate_labels(meter_kong_service_datastore_reachable,sum(service_instance_id)))' --service-name=kong::kong-cluster
+    expected: expected/metrics-single-value-label.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='aggregate_labels(meter_kong_service_http_bandwidth,sum(direction))' --service-name=kong::kong-cluster
+    expected: expected/metrics-has-value-label-bandwidth-service.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='aggregate_labels(meter_kong_service_http_status,sum(code))' --service-name=kong::kong-cluster
+    expected: expected/metrics-has-value-label-status.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='aggregate_labels(meter_kong_service_http_requests,sum)' --service-name=kong::kong-cluster
+    expected: expected/metrics-has-value.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='aggregate_labels(meter_kong_service_nginx_connections_total,sum(state))' --service-name=kong::kong-cluster
+    expected: expected/metrics-has-value-label-state.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='aggregate_labels(meter_kong_service_nginx_timers,sum(state))' --service-name=kong::kong-cluster
+    expected: expected/metrics-has-value-label-state.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='aggregate_labels(meter_kong_service_kong_latency,avg(p))' --service-name=kong::kong-cluster
+    expected: expected/metrics-has-value-label-percentile.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='aggregate_labels(meter_kong_service_request_latency,avg(p))' --service-name=kong::kong-cluster
+    expected: expected/metrics-has-value-label-percentile.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression='aggregate_labels(meter_kong_service_upstream_latency,avg(p))' --service-name=kong::kong-cluster
+    expected: expected/metrics-has-value-label-percentile.yml
+
+  # instance metrics
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_http_bandwidth --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-bandwidth.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_http_status --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-status.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_datastore_reachable --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_http_requests --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_shared_dict_bytes --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-shared-dict.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_shared_dict_total_bytes --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-shared-dict.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_memory_workers_lua_vms_bytes --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-pid.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_nginx_connections_total --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-state.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_nginx_timers --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-state.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_kong_latency --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-percentile.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_request_latency --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-percentile.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_instance_upstream_latency --service-name=kong::kong-cluster --instance-name=kong-1:8001
+    expected: expected/metrics-has-value-label-percentile.yml
+
+  # Endpoint metrics
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_endpoint_http_bandwidth --service-name=kong::kong-cluster --instance-name=kong-1:8001 --endpoint-name=sw_route
+    expected: expected/metrics-has-value-label-bandwidth-endpoint.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_endpoint_http_status --service-name=kong::kong-cluster --instance-name=kong-1:8001 --endpoint-name=sw_route
+    expected: expected/metrics-has-value-label-status.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_endpoint_kong_latency --service-name=kong::kong-cluster --instance-name=kong-1:8001 --endpoint-name=sw_route
+    expected: expected/metrics-has-value-label-percentile.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_endpoint_request_latency --service-name=kong::kong-cluster --instance-name=kong-1:8001 --endpoint-name=sw_route
+    expected: expected/metrics-has-value-label-percentile.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_kong_endpoint_upstream_latency --service-name=kong::kong-cluster --instance-name=kong-1:8001 --endpoint-name=sw_route
+    expected: expected/metrics-has-value-label-percentile.yml

test/e2e-v2/cases/kong/kong-e2e-data-mock.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This script is used to test the Kong plugin with the data mock service.
+
+# Add global plugins
+curl -i -X POST http://kong-1:8001/plugins \
+  --data name=prometheus \
+  --data config.per_consumer=true \
+  --data config.status_code_metrics=true \
+  --data config.ai_metrics=true \
+  --data config.latency_metrics=true \
+  --data config.bandwidth_metrics=true \
+  --data config.upstream_health_metrics=true
+curl -i -X POST http://kong-1:8001/plugins \
+  --data name=rate-limiting \
+  --data config.minute=5 \
+  --data config.policy=local
+
+# Add a mock service and route
+curl -i -s -X POST http://kong-1:8001/services \
+  --data name=sw_service \
+  --data url='http://httpbin.konghq.com'
+curl -i -X POST http://kong-1:8001/services/sw_service/routes \
+  --data 'paths[]=/mock' \
+  --data name=sw_route
+
+while true
+do
+  curl -s -i http://kong-1:8000/mock/anything > /dev/null
+  sleep 5
+done

test/e2e-v2/cases/kong/otel-collector-config.yaml

@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+receivers:
+  prometheus:
+    config:
+      scrape_configs:
+        - job_name: 'kong-monitoring'
+          metrics_path: "/metrics"
+          scrape_interval: 10s
+          static_configs:
+            - targets: ['kong-1:8001']
+              labels:
+                host_name: kong-cluster
+
+processors:
+  batch:
+
+exporters:
+  otlp:
+    endpoint: oap:11800
+    tls:
+      insecure: true
+service:
+  pipelines:
+    metrics:
+      receivers:
+        - prometheus
+      processors:
+        - batch
+      exporters:
+        - otlp