Add Kafka hierarchy (#11954)

docs/en/changes/changes.md

@@ -65,6 +65,9 @@
   - VIRTUAL_MQ -> RABBITMQ
 * Remove Column#function mechanism in the kernel.
 * Make query `readMetricValue` always return the average value of the duration.
+* Add Service Hierarchy auto matching layer relationships (upper -> lower) as following:
+  - KAFKA -> K8S_SERVICE
+  - VIRTUAL_MQ -> KAFKA
 
 #### UI
 

docs/en/concepts-and-designs/service-hierarchy.md

@@ -26,9 +26,11 @@ If you want to customize it according to your own needs, please refer to [Servic
 | NGINX            | K8S_SERVICE | [NGINX On K8S_SERVICE](#nginx-on-k8s_service)                     |
 | APISIX           | K8S_SERVICE | [APISIX On K8S_SERVICE](#apisix-on-k8s_service)                   |
 | ROCKETMQ         | K8S_SERVICE | [ROCKETMQ On K8S_SERVICE](#rocketmq-on-k8s_service)               |
-| VIRTUAL_MQ       | ROCKETMQ    | [VIRTUAL_MQ On ROCKETMQ](#virtual_mq-on-rocketmq)              |
 | RABBITMQ         | K8S_SERVICE | [RABBITMQ On K8S_SERVICE](#rabbitmq-on-k8s_service)               |
-| VIRTUAL_MQ       | RABBITMQ    | [VIRTUAL_MQ On RABBITMQ](#virtual_mq-on-rabbitmq)              |
+| KAFKA            | K8S_SERVICE | [KAFKA On K8S_SERVICE](#kafka-on-k8s_service)                     |
+| VIRTUAL_MQ       | RABBITMQ    | [VIRTUAL_MQ On RABBITMQ](#virtual_mq-on-rabbitmq)                 |
+| VIRTUAL_MQ       | ROCKETMQ    | [VIRTUAL_MQ On K8S_SERVICE](#virtual_mq-on-rocketmq)              |
+| VIRTUAL_MQ       | KAFKA       | [VIRTUAL_MQ On KAFKA](#virtual_mq-on-kafka)                 |
 
 - The following sections will describe the **default matching rules** in detail and use the `upper-layer On lower-layer` format. 
 - The example service name are based on SkyWalking [Showcase](https://github.com/apache/skywalking-showcase) default deployment.
@@ -162,6 +164,22 @@ If you want to customize it according to your own needs, please refer to [Servic
   - VIRTUAL_MQ.service.name: `rabbitmq.skywalking-showcase.svc.cluster.local:5672`
   - RABBITMQ.service.name: `rabbitmq::rabbitmq.skywalking-showcase`
 
+#### KAFKA On K8S_SERVICE
+- Rule name: `short-name`
+- Groovy script: `{ (u, l) -> u.shortName == l.shortName }`
+- Description: KAFKA.service.shortName == K8S_SERVICE.service.shortName
+- Matched Example:
+  - KAFKA.service.name: `kafka::kafka.skywalking-showcase`
+  - K8S_SERVICE.service.name: `skywalking-showcase::kafka.skywalking-showcase`
+
+#### VIRTUAL_MQ On KAFKA
+- Rule name: `lower-short-name-with-fqdn`
+- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }`
+- Description: VIRTUAL_MQ.service.shortName remove port == KAFKA.service.shortName with fqdn suffix
+- Matched Example:
+  - VIRTUAL_MQ.service.name: `kafka.skywalking-showcase.svc.cluster.local:9092`
+  - KAFKA.service.name: `kafka::rocketmq.skywalking-showcase`
+
 ### Build Through Specific Agents
 Use agent tech involved(such as eBPF) and deployment tools(such as operator and agent injector) detect the service hierarchy relations.
 

oap-server/server-starter/src/main/resources/hierarchy-definition.yml

@@ -51,6 +51,9 @@ hierarchy:
   RABBITMQ:
     K8S_SERVICE: short-name
 
+  KAFKA:
+    K8S_SERVICE: short-name
+
   VIRTUAL_DATABASE:
     MYSQL: lower-short-name-with-fqdn
     POSTGRESQL: lower-short-name-with-fqdn
@@ -58,6 +61,7 @@ hierarchy:
   VIRTUAL_MQ:
     ROCKETMQ: lower-short-name-with-fqdn
     RABBITMQ: lower-short-name-with-fqdn
+    KAFKA: lower-short-name-with-fqdn
 
 # Use Groovy script to define the matching rules, the input parameters are the upper service(u) and the lower service(l) and the return value is a boolean,
 # which are used to match the relation between the upper service(u) and the lower service(l) on the different layers.
@@ -88,6 +92,7 @@ layer-levels:
   NGINX: 2
   ROCKETMQ: 2
   RABBITMQ: 2
+  KAFKA: 2
 
   MESH_DP: 1
 

oap-server/server-starter/src/main/resources/ui-initialized-templates/kafka/kafka-cluster.json

@@ -219,6 +219,27 @@
 		"entity": "Service",
 		"name": "Kafka-Cluster",
 		"id": "Kafka-Cluster",
-		"isRoot": false
+		"isRoot": false,
+		"isDefault": true,
+		"expressions": [
+			"avg(meter_kafka_partition_count)",
+			"avg(meter_kafka_offline_partitions_count)",
+			"avg(meter_kafka_leader_count)",
+			"avg(meter_kafka_max_lag)"
+		],
+		"expressionsConfig": [
+			{
+				"label": "partition count"
+			},
+			{
+				"label": "offline partition count"
+			},
+			{
+				"label": "leader count"
+			},
+			{
+				"label": "max lag"
+			}
+		]
 	}
 }]