From 35e34ce8a22bd7363cef7ef1bb2b364a59a54427 Mon Sep 17 00:00:00 2001 From: mrproliu <741550557@qq.com> Date: Thu, 21 Dec 2023 10:01:57 +0800 Subject: [PATCH] SWIP-2 Collecting and Gathering Kubernetes Monitoring Data (#11681) --- docs/en/changes/changes.md | 3 +- docs/en/swip/SWIP-2.md | 121 +++++++++++++++++++++++++++++++++++++ docs/en/swip/readme.md | 3 +- 3 files changed, 125 insertions(+), 2 deletions(-) create mode 100644 docs/en/swip/SWIP-2.md diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 161f311ec2..2d0b0a931d 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -22,6 +22,7 @@ * Polish docs structure. Move customization docs separately from the introduction docs. * Add webhook/gRPC hooks settings example for `backend-alarm.md`. * Begin the process of `SWIP - SkyWalking Improvement Proposal`. -* Add `SWIP-1 Create and detect Service Hierarchy Relationship` +* Add `SWIP-1 Create and detect Service Hierarchy Relationship`. +* Add `SWIP-2 Collecting and Gathering Kubernetes Monitoring Data`. All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/202?closed=1) diff --git a/docs/en/swip/SWIP-2.md b/docs/en/swip/SWIP-2.md new file mode 100644 index 0000000000..37baa6c740 --- /dev/null +++ b/docs/en/swip/SWIP-2.md @@ -0,0 +1,121 @@ +# Motivation +SkyWalking has provided an access log collector based on the Agent layer and Service Mesh layer, +and can generate corresponding topology maps and metrics based on the data. However, the Kubernetes Layer still lacks +corresponding access log collector and analysis work. + +This proposal is dedicated to collecting and analyzing network access logs in Kubernetes. + +# Architecture Graph +There is no significant architecture-level change. Still using the Rover project to collect data and report it to +SkyWalking OAP using the gRPC protocol. + +# Propose Changes + +Based on the content in Motivation, if we want to ignore the application types(different program languages) and +only monitor network logs, using eBPF is a good choice. It mainly reflects in the following aspects: + +1. Non-intrusive: When monitoring network access logs with eBPF, the application do not need to make any changes to be monitored. +2. Language-unrestricted: Regardless of which programming language is used in the application, network data will ultimately be accessed through [Linux Syscalls](https://linasm.sourceforge.net/docs/syscalls/network.php). Therefore, we can monitor network data by attaching eBPF to the syscalls layer, thus ignoring programming languages. +3. Kernel interception: Since eBPF can attach to the kernel methods, it can obtain the execution status of each packet at L2-L4 layers and generate more detailed metrics. + +Based on these reasons and collected data, they can be implemented in SkyWalking Rover and collected and monitored based on the following steps: + +1. Monitor the network execution status of all processes in Kubernetes when the Rover system starts. +2. Periodically report data content via [gRPC protocol](https://github.com/apache/skywalking-data-collect-protocol/blob/master/ebpf/accesslog.proto) to SkyWalking OAP. +3. SkyWalking OAP parses network access logs and generates corresponding network topology, metrics, etc. + +## Limitation + +For content that uses TLS for data transmission, Rover will detect whether the current language uses libraries +such as [OpenSSL](https://www.openssl.org/). If it is used, it will asynchronously intercept relevant OpenSSL methods +when the process starts to perceive the original data content. + +However, this approach is not feasible for Java because Java does not use the OpenSSL library but performs encryption/decryption +through Java code. Currently, eBPF cannot intercept Java method calls. Therefore, it results in an inability to perceive +the TLS data protocol in Java. + +### Service with Istio sidecar scenario + +If the Service is deployed in Istio sidecar, it will still monitor each process. +If the Service is a Java service and uses TLS, it can analyze the relevant traffic generated in the sidecar (envoy). + +# Imported Dependencies libs and their licenses. + +No new library is planned to be added to the codebase. + +# Compatibility + +About the **protocol**, there should be no breaking changes, but enhancements only: + +1. `Rover`: adding a new gRPC data collection protocol for reporting the [access logs](https://github.com/apache/skywalking-data-collect-protocol/blob/master/ebpf/accesslog.proto). +2. `OAP`: It should have no protocol updates. The existing query protocols are already sufficient for querying Kubernetes topology and metric data. + +## Data Generation + +### Entity + +* service_traffic + +| column | data type | value description | +|------------|-----------|-------------------------| +| name | string | kubernetes service name | +| short_name | string | same with name | +| service_id | string | base64(name).1 | +| group | string | empty string | +| layer | string | KUBERNETES | + +* instance_traffic + +| column | data type | value description | +|------------|-----------|------------------------------------------------| +| service_id | string | base64(service_name).1 | +| name | string | pod name | +| last_ping | long | last access log message timestamp(millisecond) | +| properties | json | empty string | + +* endpoint_traffic + +| column | data type | value description | +|------------|-----------|---------------------------------------------| +| service_id | string | base64(service_name).1 | +| name | string | access log endpoint name(for HTTP1, is URI) | + +### Entity Relation + +All entity information is built on connections. If the target address is remote, the name will be resolved in the following order: + +1. If it is a pod IP, it will be resolved as pod information. +2. If it is a service IP, it will be resolved as service information. +3. If neither exists, only pod information will be displayed. + +Different entities have different displays for remote addresses. Please refer to the following table. + +| table name | remote info(display by following order) | +|-------------------|-----------------------------------------| +| service_relation | service name, remote IP address | +| instance_relation | pod name, remote IP address | + +**NOTICE**: If it is the internal data interaction within the pod, such as exchanging data between services and sidecar (envoy), +no corresponding traffic will be generated. We only generate and interact with external pods. + +#### Limitation + +If the service IP is used to send requests to the upstream, we will use eBPF to perceive the real target PodIP by +perceiving relevant [conntrack records](https://en.wikipedia.org/wiki/Netfilter#Connection_tracking). + +However, if conntrack technology is not used, it is difficult to perceive the real target IP address. +In this case, instance relation data of this kind will be **dropped,** but we will mark all discarded relationship +generation counts through a metric for better understanding of the situation. + +### Metrics + +Integrate the data into the OAL system and generate corresponding metrics through predefined data combined with OAL statements. + +# General usage docs + +This proposal will only add a module to Rover that explains the configuration of access logs, and changes in the Kubernetes module on the UI. + +In the Kubernetes UI, users can see the following additions: +1. Topology: A topology diagram showing the calling relationships between services, instances, and processes. +2. Entity Metrics: Metric data for services, instances, and processes. +3. Call Relationship Metrics: Metrics for call relationships between different entities. diff --git a/docs/en/swip/readme.md b/docs/en/swip/readme.md index 2acae7d5f9..5a75c9c5b8 100644 --- a/docs/en/swip/readme.md +++ b/docs/en/swip/readme.md @@ -57,6 +57,7 @@ All accepted and proposed SWIPs could be found in [here](https://github.com/apac ## Known SWIPs -Next SWIP Number: 2 +Next SWIP Number: 3 +- [SWIP-2 Collecting and Gathering Kubernetes Monitoring Data](SWIP-2.md) - [SWIP-1 Create and detect Service Hierarchy Relationship](SWIP-1.md) \ No newline at end of file