kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

SWIP-2 Collecting and Gathering Kubernetes Monitoring Data (#11681)

This commit is contained in:
mrproliu 2023-12-21 10:01:57 +08:00 committed by GitHub
parent 4c157b4642
commit 35e34ce8a2
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 125 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/en/changes/changes.md
View File

@ -22,6 +22,7 @@
* Polish docs structure. Move customization docs separately from the introduction docs.
* Add webhook/gRPC hooks settings example for `backend-alarm.md`.
* Begin the process of `SWIP - SkyWalking Improvement Proposal`.
* Add `SWIP-1 Create and detect Service Hierarchy Relationship`
* Add `SWIP-1 Create and detect Service Hierarchy Relationship`.
* Add `SWIP-2 Collecting and Gathering Kubernetes Monitoring Data`.
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/202?closed=1)

121
docs/en/swip/SWIP-2.md Normal file
View File

@ -0,0 +1,121 @@
# Motivation
SkyWalking has provided an access log collector based on the Agent layer and Service Mesh layer,
and can generate corresponding topology maps and metrics based on the data. However, the Kubernetes Layer still lacks
corresponding access log collector and analysis work.
This proposal is dedicated to collecting and analyzing network access logs in Kubernetes.
# Architecture Graph
There is no significant architecture-level change. Still using the Rover project to collect data and report it to
SkyWalking OAP using the gRPC protocol.
# Propose Changes
Based on the content in Motivation, if we want to ignore the application types(different program languages) and
only monitor network logs, using eBPF is a good choice. It mainly reflects in the following aspects:
1. Non-intrusive: When monitoring network access logs with eBPF, the application do not need to make any changes to be monitored.
2. Language-unrestricted: Regardless of which programming language is used in the application, network data will ultimately be accessed through [Linux Syscalls](https://linasm.sourceforge.net/docs/syscalls/network.php). Therefore, we can monitor network data by attaching eBPF to the syscalls layer, thus ignoring programming languages.
3. Kernel interception: Since eBPF can attach to the kernel methods, it can obtain the execution status of each packet at L2-L4 layers and generate more detailed metrics.
Based on these reasons and collected data, they can be implemented in SkyWalking Rover and collected and monitored based on the following steps:
1. Monitor the network execution status of all processes in Kubernetes when the Rover system starts.
2. Periodically report data content via [gRPC protocol](https://github.com/apache/skywalking-data-collect-protocol/blob/master/ebpf/accesslog.proto) to SkyWalking OAP.
3. SkyWalking OAP parses network access logs and generates corresponding network topology, metrics, etc.
## Limitation
For content that uses TLS for data transmission, Rover will detect whether the current language uses libraries
such as [OpenSSL](https://www.openssl.org/). If it is used, it will asynchronously intercept relevant OpenSSL methods
when the process starts to perceive the original data content.
However, this approach is not feasible for Java because Java does not use the OpenSSL library but performs encryption/decryption
through Java code. Currently, eBPF cannot intercept Java method calls. Therefore, it results in an inability to perceive
the TLS data protocol in Java.
### Service with Istio sidecar scenario
If the Service is deployed in Istio sidecar, it will still monitor each process.
If the Service is a Java service and uses TLS, it can analyze the relevant traffic generated in the sidecar (envoy).
# Imported Dependencies libs and their licenses.
No new library is planned to be added to the codebase.
# Compatibility
About the **protocol**, there should be no breaking changes, but enhancements only:
1. `Rover`: adding a new gRPC data collection protocol for reporting the [access logs](https://github.com/apache/skywalking-data-collect-protocol/blob/master/ebpf/accesslog.proto).
2. `OAP`: It should have no protocol updates. The existing query protocols are already sufficient for querying Kubernetes topology and metric data.
## Data Generation
### Entity
* service_traffic
| column | data type | value description |
|------------|-----------|-------------------------|
| name | string | kubernetes service name |
| short_name | string | same with name |
| service_id | string | base64(name).1 |
| group | string | empty string |
| layer | string | KUBERNETES |
* instance_traffic
| column | data type | value description |
|------------|-----------|------------------------------------------------|
| service_id | string | base64(service_name).1 |
| name | string | pod name |
| last_ping | long | last access log message timestamp(millisecond) |
| properties | json | empty string |
* endpoint_traffic
| column | data type | value description |
|------------|-----------|---------------------------------------------|
| service_id | string | base64(service_name).1 |
| name | string | access log endpoint name(for HTTP1, is URI) |
### Entity Relation
All entity information is built on connections. If the target address is remote, the name will be resolved in the following order:
1. If it is a pod IP, it will be resolved as pod information.
2. If it is a service IP, it will be resolved as service information.
3. If neither exists, only pod information will be displayed.
Different entities have different displays for remote addresses. Please refer to the following table.
| table name | remote info(display by following order) |
|-------------------|-----------------------------------------|
| service_relation | service name, remote IP address |
| instance_relation | pod name, remote IP address |
**NOTICE**: If it is the internal data interaction within the pod, such as exchanging data between services and sidecar (envoy),
no corresponding traffic will be generated. We only generate and interact with external pods.
#### Limitation
If the service IP is used to send requests to the upstream, we will use eBPF to perceive the real target PodIP by
perceiving relevant [conntrack records](https://en.wikipedia.org/wiki/Netfilter#Connection_tracking).
However, if conntrack technology is not used, it is difficult to perceive the real target IP address.
In this case, instance relation data of this kind will be **dropped,** but we will mark all discarded relationship
generation counts through a metric for better understanding of the situation.
### Metrics
Integrate the data into the OAL system and generate corresponding metrics through predefined data combined with OAL statements.
# General usage docs
This proposal will only add a module to Rover that explains the configuration of access logs, and changes in the Kubernetes module on the UI.
In the Kubernetes UI, users can see the following additions:
1. Topology: A topology diagram showing the calling relationships between services, instances, and processes.
2. Entity Metrics: Metric data for services, instances, and processes.
3. Call Relationship Metrics: Metrics for call relationships between different entities.

3
docs/en/swip/readme.md
View File

@ -57,6 +57,7 @@ All accepted and proposed SWIPs could be found in [here](https://github.com/apac
## Known SWIPs
Next SWIP Number: 2
Next SWIP Number: 3
- [SWIP-2 Collecting and Gathering Kubernetes Monitoring Data](SWIP-2.md)
- [SWIP-1 Create and detect Service Hierarchy Relationship](SWIP-1.md)