124 lines
6.6 KiB
Markdown
124 lines
6.6 KiB
Markdown
# Collecting and Gathering Kubernetes Monitoring Data
|
|
|
|
## Motivation
|
|
SkyWalking has provided an access log collector based on the Agent layer and Service Mesh layer,
|
|
and can generate corresponding topology maps and metrics based on the data. However, the Kubernetes Layer still lacks
|
|
corresponding access log collector and analysis work.
|
|
|
|
This proposal is dedicated to collecting and analyzing network access logs in Kubernetes.
|
|
|
|
## Architecture Graph
|
|
There is no significant architecture-level change. Still using the Rover project to collect data and report it to
|
|
SkyWalking OAP using the gRPC protocol.
|
|
|
|
## Propose Changes
|
|
|
|
Based on the content in Motivation, if we want to ignore the application types(different program languages) and
|
|
only monitor network logs, using eBPF is a good choice. It mainly reflects in the following aspects:
|
|
|
|
1. Non-intrusive: When monitoring network access logs with eBPF, the application do not need to make any changes to be monitored.
|
|
2. Language-unrestricted: Regardless of which programming language is used in the application, network data will ultimately be accessed through [Linux Syscalls](https://linasm.sourceforge.net/docs/syscalls/network.php). Therefore, we can monitor network data by attaching eBPF to the syscalls layer, thus ignoring programming languages.
|
|
3. Kernel interception: Since eBPF can attach to the kernel methods, it can obtain the execution status of each packet at L2-L4 layers and generate more detailed metrics.
|
|
|
|
Based on these reasons and collected data, they can be implemented in SkyWalking Rover and collected and monitored based on the following steps:
|
|
|
|
1. Monitor the network execution status of all processes in Kubernetes when the Rover system starts.
|
|
2. Periodically report data content via [gRPC protocol](https://github.com/apache/skywalking-data-collect-protocol/blob/master/ebpf/accesslog.proto) to SkyWalking OAP.
|
|
3. SkyWalking OAP parses network access logs and generates corresponding network topology, metrics, etc.
|
|
|
|
### Limitation
|
|
|
|
For content that uses TLS for data transmission, Rover will detect whether the current language uses libraries
|
|
such as [OpenSSL](https://www.openssl.org/). If it is used, it will asynchronously intercept relevant OpenSSL methods
|
|
when the process starts to perceive the original data content.
|
|
|
|
However, this approach is not feasible for Java because Java does not use the OpenSSL library but performs encryption/decryption
|
|
through Java code. Currently, eBPF cannot intercept Java method calls. Therefore, it results in an inability to perceive
|
|
the TLS data protocol in Java.
|
|
|
|
#### Service with Istio sidecar scenario
|
|
|
|
If the Service is deployed in Istio sidecar, it will still monitor each process.
|
|
If the Service is a Java service and uses TLS, it can analyze the relevant traffic generated in the sidecar (envoy).
|
|
|
|
## Imported Dependencies libs and their licenses.
|
|
|
|
No new library is planned to be added to the codebase.
|
|
|
|
## Compatibility
|
|
|
|
About the **protocol**, there should be no breaking changes, but enhancements only:
|
|
|
|
1. `Rover`: adding a new gRPC data collection protocol for reporting the [access logs](https://github.com/apache/skywalking-data-collect-protocol/blob/master/ebpf/accesslog.proto).
|
|
2. `OAP`: It should have no protocol updates. The existing query protocols are already sufficient for querying Kubernetes topology and metric data.
|
|
|
|
### Data Generation
|
|
|
|
#### Entity
|
|
|
|
* service_traffic
|
|
|
|
| column | data type | value description |
|
|
|------------|-----------|-------------------------|
|
|
| name | string | kubernetes service name |
|
|
| short_name | string | same with name |
|
|
| service_id | string | base64(name).1 |
|
|
| group | string | empty string |
|
|
| layer | string | KUBERNETES |
|
|
|
|
* instance_traffic
|
|
|
|
| column | data type | value description |
|
|
|------------|-----------|------------------------------------------------|
|
|
| service_id | string | base64(service_name).1 |
|
|
| name | string | pod name |
|
|
| last_ping | long | last access log message timestamp(millisecond) |
|
|
| properties | json | empty string |
|
|
|
|
* endpoint_traffic
|
|
|
|
| column | data type | value description |
|
|
|------------|-----------|---------------------------------------------|
|
|
| service_id | string | base64(service_name).1 |
|
|
| name | string | access log endpoint name(for HTTP1, is URI) |
|
|
|
|
#### Entity Relation
|
|
|
|
All entity information is built on connections. If the target address is remote, the name will be resolved in the following order:
|
|
|
|
1. If it is a pod IP, it will be resolved as pod information.
|
|
2. If it is a service IP, it will be resolved as service information.
|
|
3. If neither exists, only pod information will be displayed.
|
|
|
|
Different entities have different displays for remote addresses. Please refer to the following table.
|
|
|
|
| table name | remote info(display by following order) |
|
|
|-------------------|-----------------------------------------|
|
|
| service_relation | service name, remote IP address |
|
|
| instance_relation | pod name, remote IP address |
|
|
|
|
**NOTICE**: If it is the internal data interaction within the pod, such as exchanging data between services and sidecar (envoy),
|
|
no corresponding traffic will be generated. We only generate and interact with external pods.
|
|
|
|
##### Limitation
|
|
|
|
If the service IP is used to send requests to the upstream, we will use eBPF to perceive the real target PodIP by
|
|
perceiving relevant [conntrack records](https://en.wikipedia.org/wiki/Netfilter#Connection_tracking).
|
|
|
|
However, if conntrack technology is not used, it is difficult to perceive the real target IP address.
|
|
In this case, instance relation data of this kind will be **dropped,** but we will mark all discarded relationship
|
|
generation counts through a metric for better understanding of the situation.
|
|
|
|
#### Metrics
|
|
|
|
Integrate the data into the OAL system and generate corresponding metrics through predefined data combined with OAL statements.
|
|
|
|
## General usage docs
|
|
|
|
This proposal will only add a module to Rover that explains the configuration of access logs, and changes in the Kubernetes module on the UI.
|
|
|
|
In the Kubernetes UI, users can see the following additions:
|
|
1. Topology: A topology diagram showing the calling relationships between services, instances, and processes.
|
|
2. Entity Metrics: Metric data for services, instances, and processes.
|
|
3. Call Relationship Metrics: Metrics for call relationships between different entities.
|