# Nginx monitoring
## Nginx performance from nginx-lua-prometheus
The [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus) is a lua library that can be used with Nginx to collect metrics 
and expose them on a separate web page. 
To use this library, you will need Nginx with [lua-nginx-module](https://github.com/openresty/lua-nginx-module) or directly [OpenResty](https://openresty.org/).

SkyWalking leverages OpenTelemetry Collector to transfer the metrics to [OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/mal.md).

### Data flow
1. [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus) collects metrics from Nginx and expose them to an endpoint.
2. OpenTelemetry Collector fetches metrics from the endpoint expose above via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.

### Set up
1. Collect Nginx metrics and expose the following four metrics by [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus). For details on metrics definition, refer to [here](../../../../test/e2e-v2/cases/nginx/nginx.conf).
- histogram: nginx_http_latency
- gauge: nginx_http_connections
- counter: nginx_http_size_bytes
- counter: nginx_http_requests_total

2. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#docker). For details on Prometheus Receiver in OpenTelemetry Collector, refer to [here](../../../../test/e2e-v2/cases/nginx/otel-collector-config.yaml).
3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).

### Nginx Monitoring

SkyWalking observes the status, payload, and latency of the Nginx server, which is cataloged as a `LAYER: Nginx` `Service` in the OAP and instances would be recognized as `LAYER: Nginx` `instance`.

About `LAYER: Nginx` `endpoint`, it depends on how precision you want to monitor the nginx.
We do not recommend expose every request path metrics, because it will cause explosion of metrics endpoint data.

You can collect host metrics:
```
http {
  log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", ngx.var.host})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", ngx.var.host})
      metric_requests:inc(1, {ngx.var.status, ngx.var.host})
      metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.host})
  }
}
```
or grouped urls and upstream metrics:
```
upstream backend {
  server ip:port;
}

server {

  location /test {
    default_type application/json;
    return 200  '{"code": 200, "message": "success"}';
  
    log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "/test/**"})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test/**"})
      metric_requests:inc(1, {ngx.var.status, "/test/**"})
      metric_latency:observe(tonumber(ngx.var.request_time), {"/test/**"})
    }
  }
  
  location /test_upstream {
  
    proxy_pass http://backend;
  
    log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "upstream/backend"})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "upstream/backend"})
      metric_requests:inc(1, {ngx.var.status, "upstream/backend"})
      metric_latency:observe(tonumber(ngx.var.request_time), {"upstream/backend"})
    }
  }
}
```

#### Nginx Service Supported Metrics 
| Monitoring Panel        | Unit | Metric Name                                                                                   | Catalog | Description                                          | Data Source                    |
|-------------------------|------|-----------------------------------------------------------------------------------------------|---------|------------------------------------------------------|--------------------------------|
| HTTP Request Trend      |      | meter_nginx_service_http_requests                                                             | Service | The increment rate of HTTP requests                  | nginx-lua-prometheus           |
| HTTP Latency            | ms   | meter_nginx_service_http_latency                                                              | Service | The increment rate of the latency of HTTP requests   | nginx-lua-prometheus           |
| HTTP Bandwidth          | KB   | meter_nginx_service_bandwidth                                                                 | Service | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus           |
| HTTP Connections        |      | meter_nginx_service_http_connections                                                          | Service | The avg number of the connections                    | nginx-lua-prometheus           |
| HTTP Status Trend       |      | meter_nginx_service_http_status                                                               | Service | The increment rate of the status of HTTP requests    | nginx-lua-prometheus           |
| HTTP Status 4xx Percent | %    | meter_nginx_service_http_4xx_requests_increment / meter_nginx_service_http_requests_increment | Service | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus           |
| HTTP Status 5xx Percent | %    | meter_nginx_service_http_5xx_requests_increment / meter_nginx_service_http_requests_increment | Service | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus           |

#### Nginx Instance Supported Metrics
| Monitoring Panel          | Unit | Metric Name                                                                                     | Catalog  | Description                                          | Data Source                    |
|---------------------------|------|-------------------------------------------------------------------------------------------------|----------|------------------------------------------------------|--------------------------------|
| HTTP Request Trend        |      | meter_nginx_instance_http_requests                                                              | Instance | The increment rate of HTTP requests                  | nginx-lua-prometheus           |
| HTTP Latency              | ms   | meter_nginx_instance_http_latency                                                               | Instance | The increment rate of the latency of HTTP requests   | nginx-lua-prometheus           |
| HTTP Bandwidth            | KB   | meter_nginx_instance_bandwidth                                                                  | Instance | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus           |
| HTTP Connections          |      | meter_nginx_instance_http_connections                                                           | Instance | The avg number of the connections                    | nginx-lua-prometheus           |
| HTTP Status Trend         |      | meter_nginx_instance_http_status                                                                | Instance | The increment rate of the status of HTTP requests    | nginx-lua-prometheus           |
| HTTP Status 4xx Percent   | %    | meter_nginx_instance_http_4xx_requests_increment / meter_nginx_instance_http_requests_increment | Instance | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus           |
| HTTP Status 5xx Percent   | %    | meter_nginx_instance_http_5xx_requests_increment / meter_nginx_instance_http_requests_increment | Instance | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus           |

#### Nginx Endpoint Supported Metrics
| Monitoring Panel        | Unit | Metric Name                                                                                     | Catalog  | Description                                          | Data Source          |
|-------------------------|------|-------------------------------------------------------------------------------------------------|----------|------------------------------------------------------|----------------------|
| HTTP Request Trend      |      | meter_nginx_endpoint_http_requests                                                              | Endpoint | The increment rate of HTTP requests                  | nginx-lua-prometheus |
| HTTP Latency            | ms   | meter_nginx_endpoint_http_latency                                                               | Endpoint | The increment rate of the latency of HTTP requests   | nginx-lua-prometheus |
| HTTP Bandwidth          | KB   | meter_nginx_endpoint_bandwidth                                                                  | Endpoint | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus |
| HTTP Status Trend       |      | meter_nginx_endpoint_http_status                                                                | Endpoint | The increment rate of the status of HTTP requests    | nginx-lua-prometheus |
| HTTP Status 4xx Percent | %    | meter_nginx_endpoint_http_4xx_requests_increment / meter_nginx_endpoint_http_requests_increment | Endpoint | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus |
| HTTP Status 5xx Percent | %    | meter_nginx_endpoint_http_5xx_requests_increment / meter_nginx_endpoint_http_requests_increment | Endpoint | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus |

### Customizations
You can customize your own metrics/expression/dashboard panel.

The metrics definition and expression rules are found in `/config/otel-rules/nginx-service.yaml, /config/otel-rules/nginx-instance.yaml, /config/otel-rules/nginx-endpoint.yaml`.

The Nginx dashboard panel configurations are found in `/config/ui-initialized-templates/nginx`.

## Collect nginx access and error log
SkyWalking leverages [fluentbit](https://fluentbit.io/) or other log agents for collecting access log and error log of Nginx.

### Data flow
1. fluentbit agent collects access log and error log from Nginx.
2. fluentbit agent sends data to SkyWalking OAP Server using native meter APIs via HTTP.
3. The SkyWalking OAP Server parses the expression with [LAL](../../concepts-and-designs/lal.md) to parse/extract and store the results.

### Set up
1. Install [fluentbit](https://docs.fluentbit.io/manual/installation/docker).
2. Config fluent bit with fluent-bit.conf, refer to [here](../../../../test/e2e-v2/cases/nginx/fluent-bit.conf).

### Error Log Monitoring
Error Log monitoring provides monitoring of the error.log of the Nginx server.

#### Supported Metrics
| Monitoring Panel         | Metric Name                          | Catalog  | Description                               | Data Source |
|--------------------------|--------------------------------------|----------|-------------------------------------------|-------------|
| Service Error Log Count  | meter_nginx_service_error_log_count  | Service  | The count of log level of nginx error.log | fluent bit  |
| Instance Error Log Count | meter_nginx_instance_error_log_count | Instance | The count of log level of nginx error.log | fluent bit  |

### Customizations
You can customize your own metrics/expression/dashboard panel.

The log collect and analyse rules are found in `/config/lal/nginx.yaml`, `/config/log-mal-rules/nginx.yaml`.

The Nginx dashboard panel configurations are found in `/config/ui-initialized-templates/nginx`.