# RabbitMQ monitoring

SkyWalking leverages `rabbitmq_prometheus` plugin for collecting metrics data from RabbitMQ. It leverages OpenTelemetry
Collector to transfer the metrics to
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/mal.md).

## Data flow

1. The `rabbitmq_prometheus` plugin collect metrics data from RabbitMQ. Note: The RabbitMQ version is required to be
   3.8.0+.
   The `rabbitmq_prometheus` plugin is built-in since RabbitMQ v3.8.0.
2. OpenTelemetry Collector fetches metrics from `rabbitmq_prometheus` plugin via Prometheus Receiver and pushes metrics
   to
   SkyWalking OAP Server via OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to
   filter/calculate/aggregate and store the results.

## Setup

1. Setup [rabbitmq_prometheus](https://www.rabbitmq.com/prometheus.html#installation).
2. Set up [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/#kubernetes). The example
   for OpenTelemetry Collector configuration, refer
   to [here](../../../../test/e2e-v2/cases/rabbitmq/otel-collector-config.yaml).
3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).

## RabbitMQ Monitoring

RabbitMQ monitoring provides multidimensional metrics monitoring of RabbitMQ cluster as `Layer: RABBITMQ` `Service` in
the OAP. In each cluster, the nodes are represented as `Instance`.

### RabbitMQ Cluster Supported Metrics

| Monitoring Panel                                    | Metric Name                                                  | Description                                                                                                                                                                                                                                                                                  | Data Source                |
|-----------------------------------------------------|--------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------|
| Memory Available Before Publishers Blocked (MB)     | meter_rabbitmq_memory_available_before_publisher_blocked     | If the value is zero or less, the memory alarm will be triggered and all publishing connections across all cluster nodes will be blocked.                                                                                                                                                    | rabbitmq_prometheus plugin |
| Disk Space Available Before Publishers Blocked (GB) | meter_rabbitmq_disk_space_available_before_publisher_blocked | This metric is reported for the partition where the RabbitMQ data directory is stored.                                                                                                                                                                                                       | rabbitmq_prometheus plugin |
| File Descriptors Available                          | meter_rabbitmq_file_descriptors_available                    | When this value reaches zero, new connections will not be accepted and disk write operations may fail.                                                                                                                                                                                       | rabbitmq_prometheus plugin |
| TCP Sockets Available                               | meter_rabbitmq_tcp_socket_available                          | When this value reaches zero, new connections will not be accepted.                                                                                                                                                                                                                          | rabbitmq_prometheus plugin |
| Messages Ready To Be Delivered To Consumers         | meter_rabbitmq_message_ready_delivered_consumers             | Total number of ready messages ready to be delivered to consumers.                                                                                                                                                                                                                           | rabbitmq_prometheus plugin |
| Messages Pending Consumer Acknowledgement           | meter_rabbitmq_message_unacknowledged_delivered_consumers    | The total number of messages that are either in-flight to consumers, currently being processed by consumers or simply waiting for the consumer acknowledgements to be processed by the queue. Until the queue processes the message acknowledgement, the message will remain unacknowledged. | rabbitmq_prometheus plugin |
| Messages Published                                  | meter_rabbitmq_messages_published                            | The incoming message rate before any routing rules are applied.                                                                                                                                                                                                                              | rabbitmq_prometheus plugin |
| Messages Confirmed To Publishers                    | meter_rabbitmq_messages_confirmed                            | The rate of messages confirmed by the broker to publishers. Publishers must opt-in to receive message confirmations.                                                                                                                                                                         | rabbitmq_prometheus plugin |
| Messages Unconfirmed To Publishers                  | meter_rabbitmq_messages_unconfirmed                          | The rate of messages received from publishers that have publisher confirms enabled and the broker has not confirmed yet.                                                                                                                                                                     | rabbitmq_prometheus plugin |
| Messages Routed To Queues                           | meter_rabbitmq_messages_routed                               | The rate of messages received from publishers and successfully routed to the master queue replicas.                                                                                                                                                                                          | rabbitmq_prometheus plugin |
| Unroutable Messages Returned To Publishers          | meter_rabbitmq_messages_unroutable_returned                  | The rate of messages that cannot be routed and are returned back to publishers.                                                                                                                                                                                                              | rabbitmq_prometheus plugin |
| Unroutable Messages Dropped                         | meter_rabbitmq_messages_unroutable_dropped                   | The rate of messages that cannot be routed and are dropped.                                                                                                                                                                                                                                  | rabbitmq_prometheus plugin |
| Queues Total                                        | meter_rabbitmq_queues                                        | Total number of queue masters per node.                                                                                                                                                                                                                                                      | rabbitmq_prometheus plugin |
| Queues Declared                                     | meter_rabbitmq_queues_declared_total                         | The rate of queue declarations performed by clients.                                                                                                                                                                                                                                         | rabbitmq_prometheus plugin |
| Queues Created                                      | meter_rabbitmq_queues_created_total                          | The rate of new queues created (as opposed to redeclarations).                                                                                                                                                                                                                               | rabbitmq_prometheus plugin |
| Queues Deleted                                      | meter_rabbitmq_queues_deleted_total                          | The rate of queues deleted.                                                                                                                                                                                                                                                                  | rabbitmq_prometheus plugin |
| Channels Total                                      | meter_rabbitmq_channels                                      | Total number of channels on all currently opened connections.                                                                                                                                                                                                                                | rabbitmq_prometheus plugin |
| Channels Opened                                     | meter_rabbitmq_channels_opened_total                         | The rate of new channels opened by applications across all connections. Channels are expected to be long-lived.                                                                                                                                                                              | rabbitmq_prometheus plugin |
| Channels Closed                                     | meter_rabbitmq_channels_closed_total                         | The rate of channels closed by applications across all connections. Channels are expected to be long-lived.                                                                                                                                                                                  | rabbitmq_prometheus plugin |
| Connections Total                                   | meter_rabbitmq_connections                                   | Total number of client connections.                                                                                                                                                                                                                                                          | rabbitmq_prometheus plugin |
| Connections Opened                                  | meter_rabbitmq_connections_opened_total                      | The rate of new connections opened by clients. Connections are expected to be long-lived.                                                                                                                                                                                                    | rabbitmq_prometheus plugin |
| Connections Closed                                  | meter_rabbitmq_connections_closed_total                      | The rate of connections closed. Connections are expected to be long-lived.                                                                                                                                                                                                                   | rabbitmq_prometheus plugin |

### RabbitMQ Node Supported Metrics

| Monitoring Panel        | Unit | Metric Name                                                                                                                                         | Description                                                        | Data Source                |
|-------------------------|------|-----------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|----------------------------|
| Ready Messages          |      | meter_rabbitmq_node_queue_messages_ready                                                                                                            | Total number of ready messages ready to be delivered to consumers. | rabbitmq_prometheus plugin |
| Unacknowledged Messages |      | meter_rabbitmq_node_unacknowledged_messages                                                                                                         | Messages delivered to consumers but not yet acknowledged           | rabbitmq_prometheus plugin |
| Incoming Messages       |      | meter_rabbitmq_node_incoming_messages                                                                                                               | The incoming message rate before any routing rules are applied.    | rabbitmq_prometheus plugin |
| Outgoing Messages       |      | meter_rabbitmq_node_outgoing_messages_total                                                                                                         | The outgoing message rate before any routing rules are applied.    | rabbitmq_prometheus plugin |
| Publishers              |      | meter_rabbitmq_node_publisher_total                                                                                                                 | Publishers                                                         | rabbitmq_prometheus plugin |
| Consumers               |      | meter_rabbitmq_node_consumer_total                                                                                                                  | Consumers currently connect                                        | rabbitmq_prometheus plugin |
| Collections             |      | meter_rabbitmq_node_connections_total                                                                                                               | Connections currently open                                         | rabbitmq_prometheus plugin |
| Channels                |      | meter_rabbitmq_node_channel_total                                                                                                                   | Channels currently open                                            | rabbitmq_prometheus plugin |
| Queues                  |      | meter_rabbitmq_node_queue_total                                                                                                                     | Queues available                                                   | rabbitmq_prometheus plugin |
| Allocated Used          | %    | meter_rabbitmq_node_allocated_used_percent                                                                                                          | Erlang VM memory utilisation from erts_alloc perspective           | rabbitmq_prometheus plugin |
| Allocated Unused        | %    | meter_rabbitmq_node_allocated_unused_percent                                                                                                        | Erlang VM memory utilisation from erts_alloc perspective           | rabbitmq_prometheus plugin |
| Allocated Used          | MB   | meter_rabbitmq_node_allocated_used_bytes                                                                                                            | Erlang VM memory utilisation from erts_alloc perspective           | rabbitmq_prometheus plugin |
| Allocated Unused        | MB   | meter_rabbitmq_node_allocated_unused_bytes                                                                                                          | Erlang VM memory utilisation from erts_alloc perspective           | rabbitmq_prometheus plugin |
| Allocated Total         | MB   | meter_rabbitmq_node_allocated_total_bytes                                                                                                           | Erlang VM memory utilisation from erts_alloc perspective           | rabbitmq_prometheus plugin |
| Resident Set Size       | MB   | meter_rabbitmq_node_process_resident_memory_bytes                                                                                                   | Erlang VM Resident Set Size (RSS) As reported by the OS            | rabbitmq_prometheus plugin |
| Allocators              | MB   | meter_rabbitmq_node_allocated_unused_bytes <br />meter_rabbitmq_node_allocated_total_bytes <br /> meter_rabbitmq_node_process_resident_memory_bytes |                                                                    | rabbitmq_prometheus plugin |
| Allocated By Type       | MB   | meter_rabbitmq_node_allocated_by_type                                                                                                               | Allocated by allocator type                                        | rabbitmq_prometheus plugin |
| Multiblock Used         | MB   | meter_rabbitmq_node_allocated_multiblock_used                                                                                                       | Multi block used                                                   | rabbitmq_prometheus plugin |
| Multiblock Unused       | MB   | meter_rabbitmq_node_allocated_multiblock_unused                                                                                                     | Multi block used                                                   | rabbitmq_prometheus plugin |
| Multiblock Pool Used    | MB   | meter_rabbitmq_node_allocated_multiblock_pool_used                                                                                                  | Multi block pool used                                              | rabbitmq_prometheus plugin |
| Multiblock Pool Unused  | MB   | meter_rabbitmq_node_allocated_multiblock_pool_unused                                                                                                | Multi block pool unused                                            | rabbitmq_prometheus plugin |
| Singleblock Used        | MB   | meter_rabbitmq_node_allocated_singleblock_used                                                                                                      | Single block used                                                  | rabbitmq_prometheus plugin |
| Singleblock Unused      | MB   | meter_rabbitmq_node_allocated_singleblock_unused                                                                                                    | Single block unused                                                | rabbitmq_prometheus plugin |

## Customizations

You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found
in `/config/otel-rules/rabbitmq/rabbitmq-cluster.yaml, /config/otel-rules/rabbitmq/rabbitmq-node.yaml`.
The RabbitMQ dashboard panel configurations are found in `/config/ui-initialized-templates/rabbitmq`.
