67 lines
2.9 KiB
Markdown
67 lines
2.9 KiB
Markdown
# Alarm
|
|
Alarm core is driven a collection of rules, which are defined in `config/alarm-settings.yml`.
|
|
There are two parts in alarm rule definition.
|
|
1. Alarm rules. They define how metric alarm should be triggered, what conditions should be considered.
|
|
1. Webhooks. The list of web service endpoint, which should be called after the alarm is triggered.
|
|
|
|
## Rules
|
|
Alarm rule is constituted by following keys
|
|
- **Rule name**. Unique name, show in alarm message. Must end with `_rule`.
|
|
- **Indicator name**. A.K.A. metric name in oal script. Only long, double, int types are supported. See
|
|
[List of all potential metric name](#list-of-all-potential-metric-name).
|
|
- **Include names**. The following entity names are included in this rule. Such as Service name,
|
|
endpoint name.
|
|
- **Threshold**. The target value.
|
|
- **OP**. Operator, support `>`, `<`, `=`. Welcome to contribute all OPs.
|
|
- **Period**. How long should the alarm rule should be checked. This is a time window, which goes with the
|
|
backend deployment env time.
|
|
- **Count**. In the period window, if the number of **value**s over threshold(by OP), reaches count, alarm
|
|
should send.
|
|
- **Silence period**. After alarm is triggered in Time-N, then keep silence in the **TN -> TN + period**.
|
|
By default, it is as same as **Period**, which means in a period, same alarm(same ID in same
|
|
indicator name) will be trigger once.
|
|
|
|
|
|
```yaml
|
|
rules:
|
|
# Rule unique name, must be ended with `_rule`.
|
|
endpoint_percent_rule:
|
|
# Indicator value need to be long, double or int
|
|
indicator-name: endpoint_percent
|
|
threshold: 75
|
|
op: <
|
|
# The length of time to evaluate the metric
|
|
period: 10
|
|
# How many times after the metric match the condition, will trigger alarm
|
|
count: 3
|
|
# How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
|
|
silence-period: 10
|
|
|
|
service_percent_rule:
|
|
indicator-name: service_percent
|
|
# [Optional] Default, match all services in this indicator
|
|
include-names:
|
|
- service_a
|
|
- service_b
|
|
threshold: 85
|
|
op: <
|
|
period: 10
|
|
count: 4
|
|
```
|
|
|
|
## Default alarm rules
|
|
We provided a default `alarm-setting.yml` in our distribution only for convenience, which including following rules
|
|
1. Service average response time over 1s in last 3 minutes.
|
|
1. Service success rate lower than 80% in last 2 minutes.
|
|
1. Service 90% response time is lower than 1000ms in last 3 minutes
|
|
1. Service Instance average response time over 1s in last 2 minutes.
|
|
1. Endpoint average response time over 1s in last 2 minutes.
|
|
|
|
|
|
|
|
## List of all potential metric name
|
|
The metric names are defined in official [OAL scripts](../../guides/backend-oal-scripts.md), right now
|
|
metric from **Service**, **Service Instance**, **Endpoint** scopes could be used in Alarm, we will extend in further versions.
|
|
|
|
Submit issue or pull request if you want to support any other scope in alarm.
|