47 lines
3.2 KiB
Markdown
47 lines
3.2 KiB
Markdown
# Trace Sampling at server side
|
|
When we run a distributed tracing system, the trace bring us detailed info, but cost a lot at storage.
|
|
Open server side trace sampling mechanism, the metrics of service, service instance, endpoint and topology are all accurate
|
|
as before, but only don't save all the traces into storage.
|
|
|
|
Of course, even you open sampling, the traces will be kept as consistent as possible. **Consistent** means, once the trace
|
|
segments have been collected and reported by agents, the backend would do their best to don't break the trace. See [Recommendation](#recommendation)
|
|
to understand why we called it `as consistent as possible` and `do their best to don't break the trace`.
|
|
|
|
## Set the sample rate
|
|
In **agent-analyzer** module, you will find `sampleRate` setting.
|
|
|
|
```yaml
|
|
agent-analyzer:
|
|
default:
|
|
...
|
|
sampleRate: ${SW_TRACE_SAMPLE_RATE:1000} # The sample rate precision is 1/10000. 10000 means 100% sample in default.
|
|
forceSampleErrorSegment: ${SW_FORCE_SAMPLE_ERROR_SEGMENT:true} # When sampling mechanism activated, this config would make the error status segment sampled, ignoring the sampling rate.
|
|
slowTraceSegmentThreshold: ${SW_SLOW_TRACE_SEGMENT_THRESHOLD:-1} # Setting this threshold about the latency would make the slow trace segments sampled if they cost more time, even the sampling mechanism activated. The default value is `-1`, which means would not sample slow traces. Unit, millisecond.
|
|
```
|
|
|
|
`sampleRate` is for you to set sample rate to this backend.
|
|
The sample rate precision is 1/10000. 10000 means 100% sample in default.
|
|
|
|
`forceSampleErrorSegment` is for you to save all error segments when sampling mechanism actived.
|
|
When sampling mechanism activated, this config would make the error status segment sampled, ignoring the sampling rate.
|
|
|
|
`slowTraceSegmentThreshold` is for you to save all slow trace segments when sampling mechanism actived.
|
|
Setting this threshold about the latency would make the slow trace segments sampled if they cost more time, even the sampling mechanism activated. The default value is `-1`, which means would not sample slow traces. Unit, millisecond.
|
|
|
|
# Recommendation
|
|
You could set different backend instances with different `sampleRate` values, but we recommend you to set the same.
|
|
|
|
When you set the rate different, let's say
|
|
* Backend-Instance**A**.sampleRate = 35
|
|
* Backend-Instance**B**.sampleRate = 55
|
|
|
|
And we assume the agents reported all trace segments to backend,
|
|
Then the 35% traces in the global will be collected and saved in storage consistent/complete, with all spans.
|
|
20% trace segments, which reported to Backend-Instance**B**, will saved in storage, maybe miss some trace segments,
|
|
because they are reported to Backend-Instance**A** and ignored.
|
|
|
|
# Note
|
|
When you open sampling, the actual sample rate could be over sampleRate. Because currently, all error/slow segments will be saved, meanwhile, the upstream and downstream may not be sampled. This feature is going to make sure you could have the error/slow stacks and segments, but don't guarantee you would have the whole trace.
|
|
|
|
Also, the side effect would be, if most of the accesses are fail/slow, the sampling rate would be closing to 100%, which could crash the backend or storage clusters.
|