kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix out-dated docs about Kafka fetcher (#11235)

This commit is contained in:
吴晟 Wu Sheng 2023-08-17 20:47:10 +08:00 committed by GitHub
parent 5491c04a8a
commit d5bf66dea7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 2 additions and 30 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/en/changes/changes.md
View File

@ -108,5 +108,7 @@
* Add a doc to explain how to manage i18n. * Add a doc to explain how to manage i18n.
* Add a doc to explain OTLP Trace support. * Add a doc to explain OTLP Trace support.
* Fix typo in `dynamic-config-configmap.md`. * Fix typo in `dynamic-config-configmap.md`.
* Fix out-dated docs about Kafka fetcher.
* Remove 3rd part fetchers from the docs, as they are not maintained anymore.
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/181?closed=1) All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/181?closed=1)

30
docs/en/setup/backend/kafka-fetcher.md
View File

@ -29,30 +29,9 @@ kafka-fetcher:
namespace: ${SW_NAMESPACE:""} namespace: ${SW_NAMESPACE:""}
partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3} partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2} replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:false}
consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:""}
consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1} consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
``` ```
In the cluster mode, all topics have the same number of partitions. Set `"isSharding"` to `"true"` and assign the partitions to consume for the OAP server. Use commas to separate multiple partitions for the OAP server.
The Kafka Fetcher allows you to configure all the Kafka producers listed [here](http://kafka.apache.org/24/documentation.html#consumerconfigs) in property `kafkaConsumerConfig`. For example:
```yaml
kafka-fetcher:
selector: ${SW_KAFKA_FETCHER:default}
default:
bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:localhost:9092}
namespace: ${SW_NAMESPACE:""}
partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:true}
consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:1,3,5}
consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
kafkaConsumerConfig:
enable.auto.commit: true
...
```
When using Kafka MirrorMaker 2.0 to replicate topics between Kafka clusters, you can set the source Kafka Cluster alias (mm2SourceAlias) and separator (mm2SourceSeparator) according to your Kafka MirrorMaker [config](https://github.com/apache/kafka/tree/trunk/connect/mirror#remote-topics). When using Kafka MirrorMaker 2.0 to replicate topics between Kafka clusters, you can set the source Kafka Cluster alias (mm2SourceAlias) and separator (mm2SourceSeparator) according to your Kafka MirrorMaker [config](https://github.com/apache/kafka/tree/trunk/connect/mirror#remote-topics).
```yaml ```yaml
kafka-fetcher: kafka-fetcher:
@ -62,8 +41,6 @@ kafka-fetcher:
namespace: ${SW_NAMESPACE:""} namespace: ${SW_NAMESPACE:""}
partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3} partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2} replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:true}
consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:1,3,5}
consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1} consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
mm2SourceAlias: ${SW_KAFKA_MM2_SOURCE_ALIAS:""} mm2SourceAlias: ${SW_KAFKA_MM2_SOURCE_ALIAS:""}
mm2SourceSeparator: ${SW_KAFKA_MM2_SOURCE_SEPARATOR:""} mm2SourceSeparator: ${SW_KAFKA_MM2_SOURCE_SEPARATOR:""}
@ -71,10 +48,3 @@ kafka-fetcher:
enable.auto.commit: true enable.auto.commit: true
... ...
``` ```
## Other Fetcher Plugins
There are other transporter plugins. You can find these plugins from 3rd party repositories.
* [Pulsar Fetcher Plugin](https://github.com/SkyAPM/transporter-plugin-for-skywalking/blob/main/docs/en/pulsar/Pulsar-Fetcher.md)
* [RocketMQ Fetcher Plugin](https://github.com/SkyAPM/transporter-plugin-for-skywalking/blob/main/docs/en/rocketmq/Rocketmq-Fetcher.md)