kazusa
/
skywalking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Remove unreasonable default configurations for gRPC thread executor (#11639)

This commit is contained in:
kezhenxu94 2023-12-08 13:59:45 +08:00 committed by GitHub
parent f6a7791053
commit 39c21e3b45
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 24 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/en/changes/changes.md
View File

@ -6,6 +6,9 @@
#### OAP Server #### OAP Server
* Add `layer` parameter to the global topology graphQL query. * Add `layer` parameter to the global topology graphQL query.
* Add `is_present` function in MQE for check if the list metrics has a value or not. * Add `is_present` function in MQE for check if the list metrics has a value or not.
* Remove unreasonable default configurations for gRPC thread executor.
* Remove `gRPCThreadPoolQueueSize (SW_RECEIVER_GRPC_POOL_QUEUE_SIZE)`
configuration.
#### UI #### UI
* Fix the mismatch between the unit and calculation of the "Network Bandwidth Usage" widget in Linux-Service Dashboard. * Fix the mismatch between the unit and calculation of the "Network Bandwidth Usage" widget in Linux-Service Dashboard.

1
docs/en/setup/backend/backend-expose.md
View File

@ -23,7 +23,6 @@ receiver-sharing-server:
gRPCPort: ${SW_RECEIVER_GRPC_PORT:0} gRPCPort: ${SW_RECEIVER_GRPC_PORT:0}
maxConcurrentCallsPerConnection: ${SW_RECEIVER_GRPC_MAX_CONCURRENT_CALL:0} maxConcurrentCallsPerConnection: ${SW_RECEIVER_GRPC_MAX_CONCURRENT_CALL:0}
maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0} maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0}
gRPCThreadPoolQueueSize: ${SW_RECEIVER_GRPC_POOL_QUEUE_SIZE:0}
gRPCThreadPoolSize: ${SW_RECEIVER_GRPC_THREAD_POOL_SIZE:0} gRPCThreadPoolSize: ${SW_RECEIVER_GRPC_THREAD_POOL_SIZE:0}
gRPCSslEnabled: ${SW_RECEIVER_GRPC_SSL_ENABLED:false} gRPCSslEnabled: ${SW_RECEIVER_GRPC_SSL_ENABLED:false}
gRPCSslKeyPath: ${SW_RECEIVER_GRPC_SSL_KEY_PATH:""} gRPCSslKeyPath: ${SW_RECEIVER_GRPC_SSL_KEY_PATH:""}

6
docs/en/setup/backend/configuration-vocabulary.md
View File

@ -36,8 +36,7 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | searchableAlarmTags | Defines a set of alarm tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. | SW_SEARCHABLE_ALARM_TAG_KEYS | level | | - | - | searchableAlarmTags | Defines a set of alarm tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. | SW_SEARCHABLE_ALARM_TAG_KEYS | level |
| - | - | autocompleteTagKeysQueryMaxSize | The max size of tags keys for autocomplete select. | SW_AUTOCOMPLETE_TAG_KEYS_QUERY_MAX_SIZE | 100 | | - | - | autocompleteTagKeysQueryMaxSize | The max size of tags keys for autocomplete select. | SW_AUTOCOMPLETE_TAG_KEYS_QUERY_MAX_SIZE | 100 |
| - | - | autocompleteTagValuesQueryMaxSize | The max size of tags values for autocomplete select. | SW_AUTOCOMPLETE_TAG_VALUES_QUERY_MAX_SIZE | 100 | | - | - | autocompleteTagValuesQueryMaxSize | The max size of tags values for autocomplete select. | SW_AUTOCOMPLETE_TAG_VALUES_QUERY_MAX_SIZE | 100 |
| - | - | gRPCThreadPoolSize | Pool size of gRPC server. | SW_CORE_GRPC_THREAD_POOL_SIZE | CPU core * 4 | | - | - | gRPCThreadPoolSize | Pool size of gRPC server. | SW_CORE_GRPC_THREAD_POOL_SIZE | Default to gRPC's implementation, which is a cached thread pool that can grow infinitely. |
| - | - | gRPCThreadPoolQueueSize | Queue size of gRPC server. | SW_CORE_GRPC_POOL_QUEUE_SIZE | 10000 |
| - | - | maxConcurrentCallsPerConnection | The maximum number of concurrent calls permitted for each incoming connection. Defaults to no limit. | SW_CORE_GRPC_MAX_CONCURRENT_CALL | - | | - | - | maxConcurrentCallsPerConnection | The maximum number of concurrent calls permitted for each incoming connection. Defaults to no limit. | SW_CORE_GRPC_MAX_CONCURRENT_CALL | - |
| - | - | maxMessageSize | Sets the maximum message size allowed to be received on the server. Empty means 4 MiB. | SW_CORE_GRPC_MAX_MESSAGE_SIZE | 4M(based on Netty) | | - | - | maxMessageSize | Sets the maximum message size allowed to be received on the server. Empty means 4 MiB. | SW_CORE_GRPC_MAX_MESSAGE_SIZE | 4M(based on Netty) |
| - | - | remoteTimeout | Timeout for cluster internal communication (in seconds). | - | 20 | | - | - | remoteTimeout | Timeout for cluster internal communication (in seconds). | - | 20 |
@ -166,8 +165,7 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | httpMaxRequestHeaderSize | Maximum request header size accepted. | SW_RECEIVER_SHARING_HTTP_MAX_REQUEST_HEADER_SIZE | 8192 | | - | - | httpMaxRequestHeaderSize | Maximum request header size accepted. | SW_RECEIVER_SHARING_HTTP_MAX_REQUEST_HEADER_SIZE | 8192 |
| - | - | gRPCHost | Binding IP of gRPC services. Services include gRPC data report and internal communication among OAP nodes. | SW_RECEIVER_GRPC_HOST | 0.0.0.0. Not Activated | | - | - | gRPCHost | Binding IP of gRPC services. Services include gRPC data report and internal communication among OAP nodes. | SW_RECEIVER_GRPC_HOST | 0.0.0.0. Not Activated |
| - | - | gRPCPort | Binding port of gRPC services. | SW_RECEIVER_GRPC_PORT | Not Activated | | - | - | gRPCPort | Binding port of gRPC services. | SW_RECEIVER_GRPC_PORT | Not Activated |
| - | - | gRPCThreadPoolSize | Pool size of gRPC server. | SW_RECEIVER_GRPC_THREAD_POOL_SIZE | CPU core * 4 | | - | - | gRPCThreadPoolSize | Pool size of gRPC server. | SW_RECEIVER_GRPC_THREAD_POOL_SIZE | Default to gRPC's implementation, which is a cached thread pool that can grow infinitely. |
| - | - | gRPCThreadPoolQueueSize | Queue size of gRPC server. | SW_RECEIVER_GRPC_POOL_QUEUE_SIZE | 10000 |
| - | - | gRPCSslEnabled | Activates SSL for gRPC services. | SW_RECEIVER_GRPC_SSL_ENABLED | false | | - | - | gRPCSslEnabled | Activates SSL for gRPC services. | SW_RECEIVER_GRPC_SSL_ENABLED | false |
| - | - | gRPCSslKeyPath | File path of gRPC SSL key. | SW_RECEIVER_GRPC_SSL_KEY_PATH | - | | - | - | gRPCSslKeyPath | File path of gRPC SSL key. | SW_RECEIVER_GRPC_SSL_KEY_PATH | - |
| - | - | gRPCSslCertChainPath | File path of gRPC SSL cert chain. | SW_RECEIVER_GRPC_SSL_CERT_CHAIN_PATH | - | | - | - | gRPCSslCertChainPath | File path of gRPC SSL cert chain. | SW_RECEIVER_GRPC_SSL_CERT_CHAIN_PATH | - |

1
docs/en/setup/backend/grpc-security.md
View File

@ -79,7 +79,6 @@ receiver-sharing-server:
gRPCPort: ${SW_RECEIVER_GRPC_PORT:"changeMe"} gRPCPort: ${SW_RECEIVER_GRPC_PORT:"changeMe"}
maxConcurrentCallsPerConnection: ${SW_RECEIVER_GRPC_MAX_CONCURRENT_CALL:0} maxConcurrentCallsPerConnection: ${SW_RECEIVER_GRPC_MAX_CONCURRENT_CALL:0}
maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0} maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0}
gRPCThreadPoolQueueSize: ${SW_RECEIVER_GRPC_POOL_QUEUE_SIZE:0}
gRPCThreadPoolSize: ${SW_RECEIVER_GRPC_THREAD_POOL_SIZE:0} gRPCThreadPoolSize: ${SW_RECEIVER_GRPC_THREAD_POOL_SIZE:0}
gRPCSslEnabled: ${SW_RECEIVER_GRPC_SSL_ENABLED:true} gRPCSslEnabled: ${SW_RECEIVER_GRPC_SSL_ENABLED:true}
gRPCSslKeyPath: ${SW_RECEIVER_GRPC_SSL_KEY_PATH:"/path/to/server.pem"} gRPCSslKeyPath: ${SW_RECEIVER_GRPC_SSL_KEY_PATH:"/path/to/server.pem"}

1
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleConfig.java
View File

@ -77,7 +77,6 @@ public class CoreModuleConfig extends ModuleConfig {
private int gRPCThreadPoolSize; private int gRPCThreadPoolSize;
private int gRPCThreadPoolQueueSize;
/** /**
* Timeout for cluster internal communication, in seconds. * Timeout for cluster internal communication, in seconds.
*/ */

3
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
View File

@ -230,9 +230,6 @@ public class CoreModuleProvider extends ModuleProvider {
if (moduleConfig.getMaxMessageSize() > 0) { if (moduleConfig.getMaxMessageSize() > 0) {
grpcServer.setMaxMessageSize(moduleConfig.getMaxMessageSize()); grpcServer.setMaxMessageSize(moduleConfig.getMaxMessageSize());
} }
if (moduleConfig.getGRPCThreadPoolQueueSize() > 0) {
grpcServer.setThreadPoolQueueSize(moduleConfig.getGRPCThreadPoolQueueSize());
}
if (moduleConfig.getGRPCThreadPoolSize() > 0) { if (moduleConfig.getGRPCThreadPoolSize() > 0) {
grpcServer.setThreadPoolSize(moduleConfig.getGRPCThreadPoolSize()); grpcServer.setThreadPoolSize(moduleConfig.getGRPCThreadPoolSize());
} }

37
oap-server/server-library/library-server/src/main/java/org/apache/skywalking/oap/server/library/server/grpc/GRPCServer.java
View File

@ -26,9 +26,9 @@ import io.grpc.netty.NettyServerBuilder;
import java.io.IOException; import java.io.IOException;
import java.net.InetSocketAddress; import java.net.InetSocketAddress;
import java.util.Optional; import java.util.Optional;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ExecutorService; import java.util.concurrent.ExecutorService;
import java.util.concurrent.RejectedExecutionHandler; import java.util.concurrent.RejectedExecutionHandler;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
@ -50,14 +50,11 @@ public class GRPCServer implements Server {
private String privateKeyFile; private String privateKeyFile;
private String trustedCAsFile; private String trustedCAsFile;
private DynamicSslContext sslContext; private DynamicSslContext sslContext;
private int threadPoolSize = Runtime.getRuntime().availableProcessors() * 4; private int threadPoolSize;
private int threadPoolQueueSize = 10000;
public GRPCServer(String host, int port) { public GRPCServer(String host, int port) {
this.host = host; this.host = host;
this.port = port; this.port = port;
this.maxConcurrentCallsPerConnection = 4;
this.maxMessageSize = Integer.MAX_VALUE;
} }
public void setMaxConcurrentCallsPerConnection(int maxConcurrentCallsPerConnection) { public void setMaxConcurrentCallsPerConnection(int maxConcurrentCallsPerConnection) {
@ -72,10 +69,6 @@ public class GRPCServer implements Server {
this.threadPoolSize = threadPoolSize; this.threadPoolSize = threadPoolSize;
} }
public void setThreadPoolQueueSize(int threadPoolQueueSize) {
this.threadPoolQueueSize = threadPoolQueueSize;
}
/** /**
* Require for `server.crt` and `server.pem` for open ssl at server side. * Require for `server.crt` and `server.pem` for open ssl at server side.
* *
@ -92,15 +85,22 @@ public class GRPCServer implements Server {
@Override @Override
public void initialize() { public void initialize() {
InetSocketAddress address = new InetSocketAddress(host, port); InetSocketAddress address = new InetSocketAddress(host, port);
ArrayBlockingQueue<Runnable> blockingQueue = new ArrayBlockingQueue<>(threadPoolQueueSize);
ExecutorService executor = new ThreadPoolExecutor(
threadPoolSize, threadPoolSize, 60, TimeUnit.SECONDS, blockingQueue,
new CustomThreadFactory("grpcServerPool"), new CustomRejectedExecutionHandler()
);
nettyServerBuilder = NettyServerBuilder.forAddress(address); nettyServerBuilder = NettyServerBuilder.forAddress(address);
nettyServerBuilder = nettyServerBuilder.maxConcurrentCallsPerConnection(maxConcurrentCallsPerConnection)
.maxInboundMessageSize(maxMessageSize) if (maxConcurrentCallsPerConnection > 0) {
.executor(executor); nettyServerBuilder.maxConcurrentCallsPerConnection(maxConcurrentCallsPerConnection);
}
if (maxMessageSize > 0) {
nettyServerBuilder.maxInboundMessageSize(maxMessageSize);
}
if (threadPoolSize > 0) {
ExecutorService executor = new ThreadPoolExecutor(
threadPoolSize, threadPoolSize, 60, TimeUnit.SECONDS, new SynchronousQueue<>(),
new CustomThreadFactory("grpcServerPool"), new CustomRejectedExecutionHandler()
);
nettyServerBuilder.executor(executor);
}
if (!Strings.isNullOrEmpty(privateKeyFile) && !Strings.isNullOrEmpty(certChainFile)) { if (!Strings.isNullOrEmpty(privateKeyFile) && !Strings.isNullOrEmpty(certChainFile)) {
sslContext = DynamicSslContext.forServer(privateKeyFile, certChainFile, trustedCAsFile); sslContext = DynamicSslContext.forServer(privateKeyFile, certChainFile, trustedCAsFile);
nettyServerBuilder.sslContext(sslContext); nettyServerBuilder.sslContext(sslContext);
@ -109,10 +109,9 @@ public class GRPCServer implements Server {
} }
static class CustomRejectedExecutionHandler implements RejectedExecutionHandler { static class CustomRejectedExecutionHandler implements RejectedExecutionHandler {
@Override @Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) { public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
log.warn("Grpc server thread pool is full, rejecting the task"); log.warn("Task {} rejected from {}", r.toString(), executor.toString());
} }
} }

1
oap-server/server-receiver-plugin/skywalking-sharing-server-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/sharing/server/SharingServerConfig.java
View File

@ -43,7 +43,6 @@ public class SharingServerConfig extends ModuleConfig {
private int maxConcurrentCallsPerConnection; private int maxConcurrentCallsPerConnection;
private int maxMessageSize; private int maxMessageSize;
private int gRPCThreadPoolSize; private int gRPCThreadPoolSize;
private int gRPCThreadPoolQueueSize;
private String authentication; private String authentication;
private boolean gRPCSslEnabled = false; private boolean gRPCSslEnabled = false;
private String gRPCSslKeyPath; private String gRPCSslKeyPath;

3
oap-server/server-receiver-plugin/skywalking-sharing-server-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/sharing/server/SharingServerModuleProvider.java
View File

@ -125,9 +125,6 @@ public class SharingServerModuleProvider extends ModuleProvider {
if (config.getMaxConcurrentCallsPerConnection() > 0) { if (config.getMaxConcurrentCallsPerConnection() > 0) {
grpcServer.setMaxConcurrentCallsPerConnection(config.getMaxConcurrentCallsPerConnection()); grpcServer.setMaxConcurrentCallsPerConnection(config.getMaxConcurrentCallsPerConnection());
} }
if (config.getGRPCThreadPoolQueueSize() > 0) {
grpcServer.setThreadPoolQueueSize(config.getGRPCThreadPoolQueueSize());
}
if (config.getGRPCThreadPoolSize() > 0) { if (config.getGRPCThreadPoolSize() > 0) {
grpcServer.setThreadPoolSize(config.getGRPCThreadPoolSize()); grpcServer.setThreadPoolSize(config.getGRPCThreadPoolSize());
} }

4
oap-server/server-starter/src/main/resources/application.yml
View File

@ -82,7 +82,6 @@ core:
gRPCPort: ${SW_CORE_GRPC_PORT:11800} gRPCPort: ${SW_CORE_GRPC_PORT:11800}
maxConcurrentCallsPerConnection: ${SW_CORE_GRPC_MAX_CONCURRENT_CALL:0} maxConcurrentCallsPerConnection: ${SW_CORE_GRPC_MAX_CONCURRENT_CALL:0}
maxMessageSize: ${SW_CORE_GRPC_MAX_MESSAGE_SIZE:0} maxMessageSize: ${SW_CORE_GRPC_MAX_MESSAGE_SIZE:0}
gRPCThreadPoolQueueSize: ${SW_CORE_GRPC_POOL_QUEUE_SIZE:-1}
gRPCThreadPoolSize: ${SW_CORE_GRPC_THREAD_POOL_SIZE:-1} gRPCThreadPoolSize: ${SW_CORE_GRPC_THREAD_POOL_SIZE:-1}
gRPCSslEnabled: ${SW_CORE_GRPC_SSL_ENABLED:false} gRPCSslEnabled: ${SW_CORE_GRPC_SSL_ENABLED:false}
gRPCSslKeyPath: ${SW_CORE_GRPC_SSL_KEY_PATH:""} gRPCSslKeyPath: ${SW_CORE_GRPC_SSL_KEY_PATH:""}
@ -268,7 +267,6 @@ receiver-sharing-server:
gRPCPort: ${SW_RECEIVER_GRPC_PORT:0} gRPCPort: ${SW_RECEIVER_GRPC_PORT:0}
maxConcurrentCallsPerConnection: ${SW_RECEIVER_GRPC_MAX_CONCURRENT_CALL:0} maxConcurrentCallsPerConnection: ${SW_RECEIVER_GRPC_MAX_CONCURRENT_CALL:0}
maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0} maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0}
gRPCThreadPoolQueueSize: ${SW_RECEIVER_GRPC_POOL_QUEUE_SIZE:0}
gRPCThreadPoolSize: ${SW_RECEIVER_GRPC_THREAD_POOL_SIZE:0} gRPCThreadPoolSize: ${SW_RECEIVER_GRPC_THREAD_POOL_SIZE:0}
gRPCSslEnabled: ${SW_RECEIVER_GRPC_SSL_ENABLED:false} gRPCSslEnabled: ${SW_RECEIVER_GRPC_SSL_ENABLED:false}
gRPCSslKeyPath: ${SW_RECEIVER_GRPC_SSL_KEY_PATH:""} gRPCSslKeyPath: ${SW_RECEIVER_GRPC_SSL_KEY_PATH:""}
@ -580,4 +578,4 @@ ai-pipeline:
selector: ${SW_AI_PIPELINE:default} selector: ${SW_AI_PIPELINE:default}
default: default:
uriRecognitionServerAddr: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_ADDR:} uriRecognitionServerAddr: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_ADDR:}
uriRecognitionServerPort: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_PORT:17128} uriRecognitionServerPort: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_PORT:17128}

1
oap-server/server-tools/data-generator/src/main/resources/application.yml
View File

@ -34,7 +34,6 @@ core:
gRPCPort: ${SW_CORE_GRPC_PORT:11800} gRPCPort: ${SW_CORE_GRPC_PORT:11800}
maxConcurrentCallsPerConnection: ${SW_CORE_GRPC_MAX_CONCURRENT_CALL:0} maxConcurrentCallsPerConnection: ${SW_CORE_GRPC_MAX_CONCURRENT_CALL:0}
maxMessageSize: ${SW_CORE_GRPC_MAX_MESSAGE_SIZE:0} maxMessageSize: ${SW_CORE_GRPC_MAX_MESSAGE_SIZE:0}
gRPCThreadPoolQueueSize: ${SW_CORE_GRPC_POOL_QUEUE_SIZE:-1}
gRPCThreadPoolSize: ${SW_CORE_GRPC_THREAD_POOL_SIZE:-1} gRPCThreadPoolSize: ${SW_CORE_GRPC_THREAD_POOL_SIZE:-1}
gRPCSslEnabled: ${SW_CORE_GRPC_SSL_ENABLED:false} gRPCSslEnabled: ${SW_CORE_GRPC_SSL_ENABLED:false}
gRPCSslKeyPath: ${SW_CORE_GRPC_SSL_KEY_PATH:""} gRPCSslKeyPath: ${SW_CORE_GRPC_SSL_KEY_PATH:""}