From f80b03125000a2a7c8feea338fefea88e4722db6 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sun, 2 Sep 2018 23:11:56 +0800 Subject: [PATCH] Add many new chapters of v6 document. Make the document more clear. --- .../OAP-Cluster-Management.md | 43 ------------------- .../Java-Plugin-Development-Guide.md} | 2 +- docs/en/guides/README.md | 5 ++- docs/en/guides/storage-extention.md | 3 ++ docs/en/protocols/Trace-Data-Protocol.md | 2 +- docs/en/setup/backend/backend-alarm.md | 1 + docs/en/setup/backend/backend-cluster.md | 3 ++ docs/en/setup/backend/backend-oal-scripts.md | 1 + docs/en/setup/backend/backend-receivers.md | 8 ++++ docs/en/setup/backend/backend-setup.md | 33 +++++++++++--- docs/en/setup/backend/backend-storage.md | 42 ++++++++++++++++++ docs/en/setup/backend/backend-ui-setup.md | 4 +- .../setup/service-agent/java-agent/README.md | 2 +- 13 files changed, 93 insertions(+), 56 deletions(-) delete mode 100644 docs/en/concepts-and-designs/OAP-Cluster-Management.md rename docs/en/{setup/service-agent/java-agent/Plugin-Development-Guide.md => guides/Java-Plugin-Development-Guide.md} (99%) create mode 100644 docs/en/guides/storage-extention.md create mode 100644 docs/en/setup/backend/backend-alarm.md create mode 100644 docs/en/setup/backend/backend-oal-scripts.md create mode 100644 docs/en/setup/backend/backend-receivers.md create mode 100644 docs/en/setup/backend/backend-storage.md diff --git a/docs/en/concepts-and-designs/OAP-Cluster-Management.md b/docs/en/concepts-and-designs/OAP-Cluster-Management.md deleted file mode 100644 index 8a27d54a88..0000000000 --- a/docs/en/concepts-and-designs/OAP-Cluster-Management.md +++ /dev/null @@ -1,43 +0,0 @@ -# Observability Analysis Platform Cluster Management -OAP(Observability Analysis Platform) backend is a distributed system, services need to find each -other. i.e. a web service needs to find query service, level 1 aggregate service need to find -level 2 aggregate service. Cluster management is just a client-side implementation. It must work -together with a distributed coordination service, (i.e. Zookeeper, Consul, Kubernetes.) unless -running in standalone mode. - -## Architecture overview - - -### Cluster Management Plugins -By default, OAP backend provides two implementations for cluster management, which are standalone -and zookeeper. When the scale of services, which are under monitoring, is small, you can choose -the standalone mode. Otherwise, you must choose the cluster mode by zookeeper plugin, or you can -implement a new service discovery plugin for your own scene. - -### Cluster Management Interface -There are two interfaces defined beforehand in the OAP server core, which are Module register and -module query, all the cluster management plugins must implement those two interfaces. - -* Module Register: When any modules need to provide services for others, the module must do register -through this interface. -* Module Query: When any module needs to find other services, it can use this interface to retrieve -the service instance list in the certain order. - -### Process Flow Between Client and Cluster Management -The client has two ways to connect the backend, one, use the direct link by a set of backend instance -endpoint list, or you can use naming service, which considers your given list is just the seed nodes -of the whole cluster. The following graph is showing you how naming service works. You need to check -the probe documents to know which way(s) is(are) supported. -``` - Client lib Collector1 Collector2 Collector3 - (Set collector.servers=Collector2) (Collector 1,2,3 constitute the cluster) - | - +-----------> naming service ---------------------------->| - | - |<------- receive gRPC IP:Port(s) of Collector 1,2,3---<--| - | - |Select a random gRPC service - |For example collector 3 - | - |------------------------->Uplink gRPC service----------------------------------->| -``` \ No newline at end of file diff --git a/docs/en/setup/service-agent/java-agent/Plugin-Development-Guide.md b/docs/en/guides/Java-Plugin-Development-Guide.md similarity index 99% rename from docs/en/setup/service-agent/java-agent/Plugin-Development-Guide.md rename to docs/en/guides/Java-Plugin-Development-Guide.md index e493ac6f03..9d9cf4bb13 100644 --- a/docs/en/setup/service-agent/java-agent/Plugin-Development-Guide.md +++ b/docs/en/guides/Java-Plugin-Development-Guide.md @@ -158,7 +158,7 @@ SpanLayer is the catalog of span. Here are 5 values: 1. MQ Component IDs are defined and reserved by SkyWalking project. -For component name/ID extension, please follow [component library setting document](../../backend/Component-library-settings.md). +For component name/ID extension, please follow [component library setting document](../setup/backend/Component-library-settings.md). ## Develop a plugin ### Abstract diff --git a/docs/en/guides/README.md b/docs/en/guides/README.md index b4e954c8a5..b9e882f723 100644 --- a/docs/en/guides/README.md +++ b/docs/en/guides/README.md @@ -11,6 +11,7 @@ breaking any Apache rule. SkyWalking project supports many ways to extends existing features. If you are interesting in these ways, read the following guides. -- [Java agent plugin development guide](../setup/service-agent/java-agent/Plugin-Development-Guide.md). +- [Java agent plugin development guide](Java-Plugin-Development-Guide.md). This guide helps you to develop SkyWalking agent plugin to support more frameworks. Both open source plugin -and private plugin developer should read this. \ No newline at end of file +and private plugin developer should read this. +- [Storage extension development guide](storage-extention.md) diff --git a/docs/en/guides/storage-extention.md b/docs/en/guides/storage-extention.md new file mode 100644 index 0000000000..66c6fbf3db --- /dev/null +++ b/docs/en/guides/storage-extention.md @@ -0,0 +1,3 @@ +# Extend storage +SkyWalking has already provided H2 and ElasticSearch as storage solutions. In this document, you could +learn how to implement a new storage easily. \ No newline at end of file diff --git a/docs/en/protocols/Trace-Data-Protocol.md b/docs/en/protocols/Trace-Data-Protocol.md index 54652fb966..0222a223db 100644 --- a/docs/en/protocols/Trace-Data-Protocol.md +++ b/docs/en/protocols/Trace-Data-Protocol.md @@ -18,7 +18,7 @@ v2.0 [gRPC service define](https://github.com/apache/incubator-skywalking-data-collect-protocol/blob/v2.0/TraceSegmentService.proto) - UniqueId represents segmentId and globalTraceId. It have 3 parts(Longs), 1) applicationInstanceId, 2) ThreadId, 3) Timestamp + 10000 + seq(seq is in [0, 100000) ) -- Span data please refs to [Plugin Development Guide](../setup/service-agent/java-agent/Plugin-Development-Guide.md) +- Span data please refs to [Plugin Development Guide](../guides/Java-Plugin-Development-Guide.md) - Id and name both exist, please use id if possible. - operationNameId/operationName - networkAddress/networkAddressId diff --git a/docs/en/setup/backend/backend-alarm.md b/docs/en/setup/backend/backend-alarm.md new file mode 100644 index 0000000000..497e2565c3 --- /dev/null +++ b/docs/en/setup/backend/backend-alarm.md @@ -0,0 +1 @@ +# Alarm \ No newline at end of file diff --git a/docs/en/setup/backend/backend-cluster.md b/docs/en/setup/backend/backend-cluster.md index cb037dcd5a..cf4cd607d2 100644 --- a/docs/en/setup/backend/backend-cluster.md +++ b/docs/en/setup/backend/backend-cluster.md @@ -1,4 +1,7 @@ # Cluster Management +In many product environments, backend need to support high throughputs and provide HA to keep robustness, +so you should need cluster management always in product env. + Backend provides several ways to do cluster management. Choose the one you need/want. - [Zookeeper coordinator](#zookeeper-coordinator). Use Zookeeper to let backend detects and communicates diff --git a/docs/en/setup/backend/backend-oal-scripts.md b/docs/en/setup/backend/backend-oal-scripts.md new file mode 100644 index 0000000000..5945ee4da2 --- /dev/null +++ b/docs/en/setup/backend/backend-oal-scripts.md @@ -0,0 +1 @@ +# Official OAL scripts \ No newline at end of file diff --git a/docs/en/setup/backend/backend-receivers.md b/docs/en/setup/backend/backend-receivers.md new file mode 100644 index 0000000000..056a2559ef --- /dev/null +++ b/docs/en/setup/backend/backend-receivers.md @@ -0,0 +1,8 @@ +# Choose receiver +Receiver is a concept in SkyWalking backend. All modules, which are responsible for receiving telemetry +or tracing data from other being monitored system, are all being called **Receiver**. Although today, most of +receivers are using gRPC or HTTPRestful to provide service, actually, whether listening mode or pull mode +could be receiver. Such as a receiver could base on pull data from remote, like Kakfa MQ. + +We have following receivers +1. \ No newline at end of file diff --git a/docs/en/setup/backend/backend-setup.md b/docs/en/setup/backend/backend-setup.md index 4318f4e2dc..429cadd7af 100644 --- a/docs/en/setup/backend/backend-setup.md +++ b/docs/en/setup/backend/backend-setup.md @@ -27,17 +27,38 @@ core: At the same time, modules includes required and optional, the required modules provide the skeleton of backend, even modulization supported pluggable, remove those modules are meanless. We highly recommend you don't try to -switch those modules, unless you are well known SkyWalking project and its codes. +change APIs of those modules, unless you understand SkyWalking project and its codes very well. List the required modules here -1. **core** -1. **cluster** -1. **storage** -1. **query** +1. **Core**. Do basic and major skeleton of all data analysis and stream dispatch. +1. **Cluster**. Manage multiple backend instances in a cluster, which could provide high throughputs process +capabilities. +1. **Storage**. Make the analysis result persistence. +1. **Query**. Provide query interfaces to UI. +For **Cluster** and **Storage** have provided multiple implementors(providers), see **Cluster management** +and **Choose storage** documents in the [link list](#advanced-feature-document-link-list). + +Also, several **receiver** modules are provided. +Receiver is the module in charge of accepting incoming data requests to backend. Most(all) provide +service by some network(RPC) protocol, such as gRPC, HTTPRestful. +The receivers have many different module names, you could +read **Set receivers** document in the [link list](#advanced-feature-document-link-list). + +## Advanced feature document link list After understand the setting file structure, you could choose your interesting feature document. -We recommend you to read the feature documents in our following list. +We recommend you to read the feature documents in our following order. 1. [IP and port setting](backend-ip-port.md). Introduce how IP and port set and be used. 1. [Cluster management](backend-cluster.md). Guide you to set backend server in cluster mode. 1. [Deploy in kubernetes](backend-k8s.md). Guide you to build and use SkyWalking image, and deploy in k8s. +1. [Choose storage](backend-storage.md). As we know, in default quick start, backend is running with H2 +DB. But clearly, it doesn't fit the product env. In here, you could find what other choices do you have. +Choose the one you like, we are also welcome anyone to contribute new storage implementor, +1. [Set receivers](backend-receivers.md). You could choose receivers by your requirements, most receivers +are harmless, at least our default receivers are. You would set and active all receivers provided. +1. Official [OAL scripts](backend-oal-scripts.md). As you known from our [OAL introduction](../../concepts-and-designs/oal.md), +most of backend analysis capabilities based on the scripts. Here is the description of official scripts, +which helps you to understand which metric data are in process, also could be used in alarm. +1. [Alarm](backend-alarm.md). Alarm provides a time-series based check mechanism. You could set alarm +rules targeting the analysis oal metric objects. diff --git a/docs/en/setup/backend/backend-storage.md b/docs/en/setup/backend/backend-storage.md new file mode 100644 index 0000000000..b45e6a490c --- /dev/null +++ b/docs/en/setup/backend/backend-storage.md @@ -0,0 +1,42 @@ +# Backend storage +SkyWalking storage is pluggable, we have provided the following storage solutions, you could easily +use is by changing the `application.yml` + +- [**H2**](#h2) +- [**ElasticSearch 6**](#elasticsearch-6) + +## H2 +Active ElasticSearch 6 as storage, set storage provider to **H2**. + +Setting fragment example +```yaml + +``` + +## ElasticSearch 6 +Active ElasticSearch 6 as storage, set storage provider to **elasticsearch**. + +Setting fragment example +```yaml +storage: + elasticsearch: + clusterNodes: localhost:9200 + indexShardsNumber: 2 + indexReplicasNumber: 0 + # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html + bulkActions: 2000 # Execute the bulk every 2000 requests + bulkSize: 20 # flush the bulk every 20mb + flushInterval: 10 # flush the bulk every 10 seconds whatever the number of requests + concurrentRequests: 2 # the number of concurrent requests + # Set a timeout on metric data. After the timeout has expired, the metric data will automatically be deleted. + traceDataTTL: 90 # Unit is minute + minuteMetricDataTTL: 90 # Unit is minute + hourMetricDataTTL: 36 # Unit is hour + dayMetricDataTTL: 45 # Unit is day + monthMetricDataTTL: 18 # Unit is month +``` + + +## More storage solution extension +Follow [Storage extension development guide](../../guides/storage-extention.md) +in [Project Extensions document](../../guides/README.md#project-extensions) in development guide. \ No newline at end of file diff --git a/docs/en/setup/backend/backend-ui-setup.md b/docs/en/setup/backend/backend-ui-setup.md index 081a1d98d8..cd04b81aa0 100644 --- a/docs/en/setup/backend/backend-ui-setup.md +++ b/docs/en/setup/backend/backend-ui-setup.md @@ -35,7 +35,7 @@ data to backend. Also, jetty service supported in JSON format. - UI uses GraphQL(HTTP) query to access backend also in Jetty service(`core/default/rest*` in `application.yml`). Now, let's go the backend and UI setting documents. -- [Backend setting document](backend-setup.md) -- [UI setting document](ui-setup.md) +- [Backend setup document](backend-setup.md) +- [UI setup document](ui-setup.md) \ No newline at end of file diff --git a/docs/en/setup/service-agent/java-agent/README.md b/docs/en/setup/service-agent/java-agent/README.md index dc048e7325..3d8641d4e5 100644 --- a/docs/en/setup/service-agent/java-agent/README.md +++ b/docs/en/setup/service-agent/java-agent/README.md @@ -70,7 +70,7 @@ Now, we have the following known plugins. ## Plugin Development Guide SkyWalking java agent supports plugin to extend [the supported list](Supported-list.md). Please follow -our [Plugin Development Guide](Plugin-Development-Guide.md). +our [Plugin Development Guide](../../../guides/Java-Plugin-Development-Guide.md). # Test If you are interested in plugin compatible tests or agent performance, see the following reports.