From f40eff5d1545bcdbedd9f7d3516a00b26fd695f8 Mon Sep 17 00:00:00 2001 From: wu-sheng Date: Fri, 29 Dec 2017 16:23:03 +0800 Subject: [PATCH] Provide plugin development guide EN version. --- docs/README.md | 1 + docs/README_ZH.md | 2 +- docs/cn/How-to-build-CN.md | 5 +- docs/cn/Plugin-Development-Guide-CN.md | 53 +++-- docs/en/How-to-build.md | 2 +- docs/en/Plugin-Development-Guide.md | 290 +++++++++++++++++++++++++ 6 files changed, 325 insertions(+), 28 deletions(-) create mode 100644 docs/en/Plugin-Development-Guide.md diff --git a/docs/README.md b/docs/README.md index f69a04a5c..c38355652 100644 --- a/docs/README.md +++ b/docs/README.md @@ -19,6 +19,7 @@ * [Java Agent Performance Test](https://skywalkingtest.github.io/Agent-Benchmarks/) * Development Guides * [How to build project](en/How-to-build.md) + * [Plugin development guide](en/Plugin-Development-Guide.md) * Protocol * [Cross Process Propagation Headers Protocol, v1.0](en/Skywalking-Cross-Process-Propagation-Headers-Protocol-v1.md) * FAQ diff --git a/docs/README_ZH.md b/docs/README_ZH.md index d2936c9c4..1db771371 100644 --- a/docs/README_ZH.md +++ b/docs/README_ZH.md @@ -23,7 +23,7 @@ * [工程编译指南](cn/How-to-build-CN.md) * [插件开发指南](cn/Plugin-Development-Guide-CN.md) * 交互协议 - * [Cross Process Propagation Headers Protocol, v1.0 | 跨进程追踪上下文传递协议](cn/Skywalking-Cross-Process-Propagation-Headers-Protocol-CN-v1.md) + * [Cross Process Propagation Headers Protocol, v1.0 跨进程追踪上下文传递协议](cn/Skywalking-Cross-Process-Propagation-Headers-Protocol-CN-v1.md) * [SkyWalking Trace Data Protocol 探针与Collector间网络协议](cn/Trace-Data-Protocol-CN.md) * FAQ * [Trace查询有数据，但是没有拓扑图和JVM数据?](cn/FAQ/Why-have-traces-no-others-CN.md) diff --git a/docs/cn/How-to-build-CN.md b/docs/cn/How-to-build-CN.md index 734b3c57d..8159b1746 100644 --- a/docs/cn/How-to-build-CN.md +++ b/docs/cn/How-to-build-CN.md @@ -2,16 +2,17 @@ 本文档用于指导开发者，在本地开发环境中编译工程。 ### 前言 -因为工程结构和代码依赖会随版本变化，如果读者熟悉travis-ci,则可直接参考[.travis.yml](https://github.com/wu-sheng/sky-walking/blob/master/.travis.yml) +因为工程结构和代码依赖会随版本变化，如果读者熟悉travis-ci,则可直接参考[.travis.yml](../../.travis.yml) ### 编译步骤 1. 准备环境，jdk8，Maven 1. 执行`mvn clean package` +1. 生成包在`/packages`目录下，包括一个`skywalking-agent`的探针目录，以及两个collector包（.tar.gz是linux环境，.zip是windows环境） ### 在IntelliJ IDEA中编译工程 上述步骤在命令行中，能够很好的编译工程，但导入到编译器中的工程依然会有一些报错，我们需要进行几步简单的操作。 1. 在IntelliJ Terminal中，执行`mvn compile -Dmaven.test.skip=true`进行编译 1. 设置gRPC的自动生成代码目录，为源码目录 - - **apm-network/target/generated--sources/protobuf**目录下的`grpc-java`和`java`目录 + - **apm-protocol/apm-network/target/generated-sources/protobuf**目录下的`grpc-java`和`java`目录 - **apm-collector/apm-collector-remote/apm-remote-grpc-provider/target/protobuf**目录下的`grpc-java`和`java`目录 diff --git a/docs/cn/Plugin-Development-Guide-CN.md b/docs/cn/Plugin-Development-Guide-CN.md index 4014b7b3f..83f19305e 100644 --- a/docs/cn/Plugin-Development-Guide-CN.md +++ b/docs/cn/Plugin-Development-Guide-CN.md @@ -1,8 +1,8 @@ ## 插件开发指南 -本文档描述 [v3.2+](https://github.com/OpenTracing/skywalking/releases) 插件开发方法、使用的API，以及注意事项。 +这边文档描述插件的开发和贡献方法 -### 核心概念 -#### 一. Span +## 核心概念 +### 一. Span Span是追踪系统中的通用概念（有时候被翻译成埋点），关于Span的定义，请参考[OpenTracing 中文版](https://github.com/opentracing-contrib/opentracing-specification-zh/blob/master/specification.md#opentracing数据模型)。 SkyWalking作为OpenTracing的支持者，在核心实现中，与标准有较高的相似度。当然，作为实际产品的需要，我们一会扩展相关概念。 @@ -18,14 +18,17 @@ LocalSpan代表一个普通的Span,代表任意一个本地逻辑块（或方法 1.3 ExitSpan ExitSpan也可以称为LeafSpan(SkyWalking的早期版本中的称呼)，代表了一个远程服务的客户端调用。如：一次JDBC调用。 -#### 二. ContextCarrier +### 二. ContextCarrier 分布式追踪要解决的一个重要问题，就是跨进程调用链连接的问题，ContextCarrier的概念就是为了解决这种场景。 当发生一次**A->B**的网络调用时： -1. 需要在客户端生成(inject操作)ContextCarrier，并序列化成String -1. 将这个String加入RPC调用的正文（或HEAD）中，传递到服务端 +1. 创建一个空的ContextCarrier +1. 通过`ContextManager#createExitSpan`方法创建一个ExitSpan，或者使用`ContextManager#inject`，在过程中传入并初始化`ContextCarrier` +1. 将`ContextCarrier`中所有元素放入请求头（如：HTTP头）或消息正文（如 Kafka） +1. `ContextCarrier`随请求传输到服务端 1. 服务端收到后，转换为新的ContextCarrier -1. 通过提取操作（extract操作）建立关联 +1. 通过`ContestManager#createEntrySpan`方法创建EntrySpan，或者使用`ContextManager#extract`，建立分布式调用关联 + 以HTTPComponent调用Tomcat为例： 1. 客户端（HTTPComponent端） @@ -34,8 +37,7 @@ ExitSpan也可以称为LeafSpan(SkyWalking的早期版本中的称呼)，代表 CarrierItem next = contextCarrier.items(); while (next.hasNext()) { next = next.next(); - //向HTTP或者其他RPC HEAD中设置上下文 - heads.put(next.getHeadKey(), next.getHeadValue()); + httpRequest.setHeader(next.getHeadKey(), next.getHeadValue()); } ``` @@ -45,14 +47,13 @@ ExitSpan也可以称为LeafSpan(SkyWalking的早期版本中的称呼)，代表 CarrierItem next = contextCarrier.items(); while (next.hasNext()) { next = next.next(); - //从HTTP或者其他RPC HEAD中，根据指定的KEY，提取上下文 - next.setHeadValue(heads.get(next.getHeadKey())); + next.setHeadValue(request.getHeader(next.getHeadKey())); } span = ContextManager.createEntrySpan(“/span/operation/name”, contextCarrier); ``` -#### 三. ContextSnapshot +### 三. ContextSnapshot 除了跨进程的RPC调用，另外一种追踪的常见场景是跨线程保持链路连接。跨线程和跨进程有很高的相似度，都是需要完成上下文的传递工作。 所以ContextSnapshot具有和ContextCarrier十分类似的API风格。 @@ -61,8 +62,8 @@ ExitSpan也可以称为LeafSpan(SkyWalking的早期版本中的称呼)，代表 1. 将这个ContextSnapshot对象传递到B线程中 1. B线程通过ContextManager#continued操作完成上下文传递 -### 核心API -#### 一. ContextManager +## 核心API +### 一. ContextManager ContextManager提供了追踪相关操作的主入口 1. 创建EntrySpan @@ -83,7 +84,7 @@ public static AbstractSpan createExitSpan(String operationName, ContextCarrier c ``` 根据服务名，跨进程传递的ContextCarrier（空容器）和远端服务地址（IP、主机名、域名 + 端口），创建ExitSpan -#### 二. AbstractSpan +### 二. AbstractSpan AbstractSpan提供了Span内部，进行操作的各项API ```java @@ -141,17 +142,21 @@ AbstractSpan提供了Span内部，进行操作的各项API ``` Span的操作语义和OpenTracing类似。 -SpanLayer为我们的特有概念，如果是远程调用类的服务，请设置此属性，包括4个属性值 +SpanLayer为我们的特有概念，如果是远程调用类的服务，请设置此属性，包括5个属性值 +1. UNKNOWN, 默认 1. DB 1. RPC_FRAMEWORK，非HTTP类型的RPC框架，如：原生的DUBBO，MOTAN 1. HTTP 1. MQ -### 开发插件 -#### 一. 简介 +Component ID被SkyWalking项目组定义和保护。0到10000为保留值，如果你希望贡献新插件，可以在插件pull request通过，并提交的自动化 +测试用户被接收后，申请自己的组件ID。私有插件，请使用10000以上的ID，避免重复。 + +## 开发插件 +### 一. 简介 因为所有的程序调用都是基于方法的，所以插件实际上就是基于方法的拦截，类似面向切面编程的AOP技术。SkyWalking底层已经完成相关的技术封装，所以插件开发者只需要定位需要拦截的类、方法，然后结合上文中的追踪API，即可完成插件的开发。 -#### 二. 拦截类型 +### 二. 拦截类型 根据Java方法，共有三种拦截类型 1. 拦截构造函数 1. 拦截实例方法 @@ -163,7 +168,7 @@ SpanLayer为我们的特有概念，如果是远程调用类的服务，请设 当然，也可以同时支持实例和静态方法，直接继承ClassEnhancePluginDefine。但是，这种情况很少。 -#### 三. 实现自己的插件定义 +### 三. 实现自己的插件定义 我们以继承ClassInstanceMethodsEnhancePluginDefine为例（ClassStaticMethodsEnhancePluginDefine十分类似，不再重复描述），描述定义插件的全过程 1. 定义目标类名称 @@ -171,7 +176,7 @@ SpanLayer为我们的特有概念，如果是远程调用类的服务，请设 protected abstract ClassMatch enhanceClass(); ``` -ClassMatch反应类的匹配方式，目前提供三种： +ClassMatch反应类的匹配方式，目前提供四种： * byName, 通过类名完整匹配 * byClassAnnotationMatch, 通过类标注进行匹配 @@ -192,7 +197,7 @@ protected ClassMatch enhanceClassName() { ``` -2. 定义构造函数拦截点 +2. 定义方法拦截点 ```java protected InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints(); @@ -223,7 +228,7 @@ tomcat-7.x/8.x=TomcatInstrumentation * 插件名称，要求全局唯一，命名规范：目标组件+版本号 * 插件定义类全名 -#### 四. 实现拦截器逻辑 +### 四. 实现拦截器逻辑 我们继续以实现实例方法拦截为例，拦截器需要实现org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstanceMethodsAroundInterceptor。 ```java /** @@ -264,7 +269,7 @@ public interface InstanceMethodsAroundInterceptor { 可以在方法执行前、执行后、执行异常三个点，进行拦截，设置修改方法参数（执行前），并调用核心API，设置追踪逻辑。 -### 贡献插件到主仓库 +## 贡献插件到主仓库 我们鼓励大家共同贡献支持各个类库的插件。 大家需支持以下步骤执行： diff --git a/docs/en/How-to-build.md b/docs/en/How-to-build.md index 90ceda429..b21e504f8 100644 --- a/docs/en/How-to-build.md +++ b/docs/en/How-to-build.md @@ -10,5 +10,5 @@ This document helps people to compile and build the project in your maven and ID 1. Import the project as a maven project 1. Run `mvn compile -Dmaven.test.skip=true` to compile project and generate source codes. Because we use gRPC and protobuf. 1. Set **Generated Source Codes** folders. - * `grpc-java` and `java` folders in **apm-network/target/generated--sources/protobuf** + * `grpc-java` and `java` folders in **apm-protocol/apm-network/target/generated-sources/protobuf** * `grpc-java` and `java` folders in **apm-collector/apm-collector-remote/apm-remote-grpc-provider/target/protobuf** \ No newline at end of file diff --git a/docs/en/Plugin-Development-Guide.md b/docs/en/Plugin-Development-Guide.md new file mode 100644 index 000000000..2174c33d6 --- /dev/null +++ b/docs/en/Plugin-Development-Guide.md @@ -0,0 +1,290 @@ +# Plugin Development Guide +This document describe how to understand, develop and contribute plugin. + +## Concepts +### Span +Span is an important and common concept in distributed tracing system. Learn **Span** from +[Google Dapper Paper](https://research.google.com/pubs/pub36356.html) and +[OpenTracing](http://opentracing.io) + +SkyWalking supports OpenTracing and OpenTracing-Java API from 2017. Our Span concepts are similar with the paper and OpenTracing. +Also we extend the Span. + +There are three types of Span + +1.1 EntrySpan +EntrySpan represents a service provider, also the endpoint of server side. As an APM system, we are targeting the +application servers. So almost all the services and MQ-comsumer are EntrySpan(s). + +1.2 LocalSpan +LocalSpan represents a normal Java method, which don't relate with remote service, neither a MQ producer/comsumer +nor a service(e.g. HTTP service) provider/consumer. + +1.3 ExitSpan +ExitSpan represents a client of service or MQ-producer, as named as `LeafSpan` at early age of SkyWalking. +e.g. accessing DB by JDBC, reading Redis/Memcached are cataloged an ExitSpan. + +### ContextCarrier +In order to implement distributed tracing, the trace across process need to be bind, and the context should propagate +across the process. That is ContextCarrier's duty. + +Here are the steps about how to use **ContextCarrier** in a `A->B` distributed call. +1. Create a new and empty `ContextCarrier` at client side. +1. Create an ExitSpan by `ContextManager#createExitSpan` or use `ContextManager#inject` to init the `ContextCarrier`. +1. Put all items of `ContextCarrier` into heads(e.g. HTTP HEAD), attachments(e.g. Dubbo RPC framework) or messages(e.g. Kafka) +1. The `ContextCarrier` propagates to server side by the service call. +1. At server side, get all items from heads, attachments or messages. +1. Create an EntrySpan by `ContestManager#createEntrySpan` or use `ContextManager#extract` to bind the client and server. + + +Let's demonstrate the steps by Apache HTTPComponent client plugin and Tomcat 7 server plugin +1. Client side steps by Apache HTTPComponent client plugin +```java + span = ContextManager.createExitSpan("/span/operation/name", contextCarrier, "ip:port"); + CarrierItem next = contextCarrier.items(); + while (next.hasNext()) { + next = next.next(); + httpRequest.setHeader(next.getHeadKey(), next.getHeadValue()); + } +``` + +2. Server side steps by Tomcat 7 server plugin +```java + ContextCarrier contextCarrier = new ContextCarrier(); + CarrierItem next = contextCarrier.items(); + while (next.hasNext()) { + next = next.next(); + next.setHeadValue(request.getHeader(next.getHeadKey())); + } + + span = ContextManager.createEntrySpan(“/span/operation/name”, contextCarrier); +``` + +### ContextSnapshot +Besides across process, across thread but in a process need to be supported, because async process(In-memory MQ) +and batch process are common in Java. Across process and across thread are similar, because they are both about propagating +context. The only difference is that, don't need to serialize for across thread. + +Here are the three steps about across thread propagation: +1. Use `ContextManager#capture` to get the ContextSnapshot object. +1. Let the sub-thread access the ContextSnapshot by any way, through method arguments or carried by an existed arguments +1. Use `ContextManager#continued` in sub-thread. + +## Core APIs +### ContextManager +ContextManager provides all major and primary APIs. + +1. Create EntrySpan +```java +public static AbstractSpan createEntrySpan(String operationName, ContextCarrier carrier) +``` +Create EntrySpan by operation name(e.g. service name, uri) and **ContextCarrier**. + +2. Create LocalSpan +```java +public static AbstractSpan createLocalSpan(String operationName) +``` +Create LocalSpan by operation name(e.g. full method signature) + +3. Create ExitSpan +```java +public static AbstractSpan createExitSpan(String operationName, ContextCarrier carrier, String remotePeer) +``` +Create ExitSpan by operation name(e.g. service name, uri) and new **ContextCarrier** and peer address +(e.g. ip+port, hostname+port) + +### AbstractSpan +```java + /** + * Set the component id, which defines in {@link ComponentsDefine} + * + * @param component + * @return the span for chaining. + */ + AbstractSpan setComponent(Component component); + + /** + * Only use this method in explicit instrumentation, like opentracing-skywalking-bridge. + * It it higher recommend don't use this for performance consideration. + * + * @param componentName + * @return the span for chaining. + */ + AbstractSpan setComponent(String componentName); + + AbstractSpan setLayer(SpanLayer layer); + + /** + * Set a key:value tag on the Span. + * + * @return this Span instance, for chaining + */ + AbstractSpan tag(String key, String value); + + /** + * Record an exception event of the current walltime timestamp. + * + * @param t any subclass of {@link Throwable}, which occurs in this span. + * @return the Span, for chaining + */ + AbstractSpan log(Throwable t); + + AbstractSpan errorOccurred(); + + /** + * Record an event at a specific timestamp. + * + * @param timestamp The explicit timestamp for the log record. + * @param event the events + * @return the Span, for chaining + */ + AbstractSpan log(long timestamp, Map event); + + /** + * Sets the string name for the logical operation this span represents. + * + * @return this Span instance, for chaining + */ + AbstractSpan setOperationName(String operationName); +``` +Besides set operation name, tags and logs, two attributes shoule be set, which are component and layer, +especially for EntrySpan and ExitSpan + +SpanLayer is the catalog of span. Here are 5 values: +1. UNKNOWN (default) +1. DB +1. RPC_FRAMEWORK, for a RPC framework, not an ordinary HTTP +1. HTTP +1. MQ + +Component IDs are defined and protected by SkyWalking project, 0 -> 10000 IDs are reserved. If you want to contribute +a new plugin, you can ask for an official ID, after your pull request approved and automatic test cases accepted by PMC. +Please use > 10000 ID, if you are going to develop a private plugin or don't intend to contribute the plugin to community, +to avoid the ID conflict. + +## Develop a plugin +### Abstract +The basic method to trace is intercepting a Java method, by using byte code manipulation tech and AOP concept. +SkyWalking boxed the byte code manipulation tech and tracing context propagation, +so you just need to define the intercept point(a.k.a. aspect pointcut in Spring) + +### Intercept +SkyWalking provide two common defines to intercept Contructor, instance method and class method. +* Extend `ClassInstanceMethodsEnhancePluginDefine` defines `Contructor` intercept points and `instance method` intercept points. +* Extend `ClassStaticMethodsEnhancePluginDefine` definec `class method` intercept points. + +Of course, you can extend `ClassEnhancePluginDefine` to set all intercept points. But it is unusual. + +### Implement plugin +I will demonstrate about how to implement a plugin by extending `ClassInstanceMethodsEnhancePluginDefine` + +1. Define the target class name +```java +protected abstract ClassMatch enhanceClass(); +``` + +ClassMatch represents how to match the target classes, there are 4 ways: +* byName, through the full class name(package name + `.` + class name) +* byClassAnnotationMatch, through the class existed certain annotations. +* byMethodAnnotationMatch, through the class's method existed certain annotations. +* byHierarchyMatch, throught the class's parent classes or interfaces + +**Attentions**: +* Forbid to use `*.class.getName()` to get the class String name. Recommend you to use literal String. This is for +avoiding ClassLoader issues. +* `by*AnnotationMatch` doesn't support the inherited annotations. +* Don't recommend use `byHierarchyMatch`, unless it is really necessary. Because using it may trigger intercepting +many unexcepted methods, which causes performance issues and concerns. + +Example： +```java +@Override +protected ClassMatch enhanceClassName() { + return byName("org.apache.catalina.core.StandardEngineValve"); +} + +``` + +2. Define an instance method intercept point +```java +protected InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints(); + +public interface InstanceMethodsInterceptPoint { + /** + * class instance methods matcher. + * + * @return methods matcher + */ + ElementMatcher getMethodsMatcher(); + + /** + * @return represents a class name, the class instance must instanceof InstanceMethodsAroundInterceptor. + */ + String getMethodsInterceptor(); + + boolean isOverrideArgs(); +} +``` +Also use `Matcher` to set the target methods. Return **true** in `isOverrideArgs`, if you want to change the argument +ref in interceptor. + +The following sections will tell you how to implement the interceptor. + +3. Add plugin define into skywalking-plugin.def file +```properties +tomcat-7.x/8.x=TomcatInstrumentation +``` + + +### Implement an interceptor +As an interceptor for an instance method, the interceptor implements +`org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstanceMethodsAroundInterceptor` +```java +/** + * A interceptor, which intercept method's invocation. The target methods will be defined in {@link + * ClassEnhancePluginDefine}'s subclass, most likely in {@link ClassInstanceMethodsEnhancePluginDefine} + * + * @author wusheng + */ +public interface InstanceMethodsAroundInterceptor { + /** + * called before target method invocation. + * + * @param result change this result, if you want to truncate the method. + * @throws Throwable + */ + void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class[] argumentsTypes, + MethodInterceptResult result) throws Throwable; + + /** + * called after target method invocation. Even method's invocation triggers an exception. + * + * @param ret the method's original return value. + * @return the method's actual return value. + * @throws Throwable + */ + Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class[] argumentsTypes, + Object ret) throws Throwable; + + /** + * called when occur exception. + * + * @param t the exception occur. + */ + void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments, Class[] argumentsTypes, + Throwable t); +} +``` +Use the core APIs in before, after and exception handle stages. + +### Contribute plugins into Apache SkyWalking repository +We are welcome everyone to contribute plugins. + +Please follow there steps: +1. Submit an issue about which plugins are you going to contribute, including supported version. +1. Create sub modules under `apm-sniffer/apm-sdk-plugin`, and the name should include supported library name and versions +1. Follow this guide to develop. Make sure comments and test cases are provided. +1. Develop and test. +1. Send the pull request and ask for review, and provide the automatic test cases by following PMC members guides. +1. The plugin committers approves your plugins after automatic test cases provided and the tests passed in our CI. +1. The plugin accepted by SkyWalking.