Skywalking使用Zookeeper动态分配agent配置-gongstring(野渔)的博客-技术人有正能量

## 1.背景

按照Skywalking官网的客户端搭建方式，基本采取配置agent.properties文件，或者通过java -D 带参数方式（也可以直接使用环境变量进行配置），这些操作办法都属于静态配置。如果在业务高峰期，可能需要调整采样率 *agent.sample_n_per_3_secs* 的数值，只能通过重新启动agent方式更新配置信息。

那么如何才能做到通过后台，动态控制agent端的采样率、链路跨度等配置信息呢？正巧，Skywalking提供了动态更新功能，

### 1.1官网文档索引

agent支持动态代理：https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/readme/#dynamic-configurations
详细说明：
https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/configuration-discovery/

客户端允许动态更新的参数：

|Key |参数描述|参数示例|依赖插件|
|---|---|---|---|
|agent.sample_n_per_3_secs|每3秒采集链路数据,`-1`表示全部收集|-1|-|
|agent.ignore_suffix|不记录链路信息url后缀（第一个节点开始算起），如果多个，采用逗号`,`隔开|.txt,.log|-|
|agent.trace.ignore_path|不记录调用链的路径，多个可以采用逗号`,`隔开 [详细](https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/agent-optional-plugins/trace-ignore-plugin/)|`/your/path/1/**,/your/path/2/**`|apm-trace-ignore-plugin|
|agent.span_limit_per_segment|链路最大跨度|`300`|-|

也就是说，skywalking提供能动态修改上面配置的方案。那么具体要怎么才能实现呢，需要agent和Skywalking Server端共同配置才能实现。
服务端配置官网文档：https://skywalking.apache.org/docs/main/v8.6.0/en/setup/backend/dynamic-config/

SkyWalking Configurations mostly are set through application.yml and OS system environment variables. At the same time, some of them are supporting dynamic settings from upstream management system.

Right now, SkyWalking supports following dynamic configurations.

... 省略具体参数，详细见上面官网链接

| Config Key |Value Description|Value Format Example|
|---|---|---|
|configuration-discovery.default.agentConfigurations|The ConfigurationDiscovery settings，agent端动态配置key|look at [configuration-discovery.md](https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/configuration-discovery)|

其中第一列是表示key值，后面是描述及示例；

官网还有提到，服务端动态参数配置，可以支持多种模式，大致列举如下：

* `grpc` 表示可以直接通过调用grpc接口修改配置
* `zookeeper` 表示可以直接集成zookeeper实现配置刷新
* `etcd` 表示可以通过etcd配置中心的方式实现动态更新配置
* `Consul` 表示可以通过监听Consul配置进行动态更新
* `Apollo` 表示可以集成Apollo配置中心实现动态更新
* `Nacos` 表示可以集成Nacos配置中心实现动态更新
* `configmap` 表示在云原生k8s容器内部，可以通过自带配置管理功能监听来实现动态更新

默认此功能是关闭的，需要在`config/application.yml`文件中，修改如下配置，决定使用哪一个作为监听来源。注意：这里只能同时选择一种，防止出现数据不一致导致服务混乱问题。
```
configuration:
  selector: ${SW_CONFIGURATION:none}
```

当然上述配置均可以直接采用服务端修改环境变量方式实现，减少对文件的个性化修改，本文也是采用环境变量模式。

综上所属，根据查看源码流程所得，agent端在启动的时候，会开启配置变更监听，直接连接server端接口；而server端在启动的时候，会去监听所选择的配置来源，例如本文接下来介绍的使用zookeeper为例，配置修改变化下发流程如下：

zookeeper节点数据变化 --> server监听到变化 --> agent监听到变化 --> 修改本地相关参数值

### 1.2 zk节点路径

下面是`application.yml`中zk配置的采样率作为入口，寻找zk节点路径规则:

```
zookeeper:
    period: ${SW_CONFIG_ZK_PERIOD:60} # Unit seconds, sync period. Default fetch every 60 seconds.
    nameSpace: ${SW_CONFIG_ZK_NAMESPACE:/default}
    hostPort: ${SW_CONFIG_ZK_HOST_PORT:localhost:2181}
    # Retry Policy
    baseSleepTimeMs: ${SW_CONFIG_ZK_BASE_SLEEP_TIME_MS:1000} # initial amount of time to wait between retries
    maxRetries: ${SW_CONFIG_ZK_MAX_RETRIES:3} # max number of times to retry
```

对应了服务端源码：`org.apache.skywalking.oap.server.configuration.zookeeper.ZookeeperServerSettings`
```
    private String nameSpace = "/default";
    private String hostPort;
    private int baseSleepTimeMs = 1000;
    private int maxRetries = 3;
    private int period = 60;
```

其中nameSpace可以自行修改。
服务端启动监听类`org.apache.skywalking.oap.server.configuration.zookeeper.ZookeeperConfigWatcherRegister`

```

public class ZookeeperConfigWatcherRegister extends ConfigWatcherRegister {
    private final PathChildrenCache childrenCache;
    private final String prefix;

public ZookeeperConfigWatcherRegister(ZookeeperServerSettings settings) throws Exception {
        super(settings.getPeriod());
        prefix = settings.getNameSpace() + "/";
        RetryPolicy retryPolicy = new ExponentialBackoffRetry(settings.getBaseSleepTimeMs(), settings.getMaxRetries());
        CuratorFramework client = CuratorFrameworkFactory.newClient(settings.getHostPort(), retryPolicy);
        client.start();
        this.childrenCache = new PathChildrenCache(client, settings.getNameSpace(), true);
        this.childrenCache.start();
    }

@Override
    public Optional<ConfigTable> readConfig(Set<String> keys) {
        ConfigTable table = new ConfigTable();
        keys.forEach(s -> {
            ChildData data = this.childrenCache.getCurrentData(this.prefix + s);
            table.add(new ConfigTable.ConfigItem(s, data == null ? null : new String(data.getData())));
        });
        return Optional.of(table);
    }
}

```

`prefix = settings.getNameSpace() + "/";`：表示URL前缀是在上面`application.yml`中配置的nameSpace，本文环境变量配置示例`export SW_CONFIG_ZK_NAMESPACE=/skywalking/gongstring/agent`

`ChildData data = this.childrenCache.getCurrentData(this.prefix + s);`：表示监听的ZK节点路径，需要将上面的URL前缀与当前节点key合并，例如agent相关的配置，服务端配置的key`configuration-discovery.default.agentConfigurations`，所以使用zookeeper做配置下发的时候，需要在zk中找到如下节点（服务启动会自动创建，如果没有手工创建）:`/skywalking/gongstring/agent/configuration-discovery.default.agentConfigurations`

### 1.3节点数据

下面是服务启动监听node节点源码：
`org.apache.skywalking.apm.agent.core.conf.dynamic.ConfigurationDiscoveryService`

```
/**
     * Register dynamic configuration watcher.
     *
     * @param watcher dynamic configuration watcher
     */
    public void registerAgentConfigChangeWatcher(AgentConfigChangeWatcher watcher) {
        WatcherHolder holder = new WatcherHolder(watcher);
        if (register.containsKey(holder.getKey())) {
            throw new IllegalStateException("Duplicate register, watcher=" + watcher);
        }
        register.put(holder.getKey(), holder);
    }

/**
     * Process ConfigurationDiscoveryCommand and notify each configuration watcher.
     *
     * @param configurationDiscoveryCommand Describe dynamic configuration information
     */
    public void handleConfigurationDiscoveryCommand(ConfigurationDiscoveryCommand configurationDiscoveryCommand) {
        final String responseUuid = configurationDiscoveryCommand.getUuid();

if (responseUuid != null && Objects.equals(this.uuid, responseUuid)) {
            return;
        }

List<KeyStringValuePair> config = readConfig(configurationDiscoveryCommand);

config.forEach(property -> {
            String propertyKey = property.getKey();
            WatcherHolder holder = register.get(propertyKey);
            if (holder != null) {
                AgentConfigChangeWatcher watcher = holder.getWatcher();
                String newPropertyValue = property.getValue();
                if (StringUtil.isBlank(newPropertyValue)) {
                    if (watcher.value() != null) {
                        // Notify watcher, the new value is null with delete event type.
                        watcher.notify(
                            new AgentConfigChangeWatcher.ConfigChangeEvent(
                                null, AgentConfigChangeWatcher.EventType.DELETE
                            ));
                    } else {
                        // Don't need to notify, stay in null.
                    }
                } else {
                    if (!newPropertyValue.equals(watcher.value())) {
                        watcher.notify(new AgentConfigChangeWatcher.ConfigChangeEvent(
                            newPropertyValue, AgentConfigChangeWatcher.EventType.MODIFY
                        ));
                    } else {
                        // Don't need to notify, stay in the same config value.
                    }
                }
            } else {
                LOGGER.warn("Config {} from OAP, doesn't match any watcher, ignore.", propertyKey);
            }
        });
        this.uuid = responseUuid;

LOGGER.trace("Current configurations after the sync, configurations:{}", register.toString());
    }

```
该方法是将读取参数添加到待监听列表中，在agent端的，`org.apache.skywalking.apm.agent.core.sampling.SamplingService`源码中，有关于监听采样率初始化的key值，关键代码：
```
@Override
    public void boot() {
        samplingRateWatcher = new SamplingRateWatcher("agent.sample_n_per_3_secs", this);
        ServiceManager.INSTANCE.findService(ConfigurationDiscoveryService.class)
                               .registerAgentConfigChangeWatcher(samplingRateWatcher);

handleSamplingRateChanged();
    }
```

到此处，说明，需要从服务端读取并监听`agent.sample_n_per_3_secs`数据变化，同时参考官网说明：https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/configuration-discovery/

```
configurations:
  //service name
  serviceA:
    // Configurations of service A
    // Key and Value are determined by the agent side.
    // Check the agent setup doc for all available configurations.
    key1: value1
    key2: value2
    ...
  serviceB:
    ...
```

表示如果要实现动态修改功能，只需要修改zk节点的数据值为上面这个格式即可。

下面是我的配置示例。

## 2.示例配置

### 2.1环境变量

```
export SW_STORAGE=elasticsearch
export SW_STORAGE_ES_CLUSTER_NODES=192.168.0.4:9200
export SW_KAFKA_FETCHER=default
export SW_KAFKA_FETCHER_SERVERS=192.168.0.4:9093,192.168.0.4:9094,192.168.0.4:9095
export SW_NAMESPACE=gongstring
export SW_CONFIGURATION=zookeeper
export SW_CONFIG_ZK_NAMESPACE=/skywalking/gongstring/agent
export SW_CONFIG_ZK_HOST_PORT=192.168.0.4:2181
export SW_CONFIG_ZK_PERIOD=3
```

### 2.2 zookeeper节点数据配置

zk节点路径：`/skywalking/gongstring/agent/configuration-discovery.default.agentConfigurations`

```
configurations:
  dubbo.provider:
    agent.sample_n_per_3_secs: 5
  dubbo.consumer:
    agent.sample_n_per_3_secs: 2
```

### 2.3 agent配置

provider:
```
-javaagent:D:/dev/tools/skywalking/agent/skywalking-agent.jar
-DSW_AGENT_NAMESPACE=gongstring
-DSW_AGENT_NAME=dubbo.provider
-DSW_AGENT_COLLECTOR_BACKEND_SERVICES=192.168.0.4:11800
-Dplugin.dubbo.collect_provider_arguments=true
-Dplugin.dubbo.collect_consumer_arguments=true
```
consumer:
```
-javaagent:D:/dev/tools/skywalking/agent/skywalking-agent.jar
-DSW_AGENT_NAMESPACE=gongstring
-DSW_AGENT_NAME=dubbo.consumer
-DSW_AGENT_COLLECTOR_BACKEND_SERVICES=192.168.0.4:11800
-Dplugin.dubbo.collect_provider_arguments=true
-Dplugin.dubbo.collect_consumer_arguments=true
```