# zipkin-dependency

**Repository Path**: oneal/zipkin-dependency

## Basic Information

- **Project Name**: zipkin-dependency
- **Description**: 从ES读取zipkin的调用链信息，计算服务之间的依赖关系。
- **Primary Language**: Scala
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2018-08-28
- **Last Updated**: 2020-12-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

#### 编译
使用 idea 编译，打jar包。

#### 运行
1、把编译好的jar包拷贝到可提交spark-job的机器<br>
2、提交脚本如下,配置到crontab定时跑：
```bash
#!/bin/bash
spark-submit  --class ZipkinDep \
--executor-memory 1g \
--executor-cores 4 \
--num-executors  10 \
--conf spark.yarn.executor.memoryOverhead=4096 \
--conf spark.default.parallelism=100 \
--master yarn \
--deploy-mode cluster \
  /mnt/DATA_MINING/chenjianwen/jar/zipkin-dependecies/zipkin-dependency.jar zipkin_dev xxx:9200,xxx1:9200
```
- 参数一

存放调用链信息的ES index

- 参数二

ES 的地址。

<p style="border-radius:5px;border:solid;box-shadow:0 0 5px black">
<b>spark 性能优化小tips:</b><br>
并行度设置: <span style="color:gray">（决定task的数量，不设置该参数的话，executor数量再多也不会用上）</span><br>
parallelism = executor-cores * num-executors * 2~3
</p>

#### ES 索引使用
```bash
### 线上环境
PUT /_template/zipkin:dependency_template2
{
    "order": 2,
    "template": "zipkin:dependency-*",
    "settings": {
      "index": {
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "mapper": {
          "dynamic": true
        },
        "number_of_shards": "5",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "dependency": {
        "enabled": true,
        "dynamic_templates": [
          {
            "strings": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "ignore_above": 256,
                "norms": true,
                "type": "keyword"
              }
            }
          }
        ],
        "properties": {
          "parent":{
            "type": "keyword"
          },
          "callCount":{
            "type": "long"
          },
          "errorCount":{
            "type": "long"
          },
          "child":{
            "type": "keyword"
          },
          "id":{
            "type": "text",
            "fielddata": true,
            "fields": {
                "raw": { 
                  "type":  "keyword"
                }
            }
          },
          "networkdiff":{
            "type": "long"
          }
        }
      }
    },
    "aliases": {
      "dependency_template2":{}
    }
}
 
### 测试环境
PUT /_template/zipkin_dev:dependency_template2
{
    "order": 2,
    "template": "zipkin_dev:dependency-*",
    "settings": {
      "index": {
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "mapper": {
          "dynamic": true
        },
        "number_of_shards": "5",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "dependency": {
        "enabled": true,
        "dynamic_templates": [
          {
            "strings": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "ignore_above": 256,
                "norms": true,
                "type": "keyword"
              }
            }
          }
        ],
        "properties": {
          "parent":{
            "type": "keyword"
          },
          "callCount":{
            "type": "long"
          },
          "errorCount":{
            "type": "long"
          },
          "child":{
            "type": "keyword"
          },
          "id":{
            "type": "text",
            "fielddata": true,
            "fields": {
                "raw": { 
                  "type":  "keyword"
                }
            }
          },
          "networkdiff":{
            "type": "long"
          }
        }
      }
    },
    "aliases": {
      "dependency_template2":{}
    }
}

```