时序性数据库InfluxDB初探与优化

注意InfluxDB的版本号,不同版本号的语法不同,我使用的是InfluxDB2.3.0版本

官网: InfluxDB OSS 2.3 Documentation (influxdata.com)

1.常用的查询命令

2.生成配置文件

在InfluxDB中,配置文件默认是不存在的,需要执行以下命令进行生成

docker run --rm influxdb:2.3.0 influxd print-config > config.yml

具体内容如下:

assets-path: ""
bolt-path: /var/lib/influxdb2/influxd.bolt
e2e-testing: false
engine-path: /var/lib/influxdb2/engine
feature-flags: {}
flux-log-enabled: false
hardening-enabled: false
http-bind-address: :8086
http-idle-timeout: 3m0s
http-read-header-timeout: 10s
http-read-timeout: 0s
http-write-timeout: 0s
influxql-max-select-buckets: 0
influxql-max-select-point: 0
influxql-max-select-series: 0
instance-id: ""
key-name: ""
log-level: info
metrics-disabled: false
nats-max-payload-bytes: 0
nats-port: 4222
no-tasks: false
pprof-disabled: false
query-concurrency: 1024
query-initial-memory-bytes: 0
query-max-memory-bytes: 0
query-memory-bytes: 0
query-queue-size: 1024
reporting-disabled: false
secret-store: bolt
session-length: 60
session-renew-disabled: false
sqlite-path: ""
storage-cache-max-memory-size: 1073741824
storage-cache-snapshot-memory-size: 26214400
storage-cache-snapshot-write-cold-duration: 10m0s
storage-compact-full-write-cold-duration: 4h0m0s
storage-compact-throughput-burst: 50331648
storage-max-concurrent-compactions: 0
storage-max-index-log-file-size: 1048576
storage-no-validate-field-size: false
storage-retention-check-interval: 30m0s
storage-series-file-max-concurrent-snapshot-compactions: 0
storage-series-id-set-cache-size: 0
storage-shard-precreator-advance-period: 30m0s
storage-shard-precreator-check-interval: 10m0s
storage-tsm-use-madv-willneed: false
storage-validate-keys: false
storage-wal-fsync-delay: 0s
storage-wal-max-concurrent-writes: 0
storage-wal-max-write-delay: 10m0s
storage-write-timeout: 10s
store: disk
testing-always-allow-setup: false
tls-cert: ""
tls-key: ""
tls-min-version: "1.2"
tls-strict-ciphers: false
tracing-type: ""
ui-disabled: false
vault-addr: ""
vault-cacert: ""
vault-capath: ""
vault-client-cert: ""
vault-client-key: ""
vault-client-timeout: 0s
vault-max-retries: 0
vault-skip-verify: false
vault-tls-server-name: ""
vault-token: ""

3.调整Dockerfile文件

调整配置文件映射,并且设置环境变量,指定InfluxDB读取的配置文件夹,

Dockerfile模板如下:

# 时序性数据库
influxdb:
    image: influxdb:2.3.0
    container_name: influxdb
    environment:
        - DOCKER_INFLUXDB_INIT_MODE=setup
        - DOCKER_INFLUXDB_INIT_USERNAME=xxxxx
        - DOCKER_INFLUXDB_INIT_PASSWORD=xxxxx
        - DOCKER_INFLUXDB_INIT_ORG=xxxxx
        - DOCKER_INFLUXDB_INIT_BUCKET=xxxxx
        - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=xxxxx
        - INFLUXD_CONFIG_PATH=/home/   # 设置环境变量
    volumes:
        - /data/zfzn/config.yml:/home/config.yml # 设置映射路径
    ports:
        - "18086:8086"
    restart: always

可以通过api访问获取配置内容

http://{ip}:{port}/api/v2/config

4.调整配置,开始测试

首先,我们使用默认配置启动InfluxDB,执行一个大数据量查询

from(bucket: "zfzn_new")
  |> range(start: -30d)
  |> filter(fn: (r) => r["_measurement"] == "100" or r["_measurement"] == "103" or r["_measurement"] == "101" or r["_measurement"] == "102" or r["_measurement"] == "105" or r["_measurement"] == "108" or r["_measurement"] == "107" or r["_measurement"] == "106" or r["_measurement"] == "109" or r["_measurement"] == "110")
  |> fill(usePrevious: true)
  |> increase(columns: ["_value"])
  |> last()
  |> yield()

使用top查看Influxdb服务内存占用.

调整为最大查询内存1G之后:

优化常用字段:

具体说明参考官方文档: InfluxDB 配置选项

log-level: warn # 日志输出级别
query-concurrency: 5 # 允许并发执行的查询数。 默认10
query-initial-memory-bytes: 67108864 # 64 MB  为查询分配的初始内存字节数。
query-memory-bytes: 2147483648 # 2G 单个查询允许的最大内存字节数。必须大于或等于 query-initial-memory-bytes 。
query-max-memory-bytes: 8589934592 # 8G 允许查询的最大内存总字节数。 等于查询并发×单次查询内存字节数
http-read-header-timeout: 10s  #服务器应尝试读取新请求的 HTTP 标头的最长持续时间。设置为 0 表示没有超时
http-read-timeout: 10s # 读超时
http-write-timeout: 10s # 写超时 
query-queue-size: 10  # 执行队列中允许的最大查询数。当达到队列限制时，新查询将被拒绝。 默认10

storage-cache-max-memory-size: 2147483648 # 2G 控制存储引擎内存缓存的大小，以字节为单位。
storage-cache-snapshot-memory-size: 26214400 # 25M 控制存储引擎快照使用的内存大小，以字节为单位。
storage-cache-snapshot-write-cold-duration: 10m0s # 控制存储引擎冷数据快照写入的时间。
storage-compact-full-write-cold-duration: 4h0m0s # 控制存储引擎数据压缩的时间。
storage-compact-throughput-burst: 50331648
storage-max-concurrent-compactions: 0
storage-max-index-log-file-size: 1048576
storage-no-validate-field-size: false
storage-retention-check-interval: 30m0s
storage-series-file-max-concurrent-snapshot-compactions: 0
storage-series-id-set-cache-size: 0
storage-shard-precreator-advance-period: 30m0s
storage-shard-precreator-check-interval: 10m0s
storage-tsm-use-madv-willneed: false
storage-validate-keys: false
storage-wal-fsync-delay: 0s
storage-wal-max-concurrent-writes: 0
storage-wal-max-write-delay: 10m0s
storage-write-timeout: 10s
store: disk