logtail_check_point文件数据时有时无

[leizhou666]

现象如下：
pod有20+收集配置

正常情况下checkpoint文件如下：

但是间歇性会出现如下异常情况：

同时：
收集配置check_point_dump_interval时间是300秒，五分钟一次，但是上图里的checkpoint时间竟然只间隔了2分钟

启动日志里面的配置也是5分钟

[leizhou666]

麻烦大佬们帮忙看下这个问题，感谢！

[leizhou666]

还有的其他信息是，我重新指向了checkpoint文件的位置为/app/logs/ilogtail/checkpoint/logtail_check_point
而部署的ilogtail的目录为/usr/local/ilogtail

[linrunqi08]

这个是在容器里还是在主机上？

[leizhou666]

容器上

[linrunqi08]

logtail的deployment贴一下

[leizhou666]

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: ilogtail
  labels:
    k8s-app: log-collector
    app-name: log-collector
spec:
  selector:
    matchLabels:
      name: log-collector
      app-name: log-collector
  template:
    metadata:
      annotations:
        prometheus.io/port: "9249"
        prometheus.io/scrape: "true"
      labels:
        name: log-collector
        app-name: log-collector
    spec:
      volumes:
        - name: run
          hostPath:
            path: /var/run
            type: Directory
        - name: root
          hostPath:
            path: /
            type: Directory
        - name: ilogtail-log
          hostPath:
            path: /data/logs
        - name: ilog-agent-logs
          hostPath:
            path: /data/logs/ilogtail
            type: DirectoryOrCreate
        - name: ilogtail-user-config
          emptyDir: {}
      containers:
        - name: ilogtail
          env:
            - name: ALIYUN_LOG_ENV_TAGS       # add log tags from env
              value: node_name|node_ip
            - name: node_name
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: node_ip
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
            - name: ALIYUN_LOGTAIL_USER_DEFINED_ID
              value: "default"
            - name: cpu_usage_limit
              value: "2"
            - name: mem_usage_limit
              value: "1800"
            - name: max_read_buffer_size
              value: "1048576"
          image: docker.xxxxxx.com/infra/bigdata/rt/ilogtail/ilogtail-1.6.0:v0.0.2
          resources:
            limits:
              cpu: 2000m
              memory: 2Gi
              nvidia.com/gpu: 0
            requests:
              cpu: 500m
              memory: 1Gi
              nvidia.com/gpu: 0
          ports:
            - containerPort: 49999
          volumeMounts:
            - name: run
              mountPath: /var/run
            - name: root
              mountPath: /logtail_host                  # for log access on the node
              mountPropagation: HostToContainer
              readOnly: true
            - name: ilogtail-log
              mountPath: /app/logs/
            - name: ilogtail-user-config
              mountPath: /usr/local/ilogtail/user_yaml_config.d
        - name: log-agent
          volumeMounts:
            - mountPath: /app/logs/ilogtail
              name: ilog-agent-logs
            - name: ilogtail-user-config
              mountPath: /app/pipeline
          args:
            - '-n'
            - 'ilogtail'
            - '-e'
            - 'test'
            - '-c'
            - 'default'
            - '-d'
            - 'log-collector'
            - '-h'
            - $(NodeName)
          env:
            - name: NodeName
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: PodName
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
          image: docker.xxxxxx.com/infra/bigdata/rt/ilogtail/log-agent:v0.0.4
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 1000m
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 1Gi

[linrunqi08]

采集配置，和日志文件ilogtail.LOG麻烦也提供一下吧

[leizhou666]

采集配置其中之一：其他配置都差不多
enable: true
inputs:
- Type: "file_log"
  LogPath: "/app/logs/aeolus"
  FilePattern: "serverlog..log"
  ContainerFile: true
  ContainerInfo:
    K8sNamespaceRegex: "develop"
    K8sPodRegex: "^(aeolus.)$"
processors:
- Type: "processor_split_log_regex"
  SplitRegex: "^\d+-\d+-\d+.\d+:\d+:\d+.*"
  SplitKey: "content"
  PreserveOthers: true
flushers:
- Type: "flusher_kafka_v2"
  Brokers:
  - "xxxxx1.bj.baidu.internal:9092"
  - "xxxxx2.bj.baidu.internal:9092"
  - "xxxxx3.bj.baidu.internal:9092"
  Authentication:
    Kerberos:
      ServiceName: "kafka"
      Realm: "BJ.BAIDU.COM"
      UseKeyTab: true
      Username: "jd-ilogtail"
      ConfigPath: "/usr/local/kerberos/krb5.test.conf"
      KeyTabPath: "/usr/local/kerberos/jd-ilogtail.test.keytab"
  Version: "2.8.1"
  Topic: "bigdata-rt-collector-aeolus-serverlog-test"
  MaxMessageBytes: 20971520

[leizhou666]

apsara_log_conf.json：
{
"Loggers" :
{
"/" :
{
"AsyncFileSink" : "WARNING"
},
"/apsara/sls/ilogtail" :
{
"AsyncFileSink" : "INFO"
},
"/apsara/sls/ilogtail/profile" :
{
"AsyncFileSinkProfile" : "INFO"
},
"/apsara/sls/ilogtail/status" :
{
"AsyncFileSinkStatus" : "INFO"
}
},
"Sinks" :
{
"AsyncFileSink" :
{
"Compress" : "",
"LogFilePath" : "/app/logs/ilogtail/ilogtail.LOG",
"MaxDaysFromModify" : 300,
"MaxLogFileNum" : 10,
"MaxLogFileSize" : 20000000,
"Type" : "AsyncFile"
},
"AsyncFileSinkProfile" :
{
"Compress" : "",
"LogFilePath" : "/app/logs/ilogtail/snapshot/ilogtail_profile.LOG",
"MaxDaysFromModify" : 1,
"MaxLogFileNum" : 61,
"MaxLogFileSize" : 1,
"Type" : "AsyncFile"
},
"AsyncFileSinkStatus" :
{
"Compress" : "",
"LogFilePath" : "/app/logs/ilogtail/snapshot/ilogtail_status.LOG",
"MaxDaysFromModify" : 1,
"MaxLogFileNum" : 61,
"MaxLogFileSize" : 1,
"Type" : "AsyncFile"
}
}
}

[leizhou666]

启动配置：
{
"cpu_usage_limit": 4,
"mem_usage_limit": 4096,
"max_bytes_per_sec": 209715200,
"process_thread_count": 8,
"send_request_concurrency": 40,
"buffer_file_num": 200,
"buffer_file_size": 104857600,
"buffer_file_path": "",
"max_read_buffer_size": 1048576,
"check_point_dump_interval": 300,
"check_point_filename": "/app/logs/ilogtail/checkpoint/logtail_check_point"
}

[leizhou666]

2023-07-06-ilogtail.LOG

[leizhou666]

还有一个额外的信息是logtail_check_point这个文件是打到了目录/app/logs/ilogtail/checkpoint，这个目录是挂载到node磁盘上的；
而docker_path_config.json这个文件是打到默认的目录/usr/local/ilogtail/checkpoint，这个目录没有挂载到node磁盘。
我不确定是不是由于这两个文件没有放到同一个目录导致的

[leizhou666]

你这里两个container，log-agent:v0.0.4这个是做什么的呢？
log-agent是我们专门对业务pod监控的一个进程，监控pod的漂移，在对应的node上写入对应的ilogtail收集配置

[linrunqi08]

可以先把挂载拆分开，再看下吧

[leizhou666]

大佬，在使用过程中有其他用户反馈过这个情况吗？目前是我把多余的那个挂载删了，重启之后还有这个现象

[linrunqi08]

没有人反馈过，你现在的deployment再发一下？采集上来的数据有什么问题吗，比如采集重复？

[leizhou666]

好的，可能是我这边的问题，我先验证下

[leizhou666]

确认了，是我们自己的问题，我们起了两个ilogtail的daemonset，两个daemonset同时写导致的问题，抱歉。

[leizhou666]

请问下我应该提交到哪里呢

[messixukejia]

https://github.com/alibaba/ilogtail/pull/1010/files
可以参考下这个，提交个新的文件即可。

[leizhou666]

大佬，问下你们需要这个文件用来做什么

[messixukejia]

就是将一些常用的配置进行归档，方便开发者借鉴使用

[leizhou666]

好的