Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The data synchronization process with Monstache stop #733

Open
Rena078989-art opened this issue Sep 27, 2024 · 0 comments
Open

The data synchronization process with Monstache stop #733

Rena078989-art opened this issue Sep 27, 2024 · 0 comments

Comments

@Rena078989-art
Copy link

Synchronizing data from MongoDB to Elasticsearch using Monstache.
However, there is a discrepancy in the number of records: MongoDB has 26,169,353 entries, while Elasticsearch only has 24,603,904 entries.

Monstache pauses before it can complete the synchronization of all records from MongoDB to Elasticsearch.

Can you please explain the possible reasons for the discrepancy in the number of records between MongoDB and Elasticsearch?
This issue has not been resolved for more than two weeks.

Monstache configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: monstache.config02
  namespace: prod
data:
  monstache.config02.toml: |
          mongo-url = "mongodb://adm000p:[email protected]:27017/?connect=direct"
          mongo-config-url="mongodb://paos-mongocfg-0.mongodb-configdb-service.prod.svc.cluster.local:27017,paos-mongocfg-1.mongodb-configdb-service.prod.svc.cluster.local:27017,paos-mongocfg-2.mongodb-configdb-service.prod.svc.cluster.local:27017"
          elasticsearch-urls = ["https://es.prod.oke.com:9200"]
          elasticsearch-user = "elastic"
          elasticsearch-password = "Cw4Wk4R2M6bzpz4L7Nt331U0"
          elasticsearch-max-seconds = 5
          elasticsearch-max-conns = 6
          elasticsearch-retry = true
          #  elasticsearch-pem-file = "/app/pem/tls.pem"
          elasticsearch-validate-pem-file = false
          # direct-read-namespaces = ["AOS_P_AP_MG.normal-search","AOS_P_AP_MG.zenkoku-search","AOS_P_AP_MG.fav-kensaku-joken"]
          change-stream-namespaces = ["AOS_P_AP_MG.normal-search1","AOS_P_AP_MG.zenkoku-search1","AOS_P_AP_MG.normal-search3","AOS_P_AP_MG.zenkoku-search3"]

          verbose = true
          stats = true

          enable-http-server = true
          cluster-name ="AOS_P_AP_MG1"

          [[mapping]]
          namespace = "AOS_P_AP_MG.normal-search1"
          index = "normal-search1"

          [[mapping]]
          namespace = "AOS_P_AP_MG.zenkoku-search1"
          index = "zenkoku-search1"

          [[mapping]]
          namespace = "AOS_P_AP_MG.normal-search3"
          index = "normal-search3"

          [[mapping]]
          namespace = "AOS_P_AP_MG.zenkoku-search3"
          index = "zenkoku-search3"

          [gtm-settings]
          buffer-size = 16
          channel-size = 128
          buffer-duration = "1s"
          max-await-time = "5s"

          [logs]
          error = "/app/monstache"
          trace = "/app/monstache"

Monstache Deployment yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: monstache-prod-6
  namespace: prod
  labels:
    app: monstache
spec:
  selector:
    matchLabels:
      app: monstache
  replicas: 1
  template:
    metadata:
      labels:
        app: monstache
    spec:
      nodeSelector:
        name: aos-prod-oke-db-pool1
      containers:
      - name: monstache-container
        image: rwynn/monstache:6.7.17
        command:
        - "/bin/sh"
        - "-c"
        args:
        - "monstache -f /mnt/monstache.config02.toml"
        volumeMounts:
        - name: monstache-storage-prod
          mountPath: "/app"
        - name: monstache-config02
          mountPath: "/mnt"
      volumes:
      - name: monstache-storage-prod
        persistentVolumeClaim:
          claimName: monstache-storage-claim-prod-6
      - name: monstache-config02
        configMap:
          name: monstache.config02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant