Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ChangeFeed Events seem to come in out of order #70

Open
Smithx10 opened this issue Dec 16, 2020 · 5 comments
Open

ChangeFeed Events seem to come in out of order #70

Smithx10 opened this issue Dec 16, 2020 · 5 comments

Comments

@Smithx10
Copy link

It appears that the events aren't being sent in order, or are being sent out of order. The timestamps are a bit puzzling.

I believe the normal state changes would come in this order... shutting_down, down, stopped?

1st Stop

Change (2020-12-16T19:30:53.689Z) =>
    modified: state,zone_state
    state: shutting_down
    internal state: shutting_down
    object: 86586a47
Change (2020-12-16T19:30:55.691Z) =>
    modified: last_modified,state,zone_state,boot_timestamp,pid
    state: stopped
    internal state: stopped
    object: 86586a47
Change (2020-12-16T19:30:55.692Z) =>
    modified: last_modified,state,zone_state,pid
    state: down
    internal state: down
    object: 86586a47

shutting_down, stopped, down

2nd Stop

Change (2020-12-16T19:39:42.033Z) =>
    modified: last_modified,state,zone_state,boot_timestamp,pid
    state: stopped
    internal state: stopped
    object: 86586a47
Change (2020-12-16T19:39:42.037Z) =>
    modified: last_modified,state,zone_state,exit_timestamp,pid
    state: down
    internal state: down
    object: 86586a47
Change (2020-12-16T19:39:42.038Z) =>
    modified: state,zone_state,exit_status,exit_timestamp
    state: shutting_down
    internal state: shutting_down
    object: 86586a47

stopped, down, shutting_down
@kusor
Copy link
Contributor

kusor commented Dec 17, 2020

@Smithx10 CloudAPI should dispatch the VMAPI changefeed messages as they arrive. Can you take a look at CloudAPI/VMAPI logs and see if those are arriving in the right order, please?

@Smithx10
Copy link
Author

node-triton

/m/c/U/smith # ❯❯❯ itt changefeed
Change (2020-12-17T12:29:05.758Z) =>
    modified: state,zone_state,exit_status,exit_timestamp
    state: shutting_down
    internal state: shutting_down
    object: 86586a47
Change (2020-12-17T12:29:07.761Z) =>
    modified: state,zone_state,boot_timestamp,pid
    state: stopped
    internal state: stopped
    object: 86586a47
Change (2020-12-17T12:29:07.763Z) =>
    modified: last_modified,state,zone_state,exit_timestamp,pid
    state: down
    internal state: down
    object: 86586a47

CloudAPI

==> /var/svc/log/smartdc-application-cloudapi:cloudapi-8084.log <==
[2020-12-17T12:28:59.073Z] DEBUG: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: _poll: start
[2020-12-17T12:29:05.759Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: _receivedText: start
    cfText: {"changeKind":{"resource":"vm","subResources":["state","zone_state","exit_status","exit_timestamp"]},"changedResourceId":"86586a47-aaec-4874-bd40-7da727827966","resourceObject":{"uuid":"86586a47-aaec-4874-bd40-7da727827966","alias":"test0","autoboot":false,"brand":"lx","billing_id":"3feff713-e6b4-e40b-d604-b1da840f19ba","cpu_cap":200,"cpu_shares":128,"create_timestamp":"2020-12-16T19:26:18.781Z","customer_metadata":{"root_authorized_keys":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/\n"},"firewall_enabled":false,"internal_metadata":{},"last_modified":"2020-12-17T12:29:04.000Z","limit_priv":"default","max_locked_memory":2048,"max_lwps":4000,"max_physical_memory":2048,"max_swap":8192,"nics":[{"interface":"eth0","mac":"90:b8:d0:ec:90:0c","vlan_id":0,"nic_tag":"external","gateway":"10.0.0.2","gateways":["10.0.0.2"],"netmask":"255.255.255.0","ip":"10.0.0.107","ips":["10.0.0.107/24"],"network_uuid":"035edf21-dd17-4ca9-97dd-92f78c7f766d","mtu":1500,"primary":true}],"owner_uuid":"d38a23dc-5e70-49d8-9c2e-a25efe63d5a3","platform_buildstamp":"20201215T012335Z","quota":50,"resolvers":["8.8.8.8","8.8.4.4"],"server_uuid":"564d35db-b074-4340-327e-239a75d7871b","snapshots":[],"state":"shutting_down","tags":{},"zfs_filesystem":"zones/86586a47-aaec-4874-bd40-7da727827966","zfs_io_priority":128,"zone_state":"shutting_down","zonepath":"/zones/86586a47-aaec-4874-bd40-7da727827966","zpool":"zones","boot_timestamp":"2020-12-17T12:28:46.000Z","dns_domain":"local","exit_status":0,"exit_timestamp":"2020-12-16T19:39:40.859Z","kernel_version":"4.3.0","pid":81887,"tmpfs":2048,"image_uuid":"7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b"},"published":"1608208145758"}
[2020-12-17T12:29:05.759Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: change item received
    cfItem: {
      "changeKind": {
        "resource": "vm",
        "subResources": [
          "state",
          "zone_state",
          "exit_status",
          "exit_timestamp"
        ]
      },
      "changedResourceId": "86586a47-aaec-4874-bd40-7da727827966",
      "resourceObject": {
        "uuid": "86586a47-aaec-4874-bd40-7da727827966",
        "alias": "test0",
        "autoboot": false,
        "brand": "lx",
        "billing_id": "3feff713-e6b4-e40b-d604-b1da840f19ba",
        "cpu_cap": 200,
        "cpu_shares": 128,
        "create_timestamp": "2020-12-16T19:26:18.781Z",
        "customer_metadata": {
          "root_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/\n"
        },
        "firewall_enabled": false,
        "internal_metadata": {},
        "last_modified": "2020-12-17T12:29:04.000Z",
        "limit_priv": "default",
        "max_locked_memory": 2048,
        "max_lwps": 4000,
        "max_physical_memory": 2048,
        "max_swap": 8192,
        "nics": [
          {
            "interface": "eth0",
            "mac": "90:b8:d0:ec:90:0c",
            "vlan_id": 0,
            "nic_tag": "external",
            "gateway": "10.0.0.2",
            "gateways": [
              "10.0.0.2"
            ],
            "netmask": "255.255.255.0",
            "ip": "10.0.0.107",
            "ips": [
              "10.0.0.107/24"
            ],
            "network_uuid": "035edf21-dd17-4ca9-97dd-92f78c7f766d",
            "mtu": 1500,
            "primary": true
          }
        ],
        "owner_uuid": "d38a23dc-5e70-49d8-9c2e-a25efe63d5a3",
        "platform_buildstamp": "20201215T012335Z",
        "quota": 50,
        "resolvers": [
          "8.8.8.8",
          "8.8.4.4"
        ],
        "server_uuid": "564d35db-b074-4340-327e-239a75d7871b",
        "snapshots": [],
        "state": "shutting_down",
        "tags": {},
        "zfs_filesystem": "zones/86586a47-aaec-4874-bd40-7da727827966",
        "zfs_io_priority": 128,
        "zone_state": "shutting_down",
        "zonepath": "/zones/86586a47-aaec-4874-bd40-7da727827966",
        "zpool": "zones",
        "boot_timestamp": "2020-12-17T12:28:46.000Z",
        "dns_domain": "local",
        "exit_status": 0,
        "exit_timestamp": "2020-12-16T19:39:40.859Z",
        "kernel_version": "4.3.0",
        "pid": 81887,
        "tmpfs": 2048,
        "image_uuid": "7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b"
      },
      "published": "1608208145758"
    }
[2020-12-17T12:29:05.760Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: _read: start
[2020-12-17T12:29:07.763Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: _receivedText: start
    cfText: {"changeKind":{"resource":"vm","subResources":["state","zone_state","boot_timestamp","pid"]},"changedResourceId":"86586a47-aaec-4874-bd40-7da727827966","resourceObject":{"uuid":"86586a47-aaec-4874-bd40-7da727827966","alias":"test0","autoboot":false,"brand":"lx","billing_id":"3feff713-e6b4-e40b-d604-b1da840f19ba","cpu_cap":200,"cpu_shares":128,"create_timestamp":"2020-12-16T19:26:18.781Z","customer_metadata":{"root_authorized_keys":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/\n"},"firewall_enabled":false,"internal_metadata":{},"last_modified":"2020-12-17T12:29:06.000Z","limit_priv":"default","max_locked_memory":2048,"max_lwps":4000,"max_physical_memory":2048,"max_swap":8192,"nics":[{"interface":"eth0","mac":"90:b8:d0:ec:90:0c","vlan_id":0,"nic_tag":"external","gateway":"10.0.0.2","gateways":["10.0.0.2"],"netmask":"255.255.255.0","ip":"10.0.0.107","ips":["10.0.0.107/24"],"network_uuid":"035edf21-dd17-4ca9-97dd-92f78c7f766d","mtu":1500,"primary":true}],"owner_uuid":"d38a23dc-5e70-49d8-9c2e-a25efe63d5a3","platform_buildstamp":"20201215T012335Z","quota":50,"resolvers":["8.8.8.8","8.8.4.4"],"server_uuid":"564d35db-b074-4340-327e-239a75d7871b","snapshots":[],"state":"stopped","tags":{},"zfs_filesystem":"zones/86586a47-aaec-4874-bd40-7da727827966","zfs_io_priority":128,"zone_state":"installed","zonepath":"/zones/86586a47-aaec-4874-bd40-7da727827966","zpool":"zones","dns_domain":"local","exit_status":0,"exit_timestamp":"2020-12-17T12:29:06.299Z","kernel_version":"4.3.0","tmpfs":2048,"image_uuid":"7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b"},"published":"1608208147761"}
[2020-12-17T12:29:07.764Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: change item received
    cfItem: {
      "changeKind": {
        "resource": "vm",
        "subResources": [
          "state",
          "zone_state",
          "boot_timestamp",
          "pid"
        ]
      },
      "changedResourceId": "86586a47-aaec-4874-bd40-7da727827966",
      "resourceObject": {
        "uuid": "86586a47-aaec-4874-bd40-7da727827966",
        "alias": "test0",
        "autoboot": false,
        "brand": "lx",
        "billing_id": "3feff713-e6b4-e40b-d604-b1da840f19ba",
        "cpu_cap": 200,
        "cpu_shares": 128,
        "create_timestamp": "2020-12-16T19:26:18.781Z",
        "customer_metadata": {
          "root_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/\n"
        },
        "firewall_enabled": false,
        "internal_metadata": {},
        "last_modified": "2020-12-17T12:29:06.000Z",
        "limit_priv": "default",
        "max_locked_memory": 2048,
        "max_lwps": 4000,
        "max_physical_memory": 2048,
        "max_swap": 8192,
        "nics": [
          {
            "interface": "eth0",
            "mac": "90:b8:d0:ec:90:0c",
            "vlan_id": 0,
            "nic_tag": "external",
            "gateway": "10.0.0.2",
            "gateways": [
              "10.0.0.2"
            ],
            "netmask": "255.255.255.0",
            "ip": "10.0.0.107",
            "ips": [
              "10.0.0.107/24"
            ],
            "network_uuid": "035edf21-dd17-4ca9-97dd-92f78c7f766d",
            "mtu": 1500,
            "primary": true
          }
        ],
        "owner_uuid": "d38a23dc-5e70-49d8-9c2e-a25efe63d5a3",
        "platform_buildstamp": "20201215T012335Z",
        "quota": 50,
        "resolvers": [
          "8.8.8.8",
          "8.8.4.4"
        ],
        "server_uuid": "564d35db-b074-4340-327e-239a75d7871b",
        "snapshots": [],
        "state": "stopped",
        "tags": {},
        "zfs_filesystem": "zones/86586a47-aaec-4874-bd40-7da727827966",
        "zfs_io_priority": 128,
        "zone_state": "installed",
        "zonepath": "/zones/86586a47-aaec-4874-bd40-7da727827966",
        "zpool": "zones",
        "dns_domain": "local",
        "exit_status": 0,
        "exit_timestamp": "2020-12-17T12:29:06.299Z",
        "kernel_version": "4.3.0",
        "tmpfs": 2048,
        "image_uuid": "7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b"
      },
      "published": "1608208147761"
    }
[2020-12-17T12:29:07.764Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: _read: start
[2020-12-17T12:29:07.765Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: _receivedText: start
    cfText: {"changeKind":{"resource":"vm","subResources":["last_modified","state","zone_state","exit_timestamp","pid"]},"changedResourceId":"86586a47-aaec-4874-bd40-7da727827966","resourceObject":{"uuid":"86586a47-aaec-4874-bd40-7da727827966","alias":"test0","autoboot":false,"brand":"lx","billing_id":"3feff713-e6b4-e40b-d604-b1da840f19ba","cpu_cap":200,"cpu_shares":128,"create_timestamp":"2020-12-16T19:26:18.781Z","customer_metadata":{"root_authorized_keys":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/\n"},"firewall_enabled":false,"internal_metadata":{},"last_modified":"2020-12-17T12:29:06.000Z","limit_priv":"default","max_locked_memory":2048,"max_lwps":4000,"max_physical_memory":2048,"max_swap":8192,"nics":[{"interface":"eth0","mac":"90:b8:d0:ec:90:0c","vlan_id":0,"nic_tag":"external","gateway":"10.0.0.2","gateways":["10.0.0.2"],"netmask":"255.255.255.0","ip":"10.0.0.107","ips":["10.0.0.107/24"],"network_uuid":"035edf21-dd17-4ca9-97dd-92f78c7f766d","mtu":1500,"primary":true}],"owner_uuid":"d38a23dc-5e70-49d8-9c2e-a25efe63d5a3","platform_buildstamp":"20201215T012335Z","quota":50,"resolvers":["8.8.8.8","8.8.4.4"],"server_uuid":"564d35db-b074-4340-327e-239a75d7871b","snapshots":[],"state":"down","tags":{},"zfs_filesystem":"zones/86586a47-aaec-4874-bd40-7da727827966","zfs_io_priority":128,"zone_state":"down","zonepath":"/zones/86586a47-aaec-4874-bd40-7da727827966","zpool":"zones","boot_timestamp":"2020-12-17T12:28:46.000Z","dns_domain":"local","exit_status":0,"exit_timestamp":"2020-12-17T12:29:06.299Z","kernel_version":"4.3.0","pid":4294967295,"tmpfs":2048,"image_uuid":"7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b"},"published":"1608208147763"}
[2020-12-17T12:29:07.766Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: change item received
    cfItem: {
      "changeKind": {
        "resource": "vm",
        "subResources": [
          "last_modified",
          "state",
          "zone_state",
          "exit_timestamp",
          "pid"
        ]
      },
      "changedResourceId": "86586a47-aaec-4874-bd40-7da727827966",
      "resourceObject": {
        "uuid": "86586a47-aaec-4874-bd40-7da727827966",
        "alias": "test0",
        "autoboot": false,
        "brand": "lx",
        "billing_id": "3feff713-e6b4-e40b-d604-b1da840f19ba",
        "cpu_cap": 200,
        "cpu_shares": 128,
        "create_timestamp": "2020-12-16T19:26:18.781Z",
        "customer_metadata": {
          "root_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/\n"
        },
        "firewall_enabled": false,
        "internal_metadata": {},
        "last_modified": "2020-12-17T12:29:06.000Z",
        "limit_priv": "default",
        "max_locked_memory": 2048,
        "max_lwps": 4000,
        "max_physical_memory": 2048,
        "max_swap": 8192,
        "nics": [
          {
            "interface": "eth0",
            "mac": "90:b8:d0:ec:90:0c",
            "vlan_id": 0,
            "nic_tag": "external",
            "gateway": "10.0.0.2",
            "gateways": [
              "10.0.0.2"
            ],
            "netmask": "255.255.255.0",
            "ip": "10.0.0.107",
            "ips": [
              "10.0.0.107/24"
            ],
            "network_uuid": "035edf21-dd17-4ca9-97dd-92f78c7f766d",
            "mtu": 1500,
            "primary": true
          }
        ],
        "owner_uuid": "d38a23dc-5e70-49d8-9c2e-a25efe63d5a3",
        "platform_buildstamp": "20201215T012335Z",
        "quota": 50,
        "resolvers": [
          "8.8.8.8",
          "8.8.4.4"
        ],
        "server_uuid": "564d35db-b074-4340-327e-239a75d7871b",
        "snapshots": [],
        "state": "down",
        "tags": {},
        "zfs_filesystem": "zones/86586a47-aaec-4874-bd40-7da727827966",
        "zfs_io_priority": 128,
        "zone_state": "down",
        "zonepath": "/zones/86586a47-aaec-4874-bd40-7da727827966",
        "zpool": "zones",
        "boot_timestamp": "2020-12-17T12:28:46.000Z",
        "dns_domain": "local",
        "exit_status": 0,
        "exit_timestamp": "2020-12-17T12:29:06.299Z",
        "kernel_version": "4.3.0",
        "pid": 4294967295,
        "tmpfs": 2048,
        "image_uuid": "7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b"
      },
      "published": "1608208147763"
    }
[2020-12-17T12:29:07.766Z] TRACE: cloudapi/changefeed/26732 on 703eebcb-64ad-4af1-8fb4-846f8585d58c: cf: _read: start

@kusor
Copy link
Contributor

kusor commented Dec 17, 2020

@Smithx10 This time those arrive in the right order, correct? The only thing I can see there is that it's translating installed?

Anyway, in order to check if the order of events looks like something unexpected, we should turn log level to trace on VMAPI, review the logs of the publisher there (which should just match with the client) and compare with vm-agent logs into the CN where the VM ins running. Wondering this might be due to small differences when updating object values in moray, but cannot be sure w/o all the info.

@Smithx10
Copy link
Author

@kusor I attempted to set vmapi loglevel to trace and didn't see any events or trace logs... only Info.

[root@d75a1d71-1c8a-452d-8e67-31d81a1cbae6 (tiger:vmapi0) ~]# cat /opt/smartdc/vmapi/config.json  | grep trace
    "logLevel": "trace",
[root@d75a1d71-1c8a-452d-8e67-31d81a1cbae6 (tiger:vmapi0) ~]# svcadm restart vmapi
[root@d75a1d71-1c8a-452d-8e67-31d81a1cbae6 (tiger:vmapi0) ~]# svcs | grep vmapi
online          2:07:23 svc:/smartdc/site/vmapi:default
[root@d75a1d71-1c8a-452d-8e67-31d81a1cbae6 (tiger:vmapi0) ~]# date
Fri Dec 18 02:07:29 UTC 2020
[root@d75a1d71-1c8a-452d-8e67-31d81a1cbae6 (tiger:vmapi0) ~]# tail -f `svcs -L vmapi` | bunyan
[2020-12-18T02:07:24.469Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: Reindexing bucket vmapi_server_vms...
[2020-12-18T02:07:24.477Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: Bucket vmapi_server_vms reindexed successfully
[2020-12-18T02:07:24.478Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: Reindexing bucket vmapi_vm_role_tags...
[2020-12-18T02:07:24.538Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: Bucket vmapi_vm_role_tags reindexed successfully
[2020-12-18T02:07:24.538Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: Reindexing bucket vmapi_vm_migrations...
[2020-12-18T02:07:24.607Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: Bucket vmapi_vm_migrations reindexed successfully
[2020-12-18T02:07:25.542Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: countVms filter (filter=(&(!(state=destroyed))(!(state=failed))))
[2020-12-18T02:07:25.607Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: listVms filter (filter=(&(!(state=destroyed))(!(state=failed))))
[2020-12-18T02:07:25.739Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: countVms filter (filter=(&(!(state=destroyed))(!(state=failed))))
[2020-12-18T02:07:25.804Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: listVms filter (filter=(&(!(state=destroyed))(!(state=failed))))
[2020-12-18T02:07:39.896Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: countVms filter
    filter: (&(!(internal_metadata_search_array=sdc:system_role=nfsvolumestorage))(owner_uuid=d38a23dc-5e70-49d8-9c2e-a25efe63d5a3)(alias=test0*)(&(!(state=destroyed))(!(state=failed))))
[2020-12-18T02:07:39.904Z]  INFO: moray/33704 on d75a1d71-1c8a-452d-8e67-31d81a1cbae6: listVms filter
    filter: (&(!(internal_metadata_search_array=sdc:system_role=nfsvolumestorage))(owner_uuid=d38a23dc-5e70-49d8-9c2e-a25efe63d5a3)(alias=test0*)(&(!(state=destroyed))(!(state=failed))))

@kusor
Copy link
Contributor

kusor commented Dec 18, 2020

@Smithx10 if I'm not on a mistake you should either disable the config-agent service or modify the log level at /opt/smartdc/vmapi/sapi_manifests/vmapi/template in order to have the change persist config-agent possible changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kusor @Smithx10