Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node controller writes invalid instance.xml #4

Open
sjones4 opened this issue Feb 12, 2018 · 0 comments
Open

Node controller writes invalid instance.xml #4

sjones4 opened this issue Feb 12, 2018 · 0 comments
Labels
from jira

Comments

@sjones4
Copy link
Member

sjones4 commented Feb 12, 2018

Issue copied from JIRA https://eucalyptus.atlassian.net/browse/EUCA-13444

The node controller can output invalid instance.xml files when separate threads attempt concurrent updates.

An example is this sequence of writes to an instance.xml file when an instance is launched:

Pending/0.0.0.0:

<?xml version="1.0" encoding="UTF-8"?>
<instance xml-version="2">
  <hypervisor type="kvm" capability="hw" bitness="64" requiresDisk="true"/>
  <backing>
    <root type="image"/>
  </backing>
  <name>i-2850d518</name>
  <uuid>2b6861f7-252c-4685-8481-cb4991d81d38</uuid>
  <reservation>r-fd0fff91</reservation>
  <user>AIDAANVOU6GS6SFPFKX5Z</user>
  <owner>AIDAANVOU6GS6SFPFKX5Z</owner>
  <account>000783862759</account>
  <imageId></imageId>
  <kernelId></kernelId>
  <ramdiskId></ramdiskId>
  <dnsName></dnsName>
  <privateDnsName></privateDnsName>
  <instancePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518</instancePath>
  <xmlFilePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/instance.xml</xmlFilePath>
  <libvirtFilePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/instance-libvirt.xml</libvirtFilePath>
  <consoleLogPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/console.log</consoleLogPath>
  <userData></userData>
  <launchIndex>0</launchIndex>
  <hasFloppy>false</hasFloppy>
  <cpuPassthrough>false</cpuPassthrough>
  <cores>1</cores>
  <memoryKB>262144</memoryKB>
  <diskGB>3</diskGB>
  <VmType>t1.micro</VmType>
  <NicType>virtio</NicType>
  <NicDevice>br0</NicDevice>
  <rootDirective></rootDirective>
  <key doInjectKey="false" sshKey="ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4dW1AXL6L7fA3HiRH8YfSfRLEFZSYfJLFdKI9zUtTPWvCiRPY2NsLOcRbVTYkMq10CRN2ALkAviEildGIO2tEyygynYVl5zq3ACi+yv9L1aDTc+4KUV1ob6DbGu6ZV02t3Pf0d/sJ8uuYsHt4gcHpm7mVlZIgSLXCBqtJyLmfxzc9ZnJHAmZITTX4cE8XzmdLO+i0Iu8JKTeNgtV1Fr4fPA5gtI4SzAmtvwQaErTJ0T7WoKj8OOu4cYjvbo7O1Qnjk63XPO9aJHfBq2AeQX6FrGXBGTxvKGpq7h6lL/XAJyn8/8YnW3hlGBfb1hkWBfZC2NYo3fQ1zCYcQBWnfgghw== [email protected]"/>
  <os platform="linux" virtioRoot="true" virtioDisk="true" virtioNetwork="true"/>
  <groupNames>
    <name>000783862759-ba32f56b-7529-4b6b-a457-e2c5e5cb448d</name>
  </groupNames>
  <disks>
    <floppyPath></floppyPath>
    <diskPath targetDeviceType="disk" targetDeviceName="vda" targetDeviceNameVirtio="vda" targetDeviceBusVirtio="virtio" targetDeviceBus="virtio" sourceType="block" serial="euca-bdm-machine-dev-vda">/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</diskPath>
    <root device="vda"/>
    <diskPath targetDeviceType="disk" targetDeviceName="vdb" targetDeviceNameVirtio="vdb" targetDeviceBusVirtio="virtio" targetDeviceBus="virtio" sourceType="block" serial="euca-bdm-ephemeral0-dev-vdb">/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vdb</diskPath>
  </disks>
  <vbrs>
    <vbr>
      <resourceLocation>none</resourceLocation>
      <guestDeviceName>vdb</guestDeviceName>
      <sizeBytes>1073741824</sizeBytes>
      <formatName>ext3</formatName>
      <id>none</id>
      <typeName>ephemeral</typeName>
      <type>ephemeral</type>
      <locationType/>
      <format>ext3</format>
      <diskNumber>1</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vdb</backingPath>
      <preparedResourceLocation></preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-ephemeral0-dev-vdb</guestDeviceSerialId>
    </vbr>
    <vbr>
      <resourceLocation>http://[email protected]:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</resourceLocation>
      <guestDeviceName>vda</guestDeviceName>
      <sizeBytes>2147483648</sizeBytes>
      <formatName>none</formatName>
      <id>emi-ef93031f</id>
      <typeName>machine</typeName>
      <type>image</type>
      <locationType>clc</locationType>
      <format>none</format>
      <diskNumber>0</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</backingPath>
      <preparedResourceLocation>http://10.111.5.215:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-machine-dev-vda</guestDeviceSerialId>
    </vbr>
  </vbrs>
  <volumes/>
  <nics>
    <nic vlan="-1" networkIndex="-1" mac="d0:0d:28:50:d5:18" publicIp="0.0.0.0" privateIp="10.111.30.228" bridgeDeviceName="br0" guestDeviceName="vn_i-2850d518" device="0" stateName="attached" hypervisorType="kvm" osPlatform="linux" osVirtioNetwork="true"/>
  </nics>
  <states>
    <retries>0</retries>
    <stateName>Pending</stateName>
    <bundleTaskStateName>none</bundleTaskStateName>
    <bundleTaskProgress>0.0000</bundleTaskProgress>
    <createImageTaskStateName></createImageTaskStateName>
    <stateCode>14</stateCode>
    <state>Staging</state>
    <bundleTaskState>none</bundleTaskState>
    <bundleBucketExists>false</bundleBucketExists>
    <bundleCanceled>false</bundleCanceled>
    <guestStateName></guestStateName>
    <isStopRequested>false</isStopRequested>
    <createImageTaskState>none</createImageTaskState>
    <createImagePid>0</createImagePid>
    <createImageCanceled>false</createImageCanceled>
    <migrationState>none</migrationState>
    <migrationSource></migrationSource>
    <migrationDestination></migrationDestination>
    <migrationCredentials></migrationCredentials>
  </states>
  <timestamps>
    <launchTime>150188328</launchTime>
    <expiryTime>150188328</expiryTime>
    <bootTime>0</bootTime>
    <bundlingTime>0</bundlingTime>
    <createImageTime>0</createImageTime>
    <terminationRequestedTime>0</terminationRequestedTime>
    <terminationTime>0</terminationTime>
    <migrationTime>0</migrationTime>
  </timestamps>
</instance>

Pending/10.111.30.100:

<instance xml-version="2">
  <hypervisor type="kvm" capability="hw" bitness="64" requiresDisk="true"/>
  <backing>
    <root type="image"/>
  </backing>
  <name>i-2850d518</name>
  <uuid>2b6861f7-252c-4685-8481-cb4991d81d38</uuid>
  <reservation>r-fd0fff91</reservation>
  <user>AIDAANVOU6GS6SFPFKX5Z</user>
  <owner>AIDAANVOU6GS6SFPFKX5Z</owner>
  <account>000783862759</account>
  <imageId></imageId>
  <kernelId></kernelId>
  <ramdiskId></ramdiskId>
  <dnsName></dnsName>
  <privateDnsName></privateDnsName>
  <instancePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518</instancePath>
  <xmlFilePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/instance.xml</xmlFilePath>
  <libvirtFilePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/instance-libvirt.xml</libvirtFilePath>
  <consoleLogPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/console.log</consoleLogPath>
  <userData></userData>
  <launchIndex>0</launchIndex>
  <hasFloppy>false</hasFloppy>
  <cpuPassthrough>false</cpuPassthrough>
  <cores>1</cores>
  <memoryKB>262144</memoryKB>
  <diskGB>3</diskGB>
  <VmType>t1.micro</VmType>
  <NicType>virtio</NicType>
  <NicDevice>br0</NicDevice>
  <rootDirective></rootDirective>
  <key doInjectKey="false" sshKey="ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4dW1AXL6L7fA3HiRH8YfSfRLEFZSYfJLFdKI9zUtTPWvCiRPY2NsLOcRbVTYkMq10CRN2ALkAviEildGIO2tEyygynYVl5zq3ACi+yv9L1aDTc+4KUV1ob6DbGu6ZV02t3Pf0d/sJ8uuYsHt4gcHpm7mVlZIgSLXCBqtJyLmfxzc9ZnJHAmZITTX4cE8XzmdLO+i0Iu8JKTeNgtV1Fr4fPA5gtI4SzAmtvwQaErTJ0T7WoKj8OOu4cYjvbo7O1Qnjk63XPO9aJHfBq2AeQX6FrGXBGTxvKGpq7h6lL/XAJyn8/8YnW3hlGBfb1hkWBfZC2NYo3fQ1zCYcQBWnfgghw== [email protected]"/>
  <os platform="linux" virtioRoot="true" virtioDisk="true" virtioNetwork="true"/>
  <groupNames>
    <name>000783862759-ba32f56b-7529-4b6b-a457-e2c5e5cb448d</name>
  </groupNames>
  <disks>
    <floppyPath></floppyPath>
    <diskPath targetDeviceType="disk" targetDeviceName="vda" targetDeviceNameVirtio="vda" targetDeviceBusVirtio="virtio" targetDeviceBus="virtio" sourceType="block" serial="euca-bdm-machine-dev-vda">/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</diskPath>
    <root device="vda"/>
    <diskPath targetDeviceType="disk" targetDeviceName="vdb" targetDeviceNameVirtio="vdb" targetDeviceBusVirtio="virtio" targetDeviceBus="virtio" sourceType="block" serial="euca-bdm-ephemeral0-dev-vdb">/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vdb</diskPath>
  </disks>
  <vbrs>
    <vbr>
      <resourceLocation>none</resourceLocation>
      <guestDeviceName>vdb</guestDeviceName>
      <sizeBytes>1073741824</sizeBytes>
      <formatName>ext3</formatName>
      <id>none</id>
      <typeName>ephemeral</typeName>
      <type>ephemeral</type>
      <locationType/>
      <format>ext3</format>
      <diskNumber>1</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vdb</backingPath>
      <preparedResourceLocation></preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-ephemeral0-dev-vdb</guestDeviceSerialId>
    </vbr>
    <vbr>
      <resourceLocation>http://[email protected]:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</resourceLocation>
      <guestDeviceName>vda</guestDeviceName>
      <sizeBytes>2147483648</sizeBytes>
      <formatName>none</formatName>
      <id>emi-ef93031f</id>
      <typeName>machine</typeName>
      <type>image</type>
      <locationType>clc</locationType>
      <format>none</format>
      <diskNumber>0</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</backingPath>
      <preparedResourceLocation>http://10.111.5.215:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-machine-dev-vda</guestDeviceSerialId>
    </vbr>
  </vbrs>
  <volumes/>
  <nics>
    <nic vlan="-1" networkIndex="-1" mac="d0:0d:28:50:d5:18" publicIp="10.111.30.100" privateIp="10.111.30.228" bridgeDeviceName="br0" guestDeviceName="vn_i-2850d518" device="0" stateName="attached" hypervisorType="kvm" osPlatform="linux" osVirtioNetwork="true"/>
  </nics>
  <states>
    <retries>0</retries>
    <stateName>Pending</stateName>
    <bundleTaskStateName>none</bundleTaskStateName>
    <bundleTaskProgress>0.0000</bundleTaskProgress>
    <createImageTaskStateName></createImageTaskStateName>
    <stateCode>14</stateCode>
    <state>Staging</state>
    <bundleTaskState>none</bundleTaskState>
    <bundleBucketExists>false</bundleBucketExists>
    <bundleCanceled>false</bundleCanceled>
    <guestStateName></guestStateName>
    <isStopRequested>false</isStopRequested>
    <createImageTaskState>none</createImageTaskState>
    <createImagePid>0</createImagePid>
    <createImageCanceled>false</createImageCanceled>
    <migrationState>none</migrationState>
    <migrationSource></migrationSource>
    <migrationDestination></migrationDestination>
    <migrationCredentials></migrationCredentials>
  </states>
  <timestamps>
    <launchTime>150188328</launchTime>
    <expiryTime>150188328</expiryTime>
    <bootTime>0</bootTime>
    <bundlingTime>0</bundlingTime>
    <createImageTime>0</createImageTime>
    <terminationRequestedTime>0</terminationRequestedTime>
    <terminationTime>0</terminationTime>
    <migrationTime>0</migrationTime>
  </timestamps>
</instance>

Extant/Booting (invalid):

<?xml version="1.0" encoding="UTF-8"?>
<instance xml-version="2">
  <hypervisor type="kvm" capability="hw" bitness="64" requiresDisk="true"/>
  <backing>
    <root type="image"/>
  </backing>
  <name>i-2850d518</name>
  <uuid>2b6861f7-252c-4685-8481-cb4991d81d38</uuid>
  <reservation>r-fd0fff91</reservation>
  <user>AIDAANVOU6GS6SFPFKX5Z</user>
  <owner>AIDAANVOU6GS6SFPFKX5Z</owner>
  <account>000783862759</account>
  <imageId></imageId>
  <kernelId></kernelId>
  <ramdiskId></ramdiskId>
  <dnsName></dnsName>
  <privateDnsName></privateDnsName>
  <instancePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518</instancePath>
  <xmlFilePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/instance.xml</xmlFilePath>
  <libvirtFilePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/instance-libvirt.xml</libvirtFilePath>
  <consoleLogPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/console.log</consoleLogPath>
  <userData></userData>
  <launchIndex>0</launchIndex>
  <hasFloppy>false</hasFloppy>
  <cpuPassthrough>false</cpuPassthrough>
  <cores>1</cores>
  <memoryKB>262144</memoryKB>
  <diskGB>3</diskGB>
  <VmType>t1.micro</VmType>
  <NicType>virtio</NicType>
  <NicDevice>br0</NicDevice>
  <rootDirective></rootDirective>
  <key doInjectKey="false" sshKey="ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4dW1AXL6L7fA3HiRH8YfSfRLEFZSYfJLFdKI9zUtTPWvCiRPY2NsLOcRbVTYkMq10CRN2ALkAviEildGIO2tEyygynYVl5zq3ACi+yv9L1aDTc+4KUV1ob6DbGu6ZV02t3Pf0d/sJ8uuYsHt4gcHpm7mVlZIgSLXCBqtJyLmfxzc9ZnJHAmZITTX4cE8XzmdLO+i0Iu8JKTeNgtV1Fr4fPA5gtI4SzAmtvwQaErTJ0T7WoKj8OOu4cYjvbo7O1Qnjk63XPO9aJHfBq2AeQX6FrGXBGTxvKGpq7h6lL/XAJyn8/8YnW3hlGBfb1hkWBfZC2NYo3fQ1zCYcQBWnfgghw== [email protected]"/>
  <os platform="linux" virtioRoot="true" virtioDisk="true" virtioNetwork="true"/>
  <groupNames>
    <name>000783862759-ba32f56b-7529-4b6b-a457-e2c5e5cb448d</name>
  </groupNames>
  <disks>
    <floppyPath></floppyPath>
    <diskPath targetDeviceType="disk" targetDeviceName="vda" targetDeviceNameVirtio="vda" targetDeviceBusVirtio="virtio" targetDeviceBus="virtio" sourceType="block" serial="euca-bdm-machine-dev-vda">/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</diskPath>
    <root device="vda"/>
    <diskPath targetDeviceType="disk" targetDeviceName="vdb" targetDeviceNameVirtio="vdb" targetDeviceBusVirtio="virtio" targetDeviceBus="virtio" sourceType="block" serial="euca-bdm-ephemeral0-dev-vdb">/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vdb</diskPath>
  </disks>
  <vbrs>
    <vbr>
      <resourceLocation>none</resourceLocation>
      <guestDeviceName>vdb</guestDeviceName>
      <sizeBytes>1073741824</sizeBytes>
      <formatName>ext3</formatName>
      <id>none</id>
      <typeName>ephemeral</typeName>
      <type>ephemeral</type>
      <locationType/>
      <format>ext3</format>
      <diskNumber>1</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vdb</backingPath>
      <preparedResourceLocation></preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-ephemeral0-dev-vdb</guestDeviceSerialId>
    </vbr>
    <vbr>
      <resourceLocation>http://[email protected]:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</resourceLocation>
      <guestDeviceName>vda</guestDeviceName>
      <sizeBytes>2147483648</sizeBytes>
      <formatName>none</formatName>
      <id>emi-ef93031f</id>
      <typeName>machine</typeName>
      <type>image</type>
      <locationType>clc</locationType>
      <format>none</format>
      <diskNumber>0</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</backingPath>
      <preparedResourceLocation>http://10.111.5.215:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-machine-dev-vda</guestDeviceSerialId>
    </vbr>
  </vbrs>
  <volumes/>
  <nics>
    <nic vlan="-1" networkIndex="-1" mac="d0:0d:28:50:d5:18" publicIp="10.111.30.100" privateIp="10.111.30.228" bridgeDeviceName="br0" guestDeviceName="vn_i-2850d518" device="0" stateName="attached" hypervisorType="kvm" osPlatform="linux" osVirtioNetwork="true"/>
  </nics>
  <states>
    <retries>5</retries>
    <stateName>Extant</stateName>
    <bundleTaskStateName>none</bundleTaskStateName>
    <bundleTaskProgress>0.0000</bundleTaskProgress>
    <createImageTaskStateName></createImageTaskStateName>
    <stateCode>15</stateCode>
    <state>Booting</state>
    <bundleTaskState>none</bundleTaskState>
    <bundleBucketExists>false</bundleBucketExists>
    <bundleCanceled>false</bundleCanceled>
    <guestStateName></guestStateName>
    <isStopRequested>false</isStopRequested>
    <createImageTaskState>none</createImageTaskState>
    <createImagePid>0</createImagePid>
    <createImageCanceled>false</createImageCanceled>
    <migrationState>none</migrationState>
    <migrationSource></migrationSource>
    <migrationDestination></migrationDestination>
    <migrationCredentials></migrationCredentials>
  </states>
  <timestamps>
    <launchTime>150188328</launchTime>
    <expiryTime>150188328</expiryTime>
    <bootTime>150188330</bootTime>
    <bundlingTime>0</bundlingTime>
    <createImageTime>0</createImageTime>
    <terminationRequestedTime>0</terminationRequestedTime>
    <terminationTime>0</terminationTime>
    <migrationTime>0</migrationTime>
  </timestamps>
</instance>
1f</id>
      <typeName>machine</typeName>
      <type>image</type>
      <locationType>clc</locationType>
      <format>none</format>
      <diskNumber>0</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</backingPath>
      <preparedResourceLocation>http://10.111.5.215:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-machine-dev-vda</guestDeviceSerialId>
    </vbr>
  </vbrs>
  <volumes/>
  <nics>
    <nic vlan="-1" networkIndex="-1" mac="d0:0d:28:50:d5:18" publicIp="10.111.30.100" privateIp="10.111.30.228" bridgeDeviceName="br0" guestDeviceName="vn_i-2850d518" device="0" stateName="attached" hypervisorType="kvm" osPlatform="linux" osVirtioNetwork="true"/>
  </nics>
  <states>
    <retries>5</retries>
    <stateName>Extant</stateName>
    <bundleTaskStateName>none</bundleTaskStateName>
    <bundleTaskProgress>0.0000</bundleTaskProgress>
    <createImageTaskStateName></createImageTaskStateName>
    <stateCode>15</stateCode>
    <state>Booting</state>
    <bundleTaskState>none</bundleTaskState>
    <bundleBucketExists>false</bundleBucketExists>
    <bundleCanceled>false</bundleCanceled>
    <guestStateName></guestStateName>
    <isStopRequested>false</isStopRequested>
    <createImageTaskState>none</createImageTaskState>
    <createImagePid>0</createImagePid>
    <createImageCanceled>false</createImageCanceled>
    <migrationState>none</migrationState>
    <migrationSource></migrationSource>
    <migrationDestination></migrationDestination>
    <migrationCredentials></migrationCredentials>
  </states>
  <timestamps>
    <launchTime>150188328</launchTime>
    <expiryTime>150188328</expiryTime>
    <bootTime>150188330</bootTime>
    <bundlingTime>0</bundlingTime>
    <createImageTime>0</createImageTime>
    <terminationRequestedTime>0</terminationRequestedTime>
    <terminationTime>0</terminationTime>
    <migrationTime>0</migrationTime>
  </timestamps>
</instance>

Extant/Running:

<?xml version="1.0" encoding="UTF-8"?>
<instance xml-version="2">
  <hypervisor type="kvm" capability="hw" bitness="64" requiresDisk="true"/>
  <backing>
    <root type="image"/>
  </backing>
  <name>i-2850d518</name>
  <uuid>2b6861f7-252c-4685-8481-cb4991d81d38</uuid>
  <reservation>r-fd0fff91</reservation>
  <user>AIDAANVOU6GS6SFPFKX5Z</user>
  <owner>AIDAANVOU6GS6SFPFKX5Z</owner>
  <account>000783862759</account>
  <imageId></imageId>
  <kernelId></kernelId>
  <ramdiskId></ramdiskId>
  <dnsName></dnsName>
  <privateDnsName></privateDnsName>
  <instancePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518</instancePath>
  <xmlFilePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/instance.xml</xmlFilePath>
  <libvirtFilePath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/instance-libvirt.xml</libvirtFilePath>
  <consoleLogPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/console.log</consoleLogPath>
  <userData></userData>
  <launchIndex>0</launchIndex>
  <hasFloppy>false</hasFloppy>
  <cpuPassthrough>false</cpuPassthrough>
  <cores>1</cores>
  <memoryKB>262144</memoryKB>
  <diskGB>3</diskGB>
  <VmType>t1.micro</VmType>
  <NicType>virtio</NicType>
  <NicDevice>br0</NicDevice>
  <rootDirective></rootDirective>
  <key doInjectKey="false" sshKey="ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4dW1AXL6L7fA3HiRH8YfSfRLEFZSYfJLFdKI9zUtTPWvCiRPY2NsLOcRbVTYkMq10CRN2ALkAviEildGIO2tEyygynYVl5zq3ACi+yv9L1aDTc+4KUV1ob6DbGu6ZV02t3Pf0d/sJ8uuYsHt4gcHpm7mVlZIgSLXCBqtJyLmfxzc9ZnJHAmZITTX4cE8XzmdLO+i0Iu8JKTeNgtV1Fr4fPA5gtI4SzAmtvwQaErTJ0T7WoKj8OOu4cYjvbo7O1Qnjk63XPO9aJHfBq2AeQX6FrGXBGTxvKGpq7h6lL/XAJyn8/8YnW3hlGBfb1hkWBfZC2NYo3fQ1zCYcQBWnfgghw== [email protected]"/>
  <os platform="linux" virtioRoot="true" virtioDisk="true" virtioNetwork="true"/>
  <groupNames>
    <name>000783862759-ba32f56b-7529-4b6b-a457-e2c5e5cb448d</name>
  </groupNames>
  <disks>
    <floppyPath></floppyPath>
    <diskPath targetDeviceType="disk" targetDeviceName="vda" targetDeviceNameVirtio="vda" targetDeviceBusVirtio="virtio" targetDeviceBus="virtio" sourceType="block" serial="euca-bdm-machine-dev-vda">/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</diskPath>
    <root device="vda"/>
    <diskPath targetDeviceType="disk" targetDeviceName="vdb" targetDeviceNameVirtio="vdb" targetDeviceBusVirtio="virtio" targetDeviceBus="virtio" sourceType="block" serial="euca-bdm-ephemeral0-dev-vdb">/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vdb</diskPath>
  </disks>
  <vbrs>
    <vbr>
      <resourceLocation>none</resourceLocation>
      <guestDeviceName>vdb</guestDeviceName>
      <sizeBytes>1073741824</sizeBytes>
      <formatName>ext3</formatName>
      <id>none</id>
      <typeName>ephemeral</typeName>
      <type>ephemeral</type>
      <locationType/>
      <format>ext3</format>
      <diskNumber>1</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vdb</backingPath>
      <preparedResourceLocation></preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-ephemeral0-dev-vdb</guestDeviceSerialId>
    </vbr>
    <vbr>
      <resourceLocation>http://[email protected]:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</resourceLocation>
      <guestDeviceName>vda</guestDeviceName>
      <sizeBytes>2147483648</sizeBytes>
      <formatName>none</formatName>
      <id>emi-ef93031f</id>
      <typeName>machine</typeName>
      <type>image</type>
      <locationType>clc</locationType>
      <format>none</format>
      <diskNumber>0</diskNumber>
      <partitionNumber>0</partitionNumber>
      <guestDeviceType>disk</guestDeviceType>
      <guestDeviceBus>virtio</guestDeviceBus>
      <backingType>block</backingType>
      <backingPath>/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-2850d518/link-to-vda</backingPath>
      <preparedResourceLocation>http://10.111.5.215:8773/services/objectstorage/7a55e1-download-manifests-v2/DM-emi-ef93031f-one?Signature=RREesdV8XuzLmlkA5OpmzWcDF68%3D&amp;AWSAccessKeyId=AKIAABMKXXCWULSMMJ2Z&amp;Expires=1501894086&amp;x-amz-security-token=ZXVjYQABFg7Rg%2Fw7eZN4koz4dNEymsFZtyGZheake827EurJ1AGd6Zlg2LVdu9UlH7v4Ci%2FbCCDmYllGE5OumcVBcuPSDA7bBpPM6O%2BqOLSs0MfPog8hdFa6X3xZHV70EzP9TFzLapGrwDs%2BrmCIq%2F9MPoA0yFsId1FMbHYAS%2BbaFeI%2BCY8f9ySefJ1qzJhVTfWpLMed1die8xOJ%2FWTq03S5PgHqilF289qRkOXLaPAY9mD3Vfp6855ASETFKndT</preparedResourceLocation>
      <guestDeviceSerialId>euca-bdm-machine-dev-vda</guestDeviceSerialId>
    </vbr>
  </vbrs>
  <volumes/>
  <nics>
    <nic vlan="-1" networkIndex="-1" mac="d0:0d:28:50:d5:18" publicIp="10.111.30.100" privateIp="10.111.30.228" bridgeDeviceName="br0" guestDeviceName="vn_i-2850d518" device="0" stateName="attached" hypervisorType="kvm" osPlatform="linux" osVirtioNetwork="true"/>
  </nics>
  <states>
    <retries>5</retries>
    <stateName>Extant</stateName>
    <bundleTaskStateName>none</bundleTaskStateName>
    <bundleTaskProgress>0.0000</bundleTaskProgress>
    <createImageTaskStateName></createImageTaskStateName>
    <stateCode>15</stateCode>
    <state>Running</state>
    <bundleTaskState>none</bundleTaskState>
    <bundleBucketExists>false</bundleBucketExists>
    <bundleCanceled>false</bundleCanceled>
    <guestStateName>poweredOn</guestStateName>
    <isStopRequested>false</isStopRequested>
    <createImageTaskState>none</createImageTaskState>
    <createImagePid>0</createImagePid>
    <createImageCanceled>false</createImageCanceled>
    <migrationState>none</migrationState>
    <migrationSource></migrationSource>
    <migrationDestination></migrationDestination>
    <migrationCredentials></migrationCredentials>
  </states>
  <timestamps>
    <launchTime>150188328</launchTime>
    <expiryTime>150188328</expiryTime>
    <bootTime>150188330</bootTime>
    <bundlingTime>0</bundlingTime>
    <createImageTime>0</createImageTime>
    <terminationRequestedTime>0</terminationRequestedTime>
    <terminationTime>0</terminationTime>
    <migrationTime>0</migrationTime>
  </timestamps>
</instance>

The invalid file contains:

    <migrationTime>0</migrationTime>
  </timestamps>
</instance>
1f</id>
      <typeName>machine</typeName>
      <type>image</type>

which can lead to errors in the nc.log such as:

2017-08-01 22:02:00 ERROR | failed to parse XML document /var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-8334c266/instance.xml

and in the httpd-node_error_log:

/var/lib/eucalyptus/instances/work/AIDAANVOU6GS6SFPFKX5Z/i-8334c266/instance.xml:120: parser error : Extra content at the end of the document
i-ef93031f</id>
^

These example errors are not from the same occurrence of the the issue as the example instance.xml files.

To reproduce the issue you can churn 8 instances on a node controller and capture all instance.xml files using (for example):

# mkdir -p /tmp/instancexmls; inotifywait -m -r -e close_write /var/lib/eucalyptus/instances/work/ | while read path action file; do if [ "$file" == "instance.xml" ]; then MD5SUM=$(md5sum $path$file | awk '{print $1}'); INSTANCE=$(echo $path | awk -F/ '{print $(NF-1)}'); if [ ! -f "/tmp/instancexmls/$INSTANCE-$MD5SUM-instance.xml" ]; then cp -v $path$file /tmp/instancexmls/$INSTANCE-$MD5SUM-instance.xml; fi; fi; done

which will create a copy for each unique instance xml that is output. Valid instance xml files are typically 6xxx bytes, invalid files 9xxx bytes.

The invalid instance xml file in a churn test occur on launch or termination of an instance when the file can be updated by separate threads concurrently.

This issue does not always cause errors, the invalid files may be overwritten before they are read.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
from jira
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sjones4