Fix inconsistencies that result in Usage generating usage records when it should not

[winterhazel]

Description

This PR fixes some inconsistencies which results in Usage generating usage records when it should not, namely:

• Duplicated entries in the networks helper table, resulting in duplicated usage record generation;
• Port forwarding helper entries are never removed, resulting in removed port forwarding rules generating usage records;
• Load balancer helper entries are never removed, resulting in removed load balancer rules generating usage records;
• VM snapshot helper entries are never removed, resulting in removed VM snapshots generating usage records;
• Helper entries for volumes in the secondary storage are never removed after the volume gets attached to a VM, resulting in secondary storage usage records being generated for volumes that are not in the secondary storage anymore.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

Duplicated usage record generation for networks

1. I created a network
2. I added a VM to the network
3. I stopped the VM
4. I waited until the network reverted to Allocated
5. I started the VM
6. I ran the Usage job

Before the changes, after the Usage job finishes, there would be two non-removed helper entries for the same network, resulting in duplicated usage records being generated for the network. After the changes, the first helper entry was marked as removed.

Removed port forwarding rules generating usage records

1. I created a port forwarding rule
2. I ran the Usage job in order to create a helper entry for the PF rule
3. I verified, in cloud_usage.usage_port_forwarding, that the helper entry had an ID different from the PF rule's ID
4. I deleted the PF rule
5. I ran the Usage job

Before the changes, the helper entry would not get marked as removed in step 5, resulting in usage records still being generated for the removed PF rule. After the changes, the helper entry got marked as removed.

Removed load balancer policies generating usage records

1. I created a load balancer policy
2. I ran the Usage job in order to create a helper entry for the LB policy
3. I verified, in cloud_usage.usage_load_balancer_policy, that the helper entry had an ID different from the LB policy's ID
4. I deleted the LB policy
5. I ran the Usage job

Before the changes, the helper entry would not get marked as removed in step 5, resulting in usage records still being generated for the removed LB policy. After the changes, the helper entry got marked as removed.

Removed VM snapshots generating usage records

1. I took two VM snapshots of the same VM
2. I ran the Usage job in order to create helper entries for the two VM snapshots
3. I verified, in cloud_usage.usage_snapshot_on_primary, that the helper entry had IDs different from the VM's ID
4. I removed the two VM snapshots
5. I ran the Usage job

Before the changes, the helper entries would not get marked as removed in step 5, resulting in usage records still being generated for the removed VM snapshots. After the changes, the helper entries got marked as removed.

Volumes in the secondary storage that were attached still generate usage records

1. I uploaded a volume
2. I ran the Usage job in order to create a secondary storage helper entry for the volume
3. I verified, in cloud_usage.usage_storage, that the helper entry had had an ID different from the volume's ID
4. I attached the volume to a VM
5. I ran the Usage job

Before the changes, a helper entry would be created to generate secondary storage usage records for the volume (in cloud_usage.usage_storage). After attaching, another entry would be created to generate primary storage volume usage records; however, the secondary storage helper entry did not get removed, resulting in secondary storage usage records being generated for a volume that was not in the secondary storage anymore.

After the changes, the secondary storage helper entry was marked as removed.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 21 lines in your changes missing coverage. Please review.

Project coverage is 15.81%. Comparing base (46201ee) to head (a5be3ef).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...rc/main/java/com/cloud/usage/UsageManagerImpl.java	0.00%	13 Missing ⚠️
...java/com/cloud/usage/dao/UsageNetworksDaoImpl.java	0.00%	4 Missing ⚠️
...main/java/com/cloud/usage/dao/UsageVpcDaoImpl.java	0.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9888      +/-   ##
============================================
+ Coverage     15.78%   15.81%   +0.02%     
- Complexity    12565    12581      +16     
============================================
  Files          5627     5627              
  Lines        492260   492267       +7     
  Branches      63882    63886       +4     
============================================
+ Hits          77710    77832     +122     
+ Misses       406076   405912     -164     
- Partials       8474     8523      +49     

Flag	Coverage Δ
uitests	4.04% <ø> (+<0.01%)	⬆️
unittests	16.63% <0.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[DaanHoogland]

@blueorangutan package