Soft-block-bloom-filter-filter

[KevinMind]

Fixes: mozilla/addons#15014
Rleates to: mozilla/addons#15155

Description

Adds logic to generate and write bloom filters for both soft and hard blocked addons. Additionally this PR introduces logic to determine whether we should update one or both bloom filters and or a stash as multiple possible outcomes are possible now. Finally, we handle cleaning up files on a more granular level from both the local storage and remote settings.

Context

Now when we run the upload_mlbf_to_remote_settings cron job we will check for both hard and soft blocked items. It is possible to:

• do nothing
• upload a hard block filter only
• upload a hard block filter and a stash (for soft blocks)
• upload a soft block filter
• upload a soft block filter and a stash (for hard blocks)
• upload both filters

This adds a bit of complexity we need to address.

Additionally, instead of deleting all records from remote settings, we need to check for the current set of block filters and only delete records older than the older of the two.

Finally, since it is also possible to run the cron when no updates have occurred, we can safely delete mlbf cache files when that happens as there is no benefit from diffing an empty array.

Testing

This is gonna suck to test. First some preparation work.

IMPORTANT Unless soft block stashing is landed, soft blocks will be ignored.. they will not produce a stash or a filter

Setup

• Setup a local remote server here
• Set the base replace threshold to a low number (so you can trigger re-uploading of filters without creating a bunch of blocks)
• enable mlbf-soft-blocks-enabled waffle switch

src/olympia/constants/blocklist.py

BASE_REPLACE_THRESHOLD = 1

upload_mlbf_to_remote_settings

Execute this command to create a new filter/stash iteration

./manage.py cron upload_mlbf_to_remote_settings

Test Scenarios (admin)

If you've set the base_replace_threshold to 1 then testing this from the admin is the way to go. You can test all the different combinations of state changes to trigger nothing or stash and or filter.

1. Navigate to the blocklist submission page here
2. add a guid for an addon that has a lot of versions (or multiple guids to end up with a lot of versions)
3. Click submit and set various combos of the versions listed to hard/soft blocked.
4. After creating the blocks execute the in the section above from a container shell

make shell

See manage.py command above to execute

5. Expectations depend on what block versions you create for a given type

• 0 for a block type results in no stash and no filter
• 1 for a block type results in a stash with that guid:version
• 2 for a block type results in a new filter for that block type

6. If both types have 0, expect the directory is deleted after executing.. empty state
7. if both types have 1 expect a stash with both types included
8. if one type has 1 and another has 2 expect a stash for the former and a filter for the latter

Test Scenarios (django)

You can test the above scenarios from the django shell as well. In a make djshell add this code

from olympia.blocklist.models import BlockType
from olympia.amo.tests import addon_factory, block_factory, version_factory

def _blocked_addon(block_type=BlockType.BLOCKED, **kwargs):
    addon = addon_factory(**kwargs)
    block = block_factory(
        guid=addon.guid, updated_by=user, block_type=block_type
    )
    return addon, block

user = UserProfile.objects.first()

Now you can call the _blocked_addon method to create an addon with block/version of the specified type.

Ex:

_blocked_addon(block_type=BlockType.BLOCKED)
_blocked_addon(block_type=BlockType.BLOCKED)
_blocked_addon(block_type=BlockType.SOFT_BLOCKED)

If you run the cron job now, you'd expect a blocked filter and a stash with the soft blocked version added.

Verify empty stashes are not created

Using the shell approach from above, verify that if the only change is an unsigned version is blocked, that no stash is produced.

_blocked_addon(block_type=BlockType.BLOCKED, file_kw={'is_signed': False})

Expect: running the cron job produces no stash and is interpreted as a noop event.

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.