MANUAL file expiry deletion script

[cehune]

base functions for file expiry tool + the necessary python script to run it

• expiry check based on the time since last modification
• sorting through an entire directory and deleting files which have expired

Motivation

https://github.com/WATonomous/infra-config/issues/1143

Description

As to avoid race conditions, this works by copying files into a separate directory with 700 root conditions and then deleting this separate directory. The code currently handles the following edge cases:

• File naming conflicts when moved to the temp directory
• Folder by temp name already exists
• Deletes empty folders if all files inside were expired

This has been tested as follows:

• Unit tests for file expiry and file creation (need to produce for temp folder generation and file deletion)
• Manual Testing on local machine (only 10 ish files)

Unsure about how it will work when handling the entire storage folder.

Next steps

• cron automatically run script?
• better notification system (currently just prints the user whos file is deleted)
• Better decision making for what files should be deleted (ie technically expired, but a necessary part of an active folder)

[Jimmyj30]

weird thing I found that might be useful for this project? find . -ctime -30 seems to find the files in a directory that haven't been modified for 30 days, so if the output of this is empty for a certain directory, that means we can maybe delete that directory as nothing in it has been touched for 30+ days. But I do think we should only delete directories at the very "top" level of the wato-drive so we are not deleting any inactive files that are part of an ongoing project.

[ben-z]

Looking good! I think we can test this now. Feel free to change your role to ADMIN in your profile so that you have sudo access. Here's the profile editor: https://cloud.watonomous.ca/docs/utilities/profile-editor

Let's start with a machine that's not super heavily used (wato2-ubuntu1, tr-ubuntu1, delta-ubuntu2 are good choices) and a folder on that machine that is smaller like /var/lib/cluster/<uid>.

[ben-z]

Thanks for waiting hehe. Gave this a closer look. We can start testing!

[ben-z]

Looks good overall! Do we need the empty init.py files? Can we add instructions to README on how to run this tool?

[ben-z]

The structure looks good! There are some details that we should iron out, like using consistent timestamps and being efficient with filesystem accesses.

[ben-z]

Looking pretty good! Can we also run the tests in CI?

[Jimmyj30]

Looking pretty good! Can we also run the tests in CI?

Just another small comment: did we figure out how to deal with the "undefined" file info, like ctimes and stuff?

[ben-z]

Looks pretty good in general! Pretty much all just nitpicks.

I think you are looking into issues when running this on a larger folder right? Please ping me again once that's ready.

[ben-z]

LGTM! Just some nits

[ben-z]

LGTM!