[wip] 3886 – Clean up archiver tests (#411)

We had a lot of tests that were making real requests that weren't relevant to this test. So we decided to do a refactoring of the tests. What we did:
	1. mock all requests
	2. add new tests when needed
	3. use one setup and teardown for all tests
	4. skip tests related to video-archiving (we don't use this feature anymore, it will be dealt with in another ticket)
	5. general clean up

While working on the tests there were a few issues that appeared, I ended up dealing with them here (they should have been separate tickets/PRs):
	1. skip_perma_cc_archiver should return true if key.nil? (commit)
	2. mock perma_cc failed test and fix perma_cc error raising (commit)
	3. deal with sidekiq warning (commit)
	4. update how we update cache (commit)

app/models/concerns/media_archive_org_archiver.rb

@@ -6,7 +6,7 @@ module MediaArchiveOrgArchiver
   end
 
   def archive_to_archive_org(url, key_id)
-    ArchiverWorker.perform_in(30.seconds, url, :archive_org, key_id)
+    ArchiverWorker.perform_in(30.seconds, url, 'archive_org', key_id)
   end
 
   module ClassMethods

app/models/concerns/media_archiver.rb

@@ -68,7 +68,11 @@ def give_up(info = {})
 
     def notify_webhook_and_update_cache(archiver, url, data, key_id)
       settings = Media.api_key_settings(key_id)
-      Media.update_cache(url, { archives: { archiver => data } })
+
+      id = Media.get_id(url)
+      archiver_data = Pender::Store.current.read(id, :json).to_h.dig('archives', archiver).to_h
+      archiver_data.delete('error')
+      Media.update_cache(url, { archives: { archiver => archiver_data.merge(data) } })
       Media.notify_webhook(archiver, url, data, settings)
     end
 

app/models/concerns/media_perma_cc_archiver.rb

@@ -6,7 +6,7 @@ module MediaPermaCcArchiver
   end
 
   def archive_to_perma_cc(url, key_id)
-    ArchiverWorker.perform_in(30.seconds, url, :perma_cc, key_id)
+    ArchiverWorker.perform_in(30.seconds, url, 'perma_cc', key_id)
   end
 
   module ClassMethods
@@ -30,7 +30,7 @@ def send_to_perma_cc(url, key_id, _supported = nil)
           data = { location: 'http://perma.cc/' + body['guid'] }
           Media.notify_webhook_and_update_cache('perma_cc', url, data, key_id)
         else
-          data = { error: { message: response.message, code: Lapis::ErrorCodes::const_get('ARCHIVER_ERROR') }}
+          data = { error: { message: "(#{response.code}) #{response.message}", code: Lapis::ErrorCodes::const_get('ARCHIVER_ERROR') }}
           Media.notify_webhook_and_update_cache('perma_cc', url, data, key_id)
           if response&.body.include?("You've reached your usage limit")
             PenderSentry.notify(
@@ -49,6 +49,7 @@ def skip_perma_cc_archiver(perma_cc_key, url, key_id)
       if perma_cc_key.nil?
         data = { error: { message: 'Missing authentication key', code: Lapis::ErrorCodes::const_get('ARCHIVER_MISSING_KEY') }}
         Media.notify_webhook_and_update_cache('perma_cc', url, data, key_id)
+        return true
       else
         return false
       end