Imager with S3 - local cache clearing unexpectedly

[alexhillel]

I'm using Imager and integrating with an S3 bucket. Everything works really well when the local cache folder at public/imager is populated, but the files in there are being removed unexpectedly causing fresh transforms to be generated and uploaded, thus slowing page loads down significantly. I can't work out what is causing the cache removals and even if it's intended behaviour or not.

Files in the local cache are being removed within 24 hours of being generated, and I can't discern a pattern as to which files are being removed and why, or locate the cause in log files. I haven't modified cache times so I'd expect files to remain in place for two weeks as in the default config. My overridden config settings in craft/config is as follows:

$productionSettings = array(
  'imagerUrl' => 'http://s3-eu-west-1.amazonaws.com/my-bucket/transforms/',
  'awsEnabled' => true,
  'awsAccessKey' => 'AAAAAAAAAAAAAAAAAAAAAA',
  'awsSecretAccessKey' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
  'awsBucket' => 'my-bucket',
  'awsFolder' => 'transforms'
);

return array(
  '*' => array(
    'jpegoptimEnabled' => true,
    'optipngEnabled' => true,
    'format' => 'jpg',
    'allowUpscale' => false,
    'mode' => 'crop',
    'jpegQuality' => 80,
    'interlace' => true
  ),
  'mystagedomain.com' => $productionSettings,
  'mydomain.com' => $productionSettings
);

[aelvan]

It's definitely not intended behaviour. The only way for cached transforms to disappear would be if the cache is cleared manually, either via the empty cache functionality in the craft settings (there're sources for Imager image transform cache and Imager remote images cache), or via the element action in assets. Or, if something in your setup is deleting the actual files in the folders.

What kind of a server setup are you running this on?

[alexhillel]

It's a standard Laravel Forge server deployed onto a Linode VPS. It's relatively freshly provisioned and there's nothing strange installed on there from what I can recall, and it's the only site installed on that server. I'm the only one with access to the Settings tab in Craft so it won't be anybody manually regenerating caches.

There's quite a few plugins installed on the site, and the client is currently populating content so I'm noticing a lot of Imager_Jpegoptim tasks running in craft.log.

It's really baffling me. I'm currently crawling the site with a script to regenerate transforms periodically but that's not a long term solution!

[aelvan]

Ok, apart from us using DigitalOcean, we're running that setup on a ton of sites with Imager, so that should be fine.

Have you looked at the timestamps of the transformed files after they have been run through jpegoptim, could it be that it somehow changes the dates? Have you tried disabling jpegoptim?

And how did you confirm that transforms are disappearing when they shouldn't btw? It's not the client moving files around, publishing new ones, or something?

If it is something with the timestamps on the files, you could also try setting the cacheDuration and cacheDurationRemoteFiles to false in your config file. That should cache them indefinitely.

[alexhillel]

I've turned off all optimisations in the config (including jpegoptim), and also set cacheDuration and cacheDurationRemoteFiles to false. I also set the cache directory to a differently named folder (public/transforms). Unfortunately, the issue is still occurring.

My assets are divided into subdirectories, generally categorised by sections. The files in specific subdirectories are all disappearing at once (which is why I'm sure something is cleaning them out as opposed to a client moving files around as there were hundreds of files in there) - I'm keeping on eye a particular subdirectory of the cache, and everything that was previously populated in the morning is gone, and any files now in there originate from 15:30 or later. However, other subdirectories still have files from the morning in there.

The fact that I've moved the cache directory to a user-specified one, which only Imager should know about suggests that it's something triggered by Imager that's responsible, but you'll know better than me if that's not the case!

[rachelrine]

I believe I'm having this issue as well. I noticed it because my queue is full of failed jpegoptim tasks, the logs tell me my local transformed images don't exist. I haven't confirmed files are disappearing after 24 hours, but they do seem to disappear often (and I'm also the only user with cache clearing privileges).

I have an almost identical setup: Forge, Linode, AWS (I'm also using Cloudfront).