Images not uploading to AWS

[Tam]

Imager is working on my staging environment, which is on the same server as the live site. The configs are identical, but the images aren't being uploaded to S3 on the live site.

The locally stored images are being transformed and saved correctly, but the S3 images aren't.

There isn't an imager.log, and there are no errors in the craft.log. I've checked the permissions and they're all correct (staging writes to the same location, and that works).

For the time being I've disabled AWS & set my imagerUrl back to default, and everythings working as expected. Is there anything I can do to get this working on S3, or at least better debug what's wrong?

[sergeifilippov]

Same here.

[andrewmenich]

I'm having the same issue.

I turned on logging for my aws and here's the error I got: (replacing my actual bucket name with "bucketname")

REST.GET.ENCRYPTION - "GET /bucketname?encryption= HTTP/1.1" 404 ServerSideEncryptionConfigurationNotFoundError 353 - 15 - "-" "S3Console/0.4, aws-internal/3" -

[smockensturm]

Same.

I don't know about that AWS log file, though.

Inspecting the broken image links in the browser show a legitimate AWS path, but nothing is getting uploaded. I don't think Imager is even trying.

Same configuration settings as @andrewmenich here.

Craft CMS 3.0.9
Imager v2.0.1.1
PHP Version 7.0.28-1+ubuntu16.04.1+deb.sury.org+1

[aelvan]

Are you guys using bucket encryption?

https://www.cloudconformity.com/conformity-rules/S3/bucket-default-encryption.html

[smockensturm]

No. We are not using bucket encryption.

[aelvan]

@smockensturm Did you at some point change your configuration from having transforms locally to enabling AWS? If so, make sure that you clear the local imager transform cache, if the image already exists there, Imager will assume that it has already been uploaded to AWS.

What regions are you guys creating your buckets in?

[smockensturm]

If we do, we empty the cache and local imager directory. As instructed.

The region is us-east-2

[aelvan]

Are you using the same bucket in Craft? You're sure that the access is set up correctly?

Also, have you tried enabling the debug toolbar, any errors there? Imager will throw one error that says "Invalid configuration of S3 Client: (error message)" if the S3 client can't initialize, and "An error occured while uploading to Amazon S3: (error message)" if the upload failed for some reason.

[smockensturm]

Yes. Yes.

Here is the error in the log:

An error occured while uploading to Amazon S3: Encountered a permanent redirect while requesting https://mybucketname.s3.us-east-2.amazonaws.com/transforms/images/galleries/155/filename.jpg. Are you sure you are using the correct region for this bucket?

[aelvan]

Can you post your aws config settings (remove the access key and secret!)?

[smockensturm]

return array(
   'storageConfig' => [
		'aws' => [
			'accessKey' => 'XXXXX',
			'secretAccessKey' => 'XXXXXXXX',
			'region' => 'us-east-2',
			'bucket' => 'mybucketname',
			'folder' => 'transforms',
			'requestHeaders' => array(),
			'storageType' => 'standard',
			'cloudfrontInvalidateEnabled' => false,
			'cloudfrontDistributionId' => '',
		],
	],
   'storages' => ['aws'],
   'imagerUrl' => 'https://s3.amazonaws.com/mybucketname/transforms/',
);

[aelvan]

That looks correct to me, not sure what could be wrong then. All the threads I find when googling the error message, suggests that the region is wrong. Are you sure the bucket is created in us-east-2? What's the name of the region when you view the bucket in your s3 console (where it says EU (Ireland) in the attached screenshot)?

[smockensturm]

US East (N. Virginia)

[aelvan]

North Virginia is us-east-1 according to the list here: https://docs.aws.amazon.com/general/latest/gr/rande.html

[smockensturm]

Boom! That does the trick.

I don't know what is up then with the Amazon S3 url string then that says:

https://s3.console.aws.amazon.com/s3/home?region=us-east-2

Anyway. Thanks for helping us track this down.

[aelvan]

The region string in the url has nothing to do with where your buckets are, it's just where your account are I think, or where the console is loading from. Mine says us-west-2, even though I have no buckets in that region. ;) Glad we got that sorted out.

But I kinda think this issue is unrelated to what @Tam, @sergeifilippov and @andrewmenich reported. Did you guys figure out what the problem was?

[sergeifilippov]

For me the issue was resolved by clearing out the web/imager folder. 👍🏻

[nfourtythree]

I am having a strange issue with this too. Checking the local imager folder the transformed files are there, however it seems to have only uploaded 5 of the 6 images.

Local Folder

AWS Bucket

Have cleared the local imager folder and all cache and everything.

[aelvan]

@nfourtythree Anything in the debug toolbar? See my reply above on what to look for.

[nfourtythree]

@aelvan enabled debug toolbar. Killed the local folder. Reloaded the page, nothing came up in the logs in terms of errors.

Interestingly the same issue happening on another page and it is always the 6th image, that same transform that is hashed to 8033315…

But doesn't happen on all of them. So there might be three images on the page using the same set of 6 transforms. 1 image, all of the images are fine, the other two only 5 of the 6 transforms have uploaded.

Let me know if you want to talk through/screen share on slack or somethign

[nfourtythree]

Any thoughts on this @aelvan? It seems quite strange I have noticed it on a few pages of the project I am currently working on and it seems like it seems to fail to upload 1 of the transforms. So if there are 3 it would upload 2, or 5 it would upload 4 etc.

[Kashkin]

We are having the same issue as @nfourtythree. I haven't noticed a pattern, but some images don't make it to AWS.

[jwrudzin]

We are experiencing a similar issue where our imager transforms are saving properly locally, but never making it to S3.

Uploading to S3 from Craft assets works without any issue, so it doesn't appear to be a permissions problem. We are not seeing any imager or S3 errors in the debug toolbar. In our S3 logs we are seeing the following error:

<Error>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
</Error>

[karlmacklin]

I have a similar issue.
Nothing gets uploaded to AWS, and I can't find any debug info anywhere.

Anyone know where to enable the "debug toolbar"? Is that Craft, Imager or AWS specific?

The generated transforms are supposed to reside in a numbered folder, for example "300" - but that folder doesn't even exist in the bucket.

I have emptied all local imager caches.
Using Fiddler (proxy software) I can analyze all outgoing traffic and I can see that there is never any request to AWS at all. Strangely enough.
Edit: Ok maybe Fiddler isn't 100% reliable to detect this or not. Nevertheless - folder's never created.

It's weird, we can upload files to AWS with Craft. But the Imager transforms aren't uploaded, in certain environments.
Could this be a folder permission issue? We have one troublesome installation that runs nginx and php as straight processes in Windows.
We have a working installation running under Homestead (Linux Vagrant box) and that one works.

[jwb009]

Just want to echo the previous two post and say that I can't for the life of me get Imager to write to my S3 bucket. I have 1 site on my Homestead/Vagrant server that works fine, along with the staging and live versions which live on a Forge server. But the new site that I created on Homestead will not write any images to S3 (even though Craft itself uploads the images fine)

[jwb009]

Updating my post, as I fixed the issue by setting public $storages = ['aws']; in imager.php config. I had previously left it blank.

[sergeifilippov]

Updating my post, as I fixed the issue by setting public $storages = ['aws']; in imager.php config. I had previously left it blank.

@jwb009 So you've defined it above the returned configuration array?

public $storages = ['aws'];

Or did you the this as part of the returned configuration array?

...
'storages' => ['aws'],
...

[jwb009]

Above. So the relevant code looks like:

 public $storages = ['aws'];
 public $storageConfig = [
        'aws' => [
            'accessKey' => '••••••••••••',
            'secretAccessKey' => '••••••••••••••••',
            'region' => 'us-east-1',
            'bucket' => 'my-bucket-name',
            'folder' => 'imager',
            'requestHeaders' => array(),
            'storageType' => 'standard',
            'cloudfrontInvalidateEnabled' => false,
            'cloudfrontDistributionId' => '',
        ],
        'gcs' => [
            'keyFile' => '',
            'bucket' => '',
            'folder' => '',
        ],
    ];

[petetak]

I've seen this problem as well - it's much more likely to happen when you upload over 5 images at once (this seems to be reflected in the thread above)

Could it simply be an AWS upload issue and we just need to retry?

Does this line suggest that if the upload fails then the Craft queue job is not marked as 'failed' (so that it can be retried in the queue later)?

Imager-Craft/src/jobs/OptimizeJob.php

Line 63 in e644e8e

if (!$result) {

I'm not an expert in this but it looks to me like this would just fail in the event of an AWS upload fail

[petetak]

I got to the bottom of my issue which I'll write here in case it affects anyone. Images generated via the elements api were getting uploaded to S3 but to the wrong 'folder'. They were getting uploaded to s3://bucket/_imager/var/www/...... ie: including the full web root.

I tried to fix this by adding 'imagerSystemPath' to the imager config (which enabled imager to remove the the full path of the image when creating the s3 url). This still didn't work because we store images in a shared folder that doesn't get overwritten by a deploy so I had to fix that issue too. Once that was done it worked as expected.

Be sure to fully check your s3 bucket to see that missing files are not elsewhere...

[JWheeler]

@petetak -

Would you be willing to share your configuration for this task? I'd love to compare your configuration change.

[petetak]

@JWheeler sure thing. My images crops live in a folder called 'shared' whereas the craft site lives in a folder called 'current'. In order to make sure the S3 filepaths are correctly created I needed to swap out current for shared. It's not a great solution but it works...

'imagerSystemPath' => str_replace ( '/current' , '/shared' , $_SERVER['DOCUMENT_ROOT']) . '/imager/',