Amazon S3 Uploader fails because I already own the bucket?

[pstagg]

I have created an S3 bucket to upload images into from PM but when I try to upload I get the following nonsensical error:
AmazonS3   ./ruby/lib/ruby/1.8/aws/s3/error.rb:38:in `raise': Your previous request to create the named bucket succeeded and you already own it.

If I tell PM to upload into a bucket that doesn't exist that's not a problem.

Why is the uploader trying to create a bucket anyway? It got the name of the bucket from AWS in the first place when I provided my Access key id and Secret key in the first place.

FYI I can access the bucket, upload, download, edit etc. without an issue using several other tools I have and the same AIM user credentials.

Confused. Please help.

[Kirk Baker]

I have created an S3 bucket to upload images into from PM but when I try to upload I get the following nonsensical error:
AmazonS3   ./ruby/lib/ruby/1.8/aws/s3/error.rb:38:in `raise': Your previous request to create the named bucket succeeded and you already own it.

If I tell PM to upload into a bucket that doesn't exist that's not a problem.

Why is the uploader trying to create a bucket anyway? It got the name of the bucket from AWS in the first place when I provided my Access key id and Secret key in the first place.

FYI I can access the bucket, upload, download, edit etc. without an issue using several other tools I have and the same AIM user credentials.

Confused. Please help.

I'd like to see your Photo Mechanic log.  In Photo Mechanic go to the Help menu and choose "Reveal Support Data..." and moments later a Finder window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

-Kirk

[pstagg]

Thanks Kirk. PM_log.zip attached.
I can see some more interesting error in it leading up to the one I was seeing in the Upload log:

[Fri May-04-2018 21:36:27.786][b154f000][1426668] [0fe6b1bc] Exception while checking for bucket The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>images.staggarts.com</Bucket><Endpoint>images.staggarts.com.s3.amazonaws.com</Endpoint><RequestId>D54387D1481653D9</RequestId><HostId>Qz2fX3bZJ0mei2a44FkEI/oB0zE91R5I5yGUTBALXJv6k68mABsNLPWtxnIDB5wl1EoG5zp6za0=</HostId></Error>
[Fri May-04-2018 21:36:28.775][b154f000][1426692] [0fe8222c] ruby-bridge: exception: #<AWS::S3::BucketAlreadyOwnedByYou: ./ruby/lib/ruby/1.8/aws/s3/error.rb:38:in `raise': Your previous request to create the named bucket succeeded and you already own it.>

I've done some more research and a little experiment. It appears that this is the age old problem of bucket region. Several pages on the web indicate that the error(s) above a generated because "bucket region is incorrect" and I've read a few posts on this forum that suggest that PM only works with the default "US Standard" region.

As an experiment I created a bucket in the US East region with identical configurations etc to my first bucket (Asia Pacific [Sydney] - because I live in Australia). I have no problem Uploading to this bucket. "So problem solved" I here you say. Sorry but no this just means that PM's S3 bucket support has been programmed from a parochial point of view.

Looking over S3 AWS documentation etc there appears to be a misnomer that S3 bucket region makes no difference. This seems to be partially because in the S3 dashboard the phrase "S3 does not require region selection" appears under the region selection list. However when you create a bucket you get to choose a region, why?

Apparently quoted from S3 docs.

"Amazon S3 creates bucket in a region you specify. You can choose any AWS region that is geographically close to you to optimize latency, minimize costs, or address regulatory requirements. For example, if you reside in Europe, you might find it advantageous to create buckets in the EU (Ireland) or EU (Frankfurt) regions. For a list of AWS Amazon S3 regions, go to Regions and Endpoints in the AWS General Reference."

I may have misinterpreted the various docs. I've read on this so I'm happy to be proved wrong but this all suggests to me that the PM S3 Uploader really needs a way of choosing the bucket region.

[Kirk Baker]

"Amazon S3 creates bucket in a region you specify. You can choose any AWS region that is geographically close to you to optimize latency, minimize costs, or address regulatory requirements. For example, if you reside in Europe, you might find it advantageous to create buckets in the EU (Ireland) or EU (Frankfurt) regions. For a list of AWS Amazon S3 regions, go to Regions and Endpoints in the AWS General Reference."

I may have misinterpreted the various docs. I've read on this so I'm happy to be proved wrong but this all suggests to me that the PM S3 Uploader really needs a way of choosing the bucket region.

Unfortunately, the module we use for Amazon S3 is sorely out of date and has not been updated in some time.  We're switching to Amazon's own S3 implementation in PM6 so the problems that so many people have experienced with the S3 uploader in PM5 (and PM 4.6) will finally be solved.

I don't have a suggestion or a workaround for you at this time that will let you work with existing buckets.

-Kirk

[pstagg]

Thanks again Kirk,

That makes a lot of sence. I trawled through the ruby gems myself only to find that they were made of glass. I couldn’t see anywhere where I could insert a region for a bucket and the documentation on Amazon didn’t match what I was seeing. So that all tallies.

I’m beggining a journey to see what I can do with my photos and Amazon AWS. Partially as a learning exercise so glitches like this I’m happy to put down as part of the learning process. For the timebeing I have a number of other ways to upload photos to the S3 bucket. I look forward to PM6 - been a long time user and IMHO PM has been the best photo management tool on the market for a long time.