Photo Mechanic Version 6.0, build 6890 (ffcbb4c) duplicate downloads

[rcohn]

At first I thought I had done something wrong when I saw some duplicate files downloaded a few days ago, and I went through and deleted the duplicates, but yesterday PM re-downloaded the entire card even though there should have only been a limited number of new files on the card. I had changed time zones so wanted to download the remaining files and wipe the card before changing the camera time.

Deleting these duplicate files is a real mess since I have already culled a number of files and now they are downloaded as singles. Carefully fixing this is going to take me a while. However, my biggest concern is whether I did something wrong or if there is a problem with this build. First thing I'm going to do is revert to 6.0, build 6784 (409cf99) which had been rock solid. I had already had a problem with the previous build which was supposed to be fixed with 6890.

The screenshot shows image number 00016 downloaded and named on 10/27/23 and the same image numbered 03593 which was downloaded last night 11/3/23).

Using a MacBook Pro 16 inch 2021 with M1 Pro chip. Ventura 13.5.1.

I'm pretty sure this is not user error, but I could be wrong.

[Kirk Baker]

At first I thought I had done something wrong when I saw some duplicate files downloaded a few days ago, and I went through and deleted the duplicates, but yesterday PM re-downloaded the entire card even though there should have only been a limited number of new files on the card. I had changed time zones so wanted to download the remaining files and wipe the card before changing the camera time.

Deleting these duplicate files is a real mess since I have already culled a number of files and now they are downloaded as singles. Carefully fixing this is going to take me a while. However, my biggest concern is whether I did something wrong or if there is a problem with this build. First thing I'm going to do is revert to 6.0, build 6784 (409cf99) which had been rock solid. I had already had a problem with the previous build which was supposed to be fixed with 6890.

The screenshot shows image number 00016 downloaded and named on 10/27/23 and the same image numbered 03593 which was downloaded last night 11/3/23).

Using a MacBook Pro 16 inch 2021 with M1 Pro chip. Ventura 13.5.1.

I'm pretty sure this is not user error, but I could be wrong.

If you still have the image on the card, what is the modification time of the image on the card?

-Kirk

[ahoward]

I did my best to simulate your workflow and the time zone change to see if I could reproduce the issue and I was not able. Daylight savings time can mess up incremental Ingest until the card is formatted, so it's not impossible that a time change could cause the issue as well.

Carefully fixing this is going to take me a while.

It's possible that we can take care of this pretty easily using the frame value. If you want to give that a shot, I'd ask you to give our support team a call, as its an easy process to show you but a lengthy one to type out.

[rcohn]

Thank you Kirk and ahoward, I purposely have not reformatted the card. I have attached a screenshot of the image on the card in the Mac Finder. Created Friday, October 27, 2023 at 12:26 PM. I have also attached screenshots for the downloaded and renamed file and the associated .xmp file. I did delete the duplicate download so can't look at that.

Since I didn't change the time in the camera until after downloading this card I don't see how the camera could know there was a time change from Arizona to NJ. I also had this problem before the end of Daylight Savings Time on Saturday night so even if the camera has an internal knowledge of this date (which I do not believe is the case) that time change is not likely to be the cause. I will try to find the time to call the support team tomorrow, but will the person who answers know about my problem or will I have to try to explain it all again?

Thanks,

Ric Cohn

[ahoward]

I will try to find the time to call the support team tomorrow, but will the person who answers know about my problem or will I have to try to explain it all again?

Feel free to ask for Andrew, I'm in at 9 am PST.

[Kirk Baker]

Ric,

Thank you Kirk and ahoward, I purposely have not reformatted the card. I have attached a screenshot of the image on the card in the Mac Finder. Created Friday, October 27, 2023 at 12:26 PM. I have also attached screenshots for the downloaded and renamed file and the associated .xmp file. I did delete the duplicate download so can't look at that.

Since I didn't change the time in the camera until after downloading this card I don't see how the camera could know there was a time change from Arizona to NJ. I also had this problem before the end of Daylight Savings Time on Saturday night so even if the camera has an internal knowledge of this date (which I do not believe is the case) that time change is not likely to be the cause. I will try to find the time to call the support team tomorrow, but will the person who answers know about my problem or will I have to try to explain it all again?

When the system's time zone changes or when there is a change due to daylight savings time transition, the modification time of files on disks changes as well.  Ingest records the modification times of the files it copies to a file (for later use by Incremental Ingest.)  If the time offset on the system changes, the system will report modification times for files different than it did before (they'll change by some number of hours depending on the situation).  This causes Incremental Ingest to ingest files that have already been ingested.

This problem has affected Incremental Ingest for many years.  Now, if you're saying that not all of the images shot prior to your time zone change were copied again, then we may have a different situation.  We'd need to analyze the incremental ingest data files and the files on your card.

-Kirk

[rcohn]

Kirk,

If I'm understanding you correctly, it is the computer's changing its time zone that causes this? That of course happens automatically when you travel to a different time zone and connect to the internet. That is very good information to know. I will need to make sure all files are downloaded before connecting to the internet if I shoot in one time zone and don't have an opportunity to download those files before traveling to a new time zone (although harder for me to remember and deal with).

Ric

[Kirk Baker]

Ric,

If I'm understanding you correctly, it is the computer's changing its time zone that causes this? That of course happens automatically when you travel to a different time zone and connect to the internet. That is very good information to know. I will need to make sure all files are downloaded before connecting to the internet if I shoot in one time zone and don't have an opportunity to download those files before traveling to a new time zone (although harder for me to remember and deal with).

And you'll need to either remove the old images from the card or reformat it in camera, or they'll be downloaded again.

-Kirk

[rcohn]

Kirk,

I did find the duplicate file from the screen shot in the original post. I forgot that I moved them all to a separate folder rather than the trash in case I made a mistake and haven't deleted it yet. The downloaded duplicate has the same Created time as my screenshot from the card. Why it only differs by 1 hour instead of 3 is a mystery to me, but the important thing is I can see why PM would think it is a new file.

Thanks as always for your help,

Ric

[Kirk Baker]

You're welcome, Ric.

-Kirk

[rcohn]

Sorry, but duplicate downloads returned! I don't see what I could have done wrong this time. I went from NJ to Missouri. Changed camera time before I left. I downloaded all images while still in Missouri. Reformatted the CFexpress card on my return. 

I just realized that there are a number of duplicate files (but only from day one of a multi day shoot). I don't see how this could have happened other than while in Missouri. I have uploaded screen shots showing two of the images. One ends in 00070 and the other in 01479. If I had had the viewer set to Capture Time I probably would have discovered this before it became a big mess for me.

The capture time in PM are identical. However, the capture times in the Finder differ by 1 hour. Again. I did not change the camera time while in Missouri and I did not download any images on my return so I am baffled.

When I am shooting multiple long days I always download each day but only spot check the images so it is easy for me to miss this until later. This image is from my first day. the duplicate either downloaded twice when I download the first day or would have downloaded along with the new images from the second day because there are images from my third and fourth day with higher numbers. Also, the second day starts with file ending in 02720. There are no duplicates from the second through fourth days.

Normally, my projects are only one day and each project gets put in a separate folder with a unique name rather than being downloaded on multiple days using the same name with continuous numbers, so I would only see this on a shoot that encompasses more than one day.

Is there something I could have done wrong or some way to prevent this from happening again? Or is there a bug that only manifests under specific conditions?

PM 6.0, build 6890 (ffcbb4c)
Mac OS Ventura 13.6.2
Canon R5
Sony 256 GB CFexpress card

[MNeipris]

rcohn, can you send us the three following sample files to support@camerabits.com:

1. The original CR3 file of the Bald Eagle on the memory card
2. 20231213_Missouri_00070.CR3 seen in screenshot
3. 20231213_Missouri_01429.CR3 seen in screenshot

Please compress those three files into a ZIP folder before sending. The file will be too large for email attachment, so you can use a cloud hosting link with the permissions set to anyone with link can access, or use a file sharing service like WeTransfer.

[rcohn]

MNeipris, thank you for following through! Unfortunately, I reformatted the CFexpress card the files were on. I rename on download so those are all I have.

I will send you the two downloaded files via WeTransfer, but I realize they may not help without the original file on the CFexpress card.

Best regards,
Ric

[MNeipris]

Thank you for the sample files, rcohn.

Is it possible that your computer's time/time zone did not change when you physically got to Missouri?

For example, if the computer was in Airplane Mode, or the WiFi was disabled, the computer's time may still have been set to Eastern Time in New Jersey while you were physically in Missouri, in Central Time.

If the computer's time zone was still set to Eastern Time during the first Incremental Ingest, and if the computer connected to the Internet in between ingests, the computer's time zone could have updated to Central Time, which would cause Incremental Ingest to ingest the previously ingested files.

Separate from the computer's time zone, looking at the Makernotes metadata for the sample CR3 files using Exiftool, the camera's time zone was still set to New York, and Daylight Savings Time was set to On (Daylight Savings Time ended on November 5th, 2023).

[rcohn]

No, my computer showed the correct time and time zone my whole time in Missouri, and I downloaded all files while still there.

HOWEVER, now that you mention it, when I got home there was a problem with my calendar where a couple of events stayed on Midwest time when I returned to the East coast. My Apple Calendar showed it was set to Eastern Time, yet some items displayed in Midwest time. I had to call Apple support to sort it out, and the tech support person confirmed that all my preference settings were correct. The errors had to be fixed one at a time by selecting them and changing the time zone. So maybe there was a problem on Apple's part that caused this? If this was caused by a problem like this, I'm just not sure what I could have looked out for that would have allowed me to prevent this. Does this suggest anything you have seen before?

Screen shot just to show what I mean by the calendar showing the correct time zone until I clicked on the events that were stuck in the wrong time zone, then I could change to Eastern time manually.