Author Topic: Incremental ingest and time zones  (Read 7079 times)

Offline devenh

  • Sr. Member
  • ****
  • Posts: 435
    • View Profile
Incremental ingest and time zones
« on: June 22, 2012, 10:27:52 AM »
Just returned from a trip and I was using PM every day to incrementally ingest photos.  Worked fine.  I didn't ingest the last day's photos until I returned home.  When I ingested, PM read the entire card.  I suspect this is be because my laptop's time zone changed.  When I reset the laptop's time zone to where I was before, incremental ingest worked.

Using 4.6.9 under Window 7 64 bit.

Deven

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Incremental ingest and time zones
« Reply #1 on: June 22, 2012, 10:30:07 AM »
Deven,

Just returned from a trip and I was using PM every day to incrementally ingest photos.  Worked fine.  I didn't ingest the last day's photos until I returned home.  When I ingested, PM read the entire card.  I suspect this is be because my laptop's time zone changed.  When I reset the laptop's time zone to where I was before, incremental ingest worked.

Interesting.  We'll have to look into that for PM5.

-Kirk

Offline devenh

  • Sr. Member
  • ****
  • Posts: 435
    • View Profile
Re: Incremental ingest and time zones
« Reply #2 on: June 22, 2012, 10:48:37 AM »
Another thing you may want to look at is the insufficient disk space warning.  This appears to be calculated using the entire space used on the card rather than just on the images that need to be incrementally ingested.

Deven

Offline fotopan

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Re: Incremental ingest and time zones
« Reply #3 on: March 26, 2017, 03:40:53 PM »
Deven,

Just returned from a trip and I was using PM every day to incrementally ingest photos.  Worked fine.  I didn't ingest the last day's photos until I returned home.  When I ingested, PM read the entire card.  I suspect this is be because my laptop's time zone changed.  When I reset the laptop's time zone to where I was before, incremental ingest worked.

Interesting.  We'll have to look into that for PM5.

-Kirk

Was this problem solved? Because today - first day after my time zone changed (from UTC+1 to UTC+2 - daylight time saving) PM tried do ingest all photos from a card that has photos dating from November the 1st, 2016 instead of last shot only.

When I compare c:\Users\...\AppData\Roaming\Camera Bits, Inc\Photo Mechanic\ingest\incremental\{ACCF9DAD-194A-437d-82F5-8EB2EB7ABC83} file written today and the same file from a backup, I can see that today the timestamps of files created before time zone change are calculated incorrectly thus making PM downloading all the files again.

In a file written before time zone change, I can see:
DCIM\123_1101\R0001885.JPG   6619136   1477970832 -> PM thinks the file was created Tue Nov 1 3:27:06 UTC 2016

In a file written today I see:
DCIM\123_1101\R0001885.JPG   6619136   1477967232 -> PM thinks the file was created Tue Nov 1 2:27:22 UTC 2016
--
Pawel

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Incremental ingest and time zones
« Reply #4 on: March 27, 2017, 11:45:53 AM »
Deven,

Just returned from a trip and I was using PM every day to incrementally ingest photos.  Worked fine.  I didn't ingest the last day's photos until I returned home.  When I ingested, PM read the entire card.  I suspect this is be because my laptop's time zone changed.  When I reset the laptop's time zone to where I was before, incremental ingest worked.

Interesting.  We'll have to look into that for PM5.

-Kirk

Was this problem solved? Because today - first day after my time zone changed (from UTC+1 to UTC+2 - daylight time saving) PM tried do ingest all photos from a card that has photos dating from November the 1st, 2016 instead of last shot only.

When I compare c:\Users\...\AppData\Roaming\Camera Bits, Inc\Photo Mechanic\ingest\incremental\{ACCF9DAD-194A-437d-82F5-8EB2EB7ABC83} file written today and the same file from a backup, I can see that today the timestamps of files created before time zone change are calculated incorrectly thus making PM downloading all the files again.

In a file written before time zone change, I can see:
DCIM\123_1101\R0001885.JPG   6619136   1477970832 -> PM thinks the file was created Tue Nov 1 3:27:06 UTC 2016

In a file written today I see:
DCIM\123_1101\R0001885.JPG   6619136   1477967232 -> PM thinks the file was created Tue Nov 1 2:27:22 UTC 2016

Part of the problem is that getting the modification time (seconds since the OS epoch) on a file changes when your GMT offset changes.  What we'd need to do is write out the modification time adjusted to absolute GMT time and moreover determine which ingest tracking files have the old style modification times and convert them to the "new way" or everyone would have this problem.

Windows in particular does return a different modification time for a file when the GMT offset changes.  This would have to be solved first.

-Kirk

Offline fotopan

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Re: Incremental ingest and time zones
« Reply #5 on: March 27, 2017, 02:37:55 PM »
Something should be done though.  :)

I would suggest adding an information to the timestamp about UTC offset used when the timestamp was calculated - that way whenever the time zone changes and OS reports different file modification time, PM will be able to compare the time with a timestamp stored in the incremental ingest "memory" file. And it would be easy to differentiate between "old" and "new" files - the new ones would have another column with the offset added.

Also, in a situation when there are files with exactly the same filenames and sizes and only timestamps differ, I think warning user about possible ingest error would be a nice to have feature. Yesterday PM ingested several hundred files before I realized it is contaminating my photo library with duplicates (in a last few weeks I've changed the ingest renaming scheme so PM didn't try to overwrite files and kept on adding duplicate files with different file names instead).
--
Pawel

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Incremental ingest and time zones
« Reply #6 on: March 27, 2017, 02:47:22 PM »
Something should be done though.  :)

Agreed.

-Kirk

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Incremental ingest and time zones
« Reply #7 on: March 28, 2017, 04:13:59 PM »
Yes, Windows 10 Pro, 64-bit, Polish language version.

OK.  I have investigated this problem for a number of hours trying to figure out why on Windows, specifically with a FAT file system, that the modification time values can vary.  It turns out that modification times on FAT file systems are written using the local time of the computer (or camera!) that sets them.  Moreover these times are translated to UTC times by modern Windows OSes and these times are cached.  To make things even more difficult, these times are cached such that the transition from standard time to daylight savings time does not necessarily take effect when it should.

I eventually stumbled across this document from Microsoft: https://msdn.microsoft.com/en-us/library/ms724290.aspx

Quote
The FAT file system records times on disk in local time. GetFileTime retrieves cached UTC times from the FAT file system. When it becomes daylight saving time, the time retrieved by GetFileTime is off an hour, because the cache is not updated. When you restart the computer, the cached time that GetFileTime retrieves is correct. FindFirstFile retrieves the local time from the FAT file system and converts it to UTC by using the current settings for the time zone and daylight saving time. Therefore, if it is daylight saving time, FindFirstFile takes daylight saving time into account, even if the file time you are converting is in standard time.

I have been unable to find a way to get a modification time on a Windows PC that can be reliably mapped to UTC.  In my investigation, I repeatedly changed my time zone and the modification times reported by the files would change.  If I pushed my clock back to before the daylight savings time change, the modification times would also change.

I tried setting specific modification times on both a Mac and on Windows, and the only cases where the modification time would remain unchanged was when my computer's time zone and daylight savings value were the same as was in place at the moment the modification time was set.  If one tries to account for DST (and then change the time in code to fix the delta), it is possible to do the wrong thing.  For instance if I set the modification time for a file to some time in the past (when it is in standard time) but the time on my system is in daylight savings time and then later read that time when it is still daylight savings time, it is incorrect to adjust for DST.  At another time in the year when standard time is in effect, I may have to adjust in a different direction that cannot be determined because it is impossible to tell what UTC offset was in place when the modification time was set.

Basically, when it comes to disks that are formatted as FAT16 or FAT32 and used in a Windows PC, the modification times can be interpreted with significant error depending on time zone changes and daylight savings time changes.

Other file systems like Windows' NTFS filesystem always use UTC times and are completely unaffected by this issue.

Currently as implemented, incremental ingest errs on the side of copying more files than necessary, but never ignoring something that should be copied.

Given the foibles of FAT file systems, it appears that this is the best we can do without dramatically slowing down the process and using checksums (this would require reading every file on the card to get the checksums) to verify files copied by content rather than by modification time.

-Kirk

Offline fotopan

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Re: Incremental ingest and time zones
« Reply #8 on: March 31, 2017, 11:03:11 PM »
Thank you for your investigation! It looks like there really is not a good way to handle this case with no user intervention.  :(
Do you think it would be possible to implement two functions:
  • a preview of what is going to be ingested (even not necessarily in the form of icons, the list itself would suffice),
  • the ability to ignore the date when the card is scanned?
Having such a features, I could scan the card with the date comparison option on and off and see what the difference is should I suspect there are too much files going to be ingested.
--
Pawel

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Incremental ingest and time zones
« Reply #9 on: March 31, 2017, 11:25:28 PM »
Thank you for your investigation! It looks like there really is not a good way to handle this case with no user intervention.  :(
Do you think it would be possible to implement two functions:
  • a preview of what is going to be ingested (even not necessarily in the form of icons, the list itself would suffice),
  • the ability to ignore the date when the card is scanned?
Having such a features, I could scan the card with the date comparison option on and off and see what the difference is should I suspect there are too much files going to be ingested.

PM6 will have Ingest from Selection so you can always browse the card, select what you want and then Ingest.

We aren't adding features to PM5, only new camera and OS support.

-Kirk

Offline fotopan

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Re: Incremental ingest and time zones
« Reply #10 on: April 01, 2017, 12:23:14 AM »
OK, can't wait for PM 6 then!
--
Pawel