incremental ingest strangeness

[carlseibert]

Hi Kirk,

I've been having some problems with incremental ingest lately. (It's sometimes/often/too often not happening and the whole contents of a card are copied.)

Whist trying to figure out what might be going wrong, I took a look in my incremental folder in cache. Looks OK. The number of files pretty well comports with the number of cards I have.

But when I look into the files, it appears that since some date in the reasonably distant past, Photo Mechanic isn't recording the names of the RAW half of RAW/JPEG pairs. I have both Fuji and Leica cameras and both seem to be behaving this way. Old RAWS are listed; new ones aren't.

I just tried a brand-new card in the Leica and sure enough on two ingest runs, 50 files (of both types) were copied to the computer but only 25 names (just JPEGs) appear in the incremental file. So, in this case, Ingest is behaving properly, but the contents of the ingest file don't seem right.

For grins, I tried a Nikon (I don't shoot RAW/JPEG there, 'cause Nikon kindly writes full-size previews.) Three runs, all fine, both Ingest behavior-wise and contents of the ingest file-wise.

Of course, I wasn't able to reproduce - or solve - the original issue. Sigh. That only happens with a nearly-full card when I truly don't have time to wait for a thousand files to copy. ;-)

-Carl

Photo Mechanic Plus Version 6.0, build 6245 (14de08a) on Mac Mohave

[Kirk Baker]

Carl,

I've been having some problems with incremental ingest lately. (It's sometimes/often/too often not happening and the whole contents of a card are copied.)

Whist trying to figure out what might be going wrong, I took a look in my incremental folder in cache. Looks OK. The number of files pretty well comports with the number of cards I have.

But when I look into the files, it appears that since some date in the reasonably distant past, Photo Mechanic isn't recording the names of the RAW half of RAW/JPEG pairs. I have both Fuji and Leica cameras and both seem to be behaving this way. Old RAWS are listed; new ones aren't.

I just tried a brand-new card in the Leica and sure enough on two ingest runs, 50 files (of both types) were copied to the computer but only 25 names (just JPEGs) appear in the incremental file. So, in this case, Ingest is behaving properly, but the contents of the ingest file don't seem right.

Can you post a screenshot of your Ingest dialog showing all of your settings?  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your JPEG format screenshot.

Thanks,

-Kirk

[carlseibert]

Here you go...

There's no SD card inserted at the moment but otherwise it's exactly as I last used it.

[Kirk Baker]

Carl,

Thanks for the image showing your settings.  See below for discussion.

I've been having some problems with incremental ingest lately. (It's sometimes/often/too often not happening and the whole contents of a card are copied.)

Whist trying to figure out what might be going wrong, I took a look in my incremental folder in cache. Looks OK. The number of files pretty well comports with the number of cards I have.

But when I look into the files, it appears that since some date in the reasonably distant past, Photo Mechanic isn't recording the names of the RAW half of RAW/JPEG pairs. I have both Fuji and Leica cameras and both seem to be behaving this way. Old RAWS are listed; new ones aren't.

I just tried a brand-new card in the Leica and sure enough on two ingest runs, 50 files (of both types) were copied to the computer but only 25 names (just JPEGs) appear in the incremental file. So, in this case, Ingest is behaving properly, but the contents of the ingest file don't seem right.

When you ingest RAW+JPEG, only the JPEG of the pair is recorded.  If you ingest RAW+JPEG again, it should not copy the RAW files or the JPEGs again.  That said, there is an issue that occurs twice a year (when the time changes) where Ingest will copy files previously copied because the dates on the cards when interpreted with the current GMT offset causes the dates to not match the dates in the ingest data in its cache.

Of course, I wasn't able to reproduce - or solve - the original issue. Sigh. That only happens with a nearly-full card when I truly don't have time to wait for a thousand files to copy. ;-)

Did this only happen with the recent time change?

-Kirk

[carlseibert]

Hi Kirk,

Sorry about the delay getting back.

I suspected the only looking at the JPEG of RAW+JPEG thing. OK. Just another case of not being able to see the swamp for the alligators. With that distraction out of the way...

I've had three of these incidents lately.

One disk was ingested before the time change, but it's quite likely that the previous ingest was prior to the previous time change. (A card that hadn't been ingested in over six months, in other words.)

The other two were ingested post-time-change, but had pre-time-change frames that hadn't yet been ingested on them. Would it have made a difference if the cards had pre-time-change files that had once been ingested?

It seems plausible that crossing a time change could be the culprit. I was thinking about whether the events might happen after a version upgrade had occurred or about things I may have done with the cards. But the time change never occurred to me.

Is there a practicable work-around? The only thing I can think of is to suck it up and do a whole-card ingest of any card that's lingering in a camera on time-change day and delete the excess downloaded files so as to not be caught out on deadline. How practical might be is debatable. But my only other thought- temporarily changing the system time on my laptop - seems even less so. Prolly wouldn't work at all as soon as the system time corrected itself.

I guess in the days when cards were small and I had only one pair of digital cameras and shot them every day, this was a thing that just never came up. Or, if it did, there was sufficiently little data on a card that I didn't really notice. Nowadays even my 64 GB cards can eat a massive chunk of time.

Now if we stopped the silly clock-changing for real this time, this would be moot... :-)

-Carl

[Kirk Baker]

Carl,

I suspected the only looking at the JPEG of RAW+JPEG thing. OK. Just another case of not being able to see the swamp for the alligators. With that distraction out of the way...

I've had three of these incidents lately.

One disk was ingested before the time change, but it's quite likely that the previous ingest was prior to the previous time change. (A card that hadn't been ingested in over six months, in other words.)

That'd do it, but I would expect only those images more than 6 months old would be ingested in error.

The other two were ingested post-time-change, but had pre-time-change frames that hadn't yet been ingested on them. Would it have made a difference if the cards had pre-time-change files that had once been ingested?

They would be ingested again.

It seems plausible that crossing a time change could be the culprit. I was thinking about whether the events might happen after a version upgrade had occurred or about things I may have done with the cards. But the time change never occurred to me.

Is there a practicable work-around?

Workaround?  Not that I can think of.  If the filesystem reports that the modification time is different from the modification time it reported when it was ingested then the files will be copied.

As far as fixing the issue goes, I can think of two ways:

1) Record a checksum of the image in the ingest data.  Generate a checksum for files when they're being considered for ingest.  This would be foolproof, but very slow since the entire file would have to be read in order to generate the checksum.
2) Check to see if the date +/- an hour matches the ingest data.  And check to see if there have been any time change transitions between now and when they were shot.  I don't know a good way to do this that would work all over the world though.  It would take much investigation and validation.

The only thing I can think of is to suck it up and do a whole-card ingest of any card that's lingering in a camera on time-change day and delete the excess downloaded files so as to not be caught out on deadline. How practical might be is debatable. But my only other thought- temporarily changing the system time on my laptop - seems even less so. Prolly wouldn't work at all as soon as the system time corrected itself.

I guess in the days when cards were small and I had only one pair of digital cameras and shot them every day, this was a thing that just never came up. Or, if it did, there was sufficiently little data on a card that I didn't really notice. Nowadays even my 64 GB cards can eat a massive chunk of time.

Have you considered ingesting more frequently and reformatting the card in-camera before you shoot new photos?  If you have images that are 6 months old and you haven't ingested them...  They're your only copies until they get copied to your computer.

Now if we stopped the silly clock-changing for real this time, this would be moot... :-)

Right.  There may be only a couple more time changes and then the problem won't be a problem anymore.

-Kirk

[carlseibert]

Have you considered ingesting more frequently and reformatting the card in-camera before you shoot new photos?  If you have images that are 6 months old and you haven't ingested them...  They're your only copies until they get copied to your computer.

A solid point indeed.

Thanks! Stay well.