Can I convince Injest to skip files already injested?

[Mike-D]

This one is a bit difficult to describe, and I hope I don't get kicked for mentioning another product, but here goes.

When I was a PC user, I used a great little program called Downloader Pro. All it did was download files, and it was great at it.  My workflow around card dumping went like this: I would copy all the files from all cards from a wedding shoot onto a portable USB hard drive while I watched TV after the event. Then, before I went to bed, I'd plug in the USB drive, begin the download using DLP, and walk away. 

The next day, I would stick in each card, and download the files from them as well, individually (I did this just to make sure nothing was missed by the download to the USB drive).  However: Downloader Pro was smart enough to know when a file had already been downloaded, and skipped it, so dumping the 10 or so individual cards took about 5 minutes: just long enough to scan the card, and determine that there was nothing new to download.

Now, though, I'm in Macland, and Photo Mechanic does almost everything DP did.  The only thing I can't make it do is skip the injest of file already injested.  Even when I choose "Incremental ingest" is just appends an "A" to the file name, and re-ingests everything unnecessarily.

So, my question is this: Is there a way to put PM into a similar kind of "verification mode," so that it only injests new files, rather than blindly re-injesting files that are already present in the destination folder?

...Mike

[Kirk Baker]

Mike,

This one is a bit difficult to describe, and I hope I don't get kicked for mentioning another product, but here goes.

When I was a PC user, I used a great little program called Downloader Pro. All it did was download files, and it was great at it.  My workflow around card dumping went like this: I would copy all the files from all cards from a wedding shoot onto a portable USB hard drive while I watched TV after the event. Then, before I went to bed, I'd plug in the USB drive, begin the download using DLP, and walk away. 

The next day, I would stick in each card, and download the files from them as well, individually (I did this just to make sure nothing was missed by the download to the USB drive).  However: Downloader Pro was smart enough to know when a file had already been downloaded, and skipped it, so dumping the 10 or so individual cards took about 5 minutes: just long enough to scan the card, and determine that there was nothing new to download.

Now, though, I'm in Macland, and Photo Mechanic does almost everything DP did.  The only thing I can't make it do is skip the injest of file already injested.  Even when I choose "Incremental ingest" is just appends an "A" to the file name, and re-ingests everything unnecessarily.

So, my question is this: Is there a way to put PM into a similar kind of "verification mode," so that it only injests new files, rather than blindly re-injesting files that are already present in the destination folder?

If you instead use PM to Ingest your cards directly to your hard drive, then Incremental Ingest will work as you're expecting it to work.

-Kirk

[Mike-D]

It's not, though.  Like I said, it's adding an "A" to the end of my file name, and re-injesting everything.

...Mike

[Kirk Baker]

Mike,

It's not, though.  Like I said, it's adding an "A" to the end of my file name, and re-injesting everything.

You stated earlier that you copy your photos from the card to your USB hard drive via the Finder (or previously Windows Explorer) and then you used Ingest.  Is that not correct?  Are you using the "Ingest Disks" option?

-Kirk

[Mike-D]

The files are copied to the portable USB drive using it's built-in copy method (it's basically a portable backup unit).  I use injest to dump the files from this portable drive, then I use injest to dump each card. I was inserting ITPC info, perhaps PM thinks that it's a new file because of that?  I dunno.  But I can't even use DLP in parallels now because the injest feature is modifying the file somehow, beyond the simple rename.  I see that flaw has been brought up before and will be fixed in the next rev.

...Mike

[Kirk Baker]

Mike,

The files are copied to the portable USB drive using it's built-in copy method (it's basically a portable backup unit).  I use injest to dump the files from this portable drive, then I use injest to dump each card. I was inserting ITPC info, perhaps PM thinks that it's a new file because of that?  I dunno.  But I can't even use DLP in parallels now because the injest feature is modifying the file somehow, beyond the simple rename.  I see that flaw has been brought up before and will be fixed in the next rev.

When you ingest from this USB drive, are you ingesting folders or ingesting it as a disk?

-Kirk

[Mike-D]

As a disk.

[Mike-D]

Here is exactly what I'm doing:
1. copy all cards to the external portable storage device (PD70X) using it's copy method
2. attach this unit to my mac
3. injest the external portable drive as a disk, with the following settings:
      o  incremental ingest
      o  copy all photos to same destination
      o  copy directly to primary folder
      o  primary destination is a folder on a local hard disk
      o  no secondary destination
      o  copy locked and unlocked files
      o  copy RAW files only
      o  no IPTC stationary applied
      o  files renamed to the following spec: {filenamebase:4}-{serialnum:-3}
4. once the ingest completes, eject the portable drive
5. insert the CF card into to the firewire reader
6. injest the CF card as a disk, using the exact same settings as above

Result:files from the CF card are re-downloaded and the filename has a "A" appended. For example, file 2000-935.CR2 as downloaded from the portable drive gets re-downloaded from the CF card as 2000-935A.CR2

As an aside, what methodology does PM use to determine if a file is new? It can't be the file size of CRC, as PM modifies all files on download. It can't be the file date for the same reason.

...Mike 

[Kirk Baker]

Mike,

Here is exactly what I'm doing:
1. copy all cards to the external portable storage device (PD70X) using it's copy method
2. attach this unit to my mac
3. injest the external portable drive as a disk, with the following settings:
      o  incremental ingest
      o  copy all photos to same destination
      o  copy directly to primary folder
      o  primary destination is a folder on a local hard disk
      o  no secondary destination
      o  copy locked and unlocked files
      o  copy RAW files only
      o  no IPTC stationary applied
      o  files renamed to the following spec: {filenamebase:4}-{serialnum:-3}
4. once the ingest completes, eject the portable drive
5. insert the CF card into to the firewire reader
6. injest the CF card as a disk, using the exact same settings as above

Result:files from the CF card are re-downloaded and the filename has a "A" appended. For example, file 2000-935.CR2 as downloaded from the portable drive gets re-downloaded from the CF card as 2000-935A.CR2

Can I ask why you're going to the trouble of steps 5 and 6?  You've already got your photos on to your hard drive from steps 1 through 4.

As an aside, what methodology does PM use to determine if a file is new? It can't be the file size of CRC, as PM modifies all files on download. It can't be the file date for the same reason.

Photo Mechanic keeps a list of files that it has ingested from a specific disk.  Since you're ingesting a different disk (the CF card after the USB portable drive) this is why PM thinks it hasn't seen these files before.

-Kirk

[Mike-D]

The next day, I would stick in each card, and download the files from them as well, individually (I did this just to make sure nothing was missed by the download to the USB drive).  However: Downloader Pro was smart enough to know when a file had already been downloaded, and skipped it, so dumping the 10 or so individual cards took about 5 minutes: just long enough to scan the card, and determine that there was nothing new to download.

[Mike-D]

Photo Mechanic keeps a list of files that it has ingested from a specific disk.  Since you're ingesting a different disk (the CF card after the USB portable drive) this is why PM thinks it hasn't seen these files before.

-Kirk

This is kinda scary actually. Here is a scenario for you: I have my camera set to "Auto Reset" which will reset the file numbering each time a card is inserted. Now, lets say I insert a card and fill it, and ingest it...we'll call this group of files Set A.  I then delete Set A from the card, and fill it again (Set B).  Now the card is filled with files that have the same name as Set A, but are actually different files.  If all PM is doing is keeping a record of file names, then Set B will be deemed as duplicates, and will not be downloaded.

It will take longer, but the same scenario will occur if you have file numbering set to continuous.

There has to be more to it than that.

...Mike

[Kirk Baker]

Mike,

Photo Mechanic keeps a list of files that it has ingested from a specific disk.  Since you're ingesting a different disk (the CF card after the USB portable drive) this is why PM thinks it hasn't seen these files before.

This is kinda scary actually. Here is a scenario for you: I have my camera set to "Auto Reset" which will reset the file numbering each time a card is inserted. Now, lets say I insert a card and fill it, and ingest it...we'll call this group of files Set A.  I then delete Set A from the card, and fill it again (Set B).  Now the card is filled with files that have the same name as Set A, but are actually different files.  If all PM is doing is keeping a record of file names, then Set B will be deemed as duplicates, and will not be downloaded.

It will take longer, but the same scenario will occur if you have file numbering set to continuous.

As long as you re-format the card in your camera, then this scenario cannot occur.  The new list of files will not map to the old list of files even if they have the same names.

-Kirk

[Bill Kelly]

Hi Mike,

When we designed Incremental Ingest, we tried to eliminate any possibility that a newer photo could be skipped when it should have been copied.

In order to make Incremental Ingest fast, we can't perform any deep scanning of the image data, so we rely on a few file attributes able to be provided to us quickly by the operating system.  These are: the filename, the file size, and the file modification timestamp.

I would guess that these are probably the same attributes considered by Downloader Pro.

However, we felt it would be too risky to rely on these attributes alone.  As we know, the filename can repeat under the circumstances you mentioned above.  And also, the modification timestamp may be unreliable on some cameras.  Further, if one is shooting without compression, the file size will always be the same.

To guard against the potential unreliability of these attributes, we went a step further and create a file on the card called ".pmingest.dat" which helps us only skip files that have already been copied from that particular card.

This has the effect that photos on different cards are never considered duplicates of one another, and are always treated separately by Incremental Ingest.

Unfortunately, this means that Incremental Ingest is not compatible with your technique  of duplicating the photos onto a separate volume, and ingesting from both volumes.


Regards,

Bill

[Mike-D]

As long as you re-format the card in your camera, then this scenario cannot occur.  The new list of files will not map to the old list of files even if they have the same names.

-Kirk

So in order for incremental ingest to work, a card re-format is required every time a card is dumped?  This is the kind of thing that should be in the manual I think.

...Mike

[Mike-D]

Unfortunately, this means that Incremental Ingest is not compatible with your technique  of duplicating the photos onto a separate volume, and ingesting from both volumes.

I'm not sure how downloader pro does it either, but I do know that it can determine that a file has been downloaded already no matter the source, as doing so has allowed me to create this redundant download verification system in the first place. Whatever it's doing, works, and whatever PM is doing, at least in this scenario, doesn't.  I guess I'll have to stick to using it in parallels, which is a shame.

I would respectfully suggest that you find a new name for this feature, or at least fully document the many assumptions made in it's implementation.  As it is now, it seems that it's usefulness is quite limited. Comparing file names is dangerous, as PM itself renames files as they are downloaded. Comparing file sizes is pointless, because it is already acknowledged that PM modifies files as they are downloaded, which also changes the file time stamp.  In fact I can't think of any scenario that would cause this feature to reliably work, come to think of it.  Unless there is more to it than what has been said already.

...Mike