Issue with PM ingest with second card right after a game

[schlotz]

Here is the scenario:

On the field shooting and stop to ingest and find 5-10 key photos to be handed over to a runner on a thumb for the press box.  As this occurs, a big play develops, so you pick up your rig with the second card still in the camera and shoot.  Note: camera is set to record to BOTH cards for redundancy. Fast forward to end of the game and back in the press box. You re-ingest your first card (duplicate detection on so only the newer ones come across) BUT.... there are those special captures right before half-time sitting on card #2 you need. So you pop in card 2 and start to ingest.  BIG MISTAKE - PM does not recognize any of card 2 as being previously ingested and since most of the file names are the same it goes about renaming them and you have to wait for a full ingest only to find massive duplicates which you have to spend painstaking time to sort through in order to locate those few shots. Not to mention the lost time this creates when you are under deadline crunch.  ARGH!!!

Short solution (which I don't like) is to disable dual writing to both cards thus eliminating the redundancy factor which is pretty important.

Anybody have a better solution? 

[Hayo Baan]

Hmm, tough one. Some things I can think off:

* To speed up the full ingest, get a faster card reader (and make sure your computer can cope with the added speed too…).

* Assuming your images are named ending in numbers and your duplicates are named with letters, finding duplicates is quite easy. You could e.g., sort on the last letter in the filename (use a custom sort of: {filenamebase:-1}). Personally I'd use the command-line to get rid of the duplicates though (but you need to feel comfortable with that, of course)

Hope this helps,
Hayo

[schlotz]

Yup already using fast usb 3.0 readers.  Yes, finding dupes since they are all ending in a letter isn't difficult but it forces you to WAIT for the entire ingest to get done.  This is a waste of time especially when you have 700-900 photos on the card. Attempting a first pass review and selection with every other photo being a dupe is also a major PITA. Haven't spent any time with command lines. So far I just go into the OS, do the search followed by a mass delete but again, it shouldn't be necessary.  PM should allow filtering out dupes BEFORE it ingests.  Yes I know it does this but that's only on the same card.

It's somewhat hard to believe others haven't been bothered by this issue.

A simple check box in the ingest window for:  Ignore duplicate files from a different card would solve the issue.

[Kirk Baker]

Yup already using fast usb 3.0 readers.  Yes, finding dupes since they are all ending in a letter isn't difficult but it forces you to WAIT for the entire ingest to get done.  This is a waste of time especially when you have 700-900 photos on the card. Attempting a first pass review and selection with every other photo being a dupe is also a major PITA. Haven't spent any time with command lines. So far I just go into the OS, do the search followed by a mass delete but again, it shouldn't be necessary.  PM should allow filtering out dupes BEFORE it ingests.  Yes I know it does this but that's only on the same card.

It's somewhat hard to believe others haven't been bothered by this issue.

A simple check box in the ingest window for:  Ignore duplicate files from a different card would solve the issue.

There is another thread where the user had a similar issue.  Unfortunately it is not an easy problem to solve and have it be both fast and foolproof.  To make it foolproof, Ingest would have to read the entire file and produce a checksum and then compare against a list of checksums to know if the image had ever been seen before.  This would slow things down significantly when a file is rejected but not copied.  (When a file is copied anyway, then the impact is far less severe.)

The way Incremental Ingest works now is that it puts an ID file on your card and then uses that ID to look up filesystem information (name, date, file size) to determine if a file has already been ingested.  Very fast and foolproof for an individual card, but doesn't work when a card with the same files is presented to Ingest (the ID on the card will differ and a different set of data will be consulted.)

Currently Incremental Ingest is fast and foolproof for individual cards.  The other method would work among multiple cards but while it would be foolproof, it would be (possibly) slower to the point that some may complain why it takes so long for Ingest to copy only the new pictures.

Is it guaranteed that the files on the secondary card are identical in every way (filename, file size, modification date/time) or can this vary?

-Kirk

[schlotz]

Well, you can set it to be identical.  Then again you can also record raw to one and jpg to the other. Understand the approach taken, but it does create a problem for those of us on deadlines, wanting the safety of a dual card system while needing the ability to operate in the scenario I mentioned above.  From my perspective, checksums under the condition described are not necessary.  Matching filenames.ext is sufficient.  Wondering now if cloning the ID file '.pmingest.dat' to the second card might work  

At the end of the day: I would rather CameraBits recognize the scenario as an operating workflow for sports photographer's and provide a way to ignore duplicate files from a different card.

Matt

[schlotz]

Kirk,

I would still like CameraBits to come up with a solution to this problem that will allow it's customers to select a special case option to Ignore duplicate files from a different card.  It can be turned off by default AND it could be coded to force a reset to that default upon exiting PM as well.  IMO, simple filename checking is all that is required and would not overly extend the ingest process.
Please have the dev's consider this option.

Thank you!
Matt

[Kirk Baker]

Matt,

I would still like CameraBits to come up with a solution to this problem that will allow it's customers to select a special case option to Ignore duplicate files from a different card.  It can be turned off by default AND it could be coded to force a reset to that default upon exiting PM as well.  IMO, simple filename checking is all that is required and would not overly extend the ingest process.
Please have the dev's consider this option.

We may end up doing this, but it won't be for PM5.  PM6 is in active development and PM5 is in maintenance.

Can you tell me the purpose you're trying to solve by shooting to two cards simultaneously?  If it is for backup purposes then why aren't you changing out both cards simultaneously.  Leaving one card in the camera and continuing to shoot defeats that purpose.

-Kirk

[schlotz]

Kirk,

On field action sports photography is very fast paced.  When approaching half time you are editing right there on the sideline while keeping an eye on the game.  If something starts to develop you immediately put down the laptop, pick up the camera and start shooting.  Trying to manage 2-3 additional sets of cards (depending whether you are working with a 2 or 3 body rig) involves removing both, putting one from each camera in a card reader while the others then must be stored. Next, having to obtain new dual sets of cards and loading them in each camera. Of course, those in the readers have to be stored as well. All of this takes away precious time at a point where you have the least of it available.  At best it certainly complicates a process that requires being highly streamlined. 

If I were to ignore the time constraint and complication issues, running with duplicate sets does allow the backup scenario to remain intact but the deadline constraints of half time take precedence.  With the exception of a handful of photos potentially captured right before half time the backup using the 2 card process remains viable. The next deadline is end of game. No time is available to re-ingest 95% of the same photos that are on card 2, not to mention the immediate culling process of duplicates first so you don't have to review them while using PM. Ergo my fervent request for ignoring duplicate files from a different card.

Can certainly understand that PM5 is in maintenance.  Putting this solution into PM6 is the way to go.  Please push for this to be included.

Thank you,

Matt

[Kirk Baker]

Matt,

On field action sports photography is very fast paced.  When approaching half time you are editing right there on the sideline while keeping an eye on the game.  If something starts to develop you immediately put down the laptop, pick up the camera and start shooting.  Trying to manage 2-3 additional sets of cards (depending whether you are working with a 2 or 3 body rig) involves removing both, putting one from each camera in a card reader while the others then must be stored. Next, having to obtain new dual sets of cards and loading them in each camera. Of course, those in the readers have to be stored as well. All of this takes away precious time at a point where you have the least of it available.  At best it certainly complicates a process that requires being highly streamlined. 

If I were to ignore the time constraint and complication issues, running with duplicate sets does allow the backup scenario to remain intact but the deadline constraints of half time take precedence.  With the exception of a handful of photos potentially captured right before half time the backup using the 2 card process remains viable. The next deadline is end of game. No time is available to re-ingest 95% of the same photos that are on card 2, not to mention the immediate culling process of duplicates first so you don't have to review them while using PM. Ergo my fervent request for ignoring duplicate files from a different card.

Can certainly understand that PM5 is in maintenance.  Putting this solution into PM6 is the way to go.  Please push for this to be included.

Understood.  Is there some way to set up your camera to shoot only on the first card when present and only shoot to the second card when either the first card is full or is not present?  If so, then the complication of swapping two cards would be removed and the problem with Ingest would also be solved.

I suggest this as an interim solution.

-Kirk

[schlotz]

Kirk,

Appreciate the reply.  Yes (from an interim perspective) there is a way via settings but of course the backup functionality is lost which is what I'm wanting to maintain.  It's a bit of a gamble when shooting for hire to have all your eggs in one basket (so to speak).  If the card or camera takes a crash everything captured could be lost. Just not my first choice.

Thanks,
Matt