Getting duplicate files using ingest

[schlotz]

Scenario: as half-time approaches I pull a card from my two bodies and ingest.  Pick out 5-10 specific shots to send to the pressbox. Cards go back into the cameras (all files intact). At the end of the match each card is re-inserted and ingested.  Note: the Incremental Ingest: copy new photo only box IS CHECKED. The ingest process this time creates duplicates of all the first half shots. That is somewhere around 300-400 duplicates.  I'm under deadline at this juncture and this has created a real issue for me.  Not only does the end of game ingest take a long time (with the 1st half photos which it should have bypassed via the CHECKED box) but I cannot begin to review them until they all have ingested and I've spent the time deleting the duplicates. Attached is a screen shot of my ingest window (ignore the disks being shown).  With the results I just described it appears the check box function is NOT WORKING.  On the other hand I find it hard to believe this hasn't come up before so it's probably me doing something wrong. How do I stop PM from re-ingesting files it has already done, i.e. by creating 'A' versions of them?

Matt

[Hayo Baan]

Are you perhaps ingesting on a different computer/as a different user?

[schlotz]

Nope, it's my laptop. Use it field side and then after the game.

[Kirk Baker]

Matt,

Scenario: as half-time approaches I pull a card from my two bodies and ingest.  Pick out 5-10 specific shots to send to the pressbox. Cards go back into the cameras (all files intact). At the end of the match each card is re-inserted and ingested.  Note: the Incremental Ingest: copy new photo only box IS CHECKED. The ingest process this time creates duplicates of all the first half shots. That is somewhere around 300-400 duplicates.  I'm under deadline at this juncture and this has created a real issue for me.  Not only does the end of game ingest take a long time (with the 1st half photos which it should have bypassed via the CHECKED box) but I cannot begin to review them until they all have ingested and I've spent the time deleting the duplicates. Attached is a screen shot of my ingest window (ignore the disks being shown).  With the results I just described it appears the check box function is NOT WORKING.  On the other hand I find it hard to believe this hasn't come up before so it's probably me doing something wrong. How do I stop PM from re-ingesting files it has already done, i.e. by creating 'A' versions of them?

In order for PM to track what files it has ingested from a particular card, it must be able to write a file on the root of the card named ".pmingest.dat" and it must be able to create a file in its cache on your hard drive that matches the ID contained in the .pmingest.dat file.  If these operations fail, PM will fall back and err on the side of not skipping any files.

If you still have the card and you haven't formatted it, we can check for the file.  Open up the Terminal application and enter the following:

ls -al

(type a space after the -al).  Then drag and drop the camera card from the Finder onto the Terminal window.  (the path to the disk will be entered for you.)  Click back into the Terminal window and press the enter/return key.

Copy and paste the output in your reply.

-Kirk

[schlotz]

Hi Kirk,

Ahh, so that's how it keeps track.  Unfortunately both cards in my 1DX have already been formatted. Stranger yet I ran a test this morning, took 2 shots, ingested, closed out PM. Card back into 1DX, took two more shots, verified the .dat file was still present (it was), ingested again and no duplicate 'A' files were created.  So, what ever I was doing must have somehow messed this up but running the process as I described the dupes do not happen.  Really hate when I do things like this but the good news, it's now working like it should.  Only thing I can think of is that during the half time process, I was using the copy/move function inside of PM to place selected tagged photos to a sub-folder on the laptop. Could that have caused it?

Matt

[Hayo Baan]

If you copied/moved them from your memory card, that would certainly explain what happened to you. In that case you didn't use ingest and PM did not know what was ingested when next you did use ingest on the card.

Another explanation could have been a full memory card, but that wasn't the case, I think as you took more shots afterwards.

[Kirk Baker]

Matt,

Ahh, so that's how it keeps track.  Unfortunately both cards in my 1DX have already been formatted. Stranger yet I ran a test this morning, took 2 shots, ingested, closed out PM. Card back into 1DX, took two more shots, verified the .dat file was still present (it was), ingested again and no duplicate 'A' files were created.  So, what ever I was doing must have somehow messed this up but running the process as I described the dupes do not happen.  Really hate when I do things like this but the good news, it's now working like it should.  Only thing I can think of is that during the half time process, I was using the copy/move function inside of PM to place selected tagged photos to a sub-folder on the laptop. Could that have caused it?

Definitely.  The Copy/Move Photos feature does not log what photos have been copied/moved and as such, Ingest won't know what you copied/moved.

-Kirk

[schlotz]

Hi Kirk,

Still a little foggy on the copy/move scenario.  What throws me into confusion regarding your response is that you indicate PM does not log when a user runs its copy/move process and ingest does not know what has been copied or moved. With that, then ingest would have to react to what is already in the folder it is getting ready to put photos into. So I'm figuring its logic goes something like: If file exists and Incremental is off then make "A" version.  If file exists and Incremental is checked then bypass, otherwise place copy of photo into folder.

I ingested the first half of the game into the folder of /yyyy/mm/dd on my macbook at which time the card was automatically ejected.  I then used the 'copy/move' function of the PM command under "file" (the move checkbox was left unchecked, ergo a copy was done) to place the tagged few in a subfolder labeled picks.  At that point all of the photos still resided in the folder mentioned above so I'm not clear how this process would force the second ingest of this card to create "A" versions of all the first half photos since the Incremental box was checked.  If the 'move' box checked scenario was followed instead, I would think the second ingest would just place the same photos back into the folder and not use the "A" version since the original ones were no longer there. Seems to me PM was doing something more as it appeared to understand that it had previously seen the first half files, ignored the Incremental Ingest check box and automatically created "A" versions of the first half photos.

I'm probably spending too much time trying to figure this out especially since my memory of past events is less clear now than a few weeks ago when it last occurred. I want to believe I had done this both ways (copy & move) in previous games and regardless of which used, got the duplicated files. Sorry to keep bothering you, I just want to make sure I've got this understood so I don't run into it again.  Publishing deadlines tend to naw at me a bit and I need PM to consistently come through for me, i.e. make sure I don't screw myself up by doing it wrong. 

Appreciate your time.

Matt

[FVlcek]

As I understand it, you ingested first half of the game with Incremental Ingest unchecked, and second half of the game with it checked? I assume that PM saves where the last ingest ended only when the option is checked from the start. So you should have kept it checked even for the first half of the game.

I am not sure if that is the behaviour, although it would make sense to me.

[fabianlujan]

How is the file name setting on the 1DX?
Is it auto reset every time you insert a card? Continuos?

[Kirk Baker]

Matt,

Still a little foggy on the copy/move scenario.  What throws me into confusion regarding your response is that you indicate PM does not log when a user runs its copy/move process and ingest does not know what has been copied or moved. With that, then ingest would have to react to what is already in the folder it is getting ready to put photos into. So I'm figuring its logic goes something like: If file exists and Incremental is off then make "A" version.  If file exists and Incremental is checked then bypass, otherwise place copy of photo into folder.

Ingest consults its log to know what files have been copied from the source volume.  It does not care at all what is at the destination.  Since the Copy/Move feature does not update Ingest's log as to what it has copied/moved, Ingest won't know what you've done in the Copy/Move feature.

I ingested the first half of the game into the folder of /yyyy/mm/dd on my macbook at which time the card was automatically ejected.  I then used the 'copy/move' function of the PM command under "file" (the move checkbox was left unchecked, ergo a copy was done) to place the tagged few in a subfolder labeled picks.  At that point all of the photos still resided in the folder mentioned above so I'm not clear how this process would force the second ingest of this card to create "A" versions of all the first half photos since the Incremental box was checked.  If the 'move' box checked scenario was followed instead, I would think the second ingest would just place the same photos back into the folder and not use the "A" version since the original ones were no longer there. Seems to me PM was doing something more as it appeared to understand that it had previously seen the first half files, ignored the Incremental Ingest check box and automatically created "A" versions of the first half photos.

If a file exists in the destination location with the same name as a file being copied, the new file is renamed (according to your preferences it can append a letter, or digits) to avoid overwriting the file that already exists.

So if you were to Copy the same set of photos to the same location three times you would get something like this:

DSC_0001.JPG
DSC_0001A.JPG
DSC_0001B.JPG

The same thing would happen if you used Copy and then Ingest, followed by another Copy.  If a file exists in the destination location when copying/moving a file then the new file is renamed to avoid the collision.

HTH,

-Kirk

[schlotz]

I might not be communicating what I've been doing clear enough.  In no case have I been using an external app (Finder) to 'copy' files. I use PM-Ingest to grab all the files on my CF cards and put them on my hard drive. Next I use the file menu (at the top) within PM to select 'copy/move' in order to place a copy of tagged files to a subfolder.  See attached

[Hayo Baan]

This was clear to us all the time I think
This is still moving/copying though, not ingesting. Only when you use PM's ingest function will it be able to keep track of the files you got from your card and reingest them a second time. Mixing ingest and copy/move will almost always produce the duplicates you find you're having.

[schlotz]

Guess I have to stop there and just accept it.  At least my tests yesterday resulted in not creating duplicates so I'll be sure to strictly follow that process going forward.
Thanks for the help guys!

Matt