Crash during ingest, now importing duplicates

[vincentlepes]

Photo Mechanic Plus was doing a large ingest process yesterday which was nearly complete after hours of working. I left it to finish and went to bed. Today I returned to find an error that PM had crashed. So I went to try the ingest again, trusting it to sort out the duplicates and only copy over new files, as I have it set for incremental ingest. When I start the process today, it starts back at the beginning and tries to copy over the entire batch again.

What can I do to complete the ingest process without starting over and hoping there isn't another crash? I know crashes happen, but I expected PM to recognize the files it already ingested and just ingest what was remaining. Now I am left with quite the puzzle as my pre-ingest and post-ingest file structures are very different and so I can't just compare folder to folder.

I submitted the crash report recently if it helps. Thanks for any advice.

Edited to add a log file.

[ahoward]

Sorry that happened. From the log file it is hard to say why the crash happened. Though I did do some testing to try to recreate this and it does appear that at the moment, the Incremental Ingest record is updated when the Ingest completes or the user clicks the "stop" button on the Ingest Tasks window, or the Ingest simply fails but the program doesn't crash (in other words, if it does crash, it never gets the opportunity to update the incremental Ingest record). So, we will look into updating the incremental ingest record in real time rather than when the operation finishes, so that this situation can be avoided in the future. Before I get to a possible solution/workaround, I just want to confirm: Were Ingesting from a memory card, rather than attempting to Ingest the contents of an entire hard drive with the goal of building a catalog?

If you were trying to Ingest an entire hard drive to build a catalog, we would recommend against that, as you can use Scan To Catalog to build a catalog from a drive.

If you were Ingesting from a memory card, I would recommend doing the following to avoid Ingesting files that you have already Ingested:

First, open all of the folders that you Ingested to as a single contact sheet. If the Ingested created multiple folders, you can get the all to open as one contact sheet by holding the Option key as you double-click on the folder in the Photo Mechanic Navigator (or if they were all added to multiple folders but a single parent folder, you can right-click on that parent folder and choose "open folder and subfolders in new Contact Sheet"

Then, select all with Command+A and then go to Edit -> "Save Selection"  and save the text file to a place that is easy to find.

Close the contact sheet with the destination and then go to the memory card in the Navigator and right-click on that and choose "Open folder and subfolders in new Contact Sheet"

When that Contact Sheet opens, go to Edit -> "Load Selection" and then choose the file that you just saved out. Now, all items that were previously Ingested will be selected. Then, from the Edit menu, choose "Select Others (Invert)" and now you should essentially have a selection of items that were not Ingested. Now, go to the Ingest window and instead of Ingesting from "Disks", choose "From selection" and select "ingesting selected items from frontmost contact sheet" from the source list, and then perform the Ingest.

I hope this works for you and we will look into making Incremental Ingest update in real time so that this situation can be avoided in the event of a crash.

[vincentlepes]

Thank you, I will try your suggestions in the coming days and report back. The realtime incremental update sounds like a great fix, I hope it makes it in!

I have an "Incoming" folder on my active photos drive, where I just dump memory cards and then clone the drive immediately. I sort them into topical folders with short descriptive names based on the event or series they are part of. From there, I ingest in PM to name all the files with the date and folder name and sort them into folders, at which point I use the auto-catalog feature to add them into my catalog with nice readable names and structure. This workflow evolved from my not ingesting often enough and taking the time to name everything, so it helps me to keep a sort of queue this way so at least I get everything backed up before setting aside time to organize and add metadata.

My files start for example start as "Incoming/2023-04-14/Event Name/DSF005.jpg" and end up as "Ingest/2023/2023-02-15_Event_Name/2023-02-15_Event_Name_001.jpg" where the actual shoot date and folder name become the file name and structure and I let them be numbered in whatever sequence. So I'm not sure there is a 1:1 to work from anymore. In the ingest folder, I may be dumping a few SD cards with multiple days on them into one folder and letting the ingest process sort them into dates. This makes it easy to grab a month or year at a time to move them to separate archive drives over time, and also makes it easy to navigate when not in PM because the structure and filenames are descriptive.

Hopefully that answers your question--it's going from one folder on the same hard drive to another folder that is organized and at that point is also added to the catalog. Perhaps there is an easier way, I'm open to ideas. I've just been doing it like this for years in PM6 and for less time using PM+ so I could be missing opportunities to make it simpler.

[vincentlepes]

I didn't think the selection method outlined would work since all of the filenames are different, but after some pondering I found a solution. I used a third party application (Gemini) to detect duplicates between source and destination, setting it to trash all duplicates on the incoming side, which in theory will leave me with everything Photo Mechanic missed during ingest and not disrupt the catalog. My only concern is if the last file being transferred was corrupted when it crashed or something similar that I really don't know how I'd detect. I'm going to hold out on cloning the drive for a bit until I am confident everything made it through. I will run the ingest on what remains and sort out the remaining counter issues if needed later (since they won't be sequentially accurate now due to running a fresh ingest and not knowing where PM left off).

Anyway, I do like the realtime feature idea and hope that makes it in.

I have one followup question: is there any way to discover where the failed ingest may have left off? That way I might know where to start the counter and which files to check for peace of mind.

Thanks!

[ahoward]

I had posted a reply but deleted it because it contained inaccurate information about how the sequence variable works, so my apologies if you saw that and I misled you.

Really the only way to know that I can think of is going to be look at the files that got copied in their destination folders and then see what the highest sequence number that got assigned was. Unfortunately even if we were to do realtime updating of the Incremental Ingest record, this wouldn't change, as the sequence value is determined/set when you click Ingest so that you can begin a second Ingest while that one is occurring and you won't end up with your two ingests having an out of order sequence just because a file from the second Ingest copies before something from the first Ingest does.