Ingest in the background?

[PerroneFord]

I am curious if the ingest process could be run in the background.  On today's high powered i7s and such, it seems like this process could be pushed to the background successfully.

Here is a common scenario for me, and a reason I'd like to do this.  Other sports or wire/event photographers may have similar needs:

I finish shooting a game, and go to the pressroom.  I now have 3 primary responsibilities.

1. Get metadata tagged, cropped, straightened photos to the editor ASAP.  For this, I typically pull in my locked files from the camera, use stationery pads and code replacements, and deliver via FTP or network connection

2.  Get the full take onto my clients server.  This is typically done via cloud services

3.  Get the full take onto my personal office server.  This is typically done later in the night but I'd PREFER it to happen at the same time as #2

If ingest can happen in the background, then I can fulfill both #2 and #3, while I work on #1.  In half an hour to an hour, my day is done.  Versus me driving home from the game, hooking up everything, and doing a second ingest from the home office. 

In a similar scenario, I could do an ingest in the background to a local drive as well as my client's server while I work on my photos to go out on the wire or similar.  This gives me a safety backup on the fly and would be a HUGE help.

From a volume perspective in shops where editors are getting cards in, an ingest could happen to the local storage as well as the backend server all while going through the cards and pulling temporary selects to go out to the media. 

Thanks for your consideration.

[Kevin M. Cox]

Perrone,

You may have the progress dialog set to pop up while you Ingest, but that should not keep you from continuing to work in PM at that same time. You can close, minimize or change the settings to not see the dialog at all. There is also an option to show the contact sheet right away or wait until the ingest is complete to have it opened.

[fotograhn]

This thread makes me interested. Is it possible to set up PM5 so that I can ingest completely in the background? In i silent way without any kind of interruptions?

I have noticed that if I'm working with images from a specific folder in PM5 and at the same time ingesting even more images to the very same folder, then I'm being slightly interrupted when the ingest starts.

To automate as much as I have found possible, I have made the following settings to PM5, right or wrong?

1. In the Ingest Dialog, I have checked "Auto Ingest", and "Unmount After Ingest".
2. In the Preferences General Tab, I have checked "Show Ingest" on mount of disk.
3. In the Preferences Contact Sheet Tab, I have checked "Automatically rescan folder content".
4. In the Ingest Tasks Dialog, I have unchecked "Always show this when ingest starts", and I did choose "hide tasks dialog when completed" in the drop down menu.

Are there any more settings I should switch, or have I misunderstood any of them mentioned above?

Thanks for all help!

[Kirk Baker]

This thread makes me interested. Is it possible to set up PM5 so that I can ingest completely in the background? In i silent way without any kind of interruptions?

I have noticed that if I'm working with images from a specific folder in PM5 and at the same time ingesting even more images to the very same folder, then I'm being slightly interrupted when the ingest starts.

To automate as much as I have found possible, I have made the following settings to PM5, right or wrong?

1. In the Ingest Dialog, I have checked "Auto Ingest", and "Unmount After Ingest".
2. In the Preferences General Tab, I have checked "Show Ingest" on mount of disk.
3. In the Preferences Contact Sheet Tab, I have checked "Automatically rescan folder content".
4. In the Ingest Tasks Dialog, I have unchecked "Always show this when ingest starts", and I did choose "hide tasks dialog when completed" in the drop down menu.

Are there any more settings I should switch, or have I misunderstood any of them mentioned above?

Thanks for all help!

That is as silent as you can make Ingest become.  There is one portion of Ingest (the scanning phase) that is modal and it cannot be avoided.

-Kirk

[PerroneFord]

Kevin,

Try this.

1.  Insert a card with a lot of images into your reader.  Maybe 500-1000 images.  If possible, lock 10-20 of those images in your camera before inserting it into the reader.  This is my typical scenario for a game.
2.  Open a contact sheet with that card so that you can see all the images.
3.  Begin an ingest dialog and select a place for the ingest to go.  This will replicate me ingesting those images onto my editors network drive.
4.  Try to go back to the contact sheet you opened in step 1 and continue working.  All you want to do is get to those 10-20 locked images, crop them and save or send them somewhere.

I'll be curious to know if your results mirror mine.

-P

Perrone,

You may have the progress dialog set to pop up while you Ingest, but that should not keep you from continuing to work in PM at that same time. You can close, minimize or change the settings to not see the dialog at all. There is also an option to show the contact sheet right away or wait until the ingest is complete to have it opened.

[Kirk Baker]

Perrone,

1.  Insert a card with a lot of images into your reader.  Maybe 500-1000 images.  If possible, lock 10-20 of those images in your camera before inserting it into the reader.  This is my typical scenario for a game.
2.  Open a contact sheet with that card so that you can see all the images.
3.  Begin an ingest dialog and select a place for the ingest to go.  This will replicate me ingesting those images onto my editors network drive.
4.  Try to go back to the contact sheet you opened in step 1 and continue working.  All you want to do is get to those 10-20 locked images, crop them and save or send them somewhere.

I take it that the problem is that PM won't let you do #4 while you are ingesting?

-Kirk

[PerroneFord]

Perrone,

1.  Insert a card with a lot of images into your reader.  Maybe 500-1000 images.  If possible, lock 10-20 of those images in your camera before inserting it into the reader.  This is my typical scenario for a game.
2.  Open a contact sheet with that card so that you can see all the images.
3.  Begin an ingest dialog and select a place for the ingest to go.  This will replicate me ingesting those images onto my editors network drive.
4.  Try to go back to the contact sheet you opened in step 1 and continue working.  All you want to do is get to those 10-20 locked images, crop them and save or send them somewhere.

I take it that the problem is that PM won't let you do #4 while you are ingesting?

-Kirk

Correct.  It says the disk is busy, and thus I have to wait for the system to ingest all files before I can get to the few I need.  This was my impetus for asking if the ingest could be backgrounded.  Perhaps a per file interlock so that if the system is ingesting in the background, it locks only the file it's on so that I don't inadvertently modify that file while the ingest process needs it.

[Kirk Baker]

Perrone,

1.  Insert a card with a lot of images into your reader.  Maybe 500-1000 images.  If possible, lock 10-20 of those images in your camera before inserting it into the reader.  This is my typical scenario for a game.
2.  Open a contact sheet with that card so that you can see all the images.
3.  Begin an ingest dialog and select a place for the ingest to go.  This will replicate me ingesting those images onto my editors network drive.
4.  Try to go back to the contact sheet you opened in step 1 and continue working.  All you want to do is get to those 10-20 locked images, crop them and save or send them somewhere.

I take it that the problem is that PM won't let you do #4 while you are ingesting?

Correct.  It says the disk is busy, and thus I have to wait for the system to ingest all files before I can get to the few I need.  This was my impetus for asking if the ingest could be backgrounded.  Perhaps a per file interlock so that if the system is ingesting in the background, it locks only the file it's on so that I don't inadvertently modify that file while the ingest process needs it.

That can't be avoided.  Ingest needs to have unobstructed access to the files and if you were to browse the card again and make changes while the file is being ingested, then data corruption would occur.  Ingest does run as a background thread, but having multiple threads reading/writing to a given file is very perilous.

-Kirk

[Kevin M. Cox]

2.  Open a contact sheet with that card so that you can see all the images.

So you are browsing images directly off the card I take it? This is something I never do to avoid the chance of corrupting images as Kirk mentioned.

Why not simply run the Ingest process set to only download tagged images so you can quickly get your locked files onto your laptop and begin working on them? If you want to then Ingest the others while you work on this first batch you could re-start the Ingest process with un-locked images and let it work in the background.

[PerroneFord]

Well, there's a few things going on here... more than meets the eye.  I'll try to lay it out.

For personal use:

First use, I am running anywhere from 3 to 8 cameras.  I've got about 15-20 minutes to get a set of selects out.  Sometimes less if I am filing at halftime.   Working off the card is really the only way to get the speed I need.  I am shooting backups to a second card in the camera, so I'm not too afraid of corruption issues. 

Second, it is a logistical nightmare to try and manage multiple cards for multiple ingests under deadline pressure.  Yes I can do it, but man is it a pain. 

Third, sometimes the image before of after my locked file is actually the better image.  I never get to see that if I can ONLY see my locked files.  Looking at the full take, even if I am not using them all, really has a lot of advantages. 


For institutional use:

When you are brining in 4 cards per shooter for 10+ shooters, and you are trying to get selects out to media outlets, it would be immensely helpful to ingest to your local working drives as well as long term storage drives while still being able to go through the images.  I realize this may be a need beyond what most users have, but I am trying to get my school to standardize on PM for ingest purposes.  During football season it would make ALL of our lives easier.  PM is superior to the system in place now, but this background ingest would be a wonderful help if it could be worked out.

Ah well, it was just a hope.  Maybe something to consider in the future...

2.  Open a contact sheet with that card so that you can see all the images.

So you are browsing images directly off the card I take it? This is something I never do to avoid the chance of corrupting images as Kirk mentioned.

Why not simply run the Ingest process set to only download tagged images so you can quickly get your locked files onto your laptop and begin working on them? If you want to then Ingest the others while you work on this first batch you could re-start the Ingest process with un-locked images and let it work in the background.

[PerroneFord]

Perrone,

That can't be avoided.  Ingest needs to have unobstructed access to the files and if you were to browse the card again and make changes while the file is being ingested, then data corruption would occur.  Ingest does run as a background thread, but having multiple threads reading/writing to a given file is very perilous.

-Kirk

When I programmed multi-user programs that needed to access common files, it was standard fare to use lock files to signal a program that a file could not be written by another process.  It would seem to me that since the ingest is basically doing a copy procedure, that the ingest process could look at the next file in the queue, lock that file for writing, perform the copy, then unlock the file.  This would take fractions of a second to no more than 2 seconds for most files and would be nearly transparent to the user.  Nearly all workgroup software does something similar.  In fact in some instances (like when I work off SD cards) the card is locked for writing anyway so having multiple programs reading the same file should pose no issue in the least.  In this instance we are talking about two processes simply accessing a file for READ ONLY, while the ingest process writes to an alternate location. 

Are you doing a CRC check post copy to verify the ingested file is a bit match for the original?  If so, then it seems that even if there was an issue, a flag would be thrown indicating the file copy needs to be done again.

Just thinking out loud here.

[Kevin M. Cox]

Obviously we all have different needs, it may just be a difference in volume here. I realize that what works for me might not work for you and vice versa. That said:

For example NFL with a 12 minute halftime. I generally have no trouble getting to the workroom, downloading all three cameras (multiple FW800 card readers) onto my laptop, editing and getting back to the field in time. I have PM setup so that when I insert a card it automatically starts the Ingest. I probably have 25% of the card downloaded by the time I could get the Contact Sheet of the card open to view directly. I personally don't think working off the card would save me that much time.

Maybe you could speed things up with faster and/or more card readers? It just seems like you're trying to figure out a way to do something that CameraBits recommends against when there might be other ways to speed up your process.

[PerroneFord]

Yea, those FW800 card readers... you can't get those now.  Sadly, I got caught in no-man's land with my 2012 MBP purchase.  No USB3, 1 FW800, no FW800 readers available.  I got that Thunderbolt port though... WOOHOOO... and no reader EVER made for it.  Things are easier at home where I've worked around this problem with that Belkin Expressdock and the Lexar Workflow thing so everything is much faster.

I didn't quite mean for this to turn into the show it has.  It was just meant to be a basic question with a yes we can/no we can't answer.  I'm always looking for ways to speed up my workflow, and that pending $3k laptop upgrade this year is looking like my best bet.

Oh well.

(As an aside Kevin, what team(s) are you covering?)

Obviously we all have different needs, it may just be a difference in volume here. I realize that what works for me might not work for you and vice versa. That said:

For example NFL with a 12 minute halftime. I generally have no trouble getting to the workroom, downloading all three cameras (multiple FW800 card readers) onto my laptop, editing and getting back to the field in time. I have PM setup so that when I insert a card it automatically starts the Ingest. I probably have 25% of the card downloaded by the time I could get the Contact Sheet of the card open to view directly. I personally don't think working off the card would save me that much time.

Maybe you could speed things up with faster and/or more card readers? It just seems like you're trying to figure out a way to do something that CameraBits recommends against when there might be other ways to speed up your process.

[Kirk Baker]

Perrone,

That can't be avoided.  Ingest needs to have unobstructed access to the files and if you were to browse the card again and make changes while the file is being ingested, then data corruption would occur.  Ingest does run as a background thread, but having multiple threads reading/writing to a given file is very perilous.

When I programmed multi-user programs that needed to access common files, it was standard fare to use lock files to signal a program that a file could not be written by another process.  It would seem to me that since the ingest is basically doing a copy procedure, that the ingest process could look at the next file in the queue, lock that file for writing, perform the copy, then unlock the file.  This would take fractions of a second to no more than 2 seconds for most files and would be nearly transparent to the user.  Nearly all workgroup software does something similar.  In fact in some instances (like when I work off SD cards) the card is locked for writing anyway so having multiple programs reading the same file should pose no issue in the least.  In this instance we are talking about two processes simply accessing a file for READ ONLY, while the ingest process writes to an alternate location. 

Are you doing a CRC check post copy to verify the ingested file is a bit match for the original?  If so, then it seems that even if there was an issue, a flag would be thrown indicating the file copy needs to be done again.

No. Depending on what you're doing (applying IPTC Stationery Pad) the file contents can be fairly certain to be different.

File locking does not work on all devices or file systems.

-Kirk

[fotograhn]

Yea, those FW800 card readers... you can't get those now.  Sadly, I got caught in no-man's land with my 2012 MBP purchase.  No USB3, 1 FW800, no FW800 readers available....

What about this one!? Available and affordable. I'm using it myself and I'm very pleased.
http://www.amazon.co.uk/gp/aw/d/B004OXV2ZQ