Live Ingest performance - how does AP/Getty do this?

[DaveRe]

I'm running into an issue with the performance of a Live Ingest. I'm trying to develop a method whereby I can have a handful of photographers shoot using WiFi adapters on the camera, and transmit small JPEGs back to a central FTP server - and then use PM Live Ingest to rename the files, add IPTC data, and copy them to a new location for later use.

I shot a small event recently with a test config - just my two cameras and me as the sole photographer, transmitting back to my Macbook Pro. All images were saved on the local SSD. File stability monitoring was disabled. I shot the event over a course of 45 minutes. I shot a total of 1200 frames. All the files transmitted successfully to the laptop very quickly.

When the event finished, I returned to the laptop. Live Ingest was still working on images from the first 15 minutes of the event. I actually ingested all the RAW files off my cards, did my selects from the RAW files, and transmitted final images before Live Ingest finished what is essentially a file copy (with IPTC additions - something PM tends to do very quickly).

So - am I doing something wrong here? Is there a slow down in Live Ingest that prevents me from trying to use it this way?

In the end, this is similar in concept to what the AP and Getty were doing at the Olympics, but I have no idea what their actual workflow is - if someone has a clue, I'd love some more knowledge on it!

Thanks!

Dave

[Kirk Baker]

Dave,

How does LiveIngest access the photos on the FTP server?  Is that connection fast and efficient?

I'd like to see your settings.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your JPEG format screenshot.

Thanks,

-Kirk

[DaveRe]

Hi, Kirk,

Live Ingest settings attached. The laptop I'm on is the FTP server, too, right now, so it's local disk - just copying from one directory to another on a disk with plenty of space (several hundred GB).

The JPEGs I'm transmitting are 2592x1728 (Small JPEG on the 1DX) - they run about 1.7MB apiece - 2.23GB total. For reference, I just copied that directory to the same directory I was ingesting to. It took 10 seconds to copy the whole thing.

I failed to provide other pertinent info before. This is PM 5.0 build 16299, and Mac OS 10.10.2 on a Macbook Pro Retina, 16GB RAM, 750 GB SSD.

Thanks!

Dave

[attachment deleted by admin]

[DaveRe]

Kirk,

Did you have any thoughts after my last post? If need be, I'll write some Perl to do the function myself, but I'd definitely prefer if there were a way that PM would do this job for me.

Thanks, man!

Dave

[Kirk Baker]

We looked at your settings and they appear fine to us.  In order to get any insight into what is going on, you'll have to add some 'debug' flags and then do some testing.

You'll have to create a plain text file in your Photo Mechanic preferences folder.  On a Mac it is in the following location:

/Users/<your username>/Library/Preferences/com.camerabits.PhotoMechanic/

Create it there (the Finder will hide your Library folder by default, but you can use the Go To Folder command on the Go menu, enter ~/Library as the path) and name it PMDebug.txt and put the following lines in the text file:

CBFILE_IO_LOGGING ON
INGEST_LOGGING ON
COPY_MOVE_LOGGING ON

Then restart PM and attempt your Live Ingest workflow.  Then go to the Help menu and choose "Reveal Support Data..." and then post the zipped PM.log file here.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped log file.

Thanks,

-Kirk

[DaveRe]

Ok, Kirk, thanks. I'm shooting another event on Thursday night, so I'll gather this day during the gig.

[Kirk Baker]

Dave,

Ok, Kirk, thanks. I'm shooting another event on Thursday night, so I'll gather this day during the gig.

Those flags will result in a lot of data being logged.  Be sure to only turn them ON during your test and not prior to the test or performance will likely suffer.  Note that the settings only take effect on restart of Photo Mechanic.

-Kirk

[DaveRe]

You know what? I just noticed that I still had COPY_MOVE_LOGGING ON set from the permissions problem you helped me with a little while back. I wonder if that's interfering somehow. I'll test with the specified parameters below, and then run tonight with none of the debug stuff set to on, and compare/contrast.

[DaveRe]

Ok, performance was actually about the same - approx 2-3 seconds per small JPEG. When I ingested the full size RAW/JPEG combos, I got about 4 per second coming off of CF cards. PM_log.zip attached.

I also tried shutting off the Live Ingest, then restarted it with the copy pointed at a different directory, so basically PM just worked on the existing 500 or so images. That stayed about the same.

It never got as far behind as it did two weeks ago, but it still isn't performing very well.

Tonight, I'll shoot the event without the debugging stuff in place, and will report back how that goes.

[attachment deleted by admin]

[Kirk Baker]

Thanks for the log file, Dave.

From the Engineer who wrote Live Ingest:

One thing that appears over and over in the logfile:
live-ingest: failed all attempts to delete: /Users/ftponlyuser/15.3-test/BS7Rxxxx.JPG  <- various filenames
That's going to add a cost of at least +2 seconds per file (as failed delete delays for 0.25 sec for a max of 8 retries)

Is there a reason why PM can't delete the files it has transferred?  Permissions?  Some other process has the file open?

Solve that, and I expect things will become more speedy.

-Kirk

[DaveRe]

Hmm. That's interesting. The only reason is that we set it up with an "FTP only" user to allow the inbound FTP connection (without having to give my credentials out to everyone using it), but PM is running as me. So, I guess it has to delete files when it ingests them, not leave them in place?

I can work around that by simply running the FTP as whatever credentials are going to run PM - I just have to change a few things about how we're admin'ing the system. Since it's only me at the event today, I can simply log in as me, and do it that way. I'll set that up and report back post event.

Thanks!

Dave

[DaveRe]

Yeah, that definitely corrected it - it's blazing fast, now. I missed the note in the user manual about images being deleted when I first read through it, but I see it now.

[jwa]

DaveRe,
Where did you find the settings to allow it to delete the files?

[DaveRe]

jwa, PM does that by default with Live Ingest. It's a feature.