Author Topic: Speeding up Ingest on 10 Gbit network  (Read 707 times)

Offline pdizzle

  • Newcomer
  • *
  • Posts: 26
    • View Profile
Speeding up Ingest on 10 Gbit network
« on: August 21, 2023, 02:19:31 PM »
Is there anything that can be done to speed up the ingest in PM Plus over the network?  If I use PM to ingest, I top out around 1 Gbps.  If I use windows explorer and copy the files manually from the CFExpress card, I get up to around 5 Gbps.

Obviously PM is doing more than a straight copy so it isn't the most fair comparison, but seems like there is room for improvement.  Are there any settings I can tweak?

Offline Bob Russell

  • Camera Bits Staff
  • Sr. Member
  • *****
  • Posts: 269
    • View Profile
    • Camera Bits, Inc.
Re: Speeding up Ingest on 10 Gbit network
« Reply #1 on: August 21, 2023, 03:35:27 PM »
Unfortunately there are not any options to increase the copy speed to networked volumes. If you have Ingest set to Open Contact Sheets after Ingest, then this will save some time since PM won't be generating or sorting the thumbnails until the copy process is done. This menu option is at the lower right part of the Ingest dialog.

Best regards,

--Bob

Offline pdizzle

  • Newcomer
  • *
  • Posts: 26
    • View Profile
Re: Speeding up Ingest on 10 Gbit network
« Reply #2 on: August 21, 2023, 03:51:42 PM »
Hi Bob, I do have that option set.

I did some tests with a local drive and it topped out at about 2.5 Gbps, doubling my ingest to the network drive.  Why is PM better with Local drives than networked drives? PM can clearly go faster (although still not fast as a windows copy, which topped out at 6 Gbps on the local drive and 5 Gbps on the network drive).  The bandwidth is there, PM just can't utilize all of it. 

Would this be a feature request? Maybe PM could do some sort of multi thread copy to increase throughput?

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24818
    • View Profile
    • Camera Bits, Inc.
Re: Speeding up Ingest on 10 Gbit network
« Reply #3 on: August 21, 2023, 05:19:51 PM »
I did some tests with a local drive and it topped out at about 2.5 Gbps, doubling my ingest to the network drive.  Why is PM better with Local drives than networked drives? PM can clearly go faster (although still not fast as a windows copy, which topped out at 6 Gbps on the local drive and 5 Gbps on the network drive).  The bandwidth is there, PM just can't utilize all of it. 

Would this be a feature request? Maybe PM could do some sort of multi thread copy to increase throughput?

It can't be multi-threaded in the case where metadata is being applied/folders being created/sequence numbers used in naming/etc. due to the sequential nature of such operations.

Why is it slow on a networked drive?  Because of the pattern of writes (depending on the file type and your IPTC/XMP metadata settings) can be small chunks of data, and network protocols having large overhead per write operation.

Could it be sped up?  Perhaps if we did all of the writing to memory and then copied in one big chunk, it would speed up a bit, but there is still overhead in parsing the metadata of the file, expanding variables, serializing metadata, etc.  It's a lot less work to flat copy a file than what PM's Ingest does.

Feel free to make a feature request.

-Kirk

Offline pdizzle

  • Newcomer
  • *
  • Posts: 26
    • View Profile
Re: Speeding up Ingest on 10 Gbit network
« Reply #4 on: August 21, 2023, 07:41:11 PM »
Will do, I have a couple ideas that I'll throw into the feature request.