Indexing slow

[Kirk Baker]

Running now.

I can see the error messages in the DYN log - there are a number of video files within the library. As for bad data, I'm not aware of a large number of bad files in the library particularly with the most recent indexing. There are possibly a few though from the earlier years. Is this likely to cause a slowing down of indexing?

If for some reason they were to act like a roadblock (meaning a bug in PMPlus were to continuously load a bad image without moving on) then that would certainly be a problem.

I'll post the log shortly.

Great, thanks.

-Kirk

[Kirk Baker]

Matthew,

Do you have any system extensions installed that oversee network communication (packet analyzer, etc.)?

-Kirk

[Matthew]

I'm running iStat and have Backblaze installed but on pause during the indexing process.

[Kirk Baker]

Matthew,

OK, I don't know how much either of them interfere, even when paused, but sending image data (512KB) via TCP to another PMPlus process is taking 2 to 3 seconds which is incredibly slow.

On our systems it's nearly instantaneous.

-Kirk

[Matthew]

Interesting. I see this here. So this is likely iStat then? I did wonder why PM was using network bandwidth. Just assumed it was just normal...

I'll try and shut any background apps down and see if there is a speed improvement. Interestingly I've been using iStat since the first PM+ beta and haven't noticed a speed issue until now.

[Kirk Baker]

Matthew,

Interesting. I see this here. So this is likely iStat then? I did wonder why PM was using network bandwidth. Just assumed it was just normal...

I'll try and shut any background apps down and see if there is a speed improvement. Interestingly I've been using iStat since the first PM+ beta and haven't noticed a speed issue until now.

The Catalog system runs as a server on your local system.  Communication between PM and the server is via localhost TCP communication.

-Kirk

[Kirk Baker]

Matthew,

Could you please try the following?

First open a Terminal window (Applications/Utilities/Terminal.app).  Type: cd<space>  (press the space bar after you type 'cd')
Then in the Finder, find Photo Mechanic Plus.  Right-click on it and choose "Show Package Contents".  Then navigate to Contents/pmruby
Select and drag and drop the 'bin' folder onto your Terminal window.  Next, press enter.  Then copy and paste the following:

./pmruby -rsocket -e 'sv = TCPServer.new("127.0.0.1", 0); sv_port = sv.addr[1]; cl = TCPSocket.new("127.0.0.1", sv_port); svc = sv.accept; buf = ("x" * (1024**2)); nbuf=100; sz = buf.length * nbuf; t1=Time.now; [Thread.new {nbuf.times {cl.write(buf)}}, Thread.new {rsz=0; begin; rx=svc.read(65536); rsz+=rx.length; end while(rsz < sz)}].each {|th| th.join}; t2=Time.now; et=(t2-t1); puts "#{et} seconds, #{(sz/et)/(1024**2)} MB/sec"'

It should output the time and speed of your localhost transfer rate.  Please copy and paste that output in your reply.

Thanks,

-Kirk

[Matthew]

Thanks for your help with this.

I have removed iStat - no change. output below.

mh@incoming bin % ./pmruby -rsocket -e 'sv = TCPServer.new("127.0.0.1", 0); sv_port = sv.addr[1]; cl = TCPSocket.new("127.0.0.1", sv_port); svc = sv.accept; buf = ("x" * (1024**2)); nbuf=100; sz = buf.length * nbuf; t1=Time.now; [Thread.new {nbuf.times {cl.write(buf)}}, Thread.new {rsz=0; begin; rx=svc.read(65536); rsz+=rx.length; end while(rsz < sz)}].each {|th| th.join}; t2=Time.now; et=(t2-t1); puts "#{et} seconds, #{(sz/et)/(1024**2)} MB/sec"'
0.058828 seconds, 1699.8708098184538 MB/sec

[Kirk Baker]

Thanks for your help with this.

I have removed iStat - no change. output below.

mh@incoming bin % ./pmruby -rsocket -e 'sv = TCPServer.new("127.0.0.1", 0); sv_port = sv.addr[1]; cl = TCPSocket.new("127.0.0.1", sv_port); svc = sv.accept; buf = ("x" * (1024**2)); nbuf=100; sz = buf.length * nbuf; t1=Time.now; [Thread.new {nbuf.times {cl.write(buf)}}, Thread.new {rsz=0; begin; rx=svc.read(65536); rsz+=rx.length; end while(rsz < sz)}].each {|th| th.join}; t2=Time.now; et=(t2-t1); puts "#{et} seconds, #{(sz/et)/(1024**2)} MB/sec"'
0.058828 seconds, 1699.8708098184538 MB/sec

That's a good transfer rate.  We'll have to add some more logging code next week.  In the meantime, if you don't need proxies (for working with images that are offline, you can disable that in the Catalog Management window on a per-catalog basis.)

-Kirk

[Kirk Baker]

Matthew,

Could you please download and try this build? https://www.camerabits.com/download/PhotoMechanicPlusR5132_a75ecc6d.dmg

Thanks,

-Kirk

[Matthew]

Thanks Kirk.

Just installed and currently adding images from September to the library via the right click and Incude Items in Catalog option on a folder in the favourites.

Speeds for Image Preview Generation are around 3 per second. See attached.

[Kirk Baker]

Matthew,

Just installed and currently adding images from September to the library via the right click and Include Items in Catalog option on a folder in the favourites.

Speeds for Image Preview Generation are around 3 per second. See attached.

I no longer see the huge backlog of images remaining to be copied.  Preview generation speed was improved a couple of builds ago and 2.5 batches per second is quite good. (about 25 previews generated per second).

Does it seem better to you?  Would you mind running a larger Scan to Catalog to see if the Catalog Image Preview Copying ends up backlogging like before?

-Kirk

[Matthew]

I've just deleted the catalog and will start a scan on the whole library (circa. 350k images at 1.4TB) and report back.

[Kirk Baker]

I've just deleted the catalog and will start a scan on the whole library (circa. 350k images at 1.4TB) and report back.

Thank you, Matthew.

-Kirk

[Matthew]

Hi Kirk,

About 3.5 hours into this and currently showing the following.

Catalog Image Preview Copying speeds from 30 to up to 300/sec
Catalog Image Preview Generation steady at 4.2/sec
Catalog Metadata Updates steady at 2.8/sec

From my reading of this it looks like this should be complete in < 3 hours and if that's the case it's a massive improvement.

Will update you in the morning on the progress.