catalog metadata gathering NAS

[webmstrk9]

I'm new to the Photo Mechanic world and jumped in feet first as I was looking for a cataloging system. I had used Lightroom in the past but as my LR catalog grew, performance started degrade, the same can be said about CaptureOne with their catalog.

So I'm in the process of my 7th attempt to scan my NAS for my catalog, but have noticed the "catalog metadata gathering" would want to process at less than 1/sec, which would equate to approx 17+ estimated days to complete. I would come back a few hours later and it would be processing at 1.3/sec with an adjusted ETC, but return later, it would drop to .3/sec.

Do I just need to let it run its course or can I add another hamster to the wheel? I've adjusted the priority to 10 for this with the above results.

Thanks in advance

[Kirk Baker]

I'm new to the Photo Mechanic world and jumped in feet first as I was looking for a cataloging system. I had used Lightroom in the past but as my LR catalog grew, performance started degrade, the same can be said about CaptureOne with their catalog.

So I'm in the process of my 7th attempt to scan my NAS for my catalog, but have noticed the "catalog metadata gathering" would want to process at less than 1/sec, which would equate to approx 17+ estimated days to complete. I would come back a few hours later and it would be processing at 1.3/sec with an adjusted ETC, but return later, it would drop to .3/sec.

Do I just need to let it run its course or can I add another hamster to the wheel? I've adjusted the priority to 10 for this with the above results.

That seems awfully slow.  I wouldn't expect the speed of the batches to get significantly slower over time.  Is the speed of the NAS variable in general?

-Kirk

[webmstrk9]

The NAS is about 4 ft away.

On average, I would transfer 30GB worth of data in about 30-45 minutes. I don’t work directly off the NAS, it’s my archival destination.

[webmstrk9]

Checked now, it's at 2.2/sec, estimated 1.7 days

Photo Mechanic Plus Version 6.0, build 5376 (14f9534)

[Kirk Baker]

Checked now, it's at 2.2/sec, estimated 1.7 days

Photo Mechanic Plus Version 6.0, build 5376 (14f9534)

That's quite a bit more reasonable.  Did anything change?

-Kirk

[webmstrk9]

nothing changed, I came in this morning and now its .5/s ETC 7.3 days

[webmstrk9]

Just to add, I've been actively working on my MBP doing my 9-5, the "throughput" hasn't changed in the last hour

.4/sec ETC 9 days

[obelix74]

I have fought with my NAS for about 2 months now.  I have not built a successful catalog yet.  I threw in the towel and bought a 12TB USB HDD, I will copy the files over to that from the NAS and build the catalog against it.

[webmstrk9]

This is what I’m hoping to avoid, we’ll see in a few days if it finishes....

One thing I noticed is the “kernel_task” is quite busy, outbound network traffic is on average 12/mb, which was not the case before the scan

[obelix74]

What OS are you running? How is the NAS mounted? What protocol?

I run OSX and both Catalina and Big Sur are pretty bad with Samba mounts. Mac has deprecated AFS and SMB is bad. There were a few bugs in PM plus early on not dealing with disconnecting NAS drives but Kirk is phenomenal in support. I bought a license of PM plus before the discount ran out only because of him. I was confident he will fix the bugs and PM plus solves a real need for me.

There’s an app called AJA system lite on Mac that does testing of large files read and write on macs.

I get 3 MBps read on 802.11n 5GHz WiFi connected at 500Mbps and 28Mbps read when wired through GigE to the same Google wifi router. I have a ReadyNAS 104 and a 2018 maxed out Mac mini running the catalog on its primary SSD.

My NAS runs Seagate SATA 6Gb/s 3.5-Inch 4TB Desktop HDD (ST4000DM000) * 4. It’s 5900 rpm. But it’s my archival system.

My earlier machine was a custom built PC with multiple hard disks internally and not anymore.

When I moved to wired, I was able to build the catalog of 140,000 images in 3 days. It was about 10x faster but I still have one more bug with proxy building that Kirk is working on.

I thought long and hard about it. I only have two copies of images now. One in the NAS and one on the cloud (back blaze). Short of getting a NAS / DAS combination, a 12 TB desktop USB HDD provided me another local copy and quicker browsing. It was the least costly thing I could do so that helped. The HDD should arrive soon.

From what I read and benchmark, for reading large files, wifi is still very bad. PM plus has to read all the files entirely to build proxies so it’s going to be slow.

Before PM plus, I wrote my own indexing software using open source tools. I was remotely building thumbnails off the network and it took me about 20 days to just build thumbnails and copy down metadata. I started writing a search engine around it using elastic search and custom GUI but the effort involved and incremental builds of catalogs and maintaining them was a real pain. $229 was a bargain for that .

I could never get a catalog built for weeks on wifi

[webmstrk9]

Thank you for the insight! I went in blind with PM+, which is fine, I was going to purchase PM, with PM+, the additional functions/features were the added bonus.

Catalina 10.15.7
NAS via SMB
QNAP, RAID 1, dual WD RED NAS 8TB 5400 RPM drives - Archival as I have an external SSD RAID for my sandbox
2019 MBP 32GB i9 2.4GHz

Like yourself, I have been considering purchasing a 14TB external USB drive, but had missed the Black Friday sale, so I'll wait. I'll wait till PM+ crashes before I do a performance test again with BlackMagic, if hardwired is the better option, most of the time it is, I'll go that route.

What OS are you running? How is the NAS mounted? What protocol?

I run OSX and both Catalina and Big Sur are pretty bad with Samba mounts. Mac has deprecated AFS and SMB is bad. There were a few bugs in PM plus early on not dealing with disconnecting NAS drives but Kirk is phenomenal in support. I bought a license of PM plus before the discount ran out only because of him. I was confident he will fix the bugs and PM plus solves a real need for me.

There’s an app called AJA system lite on Mac that does testing of large files read and write on macs.

I get 3 MBps read on 802.11n 5GHz WiFi connected at 500Mbps and 28Mbps read when wired through GigE to the same Google wifi router. I have a ReadyNAS 104 and a 2018 maxed out Mac mini running the catalog on its primary SSD.

My NAS runs Seagate SATA 6Gb/s 3.5-Inch 4TB Desktop HDD (ST4000DM000) * 4. It’s 5900 rpm. But it’s my archival system.

My earlier machine was a custom built PC with multiple hard disks internally and not anymore.

When I moved to wired, I was able to build the catalog of 140,000 images in 3 days. It was about 10x faster but I still have one more bug with proxy building that Kirk is working on.

I thought long and hard about it. I only have two copies of images now. One in the NAS and one on the cloud (back blaze). Short of getting a NAS / DAS combination, a 12 TB desktop USB HDD provided me another local copy and quicker browsing. It was the least costly thing I could do so that helped. The HDD should arrive soon.

From what I read and benchmark, for reading large files, wifi is still very bad. PM plus has to read all the files entirely to build proxies so it’s going to be slow.

Before PM plus, I wrote my own indexing software using open source tools. I was remotely building thumbnails off the network and it took me about 20 days to just build thumbnails and copy down metadata. I started writing a search engine around it using elastic search and custom GUI but the effort involved and incremental builds of catalogs and maintaining them was a real pain. $229 was a bargain for that .

I could never get a catalog built for weeks on wifi

[obelix74]

Ah thanks. What’s your SSD RAID? I have a similar species work MacBook Pro not bought a mini for photo editing. I switched to capture one pro for editing after Adobe went all subscription but PM+ has taken over ingestion now and I use C1 as just the editor.

I have a 500GB SSD where the catalog lives. I have a 2TB SSD on USB-C where photos I process live, this has my last six months photos. The 12TB WD elements HDD (arrived today, copying files now) will become my primary volume to contain *all* photos and NAS becomes archival.

I had the same experience as you for LR catalog (very slow). My teenage daughters often simply browse through old photos. PM+ will be awesome once the catalog is built.

I have no question in my mind that your catalog will be insanely fast if you wire it. Maybe you can do it just once for a day or two.

[webmstrk9]

Years ago, I purchased this enclosure and stuck 2 1TB SSD's in them, I have another RAID enclosure with 2 500GB SSD's as well....You can say I have storage coming out of my ears....

https://www.bhphotovideo.com/c/product/1496991-REG/oyen_digital_cb3r3_sl_minipro_raid_enclsoure_silver.html

I should plan on moving/selling the dual 500GB SSD RAID as it's been sitting since I've setup the NAS.

Both RAID enclosures require power, so, I have a Samsung T5 1TB for my on the road backup to the local SSD to the MBP

Looking at the current scan of the catalog in PM+, as of now, as it's still scanning...

2017 - 86k
2018 - 53k
2019 - 41k
2020 - well... haha

EDIT: Additional Info, I just hardwired my network

[obelix74]

Thanks for the information. You need more desktop space . Man, you shoot a lot!

[webmstrk9]

Actually, I was just reminded about 68k were from a GoPro in time-lapse when I was on a 3 week road trip, so 2017 was a quiet year