Author Topic: Are there limits on the number of concurrently connected Active Catalogs?  (Read 2642 times)

Offline ejhutch

  • Member
  • **
  • Posts: 50
    • View Profile
Are there limits (practical or otherwise) to the number of catalogs that can be connected to (or open) at the same time in the Active Catalogs list?

Is it just hardware and available memory limited, or are there limits imposed by the software design or other constraints as well?

And how is the number of active catalogs constrained by the size of the respective concurrently active catalogs?

Are there suggested maximum catalog sizes, and / or numbers of concurrently open catalogs?

And is there a performance benefit for using fewer / larger catalogs, or more / smaller catalogs, especially if searching or modifying across multiple catalogs?

Does anyone have any recommendations (hopefully based on results from experience and / or testing various combinations and catalog sizes)?

The number of assets involved would be on the order of 500K or more, stored on local and network storage. Local is SSD, and space limited. Network is platters and for my purposes effectively space unconstrained, however transfer bandwidth is of course much slower than local, though still pretty fast for ethernet and wifi.

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Are there limits (practical or otherwise) to the number of catalogs that can be connected to (or open) at the same time in the Active Catalogs list?

No.  There are no limits, but management of them may become tedious for you as the user if the number of catalogs that you manage is significant.

Is it just hardware and available memory limited, or are there limits imposed by the software design or other constraints as well?

There are no hard limits in the application.  Searching across multiple active catalogs is supported but as the number of active catalogs goes up, the more queries have to be made when searching (one per catalog) and then the data has to be aggregated and sorted before it is returned to the contact sheet.

I have no idea at what point there would be too many active catalogs.

And how is the number of active catalogs constrained by the size of the respective concurrently active catalogs?

There is no such constraint.

Are there suggested maximum catalog sizes, and / or numbers of concurrently open catalogs?

No.  I regularly test with a one million item catalog and I have more than a dozen smaller catalogs used for testing.  However, it is rare that I have them all active for searching.

And is there a performance benefit for using fewer / larger catalogs, or more / smaller catalogs, especially if searching or modifying across multiple catalogs?

The larger a catalog gets, the longer it takes to add/remove items due to the size of the indexes in the database.  That said, a catalog of 1 million items is reasonably responsive.

Does anyone have any recommendations (hopefully based on results from experience and / or testing various combinations and catalog sizes)?

We feel that a catalog of one million items or fewer is performant.  I've seen a user with 1.6m items and he seemed happy with the speed.

The number of assets involved would be on the order of 500K or more, stored on local and network storage. Local is SSD, and space limited. Network is platters and for my purposes effectively space unconstrained, however transfer bandwidth is of course much slower than local, though still pretty fast for ethernet and wifi.

500k images should be no problem.  SSD is preferred as the storage location for the catalog.  Proxies do take up a fair amount of space and are optional.  You can also control their size and make them as small as 512 pixels if you desire.  At some point I'd like to make it so that the proxies and the database could be on two volumes such that the proxies could be stored on a larger but slower local drive and the catalog could be on the SSD.  Loading thumbnails and previews from a non-SSD local drive should still make for good performance.

That said, you'll find that performance with your network-stored items isn't going to be great when displaying them in search results.  Currently, if an image is available, it is always loaded directly instead of using the proxies.  Proxies are only used when the media containing the images is unavailable.  I'd like to change that and give users the option of using the proxies first on user-specified volumes.

HTH,

-Kirk

Offline ejhutch

  • Member
  • **
  • Posts: 50
    • View Profile
Wow! That was a quick response, and very thorough and well written as well. I'm impressed. Thank you!

For what it's worth, can you tell us what the specs are for your hardware that you use to test 1M asset catalogs with?

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Wow! That was a quick response, and very thorough and well written as well. I'm impressed. Thank you!

For what it's worth, can you tell us what the specs are for your hardware that you use to test 1M asset catalogs with?

Mac Pro from 2012, SSD drive, 128GB RAM.  It's my development system.  The 1m catalog would work well with much lesser hardware.  SSD is key, CPU and RAM are less important.

-Kirk