Oliver,
Does your Color Class filter have all colors selected? I can't see it in the image you posted. I'm asking because I can't reproduce the modification time sorting issue (mine happen very briskly--no delay at all). If you have even one color turned off then the metadata for all of the images must be loaded in order to perform the color class filtering in the end. I notice the number of images displayed after the sort data is gathered is 2780. When the sort completes, I see only 546 images.
Having any filters active will significantly reduce the speed of sorting unless the sort cache has up-to-date data available (PM builds its sort cache as it works with images so if PM has browsed that folder before, then browsing it again later should be much faster.) PM5 does the same thing. The sort cache isn't infallible however, if images change since the last time their sort data was cached, then it is invalidated. Also, you have control over how long PM keeps sort cache data around. If the settings dictate that old data is discarded, then PM will flush it as it ages and viewing a folder that you viewed months ago will again be slow.
-Kirk
