What sorting are you using? If it involves metadata inside the files, it will be slow since PM needs to read the file before it can sort it. Lightroom reads that data once (when you ingest the images) and stores it in its catalog so it can sort faster. This holds true for PM's catalog too (of course), but not for normal browsing where there is no database of data to fall back to.
Fastest sorting is on filename and modification time (and arrangement). The others involve reading the files.