I'd had it on sort by capture time, as I use two camera bodies.. but I've switched it to Filename sorting - performance doesn't appear to be affected by that. I have no filtering of files enabled.
I'd also disabled the Adobe DNG Converter, as it was way too slow for culling.
On further testing - it appears that it doesn't need both the contact sheet AND preview to be on the second display to work faster - but... It can run faster on the main display only as well - but..
Generally when we view the previews, the preview window has focus.. and the contact sheet thumbs are highlighted in the background.. this runs really slow on my system....
But - if I have the preview window open, but switch focus to the contact sheet, so I'm flicking between the thumbs on the contact sheet rather than the preview window, it updates the preview window MUCH faster than if I'm scrolling while the preview window is in focus...
Even faster if I change the display profile to sRGB as you suggested.
The preview window usually opens in front of the contact sheet, so try opening the windows side by side on the same display, and making sure that you're scrolling through the thumbs on the contact sheet instead of the preview window.
If anyone else with slow issues could try that?