The performance I'm seeing on similar hardware is pretty much the same as you, but it really depends. If you are opening the Preview window, waiting a few seconds for the image to resolve and then another few for it to start caching out the next files from the contact sheet, and still seeing a blurry image when you move to the next image, that would be a problem, but if you are looking at the image in the Preview window, waiting that 3-5 seconds for it to resolve, scrolling far down the Contact Sheet, and then opening an image from there in the Preview window, you're likely going to have to wait again, as it wouldn't have had the chance to cache out that arbitrary file yet, and you are essentially watching the large preview load in real time, and it is showing you a stretched version of the Contact Sheet's thumbnail until it loads the bigger version.
