I thought of other tips that may or may not help your workflow, as I sympathize with your plight.
Do the non-bracketed images that you're trying to find only come from certain photogs/cameras? If so, it takes ~2-3 seconds* to divvy up any folder into subfolders by camera model or serial#. Would that help?
If all your brackets are at one fixed shutter speed, but the singles are at different shutter speeds, that might also be a way to automate the sorting. Any other metadata cues that might help you?
Are you selecting these just by looking at the tiny thumbnails or are you using the Preview window first? If you're checking them out in the Preview window, are you aware of the 'w' shortcut when viewing images in the Preview window? If you turn off single-selection syncing**, you can do something like
1. Arrow keys to get to the first image in your session.
2. Spacebar to open the image in the Preview window.
3. Use arrow keys to move back and forth through images.
4. Press 'w' when there is an image you want to select.
5. Hit "Esc" to close the Preview window and all the individual, non-sequential images you pressed 'w' on are now selected.
(My hunch is that if you're dealing with mouse/trackpad mis-clicks a lot, this method might end up being faster for you, but just a guess. A few minutes of practice can make it very fast.)
I don't know if any of those can help at all, but that's what I thought of.
-Mick
* Steps:
cmd-a
cmd-y
check "move"
select "Subfolder"
put {serial} or {model} variable in the folder name
(Optional: create Snapshot preset to reuse in the future)
Press return
** Preferences > Contact Sheet > Uncheck "Synchronize single selection..." in the lower right