I like this refinement. Personally I'd never ever run an address-book search, but if I did, just because I've ever sent/received an email with someone doesn't mean I want to find them on here (or for them to be able to find me, if they had the reverse-notification option set).