Gmail – Get a list of all Gmail conversations with one person across several e-mail addresses

Say there's a person in my contact list that uses several email addresses – e.g., john.doe@gmail.com and john@doe.com

I list both addresses in his card at Gmail's contacts, but if I want to get a list of all our conversations, I have to:

1. switch to Contacts
2. search for his card
3. click 'list all conversations'

That works fine, but that's a bit too long.

If I type his name in search box or chat box, I can only select one of his addresses, not both.

In the past I solved this problem by creating a filter and applying a label. However, now I have several people like this, so creating labels for each of them is a bit too much.

So, it there a quicker way to list conversations from all email addresses of a single person?