You're probably going to do a lot better automating data collection based on DDG/!g searches. I really don't know how you'd score for quality, but at least take this as an example of a search whose results were not satisfactory.
Might also be the sort of thing to look at with a specific focus group. I know that the trend is to use a naive audience, but you might want to specifically recruit self-selected power-users.
I'd also design such a study to blind the result sets. I know Microsoft raised a bit of a stir claiming that their search results in blind testing were better than Google's a fair bit of the time. Problem at least personally with those results are that the two things I care about in search are relevance, and credibility in protecting user interests.
Microsoft have a very poor track record in that latter. I don't Bing. Period.
Might also be the sort of thing to look at with a specific focus group. I know that the trend is to use a naive audience, but you might want to specifically recruit self-selected power-users.
I'd also design such a study to blind the result sets. I know Microsoft raised a bit of a stir claiming that their search results in blind testing were better than Google's a fair bit of the time. Problem at least personally with those results are that the two things I care about in search are relevance, and credibility in protecting user interests.
Microsoft have a very poor track record in that latter. I don't Bing. Period.