It's trivially easy for you to greylist everything, and trawl through that everyday, and whitelist what you want and blacklist the rest. Anything whitelisted is automatically whitelisted forever. Anything blacklisted is automatically blacklisted for ever.
The problem comes when spammers forge From headers which leads to:
-1: Very many emails in the greylist everyday
-2: False positives if they use a From that you've previously accepted (ie that person gets infected)
-3: False negatives if you get a spam that you reject which was sent from an address that you really want whitelisted.
Some of the failure modes are similar to challenge-response systems.
The problem comes when spammers forge From headers which leads to:
-1: Very many emails in the greylist everyday
-2: False positives if they use a From that you've previously accepted (ie that person gets infected)
-3: False negatives if you get a spam that you reject which was sent from an address that you really want whitelisted.
Some of the failure modes are similar to challenge-response systems.