I was alive as well, and just barely starting to get into coding, so my analysis will not be very accurate.

However, in a pre-google web, most sites would add thousands of keywords to their meta tags, hoping that even when the site was not relevant for most of those keywords, they would get the click, get paid by the ads and be incentivised to create more.

And since in those days the concept of relevance (and all that the page rank brought) was still far away, the sites themselves had to catalogue themselves, thus creating the keyword spam.

I remember jumping between search engines; Excite, Altavista, Yahoo, and not getting very relevant results.