I'm not sure as a general observation the "s-curve" turns out to work that well. E.g. futurists have (with Kurzweil and his crowd a notable exception) consistently underestimated progress in computer capabilities even in the 80s and 90s when progress was insanely rapid, while over-estimating advances in AI (which is, optimistically, at the beginning of its s-curve). You could almost include Asimov in that. Similarly, Arthur C. Clarke was oblivious to miniaturization while living in the thick of it.
I think one of the factors is romance -- SF writers want certain things to happen and just wish it into being, regardless of how plausible it really is (faster than light travel, time travel, anti-gravity, artificial gravity, laser pistols).
On the reverse side they tend to ignore technologies that are either unromantic or make stories more complicated or seem to fight against cliched dramatic opportunities. (Fast travel adds to romance and drama, but fast communications detract from romance and drama.)
I think over-estimation of AI comes less from the understanding the position of the technology on the s-curve, but more from easily conceivable vision of machine behaving just like human. Nobody wants to listen to people who predict AI in future being pattern-scanners, optimizers or programing languages, because that, as you said, isn't romantic or dramatic at all.
And I would dare to argue that the AI is at the beginning of the s-curve. The field of study have been around for many decades already, many applicable results found and it is used in the industry. It's kind of like saying that current transportation technology is at the beginning of the s-curve, because I am waiting for near light-speed transportation, and current speeds are very small compared to my arbitrary expectation.
Alternately, though this may be stretching the point, you could say that '60s AI-as-human was at the _end_ of its curve, shortly to be replaced by the more practical actual AI. While specific writers have different biases--and some end up way ahead of the curve--in the aggregate I think futurists tend to overestimate the technologies of their era.
Though you're definitely right that literary biases play into it. While not not all SF writers are futurists, most futurists are trying to tell stories. Even some who aren't primarily SF writers: Kurzweil has a narrative arc to his ideas.
I think one of the factors is romance -- SF writers want certain things to happen and just wish it into being, regardless of how plausible it really is (faster than light travel, time travel, anti-gravity, artificial gravity, laser pistols).
On the reverse side they tend to ignore technologies that are either unromantic or make stories more complicated or seem to fight against cliched dramatic opportunities. (Fast travel adds to romance and drama, but fast communications detract from romance and drama.)