For simplicity, I wouldn't cache anything. Run youtube-dl with the JSON dump option to get all the sources, find the best source, put the URL into a <audio> element and that's it. At least that's how I'd do it.