It's actually not so clear-cut. The solution of presenting resources within the head of the page only works if those resources are supposed to be loaded as a result of visiting the page. That's not true of robots.txt (in fact, you want a crawler to read robots.txt before it does anything else). It's also not true of crossdomain.xml (which Flash and Java use to pre-flight cross-domain communication requests).

Personally, I'd rather require the files to be in known locations than force every client that wants the information to make a request and parse HTML.