Go has a great solution for this in the binary package:

http://golang.org/pkg/encoding/binary/#Read

I've been working on some code to parse the shapefile format, which specifies some fields in big-endian and some in little (I have no idea why), but the binary package has made it really easy to deal with.