Exactly correct. Video is on a different level of cost/complexity than images, html, etc. Video is not one of those things you can have your build pipeline automatically optimize for you as part of deploy (well at least not if you want deploys in a reasonable amount of time).

Disclaimer: Founder of service that lets you use <smartvideo> tag with raw, unfiltered videos, and automatically turns them into optimized, streaming, cdn enabled playback.