That's exactly what I was missing!

Pretty awkward to have to first drag the window you want to take along a bit though. I'd much prefer if I could do something like cmd + ctrl + arrow to mean "move the currently active window to the next space"