That is the case, and that is the point. It is very hard, in fact impossible to do for more than a few seconds without practice. Which is the problem you are solving by practicing. If you can make yourself just sit there, not scolding yourself as you drift, but always returning to silence and the breath, you can go longer and longer without thinking.
Its just like physical exercise - repetition makes you better. And just like physical exercise, it makes your entire life better.
I think I read somewhere in the book that concentration isn't all that important (& in fact you don't want to develop it too far). You're mostly trying to observe yourself lightly, & to catch yourself.
That's the same method "Happy to Burn" describes, but I still find it a case of "easier said than done", and difficult to do.