Actually, the Wikipedia article led me to help answer my own question. The second point made in this amicus curiae brief in the request to the Supreme Court to overturn Ulbrecht's sentence makes the same point that you do: that it is not OK for judges to enhance sentences by determining their own facts for the purposes of sentencing that were not affirmed in charges for which the jury found the defendant guilty. FWIW SCOTUS didn't take the case, but as least some lawyers who presumably have some idea what they're talking about agree with you.

http://www.supremecourt.gov/DocketPDF/17/17-950/34432/201802...