It's besides the point whether C pointers are more similar to Rust pointers or references. It's even true that pointers BY THEMSELVES have fewer constraints in Rust than in C . It's in the interaction between pointers and references that it's very easy to trigger undefined behavior in Rust.
Besides the fact I already mentioned about the dangers of casting pointers to references, there's also the problem that pointers are only valid as long as no operations are done with references to the same object (no interleaving). On top of it, the autoborrowing rules make it so it's not always clear when a reference is being taken (and operated upon).
So yes, in my opinion _unsafe_ Rust is significantly more difficult to get right than C.
Besides the fact I already mentioned about the dangers of casting pointers to references, there's also the problem that pointers are only valid as long as no operations are done with references to the same object (no interleaving). On top of it, the autoborrowing rules make it so it's not always clear when a reference is being taken (and operated upon).
So yes, in my opinion _unsafe_ Rust is significantly more difficult to get right than C.