"Second, shared_ptr isn't very fast, due to some design decisions like ... not being intrusive (requiring 2x the allocations)."

Not quite. You can use std::make_shared to allocate the object and the ref count in one allocation. You get improved locality of reference as an added bonus.