With a couple Google queries, you can find out that the Chrome PDF viewer has had multiple vulnerabilities that allow code to be executed in the sandboxed environment, and that there have also been Chrome vulnerabilities that allow sandboxed code to escape the sandbox.

Sandboxing is certainly a good thing, but it's not perfect. Sandboxed code still increases the attack surface. The rationale behind pdf.js is that, because it's almost entirely unprivileged JavaScript, it's unlikely that there's a way to exploit a browser in the presence of pdf.js that wouldn't work in the absence of pdf.js.

The browsers themselves have also been subject to such vulnerabilities. I dare say the attack surface of a full browser implementation is much larger than just the sandboxing, and sandboxing ideally sits behind all I trusted operations in the browser, providing a single common last line of defense.