You have to trust those miners in aggregate -- they have the power to change the rules, effectively, since the vast majority of network users will follow the majority of hashing power.
Similarly, if any one bank doesn't follow the rules, they get kicked out of the financial network, to similar effect. You don't have to trust a bank, you trust the banking (and legal) system.
If these malicious miners you speak of "change the rules", it creates a fork in the blockchain. Even if they have a majority hash rate, if the economy does not support their decision, the market value of that forked chain will be low while the unchanged fork will keep the market. A group of malicious miners can do very little to destroy bitcoin.
Miner hardware is tied to the specific proof-of-work algorithm used in the cryptocurrency they mine. They make extensive capital investments that are, at least in the case of Bitcoin, useless for any other purpose.
Similarly, if any one bank doesn't follow the rules, they get kicked out of the financial network, to similar effect. You don't have to trust a bank, you trust the banking (and legal) system.