> If hashes are uploaded to devices, they can be extracted and images that clash against it can be created.
Many organizations have the hashes, so they could leak nonetheless. Either way, I don't think that's a major problem. If the system interprets a picture of a pineapple as CSAM, you only need to produce the picture of a pineapple to defend yourself against any accusations. If clashes are too commonplace, the entire system would become unreliable and would have to be scrapped.
In any case, I have looked it up. The database is indeed on the device, but it's encrypted:
> Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child-safety organizations. Apple further transforms this database into an unreadable set of hashes, which is securely stored on users’ devices.
Overall, after reading the PDF, here is my understanding of the process:
1. Apple gathers a set of "bad hashes"
2. They upload to each device a map from a hashed bad hash to an encrypted bad hash
3. The device runs an algorithm that determines whether there are matches with hashed bad hashes
4. For each match, the device uploads a payload encrypted using a secret on-device key, and a second payload that contains a "share" of the secret key, encrypted using the neural hash and encrypted bad hash.
5. The device also periodically uploads fake shares with dummy data to obfuscate the number of matches that actually occurred. Apple can't tell fake shares from real ones unless they have enough real shares.
6. Once Apple has enough real shares, they can figure out the secret key and know which hashes caused a match.
The main concern I have, and as a non-expert, is step 2: it requires Apple to provide their key to an auditor who can cross-check with child protection agencies that everything checks out and no suspect hashes are included in the payload. In theory, that needs to be done every time a new on-device database is uploaded, but if it is done, or if child protection agencies are given the secret so that they can check it themselves, I think this is a fairly solid system (notwithstanding the specifics of the encryption scheme which I don't have the competence to evaluate).
The thresholding is also a reassuring aspect of the system, because (if it works as stated) the device can guarantee that Apple can't see anything at all until a certain number of images match, not even the count of matching images. The threshold could only be changed with an OS update.
There's certainly a lot of things to discuss and criticize about their system, but it's going to be difficult to do so if nearly no one even bothers reading about how it works. It's frustrating.
> If the system interprets a picture of a pineapple as CSAM, you only need to produce the picture of a pineapple to defend yourself against any accusations.
If the system interprets a picture of a pineapple on your phone as CSAM,
after Apple notifies the authorities they have identified child porn on your phone,
after the police detain you with the courtesies afforded to all alleged pedophiles,
after you cough up your phone’s password,
you only need to produce the picture of a pineapple to defend yourself against any accusations,
and then point out to the folks with the guns that no, you didn’t delete the child porn from your phone, look, it’s just a pineapple,
I mean, that's one imaginary scenario. On the other hand, it's quite likely that upon a match the offending picture in full res is stored in an enclave on your device, and/or encrypted in the cloud, in such a way that it cannot be deleted by the user.
If they know this attack is possible, Apple, not being idiots, will cover their asses in court by saying that a match is merely strong evidence that the user may have had CSAM on their account, but that it cannot be said for certain unless the full image is obtained by the authorities, and that the full image should be where they say it is, with the voucher made by the device.
Because of that, prosecutors are unlikely to want to move forward without better evidence: Apple may very well testify for the defence if they do, and judges will ultimately chew them out. So yeah, I suppose rashness and incompetence in some parties may lead to a very uncomfortable situation, but ultimately it is likely that the police would be reprimanded for it and that it would be a lot more cautious afterwards.
> after Apple notifies the authorities they have identified child porn on your phone,
Apple has a team that will manually vet the matches, so no pineapple picture or a fuzzy forced hash collision picture will cause the authorities to be notified.
So if you're worried someone will secretly send fake CSAM hash collision images to your phone to trigger the process, the worst that will happen is that some poor sod at Apple will get mildly inconvenienced.
They have access to a "visual derivative" (which I suppose is their way of saying "thumbnail") but it probably wouldn't help if the adversarial example is normal porn. This being said, once the authorities are contacted, they will have to work to obtain the full image, because if all they have is a thumbnail and a voucher, the evidence would probably be thrown out in court.
As for using it for other things than CSAM, well, for one, Apple would know, because the thumbnails would show political memes, so they'd have to be in on the conspiracy. They probably don't want that liability. Furthermore, the hashes are supposed to be auditable: a third party could check that they are what they are, a court order could order such an audit, and it would be suspicious for Apple to refuse. They wouldn't want to include anything that could piss off any sufficiently powerful government or, say, the EU, because they are likely to figure it out. And if they give different hashes to different citizens, that will also be obvious.
Many organizations have the hashes, so they could leak nonetheless. Either way, I don't think that's a major problem. If the system interprets a picture of a pineapple as CSAM, you only need to produce the picture of a pineapple to defend yourself against any accusations. If clashes are too commonplace, the entire system would become unreliable and would have to be scrapped.
In any case, I have looked it up. The database is indeed on the device, but it's encrypted:
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
> Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child-safety organizations. Apple further transforms this database into an unreadable set of hashes, which is securely stored on users’ devices.
Overall, after reading the PDF, here is my understanding of the process:
1. Apple gathers a set of "bad hashes"
2. They upload to each device a map from a hashed bad hash to an encrypted bad hash
3. The device runs an algorithm that determines whether there are matches with hashed bad hashes
4. For each match, the device uploads a payload encrypted using a secret on-device key, and a second payload that contains a "share" of the secret key, encrypted using the neural hash and encrypted bad hash.
5. The device also periodically uploads fake shares with dummy data to obfuscate the number of matches that actually occurred. Apple can't tell fake shares from real ones unless they have enough real shares.
6. Once Apple has enough real shares, they can figure out the secret key and know which hashes caused a match.
The main concern I have, and as a non-expert, is step 2: it requires Apple to provide their key to an auditor who can cross-check with child protection agencies that everything checks out and no suspect hashes are included in the payload. In theory, that needs to be done every time a new on-device database is uploaded, but if it is done, or if child protection agencies are given the secret so that they can check it themselves, I think this is a fairly solid system (notwithstanding the specifics of the encryption scheme which I don't have the competence to evaluate).
The thresholding is also a reassuring aspect of the system, because (if it works as stated) the device can guarantee that Apple can't see anything at all until a certain number of images match, not even the count of matching images. The threshold could only be changed with an OS update.
There's certainly a lot of things to discuss and criticize about their system, but it's going to be difficult to do so if nearly no one even bothers reading about how it works. It's frustrating.