The ordinary response to notifications like these is, to put it bluntly and accurately, a rage-mob. But here you have a corrective, a case where that doesn't make sense. Because Google operates what is considered, by an overwhelming majority of expert opinion, one of the 3 best security teams in the industry, likely exceeding in so me ways the elite of some major world governments. And they can't reliably promise, at least not in 2019, never to accidentally durably log passwords. If they can't, who else can? What are we to do with this new data point?
The issue here is meaningful, and it's useful to have a reminder that accidentally retaining plaintext passwords is a hazard of building customer identity features.
But I think it's at least equally useful to get the level set on what engineering at scale can reasonably promise today.
So far the G-Suite experience has been underwhelming to say the least. Crap interface and less fine grained control over document access than even Google drive offers for free, gmail UI just plain sucks over IMAP and so on.
And now this. As much as there is to like about this email in terms of transparency it is also very interesting for what it does not say: Apparently they can't determine with certainty whether or not the accounts were accessed.
"legacy functionality that enabled customer Domain Admins to view password"
That functionality should have never existed to begin with.
"primarily impacted system generated or admin generated passwords intended for one-time use"
Note the weasel word 'primarily', either it did or it did not potentially affect all passwords.
"We have reviewed the login information for the account(s) and have found no evidence that the unhashed passwords were misused."
No evidence does not mean it did not happen, so there apparently is a chance that even if they did not find evidence that it did happen and that their audit trails for reviewing that log data are not such that they can guarantee that nobody had access to it or viewed it.
"an internal system that logged account signup information for diagnostic purposes, also inadvertently logged the administrator’s account password in our encrypted systems in an unhashed format."
This suggests some pretty major process failures, this change was apparently found after it had already been pushed to production without review or with a review that did not catch this pretty basic mistake.
Having the best security team in the world is great but if you then have stuff like this happening you have to wonder about the processes around deployment, which are just as important as having a great security team to begin with.
Damned if you do and damned if you don't, a small company is better off relying on the likes of Google for their secure storage of mail and documents, but at the same time that's also not perfect and apparently includes some random strangers potentially having access to all of that which is something that never happened to us in the last decade or so to the best of my knowledge.
You might be aware that in Google not every application team works together with the security teams. They are supposed to work with them. That is the best practice and that helps with security compliance and review. They often do. But it is also sometimes easy to forget doing so if the team is not doing their due diligence.
It is possible that this team did not work with the security team even if it is a highly unlikely scenario. The likely scenario is that this team did work with a security team and they were aware they were supposed to hash the passwords but they made a mistake during the implementation.
I think what is being underappreciated here is that very very smart application developers can have little to no idea about security best practices. I can say this confidently from my direct experience of working with Googlers.
To be honest, as far as I'm concerned there is no reason why Google or anyone should ever even have access to the plaintext passwords, let alone log them.
There Is no reason why we can't have 2 rounds of hashing, one client side and one server side. This way even if Google is malicious, it cannot know the actual password.
If you do a round of hashing on the client side, whatever is sent to the server becomes the actual secret needed to log in to the service.
The only improvement is if you share your password between websites/services, which you shouldn't do. And a malicious service can change the javascript to exfiltrate the input password any time you log in.
In both, the client proves that it knows the password without ever sending it over the wire. The server also doesn’t store it in plaintext.
In the algorithm I’m most familiar with (SRP6a) the plaintext password never leaves the client machine either on initial signup or on later password-based authentication, and what does transit the wire can be used neither to recover the password nor to replay the login attempt. You cannot break into the account just by passively watching the messages go back and forth.
One of the posts down-thread mentioned SCRAM, which I believe qualifies as a zero-knowledge proof algorithm, not a PAKE, but that’s about all I know about it.
One nice thing that falls out of a PAKE algorithm, as opposed to a zero-knowledge proof, is that you get a cryptographic key on both sides (thus the “key exchange” part of the acronym) that the peers can use with a symmetric encryption algorithm to communicate with each other. This then lets you construct things like TLS-SRP, which is a variant of TLS (ne SSL) that uses SRP to get the symmetric key instead of X.509 certificates and an entirely different key exchange algorithm:
https://en.m.wikipedia.org/wiki/TLS-SRP
Incidentally, the use of this sort of algorithm is why so many web sites now prompt for the user name and password on separate screens. There’s usually at least a two-way exchange involved where the user name goes to the server, which provides some bit of cryptographic hoo-hah, which then lets the client prove to the server in a separate exchange that it knows the password. This goes against old guidance to ask for both at once and to reject the login if either the user name or password are incorrect, which avoids giving an attacker a way to probe for valid user names. We lose that in the name of greater security goals.
But sure, there are good solutions to this, like SCRAM. Unfortunately, there is not much point when the authentication code is controlled by the server (eg. JS served by a server)
Worth noting that the security team at Google that you are likely referring to (P0 I assume?) has nothing to do with G Suite. Google has many security teams, and I’ve seen no indication that the security team whose scope includes G Suite is particularly exceptional, if they even have one. Some teams at Google don’t work with any security team at all.
That data point is meaningless by itself, everyone makes mistakes. Most of those people have nothing like the scale for making mistakes that Google does
The issue here is meaningful, and it's useful to have a reminder that accidentally retaining plaintext passwords is a hazard of building customer identity features.
But I think it's at least equally useful to get the level set on what engineering at scale can reasonably promise today.
correction: in the world -> in the industry