Beware the Google Password Manager
Hey internet! So, someone broke into some of my accounts.
I'm taking entire responsibility for this - there's the part where I fucked up, and if I didn't fuck up, then none of this would've happened.
But there's also the part where a series of design decisions from various vendors combined into the perfect storm for me.
And we're going to talk about both! Separately! And calmly.
The part where I fucked up
I needed a macOS machine, to run CI builds.
So I rented one from one of the handful of cloud providers that do that. Because that's the only way you can run macOS without actually physically owning an Apple device.
Those cloud macOS machines usually come with Remote Desktop enabled, which for macOS means VNC. And they generate a password for you.
But VNC, especially on macOS, is slow. This is 2020, and much faster alternatives exist. Both Windows RDP and NoMachine take advantage of semi-contemporary video codecs (H264), and do a lot of other smart stuff that is completely beyond the scope of this article.
So I did the first dumb thing - I installed NoMachine.
Then I did the second dumb thing, by far the worst - I logged into a Google account from Safari. And shortly thereafter, the third dumb thing: I let Safari save the password.
I say "let", because I absolutely don't remember doing that. And that would make sense - it would make sense for this process (remembering a password) to be as frictionless as possible for a user, so that it indirectly encourages them to have strong passwords, since their browser can remember it for them.
So that's it, right - why does this article exist, clearly this is on you, etc. - get it all of your system now, this is me admitting that. I fucked up, and I should have done none of these things.
I'm not joking - just take a minute to say to yourself "ahh I can't believe someone would do something so dumb and then go on the internet and complain about something else".
Really just let it all out. It's all good.
And you know, I agree. I'm very mad at myself. I gave myself a stern talking to, and established new security practices, including frequent training.
Are we good on this?
The part where it got significantly worse
So someone was port scanning around the internet, and figured out that, on that particular IP, TCP port 4000 was open. Which is the default port for NoMachine.
They then proceeded to either guess or bruteforce the local user password, which is the one NoMachine uses for authentication. How that part happened I'm still not exactly sure, but it wasn't spectacularly strong, so, weirder things have happened.
This is why I always disable password login for sshd, and set up fail2ban, etc. - none of which were helpful in this particular case, because, hey, NoMachine! Third-party software, not really integrated with the system! What could go wrong.
So they got access to the macOS machine. And that's where the lock screen would come in, except the macOS lock screen only shows up on VNC sessions (or, you know, physical sessions), not on NoMachine sessions, since NoMachine uses the same credentials as the system's lock screen.
After going around resetting credentials and assessing the damage, I was able to reconstruct a rough timeline of some things they did.
First they checked
/etc/shadow - which, well, this is macOS Catalina, so,
why? Then they tried
sudo -l a bunch, and some other random stuff.
They somehow used XCode to edit a python file, which I've retyped for your viewing pleasure - don't execute it. They even downloaded a copy of xmr-stak-cpu, in case they needed to do some mining later.
Then they noticed there was a non-expired Google session there (that, too, is my fuckup). They went through Drive, looked at a couple of files, then switched to GMail.
At which point they were presented with a password prompt, that Safari helpfully filled out.
Since it was an "expired session" login, not a "set up a new device" login - in Google's current security model, it's called refreshing a "blessed session", there was no two-step verification (no 2FA prompt).
So everything up until now is 100% on me. Shouldn't have installed NoMachine, should have replaced the password with a stronger one, should never have logged onto a Google account on this machine in the first place, and definitely, definitely shouldn't have saved that password and not logged out.
But the reason I'm doing this write up is not just so the internet can judge poor decisions I made after a long day of fighting against CI builds. It's to examine the decisions made by Google about their password manager.
Right now, I'm a happy 1Password user. All my passwords are in a private vault, with a nice beefy master password that only I know. I've enabled 2FA on all the sites that are important to me - that's maybe 12 of them.
But that wasn't always the case. I had a life before 1Password... and Google remembers.
Before I got serious about password management (well, except that one thing we're discussing now), I just used Google Chrome's built-in password management. Go to a website, type your password, Chrome offers to save, sure, that's convenient.
And of course, Google Chrome synchronizes these passwords to their servers. So that if you log into Chrome from another device, you have access to all of these.
What I didn't realize, was that by default, your Google Account credentials are used to encrypt those passwords. Which means Google is able to decrypt those passwords.
And I should've realized that sooner - how else is a feature like Password Checkup supposed to work?
Furthermore, again, if you are on that default encryption model, there's a whole online password manager.
And that's exactly what the attacker went for.
So by this point the attack is already pretty bad. A Google account is a pretty high value target.
But using the Password Manager, and Safari's password auto-filling, they proceeded to go through my old passwords, one by one - and then eventually just grab the whole thing with the handy CSV export feature.
Using only my password.
Now, most of those passwords were outdated, since I only save things to 1Password nowadays, and I've rotated the important ones since then. And the current passwords were still mostly useless - as I have 2FA enabled on, again, the important websites - that have anything of value in them.
Except for OVH, which I only use to buy domains, so, I interact with a couple times a year. I was tired of paying bills manually, because I own maybe a dozen domains, so I connected my Paypal account, and next thing you know, the attacker has ordered themselves a shiny new dedicated server.
(The attacker was kind enough to choose a reasonably-sized instance, this could've been much, much worse).
Then the attacker made a mistake. For whatever reason, they tried using my password to log into my Google account from a Buenos Aires IP address. Possibly they wanted to grab the credentials for their brand new server, and they didn't know NoMachine had clipboard sync?
Google's defenses went up: "A vacation? That far away? In the middle of a pandemic? I don't think so". The login attempt was blocked, and Google sent me a notification about it on my phone.
Which is the notification I woke up to. Along with a 2FA SMS from Stripe.
(As far as "bad ways to wake up" go, this one ranks pretty high. Also, I later found out that the attacker saw the security notification from the compromised machine - they bailed out quickly thereafter).
It's only later that I realized they had disabled 2FA for my Google account. Accidentally, while reviewing my security settings.
I hadn't gotten a notification for that! Not by e-mail, not by phone, and the worst thing is they didn't need my 2FA credentials to disable 2FA.
After doing a significant amount of credential scrubbing, I eventually posted on Twitter: hey, uh, PSA, you don't need 2FA credentials to disable 2FA for your Google account anymore, and you don't get notified when it happens.
I was met mostly with skepticism, "of course you need 2FA to disable 2FA, that's how it works", followed by "oh god, you're right, you don't".
Eventually, a Google Security engineer showed up, and explained the following things to me.
The Google Password Security model
First off, let's address opt-in security features that I didn't have enabled:
- There is another encryption scheme for your Google-stored passwords, which is presented as "choose as passphrase". This stops your vault from being decryptable server-side, and it follows that this disables the Password Checkup feature.
I think this one may have helped - it probably would've asked for my passphrase when visiting the Google Password Manager page, and then the attacker would've been stopped right there.
- Google has an Advanced Protection Program (APP), which tightens some screws.
That one wouldn't have helped at all, since even with APP enabled, all you need is a "blessed session" (even expired) and a password to disable 2FA.
So, the Google engineer in question explained that this is all intended behavior.
First off, they're trying to reduce the amount of 2FA prompts to avoid desensitizing users to it. The idea is that if users are prompted for their 2FA credentials left and right, they'll stop thinking hard enough about why they're entering them, which renders the overall scheme less secure.
I'm sympathetic to that.
The second piece of information was that, in those particular cases where the attacker has physical (or, as was my case, remote) access to a device you've previously signed onto, the first factor is the password, but the second factor is the device's lock screen / password.
In my case, both were quickly defeated: Safari autofilled the password, and NoMachine skipped the lock screen. I enquired how did they know a lock screen was actually set up, let alone solved recently, and the answer was: "on Android, we know - for Windows & macOS, we'd probably need browser extensions".
Now, I don't know about you - I'm still pretty mad about my major fuck up, but this seems like the bigger story here.
- If you're logged into Chrome with your Google account
- Any password saved is synchronized to Google servers
- And then accessible from a webpage, including a CSV export feature
- Which asks only for your password for a "blessed session"
- At no point are 2FA credentials requested, and you don't need those either to disable 2FA.
At the end of our discussion, and after a few clarifications, the Google engineer agreed that this particular scenario was a security failure, and that he would push for changes to be made.
So, what should you do?
First off, if you're using a third-party password manager - 1Password, KeePassXC, a .txt file on your desktop, go scrub your Google Password Manager.
Even the privacy-oriented ones of us, who have switched to another browser, may have skeletons hiding in their Google account, that they can't quite get rid of completely, because, this is 2020, you can run but you can't hide.
(Furthermore, if you work at a third-party password manager company, you may want to add that step to your onboarding process, to warn users.)
If you're not using a third-party password manager, go pick a passphrase. This will make your life less convenient, but it will significantly mitigate the damage in case you, like me, end up very tired and hit a nuclear streak of bad decisions.
I didn't know APP was a thing, so some of you probably won't either - you may want to look into that, if you're interested in some additional safety.
Finally, take care of yourselves. If I hadn't been that tired, I probably wouldn't have been so careless that particular day, and none of this would've happened.
I'm looking for my next job!
August 2020 will be my last month at my current job, and I'm excited to figure out the next step.
If you're a remote-friendly company and you think we'd be a good fit, don't hesitate to get in touch!