The article I wrote yesterday on the lessons of Gawker’s massive security breach spurred a number of reactions including one I was not quite expecting: an e-mail from Gawker Media founder Nick Denton.
Denton is a controversial figure in the worlds of media and politics as documented in much of the coverage of the breach, but as a security guy I had no preconceived notions of what he was like. I have to say I appreciate his points in response to the article. For most security folks, and I’m no exception, seeing a company recover from an event like this by taking steps towards implementing a mature information security program is a positive outcome. It demonstrates that a firm has walked away from a serious security event acknowledging what happened and taking steps to mitigate the identified risks going forward.
The first thing he noted was that the piece in Forbes was “brutal but fair.” I appreciate knowing the piece was seen as fair, and didn’t intend for it to be brutal. Most security and IT compliance folks tend to tell it like it is, and I try to do the same. My first concern, an ongoing concern with coverage of security events, is that the story of a data breach is not always seen in a wide context. What I mean by that, is that people focus in on the specific security problem at fault in the breach (it was an unpatched server, it was a PHP file injection, it was a SQL injection, it was a VPN with only password authentication, etc.) rather than seeing that the strategic problem was that the affected firm was just not taking information security that seriously, that if the breach wasn’t based on one weakness, it would have been on another. That they have not hired the right leadership for security, implemented the right program, and then taken the correct tactical steps (for Gawker: exercising an incident response plan, implementing password complexity and use requirements, having an intrusion management strategy,having a data breach plan) in the context of an overall information security strategy.
My second concern or motivation is when the details of a breach do come out, and they rarely do, to document it such that not every organization has to go through the experience Gawker has, to learn what is necessary. Frankly a lot of firms have the problems Gawker has and just have not been exposed.
On to what Nick Denton had to say:
“We thought it was just the company email that had been compromised on Saturday. We didn’t take the bigger threat seriously. That was the mistake. Not denial. And we did put post up as soon as it became so clear user accounts compromised on Sunday afternoon.”
That is a fair assessment, essentially voicing that had they understood the problem from Saturday in the context of a larger attack, they would have reacted differently. That said, the going forward approach then is to have a process for reporting seemingly disparate small security issues, and having a person to report them to (in most organizations a chief information security officer or CISO) who can put them all together in the context of a larger attack. Imagine if the problem had been first identified in November via a process to audit related systems based on the one reported intrusion into the internal chat application, or if an intrusion detection system had started firing red based on the exfiltration of their entire MySQL database. It would still have been a problem, but it would have been a lesser problem, and one not first publicized by the attackers themselves.
On my criticism of Denton’s comment on users’ easy-to-crack passwords, he writes: “We’re not blaming people with weak passwords; just warning those most [at] risk.”
That’s fair. I had interpreted their mentioning “people with weak passwords” as their not fully taking responsibility for what happened in the post on their site yesterday. In other words, if you have a weak password that is easily decrypted, then you share culpability for your credentials being breached. This is of course not the case, when a password database is lost and when an obsolete encryption algorithm like DES is used, users can not share blame.
But in the context of risk he is right, weak passwords, common dictionary words, will be broken faster. With the database table of passwords widely available, it should be noted that most of the passwords will be broken eventually however, and so every user who has used a Gawker Media web site should change their password once Gawker is sure their systems are clean of any shells left behind by the attackers. Certainly any user who has used the same password elsewhere should be changing all their passwords now.
On one of his writers’ reference to readers as “peasants,” Denton writes: “The ‘peasants’ remark was a joke, taken as such by the commenters…But, yeah, confusing to a wider readership.”
Fair enough, the peasant thing is an inside joke at Gawker among writers and regular readers/commenter’s. One could make a case that the comment should never have been seen by the public at large anyway, as it is from an internal chat log from the Campfire collaboration tool used inside Gawker. This is another good reason for starting a good security program, to prevent leakage of internal employee conversations. WikiLeaks in February, as an example, may embarrass a large American bank (rumored to be Bank of America) with internal executive conversations in early 2011. It’s a hell of a lot better if the leak never happens, or is stopped in its early progress.
Gawker Media appears to be on the right track based on this note from LifeHacker’s FAQ on the breach: “We’re bringing in an independent security firm to improve security across our entire infrastructure.” Now I have to hope that the first piece of advice this independent firm provides is to establish an ongoing information security program at Gawker Media.
How do I Know If I Was Affected?
Security guru HD Moore took time out to provide a spreadsheet for users to, after hashing their e-mail address which is easily done following the instructions he provided, compare their e-mail hash with a list of hashed e-mail accounts from the Gnosis release.
Table of Hashed E-mails from the Breach: http://www.google.com/fusiontables/DataSource?dsrcid=350662
But you say, other sites have tools where I don’t have to bother with this hashing, I can just give my e-mail address and they tell me if mine was in the breach. You can do that if you like, and I’m not suggesting that any sites are doing this, but that strikes me as an easy way to get your e-mail address. From a security standpoint, checking a hash against a spreadsheet of hashes, the Moore solution seems more elegant. I understand it is not for everybody.
The earlier article I wrote speculated there would be downstream affects to the Gawker breach (not an earth shattering prediction), and it appears Twitter is the first to notice this happening.
As Mashable noted earlier, a number of users suddenly started tweeting that they lost weight with acai.
Twitter’s Del Harvey (remember her from “To Catch a Predator”?), then shared this earlier today:
There was some speculation on Twitter that libel suits against particularly caustic Gawker commenters might come out of this breach, because user names were now largely associated with actual e-mail addresses. Assuming the person was basically anonymous via only user name, but used an e-mail that clearly identified them, this might be possible. But most good Internet trolls are using nearly anonymous e-mail accounts, unless you are prepared to subpoena a major e-mail provider for IP addresses (and even that might not identify anything if an anonymous proxy is used or you may have to issue further subpoenas to Internet Service Providers) you may not get very far. So having both e-mail address and user name leads to only being a little closer to discovering someone’s true identity. It is something to watch for if it goes happen, a few of the more wild commenter’s on Gawker being “outed”.
Government / Corporate Accounts in the Gawker List
The earlier article mentioned seeing accounts for NASA, the social security administration, a UK official, an Australian official, FTC, NARA, USDA, FDA, the Library of Congress, and the Senate offices of Olympia Snowe and Bernie Sanders. Looking at the file since then, add to that e-mail accounts for people at the U.S. House of Representatives, Virginia/Michigan/Kentucky/Utah/California state governments, the postal service, the IRS, the CDC, the National Institute of Health, the Department of Justice, the NJ Port Authority, the departments of Labor/Education, the Bureau of Labor Statistics, the city of Albuquerque, and notably both the White House and the Department of Homeland Security. PBS actually published the e-mail addresses themselves, stating that they were distributed in a IRC chat frequented by members of the hacker collective Anonymous.
The folks in these lists are getting a lot of annoying e-mails if they have not already changed their accounts. Don’t use your work e-mail address to write comments on a media web site. More importantly, if you are one of the many who reuse passwords, you may have just provided access for bad actors to your corporate VPN, web accessible MS Outlook, and so forth.
The Source Code
For some reason it is being ignored in much of the reporting of the incident, but Gawker lost what appears to be most of their source code. For a web application, this is a significant piece of intellectual property. As application security expert and friend Jeremiah Grossman noted earlier today: “Gawker is going to have to rewrite their entire CMS, or purchase another. The code simply cannot be trusted.” Every possible hole that can be exploited via a coding flaw in the PHP Gawker uses to host their web properties can now be found by any bad actor willing to take the time to look for them.
For their part, the attackers themselves agree. In an interview over at thenextweb, one of the 13 members of Gnosis is quoted as saying: “T: I’d just like to point out that the disclosure of source should be taking a lot more of the stage…Gawker may as well go opensource.”
All web properties who have achieved a certain scale must have some manner of account deactivation function, and then also hopefully deactivate the account when a user requests it (there are examples out there where account deactivation exists, but accounts aren’t actually erased). Gawker is no exception and not having this available is generating complaints by its users. Having this functionality isn’t going to un-leak anyone’s e-mail or passwords, but users should have the right to say they no longer want an entity or corporation to maintain the personal information they provided. It is a reasonable nod to user privacy rights.
Web Applications and Passwords
Tom Dibble commented on the earlier article as follows, which I think quickly and pointedly sums up the right practice for web applications:
Note to web developers: if you are storing passwords in anything short of a salted one-way hash you are violating your users’ trust.
Without getting too complex, a hash function takes a string and returns a fixed size string or hash value. When it works properly, it is not possible to determine the original string from the resultant hash. For this reason, this is a good method for the storage of passwords, because it allows your web application to only store those hash values rather than actual plaintext passwords. When a user tries to login, your application hashes the password provided, and compares it with the hash previously on file for that user.
The problem with only using hashing for passwords comes with dictionary attacks that compare hash values in a captured database table with pre-computed hashes based on common passwords. A second problem is with the use of rainbow tables, a type of lookup table (data structure) that offers an even more time efficient way of determining the password represented by a hash.
To mitigate these problems, many web based applications use a “salted one-way hash” as Tom mentioned. In essence, this is concatenating a random value called a salt with the plain text password before applying the hash function. Doing this prevents any user hash from being identical, even if say two users have the same password. Every user’s password is hashed uniquely, breaking any one password does not result in a breach of many other identical passwords used by other users.
This approach is not magic, you should for example never go around saying your application’s passwords “cannot be hacked because they’re encrypted”, but it is solid approach to the problem of protecting user passwords.
The other reaction to the Gawker situation is predictably a lot of advice on passwords. The most common is “not to use the same password on different sites”. In truth, a lot of people do, there is a limitation on the number of passwords people can or will remember. A study by BitDefender back in August found that 75% of participants used the same credentials on social networking sites and e-mail. Back in February Trusteer in a similar study found that 73% of those studied used their bank password everywhere.
A few have suggested that there is a reason people have weak passwords on a site like Gawker. In essence, they mean that what is protected is not that important, the worst an attacker could do after compromising someone’s password is post comments as them. Unfortunately it has been shown that people have used similarly weak passwords on e-mail sites, such as with the pastebin hotmail password disclosure, so it is not clear that the majority of people are doing this subconsciously risk analysis and purposely using weak and strong passwords depending on the type of web site they’re using. And as noted, a whole lot of people are using the same password wherever they go.
The second piece of advice has to do with password composition, the idea that a good password is longer, contains letters and numbers, and has special characters. This advice has to do with the way password crackers work, in essence it is harder to break a longer more complex password. It is also harder to remember, so consequently people use stupid passwords like ‘password1’ or ‘abc123!’ in systems that require complex passwords, which are as insecure as dictionary words. One place I encountered used ‘letmein’ all over the place in their server environment, but that is an aside.
Finally people use information related to them personally, which is also an issue. Depending on what piece of information they use, and how publicly available that information is, they leave themselves open to targeted attacks on their accounts. Look no further than the first Gawker employee account exposed which used the person’s first name last initial as the user name, and added ‘1’ to that for the password. Such a combination is easily guessed, and can even be programmatically accounted for in a password cracking application.
Some advised coming up with complex passwords and writing them down or some method based on that. As a security person, I cannot bring myself to endorse writing down passwords, although it is probably better than having an obvious password (security people know how to check drawers and under keyboards). What I would advise is having different passwords for sites you do financial transactions on (banking, Amazon, eBay, Paypal), have sensitive information on (Facebook), could be embarrassed by if compromised (LinkedIn, Twitter), and then a cycle of passwords you use for every other blog or news site you comment on. Although frankly, the financial sites have to start embracing dual factor authentication for all of their users, password authentication does have its reasonable, well enumerated, limitations.
Another piece of good advice, get a password manager. A password manager is software that has a local store for all your other passwords which are stored encrypted with a key typically based on a user passphrase. You might think “all passwords secured by one password, how is that better?” Well having hard to break passwords in web accessible databases around the world secured by one complex password to an encrypted store on your own computer actually does work out better in most scenarios, certainly better than use of the same easily broken password in hundreds of different places. As Adam Shostack, Secure SDLC expert from Microsoft, noted on Twitter: “The lesson of Gawker isn’t “change your password” it’s “get a password manager” so you can use unique passwords per site.”
The other interesting thing to do on the security side when there is a large scale password breach (christiansingles, hotmail, rockyou) is to analyze the password complexity in a population of users. While the algorithm used in this case, DES, is obsolete, it is still the case that it is easier to crack a password that is a common dictionary word or minor variation than a complex password. Plus it is always interesting to see the most common passwords people use.
We started running a custom python script we have at Praetorian against a subset of the passwords when we noticed that Jon Oberheide at DUO Security had already done a more complete job cracking some 400,000 of the encrypted passwords published by Gnosis, already double the original number Gnosis cracked. For that reason, we’ll use his more accurate numbers below. The first meaningful number is that there are approximately 748,000 passwords that can be cracked in the Gnosis release, out of some 1.3 million lines from the database extract.
Probably the most alarming thing is the relatively high percentage of passwords composed only of lower case letters:
A reasonable immediate next step is for Gawker to require some password complexity when setting up a password on their site to leave comments.
The attackers took time out to point out the number of users using ‘password’ and ‘qwerty’ in Sunday’s release. I mentioned I was holding out for ‘123456’, the combination I use on my luggage, and I was not disappointed by the 25 most common passwords (credit: DUO Security):
Note the 301 who used the site’s name, ‘gizmodo’, as their password, a common practice. For the 307 who had ‘trustno1′ as their password, good advice.
I appreciate Nick Denton reaching out, and I think Gawker Media has started on the right track following this security event. What counts now is integrating this event into the corporate culture, using it a year and two years from now when the memory of it has lessened, to justify making the right effort, in both time and expense, to build and maintain a mature information security program.
There is one user I would like to address directly, who wrote this: “I’m in the database, but don’t seem to have my password leaked. Hoping that it stays that way, but based on the fact that I used password reset to access that account last, I’m thinking I should be good as it should be something random. Hopefully.”
This is in part what I was worried about when Gawker mentioned “people with weak passwords”, a false sense of confidence. When someone has the password file, your password has a probability of being enumerated. Please change your password.