Monday, November 02, 2015
Wednesday, October 28, 2015
We are under attack !!!
Note: this post is adapted from an amazing article at Veracode blog on "3 Easy Steps to Making Perfect Security Possible".
All of a sudden, we realized that our company was brutally hacked. More than that, it was embarrassing: someone put company emails and files on Pastebin and deleted data from the web servers.
So... What we did?
We spend most of the time answering to management how it happened. Explaining the same again and again: we have no idea, actually, and we are trying to start investigating the incident - as soon as we can leave the dozens calls with managers and focus on our job. Our initial reaction is to feel a huge, gaping hole in our chest. We felt stupid, utter helplessness. I hide a while in my office as I keep from crying. To everyone who asked, I told that I’ve got dust allergies.
:'(
I read online news stories and posts on social networks about what everyone's saying about our breach, and I felt sick to my stomach. I found a little comfort in security pros online saying everyone gets hacked eventually.
Soon we realized we need to do something. We drop everything we had to do to investigate how the attack happened (including writing dozens of reports, answering end users' emails, and posting cat pics on Facebook). We started looking at events and logs until our head pounds and we got a sharp pain right behind our eyes. And we kept stopping the investigation once on a while to answer questions from management. Later we gave up investigating and we started telling everyone asking that we don't really know how the attack happened.
For a moment I wish I had a Chinese IP address in my log, so I could just blame Chinese hackers and go back to home.
We were trying to shut down some affected servers and some network segments in order to confine the attack and avoid the attack to spread across entire network. Everyone were working like crazy at the data center. Our network topology is a mess, so shutting down a network segment means digging into dozens of routers and following hundreds of network cables to figure out which device belongs to which network.
All of a sudden, our CIO went to the data center. It is very likely the first time ever he put a foot on our data center ! He kept asking everyone what we were doing and how long it would take to have the network back to normal operation. No pressure, for sure !!! He got impressed with the huge mess we have in our data center and he claimed we will have budget to reorganize and to fix the infrastructure. Don't tell him it's too late - but you think it. He never went back to the DC again and we never saw such investment :(
Management them asks if we should contact the police. We don't actually know. We contact the police, but they don't really know what to do. We told management that they should contact partners and customers who might be affected. They replied telling us that we have to avoid the journalists. In addition, they asked me what they should say. I don't know, so I told them to tell everyone they should change their passwords.
If you have no idea on what to do, ask people to change their passwords and they will kept busy for the rest of the day !!!
We then focused on cleaning the suspected areas of the breach and running scans on the rest. We changed all the passwords we found, everywhere.
In the next day I went to the office early because I got called in for an emergency: our web server pages have been defaced. During investigation, we discovered the database is corrupt. We were happy, since we've recently invested in a kick-ass back-up solution for it. But soon we realized the last back-up actually recoverable is a month old and I kick myself for never testing the recovery system. I felt stupid and I punch the back-up machine. Then I felt stupider for hitting a computer screen.
The same thing happened the next day: I woke up with my phone buzzing in my bed. Then I found out that the web servers got tagged again. I went to work in the clothes I'd slept in and I realize that other employees avoid me in the hallways. A co-worker from my team told me that I looked like shit. The I remembered that I got into this field for a reason: I used to like security.
As next step, we check with our SOC and we realized our WAF and IDS let the attack through. I called the vendor to complain and I got told I'd configured them wrong. Then I had to remind them that they configured it. They deny it, for sure. As I got angry, I went to meet with management about looking for new security vendors. Then I found out from management that we still have over two years left on our contract with that vendor, and someone on our team told me later that the vendor is the cousin of the CEO.
So, lets go eat chocolate to deal with it.
All of a sudden, we realized that our company was brutally hacked. More than that, it was embarrassing: someone put company emails and files on Pastebin and deleted data from the web servers.
So... What we did?
We spend most of the time answering to management how it happened. Explaining the same again and again: we have no idea, actually, and we are trying to start investigating the incident - as soon as we can leave the dozens calls with managers and focus on our job. Our initial reaction is to feel a huge, gaping hole in our chest. We felt stupid, utter helplessness. I hide a while in my office as I keep from crying. To everyone who asked, I told that I’ve got dust allergies.
:'(
I read online news stories and posts on social networks about what everyone's saying about our breach, and I felt sick to my stomach. I found a little comfort in security pros online saying everyone gets hacked eventually.
Soon we realized we need to do something. We drop everything we had to do to investigate how the attack happened (including writing dozens of reports, answering end users' emails, and posting cat pics on Facebook). We started looking at events and logs until our head pounds and we got a sharp pain right behind our eyes. And we kept stopping the investigation once on a while to answer questions from management. Later we gave up investigating and we started telling everyone asking that we don't really know how the attack happened.
For a moment I wish I had a Chinese IP address in my log, so I could just blame Chinese hackers and go back to home.
We were trying to shut down some affected servers and some network segments in order to confine the attack and avoid the attack to spread across entire network. Everyone were working like crazy at the data center. Our network topology is a mess, so shutting down a network segment means digging into dozens of routers and following hundreds of network cables to figure out which device belongs to which network.
All of a sudden, our CIO went to the data center. It is very likely the first time ever he put a foot on our data center ! He kept asking everyone what we were doing and how long it would take to have the network back to normal operation. No pressure, for sure !!! He got impressed with the huge mess we have in our data center and he claimed we will have budget to reorganize and to fix the infrastructure. Don't tell him it's too late - but you think it. He never went back to the DC again and we never saw such investment :(
Management them asks if we should contact the police. We don't actually know. We contact the police, but they don't really know what to do. We told management that they should contact partners and customers who might be affected. They replied telling us that we have to avoid the journalists. In addition, they asked me what they should say. I don't know, so I told them to tell everyone they should change their passwords.
If you have no idea on what to do, ask people to change their passwords and they will kept busy for the rest of the day !!!
We then focused on cleaning the suspected areas of the breach and running scans on the rest. We changed all the passwords we found, everywhere.
In the next day I went to the office early because I got called in for an emergency: our web server pages have been defaced. During investigation, we discovered the database is corrupt. We were happy, since we've recently invested in a kick-ass back-up solution for it. But soon we realized the last back-up actually recoverable is a month old and I kick myself for never testing the recovery system. I felt stupid and I punch the back-up machine. Then I felt stupider for hitting a computer screen.
The same thing happened the next day: I woke up with my phone buzzing in my bed. Then I found out that the web servers got tagged again. I went to work in the clothes I'd slept in and I realize that other employees avoid me in the hallways. A co-worker from my team told me that I looked like shit. The I remembered that I got into this field for a reason: I used to like security.
As next step, we check with our SOC and we realized our WAF and IDS let the attack through. I called the vendor to complain and I got told I'd configured them wrong. Then I had to remind them that they configured it. They deny it, for sure. As I got angry, I went to meet with management about looking for new security vendors. Then I found out from management that we still have over two years left on our contract with that vendor, and someone on our team told me later that the vendor is the cousin of the CEO.
So, lets go eat chocolate to deal with it.
Subscribe to:
Posts (Atom)