YouTube and Gmail both down

[rogerfederer]

If FireEye, that have architecture at the level of the US intelligence agencies, have been compromized by this simple hack then it is fair to say that anybody who works in Security or a Network team here, or leads one, will be busy over the festive season resolving this.

Just removing SolarWinds in entirety isn't enough. Every piece of kit requires inspected for code changes, config changes, while triple checking firewall logs and all Security apparatus in each company.

Time to up the day rate.

[vetran]

They posted a status message on Google Docs… oh, hang on

(But seriously folks, the other week AWS had a serious outage in one of their services, and the service status dashboard relied on that service, so they couldn't tell anybody what was going on with the serious outage )

I wish people would understand that the cloud is someone else's badly managed computer.

Everyone is running towards it but what happens when these outages occur daily when accountants continue to overrule techies.

[courtg9000]

SolarWinds is a network tool that facilitates a range of services, such as the polling of devices to check configuration changes and then log them iteratively somewhere. It has a large range of addons but is considered a poor tool for the job in a modern context.

The main issue is that many companies have had the software for a decade or longer and see no reason to retire it. Companies that started in the cloud or have moved completely to serving from cloud services are very unlikely to have any form of SolarWinds running that requires patching.

The list I have seen has a very large number of companies that have been hacked through malware packaged by SolarWinds, unknowingly to them, by what is thought to have the hallmarks and evidence of Russian state sponsored hackers. This could be one of the most widespread attacks for public and private companies ever seen and details are continually emerging.

It's safe to say that if I was a betting person I would put the money on Google being among one of SolarWinds' customers, as they still run a lot of kit on premises in datacenters and have been around since the dot com boom in some form or another.

It is worth remembering that it is also very likely that the number of public entities and private companies currently sitting with an affected patch from Solarwinds from many months ago is going to be very high. That malware may already have been used to either spread, after initiating connection with the outside world and a CAC (command and control) server, or it may have been customised to automatically try various paths to infecting other services within the company networks to propagate. The latter is more common, as this avoids triggering suspicious firewall rules and deep packet inspection that is suspicious of unusual activity. The worst case scenario is that the outside world communication with the malware was via SolarWinds patching servers for months, and this traffic will not have been marked as suspicious by almost all DPI services.

In the coming days the view of this could increase so much that it is likely that most SolarWinds customers are infected and have had information stolen. Disinfecting a client of the other malware/intrusions introduced as a result of the initial hack via SolarWinds is almost impossible without shutting up shop and starting again from ground zero.

We could be witnessing one of the biggest hacks in history. As many US core public services have been impacted, including intelligence, it is an extremely bad outcome.

Enjoy your cat videos.

I had the "pleasure" of SolarWinds about 10 years ago.
It was nasty and crap back then. I am willing to bet a tenner that not much has changed in 10 years.

[ladymuck]

I think clientCo are in the process of signing up with FireEye

[ladymuck]

I wish people would understand that the cloud is someone else's badly managed computer.

Everyone is running towards it but what happens when these outages occur daily when accountants continue to overrule techies.

Cloud isn't necessarily cheaper either

[jamesbrown]

Clouds are fluffy though.

[vetran]

Cloud isn't necessarily cheaper either

Guess what the costs will be like in 5 years time when we have sold all our servers and converted the server room into an executive dining room.

[clearedforlanding]

*.avsvmcloud.com is the file you should all be searching for if using SW.

[SueEllen]

Guess what the costs will be like in 5 years time when we have sold all our servers and converted the server room into an executive dining room.

Doesn't matter as the accountant who worked that out will be gone and ripping off another company.