The best way to recover from a ransomware attack is to have a reliable and fast backup process. Here's how to do it.
According to a Sophos survey of 5,000 IT and cybersecurity leaders released in April, 59% of organizations have been hit by a ransomware attack in 2023, from which 56% paid a ransom to get their data back.
And the amounts paid were not trivial. In 63% of cases the ransom demand was for $1 million or more — $4.3 million, on average. Of the 1,097 respondents who shared their payment details, the average payment was $4 million — up from $1.5 million in 2023.
What is ransomware?
Ransomware is a type of malware that encrypts a victim’s files. The attacker then demands a ransom from the victim to restore access to the encrypted data.
Many organizations are paying ransom
According to a report released in July by Semperis, based on a survey of 900 IT and security leaders, ransomware attacks disrupted business operations for 87% of companies.
But paying ransomware is a losing game. Of those who were hit, 74% were hit multiple times, sometimes within the span of the same week. And of those who paid up, 72% paid more than once. In fact, 32% of victims paid ransoms four or more times last year.
And, to rub salt into the wound, 35% of organizations who paid up didn’t receive decryption keys or had other problems with recovering files and assets.
Ransomware has been around for a long time. Why are we still paying for it? Part of the reason is the lack of backups — specifically, the lack of usable backups. Backups must be safe from malware, quick and easy to recover, and include not just important files and databases but also key applications, configurations, and all the technology needed to support an entire business process. Most importantly, backups should be well-tested.
Here are eight steps to ensure a successful recovery from backup after a ransomware attack.
1. Keep the backups isolated
According to the Sophos survey, in 94% of cases ransomware actors attempted to compromise the backups. And 57% of those attempts were successful. When attackers were able to successfully compromise the backups, the average ransom payment was $2.3 million — compared to $1 million for companies whose backups weren’t compromised.
In addition, companies whose backups were compromised were twice as likely to pay the ransoms — 67% versus 36%. Those with compromised backups also had eight times higher recovery costs, separate from the ransom payments — $3 million versus $375,000.
“We do see some of our clients that have on-prem backups that they run themselves, as well as cloud-based ones,” says Jeff Palatt, former vice president for technical advisory services at MoxFive, a technical advisory services company. “But ideally, if someone has both, they don’t cascade. If the encrypted files get written to the local backup solution and then get replicated to the cloud, that doesn’t do you any good.”
Some cloud-based platforms include versioning as part of the product for no additional cost. For example, Office 365, Google Docs, and online backup systems like iDrive keep all previous versions of files without overwriting them. Even if ransomware strikes, and the encrypted files are backed up, the backup process just adds a new, corrupted version of the file—it doesn’t overwrite the older backups that are already there.
Technology that saves continuous incremental backups of files also means that there’s no loss of data when ransomware hits. You just go back to the last good version of the file before the attack.
2. Use write-once storage techniques
Another way to protect backups is to use storage that can’t be written over. Use either physical write-once-read-many (WORM) technology or virtual equivalents that allow data to be written but not changed. This does increase the cost of backups since it requires substantially more storage. Some backup technologies only save changed and updated files or use other deduplication technology to keep from having multiple copies of the same thing in the archive.
According to a report released in June by Veeam, many companies are already using immutable storage. According to the survey, 70% of companies use hardened disks on-premises and 89% use immutable clouds. However, of the overall backup storage used by companies, only 54% is immutable. That means that the rest is at high risk from ransomware.
3. Keep multiple types of backups
“In many cases, enterprises don’t have the storage space or capabilities to keep backups for a lengthy period of time,” says Palatt. “In one case, our client had three days of backups. Two were overwritten, but the third day was still viable.” If the ransomware had hit over, say, a long holiday weekend, then all three days of backups could have been destroyed. “All of a sudden you come in and all your iterations have been overwritten because we only have three, or four, or five days.”
Palatt suggests that companies keep different types of backups, such as full backups on one schedule combined with incremental backups on a more frequent schedule.
4. Protect the backup catalog
In addition to keeping the backup files themselves safe from attackers, companies should also ensure that their data catalogs are safe. “Most of the sophisticated ransomware attacks target the backup catalog and not the actual backup media, the backup tapes or disks, as most people think,” says Amr Ahmed, EY America’s infrastructure and service resiliency leader.
This catalog contains all the metadata for the backups, the index, the bar codes of the tapes, the full paths to data content on disks, and so on. “Your backup media will be unusable without the catalog,” Ahmed says. Restoring without one would be extremely hard or impractical. Enterprises need to ensure that they have in place a backup solution that includes protections for the backup catalog, such as an air gap.
5. Back up everything that needs to be backed up
When Alaska’s Kodiak Island Borough was hit by ransomware in 2016, the municipality had about three dozen servers and 45 employee PCs. All were backed up, says the company’s former IT supervisor Paul VanDyke, who ran the recovery effort. All servers were backed up, that is, except one. “I missed one server that had assessed property values,” he says.
The ransom demand was small by today’s standards, just half a Bitcoin, which was then worth $259. He paid the ransom, but only used the decryption key on that one server, since he didn’t trust the integrity of the systems restored with the attackers’ help. “I assumed everything was dirty,” he says. Today, everything is covered by backup technology.
Larger organizations also have a problem ensuring that everything that needs to be backed up is actually backed up. According to the Veritas survey, IT professionals estimate that, on average, they wouldn’t be able to recover 20% of their data in the event of a complete data loss. It doesn’t help that many companies, if not all companies, have a problem with shadow IT.
There’s only so much companies can do to prevent loss when critical data is sitting on a server in a back closet somewhere, especially if the data is used for internal processes. “When it comes to production, it usually hits the company’s radar somewhere,” says Watkins. “There’s a new application or a new revenue-generating service.”
Not all systems can be easily found by IT so that they can be backed up. Ransomware hits, and then suddenly things are no longer working. Watkins recommends that companies do a thorough survey of all their systems and assets. This will usually involve leaders from every function, so that they can ask their people for lists of all critical systems and data that needs to be protected.
Often, companies will discover that things are stored where they shouldn’t be stored, like payment data being stored on employee laptops. As a result, the backup project will often run concurrent with a data loss prevention project, Watkins says.
6. Back up entire business processes
Ransomware doesn’t just affect data files. Attackers know that the more business functions they can shut down, the more likely a company is to pay a ransom. Natural disasters, hardware failures, and network outages don’t discriminate either.
After they were hit by ransomware, Kodiak Island’s VanDyke had to rebuild all the servers and PCs, which sometimes included downloading and re-installing software and redoing all the configurations. As a result, it took a week to restore the servers and another week to restore the PCs. In addition, he only had three spare servers to do the recovery with, so there was a lot of swapping back and forth, he says. With more servers, the process could have gone faster.
A business process works like an orchestra, says Dave Burg, cybersecurity leader at EY Americas. “You have different parts of the orchestra making different sounds, and if they’re not in sequence with each other, what you hear is noise.”
Backing up just the data without backing up all the software, components, dependencies, configurations, networking settings, monitoring and security tools, and everything else that is required for a business process to work can make recovery extremely challenging. Companies too often underestimate this challenge.
“There’s a lack of understanding of the technology infrastructure and the interconnections,” says Burg. “An insufficient understanding of how the technology really works to enable the business.”
The biggest infrastructure recovery challenges after a ransomware attack typically involve rebuilding Active Directory and rebuilding configuration management database capability, Burg says. It used to be that if a company wanted a full backup of its systems, not just data, that it would build a working duplicate of its entire infrastructure, a disaster recovery site. Of course, doing so doubled the infrastructure costs, making it cost prohibitive for many businesses.
Today, cloud infrastructure can be used to create virtual backup data centers, one that only costs money while it is being used. And if a company is already in the cloud, setting up a backup in a different availability zone—or a different cloud—is an even simpler process. “These cloud-based hot-swap architectures are available, are cost effective, and are secure, and have a great deal of promise,” says Burg.
7. Use hot disaster recovery sites and automation to speed recovery
According to Sophos, only 35% of ransomware victims are fully recovered within a week — down from 47% in 2023 and 52% in 2022. And a third takes a month or longer to recover. “I know companies who are spending a lot of money on tapes and sending them off to Iron Mountain,” says Watkins. “They don’t have the time to wait an hour to get the tapes back and 17 days to restore them.”
A hot site, one that’s available at the switch of a key, would solve the recovery time problem. With today’s cloud-based infrastructure, there’s no reason not to have one.
“It’s a no-brainer,” says Watkins. “You can have a script that copies your infrastructure and stands it up in another availability zone or another provider altogether. Then have the automation ready to go so that you hit play. There’s no restore time, just 10 or 15 minutes to turn it on. Maybe a full day if you go through testing.”
Why aren’t more companies doing this? First, there’s a substantial cost to the initial setup, Watkins says. “Then you need that expertise in house, that automation expertise and cloud expertise in general,” he says. “Then there are things like security controls that you need to set up ahead of time.”
There are also legacy systems that don’t transfer to the cloud. Watkins points to oil and gas controllers as an example of something that can’t be replicated in the cloud.
For the most part, the initial cost of setting up the backup infrastructure should be a moot point, Watkins says. “Your cost to set up the infrastructure is much less than paying the ransomware and dealing with the reputation damage.”
For companies struggling with this, one approach could be to focus on the most critical business processes first, suggests Tanner Johnson, principal analyst for data security at Omdia. “You don’t want to buy a million-dollar lock to protect a thousand-dollar asset,” he says. “Define what your crown jewels are. Establish a hierarchy and priority for your security team.”
There’s a cultural barrier to investing proactively in cybersecurity, Johnson admits. “We are a reactionary society, but cybersecurity is finally being seen for what it is: an investment. An ounce of prevention is worth a pound of cure.”
8. Test, test, and test again
“A lot of people are approaching backups from a backup point of view, not a recovery point of view,” says Mike Golden, senior delivery manager for cloud infrastructure services at Capgemini. “You can back up all day long, but if you don’t test your restore, you don’t test your disaster recovery, you’re just opening yourself to problems.”
This is where a lot of companies go wrong, Golden says. “They back it up and go away and are not testing it.” They don’t know how long the backups will take to download, for example, because they haven’t tested it. “You don’t know all the little things that can go wrong until it happens,” he says.
It’s not just the technology that needs to be tested, but the human element as well. “People don’t know what they don’t know,” Golden says. “Or there’s not a regular audit of their processes to make sure that people are adhering to policies.”
When it comes to people following required backup processes and knowing what they need to do in a disaster recovery situation, the mantra, Golden says, should be “trust but verify.”
What steps should companies take if they’ve experienced a ransomware attack
The US Cybersecurity and Infrastructure Security Agency (CISA) has a framework for companies to follow that covers the main steps that need to be taken after a ransomware attack.
Evaluate the scope of damage: The first step is to identify all affected systems and devices. That can include on-premises hardware as well as cloud infrastructure. CISA recommends using out-of-band communications during this stage, such as phone calls, to avoid letting the attackers know that they have been discovered and what actions you are planning to take.
Isolate systems: Remove affected devices from the network or turn off their power. If there are several affected systems or subnets, take them offline at the network level, or power down switches or disconnect cables. However, powering down devices might destroy evidence stored in volatile memory, so should be a last resort. In addition, protectively isolate the most mission-critical systems that are still untouched from the rest of the network.
Triage affected systems for recovery: Prioritize systems critical for health or safety, revenue generation, and other critical business services as well as the systems that they depend on. Restore from offline, encrypted backups and golden images that have been tested to be free of infection.
Execute your notification plan: Depending on your cyber incident response and communications plan, notify internal and external teams and stakeholders. These can include the IT department, managed security service providers, cyber insurance company, corporate leaders, customers, and the public, as well as government agencies in your country. If the incident involved a data breach, follow legal notification requirements.
Containment and eradication: Collect system images and memory captures of all affected devices, as well as relevant logs and samples of related malware and early indicators of compromise. Identify ransomware variant and follow recommended remediation steps for that variant. If data has been encrypted, consult federal law enforcement for possible decryptors that may be available. Secure networks and accounts against further compromise, since the attackers may still have their original access credentials or obtained more during the breach. In addition, extended analysis should be conducted to find persistent infection mechanisms to keep them from reactivating.
How long does it take to recover from ransomware?
According to Sophos, only a minority of ransomware victims recover in a week or less. On average, 35% took less than a week. About a third took between a week and a month. And the final third, 34%, took a month or more to recover. Only 7% of victims recovered in less than a day — and 8% of victims took three months or longer.
Recovery times are significantly reduced, however, if a company has good backups.
If a company’s backups were also compromised, only 25% of companies recovered in less than a week. But if the backups were not compromised, 46% of companies took less than a week to get back on their feet.
Ransomware best practices for prevention
CISA has a detailed list of best practices for preventing ransomware.
Backups: CISA recommends maintaining offline, encrypted backups of critical data and testing these backups and recovery procedures on a regular basis. Enterprises should also have golden images of critical systems, as well as configuration files for operating systems and key applications that can be quickly deployed to rebuild systems. Companies may also consider investing in backup hardware or backup cloud infrastructure to ensure business continuity.
Incident response plan: Enterprises should create, maintain, and regularly exercise a cyber incident response plan and associated communication plan. This plan should include all legally required notifications, organizational communications procedures, and make sure that all key players have hard copies or offline versions of this plan.
Prevention: CISA recommends that companies move to a zero-trust architecture to prevent unauthorized access. Other key preventative measures include minimizing the number of services exposed to the public, especially frequently targeted services like remote desktop protocol. You should conduct regular vulnerability scanning, regularly patch and update software, implement phishing-resistant multi-factor authentication, implement identity and access management systems, change all default admin usernames and passwords, use role-based access instead of root access accounts, and check the security configurations of all company devices and cloud services, including personal devices used for work. CISA also has specific recommendations for protecting against the most common initial access vectors, such as phishing, malware, social engineering, and compromised third parties.
Related website: https://www-csoonline-com.cdn.ampproject.org/c/s/www.csoonline.com/article/571131/ransomware-recovery-8-steps-to-successfully-restore-from-backup.html/amp/
AH
Comments
Post a Comment