IT Briefcase Exclusive Interview: The Value of Security Testing

While many companies might put security testing at the bottom of their lists, it’s a critical step businesses need to take to ensure their solutions can properly protect and defend against cyber-attacks. There are numerous ways that enterprises can go about the testing of solutions, but it’s important to understand the industry best practices and recommendations when it comes to testing and evaluation. Some methods, such as open malware repositories, can cause more harm than good in the case of security testing – which is why Deep Instinct’s research team looked into the issue further.

1. How do enterprises go about testing and evaluating their security solutions?

The testing and evaluation of security solutions has been a hot topic within the cybersecurity industry, with continuous debate happening around the need for industry wide standards, tools, tactics and procedures (TTP’s) for testing. One of the most prominent ways enterprises traditionally go about testing and evaluating their security solutions is through known malware repositories. These repositories are an important source for threat intelligence providers and consumers, serving security professionals, malware researchers and others. One of their most common use-cases is for building up-to-date datasets that are to be run against security solutions during testing, evaluations and comparative studies. The results of such tests allow enterprises to rate solutions and assess overall IT-security posture. Recently, we’ve seen more IT professionals using one or two open repositories as a central, and in some cases, unique source for malware datasets.

Other than the malware datasets, many enterprises will also test false positive rates, review the solution’s architecture, deployment process, management console and evaluate its influence on performance at the end-point and network level.

2. What are the downfalls of using open repositories for testing malware?

While open repositories are mainly used as a threat intelligence source, they do present limitations such as false positives and non-malicious samples. To further examine the reliability and accuracy of open malware repositories, the research team here at Deep Instinct created a dataset of 130,000 random samples from a chosen popular repository and divided the dataset into file types and malware or non-malware samples. This was done based on Deep Instinct’s product classification, and correlated with updated multi-scanner data from multiple threat intelligence and file reputation databases. We also used the Deep Instinct’s malware classification model [1] to further refine the labeling and classification of PE files. Our research yielded the following results:

– Of the 130K files collected, only 60 percent of them were PE files (usually the most common form of malware for Windows – but this can also include false positives).

– More than a third of the files (35 percent) comprised of a mix of different types of simple text files (that don’t pose a true security risk).

– Only 34 percent were found to be relevant malicious samples for testing on Windows machines.

Almost two-thirds of the files posed no true security risk, leaving IT professionals with no indication of a solution’s ability to truly prevent or detect malware thus allowing enterprises to be attacked. This test demonstrates that creating datasets containing only randomly selected samples from free, open repositories will result in problematic, incomplete and unreliable anti-malware testing. This is the biggest downfall of blindly using an open malware repository for security testing.

3. What best practices should IT professionals follow in order to properly test security solutions?

In an effort to make sure security solutions are being tested properly, IT professionals need to test on relevant, accurate and vetted datasets that are representative of the current threat landscape. Testers should follow guidelines and continuously develop best practices that will ensure their tests are reliable, and can be used as a reference for comparison. We’ve outlined a few best practices below:

– Include Unknown Malware: Keep in mind that even the cleanest dataset originally acquired from an open, shared repository provides the tester only with seen, known malware. Known malware is less challenging, so testers should opt to include unknown malware in their tests.

– Don’t Rely Solely on Hash Blacklisting: Verify that the solution does not rely only on hash blacklisting because it is easy to bypass in the real world, and make sure the solution can detect new malware.

– Variation is Key: Make the repository as varied as possible by bundling samples from as many different families so the results are representative.

– Create Mutations: Create malware mutations based on available tools and frameworks for file mutation and manipulation.

– False Positive Rates (FPR): Make sure that the rate of false positive detections is manageable and will not create overhead your organization cannot deal with.

– Detection is only a part of a bigger picture: Products differ in many aspects, not only in detection capabilities. It is important to take other factors into consideration, including architecture, deployment, UI/UX, performance and stability.

IT professionals can use the abovementioned best practices and tools to make sure that they’re testing their security solutions on all fronts. As a result, enterprises can have peace of mind that their solutions are prepared to defend against cyber threats – especially as new malware is becoming increasingly sophisticated.

Shimon Noam Oren is Head of Cyber Intelligence at Deep Instinct, the first company to apply deep learning to cybersecurity. Shimon has extensive background in cyber security, inbound product management, intelligence, and leadership of change processes and organizational transformations. Prior to his role at Deep Instinct, he was the Head of Cyber Research for the Israel Defense Forces (IDF) elite intelligence unit.