Organizations understand the need for effective network security. The challenge, however, is how to effectively compare and contrast various products and services to find the options that will work best. A lack of industry practices or consistent testing standards makes it difficult to get a valid “apples to apples” comparison of different solutions.
In the classic movie Princess Bride, one character continuously proclaims that things are “inconceivable” even though they are happening right in front of his eyes. Finally, one of his compatriots says, “You keep using that word. I do not think it means what you think it means.”
Word matter. It’s particularly important when comparing different products or services that everyone is using the same definition for specific words. I just got back from attending the AWS re:Invent conference in Las Vegas. Walking around the Expo Hall floor and listening to the various vendor pitches can be overwhelming and confusing because many use the same terms and buzzwords, but have vastly different capabilities.
This concept also extends to the values and formulas used to calculate metrics and specifications that vendors use to market their products and services. For example, two vendors might claim that their network security appliance communicates at 1 Gbps—but if one is stating a theoretical maximum speed while the other is citing actual throughput results, the reality of the two claims is very different.
You can’t compare two outcomes if the scenarios used to produce them are different. This is a basic and essential principle. Whether you’re testing different medical treatments, or comparing different marketing campaigns, it’s important to minimize the variables other than the one thing you’re testing for.
When it comes to validating the performance of network security products, the environment and methodology is crucial. Different products need to be tested in the exact same environment and under the exact same conditions in order to obtain a valid comparison.
The lack of consistent network security performance test standards is one of the biggest challenges facing the industry—and the customers trying to make purchasing decisions. There are a handful of established players that conduct performance tests, but vendors are often able to pick and choose which ones are allowed to test their products, and frequently request specific test requirements designed to highlight specific aspects of their products.
Open, Transparent Cybersecurity Testing Standards
A new organization has been created to address these issues. NetSecOPEN was founded with the intent of creating open, transparent network security testing standards. The organization has significant momentum and support right out of the gate—with 11 prominent cybersecurity and test solutions and services providers, and testing laboratories announced as Founding Members.
According to a press release from NetSecOPEN, “The NetSecOPEN standard is designed to provide metrics that can be used to compare solutions fairly and understand the impact of different solutions on network performance under the same conditions. The goal is to examine the performance ramifications of a solution with all of that solution’s security features enabled, conveying the true costs of the solution.”
The goal of NetSecOPEN is to close the gap in performance between proprietary testing methodologies and metrics and how the network security solutions actually perform in the real world. The testing standard—which has been submitted to the IETF Benchmark Working Group—includes a real-world mix of traffic, with 400 encryption certificates and 10,000 unique URLs. It provides a more comprehensive and accurate scenario of what network security products face in the real world.
“There is great urgency for open, transparent standards for the testing of network security equipment that will prevent unfortunate surprises and aid equal comparisons of solutions,” said Brian Monkman, executive director of NetSecOPEN. “Today, security professionals face a Wild West of equipment specifications that, in the end, turn out to be mostly meaningless and virtually guarantee difficulties and disappointment upon deployment. NetSecOPEN was formed for exactly this reason. We are proud to see such progress, which brings us closer to making open network security testing standards a reality.”
The NetSecOPEN test standard is open and transparent—so every vendor and every customer knows exactly what is being tested and how. It can also adapt and evolve as network technologies and conditions change to ensure it remains an accurate reflection of real-world conditions. The most important thing, though, is that as many testing labs and cybersecurity vendors as possible embrace the NetSecOPEN standard so that prospective customers can compare and contrast network security products with confidence.