Corero
Blog & News

How Survivorship Bias Affects DDoS Reporting Part 2: Attack Vectors

In Part 1 of this series we described and got into detail about what survivorship bias actually is. The number one thing to remember is: Survivorship bias is an inaccuracy when calculating a statistic because a “survival” filter was applied before the data collection. Only the “survivors” get reported, meaning only the data that is collected in that scenario. Ultimately, the observer is missing what is missing. Meaning, you can’t always draw a clear conclusion and see the larger picture when significant data points may be absent.

When the topic of DDoS Attack Vectors is examined, survivorship bias can greatly impact results presented in all kinds of DDoS reports, often making it difficult to truly understand the overall threat landscape. Statistics that make headlines are strongly influenced by what is actually measured. Since a system can only report on what it can detect, systems that do not detect certain types of attacks will simply NOT have those vectors listed in their reports. This is the definition of survivorship bias. This explains why when it comes to DDoS Threat Intelligence reports, we often see a wide variety of claims depending on the type, size or duration of attack that is being reported. This can cause reports from different organizations or observation points to sometimes appear to contradict each other.

When the customer uses a specific mitigation strategy for certain types of attacks, the operator may choose not to send certain types of attacks or traffic to the specific mitigation. This may be because the mitigation strategy may not work for that type of attack. For example, if a customer is testing two cloud providers and provider 1 is good at TCP attacks, while provider 2 is better at UDP fragmentation attacks the customer will pick the provider based on which attacks they can mitigate. This will lead to the providers having survivorship bias in their reporting. Overall, providers will tout their success rate, but they may not be aware of all the attacks that are not handled well. Alternatively, there may have been attacks that were not sent to the cloud provider as swinging traffic to the cloud would have been too costly or ineffective (too slow).

Attack vector changes will also depend on the mitigation strategies used, while some attackers send multi-vector attacks right at the onset, others only change vectors if the attack is not successful. For the cloud example, if the on-premises equipment mitigates an attack, the customer would not need to swing traffic to the cloud. If the attack changes vectors and increases in size or has an attack leakage, that causes the customer to subsequently initiate a cloud swing, the cloud provider would only be able to see the new and subsequent vectors. In that sense, the cloud provider is much more likely to only see vectors that are not handled well by the on-premises equipment. If the cloud provider does block the subsequent vectors effectively, the on-premises equipment will not see them. Similarly, attacks reaching the on-premises equipment are the survivors of the upstream mitigations that may have already been applied. Both types of solution have a survivorship bias. When it comes to selecting certain providers for certain types of attack mitigation, it can cause those providers to see only what they are good at mitigating, effectively causing them to overstate the fraction of those attacks and their efficiency at defending against that type of DDoS.

As we have said before, to get the bigger picture it is important to gather data from multiple sources and from various solutions. There is a real chance that the individual data points might be only reporting on the “survivors”. Being mindful of the survivorship bias inherent in DDoS reporting will lead to a more informed decision on your DDoS solution choice.