Currently, multiple DDoS protection vendors regularly publish trends and analysis regarding DDoS attacks. These reports focus mostly on duration, magnitude, frequency, and attack vectors. The readers of these reports are normally checking if the data matches their current experiences or, are curious if the DDoS landscape has reached the point in which they should consider buying a DDoS solution. The fact unknown to the readers is the survivorship bias in DDoS reporting.
Survivorship bias is an inaccuracy in a statistic because a filter was applied before the data collection. Only the “survivors” get reported. The observer is missing what is missing. Some interesting examples of survivorship bias can be found on YouTube.
From the perspective of DDoS, survivor bias occurs in the reporting because attacks are mitigated and detected differently. This article discusses examples of how survivorship bias can skew the three main categories of these trend reports: duration, magnitude and frequency.
One of the most common statistics for DDoS is attack duration. The longer the duration, the longer a service may be unavailable. SLAs and other financial impacts usually worsen dramatically if duration starts to rise, especially the average duration of attacks.
The simplest example of survivorship bias effecting duration is for the rudimentary DDoS solutions that send a blackhole route upstream when a threshold is reached. Once the blackhole is sent upstream, the local solution cannot detect the attack anymore. To the local solution, the attack is over. The administrator waits a set amount of time (sometimes hours), then unblocks the attack and hopes the attack is over by then. Any statistics checking for duration will be artificially limited for attacks above the blackhole threshold. The result: The local solution will report that large attacks are shorter in duration than they actually are.
Another example is an enterprise with a hybrid on-premise and cloud DDoS solution, and they get targeted by a very large attack. To their on-premise equipment the attack lasts sixteen minutes, but when they get the cloud provider bill at the end of the month, they are in for a shock. The cloud provider says they saw the attack last 33 minutes. The total attack was actually 49 minutes in duration. The on-premise solution could no longer see the attack once it was redirected to the cloud, and the cloud provider didn’t see the traffic until it was redirected. Both DDoS vendors would report this attack, report it differently, and both have their data affected by survivorship bias.
The next category that gets a lot of interest is frequency. The more numerous the attacks, the harder it is for a SOC/NOC to keep up with redirections and/or mitigations. For cloud solutions, increasing mitigating occurrences can incur more costs as well.
Keeping with the same example as above, consider the time it takes to redirect to the cloud. Attacks that are shorter in duration than the time it takes to redirect to the cloud will never be seen by the cloud. If it takes 15 minutes to redirect to the cloud, only attacks that “survive” for more than 15 minutes will be collected in the cloud provider’s statistics. The result: The on-premise solution will report a much higher attack frequency than the cloud provider.
This brings our focus to another key variable in the attacks, magnitude. The magnitude of the DDoS is the most important factor that can lead to collateral damage for other services, systems, and even customers.
In all the preceding examples, magnitude played a big role in the bias. In the blackhole example, only LARGE attacks were redirected. In the cloud provider example, most enterprises will not redirect small attacks to the cloud due to cost. Smaller attacks will be either mitigated by the company, or perhaps even tolerated as being unmitigated, if there is no collateral damage. The result: The cloud provider average Gbps will be much higher than the average for the attacks seen at the customer site.
Knowing that a bias affects the reporting, a potential customer can consume the data with better judgement. Even with the survivorship bias, trend reports can offer guidance of which solution companies may need. The key word is trend. With one point of view, a trend of increasing DDoS attacks can be illuminating. When analyzing a trend report, be sure to look for year over year comparisons, or previous reports.
Another recommendation is to gather data from multiple sources, especially from different types of solutions. Even if you are not considering a particular type (e.g. on-premise), get a report from a vendor that specializes in it. Perhaps that data might be the missing survivors you are not seeing in the solution type you are considering (e.g. cloud).
Being mindful of the survivorship bias inherent in DDoS reporting will lead to a more informed decision on your DDoS solution choice.