IT security technologies have their origins in the investigation of statistical models. They detect anomalies in huge data volumes. A glance into the research department of RadarServices, Europe’s leading provider of continuous IT security monitoring, shows how models are tested before they are put into practice.
Determining the research area
Quick detection of anomalies in the IT landscape is the prerequisite for identifying current cyber attacks in time. However, with large enterprises, big data is an issue. To find the needle in the haystack, we must first have an idea of what that needle looks like. “Our Security Intelligence Team provides us with this information. They are the experts that keep an eye on everything that happens in our clients’ IT landscape every day. Let’s say you want to determine a certain pattern in large data volumes and ask us to create a model for that purpose,” says Andreas Esders-Kopetzky, RadarServices Research. “IT security officers of companies who are our customers may come up with similar ideas. Such concerns from all kinds of industries are brought to our attention. When we know what to look for in large data volumes, we test all eligible models from our experience, but also from completely different areas like econometrics or bioinformatics. Different customers have different demands, so we create different solution processes, accordingly,” Esders-Kopetzky continues. “Our Research Team picks up outside stimuli and develops them further. Additional patterns are evaluated, and everything is aggregated to form a superordinate strategy, in order to improve the entire IT risk detection and analysis process,” explains Christian Polster, responsible for research in the management of RadarServices.
The validation process
The Research Team’s efforts are aimed at finding a valid model to produce correct results in many different cases. “For example, our colleagues from the Security Intelligence Team might first provide us with an extract of data. We know that these data have interesting patterns. However, the systems in use did not detect those patterns automatically. So we consider a model and first evaluate what is unusual about these data. During the subsequent validation process, we try to identify where exactly anomaly and non-anomaly correlate,” Esders-Kopetzky explains.
An analysis is carried out to determine whether the error rate (false positives and false negatives) of the chosen model is acceptable. Once this comprehensive analysis and evaluation process has been successfully completed, the new model is accountable and, thus, reliable. To assure highest quality levels, this process might take months and the efforts of an entire research team with a wealth of experience in many fields.
Risk assessment before practical application
The model has been chosen and its adequacy tested with numerous data sets. To assess the risk of nondetection of anomalies despite detailed tests, a risk analysis is conducted in a separate step. Once again, it is evaluated whether the model in use detects enough anomalies without generating too much “unnecessary data” (false positives and false negatives). “Risk assessment is carried out in similar ways in each statistical scenario: for example, when a new vaccine is developed, its efficiency is weighed against any risks it harbours. We use the very same approach for risk assessment before we deploy a model,” Polster states.
Visualisation of model decisions
Understanding every detail of models and their limitations is important to win even the statistics experts’ trust in correct functionality. A key method here is a visualisation of model decisions. This is a valid option for models as long as they are not too complex. Models with high complexity, however, require comprehensive testing. These tests are used to demonstrate the inflow of data and visualise the anomalies actually shown by a model in a large number of scenarios at the end of the day. “These extensive validation, testing, risk analysis and visualisation processes are necessary to convince our security intelligence experts, our management and ourselves on a permanent basis that the model we use has been put to the acid test, and that we can rely on it to raise an alarm when it is meant to. Only then do we put it to practice and consider our work successfully completed.”
