Machine Learning Based Approach to Anomaly and Cyberattack Detection in Streamed Network Traffic Data

With the development of the fifth-generation networks and artificial intelligence technologies, new threats and challenges have emerged to wireless communication system, especially in cyber-security. In this project, the performance of a solution providing stream processing is evaluated, and its accuracy in the classification of suspicious flows in simulated network traffic is investigated. The concept of the solution is fully disclosed along with its initial evaluation in a real-world environment. At the heart of the engine are machine learning algorithms implemented using the TensorFlow library, providing the cutting edge in network intrusion detection. The tool allows easy definition of streams and implementation of any machine learning algorithm. 1.5 PROPOSED METHODOLOGY During the software development and testing phase, different network flow data formats were considered. Monitoring of network traffic is possible using the standards such as NetFlow or sFlow. These flowbased standards provide a metadata-based view of activity on the network. sFlow provides an outline of network traffic, while reducing the processing load on the network elements. This comes at the expense of its accuracy. We use Netflow for anomaly detection. NetFlow can be defined as a schema for collecting, aggregating and logging the data about the network traffic. NetFlow collectors can be deployed as either hardware or software-based probes. The tool that is used to collect the traffic from the machines in the experimental environment is an open-source solution available for the Unix environment, called fProbe. The process starts from the moment the NetFlow data is delivered from the probe to the collector – i.e., dedicated software or hardware that collects information from probes. A software-based collector is used, which can collect and export NetFlow flows generated by gateways, switches, border routers or any other device that can export flow data in NetFlow v5/v9. The collector is connected to the streaming tool, i.e., Apache Kafka. The collector then allows manipulating both the input and outputting schema of the NetFlow frame. The redundant parameters that were not needed in the process of machine learning or prediction can be omitted immediately, resulting in less data passing through the stream and lowering the overall system load.