The constant evolution of cyber threats requires innovative approaches to safeguard information networks. Intrusion Detection Systems (IDS) play a pivotal role in securing networks by monitoring and analyzing network traffic to identify suspicious activity. With the growing reliance on network technology and the proliferation of IoT devices, cyber attacks have become more sophisticated and frequent. These attacks not only target the IoT infrastructure but also pose severe threats to national security, corporate data integrity, and public safety. A robust IDS is essential to detect and prevent these malicious activities. Traditional IDS models predominantly rely on supervised learning techniques, requiring extensive datasets that include both normal and attack instances for training. However, this dependency on historical data makes them less effective against zero-day or previously unknown attacks. To address these challenges, researchers have explored various machine learning approaches, including semi-supervised techniques, to enhance IDS’s capability in detecting unknown attacks.
1. Create Attack Combinations
To evaluate the effectiveness of a supervised learning model in detecting unknown attacks, we first need to create different sets of attack types to be excluded from the datasets. This process involves formulating various combinations such as individual attacks (a1, a2, a3, a4), pairs of attacks ({a1, a2}, {a1, a3}, {a1, a4}, {a2, a3}, {a2, a4}, {a3, a4}), triplets of attacks ({a1, a2, a3}, {a1, a3, a4}, {a2, a3, a4}, {a1, a2, a4}), and other more complex combinations. By purposefully excluding certain attack types from the training datasets, we can assess how well the model performs when it encounters unfamiliar attack patterns during the testing phase.
Creating these combinations is crucial for understanding the model’s ability to generalize beyond its training data. This step examines whether the model can detect unseen attacks efficiently. The methodology involves systematically removing specific sets of attacks from the training datasets and generating new combinations for analysis. For each combination, we will perform 10 stratified runs to ensure a balanced assessment of model performance across different scenarios. By evaluating the model’s accuracy in these test scenarios, we can gain insights into its robustness and capability to identify zero-day attacks.
2. Conduct Stratified Runs
After creating the attack combinations, the next step is to conduct stratified runs to evaluate the model’s performance. For each combination, we perform 10 stratified runs and compute the average accuracy over these 10 folds. Stratified runs ensure that each fold of the dataset maintains the same proportion of each class, providing a reliable estimate of the model’s performance. This systematic approach helps eliminate biases that might arise from an imbalanced dataset, leading to more accurate and consistent evaluation metrics.
In addition to computing average accuracy, it’s essential to calculate other performance metrics such as average precision, recall, and F1-score for each combination length (one, two, three, or four attacks). The precision metric helps assess how many of the detected attacks are actual attacks, while recall measures the model’s ability to identify all true attacks. The F1-score balances precision and recall, providing a comprehensive evaluation of the model’s overall performance. For instance, by summing the 10-fold performance metrics of individual attack combinations (a1, a2, a3, a4) and dividing by the number of combinations, we can derive the average performance for single attack combinations. This methodology applies similarly to combinations of two, three, and four attack types, allowing for a thorough assessment of the model’s ability to detect unknown attacks.
3. Visualize Results
The final step is to visualize the results to clearly understand how the model performs with different attack combinations. By graphing accuracy and F1-score values for each combination length (one through four attacks), we can easily spot trends and patterns in the model’s ability to identify unknown attacks. This visual representation offers valuable insights into where the model excels or struggles when facing new attack patterns.
Visualizing outcomes helps compare the effectiveness of various strategies, such as supervised learning with and without noise data, and semi-supervised learning techniques like usfAD. The graph emphasizes how the model’s performance shifts when different groups of attacks are excluded from the training datasets. It can show sharp declines in accuracy and F1-scores as more attack types are left out during training, highlighting the challenges traditional supervised models face. This visualization is crucial in demonstrating the superior consistency of semi-supervised methods, such as usfAD, which maintain stable accuracy and F1-scores even when the test data includes unknown attacks.
In summary, systematically evaluating and visualizing different attack combinations gives a comprehensive view of the model’s strengths and weaknesses in detecting zero-day attacks. This method underscores the necessity for innovative techniques like usfAD to improve the effectiveness of Intrusion Detection Systems (IDS) and adapt to the dynamic nature of cyber threats.
