Anomaly detection on network logs using Isolation Forest and ElasticSearch

🔍 description

This POC connects to an Elasticsearch instance to collect network logs, applies anomaly detection using the Isolation Forest algo, and exports detected anomalies back to Elasticsearch. for further analysis.

🚀 features

• Connect to an Elasticsearch server
• Retrieves up to 1000 logs from the filebeat-* index.
• Detects anomalies using the isolation forest model
• exports anomalies into a dedicated index : anomalies-ml

📦 installation

sudo apt update
sudo apt install python3 python3-pip python3-venv -y

python3 -m venv venv
source venv/bin/activate

pip install -r requirements.txt

# create .env file

python collect_logs.py

⚙️elasticsearch Configuration

Make sur to have Elasticsearch running on localhost:9200

Create an index pattern

1. Open Kibana in your web browser (usually at http://IP_ELASTIC_SERVER:5601).
2. Go to the "Management" section.
3. Click on "Index Patterns".
4. Click on "Create index pattern".
5. Enter anomalies-ml as the index pattern.
6. Go to Dashboard and create a new dashboard.
7. Create a new visualization.
8. Select the "Bar vertical" visualization type.
9. Select the anomalies-ml index pattern.
10. Drag and drop the @timestamp field into the "X-axis" section.
11. Drag and drop the anomaly field into the "Y-axis" section and select "Count" as the aggregation.
12. Click on "Save" to save the visualization.