How to identify attacks on your server and mitigate it.

Giedrius

This article explains how to check if your server is facing an attack and how to manage it. The steps below are the suggestions that you can use to check your server and see if it could be under some sort of attack.

One of the first things you could do in case you feel the server/website and its tasks are running slowly is to simply check your server load, it could be done with the  following commands:

If you are not sure or you don't know, you can find out the number of processors using these commands:

grep processor /proc/cpuinfo | wc -l

nproc

These commands will show just a number of your processors on the server. For example 1, 2, 4 and etc

Now you can check the server load using the other commands:

uptime

cat /proc/loadavg

The result of this command is very simple, but you could notice some anomalies here. An example of the uptime command output:

The output of the command shows how long is your server is up, how many users are currently logged in, and the load average, which provides values of 1 minute, 5 minutes, and 15 minutes. Considering we have 1 processor, the load values seem normal and not out of the ordinary. However, if the load average would be something like 1.50, 2.23, 8.14, this could be identified as a potential issue. The 1.00 stands for 100% of CPU utilization. So if the values are 1.50, 2.23, 8.14, they could be read as follows:

• 1.50 meaning that the CPU is overloaded by 50% during the last 1 minute;
• 2.23 meaning that the CPU is overloaded by 123% during the last 5 minutes;
• 8.14 meaning that the CPU is overloaded by 714% during the last 15 minutes.

While 50% or even 123% in this short time could be acceptable, the 3rd value should raise a concern. However, seeing the values decreasing could translate that the 'issue' is disappearing. While if the load values were 8.14, 2.23, 1.50 this could say that the load is just increasing right now.

The load values could show a potential issue regarding the server being attacked. The server load can rise according to the requests/traffic your server receives, however it could not be related to attacks at all, so we should check further and the next step is network traffic.

In a Linux environment, we could use netstat. This command could help to monitor the network and identifying possible issues. Here are a few general commands:

The command shows the number of connections each IP has with the server. It helps to identify the IPs that are connecting to the server most often. A single IP could be accessing your server a few times. For example, if you host some online resources, it might be normal to receive multiple connections from the same IP, as some smaller companies or networks with a shared IP are accessing your online resource. So you should include these possibilities depending on your server. However, let's say that one IP has 100+ connections to your server, this might seem suspicious and you should recheck it.

netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -r

A few more netstat commands for general monitoring.

This command shows all the active connections to the server:

netstat -na

This command shows all the specific traffic(in example port :80) by IPs :

netstat -an | grep :80 | sort

This command shows the number of connections by IP via TCP and UDP ports:

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

All these commands as well as other options that netstat provides can help you to identify if there is something incorrect on your server network.

Now let's say you visited your website and noticed it runs slowly, you get 5xx errors. You then access the server and notice it's responding slower than usual - this could be a moment you can start checking your server load first. It could confirm something is wrong if the load is heavy. Then you could check the connections to your server and perhaps identify that a few IPs are connecting to the server hundreds of times. If these IPs are unknown, this could be considered as an attack against your server.

The next phase should be mitigating the attacks. Here are a few methods on how to do that.

If you noticed an unknown IP or a few that are just generating connections to your server and you want to stop that, you can block it using iptables, for example:

iptables -A INPUT -s 111.11.1.1 -j DROP(or REJECT)

service iptables save

service iptables restart

More information about how to use iptables in this tutorial.

Another thing is if you stopped everything, but yet the server is slow, there could be many Apache processes stuck due to previous connection attempts, you may try to simply kill them all and then start the Apache freshly(this usually takes ups to a few minutes or quicker):

killall -KILL httpd

service httpd start

Note. This article is a simple way to check and try to find out if your server is under attack when you have any suspicions. There are many other tools, methods, and procedures that you can use as well. This is just something to get you started.