• Symptoms of compromised server
  1. When servers are invaded by inexperienced attackers or automated attack programs, they often consume 100% of the resources. They may consume CPU resources to mine digital currencies or send spam, or they may consume bandwidth to launch DoS attacks.
  • So the first manifestation of the problem is that the server “slowed down”. This may be due to the slow opening of pages on the website, or the e-mail taking a long time to send out.
  • So what should you look at?

Check 1-Who is currently logged in?

You must first check who is currently logged on to the server. It is not complicated to find that the attacker logs in to the server to operate.

The corresponding command is w . Running w will output the following results:

08:32:55 up 98 days, 5:43, 2 users, load average: 0.05, 0.03, 0.00


root pts / 0 08:26 0.00s 0.03s 0.02s ssh root @ coopeaa12

root pts / 1 08:26 0.00s 0.01s 0.00sw

The first IP is a British IP, and the second IP is a Vietnamese IP. This is not a good sign.

  1. Stop and take a deep breath, don’t panic just kill their SSH connection. Unless you can prevent them from entering the server again, they will come in quickly and kick you out in case you go back again.
  2. Please refer to the section “What to do after an invasion” at the end of this article to see what to do if you find evidence of an invasion.
  3. The whois command can take an IP address and tell you all the information of the organization registered with the IP, including the country information.

Check 2-Who ever logged in?

The Linux server records which users, from which IP, when they logged in and how long they logged in. Use the last command to view this information.

  1. The output looks like this:root pts / 1 Thu Nov 30 08:26 still logged in

    root pts / 0 Thu Nov 30 08:26 still logged in

    root pts / 1 Thu Nov 30 08: 24-08: 26 (00:01)

    root pts / 0 Wed Nov 29 12:34-12:52 (00:18)

    root pts / 0 Mon Nov 27 13:32-13:53 (00:21)

    Here you can see that the UK IP and Vietnam IP appear alternately, and the top two IPs are still logged in. If you see any unauthorized IP, then please refer to the last chapter.

  2. The history after login will be recorded in the binary / var / log / wtmp file (LCTT Annotation: The author should write it wrong here, modify it according to the actual situation), so it is easy to delete. Usually the attacker will delete this file directly to cover up their attacks. Therefore, if you run the last command and only see your current login, then this is a bad signal.
  3. If there is no login history, please be careful and continue to pay attention to other clues of the invasion.

Check 3-Review command history\

  1. Attackers at this level usually do not pay attention to obscuring the history of commands, so running the history command will show everything they have done. You must pay attention to whether you use wget or curl commands to download unconventional software such as spam robots or mining programs.
  2. The command history is stored in the ~ / .bash_history file, so some attackers will delete the file to hide what they did. As with the login history, if you run the history command but nothing is output, it means that the history file has been deleted. This is also a bad signal, you need to check the server very carefully. (LCTT translation, if there is no command history, it may be your configuration error.)

Check 4-Which processes are consuming CPU?

  1. The kind of attackers you often encounter often don’t cover up what they do. They will run some processes that consume CPU in particular. This makes it easy to discover these processes. Just run top and look at the first few processes.
  2. This can also show those attackers who are not logged in. For example, someone may be using unprotected mail scripts to send spam.
  3. If you do n’t know the top process, you can Google the process name or use losf and strace to see what it does.

Using these tools, the first step is to copy the PID of the process from top and then run:

    1. strace -p PID

    This will show all system calls called by the process. It will produce a lot of content, but this information can tell you what this process is doing.

    1. lsof -p PID

    This program will list the files opened by the process. You can understand what it is doing by viewing the files it accesses.

Check 5-Check all system processes

Unauthorized processes that do not consume CPU too much may not show up in top , but it can still be listed by ps . The command ps auxf can display clear enough information.

  1. You need to check every unknown process. Running ps frequently (which is a good habit) can help you find strange processes.

Check 6-Check the network usage of the process

Iftop ‘s function is similar to top , it will arrange the process of sending and receiving network data and their source and destination addresses. Processes like DoS attacks or spam robots are easily displayed at the top of the list.

Check 7-Which processes are listening for network connections?

Usually an attacker will install a backdoor program to listen to the network port to accept commands. The process will not consume CPU and bandwidth during the waiting period, so it is not easy to find through commands such as top .

  1. The lsof and netstat commands will list all the networking processes. I usually let them bring the following parameters:

lsof -i

netstat -plunt

You need to pay attention to those processes in the LISTEN and ESTABLISHED state, these processes are either waiting for connection (LISTEN), or already connected (ESTABLISHED). If you encounter a process you do n’t know, use strace and lsof to see what they are doing.

What should I do after being hacked?

  1. First, do n’t be nervous, especially when the attacker is logged in. You need to regain control of the machine before the attacker is alerted that you have discovered him. If he finds that you have discovered him, then he may lock you from logging on to the server, and then start ruining the corpse.
  2. If your technology is not very good then just shut it down. You can run one of the two commands shutdown -h now or systemctl poweroff on the server . You can also log in to the hosting provider’s control panel to shut down the server. After shutting down, you can start to configure the firewall or consult the supplier for advice.
  3. If you are confident in yourself and your hosting provider also provides upstream firewalls, then you only need to create and enable the following two rules based on this:
    1. Only allow SSH login from your IP address.

    Block anything but this, not just SSH, but any protocol on any port.This will immediately close the attacker’s SSH session, leaving only you with access to the server.

  4. If you cannot access the upstream firewall, you need to create and enable these firewall policies on the server itself, and then use the kill command to close the attacker’s SSH session after the firewall rules take effect . (LCTT Annotation: The local firewall rules may not block the established SSH session, so for security reasons, you need to manually kill the session.)
  5. Finally, there is a way, if supported, to log in to the server through an out-of-band connection such as a serial console, and then stop the network function through systemctl stop network.service . This will close the network connections on all servers, so you can slowly configure those firewall rules.
  6. After regaining control of the server, don’t think everything will be fine.Don’t try to repair this server, and then use it. You never know what the attacker has done, so you can never guarantee that this server is still secure.
  7. The best way is to copy out all the data and then reinstall the system. (LCTT Annotation: Your program is not trusted at this time, but the data is generally ok.)


Codec Networks provides IT Trainings from EC Council CEH ECSA, LPT, CHFI, Network Security, Penetration Testing, ISACA, ISC2, PECB ISO 27001LA LI, Cisco Networking CCNA CCNP, Linux Administration RHCE, Prog Languages JAVA, Advanced Java, android development. We also offer B2B Industry Solutions and Services in IT | Information|Cyber Security in Delhi NCR India.

View all posts by

Leave a Reply

Your email address will not be published. Required fields are marked *