Linux VPS Server Setup, Security, and Optimization

This guide walks through the secure setup of a RHEL-based VPS server step by step. Every command is explained so you know what you're doing and why.

Before You Begin

A new VPS server is visible on the network from the moment it boots. Automated bots continuously scan the entire internet and will find your server within minutes. That's why security must be configured before you install anything else.

You'll need:

• Your VPS server's IP address
• The root passwords (provided by your hosting provider)
• A terminal on your local machine

1. Log in and update the system

Log into the server for the first time as the root user. This is the only time you'll use the root account directly.

ssh root@SERVER_IP

Update the system immediately. Many server breaches exploit known vulnerabilities that could have been prevented with a simple update.

dnf update -y    # update all packages to the latest version (-y auto-accepts)

2. Create a new user

The root user has unlimited privileges. If an attacker gets in as root, the entire server is immediately compromised. Create a regular user for daily use.

adduser username      # create a new user
passwd username       # set a password (RHEL doesn't prompt for it automatically)

Grant the user wheel group privileges.

usermod -aG wheel username    # add the user to the wheel group

Switch to the new user and verify that sudo works.

su - username
sudo whoami    # should print: root

3. SSH key-pair

Password-based login is vulnerable to brute-force attacks, where a bot tries thousands of passwords per second. An SSH key pair is a cryptographically secure alternative, without the correct key, no one gets in. Do this on your own machine, not on the server.

ssh-keygen -t ed25519 -C "your@email.com"    # create a key pair

Copy the public key to the server.

ssh-copy-id username@SERVER_IP    # add the key to the server

4. Firewall

The firewall determines which network traffic is allowed to the server. First block everything, then allow only necessary ports.

Firstly install firewall.

sudo dnf install firewalld -y

sudo systemctl enable firewalld    # start automatically at boot

Open the necessary ports. Remember to use the same SSH port you chose in step 4.

sudo firewall-cmd --permanent --add-port=2222/tcp    # SSH (custom port)
sudo firewall-cmd --permanent --add-service=http     # HTTP
sudo firewall-cmd --permanent --add-service=https    # HTTPS
sudo systemctl start firewalld     # start immediately

Reload settings.

sudo firewall-cmd --reload    # apply changes

Check status.

sudo firewall-cmd --list-all    # shows active rules

4. Hardening SSH settings

For nano commands to work in this enviroment, nano needs to be installed.

sudo dnf install nano -y

Now that key-based login works, let's lock down SSH further.

sudo nano /etc/ssh/sshd_config

Find these lines in the file and modify them as follows. Changing the port from the default 22 significantly reduces automated scanning attacks, as most bots only target port 22.

Port 2222                   # change default port 22 → reduces bot scanning
PermitRootLogin no          # disable root login entirely
PasswordAuthentication no   # disable password login
PubkeyAuthentication yes    # ensure key-based login is enabled

Save the file (Ctrl+X, then Y, then Enter) and restart SSH.

sudo systemctl restart sshd

Important!

Do not close your current connection before testing the new port. Open a new terminal window and try logging in.

ssh -p 2222 username@SERVER_IP

6. Fail2ban

Fail2ban monitors logs and automatically blocks IP addresses that attempt to log in too many times unsuccessfully. This is an important additional layer on top of the firewall.

Installation requires the EPEL repository.

sudo dnf install epel-release -y
sudo dnf install fail2ban -y

Create a local configuration file. Use jail.local instead of editing jail.conf directly, since updates could overwrite the latter.

sudo nano /etc/fail2ban/jail.local

Write the following into the file. Remember to change the port to match your SSH port.

[DEFAULT]
bantime  = 1h     # IP is blocked for one hour
findtime = 10m    # monitoring time window
maxretry = 5      # number of allowed failures

[sshd]
enabled = true
port    = 2222    # your SSH port

Start Fail2ban and set it to start automatically.

sudo systemctl enable fail2ban    # start automatically on reboot
sudo systemctl start fail2ban     # start immediately
sudo fail2ban-client status sshd  # check status

7. Automatic security updates

Manual updates are easy to forget. unattended-upgrades installs security updates automatically in the background.

sudo dnf install dnf-automatic -y

Now enable the automatic updates.

sudo systemctl enable --now dnf-automatic.timer    # scheduled updates

Optionally edit the settings.

sudo nano /etc/dnf/automatic.conf

Inside the file.

apply_updates = yes    # install updates automatically

8. Swap memory

VPS servers often have limited RAM. Swap is disk space used as backup memory when RAM runs out. Without it, the server may crash during traffic spikes. Create a 2GB swap file. Adjust the size as needed 1G is enough for small servers, larger ones may use 4G.

sudo fallocate -l 2G /swapfile    # reserve 2 gigabytes from disk
sudo chmod 600 /swapfile          # only root may access
sudo mkswap /swapfile             # format as swap
sudo swapon /swapfile             # activate

Make the setting permanent by adding swap to /etc/fstab, which defines what gets mounted at boot.

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Adjust the swappiness value. The default (60) moves data to swap fairly aggressively. A value of 10 keeps data in RAM longer and uses swap only when necessary.

echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p    # load settings immediately

9. Backups

Security hardening won’t help if your disk fails or you accidentally delete important files. Backups are essential.

Install rsync, an efficient file synchronization tool.

sudo dnf install rsync -y

Create a script that automatically copies the most important directories. The script saves copies in date-stamped folders and automatically cleans up versions older than 30 days.

sudo nano /usr/local/bin/backup.sh

Add the following content:

#!/bin/bash

DATE=$(date +%Y-%m-%d)    # today's date
DEST="/backup/$DATE"      # destination folder

mkdir -p $DEST                          # create folder if missing
rsync -av /etc/ $DEST/etc/              # system configuration
rsync -av /var/www/ $DEST/www/          # web content
rsync -av /home/ $DEST/home/           # users

find /backup -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \;    # remove copies older than 30 days

Make the script executable and schedule it to run every night at 03:00.

sudo chmod +x /usr/local/bin/backup.sh
sudo crontab -e

Add this line at the end of the cron editor and save:

0 3 * * * /usr/local/bin/backup.sh    # run every night at 03:00

10. Checklist

sudo firewall-cmd --list-all               # firewall on and rules correct?
sudo systemctl status sshd                 # SSH working?
sudo systemctl status fail2ban             # Fail2ban working?
sudo swapon --show                         # swap active?
sudo systemctl status dnf-automatic.timer  # automatic updates OK?