If you is using cloud VPS like AWS or DigitalOcean, they making journald write logs in RAM. This meaning when you reboot your Linux, all logs is gone forever. This is very bad for security because if a hacker come to your server and do bad things, then he reboot the server, you cannot see what he did. We need to make the logs stay in the hard disk. In this tutorial, I show you how to do this persistent logging and also how we can send the logs to another server far away so even if the hacker delete your server, you still have the proof. I will explain everything step by step so you do not get confused.
First, we must make the local journald configuration to save everything on the hard disk. We do this by creating a special config file. Do not edit the main config file because if update come, it can overwrite your changes. Instead, we make a directory for custom configs. Run this command in your terminal to create the folder:
sudo mkdir -p /etc/systemd/journald.conf.dNow we need to write our settings inside this folder. We make a file called cfg-persistent.conf. You can use this command to write the configurations directly. It is very easy:
sudo tee /etc/systemd/journald.conf.d/cfg-persistent.conf > /dev/null <<'EOF'
[Journal]
Storage=persistent
ForwardToSyslog=no
MaxRetentionSec=1month
SystemMaxUse=1G
Compress=yes
Seal=yes
EOFI will now explain what every line of this config file is doing for your server.
- The Storage=persistent line is the most important one. It tell journald that it must write the logs on the hard disk inside the folder /var/log/journal. If this folder is not there, journald will create it automatically. This make sure logs do not disappear when you turn off or restart the machine.
- The ForwardToSyslog=no line is for making the server faster. Usually, journald send logs to rsyslog too. But we do not need this because we only use journald. Turning this off save some CPU power and disk write operations.
- The MaxRetentionSec=1month line is for deleting old logs. We do not want the hard disk to become full because of logs from one year ago. This setting make sure logs older than one month is deleted automatically.
- The SystemMaxUse=1G line is also for protecting the disk. It say that journald can only use one gigabyte of disk space. If the logs reach more than one gigabyte, the oldest logs will be deleted to make space for new logs.
- The Compress=yes line tell the system to compress the log files. Journald will make the files smaller so you can save more space on your hard disk.
- The Seal=yes line is very cool for security. It use a technology called Forward Secure Sealing. This make a special key that changes every second. If an attacker hack your server and try to change or delete old logs to hide his track, the system will know because the seal key will not match anymore.
After you write the configuration, nothing change yet. You must restart the journald service so it can read the new settings you just wrote. You do this with this command:
sudo systemctl restart systemd-journaldNow you can check if journald is writing the logs to the hard disk. You can run this command to see if the new folder is created:
ls -ld /var/log/journal/*/If you see a folder with a long name made of numbers and letters, it means it is working. The long name is the machine ID of your Linux server.
If you want to use the Seal=yes feature, you must generate the key. If you do not generate the key, sealing will not work properly. To make the key, you must run this command in your terminal:
sudo journalctl --setup-keysWhen you run this, it will show you a secret key on the screen. You must write down this key on a paper or save it in a very safe password manager. Do not save it on the same server because if a hacker get in, he can steal the key and modify the logs anyway. If you suspect someone changed the logs, you can verify them using this key with the command journalctl –verify.
Now, we talk about sending logs to another server. Having logs on the same server is good, but if a hacker get root access, he can delete the whole server. So, we must send a copy of the logs to a remote server. We will use a tool called systemd-journal-upload.
First, you must install this tool on your client server. If you use Fedora or Red Hat or Rocky Linux, you can run this command:
sudo dnf5 install -y systemd-journal-remoteIf you are using Ubuntu or Debian, you must run a different command like sudo apt-get install systemd-journal-remote.
After you install it, we need to configure it to send logs to our central log server. We write the configuration in the file /etc/systemd/journal-upload.conf. Run this command to do it:
sudo tee /etc/systemd/journal-upload.conf > /dev/null <<'EOF'
[Upload]
URL=https://logs.example.com:19532
ServerKeyFile=/etc/ssl/private/upload-key.pem
ServerCertificateFile=/etc/ssl/certs/upload-cert.pem
TrustedCertificateFile=/etc/ssl/certs/log-ca.pem
EOFLet me explain what these options mean so you understand what is happening.
The URL line is the address of your remote log server. It use HTTPS because we want the logs to be encrypted when they travel over the internet. If you do not encrypt them, someone can sniff your network and read your private logs. The port is usually 19532.
The ServerKeyFile and ServerCertificateFile is the SSL certificates for your client server. The central server need to know who is sending the logs. It will only accept logs from servers that have the correct certificate.
The TrustedCertificateFile is the certificate of the Certificate Authority that signed the server certificate. This make sure your client is talking to the real log server and not a fake server made by a hacker.
You must put the real certificates in those folders before you start the service. You can use Let is Encrypt or you can make your own private Certificate Authority using tools like easy-rsa or openssl.
Once you have put the certificates in the correct places, you can start the upload service and make it start automatically every time the server boots up. Run this command:
sudo systemctl enable --now systemd-journal-uploadTo make sure the service is running and there is no error, you can check its status with this command:
sudo systemctl status systemd-journal-uploadIf you see green text saying active running, then everything is good. If you see errors, you must check your certificate paths or check if the central server is turned on and can accept connections on port 19532.
On the receiving server, you also need to configure systemd-journal-remote to listen for incoming logs. You do this by enabling the socket service on the receiver server with the command sudo systemctl enable –now systemd-journal-remote.socket. The receiving server will then write all incoming logs to a folder like /var/log/journal/remote/.
This setup is very strong. Even if an attacker get into your main server, they cannot edit the logs because of the Forward Secure Sealing key. And even if they try to delete the system, the logs is already sent to your remote log server over encrypted HTTPS connection.
In conclusion, we have changed journald from volatile storage to persistent storage on the hard disk. We set up limits so the disk does not get full, and we enabled sealing for security. We also configured the server to upload all logs to a remote server using TLS certificates for encryption and authentication. This gives you a very secure logging system for your infrastructure.