How See Hidden SELinux Errors When Your Server Is Broken

Many times when you work with Linux, especially CentOS or Red Hat, you will have problem where your website or database not want to start. You think it is SELinux because when you turn off SELinux with setenforce 0, it work. But when you look at the audit log file, you see absolutely nothing. This is because SELinux have a secret feature called dontaudit what hide many errors so your log file do not get filled with too much trash text. In this guide, I will show you how to disable this hide feature so you can find the real error and then how to turn it back on so your system stay safe and clean.

To understand why we need this, we must know how SELinux works inside the Linux kernel. SELinux is security system that check everything. Every file have label, every program have label, and every network port have label. If a program try to access a file, SELinux checks its policy database. If the policy does not allow it, SELinux blocks it.

But many program in Linux are designed to try many things. For example, a program might try to read some system configuration files just to see if they exist, even if the program does not really need them to run. If SELinux write a log line every time a program try to touch something it does not need, your log file at /var/log/audit/audit.log will get very big, very fast. It can write gigabytes of text in just one day.

To prevent this big log problem, the developers of SELinux put dontaudit rules inside the security policies. These rules basically tell the system if this specific program try to do this specific thing, block it, but do not write any message in the log file because we already know it is not important.

This is very good for keeping the log clean, but it is very bad when you are trying to make a new program work or when you change the folder of your database. Sometimes, the first block that SELinux does silently will make the program fail in a different way later. Because the first block was silent, you only see the later errors, or maybe you see no errors at all. This is why you must know how to make SELinux talkative for a short time.

Here is the step by step tutorial to find these hidden errors. Please follow the steps carefully so you do not break your system log files.

Step 1: Disable the dontaudit rules temporarily

First, we must tell SELinux to stop hiding the blocked actions. We need to open our terminal and use root privileges. We run this command:

sudo semodule -DB

Let me explain what this command is doing. The semodule command is the main tool we use to manage SELinux policy modules. The -D option means disable, and specifically here it means disable the dontaudit rules in the policy. The -B option means rebuild, which tells the system to rebuild the policy database with this change.

When you run this command, it might take some seconds to finish. Do not cancel it. The computer is reading all the policy files and rebuilding them without the dontaudit rules. Once it is done, your terminal will show the normal prompt again. Now, every single block that SELinux does will be written directly to the audit log, even the very noisy ones.

Step 2: Recreate the error on your system

Now that the security guard is not silent anymore, you must make the error happen again. Go to your terminal and try to start your broken service or run your script. For example, if your web server Apache is having problem, run this command:

sudo systemctl restart httpd

Or if you have a custom script that is failing, run it now:

./myscript.sh

You must do the exact action that was failing before. When the program try to do the action and get blocked, SELinux will now write the details into the log file because we disabled the silent rule in the first step.

Step 3: Search the audit log for the hidden errors

Now we must find the error message in the audit log. The audit log file is located at /var/log/audit/audit.log. It has a lot of confusing text, so we do not want to read it with normal text editors. Instead, we use a special tool called ausearch which is made for searching audit logs. Run this command:

sudo ausearch -i -if /var/log/audit/audit.log -m AVC --start recent

Let me explain all parts of this command:

• The sudo is for admin permission.
• The ausearch is the name of the tool.
• The -i is very useful because it translate the system numbers into human words. For example, it will show user names instead of user ID numbers, and path names instead of raw numbers.
• The -if /var/log/audit/audit.log tell the tool to read from this specific audit log file.
• The -m AVC tells it to search only for AVC messages. AVC means Access Vector Cache, which is where SELinux log all its blocks.
• The –start recent is very important because it tells the tool to only show the logs that happened in the last few minutes. This prevents you from seeing old errors from last week.

When you run this command, look at the output. You will see lines containing denied. It will tell you which process tried to access which file, what label the process had, and what label the file had. This will show you the real reason why your program is not working.

Step 4: Enable the dontaudit rules again

After you find the error and write down the information, you must turn the hiding feature back on. If you forget this step, your server will continue to write millions of noisy logs, and your hard drive can get full very quickly. Run this command to make SELinux quiet again:

sudo semodule -B

This command only has the -B option. It tells the system to rebuild the policy database again. Because we did not put -D this time, the system will put the dontaudit rules back into the policy. Your system log will go back to being clean and normal.

Let us look at a real example so you can see how this works in real life. Imagine you are running a web server using Nginx. You want Nginx to read files from a folder inside a user home directory, like /home/user/mywebsite.

You configure Nginx and then you try to access the website. But you get a 403 Forbidden error on your browser.

You look at the Nginx error log, and it says permission denied.

Then you look at /var/log/audit/audit.log to see if SELinux is blocking Nginx. But you see no AVC denial messages for Nginx. You check the folder permissions, and they are correct. You feel very confused because you do not know why Nginx cannot read the files.

This is because SELinux has a dontaudit rule for Nginx trying to read files in the home directory. SELinux does this because many programs accidentally try to read home directories, and the developers did not want the logs to be spammed.

Now, you follow our tutorial. You run:

sudo semodule -DB

Then you try to open the website in your browser again to trigger the 403 error.

Next, you run the search command:

sudo ausearch -i -if /var/log/audit/audit.log -m AVC --start recent

Suddenly, you see a message in the log! It says Nginx which has the label httpd_t tried to read index.html which has the label user_home_t, but SELinux blocked it.

Now you know exactly what is wrong. The label on the file is wrong for a web server. You can fix this by changing the label of the folder using semanage fcontext and restorecon commands, or by enabling the SELinux boolean httpd_enable_homedirs.

Once you fix the label or the boolean, you test the website again, and it works.

Finally, you run:

sudo semodule -B

This turns the silent rules back on, and your system is clean again.

Using this method is much better than turning off SELinux completely. Many people get frustrated with SELinux and just run setenforce 0 to disable it forever. But this is bad for security. SELinux protects your server if a hacker finds a bug in your web application. If SELinux is on, the hacker cannot access other parts of the system. If you turn it off, the hacker can do whatever they want.

By using the semodule -DB command, you can find the exact problem and fix it properly without making your server unsafe. It is a good tool for any system administrator.

In conclusion, the dontaudit rules in SELinux are very helpful for keeping our system logs clean, but they can make troubleshooting very hard when we do not know why a program is failing. By temporarily disabling these rules with semodule -DB, reproducing the issue, searching the log with ausearch, and then enabling them again with semodule -B, we can find and fix hidden security blocks safely. This keeps our system secure and our logs readable.