Centralized syslog server for vSphere environment with CentOS 7 and rsyslog

Couple of years ago we were asked to provide a working solution for centralized syslog collector. My customer had very tough security rules, and they were strictly against any appliance based, out of the box solution. They needed to have full control over the OS and applications installed. The main condition was: Solution should use free and open source operating system and applications. At that time, my solution was based on CentOS 5 and rsyslog. Recently I reviewed it, improved it, added some nice features, and now it ready to be published. In two words: We will have a CentOS 7 with rsyslog, listening on TCP port 514, which will receive logs from ESXi servers and vCenter server. Logs will be kept for 90 days. Solutions is quite flexible and can be changed based on your needs.

Note: when writing this article I assume that reader has basic understanding of Linux operating systems, so some of basic steps will be skipped.

So lets start.

Downloading and installing CentOS 7

To download CentOS 7 click Get CentOS 7 Now link on CentOS home page. You can download the full DVD, but if you want to avoid 4+ GB download, you can click the alternative downloads link, and you will see much more download options. In my example I will be using minimal installations ISO, and all additional packages will be installed from internet. I will not go through CentOS installation process, because it is very straight forward, almost like installing ESXi. You just need to select Time Zone, select disk to use, and set root password. Nevertheless, if you will have problems at this step, fell free to leave a comment bellow.

Steps to take on first login to CentOS 7

Once CentOS is installed, I strongly recommend to perform update. The ISO images are generated on each release, and the packages included are not the latest version.  To perform update on CentOS execute the following command:

yum -y update

After update is done, it is right time install some useful tolls like for example VIM text editor, nano text editor and even full  Midnight Commander. Use yum to install those.

yum -y install vim mc nano

Note: In my examples I will be using vim text editor, if you are not familiar with it or you don’t feel comfortable using it, you can use mcedit or nano.

Configuring rsyslog

Now, when we have all out tools set up, we will proceed with configuring rsyslog. The default configuration file for rsyslog is /etc/rsyslog.conf. But additional configuration files can be created in /etc/rsyslog.d/ directory. This is done to simplify configuration process, as each application will create its own  file, and you will not end up with huge, unreadable config.

Note: All configuration can be done in one /etc/rsyslog.conf file. The separation to different config files is done for sake of simplicity.

We will be creating 2 RuleSets, one for local logging and another one for remote. Local RuleSet will be responsible for storing local logs of rsyslog linux server, and the remote will serve for storing our vSphere logs. First we will need to edit global /etc/rsyslog.conf  file, and add 2 lines there. $RuleSet local and DefaultRuleset local To show the placement of these lines, I will post here the config file as it should looks after adding those.

 
#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### RULES ####

#Create Local RuleSet in order to store Logs of syslog server itself as usual and keep remote logs from beeing logged to default log files.
$RuleSet local
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure
# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog
# Log cron stuff
cron.*                                                  /var/log/cron
# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler
# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

#Make local RuleSet default one
$DefaultRuleset local

Note: By default there is a lot of commented out config parameters in rsyslog.conf. Those were removed in this example  to make config file more readable.

After adding these 2 lines into main config, we will create our config for remote logging of vSphere environment. The file which we will create will be called vsphere.conf and will be placed in /etc/rsyslog.d/ directory. Use your favorite editor to create it in my case it will be vim.

vim /etc/rsyslog.d/vsphere.conf

Here is content of that file.

# Provides TCP syslog reception
$ModLoad imtcp

#### Create Templates for Log parsing and storing using predefined Folder structure
template(name="TIMESTAMP" type="string" string="%timegenerated:8:25%||%HOSTNAME%||%syslogtag%||%msg%\n")
template(name="REMOTEESX" type="string"  string="/logs/esx/%fromhost-ip%/%$YEAR%-%$MONTH%/%$DAY%.%$MONTH%-messages.log")
template(name="VCENTER" type="string" string="/logs/vcenter/%fromhost-ip%/%$YEAR%-%$MONTH%/%$DAY%.%$MONTH%-messages.log")

#Defining remote logging RuleSet
$RuleSet remote

### The following rule is created to store logs coming from vCenter with different parameters
## If log is coming from vcenter with IP 10.12.14.24 it will be stored using VCENTER template.
if $fromhost == '10.12.14.24' then {
*.* ?VCENTER
}
else {
#All other logs will be stored using TIMESTAMP and REMOTESX templates.
*.* ?REMOTEESX;TIMESTAMP
}

### bellow parameters are telling rsyslog to listen TCP port 514. All events coming to this port will be parsed using remote RuleSet.
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514

Templates are using by rsyslog as a control mechanism. With templates you can define a lot of things, like for example where and how to store your logs, you can use templates to parse logs in case of need. There are three templates we will be using in this config.

• TIMESTAMP – This  template will parse all the lines coming from sources corresponding to certain rule and perform custom formatting. This is done to make log files more readable. You can disable this by simply removing TIMESTAMP from rule which uses it.
• REMOTEESX – The REMOTEESX template tells rsyslog to store all logs from sources corresponding to certain rule in a /logs/esx/ directory structure. logs from each host will be stored in separate folder and logs from each day will be stores in folder corresponding to that day.
• VCENTER – This does same as REMOTEESX but all logs are stored in /logs/vcenter/ directory.

More on using templates in official documentation for rsyslog here.

Now, when the config files are ready, we can proceed with the rest of the configuration.

Configuring Firewall

To allow rsyslog to receive communications on TCP port 514 we need to enable that port on firewall. CentOS 7 uses new approach to contol firewall config. Actually it is not that new and it was used in Fedora for quite some time, but it was implemented to RHEL 7 and CentOS 7 as well. I fount is very comfortable to use. Here is an article which can help you to get started with it and of course link to official RedHat documentation here.

Firewalld is using so called services. Services are defined by using config files in xml format. There is a set of predefined services, and config files for those are stored in /etc/lib/firewalld/services/ directory. If you need to create a customer service xml, you need to store it in /etc/firewalld/services/ directory.

What I suggest to do in our case, is to copy one of predefined service files from /etc/lib/firewalld/services/ (for example ssh.xml) to /etc/firewalld/services/ and modify as needed.

cp /usr/lib/firewalld/services/ssh.xml /etc/firewalld/services/rsyslog.xml

Use your favorite editor to edit the copied file. As you know, in my case I use vim.

vim /etc/firewalld/services/rsyslog.xml

The config should look like this.

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>rsyslog</short>
  <description>Centralized rsyslog server for vSphere environment.</description>
  <port protocol="tcp" port="514"/>
</service>

Once you have the service configuration ready, it is time to enable that service using firewall-cmd command.

Add rsyslog service to public zone.

firewall-cmd --permanent --zone=public --add-service=rsyslog

Reload firewall configuration

firewall-cmd --relaod

To see if service was enabled execute command bellow and make sure rsyslog is listed.

firewall-cmd --list-services

Creating folder structure

As you noticed, in config file of rsyslog we created 2 templates one called VCENTER and another called REMOTEESX. To make those work, we need to prepare some folder structure for them. First of all we need to create /logs/ directory.

To create folder execute

mkdir /logs

But just creating folder is not enough. We are in CentOS and, as in many other Linux distributions, there is one very advanced security feature called SELinux which will not allow rsyslog service to write into that folder. Unless we will assign proper context to it of caurse. Some people suggest disabling SELinux after installation, but lets not be Dumb(sorry if I hurt anyone’s feelings), people would not create that feature, if you would need to disable it right from the beginning.

We will do it in simple and correct way. We know that rsyslog can write to /var/log/ directory by default, which means that /var/log/ has all the correct things to it. So what if we will just copy the needed contexts from /var/log/ to /logs/ ?

Copy SELinux context from one directory to another.

chcon --reference /var/log /logs

To check SELinux context before and after the change use the following command

ls -Zd /logs

Configuring Log rotation

We will use built-in logrotate service to rotate our logs. As rsyslog, logrotate has config files are stored in /etc. The main config file is /etc/logrotate.conf. Additional config files need to be created in /etc/logrotate.d/ directory. There is a cron job scheduled  to run every day and rotate the log. The cron job script is located in /etc/cron.daily/ in case you will ever want to edit it.

Requirement for my customer was to keep all log files for 90 days. In my example I will configure logrotate to fulfill that requirement.

Fire up you preferred editor and create file /etc/logrotate.d/vsphere. Here is what I have in it.

/logs/esx/*/*/*.log {
        rotate 90
        daily
        notifempty
        compress
}
/logs/vcenter/*/*/*.log {
        rotate 90
        daily
        notifempty
        compress
}

I think config is self explanatory, so will not go into details. If some part of it is not clear, feel free to leave a comment and I will com back to you.

Restarting rsyslog service

There is another change in CentOS 7. The former service and chkconfig commands are not used any more, as systemd is now used for services initialization. So, from now on, to control services, we will use systemctl command.

As all configurations are in place, lets restart rsyslog service.

systemctl restart rsyslog

And check status of it once restarted

systemctl status rsyslog

What’s next

Now, when you have rsyslog server ready, you need to configure your ESXi servers to send the logs. I will not describe here how to do it, as it is very well described in VMware KB 2003322.

In my Next article called “Sending vCenter Logs to Centralized Syslog Server using NXlog”  I describe, how to send Log files from you vCenter Server installed on windows to this centralized syslog server.

Post Scriptum

I would like to express my thanks for helping in creating in this article to:

1. Stanislav Jurena –  for his help one initial config of rsyslog server and for his friendly support.
2. Google –  for leading me to a solutions when I was lost 🙂
3. TheVirtualist.org team, for their opinions and comments.

The following two tabs change content below.

• Bio
• Latest Posts

Aram Avetisyan

Aram Avetisyan is an IT specialist with more than 18 years experience. He has rich background in various IT related fields like Cloud, Virtualization and SDN. He holds several industry level certifications including but not limited to VCIX-DCV, VCIX-NV. He is also a vEXPERT in years 2014-2021.

Latest posts by Aram Avetisyan (see all)

• Make Youtube Videos About Technology? Why not… The Cross-Cloud Guy - October 7, 2021
• Automating (NSX-T) REST API using Ansible URI module - December 29, 2020
• Quick Reference: Create Security Policy with Firewall Rules using NSX-T Policy API - May 4, 2020