Why downtimes occur X hours later in icinga monitoring server installed on CentOS/Redhat Linux systems

In this mini post I’ll show you how to fix and set the correct timezone on your icinga monitoring server with icinga-web V1 installed on CentOS Linux systems. I’m using icinga to monitor all my Linux servers and it works very good, I always receive mails when any error occurs. But suddenly I discovered that there is an error occurs at one of my servers due to a cronjob I configured on that server “it was a temporary error during the cronjob lifetime”, but the mail I receive for this error always one hour later than the cronjob that causes it. Now I can say that I’ve a big issue with timing on my icinga monitor server due to a wrong timezone configured on either the Linux system or on the php.ini or on icinga-web files.

So, How can I fix the timezone on icinga monitoring server?

To fix the timezone on your icinga monitoring server, In three steps, I’ll set the Linux system timezone first, then I’ll set the correct timezone in php.ini and finally I’ll fix the icinga-web timezone on two files, so let’s start:

Step 1: Setting up the correct Linux timezone.

You can use our previously posted article on setting up the timezone on CentOS  Linux systems, here’s it’s url:

• Setting up timezone on CentOS / RHEL Linux systems.

Now, after I fixed my OS “CentOS 7 Linux” timezone, here’s my current timezone:

# timedatectl 
 Local time: Wed 2016-11-30 08:11:17 SAST
 Universal time: Wed 2016-11-30 06:11:17 UTC
 RTC time: Wed 2016-11-30 06:11:17
 Time zone: Africa/Johannesburg (SAST, +0200)
 NTP enabled: yes
 NTP synchronized: yes
 RTC in local TZ: no
 DST active: n/a

Now, let’s go and fix the php.ini timezone setting.

Step 2: Setting up and fixing the timezone in php.ini

Setting up the Linux OS timezone only will not fix the icinga timezone issue, I need to continue and set the correct timezone in  php.ini .

Here’s a small image for icinga-web interface showing the timezone before editing php.ini:

As you see  my timezone in php.ini was “Africa/Cairo“.

Now, edit the php.ini and set the value for “date.timezone” directive to your timezone as follow:

# vim /etc/php.ini
date.timezone = Africa/Johannesburg

Now, After we set the correct timezone on php.ini, We must restart Apache web server, as I’m using CentOS 7 I’ll run the following command:

# systemctl restart httpd

Now, here’s a small image for icinga-web interface showing the timezone after editing php.ini:

Now, let’s go and fix the timezone setting in icinga-web files.

Step 3: Setting up and fixing the timezone in icinga-web files

Icinga-web has two configuration files that we can set value for “date.timezone“, those files are:

1. app/modules/AppKit/config/module.xml
2. app/config/translation.xml

But where exactly those two files exist? use the following find command to find their paths.

# find / -type f -name module.xml
/usr/share/icinga-web/app/modules/Web/config/module.xml
/usr/share/icinga-web/app/modules/TestDummy/config/module.xml
/usr/share/icinga-web/app/modules/Api/config/module.xml
/usr/share/icinga-web/app/modules/AppKit/config/module.xml
/usr/share/icinga-web/app/modules/Cronks/config/module.xml
/usr/share/icinga-web/app/modules/Config/config/module.xml
/usr/share/icinga-web/app/modules/Reporting/config/module.xml

So, the first file found at “/usr/share/icinga-web/app/modules/AppKit/config/module.xml” on CentOS Linux systems.

# find / -type f -name translation.xml
/etc/icinga-web/conf.d/translation.xml
/usr/share/icinga-web/app/config/translation.xml
/usr/share/icinga-web/app/modules/TestDummy/config/translation.xml

And the second file exists at “/usr/share/icinga-web/app/config/translation.xml” on CentOS Linux systems.

All needed from you is to set the “date.timezone” found on “/usr/share/icinga-web/app/modules/AppKit/config/module.xml” to the same timezone you set before for the system and in php.ini

Here’s the default date.timezone value found in this file:

# grep "date.timezone" /usr/share/icinga-web/app/modules/AppKit/config/module.xml
<ae:parametername="date.timezone">GMT</ae:parameter>

# vim /usr/share/icinga-web/app/modules/AppKit/config/module.xml
 <ae:parameter name="date.timezone">Africa/Johannesburg</ae:parameter>

# grep timezone /usr/share/icinga-web/app/config/translation.xml
<available_locales default_locale="en" default_timezone="GMT">

Hints:
1. Do not modify the second file "/usr/share/icinga-web/app/config/translation.xml" leave it with it's default installation values.
2. If we can not set the timezone in php.ini, only in this case we have to set the default_timezone in this file.

Now, we need to restart icinga service, run the following command:

# /etc/init.d/icinga restart

And finally, icinga also has a timezone setting (which is normally unconfigured and defaults to the system timezone):

# grep "use_timezone" /etc/icinga/icinga.cfg
#use_timezone=US/Mountain
#use_timezone=Australia/Brisbane

Why do I change the date.timezone only in module.xml and leave translation.xml unchanged?

To avoid having double configuration of timezone in Icinga-web, Which in turn will give a wrong time. Here’s more explanation:

The date.timezone sets the timezone value. Translation default_timezone is only to convert the timezone into other values.

The best way if you’re running all components at the same co-ordinates: Keep translation.xml default_timezone to its default value of “GMT”. In this case “GMT” means that your base timezone is the same and you do not need conversion between them.