See also: Compilation issues.
zabbix_proxy on MySQL versions 8.0.0-8.0.17 fails with the following "access denied" error:
[Z3001] connection to database 'zabbix' failed: [1227] Access denied; you need (at least one of) the SUPER, SYSTEM_VARIABLES_ADMIN or SESSION_VARIABLES_ADMIN privilege(s) for this operation
That is due to MySQL 8.0.0 starting to enforce special permissions for setting session variables. However, in 8.0.18 this behavior was removed:
The workaround is based on granting additional privileges to the zabbix
user:
For MySQL versions 8.0.14 - 8.0.17:
For MySQL versions 8.0.0 - 8.0.13:
The sql_mode
setting in MySQL/MariaDB must have the "STRICT_TRANS_TABLES" mode set. If it is absent, the Áú»¢¶Ä²© database upgrade will fail (see also ).
Upgrading Áú»¢¶Ä²© may fail if database tables were created with MariaDB 10.2.1 and before, because in those versions the default row format is compact. This can be fixed by changing the row format to dynamic (see also ).
In dual-stack environments (systems configured to support both IPv4 and IPv6), the hostname localhost
typically resolves to both IPv4 and IPv6 addresses. Due to the common prioritization of IPv6 over IPv4 by many operating systems and DNS resolvers, Áú»¢¶Ä²© templates may fail to work correctly if the service being monitored is configured to listen only on IPv4.
Services that are not configured to listen on IPv6 addresses may become inaccessible, leading to monitoring failures. Users might configure access correctly for IPv4 but still face connectivity issues due to the default behavior of prioritizing IPv6.
A workaround for this is to ensure that the services (Nginx, Apache, PostgreSQL, etc.) are configured to listen on both IPv4 and IPv6 addresses, and Áú»¢¶Ä²© server/agent is allowed access via IPv6. Additionally, in Áú»¢¶Ä²© templates and configurations, use localhost
explicitly instead of 127.0.0.1
to ensure compatibility with both IPv4 and IPv6.
For example, when monitoring PostgreSQL with the template, you may need to edit the pg_hba.conf
file to allow connections for the zbx_monitor
user. If the dual-stack environment prioritizes IPv6 (system resolves localhost to ::1
) and you configure localhost
but only add an IPv4 entry (127.0.0.1/32
), the connection will fail because there is no matching IPv6 entry.
The following pg_hba.conf
file example ensures that the zbx_monitor
user can connect to any database from the local machine using both IPv4 and IPv6 addresses with different authentication methods:
# TYPE DATABASE USER ADDRESS METHOD
host all zbx_monitor localhost trust
host all zbx_monitor 127.0.0.1/32 md5
host all zbx_monitor ::1/128 scram-sha-256
If necessary, you can also use the IPv4 address (127.0.0.1
) directly when configuring the template macro for the connection string.
With EPEL repository installed and enabled, installing Áú»¢¶Ä²© from packages will lead to EPEL Áú»¢¶Ä²© packages being installed rather than official Áú»¢¶Ä²© packages.
In this case uninstall Áú»¢¶Ä²© packages from EPEL, i.e.:
Block Áú»¢¶Ä²© packages from EPEL. Add the following line in the /etc/yum.conf
file:
Install Áú»¢¶Ä²© server again:
Notice that official Áú»¢¶Ä²© packages have the word release
in their version string:
When installing Áú»¢¶Ä²© from Red Hat Enterprise Linux packages on environments, ensure access to required repositories and dependencies. Áú»¢¶Ä²© packages depend on libOpenIPMI.so
and libOpenIPMIposix.so
libraries, which are not provided by any package in the default package manager repositories enabled on UBI systems and will result in installation failures.
The libOpenIPMI.so
and libOpenIPMIposix.so
libraries are available in the OpenIPMI-libs
package, which is provided by the redhat-#-for-<arch>-appstream-rpms
repository. Access to this repository is curated by subscriptions, which, in the case of UBI environments, get propagated by mounting repository configuration and secrets directories of the RHEL host into the container file-system namespace.
For more information, see .
When upgrading Áú»¢¶Ä²© on Red Hat Enterprise Linux or its derivatives, you may encounter an expired signing key issue for packages on . When a signing key expires, attempts to verify package signatures will result in an error indicating that the certificate or key is no longer valid. For example:
error: Verifying a signature using certificate D9AA84C2B617479C6E4FCF4D19F2475308EFA7DD (Áú»¢¶Ä²© LLC (Jul 2022) <[email protected]>):
1. Certificate 19F2475308EFA7DD invalid: certificate is not alive
because: The primary key is not live
because: Expired on 2024-07-04T11:41:23Z
2. Key 19F2475308EFA7DD invalid: key is not alive
because: The primary key is not live
because: Expired on 2024-07-04T11:41:23Z
To resolve such issues, manually reinstall the latest zabbix-release
package for your specific variant of RHEL (replace the link below with the correct one from ).
For example, on RHEL 9, run:
rpm -Uvh https://repo.zabbix.com/zabbix/7.4/release/rhel/9/noarch/zabbix-release-latest.el9.noarch.rpm
Then, update the repository information:
For more information, see .
PostgreSQL versions 9.6-12 use too much memory when updating tables with a large number of partitions. This issue manifests itself when Áú»¢¶Ä²© updates trends on systems with TimescaleDB if trends are split into relatively small (e.g. 1 day) chunks. This leads to hundreds of chunks present in the trends tables with default housekeeping settings - the condition where PostgreSQL is likely to run out of memory.
The issue has been resolved since Áú»¢¶Ä²© 5.0.1 for new installations with TimescaleDB, but if TimescaleDB was set up with Áú»¢¶Ä²© before that, please see for the migration notes.
This issue manifests when TimescaleDB 2.5.0/2.5.1 is used. It has been resolved since TimescaleDB 2.5.2.
For more information, please see .
Database TLS connection is not supported with the 'verify_ca' option for the DBTLSConnect parameter if MariaDB is used.
When running under high load, and with more than one LLD worker involved, it is possible to run into a deadlock caused by an InnoDB error related to the row-locking strategy (see ). The error has been fixed in MySQL since 8.0.29, but not in MariaDB. For more details, see .
Events may not get correlated correctly if the time interval between the first and second event is very small, i.e. half a second and less.
PostgreSQL 11 and earlier versions only support floating point value range of approximately -1.34E-154 to 1.34E+154.
Various Áú»¢¶Ä²© processes may randomly crash on startup on the NetBSD versions 8.X and 9.X. That is due to the too small default stack size (4MB), which must be increased by running:
For more information, please see the related problem report: .
Áú»¢¶Ä²© agent 2 does not support lookaheads and lookbehinds in regular expressions due to the standard Go regexp library limitations.
IPMI checks will not work with the standard OpenIPMI library package on Debian prior to 9 (stretch) and Ubuntu prior to 16.04 (xenial). To fix that, recompile OpenIPMI library with OpenSSL enabled as discussed in .
Some Linux distributions like Debian, Ubuntu do not support encrypted private keys (with passphrase) if the libssh2 library is installed from packages. Please see for more details.
When using libssh 0.9.x on some Linux distributions with OpenSSH 8, SSH checks may occasionally report "Cannot read data from SSH server". This is caused by a libssh (). The error is expected to have been fixed by a stable libssh 0.9.5 release. See also for details.
Using the pipe "|" in the SSH script may lead to a "Cannot read data from SSH server" error. In this case it is recommended to upgrade the libssh library version. See also for details.
MySQL unixODBC driver should not be used with Áú»¢¶Ä²© server or Áú»¢¶Ä²© proxy compiled against MariaDB connector library and vice versa, if possible it is also better to avoid using the same connector as the driver due to an . Suggested setup:
PostgreSQL, SQLite or Oracle connector → MariaDB or MySQL unixODBC driver MariaDB connector → MariaDB unixODBC driver MySQL connector → MySQL unixODBC driver
See for more information and available workarounds.
XML data queried from Microsoft SQL Server may get truncated in various ways on Linux and UNIX systems.
It has been observed that using ODBC checks for monitoring Oracle databases using various versions of Oracle Instant Client for Linux causes Áú»¢¶Ä²© server to crash. See also: , .
If using FreeTDS UnixODBC driver, you need to prepend a 'SET NOCOUNT ON' statement to an SQL query (for example, SET NOCOUNT ON DECLARE @strsql NVARCHAR(max) SET @strsql = ....
). Otherwise, database monitor item in Áú»¢¶Ä²© will fail to retrieve the information with an error "SQL query returned empty result".
See for more information.
The request method parameter, used only in HTTP checks, may be incorrectly set to '1', a non-default value for all items as a result of upgrade from a pre-4.0 Áú»¢¶Ä²© version. For details on how to fix this situation, see .
Áú»¢¶Ä²© server leaks memory on some Linux distributions due to an when "SSL verify peer" is enabled in web scenarios or HTTP agent. Please see for more information and available workarounds.
There is a bug in fping versions earlier than v3.10 that mishandles duplicate echo replay packets. This may cause unexpected results for icmpping
, icmppingloss
, icmppingsec
items. It is recommended to use the latest version of fping. Please see for more details.
When containers are running in rootless mode or in a specific-restrictions environment, you may face errors related to fping execution when performing ICMP checks, such as fping: Operation not permitted
or all packets to all resources lost.
To fix this problem add --cap-add=net_raw
to "docker run" or "podman run" commands.
Additionally fping execution in non-root environments may require sysctl modification, i.e.:
where "1995" is the zabbix GID. For more details, see .
If the OpenBSD operating system is used, a use-after-free bug in the Net-SNMP library up to the 5.7.3 version can cause a crash of Áú»¢¶Ä²© server if the SourceIP parameter is set in the Áú»¢¶Ä²© server configuration file. As a workaround, please do not set the SourceIP parameter. The same problem applies also for Linux, but it does not cause Áú»¢¶Ä²© server to stop working. A local patch for the net-snmp package on OpenBSD was applied and will be released with OpenBSD 6.3.
Spikes in SNMP data have been observed that may be related to certain physical factors like voltage spikes in the mains. See more details.
The "net-snmp-perl" package, needed for SNMP traps, has been removed in RHEL 8.0-8.2; re-added in RHEL 8.3.
So if you are using RHEL 8.0-8.2, the best solution is to upgrade to RHEL 8.3.
Please also see for more information.
Instances of a Áú»¢¶Ä²© server alerter process crash have been encountered in RHEL 7. Please see for details.
When upgrading Áú»¢¶Ä²© agent 2 (version 6.0.5 or older) from packages, a plugin-related file conflict error may occur. To fix the error, back up your agent 2 configuration (if necessary), uninstall agent 2 and install it anew.
On RHEL-based systems, run:
On Debian-based systems, run:
For more information, see .
It has been observed that frontend locales may flip without apparent logic, i. e. some pages (or parts of pages) are displayed in one language while other pages (or parts of pages) in a different language. Typically the problem may appear when there are several users, some of whom use one locale, while others use another.
A known workaround to this is to disable multithreading in PHP and Apache.
The problem is related to how setting the locale works : locale information is maintained per process, not per thread. So in a multi-thread environment, when there are several projects run by same Apache process, it is possible that the locale gets changed in another thread and that changes how data can be processed in the Áú»¢¶Ä²© thread.
For more information, please see related problem reports:
bcdiv
function of BC Math functions)Changes to Daylight Saving Time (DST) result in irregularities when displaying X axis labels (date duplication, date missing, etc).
When using sum aggregation in a graph for period that is less than one hour, graphs display incorrect (multiplied) values when data come from trends.
For some frontend languages (e.g., Japanese), local fonts can cause text overlapping in graph legend. To avoid this, use version 2.3.0 (or later) of PHP GD extension.
log[]
and logrt[]
items repeatedly reread log file from the beginning if file system is 100% full and the log file is being appended (see for more information).
Áú»¢¶Ä²© server generates slow SELECT
queries in case of non-existing values for items. This is known to occur in MySQL 5.6/5.7 versions (for an extended discussion, see ), and, in specific cases, may also occur in later MySQL versions. A workaround to this is disabling the or optimizer in MySQL. Note, however, that this workaround may not fix all issues related to slow queries.
When opening a link to Áú»¢¶Ä²© frontend page that contains filter settings, including the time selector, the filter is automatically saved in the database for the user, replacing the previously saved filter and/or time selector settings for that page. These settings remain active until the user manually updates or resets them.
Due to a net-snmp bug, IPv6 address may not be correctly displayed when using SNMPv3 in SNMP traps. For more details and a possible workaround, see .
A failed login attempt message will display only the first 39 characters of a stored IP address as that's the character limit in the database field. That means that IPv6 IP addresses longer than 39 characters will be shown incompletely.
Non-existing DNS entries in a Server
parameter of Áú»¢¶Ä²© agent configuration file (zabbix_agentd.conf) may increase Áú»¢¶Ä²© agent response time on Windows. This happens because Windows DNS caching daemon doesn't cache negative responses for IPv4 addresses. However, for IPv6 addresses negative responses are cached, so a possible workaround to this is disabling IPv4 on the host.
There are some known issues with YAML export/import:
Frontend setup wizard cannot save configuration file on SUSE with NGINX + php-fpm. This is caused by a setting in /usr/lib/systemd/system/php-fpm.service unit, which prevents Áú»¢¶Ä²© from writing to /etc. (introduced in ).
There are two workaround options available:
In some cases, Apache or NGINX may prevent the Authorization header in API requests from reaching Áú»¢¶Ä²©. This can cause authentication issues when using Áú»¢¶Ä²© API or single sign-on (SSO) services, such as SAML with Okta.
To address this, update your web server's configuration.
For Apache, if you are using it as a reverse proxy (non-CGI setup), add the following directive to /etc/httpd/conf/httpd.conf
(on RHEL-based systems) or /etc/apache2/apache2.conf
(on Debian/Ubuntu):
If Apache directly executes scripts to handle requests (e.g., by using mod_cgi), add the following directive instead:
In contrast, NGINX handles the Authorization header automatically. However, if NGINX is acting as a reverse proxy, you may explicitly forward the Authorization header by adding the following directives to /etc/nginx/nginx.conf
(for your Áú»¢¶Ä²© frontend location):
...
location / {
...
proxy_set_header Authorization $http_authorization;
proxy_pass http://backend_server;
...
}
After updating the configuration, restart you web server.
For more details, see:
Though in most cases, Áú»¢¶Ä²© web service can run with Chromium, on Ubuntu 20.04 using Chromium causes the following error:
Cannot fetch data: chrome failed to start:cmd_run.go:994:
WARNING: cannot create user data directory: cannot create
"/var/lib/zabbix/snap/chromium/1564": mkdir /var/lib/zabbix: permission denied
Sorry, home directories outside of /home are not currently supported. See https://forum.snapcraft.io/t/11209 for details.
This error occurs because /var/lib/zabbix
is used as a home directory of user 'zabbix'.
When Áú»¢¶Ä²© detects that the backend database is inaccessible, it sends a notification and continues attempting to connect. For certain database engines, specific error codes are recognized. In MySQL, these recognized error codes include:
Additionally, when using Áú»¢¶Ä²© with a MySQL installation on Azure, the generic error message [9002] Some errors occurred may appear in Áú»¢¶Ä²© logs. This message is sent by the database to the Áú»¢¶Ä²© server or proxy. To determine the cause of the error, please consult the Azure logs.
In Áú»¢¶Ä²© 6.0 support for PCRE2 has been added. Even though PCRE is still supported, Áú»¢¶Ä²© installation packages for RHEL 7 and newer, SLES (all versions), Debian 9 and newer, Ubuntu 16.04 and newer have been updated to use PCRE2. While providing many benefits, switching to PCRE2 may cause certain existing PCRE regexp patterns becoming invalid or behaving differently. In particular, this affects the pattern ^[\w-\.]. In order to make this regexp valid again without affecting semantics, change the expression to ^[-\w\.] . This happens due to the fact that PCRE2 treats the dash sign as a delimiter, creating a range inside a character class.
The maps in the Geomap widget may not load correctly, if you have upgraded from an older Áú»¢¶Ä²© version with NGINX and didn't switch to the new NGINX configuration file during the upgrade.
To fix the issue, you can discard the old configuration file, use the configuration file from the current version package and reconfigure it as described in the download instructions in section e. Configure PHP for Áú»¢¶Ä²© frontend.
Alternatively, you can manually edit an existing NGINX configuration file (typically, /etc/zabbix/nginx.conf). To do so, open the file and locate the following block:
Then, replace this block with:
location ~ /(api\/|conf[^\.]|include|locale) {
deny all;
return 404;
}
location /vendor {
deny all;
return 404;
}
As global variables are shared across different webhook calls, the following code will result in the tag value counter gradually increasing:
try
{
aa = aa + 1;
}
catch(e)
{
aa = 0;
}
result = {
'tags': {
'endpoint': aa
}
};
return JSON.stringify(result);
Using local variables instead of global ones is recommended to make sure that each script operates on its own data and that there are no collisions between simultaneous calls.
Microsoft documentation states that systems with fewer than 64 logical processors always have a single processor group, Group 0. However, Áú»¢¶Ä²© users have reported a rare bug , when there are two processor groups on systems with 64 or less logical processors. This resulted in having the "(n)" performance counters for only one processor group out of two. The actual root cause of this bug is not known. However, a similar case was described at , and the root cause there was in interoperation between BIOS and Windows.
Filters (e.g., in Data collection → Maintenance) may not function correctly when applied to entities containing certain Unicode characters (e.g., ȼ, ɇ). This issue arises due to how the default utf8mb4_bin collation for MySQL or MariaDB databases handles sorting and comparison of Unicode characters.
To address this limitation, users can change the collation of database columns to alternatives such as utf8mb4_0900_bin, utf8mb4_0900_ai_ci, or utf8mb4_unicode_520_ci. Note, however, that changing the collation may cause unexpected behavior in the handling of empty spaces, as well as sorting and filtering for other characters.
For more information on changing collations, see or . For details on collation differences, see in MySQL documentation.
Accessing the Áú»¢¶Ä²© web frontend with a role other than Super Admin may result in the message: "System error occurred. Please contact Áú»¢¶Ä²© administrator.". This issue affects installations using MariaDB versions 10.5.1 through 10.5.9.
To avoid this issue, update MariaDB to a version later than 10.5.9. For more details, see .
If you suspect your Áú»¢¶Ä²© installation is using too much memory, you can use memory profiling feature to investigate Áú»¢¶Ä²© server/proxy memory consumption.
1. When installing Áú»¢¶Ä²© from sources, configure additional flags:
The -std=gnu99
flag is required for building Áú»¢¶Ä²© server, Áú»¢¶Ä²© proxy, or Áú»¢¶Ä²© agent. The -g
flag adds extra debugging information, while -O0
disables optimizations, which can interfere with tcmalloc's profiling.
2. Set the following environment variables before starting the Áú»¢¶Ä²© server. These variables tell tcmalloc how to track and report memory usage:
LD_PRELOAD="/usr/lib/aarch64-linux-gnu/libtcmalloc.so" \
HEAPPROFILE=./heap_profile \
HEAP_PROFILE_ALLOCATION_INTERVAL=0 \
HEAP_PROFILE_INUSE_INTERVAL=4294967296 \
HEAPPROFILESIGNAL=5 \
MALLOCSTATS=1 \
./sbin/zabbix_server -f -c /etc/zabbix/zabbix_server.conf
3. Trigger a profile dump by sending signal 5 to the target process. Replace 1234 with the actual process ID (PID):
4. Print the generated profile:
pprof-symbolize -text ./sbin/zabbix_server ./heap_profile.0001.heap
Using local file ./sbin/zabbix_server.
Using local file ./heap_profile.0001.heap.
Total: 1078.1 MB
1076.8 99.9% 99.9% 1076.8 99.9% zbx_malloc2
1.0 0.1% 100.0% 1.0 0.1% __GI___strdup
0.2 0.0% 100.0% 0.2 0.0% CRYPTO_zalloc@@OPENSSL_3.0.0
0.1 0.0% 100.0% 0.1 0.0% OPENSSL_LH_insert@@OPENSSL_3.0.0
0.0 0.0% 100.0% 0.0 0.0% zbx_realloc2
0.0 0.0% 100.0% 0.1 0.0% PKCS7_decrypt@@OPENSSL_3.0.0
0.0 0.0% 100.0% 0.0 0.0% find_best_tree_node
0.0 0.0% 100.0% 0.0 0.0% CRYPTO_strndup@@OPENSSL_3.0.0
...
0.0 0.0% 100.0% 0.0 0.0% preprocessing_flush_value
0.0 0.0% 100.0% 1074.0 99.6% preprocessor_add_request
In this example, zbx_malloc2 is responsible for almost all memory allocations.
See also:
-std=gnu99
, -g
, -O0
, etc.).