Cowboy Denny Posted November 17, 2022 Share Posted November 17, 2022 This forum topic is my journey figuring out why one of our BIG-IP GTM's (BIG-IP DNS) was disconnected from the sync group These broadcast messages were blowing up the cli Broadcast message from systemd-journald@usfnt1slbgtm99.thezah.corp (Wed 2022-02-23 12:18:52 CST): logger[16928]: Re-starting zrd The BIG-IP GTM system may occasionally log the following error message to the console and to the /var/log/daemon.log file: Re-starting zrd tmsh show sys service zrd zrd finish (pid 26149) 0 seconds, 21 starts, 6209 restarts The zrd process is the ZoneRunner daemon. The ZoneRunner utility is used to create, manage, and maintain DNS files on the BIG-IP GTM system. Various issues may cause the zrd process to restart. The most common issues include the following: Syntax errors in the BIND configuration file Syntax errors in the BIND zone files ZoneRunner records do not match BIND zone files To determine why the zrd process is restarting, perform the following procedure: Log in to the command line. If the Re-starting zrd error message is repeatedly logged to the console, you can prevent the zrd process from attempting to restart by typing the following command: bigstart stop zrd After the zrd process halts, use a text reader such as the less command to review the the /var/log/gtm file for error messages related to the named.conf file, or specific zone file. For example: less /var/log/gtm For example, the following log entry indicates the journal file for the test.com zone is out of sync with the zone file. The error message also indicates that the test.com zone was not found. This behavior is caused by the errant space in the zone name in the named.conf file: zrd[26232]: 0115020b:3: Errors in config file zone siterequest.com/IN: loaded serial 55. zone m1.test.net/IN: loaded serial 2009020301. db.external.test.com.:3: ignoring out-of-zone data (test.com). zone test.\032com/IN: journal rollforward failed: journal out of sync with zone. externa. l/test. com./IN: not found. Depending on the error message found in the /var/log/gtm file, you may need to refer to one of the following articles: If an error message indicates a syntax issue with the named.conf file, refer to K6963: Managing the BIG-IP BIND configuration file. If an error message indicates a syntax issue with a BIND zone file, refer to K7032: Freezing zone files to allow manual update to ZoneRunner-managed zone files. If an error message indicates that ZoneRunner records do not match BIND zone files, refer to K5738: ZoneRunner records do not match BIND zone files. After you have fixed the source of the error, restart the zrd process by typing the following command: bigstart start zrd error: zone domain.com/IN/external: journal rollforward failed: journal out of sync with zone The zrd process exits when it encounters this error message. Additionally, you cannot restart the zrd process until you resolve the error condition. Manually deleting the zone journal files from the BIG-IP DNS/GTM system FIRST identify what zones you have issues with: grep named /var/log/daemon.log Feb 23 12:00:49 usfnt1slbgtm99.thezah.corp err named[13617]: 23-Feb-2022 12:00:49.713 general: error: zone thezah.com/IN/external: journal rollforward failed: journal out of sync with zone Feb 23 12:00:49 usfnt1slbgtm99.thezah.corp err named[13617]: 23-Feb-2022 12:00:49.720 general: error: zone thezah.com/IN/external: not loaded due to errors. Feb 23 12:00:49 usfnt1slbgtm99.thezah.corp err named[13617]: 23-Feb-2022 12:00:49.723 general: error: zone int.thezah.com/IN/external: journal rollforward failed: journal out of sync with zone Feb 23 12:00:49 usfnt1slbgtm99.thezah.corp err named[13617]: 23-Feb-2022 12:00:49.723 general: error: zone int.thezah.com/IN/external: not loaded due to errors. NOTE: You may have too many errors that filled up the latest daemon.log so you may have to search the older files which you can do with: zgrep named /var/log/daemon* In the above example you can see the zones that are having issues are thezah.com and int.thezah.com Location zone files: ls -ltrh /var/named/config/namedb ls -ltrh /var/named/config/namedb total 128K -rw-r--r--. 1 named named 337 Oct 12 2014 db.external.abc. -rw-r--r--. 1 named named 350 Sep 12 2018 db.external.int.thezah.com.https. -rw-r--r--. 1 named named 14K Nov 15 19:48 db.external.int.thezah.com.~ -rw-r--r--. 1 named named 3.3K Nov 15 19:49 db.external.thezah.com.~ -rw-r--r--. 1 named named 984 Nov 15 19:53 db.external.eventguyz.com..jnl -rw-r--r--. 1 named named 7.5K Nov 15 19:53 db.external.thezah.corp..jnl -rw-r--r--. 1 named named 1.7K Nov 15 19:53 db.external.cowboydenny.com..jnl -rw-r--r--. 1 named named 4.1K Nov 15 20:04 db.external.thezah.corp. -rw-r--r--. 1 named named 469 Nov 15 20:05 db.external.eventguyz.com. -rw-r--r--. 1 named named 881 Nov 15 20:05 db.external.cowboydenny.com. -rw-r--r--. 1 named named 22K Nov 16 21:22 db.external.int.thezah.com..jnl -rw-r--r--. 1 named named 11K Nov 16 21:22 db.external.thezah.com..jnl -rw-r--r--. 1 named named 3.4K Nov 16 21:33 db.external.thezah.com. -rw-r--r--. 1 named named 14K Nov 16 21:34 db.external.int.thezah.com. Stop ZonerRunner: tmsh stop /sys service zrd Stop named: tmsh stop /sys service named Remove the journal files (.jnl) for the affected zones rm /var/named/config/namedb/db.external.thezah.com..jnl rm /var/named/config/namedb/db.external.int.thezah.com..jnl Start services backup tmsh start /sys service named tmsh start /sys service zrd Link to comment Share on other sites More sharing options...
Cowboy Denny Posted November 17, 2022 Author Share Posted November 17, 2022 Another good F5 article that references: zrd process exits upon the 'journal rollforward failed: journal out of sync with zone' Link to comment Share on other sites More sharing options...
Recommended Posts