How To Force logadm To Rotate A “Busy” Log

If logadm tells you that a “log file is too busy to rotate” as in below:

logadm: Warning: command failed: /bin/sh -c /usr/sbin/fmadm -q rotate errlog &&
mv /var/fm/fmd/errlog.0- /var/fm/fmd/errlog.0
fmadm: failed to rotate errlog: log file is too busy to rotate (try again later)

Or if you have received an e-mail saying essentially the same thing – a log file is too busy to rotate:

Your "cron" job on solsvr10_02
/usr/sbin/logadm

produced the following output:

logadm: Warning: command failed: /bin/sh -c /usr/sbin/fmadm -q rotate errlog &&
mv /var/fm/fmd/errlog.0- /var/fm/fmd/errlog.0
fmadm: failed to rotate errlog: log file is too busy to rotate (try again later)

And you’ve noticed that /var is constantly at a high utilization or even full, then chances are you’ve been hit by a known Solaris 10 bug 6797442. This bug causes fmd, the Solaris Fault Management daemon, to log an “ereport.io.ddi.fm-capability” event in /var/fm/fmd/errlog tens, even hundreds, of times per second. This much activity keeps logadm from rotating the fmd errlog when it reaches the defined size causing the /var filesystem to blow out.

The fix is to install patch 141874-10 SunOS 5.10: fp patch. However, this patch requires a reconfiguration reboot on the server. If a reboot is not possible, you can work around the issue by stopping fmd, deleting the errlog file and restarting fmd.

# svcs fmd
STATE          STIME    FMRI
online          3:10:36 svc:/system/fmd:default
#
# svcadm disable fmd
#
# svcs fmd
STATE          STIME    FMRI
disabled        8:00:31 svc:/system/fmd:default
#
# rm /var/fm/fmd/errlog
#
# svcadm enable fmd
#
# svcs fmd
STATE          STIME    FMRI
online          8:01:06 svc:/system/fmd:default
#

This is fine if you catch /var filling up during the day. But what if it happens in the middle of the night? That’s when logadm comes to the rescue. Run this command to replace the default /var/fm/fmd/errlog entry in /etc/logadm.conf with one that does all of the above for you,

# logadm -M '/usr/sbin/svcadm disable fmd && sleep 30 && mv $file $nfile && \
/usr/sbin/svcadm enable fmd' -N -s 10m -w /var/fm/fmd/errlog
#
# grep errlog /etc/logadm.conf
# /var/fm/fmd/errlog -M '/usr/sbin/svcadm disable fmd && sleep 30 && mv $file $nfile &&
/usr/sbin/svcadm enable fmd' -N -s 10m
#

Now you can a good night’s sleep. When you wake up tomorrow read How To Manage Log Files With logadm to learn more about logadm.