UNIX Hints & Hacks

ContentsIndex

Chapter 4: System Monitoring

 

Previous ChapterNext Chapter

Sections in this Chapter:

   

4.1 Monitoring at Boot Time

 

4.5 Mail a Process

 

4.9 Monitoring with ping

 

4.2 Starting with a Fresh Install

 

4.6 Watching the Disk Space

 

4.10 Monitoring Core Files

 

 

4.3 Monitor with tail

 

4.7 Find the Disk Hog

 

4.11 Monitoring Crash Files

 

 

4.4 Cut the Log in Half

 

4.8 Watching by grepping the Difference

 

4.12 Remember Daylight Savings Time

 

 

 

4.6 Watching the Disk Space

4.6.1 Description

4.6.1 Description

Use the df command to keep an eye on the disk space.

Flavors: AT&T, BSD

Shells: All

Syntax:

df -[kl] [-v] [filesystem]
bdf -[k] [filesystem]

The df command has different variances depending on the flavor you are using. In the examples that I will discuss, the BSD form of df is used to display the available space:

% df -kl
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t1d0s0    1203886  896255  187251    83%    /
/dev/dsk/c0t1d0s6     634354  194243  376681    34%    /usr
/dev/dsk/c0t2d0s0    3941764 3420328  324351    91%    /local
/dev/dsk/c0t0d0s6    3692074  591902 2730972    18%    /mnt

Check your man pages for the syntax that is comparable to the BSD form; it can take the form as one of the following:

% df
% df -k
% df -v
% bdf -k

Example One: Get the Percentage of Space

Monitoring the percentage of disk space being used can help keep the disk from filling up.

# df -k | awk '{ print $6"\t"$5}'

This simple command displays fields 5 and 6 from the output of the df command. It also places a tab in between the two fields, where field 6 is the name of the mounted filesystem and field 5 is the percentage of disk space being used.

Note - The field numbers on your system might be different. The goal is to extract the filesystem mounted and the percentage of disk space used.


Mounted capacity
/       83%
/usr    34%
/local  91%
/mnt    18%

You can write a script to monitor the percentage of disk space being used. After a certain predefined threshold limit is hit, mail a notification to the administrator.

#! /bin/sh
df -kl | grep -iv filesystem |  awk '{ print $6" "$5}' | while read LINE; do
 PERC=`echo $LINE | cut -d"%" -f1 | awk '{ print $2 }'`
 if [ $PERC -gt 98 ]; then
   echo "${PERC}% ALERT"  | Mail -s "${LINE} on `hostname` is almost full"  admin@rocket.ugu.com
 fi
done

Line 1: Define the shell to be used.

Line 2: Collect the amount of free space from the df command and strip out the header information. From the data collected, output the name of the mounted filesystem and the percentage of disk space used. Read in each line collected one by one.

Line 3: Take each line, strip out the character %, and store only the percentage value of the mounted filesystem into the variable $PERC.

Line 4: Check whether the value stored in the variable $PERC is greater than the defined threshold limit ( 98).

Line 5: If $PERC is greater than the threshold limit, send an alert to the administrators via email.

The script could be executed from the crontab every 30 minutes throughout the day:

5,35 * * * * /usr/local/bin/dfmon

The output then shows up in the administrator's mailbox. Here is a look at what is displayed when you read your mail after the script runs:

     Mailbox is '/var/mail/admin' with 3 messages [ELM 2.4 PL24 PGP2]
N  1   Oct 20 Unix Guru Universe  (30)   A HOT UNIX TIP
N  2   Oct 21 root                (10)   /local 99% on moon is almost full
   3   Oct 20 Harry Beeson        (25)   Need Unix Help.

When the email is sent off to the system administrator, the subject header provides direct information about the problem. Displayed is the filesystem that is filling up, the percentage full, and the host that the disk lives on.

Message 2/3  From root@moon.ugu.com             Oct 29, 98 00:28:53 am -0400
Date: Wed, 29 Oct 1998 00:28:53 -0413 (EDT)
To: admin@rocket.ugu.com
Subject: /local 99% on moon is almost full
99% ALERT

Example Two: Get the Amount of Free Space

If you need to track the amount of free space available, use the df command. The defined available free space field, is field 5 from the sample version of df you are using.

$ df -k | awk '{ print $6"\t"$4 }'
Mounted avail
/       187251
/usr    376681
/local  324351
/mnt    2730972

To monitor a filesystem that might drop below 100MB, modify the previous script from Example One. In this version, you can redirect the alert out to a log file instead of a mail message. If you expect the filesystem to grow and shrink in size, constantly going over and under the threshold limit, it might be easier to monitor the results from a file rather than to send constant email notifications.

#! /bin/sh
df -kl | grep -iv filesystem |  awk '{ print $6" "$4}' | while read LINE; do
 FSPC=`echo $LINE | awk '{ print $2 }'`
 if [ $FSPC -lt 100000 ]; then
   echo "`date` - ${LINE} space left on `hostname` " >> /var/log/df.log
 fi
done

Line 1: Define the shell to be used.

Line 2: Collect the amount of free space from the df command and strip out the header information. From the data collected, output the name of the mounted filesystem and the amount of free disk space left. Read in each line collected, one by one.

Line 3: Take each line and store only the available free disk space value of the mounted filesystem into the variable $FSPC.

Line 4: Check whether the value stored in the variable $FSPC is less than the defined threshold limit (100MB).

Line 5: If $FSPC is less than the threshold limit, output the results to the /var/log/df.log file.

As the script is executed, any alerts are appended to the file /var/log/df.log with a date stamp attached to it. This allows for easy monitoring of the log file.

The script can be added to the crontab so that it is executed every 30 minutes or less, depending on your environment:

5,35 * * * * /usr/local/bin/dfmail

Reason

Monitoring the disk's activity and growth is important to users and management within your environment. Being able to project growth patterns and lack of disk space helps in the future to set up the planning stages for new space requirements.

Real World Experience

Two forms of disk space monitoring were discussed: percentage and availability. Each method needs to be examined on an individual basis for the environment you are working in. In the end you could end up with more emails than you ever imagined.

The percentage method works well with filesystems that are not expected to fluctuate radically from one end of the spectrum to the other in available free disk space. It works well with filesystems that continuously keep growing and expanding in size. The only thing to watch out for is, with filesystems growing into the gigabytes, a 99% reading could actually equal one full gigabyte of disk space.

When monitoring the available disk space that remains, you have much more control over your threshold limit. This method works well on single root filesystems or a data disk. If your threshold was set to 50MB, that might be fine for your root and user filesystems. Your data disk, however, might need a threshold setting of 200MB or 300MB. If you wait until 50MB, it might be too late, and users might start experiencing unexpected problems.

Other Resources

Man pages:

bdf, cron, crontab, df

UNIX Hints & Hacks

ContentsIndex

Chapter 4: System Monitoring

 

Previous ChapterNext Chapter

Sections in this Chapter:

   

4.1 Monitoring at Boot Time

 

4.5 Mail a Process

 

4.9 Monitoring with ping

 

4.2 Starting with a Fresh Install

 

4.6 Watching the Disk Space

 

4.10 Monitoring Core Files

 

 

4.3 Monitor with tail

 

4.7 Find the Disk Hog

 

4.11 Monitoring Crash Files

 

 

4.4 Cut the Log in Half

 

4.8 Watching by grepping the Difference

 

4.12 Remember Daylight Savings Time

 

 

 

© Copyright Macmillan USA. All rights reserved.