Perl Script for Automatically Backing Up Log Files

This Perl script was written for backing up system log files in the /tmp/ directory and tomcat6 log files in the /usr/share/tomcat6/logs/ directory.

Modify the code to suite your personal needs. Hopefully this will provide a helpful platform to build of off!

**********Begin Perl Script for Log Backup***********

#!/usr/bin/perl -w

use strict;

########################################################################
#
# Created by Andrew Stout <stout.andrew@ymail.com>
# Origination Date: Jan 24, 2011
#
# Current Version: 1.1.2
# Release Date: Mar 13, 2012
#
# This Perl application archives Linux system and Apache Tomcat log files for
# system and network administrators. Each file is compressed and renamed with a
# timestamp, that identifies when the compression occured, appended as a prefix
# to the original filename and .tar.gz as the suffix, representing the compression
# methods used. These files are then relocated to the /home/logArchive/ directory
# for local-system storage
#
# To automate this archive script, as root user:
# 1. cp or mv this script (archiveLogs_1.1.2.pl) to /etc/bin/
# 2. Change the user and group ownership of this script to root, with chown, and
# set the permissions to allow read, write, and execute by the root user, with
# chmod.
# 3. Open /etc/crontab and add '1 0 * * *  root archiveLogs.pl', then save and close  # the file.
# 4. In 'yast runlevel', enable the cron service, so that it will automatically start in
# the event of a reboot. This will ensure the archival script remains automated.
#
########################################################################

# Designates and makes, if not already in existance, the target directory for archiving
# compressed log files
my $destination="/home/logArchive";
system "mkdir $destination";

#### .log Evaluation
# This opens the specified system directory and reads the files into the DIR filehandle,
# then finds the files ending in ".log" and places them into an array (a list). The contents
# of this array are then evaluated for size to determine whether or not they are large enough
# to archive
chdir "/tmp"
    or die "cannot chdir to ../tmp/: $!";
opendir DIR, "/tmp/";
my @logEntry = grep {/.+\.log$/} (readdir DIR);
foreach my $filename (@logEntry) {
        my $size = (stat($filename))[7];

    # Makes a copy of the orginal filename and then removes the extension,
    # while preserving the original. The truncated filename will be inserted
    # into the new archival name, once the file has been compressed
    my $filenameTrunc = $filename;
    $filenameTrunc =~ s/^(.+)\.log$/$1/;

        # Ensures that existing log files (ones created automatically by the system) have
        # the correct owner:group and permissions settings to grant the configured system
        # users writable access to the logs. This is generally only necessary before the log
        # file has grown large enough to be archived for the first time. After that, this
        # script will automatically replace the log files and configure the proper ownership
        # and permissions. The permissions will allow the ownwer to read and write to the file,
        # members of the same group to read and write to the file, and other users to only
        # read the file, but not write. This is based on a three-bit binary activation for
        # "rwx", with read = 4, write = 2 (r+w=6), and execute = 1.
        if ($size < 1_048_576) {
                system "chown ido.emf-fiscal.net:mercury $filename && chmod 664 $filename";
        }

    # EValuates the file size, then compresses and archives it if it is larger than 1MB.
    # If the file is smaller than 1MB, it leaves it alone.
    if ($size >= 1_048_576) {
        # Extracts the year, month, day, hour, minute, and second elements from the
        # localtime function, in that order, and formats the year to be four digits,
        # with every other element being two digits, with a preceding 0, if necessary.
        # This time stamp is then placed at the beginning of the new filename, when
        # the log file is compressed
        (my $Y, my $m, my $d, my $H, my $M, my $S) = (localtime)[5,4,3,2,1,0];
        my $timeStamp = sprintf
                "%4d%02d%02d%02d%02d%02d",
                ($Y+1900, $m+1, $d, $H, $M, $S);
           
                # Shorten the code by assigning a temporary filename for the original file to
        # $tmpFile. The original file will be moved to this temporary name, if it does
        # not exist, and the original file will be emptied for new entries. Compression
        # will then be performed on the new temp file
        my $tmpFile = "$timeStamp.$filenameTrunc.tmp";

        # Checks whether or not the temporary filename already exists. If not, it moves
        # the original file to the new filename, then empties the original log file.
        # The compression and renaming, for the log archive is performed on the new
        # temporary file, then the temporary file is deleted automatically. This prevents
        # the potential loss of data entries, which could occur if the compression were
        # performed on the original file at the same time as new data was being written.
        if (-e $tmpFile) {
            warn "Can't rename $filename to $tmpFile: $tmpFile already exists\n";
        } elsif (rename $filename, $tmpFile) {
            system "touch $filename && chown ido.emf-fiscal.net:mercury $filename && chmod 664 $filename";
            system "tar -czf $timeStamp.$filenameTrunc.tar.gz $tmpFile && echo $filename && rm $tmpFile";
            system "mv $timeStamp.$filenameTrunc.tar.gz $destination";
        } else {
            warn "rename $filename to $tmpFile failed: $!\n";
        }
        }
}

#### .out Evaluation
# This opens the specified system file directory and reads the files into the DIR filehandle,
# then finds the files ending in ".out" and places them into an array (a list). The contents
# of this array are then evaluated for size to determine whether or not they are large enough
# to archive
chdir "/usr/share/tomcat6/logs"
    or die "cannot chdir to ..tomcat6/logs/: $!";
opendir DIR, "/usr/share/tomcat6/logs/";
my @outFiles = grep {/.+\.out$/} (readdir DIR);
foreach my $filename (@outFiles) {
    my $size = (stat($filename))[7];
   
    # Makes a copy of the orginal filename and then removes the extension,
    # while preserving the original. The truncated filename will be inserted
    # into the new archival name, once the file has been compressed
    my $filenameTrunc = $filename;
    $filenameTrunc =~ s/^(.+)\.out$/$1/;

        # Ensures that existing log files (ones created automatically by the system) have
        # the correct owner:group and permissions settings to grant the configured system
        # users writable access to the logs. This is generally only necessary before the log
        # file has grown large enough to be archived for the first time. After that, this
        # script will automatically replace the log files and configure the proper ownership
        # and permissions. The permissions will allow the ownwer to read and write to the file,
        # members of the same group to read and write to the file, and other users to only
        # read the file, but not write. This is based on a three-bit binary activation for
        # "rwx", with read = 4, write = 2 (r+w=6), and execute = 1.
        if ($size < 1_048_576) {
                system "chown tomcat:tomcat $filename && chmod 664 $filename";
        }

    # Evaluates the file size, then compresses and archives it if it is larger than 1MB.
    # If the file is smaller than 1MB, it leaves it alone.
    if ($size >= 1_048_576) {
                # Extracts the year, month, day, hour, minute, and second elements from the
        # localtime function, in that order, and formats the year to be four digits,
        # with every other element being two digits, with a preceding 0, if necessary.
        # This time stamp is then placed at the beginning of the new filename, when
        # the log file is compressed
        (my $Y, my $m, my $d, my $H, my $M, my $S) = (localtime)[5,4,3,2,1,0];
        my $timeStamp = sprintf
                "%4d%02d%02d%02d%02d%02d",
                ($Y+1900, $m+1, $d, $H, $M, $S);
           
        # Shorten the code by assigning a temporary filename for the original file to
        # $tmpFile. The original file will be moved to this temporary name, if it does
        # not exist, and the original file will be emptied for new entries. Compression
        # will then be performed on the new temp file
        my $tmpFile = "$timeStamp.$filenameTrunc.tmp";

        # Checks whether or not the temporary filename already exists. If not, it moves
        # the original file to the new filename, then empties the original log file.
        # The compression and renaming, for the log archive is performed on the new
        # temporary file, then the temporary file is deleted automatically. This prevents
        # the potential loss of data entries, which could occur if the compression were
        # performed on the original file at the same time as new data was being written.
        if (-e $tmpFile) {
            warn "Can't rename $filename to $tmpFile: $tmpFile already exists\n";
        } elsif (rename $filename, $tmpFile) {
            system "touch $filename && chown tomcat:tomcat $filename && chmod 664 $filename";
            system "tar -czf $timeStamp.$filenameTrunc.tar.gz $tmpFile && echo $filename && rm $tmpFile";
            system "mv $timeStamp.$filenameTrunc.tar.gz $destination";
        } else {
            warn "rename $filename to $tmpFile failed: $!\n";
        }
    }
}

# Evaluate the age of files in the log archive and move those that are a year old to a new location
chdir "/home/logArchive"
        or die "cannot chdir to ..logArchive/: $!";
opendir DIR, "/home/logArchive/";
my @archiveFiles = grep {/.+\.out$/} (readdir DIR);
foreach my $filename (@archiveFiles) {
        my $size = (stat($filename))[7];
}

system "/etc/init.d/apache2 restart";
system "/etc/init.d/tomcat6 restart";

**************End Perl Script for Log Backup*************

Tutorial: Linux File Permissions and Ownership

            WARNING: Running system tasks as the root user is very dangerous! Files deleted from the command line cannot be recovered and serious system damage can occur! Use the root user account with extreme caution!

            This tutorial uses the yourFile file to explain Linux file permissions and ownership. This explanation can be applied to almost any Linux file system, because permissions and ownership formatting is pretty standardized across the Linux platform.

Identifying Permissions and Ownership

Example 1: Original

• -rw------- 1 root root 56318 Sep 21 16:40 yourFile

Example 2: After changing permissions with chmod and ownership with chown

• -rwxrwxrwx 1 tester users 56318 Sep 21 16:40 yourFile

            In Example 1, above, we see the file type (shown in the first position as a '-'); the access permissions (shown in the latter nine positions as 'rw-------'); the owner of the file, referred to as the user (root); the user's group (root); the file size in bytes (56318); the date and time the file was last modified (Sep 21 16:40); and the name of the file itself (yourFile).

            According to the details listed in Example 1, the file is a regular file (-) and the owner has permission to read and write to the file, but cannot execute it (rw-). However, no one else has permission to access the file ('---' and '---'), except for the root user, who automatically has access (regardless of who the file belongs to).

            Listed after the permissions, we see 'root root'. The first 'root' (on the left) represents the owner of the file. The owner is actually referred to as the user, in the context of assigning permissions (more on this later). The second 'root' (on the right) represents the group that the owner belongs to. This example can be confusing, because both the owner and the group have the same name.
            It is easier to understand ownership by looking at Example 2, in which the owner is 'tester' and the owner's group is 'users'.

File Type 

           As stated, the first position in Example 1, above, is a hyphen (-) and identifies the file type. Typically speaking, the file type will never be changed, so we will list the possible values for this position, but not discuss how to change them: 

Table 1: File Types

• - = regular file
• d = directory
• l = symbolic link (similar to a shortcut that is used on a MS Windows desktop)
• s = socket (configured IP address and port number combo)
• b = block device
• c = character device
• p = named pipe

Permissions

            There are three sets of permissions, following the file type, and each set includes three letters that identify the individual permissions of read, write, and execute: 

Table 2: Permissions Sets

• The first set of 'rwx' is for the user (u), also known as the owner of the file.
• The second set of 'rwx' is for the user's group (g).
• The third set of 'rwx' is for other (o), also known as “the rest of the world”.

There are four possible values associated with the file permissions: 

Table 3: Permissions Types

• - = permission is turned off (the hyphen has a different definition when used with permissions, than it does for file types)
• r = read (able to open the file to read its contents)
• w = write (able to modify the contents of the file)
• x = execute (able to cause the file to perform a task, such as running a utility or application).

            The presence of a letter (r, w, x) in one of the positions means that the permission is turned on within that specific set (but not necessarily within the other two), but a hyphen (-) means that the permission is turned off. In other words, if the file details show '-rwxrwxrwx', as in Example 2, then the file is fully accessible to everyone. However, if the file details show '-rw-------', as in Example 1, then only the owner (u) has permission to access the file, and then only to read it, or modify its contents. The rest of the users within the owners group (g), and every other individual (o), are barred from accessing the contents of the file or executing it.

Ownership (Users and Groups)

            Each user on a Linux system has their own user account (with log-in and home directory), but each user is also assigned to a group of similar users. The groups help to categorize users according to their system access levels. In fact, a new user is placed in the 'users' group by default, whereas the root user (i.e. super user or administrator) is in its own group called 'root'. A root user is allowed to have much more access to system files and processes than a standard user, which is why they are assigned to a different group.

            For example, if there is a group called 'developers' and a group called 'installers' on the same system, then the programmers can be assigned to the developers group, giving them full access to source code and development utilities, but the installers group may only have access to PDF files containing configuration and installation procedures for the same software. There is no concern about the installers modifying or deleting source code by accident, because the files are inaccessible to them.

Modifying Permissions and Ownership

Changing File Permissions by Lettered Designations: 

Example 3: chmod Syntax for Lettered Designations (to change file permissions)

• Example 3a: Original File
• -rw------ 1 root root 56318 Sep 21 16:40 yourFile
• Example 3b: Basic Syntax
• chmod [ugo][+-=][rwx] [filename]
• Example 3c: Changing User (u), Group (g), and Other (o) Permissions to Allow Full Access for read (r), write (w), and execute (x)
• chmod ugo=rwx yourFile
• Example 3d: Results of Permissions Change, Part 1
• -rwxrwxrwx 1 root root 56318 Sep 21 16:40 yourFile
• Example 3e: Removing write and execute Permissions from User, Group, and Other
• chmod ugo-wx yourFile
• Example 3f: Results of Permissions Change, Part 2
• -r--r--r-- 1 root root 56318 Sep 21 16:40 yourFile
• Example 3g: Add write Permissions to User and Group
• chmod ug+w yourFile
• Example 3h: Results of Permissions Change, Part 3
• -rw-rw-r-- 1 root root 56318 Sep 21 16:40 yourFile

            Example 3, above, shows syntax options for using chmod to change file permissions with lettered designations (see Examples 3a to 3h for sequential permissions changes). The letters before the equal sign (=) represent the permissions combinations for user (u), user's group (g), and other (o), in that order. The letters after the equal sign (=) represent the permissions, themselves: read (r), write (w), and execute (x).

To change file permissions using lettered designations:

1.    Type chmod at the command line.

2.    Choose the permissions sets to be included; either user (u), group (g), or other (o).

3.    Next, identify the function to be performed with the designated permissions; either add (+) them to the existing permissions, remove (-) them from the existing permissions, or choose to have the designated permissions equal (=) the final status of the chosen sets (additions or removals are performed automatically to match the designated permissions).

4.    After designating the task to be performed, designate the individual permissions (r, w, x) that the task will affect.

5.    Finally, add the name of the file for which the permissions will be modified.

            The challenge to modifying permissions with lettered designations is that if different sets (u, g, or o) need to be assigned different combinations of permissions, the chmod command would have to be executed for each unique set of permissions, which means it may have to be run three times in order to get all the sets configured correctly.

            There is an easier way to modify permissions, though, and that is by using binary designations, shown below.

Changing File Permissions by Binary Designations: 

Example 4: chmod Syntax for Binary Designations (to change file permissions)

• Example 4a: Original File
• -rw------- 1 root root 56318 Sep 21 16:40 yourFile
• Example 4b: Basic Syntax
• chmod [###] [filename]
• Example 4c: Changing User permissions to 'rwx' (7), Group permissions to 'r-x' (5), and Other permissions to 'r--' (4)
• chmod 754 yourFile
• Example 4d: Results of Permissions Change
• -rwxr-xr-- 1 root root 56318 Sep 21 16:40 yourFile
• Example 4e: A Sampling of Other Binary Permissions Combinations
• chmod 777 [filename]                       =          -rwxrwxrwx
• chmod 500 [filename]                       =          -r-x------
• chmod 751 [filename]                       =          -rwxr-x--x
• chmod 711 [filename]                       =          -rwx--x--x
• chmod 110 [filename]                       =          ---x--x---

            If the permissions for User and Group need to be different from Other, using lettered designations with chmod would require the command to be executed for each unique set of permissions (in other words, two times). So if User and Group had the same permissions, but Other had a different combination, then User and Group could be configured simultaneously with chmod, then chmod would have to be executed a second time, for Other.

            The repetitious use of chmod can be resolved by using binary designations to set permissions, instead of using lettered designations (see Examples 4a to 4e, above, for the chown syntax). Binary equivalents make it so that all three sets of permissions can receive unique settings, simultaneously. For example: 'chmod 754 [filename]' would set the permissions to '-rwxr-xr--', otherwise, you would have to run the chmod command three times for u=rwx, g=rx, and o=r. As you can see, the binary designations are much more efficient.

 Understanding Binary:

            Each number in the three-digit permissions parameter represents a three-bit designations for one of the 'rwx' permissions sets, seen as the lettered designations in the file details. Therefore, the three numbers represent the permissions combinations for user (u), user's group (g), and other (o), in that order. The permissions represent read (r), write (w), and execute (x).

           Each three-bit binary designation can range from 0 to 7: 

Table 4: Permissions with Binary Designations

• 0 = --- (no permissions)
• 1 = --x
• 2 = -w-
• 3 = -wx
• 4 = r--
• 5 = r-x
• 6 = rw-
• 7 = rwx (or full permissions)

            Example 4, above, shows syntax options for using chmod to change file permissions with binary designations, and Table 4 identifies the binary-to-letter conversions.

To change file permissions using binary designations:

1.    Type chmod at the command line.

2.    Determine the permissions-combination needed for each of the three sets and refer to Table 4 to determine the binary equivalent.

3.    Enter all three binary determinations after chmod, without spaces between them.

4.    Finally, add the name of the file for which the permissions will be modified.

Changing File Ownership: 

Example 5: chown syntax (to change file ownership)

• Example 5a: Original File
• -rw------- 1 root root 56318 Sep 21 16:40 yourFile
• Example 5b: Basic Syntax
• chown [user]:[group] [filename]
• Example 5c: Changing User to 'tester' and Group to 'users'
• chown tester:users yourFile
• Example 5d: Result of Ownership Change
• -rw------- 1 tester users 56318 Sep 21 16:40 yourFile

            There are times in which the user (owner) or group assigned to a file will need to be changed, such as when the root user (also known as the super user) pulls a file from one computer system to another (which can automatically set the file ownership to root:root, as in Example 1 or Example 5, above). If the file needs to be accessible by a test group, as in our examples, above, then it should not be owned by the root user. In this case the root user would use chown to reassign the file ownership so that the necessary people can access and modify it.
            In these examples, we are using 'tester' as the owner and 'users' as the group. To change ownership of a file, we use the chown command (see Examples 5a to 5d, above, for the chown syntax).

To change file ownership:

1.    Type chown at the command line.

2.    Enter the new user name, followed by a colon (:).

3.    After the colon (without a space to separate them), enter the new group name.

4.    Finally, add the name of the file for which the ownership will be modified.

            However, if you are logged in to the system as a regular user, then once you change the user or group ownership of a file, you may lose all access to that file, depending on the permissions that are set. Due to this, it may be necessary to modify permissions before modifying ownership, or it may be necessary to log in as the root user (super user) to modify the file, because the root user will retain the ability to the access the file even after the ownership has been changed.

            WARNING: Running system tasks as the root user is very dangerous! Files deleted from the command line cannot be recovered and serious system damage can occur! Use the root user account with extreme caution!