|
UNIX Hints & Hacks |
 |
||||||||||||||||||||||||||||||||||||
|
Chapter 6: File Management |
|
||||||||||||||||||||||||||||||||||||
|
In the interconnected mesh of systems that can exist on your LAN, there are times when migrating files over from one host to another is necessary. There are several ways to copy files to a remote host. Here are a few ways to achieve this using rcp, tar, and ftp, and through network file system (NFS) mount points.
Flavors: AT&T, BSD
Shells: All
Syntax:
rcp -[pr] [source] [host:][destination]
For this to work, the local system and remote system must have a trusted relationship between the local and remote hosts. The level of trust is dependent on what type of account is performing the cp command.
This is the easiest of all the commands. It is similar to the cp command. Simply provide the source of the files to copy, the remote host, and the destination on that remote host where you want the files. The host and destination need to be separated by a colon. If the user joedee has files on a system called rocket , and wants to copy them to a remote system named planet , the command is
# rcp -pr joedee planet:/home
The command executes a remote copy that attempts to retain all permissions and ownership ( p) on the files under the directory joedee while the files are recursively ( r) copied into /home on the remote system planet.
Flavors: AT&T, BSD
Shells: All
Syntax:
tar -cvf [source] [destination] | rsh hostname '(cd [dir];tar -xvfB [source] )'
A copy can be made of an entire directory structure to a remote host using the tar command. Although it might look ugly, it is easy to understand. The process involves archiving the data, sending it to a remote host, and extracting it in the appropriate directory. If you want to copy the joedee home directory to a remote host called planet using tar, the command is
# tar cf - joedee | rsh planet '( cd /home; tar xfB - )'
The first tar command creates ( cf) an archive of joedee and places it into a buffer, instead of creating a file or writing it off to a tape. While the data is in the buffered area ( -), it is sent to the remote host. When the data arrives at the remote host, the command changes to the directory /home and begins to extract ( xfB) the contents of the archive from the buffer where it resides.
A lowercase v option makes the tar command verbose the output of the data as the file is being archived ( cvf) or extracted ( xvf). If there are a large number of files to copy you might want to use the v option on one of tar commands but not both. Otherwise you end up with a lot of extraneous output. When deciding which verbose option to use, I like to apply the verbose to the extraction process. This helps to verify that the files are reaching their target destination. If you are on a slow network, the output of all the data can actually slow the copy process down by sending the resulting output of the process to STDOUT. If this is the case don't use the verbose option at all. There are other ways to monitor the data being copied that will be discussed later in this section.
The capital letter B is used to apply a blocking factor for the data as it crosses through pipes, networks, or other channels where record blocking can not be maintained. If this is not applied, a result could be a loss in data packets and the command will halt its processing of the command.
# tar cf - joedee | rsh planet '( cd /home; tar xf - )' /home tar: tape blocksize error
In most cases, if the B is left off, a tape blocksize error occurs and the copy will halt.
Flavors: AT&T, BSD
Shells: All
Syntax:
cp -pr [source] [destination] tar -cfB [source] [destination] | (cd [dir]; tar -cvfB [source] ) find dir -print | cpio -pudv destination
Although it is faster to copy files directly from one system to another when the two system are trusted, it is easier, but quite a bit slower, to copy files over NFS from the local system. It is necessary to have write permissions granted to a working account through the NFS mount point that is being exported from the remote host.
You can then use any cp, tar, or cpio to get the files over to the remote system. To do a straight copy and attempt to retain the permission while recursing down the subdirectories use the basic cp command that was discussed in Example One in section 6.1, "Copy Files with Permissions and Time Stamps."
# cp -pr jodee /hosts/planet/home
The cpio command from Example Three in section 6.1 can be used here as well.
# find jodee -print | cpio -pumdv /hosts/planet/home
As you can see there are many ways to get the files over. Using tar is yet one more way through the NFS mount point. This is the same command found in Example Two in section 6.1.
# tar -cf - project | (cd /disk2; tar -xvf - )
Many environments have policies in place that keep system administrators from exporting filesystems and allowing hosts to be trusted to one another. In these cases, you might have no choice but to resort back to the old FTP protocol. FTP is fast and reliable. Some parts of FTP can be considered a security risk, though.
The straightforward approach is to archive the data and FTP it over to the remote system. To FTP the home directory of joedee, you first tar the files into an archive, copy the archive over to the remote system with ftp, and then extract it into the new area.
# cd /home # tar -cf /tmp/jodee.tar joedee
ftp planet Connected to planet 220 planet FTP server ready. Name (planet:joedee): root 331 Password required for joeedee. Password: 230 User joedee logged in. ftp> cd /usr/tmp 250 CWD command successful. ftp> bin ftp> put joedee.tar 200 PORT command successful. 150 Opening ASCII mode data connection for 'jodee.tar'. 226 Transfer complete. local: joedee.tar remote: joedee.tar 2772 bytes sent in 0.014 seconds (2e+02 Kbytes/s ftp> quit 221 Goodbye.
|
Note - Access to FTP under the root account should never be allowed for security reasons. It provides complete access to the system if root is compromised. All modifications that require root access should be done from a shell and not from FTP. Place the word root in the /etc/users file to disable the root account from using FTP. |
The tar file is copied over to the new location on the remote system planet with a normal account. It should be placed in an easily accessible area that has enough disk space for the archived file. The file can then be extracted into the new area.
# cd /home # tar xf /usr/tmp/joedee.tar .
To help automate this process, write a script to create an archive of a directory and FTP it over to a remote system. This script can be considered dangerous and a security risk. It does require an exposed password to an account on the remote system. When the ftptar script is executed (with permissions set to 700), two arguments are passed to it. The first is the name of the source directory followed by the name of the remote host.
rocket 22% vi ftptar
#! /bin/sh tar cf $1.tar $1 echo "machine $2 login joedee password b0ssdq" > /usr/people/joedee/.netrc chmod 700 /usr/people/joedee/.netrc ftp $2 <<EOF bin cd /usr/tmp put $1.tar EOF rm -f /usr/people/joedee/.netrc
Line 1: Define the scripting language to use.
Line 2: Archive the file using the name passed to it by STDIN ( $1).
Line 3: Create the .netrc file in the current account's home directory that allows automating the ftp login process. Passed to the ftp process is the hostname passed in by STDIN ( $2), the account to access, and the password to use. (This file can be set up manually in ~/.netrc instead of in the ftptar script, for a little more security.)
Line 4: Lock down the permissions to the .netrc file. This is a must, not only for security but for ftp to use the file.
Line 5: Establish an FTP connection to the system that is passed in to STDIN ($2), and start processing the following lines through the FTP connection.
Line 6: Change to binary mode for the file transfer.
Line 7: Change to the temporary directory to store the archived file.
Line 8: Put the file on the remote host with the filename that is passed to it by STDIN ( $2).
Line 9: Exit FTP and complete the script.
Line 10: Clean up and remove the .netrc files for safety.
Copy the files from the home directory of joedee and onto planet. ftptar archives the joedee directory into an archived file, ftp the file over to planet, and put it in the directory /usr/tmp.
rocket 23% cd /home rocket 24% ftptar joedee planet
The script does not display any output. When you go to planet and look at the /usr/tmp directory, the archived file joedee.tar is created.
planet 1% ls -al /home drwxr-xr-x 9 root sys 512 Jun 16 1997 . drwxr-xr-x 7 root sys 512 Nov 5 16:22 .. drwxr-xr-x 5 ddelmar user 512 Jun 16 1997 ddelmar -rw-r--r-- 4 joedee user 1255712 Nov 16 22:00 joedee.tar drwxr-xr-x 8 kxnewm user 512 Jul 1 13:03 kxnewm
Files need to be relocated from one host to another. Like other tasks in UNIX there is more than one way to do execute the transfer. Choosing the appropriate method depends on you configuring your environment to optimize your network and your hosts in the safest and most-efficient way.
When you transfer large amounts of data across the network, it is best to log in to the remote system and verify that the transfer of files is working properly. You can do three things to verify that all is going well.
If you are piping the tar command through a remote shell to be extracted on the remote system, also check the process table and verify that there is a tar process running for the files that are being copied.
# ps -ef | grep tar joedee 5835 5829 0 20:46:09 ? 0:00 csh -c (cd /home; tar xfB -) joedee 5839 5835 0 20:46:09 ? 0:02 tar xfB -
Verify that the command executed from the local system created a directory on the remote machine and files being copied into the directory.
# ls -la /home/joedee total 48 drwxrwxr-x 2 joedee user 85 Nov 17 12:32 . drwxr-xr-x 35 root sys 4096 Nov 17 16:02 .. -rw-r--r-- 1 joedee user 689 Nov 11 18:58 .cshrc -rw-r--r-- 1 joedee user 704 Nov 11 18:58 .login -rwxr-xr-x 1 joedee user 154 Nov 17 12:46 .nologin -rw-r--r-- 1 joedee user 638 Nov 11 18:58 .profile drwxr-xr-x 1 joedee user 512 Nov 11 18:58 data
Check the disk usage on the directory and files being transferred and watch to make sure that it is growing in size.
# du -ks . 22876 . # du -ks . 23820 . # du -ks . 25356 .
This is especially necessary when you are not applying any verbose action of the command to display the progress that the transfer is making.
Man pages:
cpio, cp, du, find, ftp, rcp, rsh, tar
|
UNIX Hints & Hacks |
|
||||||||||||||||||||||||||||||||||||
|
Chapter 6: File Management |
|
||||||||||||||||||||||||||||||||||||
|
© Copyright Macmillan USA. All rights reserved.