Perforce is a revision control system. It uses a client-server model that allows developers to concurrently edit content that can then be atomically submitted to and stored on the server. The server generally stores content in RCS format or compressed binary files, while meta-data about file revisions, users and everything else are stored in Berkley database format.
The Perforce System Administrators guide gives a procedure for backing up all the data stored in a Perforce system. However, the prescribed method has a number of failings that mean it can be unsuitable in some circumstances, typically with very large installations. This document describes the problems with the Perforce backup method and then presents alternative methods that have been implemented and used in a production environment under Linux. Critically the methods presented allow backing up of very large sites without compromising server availability and have been used and refined in a production environment.
The System Administrators Guide gives 4 steps to backup a Perforce server. These are as follows:
p4 verify //...
p4d -jc
The first step ensures that digests stored in the database match a digest computed from the versioned files on the disk. This will discover any corruption or unexpected change to the versioned files since their submission, and is important to ensure that the data being backed up is intact.
The second step is where the database files are safely dumped to disk. Critically this command both truncates the transaction journal and creates a checkpoint while the database is locked, thus ensuring that the database is in a consistent state when dumped. While this step is running queries to the server will be blocked since the tables are locked.
The final two steps relate mainly to the running of the backup software itself. Following the verify and p4d -jc commands, the database is presented in a form that can be safely copied to other storage and backed up.
While this solution works and does provide safe and reliable backups, it is not necessarily ideal for large installations. Typically the first step can take a long time, and if executed with a wide view of the depot, as given in the example, will consume a large amount of memory on the server while running. Depending on the amount of memory the Perforce server has available to it, this alone can impact performance by causing excessive swapping.
The largest problem for this scheme is that the p4d -jc command may take a long time to complete, and that the database is locked throughout this operation. This means that any client requests to the server will not progress until this part of the backup has completed. For small servers where the checkpoint is quick to generate this is rarely a problem. However, as the database grows with files, revisions, clients and labels, the amount of data to be dumped increases as does the time it takes for the operation to complete. Critically this step was found to be taking over an hour, leaving the server unusable for interactive operations during this period.
The Perforce Backup Solution has failings in the verification step and then the journal truncation/checkpoint creation phases. The following sections describe alternative approaches that overcome these problems.
The only issues here are that the verify process may consume a large amount of memory on the server and also be very slow to complete. To address memory consumption, the simplest thing to do is to break up the verify into sections such that each depot and directory is verified individually. This reduces the memory used by the operation, and can simply be scripted as in the following example:
P4="p4 -u $P4USER -P $P4PASSWD" # Iterate the depots for DEPOT in `$P4 -Ztag -s depots | grep "info1: name" | cut -d " " -f 3`; do # Check any files right under the depot if $P4 -s files //$DEPOT/* | grep -q "^info: " ; then $P4 verify -q //$DEPOT/* fi # Now each directory in the depot in turn $P4 dirs //$DEPOT/* | while read DIR ; do $P4 verify -q "$DIR/..." done done
Script 1: Verification of a complete server is sections.
The script simply iterates each defined depot and verifies each directory under the depots in turn. Depending on the depth of the directory structures and the number of files that need verifying, iterating directories at a further level may also be beneficial. Another trick to reduce the memory consumed by the command is to verify all files beginning with 'a', then 'b' and so on. This can be done by passing the verify command appropriate wildcards such as //.../a*.
The second problem with verification is that it can take a very long time to complete is there are a large number of files on the server. Much of this comes down to raw IO and computation of the version file digest. However, one problem is that the server itself may actually verify some files multiple times due to lazy branching. Lazy branching is a technique used by the Perforce server to saves disk space when branching files. When a branch is submitted to Perforce, the server doesn't automatically create copies of the version files on disk. Instead it updates its database to indicate that a new file has been created in the depot, but points to the existing file on disk, creating a lazy branch. Unfortunately when selecting files to verify, the command uses the database to find which files need verifying and as such may verify a lazy branched file multiple times.
To overcome this problem we can search for version files on disk and ask that these are verified. This ensures that they are verified just once, and the following command can do this:
cd $P4ROOT && find . -name "*,[vd]" | sed "s/,[vd]$//" | sed "s/^\./\//" | p4 -x- verify
Script 2: Verification of versioned files once only.
This pipeline essentially finds all files or directories ending in ',v' or ',d' which are used by Perforce to store RCS text files and binary files respectively. The filename is then altered to remove the extension and prepend a forward slash to give the filenames in depot syntax. The result is then piped into the verify command.
A weakness of this scheme is that if a RCS file is completely removed, it will not be detected. Additionally if any extra files exist on disk that are not known to the Perforce server, warnings will be produced. This is probably beneficial except after obliterations of binary files where the Perforce server will leave empty directories with the ',d' extension (such empty directories can however be manually deleted). Because of this weakness, it is advisable to perform a verification of the full database using the first approach on a regular schedule.
The p4d -jc maybe unacceptable for large installations where it may prevent access to the server for the duration of its execution. This can be particularly problematic because this duration is proportional to the size of the database. This command does however have the nice property that the generated checkpoint is coincident in time to the journal rotation; should recovery be required at some point, the checkpoint can be used with subsequent journals to rebuild the database as it stood at any point in time.
Ideally we would still simultaneously rotate the journal and create a checkpoint, but without requiring the database to be locked for any long period. Essentially a checkpoint can be generated offline from a copy of the database files (db.*), so a method of grabbing these files while truncating the journal would meet this requirement. Perforce helps somewhat here by providing the p4d -c command. This instructs Perforce to take all locks on the database files, then execute some command before releasing the locks again. This can be used to take a consistent snapshot of the database files, from which a checkpoint can be generated without locking the database:
p4d -r $P4ROOT -c "cp $P4ROOT/db.* /tmp" nice p4d -r /tmp -jd
Script 3: Creating a checkpoint from a consistent snapshot.
The second command is shown to be called under nice since the example assumes this to be running on the same machine as the Perforce server, and hence allows other processes to utilise the CPU(s) in preference to this command.
While good, this unfortunately doesn't also truncate the journal. This would make it difficult to apply subsequent journals to the generated checkpoint. Ideally we would therefore be able to do something such as the following:
# This does not work p4d -r $P4ROOT -jj -c "cp $P4ROOT/db.* /tmp" nice p4d -r /tmp -jd
Script 4: Checkpoint with journal rotation. This does not work since p4d fails to interpret the command line as would be required.
Ideally the Perforce server would take locks, truncate the journal, run the command and then release the locks. However, it does not support this at present. The following can also be considered:
# This does not work p4d -r $P4ROOT -c "p4d -r $P4ROOT -jj; cp $P4ROOT/db.* /tmp" nice p4d -r /tmp -jd
Script 5: Checkpoint with journal rotation. This does not work due to deadlock from the database file locking.
Unfortunately here the outer p4d correctly takes the locks on the database files hence causing the inner p4d to block indefinitely as well as any other instance of p4d that may attempt to take locks on the database. It is not a good idea to run this on a live server.
A workaround is to carefully interfere with Perforce file locking. Using the strace command it is easy to see how files are locked by p4d during journal rotation.
...
[pid 4322] open("db.counters", O_RDWR|O_CREAT, 0666) = 4
[pid 4322] fstat(4, {st_mode=S_IFREG|0640, st_size=16384, ...}) = 0
[pid 4322] pread(4, "\266\355\0\0\2\0\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192, 0) = 8192
[pid 4322] flock(4, LOCK_EX) = 0
[pid 4322] open("db.logger", O_RDWR|O_CREAT, 0666) = 5
[pid 4322] fstat(5, {st_mode=S_IFREG|0640, st_size=16384, ...}) = 0
[pid 4322] pread(5, "\266\355\0\0\2\0\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192, 0) = 8192
[pid 4322] flock(5, LOCK_EX) = 0
[pid 4322] open("db.user", O_RDWR|O_CREAT, 0666) = 6
[pid 4322] fstat(6, {st_mode=S_IFREG|0640, st_size=114688, ...}) = 0
[pid 4322] pread(6, "\266\355\0\0\2\0\0\0\0 \0\0\0\0\0\0\r\0\0\0\2\0\0\0\0\0"..., 8192, 0) = 8192
[pid 4322] flock(6, LOCK_EX) = 0
...
Listing 1: Fragment of strace output from p4d -jj.
Essentially each database file is opened in a predefined order (to ensure no risk of deadlock within the Perforce server process) and locked using flock(). This can be intercepted under Linux by setting the LD_PRELOAD environment variable such that a library is dynamically linked against the executable before it is ran. A small library was therefore developed to discover attempted locking of the database files and to pretend to perform locking, but to not actually do anything. The library was named flock.so and so the command to create a backup snapshot becomes as follows:
# This does work very well! p4d -r $P4ROOT -c "LD_PRELOAD=flock.so p4d -r $P4ROOT -jj; cp $P4ROOT/db.* /tmp" nice p4d -r /tmp -jd
Script 6: Checkpoint with journal rotation.
This now allows the journal to be rotated and the db files to be copied while the database is locked and hence consistent. The source code for flock can be downloaded and contains instruction on how it can be built.
The scheme described allows journal rotation in a context where any other command can be executed without the database state changing. The examples given copy the database files, which still may take some time for large databases. A better approach is therefore to use a snapshotting file system that can make a near instantaneous copy of a filesystem using copy-on-write techniques or other tricks.
Under Linux, both EVMS and LVM allow snapshots of filesystems to be taken with the need for no additional hardware providing that there is sufficient free space on disk. LVM2 under RedHat ES4 has been found to work very effectively in a production environment. The following bash functions have been used to manipulate checkpoints under the Perforce server when needed:
#!/bin/bash
# snapshot_create <lvname> <command>
# Create a snapshot of the logical volume <lvname> using the remaining
# space on the volume group. If <command> is specified, run the command
# before creating the snapshot, but while Perforce locks are held.
#
# Returns 0 if the volume was created without error.
#
function snapshot_create
(
local freespace=`sudo vgdisplay -c $VOLGROUP | cut -d : -f 16`
if [ -z "$2" ] ; then
$P4D -c "sync; sudo lvcreate -s -l $freespace -n p4snap /dev/$VOLGR
else
$P4D -c "$2; sync; sudo lvcreate -s -l $freespace -n p4snap /dev/$V
fi
# Call lvdisplay to set the exit code
sudo lvdisplay /dev/$VOLGROUP/p4snap > /dev/null
)
# snapshot_delete
# Delete the snapshot volume
function snapshot_delete
{
sudo umount /dev/$VOLGROUP/p4snap
sudo lvremove -f /dev/$VOLGROUP/p4snap
}
# Example usage:
# Create a snapshot of the depot with the journal truncated:
snapshot_create p4depot "LD_PRELOAD=flock.so $P4D -jj"
Script 7: LVM2 snapshot functions with Perforce.
The above functions and example allow for atomic journal truncation and snapshot creation. Once the snapshot is created, it can be mounted and used to produce a checkpoint without holding database locks. Additionally the snapshot allows for other checks to be ran on the database; the Perforce verification functions that are provided are:
p4d -xv : Low level verification of database file structures.
p4d -xx : Table level consistency cross checks.
Note that in the example, the LVM functions are called under sudo. This is not essential if the script is running with root privileges, but use of sudo maybe preferable to avoid running the server with more privileges than it requires.
Finally it is important to note that the status code of the checkpoint creation command, p4d -jc or p4d -jd, should be checked to ensure that it is zero. This indicates that no errors were encountered during checkpoint creation, and adds a little more robustness to the scheme.
Having backed up the database, the version controlled files can then be backed-up as desired. It does not matter that some of the versioned files could potentially be newer than the database checkpoint since the server ignores file revisions that are not represented in the database, replacing them upon submission of newer revision as required.
Having already used snapshots for the backup of the database, this method can also be used to backup the versioned files, although snapshotting is not essential. In this case there is no issue with database consistency, so simply issuing the snapshot commands under p4d -c is satisfactory. The snapshot_create function in Script 7 allows for creation of a snapshot under p4d if no command argument is supplied.
It is worth noting that this system is being used on a real server in a production environment where interactive use of Perforce is required almost continually. The following gives some data about the setup which is used to provide services for about 540 users in varying locations and time zones.
| Hardware | HP Proliant DL380, quad Xeon 2.80 GHz |
|---|---|
| Operating System | RedHat ES4 |
| RAM | 12GB |
| Disks | 6 x 146GB Ultra320 SCSI drives in RAID 0+1 configuration. |
| Storage Controller | Smart Array 6i with BBWC + 196Meg cache. |
Table 1: Server configuration overview.
|
|
Table 2: Perforce database file sizes.
Checkpoint creation using p4d -jc takes around 1 hour for these database files, during which no queries are handled by the server. Using the snapshotting technique with the flock library, a snapshot is taken in around 0.4 seconds. The checkpoint is then created from the snapshot under nice without noticeably impairing performance, taking around 80 minutes to complete.
The server contains 19,915,718 revision controlled files as reported by p4 files //.... On disk this is stored as 937,268 physical files, consuming approximately 70 GB or storage. Verification these files in full using p4 verify for each depot in turn, as in Script 1, takes just over 12 hours. Verification of only the physical files, as in Script 2, takes only 2 hours.
The source to flock and a Perforce server verification script can be downloaded. These are released as public domain. This means that there is no copyright and anyone is able to take a copy for free and use it as they wish, with or without modifications, and in any context they like, commercially or otherwise. The only limitation is that I don't guarantee that the software is fit for any purpose or accept any liability for it's use or misuse - the software is without warranty.