# Developer Information

Subversion is an open-source software project developed under an Apache-style software license. The project is financially backed by CollabNet, Inc., a California-based software development company. The community that has formed around the development of Subversion always welcomes new members who can donate their time and attention to the project. Volunteers are encouraged to assist in any way they can, whether that means finding and diagnosing bugs, refining existing source code, or fleshing out whole new features.

This chapter is for those who wish to assist in the continued evolution of Subversion by actually getting their hands dirty with the source code. We will cover some of the software's more intimate details, the kind of technical nitty-gritty that those developing Subversion itself—or writing entirely new tools based on the Subversion libraries—should be aware of. If you don't foresee yourself participating with the software at such a level, feel free to skip this chapter with confidence that your experience as a Subversion user will not be affected.

## Layered Library Design

Subversion has a modular design, implemented as a collection of C libraries. Each library has a well-defined purpose and interface, and most modules are said to exist in one of three main layers—the Repository Layer, the Repository Access (RA) Layer, or the Client Layer. We will examine these layers shortly, but first, see our brief inventory of Subversion's libraries in . For the sake of consistency, we will refer to the libraries by their extensionless Unix library names (e.g.: libsvn_fs, libsvn_wc, mod_dav_svn).

### A Brief Inventory of the Subversion Libraries

Library | Description
--- | ---
libsvn_client | Primary interface for client programs
libsvn_delta | Tree and text differencing routines
libsvn_fs | The Subversion filesystem library
libsvn_ra | Repository Access commons and module loader
libsvn_ra_dav | The WebDAV Repository Access module
libsvn_ra_local | The local Repository Access module
libsvn_ra_svn | A custom protocol Repository Access module
libsvn_repos | Repository interface
libsvn_subr | Miscellaneous helpful subroutines
libsvn_wc | The working copy management library
mod_authz_svn | Apache authorization module for Subversion repositories access via WebDAV.
mod_dav_svn | Apache module for mapping WebDAV operations to Subversion ones

The fact that the word miscellaneous only appears once in is a good sign. The Subversion development team is serious about making sure that functionality lives in the right layer and libraries. Perhaps the greatest advantage of the modular design is its lack of complexity from a developer's point of view. As a developer, you can quickly formulate that kind of big picture that allows you to pinpoint the location of certain pieces of functionality with relative ease.

Another benefit of modularity is the ability to replace a given module with a whole new library that implements the same API without affecting the rest of the code base. In some sense, this happens within Subversion already. The libsvn_ra_dav, libsvn_ra_local, and libsvn_ra_svn all implement the same interface. And all three communicate with the Repository Layer—libsvn_ra_dav and libsvn_ra_svn do so across a network, and libsvn_ra_local connects to it directly.

The client itself also highlights modularity in the Subversion design. While Subversion currently comes with only a command-line client program, there are already a few other programs being developed by third parties to act as GUIs for Subversion. Again, these GUIs use the same APIs that the stock command-line client does. Subversion's libsvn_client library is the one-stop shop for most of the functionality necessary for designing a working Subversion client (see ).

### Repository Layer

When referring to Subversion's Repository Layer, we're generally talking about two libraries—the repository library, and the filesystem library. These libraries provide the storage and reporting mechanisms for the various revisions of your version-controlled data. This layer is connected to the Client Layer via the Repository Access Layer, and is, from the perspective of the Subversion user, the stuff at the other end of the line.

The Subversion Filesystem is accessed via the libsvn_fs API, and is not a kernel-level filesystem that one would install in an operating system (like the Linux ext2 or NTFS), but a virtual filesystem. Rather than storing files and directories as real files and directories (as in, the kind you can navigate through using your favorite shell program), it uses a database system for its back-end storage mechanism. Currently, the database system in use is Berkeley DB. The choice of Berkeley DB brought several automatic features that Subversion needed, such as data integrity, atomic writes, recoverability, and hot backups. However, there has been considerable interest by the development community in giving future releases of Subversion the ability to use other back-end database systems, perhaps through a mechanism such as Open Database Connectivity (ODBC).

The filesystem API exported by libsvn_fs contains the kinds of functionality you would expect from any other filesystem API: you can create and remove files and directories, copy and move them around, modify file contents, and so on. It also has features that are not quite as common, such as the ability to add, modify, and remove metadata (properties) on each file or directory. Furthermore, the Subversion Filesystem is a versioning filesystem, which means that as you make changes to your directory tree, Subversion remembers what your tree looked like before those changes. And before the previous changes. And the previous ones. And so on, all the way back through versioning time to (and just beyond) the moment you first started adding things to the filesystem.

All the modifications you make to your tree are done within the context of a Subversion transaction. The following is a simplified general routine for modifying your filesystem:

1. Begin a Subversion transaction.

2. Make your changes (adds, deletes, property modifications, etc.).

3. Commit your transaction.

Once you have committed your transaction, your filesystem modifications are permanently stored as historical artifacts. Each of these cycles generates a single new revision of your tree, and each revision is forever accessible as an immutable snapshot of the way things were.

#### The Transaction Distraction

The notion of a Subversion transaction, especially given its close proximity to the database code in libsvn_fs, can become easily confused with the transaction support provided by the underlying database itself. Both types of transaction exist to provide atomicity and isolation. In other words, transactions give you the ability to perform a set of actions in an all or nothing fashion—either all the actions in the set complete with success, or they all get treated as if none of them ever happened—and in a way that does not interfere with other processes acting on the data.

Database transactions generally encompass small operations related specifically to the modification of data in the database itself (such as changing the contents of a table row). Subversion transactions are larger in scope, encompassing higher-level operations like making modifications to a set of files and directories which are intended to be stored as the next revision of the filesystem tree. If that isn't confusing enough, consider this: Subversion uses a database transaction during the creation of a Subversion transaction (so that if the creation of Subversion transaction fails, the database will look as if we had never attempted that creation in the first place)!

Fortunately for users of the filesystem API, the transaction support provided by the database system itself is hidden almost entirely from view (as should be expected from a properly modularized library scheme). It is only when you start digging into the implementation of the filesystem itself that such things become visible (or interesting).

Most of the functionality provided by the filesystem interface comes as an action that occurs on a filesystem path. That is, from outside of the filesystem, the primary mechanism for describing and accessing the individual revisions of files and directories comes through the use of path strings like `/foo/bar`, just as if you were addressing files and directories through your favorite shell program. You add new files and directories by passing their paths-to-be to the right API functions. You query for information about them by the same mechanism.

Unlike most filesystems, though, a path alone is not enough information to identify a file or directory in Subversion. Think of a directory tree as a two-dimensional system, where a node's siblings represent a sort of left-and-right motion, and descending into subdirectories a downward motion. shows a typical representation of a tree as exactly that.

#### Files and directories in two dimensions

Of course, the Subversion filesystem has a nifty third dimension that most filesystems do not have—Time! We understand that this may come as a shock to sci-fi fans who have long been under the impression that Time was actually the fourth dimension, and we apologize for any emotional trauma induced by our assertion of a different theory. In the filesystem interface, nearly every function that has a `path` argument also expects a `root` argument. This svn_fs_root_t argument describes either a revision or a Subversion transaction (which is usually just a revision-to-be), and provides that third-dimensional context needed to understand the difference between `/foo/bar` in revision 32, and the same path as it exists in revision 98. shows revision history as an added dimension to the Subversion filesystem universe.

#### Versioning time—the third dimension!

As we mentioned earlier, the libsvn_fs API looks and feels like any other filesystem, except that it has this wonderful versioning capability. It was designed to be usable by any program interested in a versioning filesystem. Not coincidentally, Subversion itself is interested in that functionality. But while the filesystem API should be sufficient for basic file and directory versioning support, Subversion wants more—and that is where libsvn_repos comes in.

The Subversion repository library (libsvn_repos) is basically a wrapper library around the filesystem functionality. This library is responsible for creating the repository layout, making sure that the underlying filesystem is initialized, and so on. Libsvn_repos also implements a set of hooks—scripts that are executed by the repository code when certain actions take place. These scripts are useful for notification, authorization, or whatever purposes the repository administrator desires. This type of functionality, and other utilities provided by the repository library, are not strictly related to implementing a versioning filesystem, which is why it was placed into its own library.

Developers who wish to use the libsvn_repos API will find that it is not a complete wrapper around the filesystem interface. That is, only certain major events in the general cycle of filesystem activity are wrapped by the repository interface. Some of these include the creation and commit of Subversion transactions, and the modification of revision properties. These particular events are wrapped by the repository layer because they have hooks associated with them. In the future, other events may be wrapped by the repository API. All of the remaining filesystem interaction will continue to occur directly via the libsvn_fs API, though.

For example, here is a code segment that illustrates the use of both the repository and filesystem interfaces to create a new revision of the filesystem in which a directory is added. Note that in this example (and all others throughout this book), the `SVN_ERR` macro simply checks for a non-successful error return from the function it wraps, and returns that error if it exists.

#### Using the Repository Layer

    /* Create a new directory at the path NEW_DIRECTORY in the Subversion
       repository located at REPOS_PATH.  Perform all memory allocation in
       POOL.  This function will create a new revision for the addition of
       NEW_DIRECTORY.  */
    static svn_error_t *
    make_new_directory (const char *repos_path,
                        const char *new_directory,
                        apr_pool_t *pool)
    {
      svn_error_t *err;
      svn_repos_t *repos;
      svn_fs_t *fs;
      svn_revnum_t youngest_rev;
      svn_fs_txn_t *txn;
      svn_fs_root_t *txn_root;
      const char *conflict_str;

      /* Open the repository located at REPOS_PATH.  */
      SVN_ERR (svn_repos_open (&repos, repos_path, pool));

      /* Get a pointer to the filesystem object that is stored in
         REPOS.  */
      fs = svn_repos_fs (repos);

      /* Ask the filesystem to tell us the youngest revision that
         currently exists.  */
      SVN_ERR (svn_fs_youngest_rev (&youngest_rev, fs, pool));

      /* Begin a new transaction that is based on YOUNGEST_REV.  We are
         less likely to have our later commit rejected as conflicting if we
         always try to make our changes against a copy of the latest snapshot
         of the filesystem tree.  */
      SVN_ERR (svn_fs_begin_txn (&txn, fs, youngest_rev, pool));

      /* Now that we have started a new Subversion transaction, get a root
         object that represents that transaction.  */
      SVN_ERR (svn_fs_txn_root (&txn_root, txn, pool));

      /* Create our new directory under the transaction root, at the path
         NEW_DIRECTORY.  */
      SVN_ERR (svn_fs_make_dir (txn_root, new_directory, pool));

      /* Commit the transaction, creating a new revision of the filesystem
         which includes our added directory path.  */
      err = svn_repos_fs_commit_txn (&conflict_str, repos,
                                     &youngest_rev, txn, pool);
      if (! err)
        {
          /* No error?  Excellent!  Print a brief report of our success.  */
          printf ("Directory '%s' was successfully added as new revision "
                  "'%" SVN_REVNUM_T_FMT "'.\n", new_directory, youngest_rev);
        }
      else if (err->apr_err == SVN_ERR_FS_CONFLICT)
        {
          /* Uh-oh.  Our commit failed as the result of a conflict
             (someone else seems to have made changes to the same area
             of the filesystem that we tried to modify).  Print an error
             message.  */
          printf ("A conflict occurred at path '%s' while attempting "
                  "to add directory '%s' to the repository at '%s'.\n",
                  conflict_str, new_directory, repos_path);
        }
      else
        {
          /* Some other error has occurred.  Print an error message.  */
          printf ("An error occurred while attempting to add directory '%s' "
                  "to the repository at '%s'.\n",
                  new_directory, repos_path);
        }

      /* Return the result of the attempted commit to our caller.  */
      return err;
    }

In the previous code segment, calls were made to both the repository and filesystem interfaces. We could just as easily have committed the transaction using `svn_fs_commit_txn`. But the filesystem API knows nothing about the repository library's hook mechanism. If you want your Subversion repository to automatically perform some set of non-Subversion tasks every time you commit a transaction (like, for example, sending an email that describes all the changes made in that transaction to your developer mailing list), you need to use the libsvn_repos-wrapped version of that function—`svn_repos_fs_commit_txn`. This function will actually first run the `pre-commit` hook script if one exists, then commit the transaction, and finally will run a `post-commit` hook script. The hooks provide a special kind of reporting mechanism that does not really belong in the core filesystem library itself. (For more information regarding Subversion's repository hooks, see .)

The hook mechanism requirement is but one of the reasons for the abstraction of a separate repository library from the rest of the filesystem code. The libsvn_repos API provides several other important utilities to Subversion. These include the abilities to:

1. create, open, destroy, and perform recovery steps on a Subversion repository and the filesystem included in that repository.

2. describe the differences between two filesystem trees.

3. query for the commit log messages associated with all (or some) of the revisions in which a set of files was modified in the filesystem.

4. generate a human-readable dump of the filesystem, a complete representation of the revisions in the filesystem.

5. parse that dump format, loading the dumped revisions into a different Subversion repository.

As Subversion continues to evolve, the repository library will grow with the filesystem library to offer increased functionality and configurable option support.

### Repository Access Layer

If the Subversion Repository Layer is at the other end of the line, the Repository Access Layer is the line itself. Charged with marshalling data between the client libraries and the repository, this layer includes the libsvn_ra module loader library, the RA modules themselves (which currently includes libsvn_ra_dav, libsvn_ra_local, and libsvn_ra_svn), and any additional libraries needed by one or more of those RA modules, such as the mod_dav_svn Apache module with which libsvn_ra_dav communicates or libsvn_ra_svn's server, `svnserve`.

Since Subversion uses URLs to identify its repository resources, the protocol portion of the URL schema (usually `file:`, `http:`, `https:`, or `svn:`) is used to determine which RA module will handle the communications. Each module registers a list of the protocols it knows how to speak so that the RA loader can, at runtime, determine which module to use for the task at hand. You can determine which RA modules are available to the Subversion command-line client, and what protocols they claim to support, by running `svn --version`:

    $ svn --version
    svn, version 1.0.1 (r9023)
       compiled Mar 17 2004, 09:31:13

    Copyright (C) 2000-2004 CollabNet.
    Subversion is open source software, see http://subversion.tigris.org/
    This product includes software developed by CollabNet (http://www.Collab.Net/).

    The following repository access (RA) modules are available:

    * ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
      - handles 'http' schema
      - handles 'https' schema
    * ra_local : Module for accessing a repository on local disk.
      - handles 'file' schema
    * ra_svn : Module for accessing a repository using the svn network protocol.
      - handles 'svn' schema

#### RA-DAV (Repository Access Using HTTP/DAV)

The libsvn_ra_dav library is designed for use by clients that are being run on different machines than the servers with which they communicating, specifically servers reached using URLs that contain the `http:` or `https:` protocol portions. To understand how this module works, we should first mention a couple of other key components in this particular configuration of the Repository Access Layer—the powerful Apache HTTP Server, and the Neon HTTP/WebDAV client library.

Subversion's primary network server is the Apache HTTP Server. Apache is a time-tested, extensible open-source server process that is ready for serious use. It can sustain a high network load and runs on many platforms. The Apache server supports a number of different standard authentication protocols, and can be extended through the use of modules to support many others. It also supports optimizations like network pipelining and caching. By using Apache as a server, Subversion gets all of these features for free. And since most firewalls already allow HTTP traffic to pass through, sysadmins typically don't even have to change their firewall configurations to allow Subversion to work.

Subversion uses HTTP and WebDAV (with DeltaV) to communicate with an Apache server. You can read more about this in the WebDAV section of this chapter, but in short, WebDAV and DeltaV are extensions to the standard HTTP 1.1 protocol that enable sharing and versioning of files over the web. Apache 2.0 comes with mod_dav, an Apache module that understands the DAV extensions to HTTP. Subversion itself supplies mod_dav_svn, though, which is another Apache module that works in conjunction with (really, as a back-end to) mod_dav to provide Subversion's specific implementations of WebDAV and DeltaV.

When communicating with a repository over HTTP, the RA loader library chooses libsvn_ra_dav as the proper access module. The Subversion client makes calls into the generic RA interface, and libsvn_ra_dav maps those calls (which embody rather large-scale Subversion actions) to a set of HTTP/WebDAV requests. Using the Neon library, libsvn_ra_dav transmits those requests to the Apache server. Apache receives these requests (exactly as it does generic HTTP requests that your web browser might make), notices that the requests are directed at a URL that is configured as a DAV location (using the `Location` directive in `httpd.conf`), and hands the request off to its own mod_dav module. When properly configured, mod_dav knows to use Subversion's mod_dav_svn for any filesystem-related needs, as opposed to the generic mod_dav_fs that comes with Apache. So ultimately, the client is communicating with mod_dav_svn, which binds directly to the Subversion Repository Layer.

That was a simplified description of the actual exchanges taking place, though. For example, the Subversion repository might be protected by Apache's authorization directives. This could result in initial attempts to communicate with the repository being rejected by Apache on authorization grounds. At this point, libsvn_ra_dav gets back the notice from Apache that insufficient identification was supplied, and calls back into the Client Layer to get some updated authentication data. If the data is supplied correctly, and the user has the permissions that Apache seeks, libsvn_ra_dav's next automatic attempt at performing the original operation will be granted, and all will be well. If sufficient authentication information cannot be supplied, the request will ultimately fail, and the client will report the failure to the user.

By using Neon and Apache, Subversion gets free functionality in several other complex areas, too. For example, if Neon finds the OpenSSL libraries, it allows the Subversion client to attempt to use SSL-encrypted communications with the Apache server (whose own mod_ssl can speak the language). Also, both Neon itself and Apache's mod_deflate can understand the deflate algorithm (the same one used by the PKZIP and gzip programs), so requests can be sent in smaller, compressed chunks across the wire. Other complex features that Subversion hopes to support in the future include the ability to automatically handle server-specified redirects (for example, when a repository has been moved to a new canonical URL) and taking advantage of HTTP pipelining.

#### RA-SVN (Custom Protocol Repository Access)

In addition to the standard HTTP/WebDAV protocol, Subversion also provides an RA implementation that uses a custom protocol. The libsvn_ra_svn module implements its own network socket connectivity, and communicates with a stand-alone server—the `svnserve` program—on the machine that hosts the repository. Clients access the repository using the `svn://` schema.

This RA implementation lacks most of the advantages of Apache mentioned in the previous section; however, it may be appealing to some sysadmins nonetheless. It is dramatically easier to configure and run; setting up an `svnserve` process is nearly instantaneous. It is also much smaller (in terms of lines of code) than Apache, making it much easier to audit, for security reasons or otherwise. Furthermore, some sysadmins may already have an SSH security infrastructure in place, and want Subversion to use it. Clients using ra_svn can easily tunnel the protocol over SSH.

#### RA-Local (Direct Repository Access)

Not all communications with a Subversion repository require a powerhouse server process and a network layer. For users who simply wish to access the repositories on their local disk, they may do so using `file:` URLs and the functionality provided by libsvn_ra_local. This RA module binds directly with the repository and filesystem libraries, so no network communication is required at all.

Subversion requires that the server name included as part of the `file:` URL be either `localhost` or empty, and that there be no port specification. In other words, your URLs should look like either `file://localhost/path/to/repos` or `file:///path/to/repos`.

Also, be aware that Subversion's `file:` URLs cannot be used in a regular web browser the way typical `file:` URLs can. When you attempt to view a `file:` URL in a regular web browser, it reads and displays the contents of the file at that location by examining the filesystem directly. However, Subversion's resources exist in a virtual filesystem (see ), and your browser will not understand how to read that filesystem.

#### Your RA Library Here

For those who wish to access a Subversion repository using still another protocol, that is precisely why the Repository Access Layer is modularized! Developers can simply write a new library that implements the RA interface on one side and communicates with the repository on the other. Your new library can use existing network protocols, or you can invent your own. You could use inter-process communication (IPC) calls, or—let's get crazy, shall we?—you could even implement an email-based protocol. Subversion supplies the APIs; you supply the creativity.

### Client Layer

On the client side, the Subversion working copy is where all the action takes place. The bulk of functionality implemented by the client-side libraries exists for the sole purpose of managing working copies—directories full of files and other subdirectories which serve as a sort of local, editable reflection of one or more repository locations—and propagating changes to and from the Repository Access layer.

Subversion's working copy library, libsvn_wc, is directly responsible for managing the data in the working copies. To accomplish this, the library stores administrative information about each working copy directory within a special subdirectory. This subdirectory, named `.svn`, is present in each working copy directory and contains various other files and directories which record state and provide a private workspace for administrative action. For those familiar with CVS, this `.svn` subdirectory is similar in purpose to the `CVS` administrative directories found in CVS working copies. For more information about the `.svn` administrative area, see in this chapter.

The Subversion client library, libsvn_client, has the broadest responsibility; its job is to mingle the functionality of the working copy library with that of the Repository Access Layer, and then to provide the highest-level API to any application that wishes to perform general revision control actions. For example, the function `svn_client_checkout` takes a URL as an argument. It passes this URL to the RA layer and opens an authenticated session with a particular repository. It then asks the repository for a certain tree, and sends this tree into the working copy library, which then writes a full working copy to disk (`.svn` directories and all).

The client library is designed to be used by any application. While the Subversion source code includes a standard command-line client, it should be very easy to write any number of GUI clients on top of the client library. New GUIs (or any new client, really) for Subversion need not be clunky wrappers around the included command-line client—they have full access via the libsvn_client API to same functionality, data, and callback mechanisms that the command-line client uses.

#### Binding Directly—A Word About Correctness

Why should your GUI program bind directly with a libsvn_client instead of acting as a wrapper around a command-line program? Besides simply being more efficient, this can address potential correctness issues as well. A command-line program (like the one supplied with Subversion) that binds to the client library needs to effectively translate feedback and requested data bits from C types to some form of human-readable output. This type of translation can be lossy. That is, the program may not display all of the information harvested from the API, or may combine bits of information for compact representation.

If you wrap such a command-line program with yet another program, the second program has access only to already-interpreted (and as we mentioned, likely incomplete) information, which it must again translate into its representation format. With each layer of wrapping, the integrity of the original data is potentially tainted more and more, much like the result of making a copy of a copy (of a copy …) of a favorite audio or video cassette.

## Using the APIs

Developing applications against the Subversion library APIs is fairly straightforward. All of the public header files live in the `subversion/include` directory of the source tree. These headers are copied into your system locations when you build and install Subversion itself from source. These headers represent the entirety of the functions and types meant to be accessible by users of the Subversion libraries.

The first thing you might notice is that Subversion's datatypes and functions are namespace protected. Every public Subversion symbol name begins with `svn_`, followed by a short code for the library in which the symbol is defined (such as `wc`, `client`, `fs`, etc.), followed by a single underscore (`_`) and then the rest of the symbol name. Semi-public functions (used among source files of a given library but not by code outside that library, and found inside the library directories themselves) differ from this naming scheme in that instead of a single underscore after the library code, they use a double underscore (`__`). Functions that are private to a given source file have no special prefixing, and are declared `static`. Of course, a compiler isn't interested in these naming conventions, but they help to clarify the scope of a given function or datatype.

### The Apache Portable Runtime Library

Along with Subversion's own datatype, you will see many references to datatypes that begin with `apr_`—symbols from the Apache Portable Runtime (APR) library. APR is Apache's portability library, originally carved out of its server code as an attempt to separate the OS-specific bits from the OS-independent portions of the code. The result was a library that provides a generic API for performing operations that differ mildly—or wildly—from OS to OS. While the Apache HTTP Server was obviously the first user of the APR library, the Subversion developers immediately recognized the value of using APR as well. This means that there are practically no OS-specific code portions in Subversion itself. Also, it means that the Subversion client compiles and runs anywhere that the server does. Currently this list includes all flavors of Unix, Win32, BeOS, OS/2, and Mac OS X.

In addition to providing consistent implementations of system calls that differ across operating systems, Subversion uses ANSI system calls and datatypes as much as possible. APR gives Subversion immediate access to many custom datatypes, such as dynamic arrays and hash tables. Subversion uses these types extensively throughout the codebase. But perhaps the most pervasive APR datatype, found in nearly every Subversion API prototype, is the `apr_pool_t`—the APR memory pool. Subversion uses pools internally for all its memory allocation needs (unless an external library requires a different memory management schema for data passed through its API), Neon and Berkeley DB are examples of such libraries. and while a person coding against the Subversion APIs is not required to do the same, they are required to provide pools to the API functions that need them. This means that users of the Subversion API must also link against APR, must call `apr_initialize()` to initialize the APR subsystem, and then must acquire a pool for use with Subversion API calls. See for more information.

### URL and Path Requirements

With remote version control operation as the whole point of Subversion's existence, it makes sense that some attention has been paid to internationalization (i18n) support. After all, while remote might mean across the office, it could just as well mean across the globe. To facilitate this, all of Subversion's public interfaces that accept path arguments expect those paths to be canonicalized, and encoded in UTF-8. This means, for example, that any new client binary that drives the libsvn_client interface needs to first convert paths from the locale-specific encoding to UTF-8 before passing those paths to the Subversion libraries, and then re-convert any resultant output paths from Subversion back into the locale's encoding before using those paths for non-Subversion purposes. Fortunately, Subversion provides a suite of functions (see `subversion/include/svn_utf.h`) that can be used by any program to do these conversions.

Also, Subversion APIs require all URL parameters to be properly URI-encoded. So, instead of passing file:///home/username/My File.txt as the URL of a file named `My File.txt`, you need to pass file:///home/username/My%20File.txt. Again, Subversion supplies helper functions that your application can use—`svn_path_uri_encode` and `svn_path_uri_decode`, for URI encoding and decoding, respectively.

### Using Languages Other than C and C++

If you are interested in using the Subversion libraries in conjunction with something other than a C program—say a Python script or Java application—Subversion has some initial support for this via the Simplified Wrapper and Interface Generator (SWIG). The SWIG bindings for Subversion are located in `subversion/bindings/swig` and are slowly maturing into a usable state. These bindings allow you to call Subversion API functions indirectly, using wrappers that translate the datatypes native to your scripting language into the datatypes needed by Subversion's C libraries.

There is an obvious benefit to accessing the Subversion APIs via a language binding—simplicity. Generally speaking, languages such as Python and Perl are much more flexible and easy to use than C or C++. The sort of high-level datatypes and context-driven type checking provided by these languages are often better at handling information that comes from users. As you know, humans are proficient at botching up input to a program, and scripting languages tend to handle that misinformation more gracefully. Of course, often that flexibility comes at the cost of performance. That is why using a tightly-optimized, C-based interface and library suite, combined with a powerful, flexible binding language, is so appealing.

Let's look at an example that uses Subversion's Python SWIG bindings. Our example will do the same thing as our last example. Note the difference in size and complexity of the function this time!

#### Using the Repository Layer with Python

    from svn import fs
    import os.path

    def crawl_filesystem_dir (root, directory, pool):
      """Recursively crawl DIRECTORY under ROOT in the filesystem, and return
      a list of all the paths at or below DIRECTORY.  Use POOL for all
      allocations."""

      # Get the directory entries for DIRECTORY.
      entries = fs.dir_entries(root, directory, pool)

      # Initialize our returned list with the directory path itself.
      paths = [directory]

      # Loop over the entries
      names = entries.keys()
      for name in names:
        # Calculate the entry's full path.
        full_path = os.path.join(basepath, name)

        # If the entry is a directory, recurse.  The recursion will return
        # a list with the entry and all its children, which we will add to
        # our running list of paths.
        if fs.is_dir(fsroot, full_path, pool):
          subpaths = crawl_filesystem_dir(root, full_path, pool)
          paths.extend(subpaths)

        # Else, it is a file, so add the entry's full path to the FILES list.
        else:
          paths.append(full_path)

      return paths

An implementation in C of the previous example would stretch on quite a bit longer. The same routine in C would need to pay close attention to memory usage, and need to use custom datatypes for representing the hash of entries and the list of paths. Python has hashes (called dictionaries) and lists as built-in datatypes, and provides a wonderful selection of methods for operating on those types. And since Python uses reference counting and garbage collection, users of the language don't have to bother themselves with allocating and freeing memory.

In the previous section of this chapter, we mentioned the `libsvn_client` interface, and how it exists for the sole purpose of simplifying the process of writing a Subversion client. The following is a brief example of how that library can be accessed via the SWIG bindings. In just a few lines of Python, you can check out a fully functional Subversion working copy!

#### A Simple Script to Check Out a Working Copy.

    #!/usr/bin/env python
    import sys
    from svn import util, _util, _client

    def usage():
      print "Usage: " + sys.argv[0] + " URL PATH\n"
      sys.exit(0)

    def run(url, path):
      # Initialize APR and get a POOL.
      _util.apr_initialize()
      pool = util.svn_pool_create(None)

      # Checkout the HEAD of URL into PATH (silently)
      _client.svn_client_checkout(None, None, url, path, -1, 1, None, pool)

      # Cleanup our POOL, and shut down APR.
      util.svn_pool_destroy(pool)
      _util.apr_terminate()

    if __name__ == '__main__':
      if len(sys.argv) != 3:
        usage()
      run(sys.argv[1], sys.argv[2])

Subversion's language bindings unfortunately tend to lack the level of attention given to the core Subversion modules. However, there have been significant efforts towards creating functional bindings for Python, Perl, and Java. Once you have the SWIG interface files properly configured, generation of the specific wrappers for all the supported SWIG languages (which currently includes versions of C#, Guile, Java, MzScheme, OCaml, Perl, PHP, Python, Ruby, and Tcl) should theoretically be trivial. Still, some extra programming is required to compensate for complex APIs that SWIG needs some help generalizing. For more information on SWIG itself, see the project's website at http://www.swig.org/.

## Inside the Working Copy Administration Area

As we mentioned earlier, each directory of a Subversion working copy contains a special subdirectory called `.svn` which houses administrative data about that working copy directory. Subversion uses the information in `.svn` to keep track of things like:

- Which repository location(s) are represented by the files and subdirectories in the working copy directory.

- What revision of each of those files and directories are currently present in the working copy.

- Any user-defined properties that might be attached to those files and directories.

- Pristine (un-edited) copies of the working copy files.

While there are several other bits of data stored in the `.svn` directory, we will examine only a couple of the most important items.

### The Entries File

Perhaps the single most important file in the `.svn` directory is the `entries` file. The entries file is an XML document which contains the bulk of the administrative information about a versioned resource in a working copy directory. It is this one file which tracks the repository URLs, pristine revision, file checksums, pristine text and property timestamps, scheduling and conflict state information, last-known commit information (author, revision, timestamp), local copy history—practically everything that a Subversion client is interested in knowing about a versioned (or to-be-versioned) resource!

#### Comparing the Administrative Areas of Subversion and CVS

A glance inside the typical `.svn` directory turns up a bit more than what CVS maintains in its `CVS` administrative directories. The `entries` file contains XML which describes the current state of the working copy directory, and basically serves the purposes of CVS's `Entries`, `Root`, and `Repository` files combined.

The following is an example of an actual entries file:

#### Contents of a Typical `.svn/entries` File

    <?xml version="1.0" encoding="utf-8"?>
    <wc-entries
       xmlns="svn:">
    <entry
       committed-rev="1"
       name="svn:this_dir"
       committed-date="2002-09-24T17:12:44.064475Z"
       url="http://svn.red-bean.com/tests/.greek-repo/A/D"
       kind="dir"
       revision="1"/>
    <entry
       committed-rev="1"
       name="gamma"
       text-time="2002-09-26T21:09:02.000000Z"
       committed-date="2002-09-24T17:12:44.064475Z"
       checksum="QSE4vWd9ZM0cMvr7/+YkXQ=="
       kind="file"
       prop-time="2002-09-26T21:09:02.000000Z"/>
    <entry
       name="zeta"
       kind="file"
       schedule="add"
       revision="0"/>
    <entry
       url="http://svn.red-bean.com/tests/.greek-repo/A/B/delta"
       name="delta"
       kind="file"
       schedule="add"
       revision="0"/>
    <entry
       name="G"
       kind="dir"/>
    <entry
       name="H"
       kind="dir"
       schedule="delete"/>
    </wc-entries>

As you can see, the entries file is essentially a list of entries. Each `entry` tag represents one of three things: the working copy directory itself (called the this directory entry, and noted as having an empty value for its name attribute), a file in that working copy directory (noted by having its kind attribute set to `"file"`), or a subdirectory in that working copy (kind here is set to `"dir"`). The files and subdirectories whose entries are stored in this file are either already under version control, or (as in the case of the file named `zeta` above) are scheduled to be added to version control when the user next commits this working copy directory's changes. Each entry has a unique name, and each entry has a node kind.

Developers should be aware of some special rules that Subversion uses when reading and writing its `entries` files. While each entry has a revision and URL associated with it, note that not every `entry` tag in the sample file has explicit revision or url attributes attached to it. Subversion allows entries to not explicitly store those two attributes when their values are the same as (in the revision case) or trivially calculable from That is, the URL for the entry is the same as the concatenation of the parent directory's URL and the entry's name. (in the url case) the data stored in the this directory entry. Note also that for subdirectory entries, Subversion stores only the crucial attributes—name, kind, url, revision, and schedule. In an effort to reduce duplicated information, Subversion dictates that the method for determining the full set of information about a subdirectory is to traverse down into that subdirectory, and read the this directory entry from its own `.svn/entries` file. However, a reference to the subdirectory is kept in its parent's `entries` file, with enough information to permit basic versioning operations in the event that the subdirectory itself is actually missing from disk.

### Pristine Copies and Property Files

As mentioned before, the `.svn` directory also holds the pristine text-base versions of files. Those can be found in `.svn/text-base`. The benefits of these pristine copies are multiple—network-free checks for local modifications and difference reporting, network-free reversion of modified or missing files, smaller transmission of changes to the server—but comes at the cost of having each versioned file stored at least twice on disk. These days, this seems to be a negligible penalty for most files. However, the situation gets uglier as the size of your versioned files grows. Some attention is being given to making the presence of the text-base an option. Ironically though, it is as your versioned files' sizes get larger that the existence of the text-base becomes more crucial—who wants to transmit a huge file across a network just because they want to commit a tiny change to it?

Similar in purpose to the text-base files are the property files and their pristine prop-base copies, located in `.svn/props` and `.svn/prop-base` respectively. Since directories can have properties, too, there are also `.svn/dir-props` and `.svn/dir-prop-base` files. Each of these property files (working and base versions) uses a simple hash-on-disk file format for storing the property names and values.

## WebDAV

WebDAV (shorthand for Web-based Distributed Authoring and Versioning) is an extension of the standard HTTP protocol designed to make the web into a read/write medium, instead of the basically read-only medium that exists today. The theory is that directories and files can be shared—as both readable and writable objects—over the web. RFCs 2518 and 3253 describe the WebDAV/DeltaV extensions to HTTP, and are available (along with a lot of other useful information) at http://www.webdav.org/.

A number of operating system file browsers are already able to mount networked directories using WebDAV. On Win32, the Windows Explorer can browse what it calls WebFolders (which are just WebDAV-ready network locations) as if they were regular shared folders. Mac OS X also has this capability, as do the Nautilus and Konqueror browsers (under GNOME and KDE, respectively).

How does all of this apply to Subversion? The mod_dav_svn Apache module uses HTTP, extended by WebDAV and DeltaV, as one of its network protocols. Subversion uses mod_dav_svn to map between Subversion's versioning concepts and those of RFCs 2518 and 3253.

For a more thorough discussion of WebDAV, how it works, and how Subversion uses it, see . Among other things, that appendix discusses the degree to which Subversion adheres to the generic WebDAV specification, and how that affects interoperability with generic WebDAV clients.

## Programming with Memory Pools

Almost every developer who has used the C programming language has at some point sighed at the daunting task of managing memory usage. Allocating enough memory to use, keeping track of those allocations, freeing the memory when you no longer need it—these tasks can be quite complex. And of course, failure to do those things properly can result in a program that crashes itself, or worse, crashes the computer. Fortunately, the APR library that Subversion depends on for portability provides the apr_pool_t type, which represents a pool from which the application may allocate memory.

A memory pool is an abstract representation of a chunk of memory allocated for use by a program. Rather than requesting memory directly from the OS using the standard `malloc()` and friends, programs that link against APR can simply request that a pool of memory be created (using the `apr_pool_create()` function). APR will allocate a moderately sized chunk of memory from the OS, and that memory will be instantly available for use by the program. Any time the program needs some of the pool memory, it uses one of the APR pool API functions, like `apr_palloc()`, which returns a generic memory location from the pool. The program can keep requesting bits and pieces of memory from the pool, and APR will keep granting the requests. Pools will automatically grow in size to accommodate programs that request more memory than the original pool contained, until of course there is no more memory available on the system.

Now, if this were the end of the pool story, it would hardly have merited special attention. Fortunately, that's not the case. Pools can not only be created; they can also be cleared and destroyed, using `apr_pool_clear()` and `apr_pool_destroy()` respectively. This gives developers the flexibility to allocate several—or several thousand—things from the pool, and then clean up all of that memory with a single function call! Further, pools have hierarchy. You can make subpools of any previously created pool. When you clear a pool, all of its subpools are destroyed; if you destroy a pool, it and its subpools are destroyed.

Before we go further, developers should be aware that they probably will not find many calls to the APR pool functions we just mentioned in the Subversion source code. APR pools offer some extensibility mechanisms, like the ability to have custom user data attached to the pool, and mechanisms for registering cleanup functions that get called when the pool is destroyed. Subversion makes use of these extensions in a somewhat non-trivial way. So, Subversion supplies (and most of its code uses) the wrapper functions `svn_pool_create()`, `svn_pool_clear()`, and `svn_pool_destroy()`.

While pools are helpful for basic memory management, the pool construct really shines in looping and recursive scenarios. Since loops are often unbounded in their iterations, and recursions in their depth, memory consumption in these areas of the code can become unpredictable. Fortunately, using nested memory pools can be a great way to easily manage these potentially hairy situations. The following example demonstrates the basic use of nested pools in a situation that is fairly common—recursively crawling a directory tree, doing some task to each thing in the tree.

### Effective Pool Usage

    /* Recursively crawl over DIRECTORY, adding the paths of all its file
       children to the FILES array, and doing some task to each path
       encountered.  Use POOL for the all temporary allocations, and store
       the hash paths in the same pool as the hash itself is allocated in.  */
    static apr_status_t
    crawl_dir (apr_array_header_t *files,
               const char *directory,
               apr_pool_t *pool)
    {
      apr_pool_t *hash_pool = files->pool;  /* array pool */
      apr_pool_t *subpool = svn_pool_create (pool);  /* iteration pool */
      apr_dir_t *dir;
      apr_finfo_t finfo;
      apr_status_t apr_err;
      apr_int32_t flags = APR_FINFO_TYPE | APR_FINFO_NAME;

      apr_err = apr_dir_open (&dir, directory, pool);
      if (apr_err)
        return apr_err;

      /* Loop over the directory entries, clearing the subpool at the top of
         each iteration.  */
      for (apr_err = apr_dir_read (&finfo, flags, dir);
           apr_err == APR_SUCCESS;
           apr_err = apr_dir_read (&finfo, flags, dir))
        {
          const char *child_path;

          /* Clear the per-iteration SUBPOOL.  */
          svn_pool_clear (subpool);

          /* Skip entries for "this dir" ('.') and its parent ('..').  */
          if (finfo.filetype == APR_DIR)
            {
              if (finfo.name[0] == '.'
                  && (finfo.name[1] == '\0'
                      || (finfo.name[1] == '.' && finfo.name[2] == '\0')))
                continue;
            }

          /* Build CHILD_PATH from DIRECTORY and FINFO.name.  */
          child_path = svn_path_join (directory, finfo.name, subpool);

          /* Do some task to this encountered path. */
          do_some_task (child_path, subpool);

          /* Handle subdirectories by recursing into them, passing SUBPOOL
             as the pool for temporary allocations.  */
          if (finfo.filetype == APR_DIR)
            {
              apr_err = crawl_dir (files, child_path, subpool);
              if (apr_err)
                return apr_err;
            }

          /* Handle files by adding their paths to the FILES array.  */
          else if (finfo.filetype == APR_REG)
            {
              /* Copy the file's path into the FILES array's pool.  */
              child_path = apr_pstrdup (hash_pool, child_path);

              /* Add the path to the array.  */
              (*((const char **) apr_array_push (files))) = child_path;
            }
        }

      /* Destroy SUBPOOL.  */
      svn_pool_destroy (subpool);

      /* Check that the loop exited cleanly. */
      if (apr_err)
        return apr_err;

      /* Yes, it exited cleanly, so close the dir. */
      apr_err = apr_dir_close (dir);
      if (apr_err)
        return apr_err;

      return APR_SUCCESS;
    }

The previous example demonstrates effective pool usage in both looping and recursive situations. Each recursion begins by making a subpool of the pool passed to the function. This subpool is used for the looping region, and cleared with each iteration. The result is memory usage is roughly proportional to the depth of the recursion, not to total number of file and directories present as children of the top-level directory. When the first call to this recursive function finally finishes, there is actually very little data stored in the pool that was passed to it. Now imagine the extra complexity that would be present if this function had to `alloc()` and `free()` every single piece of data used!

Pools might not be ideal for every application, but they are extremely useful in Subversion. As a Subversion developer, you'll need to grow comfortable with pools and how to wield them correctly. Memory usage bugs and bloating can be difficult to diagnose and fix regardless of the API, but the pool construct provided by APR has proven a tremendously convenient, time-saving bit of functionality.

## Contributing to Subversion

The official source of information about the Subversion project is, of course, the project's website at http://subversion.tigris.org/. There you can find information about getting access to the source code and participating on the discussion lists. The Subversion community always welcomes new members. If you are interested in participating in this community by contributing changes to the source code, here are some hints on how to get started.

### Join the Community

The first step in community participation is to find a way to stay on top of the latest happenings. To do this most effectively, you will want to subscribe to the main developer discussion list (dev@subversion.tigris.org) and commit mail list (svn@subversion.tigris.org). By following these lists even loosely, you will have access to important design discussions, be able to see actual changes to Subversion source code as they occur, and be able to witness peer reviews of those changes and proposed changes. These email based discussion lists are the primary communication media for Subversion development. See the Mailing Lists section of the website for other Subversion-related lists you might be interested in.

But how do you know what needs to be done? It is quite common for a programmer to have the greatest intentions of helping out with the development, yet be unable to find a good starting point. After all, not many folks come to the community having already decided on a particular itch they would like to scratch. But by watching the developer discussion lists, you might see mentions of existing bugs or feature requests fly by that particularly interest you. Also, a great place to look for outstanding, unclaimed tasks is the Issue Tracking database on the Subversion website. There you will find the current list of known bugs and feature requests. If you want to start with something small, look for issues marked as bite-sized.

### Get the Source Code

To edit the code, you need to have the code. This means you need to check out a working copy from the public Subversion source repository. As straightforward as that might sound, the task can be slightly tricky. Because Subversion's source code is versioned using Subversion itself, you actually need to bootstrap by getting a working Subversion client via some other method. The most common methods include downloading the latest binary distribution (if such is available for your platform), or downloading the latest source tarball and building your own Subversion client. If you build from source, make sure to read the `INSTALL` file in the top level of the source tree for instructions.

After you have a working Subversion client, you are now poised to checkout a working copy of the Subversion source repository from http://svn.collab.net/repos/svn/trunk/: Note that the URL checked out in the example above ends not with `svn`, but with a subdirectory thereof called `trunk`. See our discussion of Subversion's branching and tagging model for the reasoning behind this.

    $ svn checkout http://svn.collab.net/repos/svn/trunk subversion
    A  HACKING
    A  INSTALL
    A  README
    A  autogen.sh
    A  build.conf
    …

The above command will checkout the bleeding-edge, latest version of the Subversion source code into a subdirectory named `subversion` in your current working directory. Obviously, you can adjust that last argument as you see fit. Regardless of what you call the new working copy directory, though, after this operation completes, you will now have the Subversion source code. Of course, you will still need to fetch a few helper libraries (apr, apr-util, etc.)—see the `INSTALL` file in the top level of the working copy for details.

### Become Familiar with Community Policies

Now that you have a working copy containing the latest Subversion source code, you will most certainly want to take a cruise through the `HACKING` file in that working copy's top-level directory. The `HACKING` file contains general instructions for contributing to Subversion, including how to properly format your source code for consistency with the rest of the codebase, how to describe your proposed changes with an effective change log message, how to test your changes, and so on. Commit privileges on the Subversion source repository are earned—a government by meritocracy. While this may superficially appear as some sort of elitism, this earn your commit privileges notion is about efficiency—whether it costs more in time and effort to review and apply someone else's changes that are likely to be safe and useful, versus the potential costs of undoing changes that are dangerous. The HACKING file is an invaluable resource when it comes to making sure that your proposed changes earn the praises they deserve without being rejected on technicalities.

### Make and Test Your Changes

With the code and community policy understanding in hand, you are ready to make your changes. It is best to try to make smaller but related sets of changes, even tackling larger tasks in stages, instead of making huge, sweeping modifications. Your proposed changes will be easier to understand (and therefore easier to review) if you disturb the fewest lines of code possible to accomplish your task properly. After making each set of proposed changes, your Subversion tree should be in a state in which the software compiles with no warnings.

Subversion has a fairly thorough You might want to grab some popcorn. Thorough, in this instance, translates to somewhere in the neighborhood of thirty minutes of non-interactive machine churn. regression test suite, and your proposed changes are expected to not cause any of those tests to fail. By running `make check` (in Unix) from the top of the source tree, you can sanity-check your changes. The fastest way to get your code contributions rejected (other than failing to supply a good log message) is to submit changes that cause failure in the test suite.

In the best-case scenario, you will have actually added appropriate tests to that test suite which verify that your proposed changes work as expected. In fact, sometimes the best contribution a person can make is solely the addition of new tests. You can write regression tests for functionality that currently works in Subversion as a way to protect against future changes that might trigger failure in those areas. Also, you can write new tests that demonstrate known failures. For this purpose, the Subversion test suite allows you to specify that a given test is expected to fail (called an `XFAIL`), and so long as Subversion fails in the way that was expected, a test result of `XFAIL` itself is considered a success. Ultimately, the better the test suite, the less time wasted on diagnosing potentially obscure regression bugs.

### Donate Your Changes

After making your modifications to the source code, compose a clear and concise log message to describe those changes and the reasons for them. Then, send an email to the developers list containing your log message and the output of `svn diff` (from the top of your Subversion working copy). If the community members consider your changes acceptable, someone who has commit privileges (permission to make new revisions in the Subversion source repository) will add your changes to the public source code tree. Recall that permission to directly commit changes to the repository is granted on merit—if you demonstrate comprehension of Subversion, programming competency, and a team spirit, you will likely be awarded that permission.