\input texinfo   @c -*-texinfo-*-

@setfilename internals.info
@setcontentsaftertitlepage
@setshortcontentsaftertitlepage
@c @setchapternewpage odd
@footnotestyle end
@set TITLE OpenCM Internals Manual
@set AUTHOR Jonathan S. Shapiro
@set MODYEAR 2000
@set OPENCMURL http://www.eros-os.org
@settitle @value{TITLE}

@ifinfo
This file documents the design and internals of the OpenCM configuration
management system.

Copyright @value{MODYEAR}, The EROS Group, LLC.

Permission is granted to redistribute this document without royalty or
fee, provided that this copyright notice remains intact.

@end ifinfo
@c
@c  This title page illustrates only one of the
@c  two methods of forming a title page.

@titlepage
@title @value{TITLE}
@c @subtitle SUBTITLE-IF-ANY
@c @subtitle SECOND-SUBTITLE
@author @value{AUTHOR}

@c  The following two commands
@c  start the copyright page.
@page
@vskip 0pt plus 1filll
Copyright @copyright{} @value{MODYEAR}, The EROS Group, LLC and Johns
Hopkins University 

Permission is granted to redistribute this document without royalty or
fee, provided that this copyright notice remains intact.
@end titlepage

@html
<center>
<h1>@value{TITLE}</h1>
<p>
<em>@value{AUTHOR}</em>
</p>
</center>
@end html

@ifhtml
@html
<table>
<tr valign=top>
<td width="10%">&nbsp;</td>
<td>
@end html
Copyright @copyright{} @value{MODYEAR}, The EROS Group, LLC.
Permission is granted to redistribute this document without royalty or
fee, provided that this copyright notice remains intact.
@html
</td>
<td width="10%">&nbsp;</td>
</tr>
</table>
@end html
@end ifhtml

@ifnottex
@node Top
@top
This document describes the internals and the "theory of operation" of
the OpenCM configuration management system. It's purpose is to provide
enough information for others to successfully modify OpenCM, and  to
provide rationale for certain aspects of the design.

This document applies to versions 0.1 and later of OpenCM (Persistent
Configuration Management System), which is most commonly installed as a
program named @command{opencm}.
@end ifnottex

@menu
* Introduction::     What this manual provides.
* Overview::         Overview of the OpenCM Design.
* The Repository::   What is in the repository.
* The Workspace::    How OpenCM manages the workspace.
* Atomic Commit::    How commits work.
* Permissions::      How access control works.
  ...
  ...
* Scripting::        The OpenCM scripting language.
* Rationale::        Why OpenCM embeds some decisions in the tool.
* Futures::          Things that need to be done.
@end menu

@node Introduction
@chapter Introduction
@cindex Introduction to OpenCM

This manual describes the internals and the theory of operation for
OpenCM. The audience is people who want to understand in detail how OpenCM
actually works, and @emph{especially} those who may decide to enhance
OpenCM in some say. The document attempts to answer both @emph{what} OpenCM
does and, where there is some reason to wonder, @emph{why} OpenCM operates
in a particular way.

@section Antecedents

Several previous systems have inspired various aspects of OpenCM. CVS has
probably had the greatest impact. Like CVS, OpenCM does all locking at
checkin time; the workspace can always be modified. Also like CVS, OpenCM
is a client/server system (though in the CVS design this was an
afterthought).  OpenCM is in some sense a direct response to three flaws
in CVS:

@itemize @bullet
@item
CVS does not really support configurations. It does versions of files,
and can tag these versions, but it doesn't really know what set of what
versions constitutes a consistent set.
@item
CVS does not really support @emph{rename}.
@item
CVS does not support nested or replicated respositories well.
@item
CVS does not support disconnected operation. I can change a file while
disconnected, but there is no way to commit intermediate states.
@item
CVS does not provide for per-branch and per-project access
controls. While the @command{cvs_acls} program tries to provide this, the
implementation is incomplete and does not restrict access to things like
the history log.
@end itemize

OpenCM borrows its merge management strategy from
@iftex
\break
@end iftex
@uref{http://xcf.berkeley.edu/~jmacd/prcs.html, PRCS}. The work that Josh Macdonald has
done on merge algorithms is the most comprehensive thinking about this I
know about. At some point, OpenCM may also adopt Josh's mechanism for
delta compression and storage, though this is less certain and is really
an implementation detail.

OpenCM borrows ideas about naming from @uref{http://www.waterken.com,
Droplets}, which names entities by using their SHA-1 hash. Indeed,
thinking about the droplets and discussions with Tyler Close about their
applications to security in open web servers was what originally led me
to think a system like OpenCM was viable.

@section Motivation

In the final analysis, the reason I started OpenCM was somewhere between
the need for rename and the need for nested repositories. I have been
bitten by CVS's lack of support for rename in the work on the
@uref{http://www.eros-os.org, EROS} operating system. Older
versions of EROS can still be checked out, but the file names are all
wrong and the resulting collection of files won't build.  Also, I tend
to do a lot of work on my portable on an airplane somewhere. I really
wanted a system that would let me do local checkins where the whole
chain could be merged back into the repository later. Finally, I was
interested in the situation where you might want a repository that could
be shared with a co-developer without fully publishing the repository.

So: OpenCM.

@node Overview
@chapter Overview
@cindex Overview of OpenCM

OpenCM is a client/server system. The client manages the workspace, and
manages the relationship between the state of the workspace and the
state that is currently checked in (if any). The server manages
long-term storage and reproducible configurations.

More precisely, OpenCM is an @emph{application} that is constructed on
top of a cryptographically secured archival store. There is a single
logical object space that is distributed across multiple servers. This
object space can be cached, allowing disconnected operation. As a
result of the object space design, OpenCM servers can be arranged in
either hierarchical or "peer to peer" relationships. In the
hierarchical arrangement, one server is conceptually "definitive" and
the second acts as a (lazy) cache of this definitive server. In the
peer relationship, two servers mirror each other. This mirroring can
be done selectively.

Behind the scenes, all of these relationships are actually peer to
peer. When you check in a change to a branch on your notebook, the
server on the notebook is modifying a branch on the notebook. This branch
will later be replicated to the "upstream" server. Once the replication
is successfully completed, control of the branch
can be transferred to the upstream server.

Every branch 
has a unique server that ``owns'' that branch
This is the only server where a revision to that particular
branch may be made. Similarly, every project has an owning server where
revisions to the project as a whole can be made. If you can read some
existing branch of a project, you can create a private, modifiable
branch derived from it.

@section Entities and Sequences

@emph{This description has been supersceded by subsequent changes in
the design. A current description can be found in
@uref{http://www.opencm.org/papers/opencm2003.pdf, OpenCM One Year Later}}.

There are two kinds of objects in the OpenCM repository: entities and
sequences.@footnote{The name ``sequence'' is unfortunate and is going
away, as access control lists are modifiable and are not sequences. A
better term would be ``mutables,'' but for some reason people seem to
find this term confusing.}
Entities are uninterpreted byte vectors. A given version of
a file is an entity, as is a given configuration. While the server is
free to store things cleverly, entities in the repository are never
modified.
Conceptually,
checking in a new version of a file creates a new entity. The fact that
delta compression is possible and that the server uses it has no bearing
on the @emph{semantics}, which is that the two entities are
independent. Entities have no access rights beyond knowing their
name. If you can name an entity, and you can connect to a server that
has that entity, you can get it from the server. Note,
however, that entity names are generated by constructing a cryptographic
hash. Someone can @emph{tell} you the name of an entity, but you won't
guess one successfully in your lifetime (sorry).

Sequences are modifiable, such as branches and projects.@footnote{We may
someday have modifiable objects that @emph{aren't} sequences, such as
groups, but for the moment it turns out that all modifiable objects are
sequences.} A branch can be modified by appending a new configuration (a
new version of the branch) or by changing its access rights. A project
can be modified by changing its description or access rights.  Where
entities are named by their SHA-1 cryptographic hash, sequences are
named by random numbers.

Modifiable objects always have an @dfn{owner}, which is the server that
controls modifications to that object. The owner is the only server at
which the modifiable object can be changed, and it is considered the
definitive source for that object. For example, a branch can only be
modified on the server that manages that branch. If two servers have the
same version of a modifiable object, and one owns the object, it can
transfer ownership to the second.

@section Clients and Servers

The client manages the user's working files, which are known as the
@dfn{workspace}. Files in the workspace are always modifiable, and no
attempt is made to do any locking for files in the workspace. The job of
the OpenCM client is to populate the workspace and to determine which
files in the workspace are modified.

Conceptually, a configuration management system manages collections of
objects. As far as the server is 
concerned, the objects themselves consist of uninterpreted bytes. The
server has no knowledge of the semantics of the objects, and never
will. It merely keeps track of which sets of entities constitute
a coordinated set. A @dfn{configuration} consists of a coordinated set
of objects plus their metadata.@footnote{In some cases, the metadata is
actually stored with the entity.}  A OpenCM @dfn{branch} is merely a
sequence of configurations. Finally, the server manages a notion of a
@dfn{project}, which is merely a collection of (somehow related)
branches.

By metadata, I mean facts @emph{about} the objects.  While the server
stores the metadata for entities, it does not @emph{interpret} that
metadata. All interpretation of the metadata is performed by the
client. Also, all knowledge of object semantics (e.g. file semantics)
resides in the client. In part, this is because the needs of the client
depend on the activity being managed. For example, the OpenCM repository
could be used to manage functions as easily as files. No current client
(user agent) does this.

The server does know about the semantics of projects and sequences. It
must, in order to implement the atomicity of checkins.

At some point, it will be useful to introduce a "middle" layer on the
server side. For example, it is easier to generate user-comprehensible
deltas if you have some knowledge of the content. All of this should be
script-driven, and the @emph{semantics} of the server should never rely
on such knowledge.

@section Cryptographic Names and Pet Names

OpenCM uses cryptographic hashes or random numbers for all object names in
the repository. These are referred to as the @dfn{true names} of the
objects, as distinct from the "pet names" (see below).

This section describes pros and cons of such a design.

@subsection Advantages
Using cryptographic hashes and random numbers has a
number of useful properties:

@itemize @bullet
@item
Because file names are essentially random, they are (pragmatically)
unguessable. This provides a degree of security. You can only guess the
name of an entity if you already konw what it contains, and there isn't
an easy way to guess the name of a sequence. The main risk here is that
somebody will @emph{tell} you the name of an object, but if they are in
a position to do that they are also in a position to send you the
object.

That is, cryptographic names provide exactly the degree of security that
is actually enforceable. The sources of failure in security have to do
with the (mis)behavior of the users, and these cannot realistically be
controlled by the computer.

@item
There is essentially no likelihood of name collision. This means that
servers can cross-replicate easily. Contrast this with file names: how
many programs have you seen that included a file named @code{main.c} or
@code{globals.h}?

@item
There is a level of indirection between the file name and the entity
name. This is essential to do a correct implementation of rename. Most
other systems do this by introducing a mapping table for the renamed
objects. There is nothing wrong with this, but a mapping from file names
to file names doesn't gain the other advantages of universally unique
names.

@item
Not everything managed by a CM system is a file! Indeed, the OpenCM server
doesn't do anything with the "workspace name" of an object -- it is
merely a string to be stored. This means that the workspace name can
equally well be a programming language identifier, a class name, a URL,
etc. etc.

@item
Using a namespace on the server that doesn't depend on file names lets
us avoid problems with broken file systems, such as MacOS and Windows,
which are caseless or impose length limits.

@item
Use of universally unique names directly supports atomic commits.
@end itemize

One consequence is that unless you have your hands on a branch file (or
the name of a branch file), you won't be able to download the
configurations or the entities that make up that branch.

A second useful consequence is that replication can be done using
@code{rsync} or a similar program, because there are no
collisions. Selective replication requires that the file name generation
algorithm be modified, but does not require modification to the file
exchange logic.

@subsection Disadvantages

The main disadvantage to cryptographic names is that you cannot look at
the repository content and immediately understand what's where. In
particular, if you manage to lose the branch file you have a definite
problem reconstructing which configurations are which and what names go
with the entities.

My personal take is that these disadvantages are not compelling. SHA-1
hashes are self-checking: by recomputing the hash you can determine if
the entity you have is a valid copy. Random numbers are not self
checking, but most of the objects that are named using random numbers
can be reconstructed from other information in the repository. This
makes them relatively easy to check.

Some people have argued that the use of binary formats is risky, but I'm
not compelled by this argument. In practice, you don't lose the middle
of a file very often, and the binary format is easier to parse. I
concluded that parser mistakes were more likely than file system
mistakes. This is especially true given that files are never updated in
place by the OpenCM server.

@subsection Pet Names
@cindex Pet Names
The main problem with cryptographic names is that human beings cannot
remember them. (Exercise for the reader: try memorizing your RSA public
key.) The OpenCM solution for this is due to Mark Miller, and was
originally used at @uref{http://www.communities.com, Electronic
Communities Inc}. It is known as "pet names". A similar idea is being
used in SPKI.  Here is the idea:

Every project has an "official" human-readable name. This is called it's
@dfn{nickname}. For example, the nickname for the OpenCM project is
@kbd{opecm}. In OpenCM, both projects and branches have nicknames. The
nickname provides a way for human beings to refer to the project in a
comprehensible way.  The problem is that these names can
collide. Somebody else could start a project and give it the nickname
@kbd{opencm}. The solution to this is "pet names."

When a server first gets a copy of a project or a branch, it examines
that object to learn its nickname. Let's imagine that the nickname for
the new project is @kbd{myproject}. A pet name is constructed as follows:
follows:
@itemize @bullet
@item
If there is no current project (branch) whose pet name is
@code{myproject}, then @code{myproject} is used as the pet name for the
new project.
@item
If the pet name @code{myproject} is taken, then @code{myproject.1},
@code{myproject.2}, ... are tried in turn until an available nickname is
found.
@end itemize

If you have appropriate permissions on the server, you can change the
pet name of a project. The nickname of the project cannot be altered.

This provides a name that is fairly easy for a person to remember. On
the other hand, it ensures that name collisions on @emph{your} server
don't mess up @emph{my} server. Finally, it lets me decide what name I
wish to use for each project and branch I work with.

The pet name also subsumes certain cases of ``tagging.'' To remember a
particular version purely for your own reference, you can simply create
a new pet name for that version.
@node The Repository
@chapter The Repository
@cindex Repository Structure

The OpenCM repository is independent of the client. The client speaks to a
generic repository interface, which may be backed by a file repository,
a remote repository, a compressed repository, a DAV repository (maybe),
or some other implementation. The defining characteristics of a OpenCM
repository are (1) it can respond appropriately to the client/server
boundary API, and (2) it respects the naming model inherent in that API.

At present, there is no distinguished server to server API. There are
reasons to want such an API in order to facilitate compression and delta
storage, but the current architecture does not define one. Fortunately,
most of the information that is useful for delta management is a side
effect of information that is already stored in the branch objects.

@menu
* The File Repository:: A simple testing repository.
@end menu

@node The File Repository
@section The File Repository
The only repository type currently implemented is the @dfn{File
Repository}. This repository is implemented as a tree of files with no
compression or delta management, and exists primarily for the purpose of
testing the designed API.

A file repository is rooted at some top-level directory. The contents of
this directory are:

@example
@group
<ROOT>
  OpenCM-VERSION  @r{The version of the OpenCM server.}
  entities/     @r{Directory containing all entities.}
  projects/     @r{Directory of projects.}
  petnames/     @r{Directory of project pet names.}
@end group
@end example

In future implementations, it may be that each project should have a
distinct entities subdirectory to reduce the likelihood of collisions.

At present, each file in the petnames directory works as follows: the
file name is the pet name of the project/branch, and the content of the
file is the true name of that project/branch. This is a bug, as it
causes pet names to be restricted by the file system naming convention,
which is a problem. It should be replaced by a format in which each file
in the petnames/ directory contains a (pet name, true name)
pair.

A better implementation of pet names might use a database of some
form. The primary design goal of the ``file repository'' is to provide
a simple, quick implementation that will let OpenCM be used quickly.

Project directories are in turn subdivided:

@example
@group
<ROOT>/projects/XXXX
  description   @r{Description of the project (a sequence).}
  branches/     @r{Directory of branches.}
  petnames/     @r{Directory of branch pet names.}
@end group
@end example


@node The Workspace
@chapter The Workspace
@cindex The Workspace

The workspace has a similar conceptual structure to the repository. On
the client side, there is some directory that is the ``top'' of the work
area. All of the files that were checked out can be found somewhere
underneath this directory. For OpenCM itself, the file tree structure
looks like:
@example
@group
<TOP>/README
<TOP>/doc
<TOP>/doc/internals.texi
<TOP>/src
<TOP>/src/Entity.h
...
@end group
@end example

Also in the top directory is a single file named
@file{.opencm}. This file records what was checked out, and is used
to determine which files have been changed by the user.

@section Content of the @file{.opencm} File

When it is finally checked in, the state of the workspace will become a
@code{Change} object, and this new @code{Change} object will be appended
to some @code{Branch} object. A @code{Change} object consists of an
author, a commit time, a description, and a set of @code{Entity}
objects. Each @code{Entity} object contains some amount of metadata, the
@emph{predecessors} (a set of entities) of this entity, the
@emph{type} of the entity (directory, ASCII, binary, etc.), the @emph{name}
of the entity (e.g. @code{src/Entity.h}) and the @emph{content} of the
object. The content is stored in an @code{EntityBits} object.

The @file{.opencm} file contains an analogous set of objects. The file
itself is merely a serialization of a @code{PendingChange} object. Just
as a @code{Change} object consists of a set of @code{Entity} objects, a
@code{PendingChange} object consists of a set of @code{WsEntity} objects
(for ``Workspace Entities''). The @code{PendingChange} object also
records the project, branch, and version from which this workspace was
checked out.

@section Content of @code{WsEntity} Objects

The interesting part in all of this is the @code{WsEntity} object,
because this is the object that the merge logic is built around. Keep in
mind that there are three categories of changes that can occur to an
entity in the workspace. These are:

@itemize @bullet
@item 
A file can be edited (by the user) in the workspace. This is detected by
building an @code{EntityBits} object from the file and comparing the
truename of the resulting object to the truename in the associated
@code{Entity} object.

@item 
An entity can be added or removed. Logically, this is really a change to
the containing @code{PendingChange} object. However, we wish to support
a limited form of ``undo'' capability for these operations. Because we
wish to undo, a ``deleted'' entity is really only @emph{provisionally}
deleted until the workspace changes are committed.

@item 
A merge can be performed. This may or may not result in an actual
edit to the file. For example, the changes that would be applied by the
merge may already be present in the
file. A merge @emph{always} results in a revision to the predecessor
list for the Entity associated with this file.
@end itemize

The @code{WsEntity} structure contains five fields:

@itemize
@item
The @code{fsName} field identifies the name in the workspace of the
file. Unless the file is newly added, this is simply a copy of the name
in recorded in the @code{baseEntity} field.

@item
The @code{type} field identifies the type (ascii, symlink, directory,
binary) of the file. Unless the file is newly added, this is simply a
copy of the type in recorded in the @code{baseEntity} field.

@item
The @code{baseEntity} field contains a copy of the entity from which the
current state of the workspace is derived. That is, the state in the
workspace can always be thought of as @code{baseEntity} plus a possible
merge plus some user-supplied edits.

@item
The @code{mergeEntity} field indicates the identity of the entity (if
any) that has been merged into the current workspace.

@item
The @code{flags} field indicates whether the file has been added or
removed in the current editing session.

@c  @item
@c  The @code{pickledBits} field is usually empty. If a file has been
@c  removed, however, the @code{pickledBits} field holds the content of the
@c  file at the time of removal. This allows @command{opencm} to restore the
@c  contents of the file if the remove is ``undone'' before the current
@c  state of the workspace is committed.
@end itemize

@noindent
Detecting that the content of a file has changed is straightforward
(simply compare the current state in the workspace to the recorded state
in the @code{baseEntity}.  To understand how the other types of
modifications are handled, the next sections describe these actions in
greater detail.

@section Adding and Removing Entities

A newly added file causes a new @code{WsEntity} object to be created
that (conceptually) has no @code{baseEntity}. If a file is added
multiple times without an intervening @emph{commit} or @emph{remove},
the second and subsequent additions may cause the ``type'' field to be
revised.

In abstract, it is possible to normalize file addition by treating new
files as derived from the universal ``empty'' entity. Unfortunately,
this would have the effect of creating a common ancestor among unrelated
files, which would mess up the merge logic.  What we do instead is
assign no @code{baseEntity} to these objects and handle them specially.

A removed file causes the @code{WsEntity} flags field to be marked as
``removed,'' but the associated @code{WsEntity} object is not
erased. The content of the removed file remains unchanged in the
workspace. This is done so that the @emph{remove} can be undone by a
subsequent @emph{add}. If a file is removed multiple times without an
intervening @emph{commit} or @emph{add}, the second and subsequent
removals have no effect.

Adds and removes cancel each other. If a file is removed and then added
again, its corresponding @code{WsEntity} object will appear as though
the operations never happened.

When the time comes to commit the current work in progress, OpenCM will
omit any files that have been deleted from the committed change. The
state diagram looks like:  

@example
@group
      (add)         (remove)            (commit)
no   -------> new  ---------> deleted  ----------> no
file          file <--------- new  file            file
                      (add)

          (remove)            (commit)
existing ---------> deleted  ---------> existing
file     <--------  file                file
           (add)      |
                      |(commit)
                      v
                    no file
@end group
@end example

To summarize the behavior at commit time, here are the rules:

@enumerate
@item
If an file is flagged as ``deleted,'' no @code{Entity} for this file
will be included in the configuration created by the commit.
@item
If a file is flagged as ``new,'' a @code{Entity} object for this file
will be included in the configuration created by the commit. This
@code{Entity} will have no predecessors.
@item
If the content of a file has changed since it was checked out (i.e. if
the SHA-1 hash of the current file content and the truename of the
original @code{EntityBits} object do not match), a new @code{Entity}
object will be included in the configuration created by the commit. This
@code{Entity} object will have @code{baseEntity} as a predecessor.
@item
If a merge has been performed, a new @code{Entity}
object will be included in the configuration created by the commit. This
@code{Entity} object will have both @code{baseEntity} and the merged
@code{Entity} as its predecessors. Note that this new @code{Entity} will
be created regardless of whether the content of the file in the
workspace has changed. This is because we must record the fact that a
merge has been performed.
@end enumerate

@noindent
At present, @command{opencm} permits multiple @emph{update} operations to
be performed, but will not permit multiple @emph{merge} operations to be
performed between commits. This is a bug, and should be fixed in a later
release. The delay is to ensure that the implementation captures
everything that is needed.

@section Merge Handling

Merge handling is easily the most complex bit of code in
@command{opencm}. At the moment, merge handling is incomplete. The common
cases are handled adequately, but I am still thinking through some of
the more obscure cases.

While the implementations are very similar, @command{opencm} divides
merges into two types:

@enumerate A
@item
@b{General Merge}, in which the merged version may be an arbitrary
version, so long as that version is @emph{not} an ancestor of the
current state.

@item
@b{Update}, where the merged version is a successor to the current
@code{baseEntity}. Unless a general merge has occurred this is
straightforward. Once a general merge has occurred this becomes quite
tricky.
@end enumerate

These cases are different because the management of the predecessor
logic differs.

A ``merge'' is the joining of two versions in two branches.

@subsection Update

Every file in the work area has some @code{baseEntity} from which it was
derived. This @code{baseEntity} object is associated with some particular
project, branch, and version. The file in the workspace can be thought
of as the content of the entity plus some (possibly empty) delta. The
delta is the edits (if any) that have been performed in the workspace.

The @emph{update} operation deals with the case where the branch has
been updated while the user has performed edits:

@example
@group
     ----> workspace A --
    /                    \ (commit)
   /                      \
mybranch:5  ............> mybranch:6 ...........> mybranch:7
   \                       \                  /
    \                       \ (update)       /(commit)
     ----> workspace B ------+-----> updated workspace B
@end group
@end example

@noindent Conceptually, what happens in the update is that the deltas
(the user changes) are saved aside, the workspace is checked out from
the more recent version in the branch, and the the deltas are
reapplied. In practice a three way merge is done to implement this.

Imagine that a @code{file} has been modified in both workspace A and
workspace B. When workspace A is committed, a new entity
@code{mybranch:6:thefile} is created with @code{mybranch:5:thefile} as
its predecessor. This is done because workspace A is derived from
@code{mybranch:5}.

When workspace B commits, there are two stories we could tell:
@enumerate
@item
The new version, @code{mybranch:7:file} is derived from both 
@code{mybranch:5:thefile} and @code{mybranch:6:thefile}, and should have
both entities listed as its predecessors.

@item
The new version, @code{mybranch:7:file} is derived only from 
@code{mybranch:6:thefile}, and should @emph{not} have
@code{mybranch:5:thefile} as a predecessor.
@end enumerate

Because @code{mybranch:6:file} is a successor to @code{mybranch:5:file},
we drop @code{mybranch:5:file}. The general rule is that for any two
predecessors (a,b), if @b{b} is a successor to @b{a} and both @b{a} and
@b{b} are on the same branch, we drop @b{a} from the predecessor list.

@subsection General Merge

The general merge case is different from the update case because the
version we are merging in is not a successor to the
@code{baseEntity}. To capture the difference with yet more ASCII art:

@example
@group
       --> workspace:0 ---+-> workspace:1
      /                   /
     /                   /  [UPDATE]
mybranch:1  ----> mybranch:2


       --> branch:76
      /       \
     /         \
branch:75       ----> workspace:0----+-->workspace:1
     \                              /
      \                            / [MERGE]
       -------------------> mybranch:1
@end group
@end example

@noindent Note that @code{workspace:0} is derived from
@code{branch:76}. While @code{branch:76} and @code{mybranch:1} share a
common ancestor, neither can be said to be a successor to the
other. That is what makes this a general merge as opposed to an update.

If, following the merge of @code{mybranch:1} into the workspace, the
workspace is subsequently committed, it will create @code{branch:77}. at
that point, our evolution graph will look like:

@example
@group
       --> branch:76
      /          \
     /            \
branch:75          +---> branch:77
     \            /
      \          /
       --> mybranch:1
@end group
@end example

When @code{branch:77} is created by the workspace commit, entities in
@code{branch:77} will list both @code{branch:76} and @code{mybranch:1}
as predecessors.

@emph{The current implementation does not permit more than one @emph{merge}
operation to be performed between commits.}

@noindent While @code{branch:77} is a successor to @code{mybranch:1}, an 
@emph{update} performed on the workspace will @emph{not} incorporate the
state of @code{branch:77} into the workspace because it is on a
different branch.

@subsection Update, Take Two

There is one special circumstance in which @command{opencm} will allow a
repeated merge, which is when the second merge is a successor of the
first. When the resulting workspace is finally committed, knowlege of
the intervening merges is not recorded.

@example
@group
       --> branch:76 ---> branch:77
      /          \           \
     /            \           \
branch:75 --> workspace ---> workspace ...
     \              /           /
      \            /           /
       --> mybranch:1 ----> mybranch:2
@end group
@end example

@subsection Why the Complications?

It would be very nice to support merges from multiple branches into a
single workspace. There are @emph{lots} of reasons to want to support
this, and it's a very reasonable thing to do.

The problem is that we need to ensure that the nearest common ancestor of two
entities is always unique. So long as each entity has at most two
ancestors this is ensured. Once we allow three ancestors (e.g. an
update, a merge, and another merge), it becomes possible that there will
be two equally good choices for nearest common ancestor in a
merge.

In practice, these ancestors may not be equally good choices for
performing a merge. While we could certainly choose the one that results
in a merge triple with the shortest edit distance, this is expensive to
compute.

The alternative design would be to preserve the ``two ancestors'' rule
by creating intermediate @code{Entity} objects at each merge. These
intermediate entities would never be directly referenced by any
@code{Change} object, but would be part of the ancestry graph for the
individual entities. The main problem with it is that now we actually
have to record the file contents associated with these interim
states. We are in fact doing all the work of a @emph{commit} operation;
we're just not concatenating the result onto any @code{Branch}. We need
to do this because these intervening states may turn out to be the
best nearest ancestor for some future merge, and if we don't record the
file contents we will be unable to perform the merge.

Conceptually, I feel that creating these intervening @code{Change}
objects is the proper way to handle multiple merges. The resulting
@code{Change} objects can be stored in the @file{.opencm} file, and can be
uploaded to the repository as part of the commit processing when the
workspace is finally committed. At the user's level, these intervening
@code{Change} records will never really be seen.

Recording these intervening @code{Change} objects, however, introduces a
further complication, which is that we must now deal with the
possibility that an @code{Entity} might exist only within the
@file{.opencm} file because it has not yet been uploaded to any
repository. This changes the search strategy for locating entities,
which isn't inherently bad but adds a complication to the current
logic. I'ld like to get the current logic working before undertaking any
new complications.

With all that said, here are two further things to say:
@itemize @bullet
@item
Introduction of this change later will not break existing clients or
their associated merge logic.
@item
The resulting data structure does not in any way violate the current
repository schema.
@end itemize

@noindent
Given this, I'm simply planning to defer the implementation of multiple
merge.

@node Atomic Commit
@chapter Atomic Commit
@cindex Atomic Commit
@cindex Update

Because OpenCM uses universally unique names, atomic updates are greatly
simplified. The following description of how "opencm commit" operates
illustrates how universally unique names are leveraged.

When a commit occurs, a check is first performed to ensure that the
workspace is current. A description of how this is done is deferred to a
later section. For the moment, assume that the workspace is up to date.

If the workspace is up to date, then the following steps are performed:
@enumerate
@item
New files must be uploaded to the server.
@item
A new configuration file (a list of the files that are part of the
configuration) must be uploaded to the server.
@item 
The new configuration is appended to a local copy of the current branch.
@item
The revised branch file is uploaded to a temporary file on the server.
@item
After a consistency check, 
the temporary file is renamed atomically to replace the existing branch
file.
@end enumerate

Because entities have universally unique names it is okay for this
sequence of operations to fail at any point. Here are the possible
consequences of failure:

@itemize @bullet
@item
Some (possibly all) of the modifications involved will be uploaded, but
will not be referenced by any branch. This simply reduces the amount of
work that must be done for the next upload.
@item
An upload of a single entity may die before completing. Either the
resulting file will not pass an SHA-1 check (and can therefore to be
deleted by the server), or it is known to be a temporary file (and can
therefore be deleted by the server). Note that there is never more than
one temporaty file during the upload.
@item
The server might crash before the rename, in which case the temporary
file will be deleted on restart and the operation will not complete.
@end itemize

The update is committed when the branch file has been successfully
renamed by the owning server.


@node Permissions
@chapter Permissions
@cindex Permissions
@cindex Access Control

This is a big gaping problem that I need to work on. At the moment, the
answer is: either you can connect to the server or you can't. The
following description contains the @emph{intended} design.

First, we are only concerned about protection of sequences. Entities are
not protected at all. If a user is entitled to examine (modify) the
state of a configuration, they should be able to examine (modify)
@emph{all} of that state. The intuition here is that the logical unit
being protected is a branch (or project), not a file within the branch.

The effect is that there are two ``layers'' of protection in the
system. Assuming you can talk to a server at all, you can ask for a list
of projects and then for branches within those projects (more on this in
a moment). Since the entity names are cryptographically sparse, there is
no practical way to guess an entity name -- you have to already know
it. This means that @emph{reading} of entities is controlled by
cryptographic sparsity, and @emph{writing} of entities is controlled by
the restriction that objects are immutable. The worst you can do with
entities is to upload a bunch of garbage to a server. However, this
garbage won't be referenced by a branch or project, and will be
garbage collected once your session terminates.

Branches and projects are protected by access control lists. In order to
read or write a branch (project), the user must appear on the access
control list for that branch or project. OpenCM maintains separate access
control lists for read access, for write access, and for control
access. Read and write access do what you expect. Control access
determines who can revise the access control lists.

In the original design, the idea was to use public/private key pairs as
the access control mechanism. Since the RSA patent expires in October
2000 and is widely known and tested, this seems like a fine
mechanism. There are two disadvantages to this approach:

@itemize @bullet
@item
As far as I know web browsers do not currently do key management in a
suitable way. If there is an appropriate means to integrate, say, X.509
certificates into the web browser, then the public key approach provides
a sort of ``universal solvent'' for access control.
@item
Some sites already have authentication mechanisms (e.g. UNIX passwords)
in place and are fond of these existing mechanisms. In particular they
wish to centrally revoke such authorities. I have several rough ideas on
how to manage this, but none that are fully developed at this time.
@end itemize

The compelling advantage to public/private key pairs is that the
validation can be performed by @emph{any} server. If the access list
says that ``anything signed by certificate A'' can write the object, and
it knows the @emph{public} half of the certificate, then it can validate
the access without access to any central source of information -- the
operation is either signed or it isn't.

So now the problem is to ensure that all the servers obey the protocol
about updates. The notional design is that we must trust the owning
server with respect to the objects that it owns. This server is the only
legitemate source of modifications to the mutable sequences that it
owns. If we imagine that each revision is signed by the owning server,
then the authenticity of revisions can be verified.

@node Scripting
@chapter Scripting
 
@emph{This is not implemented in the current OpenCM system. It is
anticipated for the second release.}

Projects are not ``policy free.'' At one extreme, they may set standards
about testing or about formatting and indentation. At the other they may
wish to cause metadata-related ``variables'' to be expanded in files as
they are checked out.

OpenCM provides support for this by means of an embedded scripting
language, which is a variant of Scheme.  Building the scripting
language into the OpenCM client ensures that there is a compatible
scripting language across all platforms. Because Scheme is a real
language, it is possible to implement fairly sophisticated policies
within the scripting language.

Scheme procedures are called when certain triggering events occur. For
example, there are pre- and post-commit triggers for entities. Neither
the triggers nor the interface for these events and calls is presently
defined. This is an open project that could usefully be taken up by
someone.

The script code itself comes from an object in the repository rather
than from the local machine.  In addition to extracting the objects that
make up the application, the OpenCM client extracts the script code from
the repository and
runs it. There is a versioning and consistency issue hiding here that
needs to be thought through:

@itemize @bullet
@item
Policy may mandate that
when the script is changed all subsequent versions follow the new
script.

@item
Reproducibility mandates that when you check out an old version it must
run the script that was active at the time it was created. Otherwise you
won't get the same output that would have been obtained at that time.
@end itemize

This set of competing requirements is why the scripts themselves must be
managed as part of the project.

Event handlers may be set based on both the trigger and the ``type'' of
the object. For example, variables may be expanded in text objects but not in
binary objects.


@node Rationale
@chapter Rationale
@cindex Rationale Notes

The following sections describe the design rationale for various
design decisions made in OpenCM.

@menu
* On Embedded Policies:: Why OpenCM is not policy-neutral.
* Scripting Language:: The choice of Scheme.
@end menu


@node On Embedded Policies
@section On Embedded Policies
 
The most argued aspects of the OpenCM design will undoubtedly be those
where I have embedded some policy in the design of the tool. In a few
places, I have made design decisions based on judgements about how the
tool will be used in the field. Some of these judgements will be wrong
and will need to be revised before release 1.0, but it is necessary that
they be made.

Configuration management systems are social support systems, and social
support systems are not policy-neutral. Configuration management systems
are one of the bedrock systems on which everything else in the software
universe is built. They are often used by people who don't understand
software process management very well. Many of these people believe that
they understand process management perfectly, and only learn otherwise
when they are well into a project. As painful as it may be to use a tool
that embeds the wrong policies, it is even more painful to recover from
the absence of policy when you are a year and a half into a project.

It is not impossible to change the policies that are built into
OpenCM. Doing so is challenging. @emph{This is intentional.} It
is easier to change the policies that are implemented in Scheme, which
is the OpenCM scripting language. The design attempts to strike a balance
in which the configuration management system can be tuned to the process
needs of a particular project, but it is not @emph{so} easy to tune that
this is undertaken lightly.

To the extent that I could see how to do so, I built policy using the
scripting language rather than building it into the code of OpenCM. The
necessary exceptions to this include the basics of entities and
sequences, handling of logging (though this probably could be relegated
to scripting), and the choice of a scripting language.

At some point you will decide that some policy built into OpenCM is
wrong. People are already saying this about the choice of Scheme as a
scripting language. To those of you who want to change the policies, I
have several words of advice.

@enumerate
@item
@strong{Don't.}
Even if your new policy is better, OpenCM must forever support those
policies that exist today in a backwards compatible way. Is your revised
policy so compelling that it is worth the cost of maintaining two
policies indefinitely?

@item
@strong{Don't.}
Policies are really really easy to get wrong. Every built-in policy
will screw us up at some point.

@item
@strong{Don't.}
You really are as clever as you think you are, but the users out there
probably aren't. However beautiful and wonderful your policy may be,
people will not use it correctly. Policies need to work in the field in
the hands of your average bear.

@item
@strong{Don't.}
Good policies are usually worse than no policies.

@item
@strong{Go Public.}
If all of the above hasn't convinced you, at least raise the proposal as
a discussion on the opencm-dev mailing list @email{opencm-dev@@eros-os.org}.
That way, lots of people can help figure out if the change is
worthwhile.
@end enumerate

One of the reasons that this manual includes rationale is to
@emph{discourage} certain kinds of strongly advocated
changes that I think are unwise. Some of these are discussed in the
sections that follow.

@node Scripting Language
@section Scripting Language

One of the built-in design decisions is the choice of scripting
language: Scheme. Many people argued for personal favorites including
Perl, Java, and Python. Somebody will undoubtedly integrate one or more
of these languages into OpenCM at some point. I chose Scheme for several
reasons:

@itemize @bullet
@item
The OpenCM system wants a scripting language that executes the same way on
clients that use different operating systems.
@item
Scripts will run on client machines, much as Javascript runs in
browsers. The scripting language @emph{must} be secure.
@item
To understand the security of a language, it's semantics needs to be
clearly expressed in a formal way.
Scheme and ML are the only readily available languages whose
semantics are expressed appropriately.
@item
Scheme is easily learned, and is already widely known.  Most
undergraduate CS programs use it as a teaching language.
@item
Scheme was easier to integrate than Python, which was my next choice.
@item
The language is @emph{tiny}, which makes it easy to implement, but it is
also extremely powerful.
@item
The language is perceived by users as a real programming language, which
means that they are less likely to do quick and dirty hacking in it.
@item
I'm not happy with the debugging support for some of the other choices.
@end itemize

In choosing Scheme, I don't mean to reject Python, Perl, Java or
anything else that has been suggested to date.@footnote{Actually, I
@emph{do} reject Visual Basic Script, but that's purely personal.}
Rather, I'm making a design choice that meets the requirements. I
@emph{do} argue that more is not better: there should be a single
scripting language for OpenCM.

Some have argued that Perl is more widely known, or that Python is a
better choice -- particularly given that I've had to extend the standard
Scheme library. Actually, I think I could be convinced about Python, but
the Scheme core was easier to integrate.

Perhaps my view is wrong, perhaps not. By articulating rationale we can
have an informed discussion about this before doing something that will
really hurt for a long time.


@node Futures
@chapter Futures and Things To Do
@cindex Future of OpenCM

@contents
@bye
