head	1.1;
access;
symbols
	TIGRIS_1_1_0RC2:1.1
	TIGRIS_1_1_0RC1:1.1
	TIGRIS_1_1:1.1.0.14
	TIGRIS_1_0_8:1.1
	TIGRIS_1_0_8RC3:1.1
	TIGRIS_1_0_8RC2:1.1
	TIGRIS_1_0_8RC1:1.1
	TIGRIS_1_0_7:1.1
	TIGRIS_1_0_7RC3:1.1
	TIGRIS_1_0_7RC2:1.1
	TIGRIS_1_0_7RC1:1.1
	TIGRIS_1_0_6:1.1
	TIGRIS_1_0_6RC5:1.1
	TIGRIS_1_0_6RC4:1.1
	TIGRIS_1_0_6RC3:1.1
	TIGRIS_1_0_6RC2:1.1
	TIGRIS_1_0_6RC1:1.1
	TIGRIS_1_0_5:1.1
	TIGRIS_1_0_5RC6:1.1
	TIGRIS_1_0_5RC5:1.1
	TIGRIS_1_0_5RC4:1.1
	TIGRIS_1_0_5RC3:1.1
	TIGRIS_1_0_5RC2:1.1
	TIGRIS_1_0_5RC1:1.1
	TIGRIS_1_0_4:1.1
	TIGRIS_1_0_3:1.1
	TIGRIS_1_0_2:1.1
	TIGRIS_1_0_1:1.1
	TIGRIS_1_0:1.1.0.16
	TIGRIS_1_0_0:1.1
	TIGRIS_1_0_0_RC1:1.1.0.12
	dlr:1.1.1
	TIGRIS_0_9_2_4:1.1
	TIGRIS_0_9_2_3:1.1
	HELM_PEER_PORT_BRANCH:1.1.1.1
	TURBINE_PEER:1.1
	TIGRIS_0_9_2_1:1.1
	TIGRIS_0_9_2:1.1.0.10
	TIGRIS_0_9_0:1.1.0.8
	TIGRIS_0_8_4:1.1.0.6
	TIGRIS_NOV_12_2000:1.1
	OLDHELM:1.1.0.4
	TIGRIS_NOV_11_2000:1.1
	TIGRIS_SEP_13_2000:1.1.0.2
	TIGRIS_BASELINE:1.1;
locks; strict;
comment	@# @;


1.1
date	2000.05.25.02.07.13;	author jrobbins;	state Exp;
branches
	1.1.1.1;
next	;

1.1.1.1
date	2001.02.21.00.37.08;	author dlr;	state Exp;
branches;
next	;


desc
@@


1.1
log
@initial checkin
@
text
@The Design of the Tigris Prototype
by Alexei Kosut <akosut@@collab.net>

Revision: September 2, 1999

This document attempts to describe my design goals and ideas during
the eight weeks I spent designing and implementing the prototype
Tigris system in the summer of 1999. It sets forth my ideas for the
system, how the actual implementation varies from them, and how I
envision further development on the project takes place. It is aimed
at developers who are familiar with, and can understand, the code
(which is written in a myriad of Java, Perl and C). It assumes a
familiarity with the Tigris project and it's goals, which are
described in the Tigris Specification document. It also leaves out
details about the implementation and the system that can be obtained
from the INSTALL document, or from simply using the basic system.

The basics: Tigris is composed of four parts: the front-end servlets,
the database, the back-end configuration Perl scripts, and the actual
Tigris tools (CVS, qmail, and so forth). A diagram might look
something like this:

  +--------------------------------+ +-----------------------------------+
  |         Web browser            | |  Client tools (CVS, mail, etc...) |
  +--------------------------------+ +-----------------------------------+
                 |             |                           |
                 |             +----------------------+----+
                 |                                    |
  +---------------------------------------+           |
  |    Admin interface (Java servlets)    |           |
  +---------------------------------------+           |
                 |                                    |
  +=============================================+     |               
  |               MySQL Database                |     |
  +=============================================+     |
                  |                   |               |
  +--------------------------+        |               |
  |  Configuration scripts   |        |               |
  |          (Perl)          |        |               |
  +--------------------------+        |               |
                  |                   |               |
  +---------------------------------------------------------------------+
  | Back-end Tigris tools (Mail, Web, CVS server, bug database, etc...) |
  +---------------------------------------------------------------------+

The core Tigris component is a MySQL database, which stores all the
relevant configuration information about the Tigris system: which
users, project, modules, mailing lists, Web sites, etc..., are
configured, how they are set up, etc... Note that in most cases, the
database does not contain the actual data backed by Tigris (e.g., the
files maintained by CVS, but bug database records (although some tools
may use a different database run through MySQL to store such
records)), but simply the information needed to configure these tools.

Data is at the same time pushed into and pulled from the database. On
the front end, the content of the database is manipulated by a set of
Java servlets. It was my original intention that these could be
swapped out for any other SQL-based tool, such as a Win32 GUI
application, or an SNMP tool, but in the process of creating the
prototype, I put a lot more logic into the servlets than I had
originally intended, making this objective a lot harder to obtain;
however, since it was never seen as a real need for Tigris, it was
much easier to design the system this way.

On the back end, configuration information from the database is pushed
into the tools that need configuring. Tigris is, in its essence, a set
of off-the-shelf software development tools such as CVS. What Tigris
does is enables a single point of configuration and
administration. However, since most of the tools were not set up with
this goal in mind, it is necessary to turn the information stored in
the Tigris database into configuration information that the tools can
understand. This has been done in one of two ways:

The first, and simplest, method, is to simply modify the tool itself
to contact the database and request the information. I tried to avoid
this approach as much as possible, because patching software that I do
not control means that as that software develops further, the patch
becomes out of date, and either further development must be placed
into updating the patch, or one must stick with an older version of
the software. The only part of the Tigris prototype that I used this
method for was CVS. This is because CVS, unfortunately, did not
support the level of configuration I needed out of the box, so I had
to patch it extensively anyway. It was easier to simply have it
contact the database than set up a generic configuration
system. Because CVS was recently acquired by Sourcegear, and I imagine
they have vast plans for its future, I felt relatively secure in
modifying CVS, since the CVS 1.10 tree has been fairly dead for some
time.

The second approach, which I used for the other tools used in the
Tigris prototypes, is a set of Perl scripts (modules in the Tigris::
namespace) that contact the database, determine what has changed since
the last update, and spit out a set of flat files or use other,
standard, configuration methods, to set up the tools. These scripts
are automatically run by the servlets when changes are made, and they
keep the Tigris system up to date. This method has the advantage that
the tool itself need not be modified to work with Tigris, and that the
way the Tigris database specifies the configuration can be changed
without modifying the back-end tool itself. The disadvantage is that
it introduces an extra layer of complexity, that may at times be
undesirable.

The entire Tigris system was designed to be extensible, although not
in the script technical sense that new peices can be "plugged in"
without modifying the rest of Tigris. However, the layout of the
system does lend itself to easily adding new database tables, servlets
or back-end tools without destroying the Tigris system as a whole. The
main intent was to allow rapid development, and a high degree of local
customiziation, without the overhead of a rigid API or the development
time needed to create a large plugin architecture. It may be that, in
the future, something of this sort may be desirable. At this time, it
does not seem to be the case.

The MySQL database is layed out in a fairly ad hoc manner; each table
generally represents some type of 'object' in the Tigris system. One
table for the users, one for the projects, modules, mailing lists, Web
sites, etc... A new service can be added to the database simply by
creating a table for it. If the new service can be added per-project,
you simply need a 'project' column to identify the project the service
is in. Note that for all per-module database records, there need to be
both 'module' and 'project' columns, as modules are part of a specific
project.

There are also three privilege tables, domain_privs, project_privs and
module_privs, which specify a user's privilege for the whole domain,
or for projects or modules. These contain key columns (username,
username/project, username/project/module) and a number of additional
columns that each represent a privilege. For example, checkout_priv
(present in all three tables) enables read access to CVS. To add a new
privilege to the database (e.g., if a new service is added, and access
control to it is desired), simply add a new column to the proper
table (you will also have to modify the Java classes that deal with
permissions, as described below).

Also worth noting is the domain_config table, which provides random
key/value strings that are used to configure the Tigris system. Most
of these might normally be present in a flat configuration file, but
since many of the values are used throughout the Tigris system, in
both Perl, Java and C code, accessing the configuration information
would require writing the code three times. Since all the Tigris
components can talk to the database (either directly or indirectly),
putting this information directly into a database table allows all the
tools to share the same configuration system.

Access to the database is restricted to a single user; actual Tigris
user authentication is not done by the database but by other tools
accessing the database and examining the domain_users table. This
contains the username, password (encrypted using the MySQL PASSWORD()
routine), real name, email address, etc... Any code wishing to
authenticate a user connects to the database (the Java servlets and
Perl modules I have written automate this process - other languages
will have to roll their own) and verifies this information.

The Java servlets are designed using WebMacro, a template-based
servlet system that allows you to seperate the content (HTML) output
of the servlet from the code/logic needed to generate that
content. All the servlets that should be subclassed or used by other
classes have extensive javadoc documentation; reading through the HTML
files in tigris/java/html is highly recommended. Especially the
classes (org.tigris.)TigrisServlet, TigrisContext, TigrisRequest and
TigrisPrivs.

Note that I developed the servlets using Java 2, and I make heavy use
of the Collections API. Later, I had the need to install the Tigris
prototype on a platform that only supported the 1.1 JDK (FreeBSD), so
I had to compromise: I modified Tigris to use no Java 2 features
except the Collections classes, and then extracted the Collections
classes from the JDK 1.2 jarfile and added them to the 1.1
classpath. This turns out to work (see the INSTALL file for more
details). I would recommend that all further development be done using
a Java 2 platform (e.g., JDK 1.2), although maintaining the semi-1.1
support may not be a bad idea for a while.

Each servlet extends from TigrisServlet, and provides a handleTigris()
method to implement whatever it is that servlet does (see the
TigrisServlet docs). It is handed three objects, one that represents
the WebMacro context for the request (see the WebMacro documentation),
one that represents the current Tigris (as a whole) context - i.e.,
the database connection, and one that represents information about the
current Tigris request. This last one allows easy access to
information about the current user and the permissions that user
has. The servlet should return, after verifying that everything is
proper and correct, and setting any necessary WebMacro context
variables, the name of a WebMacro template file, which are kept in
tigris/java/templates. There is approxamately one of these per
servlet, and contain the HTML that is generated. The prototype Tigris
uses fairly simple HTML, since the focus was on functionality, not
appearance. These files can, however, be changed without even looking
at the Java servlets, if so desired.

The Tigris permissions are controlled by the UserPrivs, ProjectPrivs
and ModulePrivs classes, which correspond to the domain_privs,
project_privs and module_privs tables in the database,
respectively. These contain knowledge of which columns are in those
tables, so if the tables are modified, the classes must be as well
(this automatically adds support for modifying these privileges to the
appropriate servlets). These classes subclass from TigrisPrivs, which
allows generic control of permission tables in the Tigris database. It
can even be used for some non-permission tables, e.g., it is used in
the mailing list servlets for maintaining the list of ezmlm mailing
list options, which, like servlets, are a long list of binary (I use
SQL type enum('N','Y')) columns.

Probably the best way to understand the servlet system is to read
through some of the servlets; the code is well-commented, and the
method calls are documented using javadoc.

On the other side of the database are Perl modules, in the Tigris::
namespace, that handle updating the configuration files based on the
database. These are also well-commented, and are probably work looking
through when learning how Tigris works. Unfortunately, I never got
around to writing perldoc documentation for these modules, but the
comments in the code itself should suffice. The main modules of
importance are Tigris::Database, Tigris::Config, Tigris::Updater and
Tigris::FileSync. Tigris::Database and Tigris::Config provide access
to the database and the domain configuration parameters for all the
modules. Tigris::Updater is the base class for the modules that
actually implement the database-to-configurationtranslators. Subclasses implement various methods, the most important
of which is update(), to enable a common interface to the Perl
scripts. Reading through Tigris::Update.pm is a must before creating
or editing the Perl modules. Tigris::FileSync is a handy utility
module used by most of the Perl modules - it automatically takes care
of maintaining files based on a table in the database, and providing
hooks for modifying/deleting/adding these files when the database
table changes.

The Perl modules are called by a CGI script located in
tigris/perl/cgi/update.cgi. Originally, this was a standard executable
Perl script, but I ran into problems getting Java to execute it - Java
isn't good at executing programs that aren't Java code. However, it is
very good at talking to Web servers, and Web servers are pretty good
at running programs, so that was my solution. The script is setuid,
and changes uid based on what the given module needs. Currently,
Tigris uses three uids, one for CVS, one for qmail and one for
Apache. I guess these aren't really necessary, but I was thinking
ahead to a much more secure version of Tigris, where bottling up the
functionality of each section into a restricted environment might be
useful (I had just finished installing qmail at the time I thought of
this, with the obvious influences). The update.cgi script contains a
list of the Perl modules to run; when new modules are added to Tigris,
they must also be added to the update.cgi script.

In the eight weeks I spent on the Tigris prototype, I was able to get
a fair bit of the basic system working, and work out a number of the
kinks in getting simple CVS, mail and Web control up, as well as a
framework for further extensibility. My thoughts for near-future
Tigris goals are as follows, and I'd hope that whomever works on the
Tigris in the future can implement them easily:

Obviously, more back-end tools are needed. A bug database, for
example, as well as Web-based CVS browsers, etc... The spec.txt Tigris
Specification document goes into great detail as to what software
Tigris was designed to support, as well as some idea of their
priority, although I imagine this will change as people begin to use
this software.

Greater flexiblity in the system would be nice; I had originally
wanted mailing lists to be per-project (as they are), but with the
option of per-module sublists (e.g., for CVS mailing lists specific to
each module). I never looked into this much, but I still think it
would be a nice thing to implement. Further, a lot more toggles are
needed to customize the CVS tools (e.g., turning on/off automatic Web
site updates), and even the ezmlm stuff isn't complete - some of the
options can't be controlled by ezmlm-make or by email-based remote
admin, and you have to edit the files directly (although which
features these are are very unclear - ezmlm is not a simple system to
understand).

Qmail is the one tool not configured by Tigris - the control rcpthosts
and virtualdomains files are not updated by Tigris, the sysadmin has
to do it manually. I didn't bother doing this because adding new
projects necessitates new DNS records as well, and I did not want to
deal with updating DNS for the prototype. Ideally, Tigris should do
both, so that new projects can be added completely at the command
line.

I had also wanted to implement "anonymous user" support. In the
database, this would probably be no more than creating a user named
anonymous with a blank password, but I envisioned the servlet support
for this being specific. In particular, anon would not be listed with
other users in the user lists, but a specific "Allow anonymous access"
checkbox would be present in the service control servlet pages. This
is not a hard thing to do, but I simply ran out of time and there were
more pressing issues.

At any rate, hopefully this document has provided some insight into
the design principles that went into the implementation of Tigris. I
don't know if I'll be involved with the further development of Tigris
(probably not, at least at first), but I welcome any questions or
comments at <akosut@@collab.net>. Enjoy!

@


1.1.1.1
log
@Replacing the HEAD with the HELM_PEER_PORT_BRANCH.
@
text
@@

