Beyond The Config File: User-Friendly Configuration For Web Apps

Introduction: The Three Stages Of App

Oftentimes, a very small or very new web application will keep all of its configuration settings inside the code itself. Hopefully this is done as constant definitions in some obvious place. To change the configuration, a user must become a developer of the application. For example:

#Whether or not to enable image uploads
ENABLE_IMAGE_UPLOADS = 1

#The maximum size of an uploaded image, in kilobytes
MAXIMUM_IMAGE_SIZE = 100

A larger, more flexible, or more mature application may keep its configuration settings in a text file loaded by the application at runtime. When configuring such an application a user is changing data, rather than code, and acting as an administrator rather than a developer. Python's ConfigParser module provides built-in support for this level of flexibility, for example:

[Images]
enable-image-upload=1  #Whether or not to enable image uploads
maximum-image-size=100 #Maximum size of an uploaded image, in kb

The natural next step toward better usability is to provide an interface to the configuration file from within the application itself, using the same framework and UI standards found in the rest of the application. This allows a user to configure the application while remaining just that--a user:

Enable image uploads?
Maximum image size? kb

The best way to take this step is to use a configuration framework: an end-user interface to an application's configuration, which looks like the rest of the application. This makes the application easier to use, increasing the number of potential users, without making the developer's job more difficult. In an application which has user accounts, the same configuration framework can drive a system for gathering user preferences.

The topic of this paper is configuration frameworks for web applications in Python. The lessons are applicable elsewhere, but the web environment is conducive to dynamically generated user interfaces, and web applications are most likely to have nontechnical users. A configuration framework can be implemented in any language, but I have found Python ideal because of its strong support for classes and reflection.

Even within the Python world, web application environments differ widely, so it is not my intention to evangelize one true configuration framework. This paper will focus on the components common to any configuration framework rather than evangelizing one true configuration framework. The second half of this paper will look at one implementation of a configuration framework, called I Want Options, which can be used as an example or extended to fit within a given Web framework.

Advantages of Configuration Frameworks

Safer, more user-friendly

A configuration framework is easier and safer to use than a text editor on a flat file. The structure provided by selection boxes and checkboxes prevents some bad values from being introduced at all. For freeform text values, the interaction provided by form submission gives immediate feedback about invalid settings. In contrast, the often undocumented syntax rules for flat-file settings are applied at runtime, and the user must switch tools to see if changes they made were valid. ⁰

Self-documenting

The documentation for a configuration option can be stored as part of the setting definition, analogous to the way a method's pydoc documentation is stored with the method. At runtime, it can be displayed alongside the option's setting control. This makes it easy for the user to find the relevant documentation, and easy for the developer to keep that documentation up to date. If an option requires more explanation than is appropriate for a configuration screen, or requires reference material, a link to additional online documentation can be provided.

A configuration framework also makes the application as a whole more self-documenting. Adding a tweakable feature to the application involves adding a controlling option to the configuration framework, which immediately shows up in the user interface. Even if the new feature is disabled by default, the administrator-as-user can find it through the user interface. Without a configuration framework, the administrator needs to consult the documentation or the sample configuration file to discover that a new version has the new feature. When an application is undergoing continuous development, a configuration framework greatly eases the transitions between versions.

Fits the web application philosophy

A configuration framework furthers the goals which drive people to implement web applications as opposed to other types of application.

For instance, one common goal of a web-application is to allow non-technical users to perform specific tasks. A configuration framework makes the task of application administration much more pleasant, even possible, for non-technical users.

Users who would never dare alter a text configuration file from its defaults are much less reluctant to use a helpful web interface to that same file; they trust a web interface to do the right thing. On the psychological level, configuration settings stop being application data and become a special type of user data--a much less scary thing.

Even experienced users may make fuller use of configuration options through an intuitive front-end than through the semantics-free interface of a text editor. The full power of a text editor is rarely needed, since almost all web application settings are less complicated than, say, procmail recipes.

Another goal of web applications is to make services accessible from the lowest-common-denominator platform of the browser. To the extent possible, it's desirable to move configuration tasks onto the web as well, to take advantage of this same accessibility.

Testimonial

My weblog application, NewsBruiser, was once a 98-pound configuration weakling. I had started with a small configuration file, but as the application grew, the configuration file became bloated, full of random options, hard to manage. Then I wrote a configuration framework. Now the configuration file is so simple it can be generated by a setup script, and all other administration is handled from a CGI. It's greatly reduced setup effort and flattened the learning curve.

Anatomy of a Configuration Framework

Any configuration framework will contain the following four pieces:

Option Definition Store

This is a mechanism for defining the configuration options to be set. Unless the option definitions themselves can change at runtime, a flat file in some format is best for this.

At an absolute minimum, the option definition store must contain the following set of properties for each option:

• A name, for use as the key in the key-value pairs in the configuration store.
• A type: boolean, text, etc., or a custom class. This defines the option's semantics.

Almost any configuration framework should also define these additional properties for each option:

• A short label, for use in the display interface.
• A description or explanation, for use in the display interface.
• A default value, for use when the user has not expressed a preference.

Options of certain classes may have additional properties: for instance, a selection option will have a list of valid selection values and their descriptions. An option of a custom type may have arbitrary additional properties.

Configuration Store

This is a mechanism for storing the current values for options, as set by the user. This can be one or more flat files in any format, a database table, or any other data store compatible with the larger application framework.

Option Display and Semantics

So far we have considered how to represent aspects of different options--their names, descriptions, and values. Now we turn to the problem of representing different types of option, each type having different display, storage, and validation requirements. The most obvious example is of a BooleanOption whose interface is a checkbox, versus a StringOption whose interface is a text box and which may enforce some maximum length requirement specified on a per-option basis.

The option definition and configuration stores, which distinguish between options, are best handled as data, but the things that distinguish between types of options are best handled as code. The best way to implement different options is with a subclass of a base Option class for each type of option. Particularly strange options may have their own class, which no other option shares, but most options will implement one of a few common option classes (BooleanOption, StringOption, etc.).

Edit and display interfaces

Each type of option needs an editing interface: a way of rendering GUI controls which a user can use to change the option's value. It is here, for instance, that the boolean option renders its checkbox, the selection option its drop-down selection, etc.

If your application is one in which a user might be able to see but not edit an option ¹, you also need a non-editable display interface for each option. For example, the non-editable display interface for a boolean option might display a "Yes" or "No" instead of a checkbox.

Option semantics

There are several other interfaces which it is useful for option classes to implement:

• A value retrieval interface--a method to return the current value of this option in the format most useful to the application. For instance, the boolean option class will return a boolean even if the value is stored differently internally.
• A validation interface--a method to determine whether a proposed value is acceptable for an option. For instance, an option value representing an integer should be able to reject values that cannot be coerced into integer form. One option may have different acceptable values depending on the value of other options; this interface allows you to enforce such dependencies. ²
• A relevancy interface--a method to determine whether the application is in a state such that this option can be set by the user. Certain option values may disable other options, or a user may not have permission to edit certain options; this interface lets you reduce UI clutter by not displaying settings whose values are currently irrelevant or immutable.

User Interface

Of course, the whole reason you are going through all this trouble is so users can edit your application's configuration from within the application. So you need a user interface that looks like the rest of your application.

Details differ between application frameworks, but a configuration interface CGI generally implements the following logic:

Retrieve any submitted values from the CGI form data.

For each submitted value:
  Validate value and gather any errors

If there are no errors:
  For each submitted value:
    Change the value for this setting to the submitted value
Else:
  Print the errors

Print the editing form with the current values for each setting,
 overridden by any submitted form values. This is so that submitting
 a form with invalid option values doesn't destroy all the work the
 user put into the other values in that submission.

option=maximumImageSize
displayName=Maximum image size?
type=ByteSize
unit=kb
defaultValue=100
maxValue=10000

maximumImageSize=100

class ByteSizeOption(IntegerOption):
   def printEditControl(self):
       IntegerOption.printEditControl(self)
       print self.getUnitAbbreviation()

   def getUnitAbbreviation(self):
       return self.unit

class ConfigurationCGI(BaseCGI):

     def processRequest():
         ...

Optional Feature: Option Grouping

It often makes sense to allow options to be grouped into sets of similar options, so that the user is not overwhelmed by a myriad of HTML controls. For example, an application might set all of its cosmetic settings (such as HTML colors and greeting messages) in one option group, its security policy settings in another, its locale and timezone settings in a third, and so on.

Option sets can be separated visually, or separately accessible through a tabbed interface in the configuration GUI. Configuration frameworks which handle relatively few or very cohesive settings need not have sets, but it is useful for breaking up into coherent sections a page which would otherwise contain dozens of checkboxes.

Limitations of Configuration Frameworks

Some information is best left out of the configuration framework. Information about system and data paths, and other options necessary to the running of the application, should probably still be controlled with hand-edited configuration files. It should be impossible for a user to break the application through the interface by choosing a bad value for a configuration option. Properly implementing validation for such options can mitigate this concern somewhat, but this is not always possible.

I Want Options: An Extensible Configuration Framework

I Want Options (IWO) is a Python configuration framework which can be used directly, if you don't mind things looking ugly and being stored in flat files, or extended to fit in with your web framework if otherwise. It provides all necessary parts of a configuration framework, and can be extended to fit in with the UI standards and storage mechanisms of the rest of your application. This section reveals how IWO implements each necessary part of a configuration framework.

OptionConfig: The Option Definition Store

IWO defines a class called OptionConfig which instantiates OptionGroup and Option objects defined in a flat file in the following simple format:

group=Group1
Description of group #1

option=option1
type=selection
displayName=Option #1
description=This is option #1
values=value1,value2
valueDescriptions=Selection value #1,Selection value #2
default=value1

option=option2
...

group=Group2
...

The relevant method is loadOptionDefinitions; you can override this method in a subclass to read in an option configuration store in the ConfigParser format, ³ the ZConfig schema format, an XML format of your choosing, or from a database.

An option's properties become members of the corresponding Option object. The following option properties have special meaning to I Want Options:

• option: This is required and must come first in an option definition. The value is the internal name of this option, which need not be human-readable.
• type: The type of the option; an alias for a preloaded type (e.g. "string"), or the name of an Option subclass.
• displayName: A short, human-readable name for the option.
• description: A longer description of the option. Should explain the option, and link to more documentation if the option can't be explained in just a sentence or two.
• default: The default value of this option.
• group: Ordinarily, an option belongs to the group most recently defined in the configuration store. However, you can override this by explicitly specifying a group for an option. This is useful when you have multiple configuration stores, and an option in store B wants to belong to a group defined way back in store A.

Other types of option, like the SelectionOption, define additional properties, such as values and valueDescriptions. An option may have arbitrary properties besides the ones listed above, and the Option subclass or the OptionWrapper class (see below) may use them in any way imaginable.

Context: The Configuration Store

Out of necessity, the configuration store is application-specific, so IWO provides an abstract class called Context which is used as the interface to the configuration store. This class is so called because it is the application-specific context in which configuration options are set. The idea is to change the appropriate object in an existing application to extend Context. For instance, if you are using IWO to do user preferences, then the context is the user, because each user has their own preferences. In a weblog application like NewsBruiser, the context is the weblog, because each weblog has its own settings. In an application which uses IWO to do global configuration, the context is the entire site. ⁴

FileBackedContext is a simple implementation of Context which writes key-value pairs to a prespecified flat file. It's used extensively in the example applications, and its simplicity is its primary virtue.

Option: The Option Display and Semantics

IWO defines a base Option class with many subclasses. Option contains an editing interface (renderEditControl), a value retrieval interface (getValue), a relevancy interface (isRelevant), and a validation interface (validate).

The sundry subclasses of Option (StringOption, BooleanOption, EmailOption, etc.) mainly provide different implementations of the editing, value retrieval, and validation interfaces. The relevancy of an attribute is generally decided on a per-attribute basis, not on a per-attribute-type basis. A small counterexample: SelectionOption decides that it is not relevant if it has only one valid selection value.

ConfigurationCGI: The User Interface

This is a simple CGI which allows the user to edit the options in an option group, and to switch between groups. It also does validation and error reporting. However, it's very primitive, and is more useful for demonstration and example purposes than for actual use; I recommend you implement similar CGI logic in the application framework you're using.

OptionGroup: The Option Grouping Implementation

OptionGroup is an implementation of the optional grouping feature. OptionGroup objects are instantiated by the OptionConfig object as it reads from the option definition store, and stored in a hash by OptionGroup name. An OptionGroup contains a list of the Option objects defined as being part of that group, in the order they were defined in the configuration store.

OptionWrapper: Everything Else

If you were implementing your own configuration framework, everything could go into the four sections listed above. Since I Want Options is intended to be reusable, it needs a subclassable container for application-specific logic, so that to get some minor change in functionality you don't have to subclass OptionGroup and every Option subclass. OptionWrapper is that container.

Many OptionGroup and Option methods delegate to the OptionWrapper, and by overriding its methods in a subclass you can:

• Change the rendering of the edit interface however you want.
• Set general relevancy rules for your attributes: for instance, attributes with a certain custom property may only be relevant if the user has certain permissions, or if the application as a whole is in a particular state.
• Transparently alter the text of option or option group display text, or of option default values, from the values stored in the option definition store. For instance, the default title of a user's page in the application might incorporate the name of the user; you would store the default title as something like "%U's Page", and override getOptionDefault to replace "%U" with the current user's name.

Conclusion

Using a configuration framework to configure your application is a simple way to give a boost to its usability in an area you may not have thought needed any boosting. Adding a configuration framework not only lowers the barriers to use, it helps dispel the image of web applications as unfriendly to administer. You may even find that the immediate exposure of new functionality through the configuration interface gives you new ideas and helps you better organize your application.

Downloads

I Want Options is available for download at http://www.crummy.com/devel/PyCon2003/IWantOptions.py. The example applications used in this demonstration are available for download at http://www.crummy.com/devel/PyCon2003/examples.tar.gz.

Toy examples

Four simple example applications are included in the tarball linked above.

• A demonstration of the Three Stages Of App: a simple random text generator, implemented three times. The first implementation uses hard-coded constants, the second uses a configuration file, and the third uses I Want Options to provide a configuration interface.
• A simple portal application which uses I Want Options both for site options and user preferences.

Real-world examples

NewsBruiser (http://newsbruiser.tigris.org) uses I Want Options for its configuration framework. Look in the Options.py file for custom Option subclasses and an extension of OptionWrapper. The Notebook class extends Context, and CoreCGIs.ConfigureCGI implements the configuration logic.

Bugzilla (http://bugzilla.mozilla.org) has a configuration framework in which both the option definition store and the configuration store consist of Perl code.

The Scarab artifact tracker (http://scarab.tigris.org/) uses a configuration framework to present an interface for gathering data matching a user-defined schema. Because the options themselves are defined at runtime, the configuration options are limited to predefined classes such as text entries, booleans, and selection drop-downs.

Zope provides a similar configuration framework which allows users to define, at runtime, custom fields for data gathering. (See http://www.zope.org/Documentation/Guides/ZCMG-HTML/ZCMG.4.2.html for an example) Again, because the options are set dynamically, the configuration options are limited to predefined option types.

Intake for Java (http://jakarta.apache.org/turbine/fulcrum/howto/intake-service.html), a subproject of Apache's Fulcrum project, provides a generic option definition store, validation interface, and hooks into a configuration store. Intake allows developers to build entire applications, not just configuration frameworks, on the model expounded here.

ZConfig (http://cvs.zope.org/%2Acheckout%2A/ZODB3/ZConfig/doc/zconfig.pdf?rev=HEAD) can be used as an option definition store and the backend to a configuration store. It also defines validation semantics for many types of option.