Applying the wftk: CRM -- (no-frills bug tracking)

[ wftk documentation home ] [ CRM example home ]

Contents (linked entries have been written) is a searchable database of machine tools which has been in continuous operation since 1995. I started working with them in November of 1995 and have stuck with them ever since, through their acquisition in the dot-com boom and their subsequent de-acquisition shortly after said boom. The site is based on AOLserver and the code itself is written in Tcl. There are two major sections of functionality; the "site proper," which is the part in operation since 1995 (much modified) and the searchable database itself, consisting of about 120,000 lines of code; and the "self-maintenance" portion, started early in 2002 and now in production, which is a collaborative data management application allowing the industry community to contribute to accuracy in the database. The self-maintenance application is partially wftk-based and consists of about 30,000 lines of really nitpicking code so far, and we've only addressed maintenance of information about companies (i.e. nothing about machine tools yet).

Since I'm the only programmer attached to the project, this means I have a lot of issues to keep track of. And if you've followed the course of development of the wftk, you know that I'm not the most organized of people. So an issue tracking system is a really logical thing to implement for my work at Techspex. This page represents a chronicle of my attempt to do this; since much of the implementation will require development of wftk to work, you can expect progress here to be really spotty. (But again, if you've been following the wftk, you're used to that.)

The basic issue tracking system

The issue tracking system per se consists of a single list of issues. Since there is only one customer, there's no need to track customers. Since there are only two really meaningful subsystems, I see no need to muddy the waters by tracking them, either. So this is an example of what I've been thinking of as a "simple bug-tracking system," and it's a good place to start working on a real live system.

I call the single issues list "issues", and I end up with a site description file like this:

<list id="issues" defn="issues.xml" order="priority desc, date"/>
This leaves me to configure the issues list in the definition file issues.xml, which I initially set up as follows:
<field id="id" special="key"/>
<field id="date" type="date" special="add"/>
<field id="expected" type="date"/>
<field id="desc"/>
<field id="notes" type="wiki" rows="10" cols="80"/>>
This gives me an ID which the list adaptor will ensure is a unique key (the special="key" part), a date which will be filled in automatically when the issue is opened (the special="add" part), an expected date of completion, a title, and a general notespace of type="text". This doesn't mean I can't add arbitrary other data values, including attached files, as appropriate to each issue, but it does define the minimal set of values along with some information about how to build an edit screen.

I also create a directory named "issues" in my main directory; this will store each issue file, which will be a localdir XML file by default. Note that I could have chosen a different storage area for the issues list.

The "order" attribute on the list definition defines the default order for issue listings. This could be omitted entirely to save some overhead, but it makes a lot of sense to use it, especially once we get into publishing below.

The notes field is a "Wiki text"; it is edited as a textarea (with the rows and columns as specified in its field definition) but formatted as structured text. My inspiration for this was the WikiWikiWeb concept which you can easily find in Google; the point is to make it simple to produce formatted text, links and all, without typing HTML. In my structured text setup, skipped lines are paragraph breaks, asterisks introduce links, and so forth. True to wftk form, I haven't yet documented this (partly because I expect it to remain in flux for a while.) The point is that the notes are designed to be quick ways to create formatted content.

So far, time expended (ignoring the time I'm taking to document this) has been under ten minutes. If I had a fully functional front-end, I'd really already have a simple issue tracking system, because the wftk already supports tasks against objects (like issues), indexing of tasks, and so forth. But I don't have a front-end yet, just pieces of them. Let's talk about the front end a little.

Front end planning

Presentation of issues can be seen in terms of use cases, more or less as follows: Of these, the status pages can easily be thought of as either live database queries or as published static HTML pages. Since I haven't involved a database with this system yet, that leaves me with published static HTML pages. I'll want to create two publishers which will be called whenever the list of issues is changed. The first will be a list template to publish a sorted list of all active issues; the second will operate for each issue and will publish a static representation of that issue. The static issue page will also include hooks for actions to be taken.

I kind of like this model for active site development. Information that doesn't change often can be written a single time and then handed out to many viewers with only minor processor overload. This would, therefore, scale very well. But it still permits the normal navigation of the data to be used to find items to modify and actions to take, so that (active) users of the site needn't keep two pictures of the data in their heads at once.

A second advantage of static HTML (or other static data representations, like an XML file of some sort) is that it lends itself to publishing on some system other than the one actively hosting the bug tracker. In the Techspex case, this doesn't apply, but it can be seen as a distinct advantage.

At any rate, the front end for this system breaks conveniently into two portions: static HTML publishers for system and issue status pages, plus active maintenance code to make actual changes to the issue list. The first is well-supported by code current in December 2002, while the latter category is only spottily supported -- at the moment, it is partially addressed by the CGI front end, but that's the extent. So as part of this exercise, I'm going to address myself first to fixing up the CGI front end sufficiently to work for this system, but I'd also like to start laying out a native AOLserver front end, since that's the HTTP server used by Techspex and thus I'd like to use it in other Techspex applications.

Publishing static HTML for system status

Let's go ahead and define our publishers for the issue list. This is really pretty easy; first we define two pages to provide a place for the publishers to publish to, and then we provide the publishers. We modify the site.opm file to look like this:
<list id="issues" defn="issues.xml"/>
<page id="issues" page="index.html" displays="issues"/>
<page id="issue" page="issue[id].html"/>
<publish id="issues" from="issues" to="issues" defn="index_template.xml"/>
<publish id="issue" from="issues" to="issue" defn="issue_template.xml"/>
<layout id="default" defn="layout.xml"/>
The pages could be used for a navigation interface, but since we only have one datasource that'd be fairly pointless. So instead they just provide a place to define the URL used by each publisher. The publisher elements define the dataflow in this system, which is strictly from the single list into two sets of pages (one of which happens to be a single page.) The layout defines the overall HTML template to be used for all pages on the site; again, this system involves so few pages there would be no point in being more sophisticated with layout types.

Note that the page value for the page "issue" contains square brackets like this: issue[id].html. This is something I call a flat template (because it's not an XML template like the ones we're about to define below) and it gives us a simple way to specify how to build a filename for the static HTML file associated with any particular object: we simply replace [id] with the value of the "id" field for the object. Nifty.

The "displays" attribute on the issues page is there to tell the CGI data management interface that when displaying the full list after modification, it should use that page. This allows the published page to be an effective center for navigation when editing the list.

So now all we have to do is define three templates: the overall site layout, the issue list template, and the issue detail template. Since this system isn't really for public consumption, there's very little reason to pretty it up, so I'm going to keep all of this to a bare minimum to make it easier to see what's where (and also to save me some development time.)

All layouts are effectively templates, that is, they are XML which is run through the xml_template library and then written using xml_writehtml, which makes sure, for instance, that the XML element <br/> gets turned into <br> and the like. Slots in the templates are filled using template:value elements. For the general layout, there is one slot named "content" defined by the system; that will be the default target of whatever page is published. A layout may also optionally define other value slots, which can be filled in using a couple of different mechanisms, only one of which I'll be using here, to fill in the title slot.

Here's the general layout:

<layout type="page">
   <title><template:value name="title"/></title>
<body bgcolor="#FFFFFF" link="#000099" vlink="#990000" alink="990000">
<template:value name="pagetitle"/>
<template:value name="content"/>
This gives us three slots for each page: a title slot, which is just going to be a value; a page title slot, which will contain some formatting for the title (because I don't want the index page to look exactly like an issue page); and finally, the content itself, which will be filled in with whatever is appropriate for the page at hand.

We have two different basic kinds of page here; one lists entries from a list as a table, while the other publishes an individual object. A list template must contain a template:list element, which will connect with the list in question, retrieve it, order it, and proceed.

The list template is the longest file I've quoted yet, but it's still pretty easy to understand:

<template replaces="title">Techspex issue tracker</template>
<template replaces="pagetitle"><h2>Techspex issue tracker</h2></template>
The Techspex issue tracker lists active issues.  Click an issue to see or
modify it.
<td bgcolor="eeeeee">Priority</td>
<td bgcolor="eeeeee">Date entered</td>
<td bgcolor="eeeeee">Description</td>
<template:list order="priority desc, date">
<td align="right"><template:value name="priority"/></td>
<td><template:value name="date" format="date"/></td>
<td><template:value name="desc" format="link:issue[id].html"/></td>
<template:value format="link:repmgr_cgi?mode=new&amp;list=issues">Add new issue</template:value>
Note in particular that the top-level element of this template is not a "template" element, but a "list" element, denoting that it is a list of templates. The individual templates are then used to replace the values of the slots in the layout; if a template is not named, it is assumed to be the content template and will replace the content value in the layout. Clear as mud, right?

Order in list retrieval is specified using an SQL-standard order specifier. If the underlying storage is an RDBMS, then this spec is used directly when building the query; otherwise, it's passed to the xmlobj_list_sort function by non-RDBMS adaptors.

When we insert a single test task into the list with the above index template, the result looks like this:

For the detail, we build a similar very simple template as follows:

<template replaces="title">Techspex issue <template:value name="id"/>: <template:value name="desc"/></template>
<template replaces="pagetitle"><h2><template:value name="desc"/></h2></template>
<tr><td>ID:</td><td><template:value name="id"/></td></tr>
<tr><td>Priority:</td><td><template:value name="priority"/></td></tr>
<tr><td>Date entered:</td><td><template:value name="date" format="date"/></td></tr>
<tr><td colspan="2"><u>Notes</u><br/><template:value name="notes"/></td></tr>
<template:value format="link:repmgr_cgi?mode=edit&amp;list=issues&amp;key=[id]">Edit this entry</template:value>
<a href="index.html">Back to list</a>
Again, this template is a list of three templates to cover our three layout slots. The layout of this detail is pretty boring, and there is as yet no specification of tasks to be done, history, attachments, or indeed much of anything, because there isn't much code there to format any of that. That will be forthcoming soon, I hope, but it's a wftk "soon", so it may be a year before I really get down to it.

The test task referenced above ends up looking like this:

This much gives us a basis for working with the system, even if there is as yet no mature Web interface to manage data; to date, when using the publishing code of the repository manager, I either enter the data by hand through the repmgr command-line interface plus a text editor invoked by the CLI, or I generate the data using Perl scripts running on the system, and simply use the repmgr "publish" command to run off a set of pages showing the current state of the system. I can use either of those approaches with this system as well. But naturally, a Web interface for data management will make the system far, far easier to use.

Data management interface (adding/modifying issues)

Even though my heart wants to build a native AOLserver interface for this system, my head is telling me there's no chance I'll get around to that in a timely fashion. So instead I'm going to recycle the repmgr CGI interface, which is already half-done as of this writing (December 2002). I'm hoping that using this system on a more or less daily basis will motivate me to make improvements to the CGI interface (most of which will probably be easily wrapped in Tcl to create an AOLserver-native version.)

The issue detail above links to repmgr.cgi to "Edit this entry" -- that and the analogous "Add issue" link on the index page are our means of entry to the data management interface. Beyond making sure that a CGI symlink is present in the repository directory, there's nothing more to specify in order to provide CGI access to the system. Once alternative front ends are available, they will likewise be precisely this easy to set up, since they all work off the same site definition file we use for publishing information. At most we need to set configuration parameters (like the "displays" attribute on the index page) in order to modify the default way in which a front end works.

Resolving issues and archiving resolved issues

At first glance, resolving issues is simple -- we just delete the issue. However, this really isn't a great way to handle resolution, for two reasons (neither of which is that the CGI interface doesn't delete things yet as of this writing). First, when a list entry is deleted, the publisher doesn't remove the file it was published to. If someone has bookmarked the page, a fresh visit won't tell them that the issue has been resolved; it will simply have vanished from the master list of open issues, leaving a fossil detail page in place. Moreover, deletion doesn't provide any history at all -- when was the issue resolved? How long did the resolution take?All in all, maintaining a database of only active issues is a weak solution.

The best way to handle this is to build a state machine for the issues list. If we have two states (say, "active" and "resolved") then we can specify archival when the issue reaches the "resolved" state. Archival moves the object into a second list, where it can be published using completely different templates, if necessary. This is rather handy, since the template for a resolved issue can be made visibly different, making it easy to see that something has changed.

The list template for resolved issues can also publish to a separate page, and can sort by something other than priority -- say, by resolution date. And if we specify the archival list with an "add" date, it will automatically mark the date of resolution of the issue.

Another benefit the state machine gives us is that we can define more states -- for instance, we could define an "emergency" state for an issue. The emergency state could be triggered by an automatic process which scans active issues for signs of neglect, and entry of the emergency state could notify management, or page a senior developer, or ... whatever. The "whatever" is the whole point of a workflow toolkit, after all.

To set all this up, we add a new list "issue_archive", and define the state machine for the issues list. Both of these are done in site.opm (although the state machine could just as well be defined in the defn file for the issues list, issues.xml -- the definition is merged with the information already in site.opm, so it doesn't really matter where we define the state machine.) The resulting site.opm, which is getting a little more ungainly as a simple example, looks like this:

<list id="issues" defn="issues.xml" order="priority desc, date">
<state id="active" label="Active"/>p
<state id="urgent" label="Urgent"/>
<state id="resolved" label="Resolved" archive-to="issue_archive"/>
<page id="issues" page="index.html" displays="issues"/>
<page id="issue" page="issue[id].html"/>
<publish id="issues" from="issues" to="issues" defn="index_template.xml"/>
<publish id="issue" from="issues" to="issue" defn="issue_template.xml"/>

<list id="issue_archive" defn="issue_archive.xml" order="resolved desc"/>
<page id="issue_archive_list" page="old_issues.html"/>
<page id="issue_archive_detail" page="issue[id].html"/>
<publish id="issue_archive_list" from="issue_archive" to="issue_archive_list" defn="archive_template.xml"/>
<publish id="issue_archive_detail" from="issue_archive" to="issue_archive_detail" defn="archive_detail_template.xml"/>

<layout id="default" defn="layout.xml"/>
I slipped in an extra state "urgent" just to show where it might go; if you wanted to use that, you'd have to display it in some way by modifying the display templates for the issues list, plus you'd probably want to sort on the state in order to have urgent issues float to the top of the list. But I don't want to spend time on that right now. Similarly, I'm not going to show you the templates for the issue_archive list, because they're essentially the same as those for the issues list.

The "archive-to" attribute on the resolved state indicates that this state is a sink -- that is, when the object hits this state, it will be moved entirely into the issue_archive list. An archival state circumvents any workflow attached to an object, and archival removes all active workflow (or at least, these statements will be true once the integration of the repository manager with the wftk core is complete.) However, the addition to the new list can invoke whatever workflow is required in the context of the new list. From a state-machine point of view, an archival transition can be seen as a compartmentalization of the state machine into manageable submachines (one per list.)

The issue archive has no state machine, and thus cannot transition back to the issues list. There's a case to be made either for the ability to bring an issue back to life by making it an active issue, or forcing the system to create a new issue instead -- I've opted for the latter, mostly because it's simpler. It could easily be changed, of course, by defining an "archiving" state machine for issue_archive.

Interaction with CVS

Since I manage the code for using CVS, I'd like some integration between CVS and the issue tracking system. The real question is what form this integration should take; I have a number of interesting possibilities, none of which are actually working yet. Anyway, eventually I'll get around to exploring all this. Watch this space for further details.

Where from here?

There are naturally an infinite number of features which one can incorporate into any system, and the bug tracker (even the simple one!) is no exception. I haven't had much time to write anything down, but eventually there will be more.

Copyright (c) 2002
Vivtek. Please see the licensing terms for more information.