For some time, in the context of my workflow toolkit, I've been thinking intensively about UI design in wxPython.
See, once I was embroiled in a rather extensive project developing a GUI application under wxPython, and frankly, the UI was unmanageable. It had been developed with some IDE tool or another, but the output was Python code. It was horrible, trying to find what was what and on which panel it was developed and what its ID was -- ugh! This was back in about 2001.
At that point, I hadn't really started integrating wftk into Python yet, but I dabbled in it over the next couple of years, always with the notion that the UI is most sensibly defined in XML, and that a sensible UI manager would then take that definition and build all the objects needed to implement it in wxPython (or, for instance, online in a portal or something). And since that time, other people have naturally had many of the same ideas, and you see this implemented. But I've always wanted to finish my own implementation.
The current app for Anappaweek.com that I'm working on is, of course, a GUI app (at least, some of the time.) And so naturally I have relived my need for my UI design notion -- and in the context of working on the file tagger, I intend to start implementing the UI module. On that note, here is a tentative UI definition sketch for the file tagger. Ideally, we could use this XML not only to generate the app itself, but also to generate documentation for the UI design (by transforming it with XSLT into SVG, for instance; wouldn't that be indescribably cool?)
All of this is, of course, subject to radical change. Here goes:
<frame>
<tabset>
<tab label="Cloud">
<html>
</tab>
<tab label="Files">
<splitter (some kind of parameters)>
<panel>
<radio-group>
<radio value="something" label="All"/>
<radio value="something" label="Some"/>
</radio-group>
<listbox/>
<button label="Show"/>
</panel>
<listcontrol>
<col label="Name"/>
<col label="Tags"/>
<col label="Description"/>
</listcontrol>
</tab>
</tabset>
</frame>
I already have a framework for that definition to go into -- I wrote that in, like, 2002 or so. But I never got further than definition of menus. So here, I'm going to implement frames, and at least one dialog.
Note that what's utterly missing from this is any reference to code to handle events. That will come later, when I see what has to be defined where to get all this to work.
And on that note, I close.
I posted the second appaweek project last week, based on the wxpywf framework from the wftk, as you know. And this week I've started mulling over app #3, and since I'd upgraded from Python 2.1 to 2.4, my old (ancient) copy of the McMillan installer was way outdated. Turns out that McMillan disappeared from the Net in about 2004, but some enterprising souls (headed by a guy in Puerto Rico, no less, at UPR!) have taken up the gauntlet and are continuing development under the name PyInstaller.
So that was good, but when I got down to business compiling the little filemonitor work-in-progress, I found that the wx libraries have now grown in size, with the result that the (already somewhat heavy) 2MB size of a simple compiled wx app has now become 5MB! Wow! (Of course, that includes the runtime and everything, but still...)
I don't want that kind of stuff bogging down my server, and already there has been respectable traffic around the appaweek projects. But there is an awful lot of specialized code involved in this venture: wx itself, of course, and the wftk's xmlapi and repmgr libraries, their Python wrappers, and the wftk and wxpywf OO wrappers around those. That's a lot of stuff to make people install, especially on Windows when they may simply have to install Python to start with! Which is why I wanted to do the compilation thing in the first place, of course.
So here's what I thought to do -- and this is what wxpywf was always intended to do: I brought the PyPop app up to date so that it displays the XML-defined UI just fine. And it would then be possible to do dynamic importation of a module to complete the application; PyPop already contains Python and all the libraries necessary to run things, so that would work fine. And yes, the UI for the filetagger app (app #2) displayed fine, although it naturally didn't do anything except display.
But then it occurred to me: Wouldn't it be nice if wxpywf could look at the XML definition of the UI and go ahead and build the CLI definition? And then I realized that if PyPop could (1) interpret the command line and put the results into the context, (2) load the data file and do the necessary Python indexing of it, and (3) build a Python CLI on the fly and link it to the frame, then, then...
Then the entire app would be in one XML file!
And a little bitty one, too, a whole app in about 20KB. That's a quick download! And then:
If the download of the app were managed through PyPop then we'd really have a Swiss Army Knife for software! And of course, more serious Python requirements could always import a Python file for better clarity, leaving the main application to define the CLI glue holding it all together.
Then envision this: imagine an application repository, and imagine the PyPop tool as a custom-thick-client "browser" of sorts. The local PyPop installation would manage common dialogs and applications in a versioned repository, and could check for updates every time you ran an app (or on a schedule). You could design your own UIs and submit them as project requests, and have a programmer fill in the hard parts. Every app built on this framework already knows about data management from the repmgr and about workflow from the wftk, it has logging and macro capabilities built in, and it runs Python. It could run Perl if you wanted to install it -- Perl's easy to embed in Python.
You'd know where all your little apps were, and you could trade them through the central repository or you could set them up for download yourself.
That would be pretty neat. Pretty neat indeed!
Wow. This wasn't something I'd budgeted time for, but it works. I have reimplemented the filetagger app as a single-file definition, including action tags using a very convenient abbreviated notation for wxpywf-relevant Python. For instance, the code to execute a command against the frame is
self.frame.do(context, 'update_list')
in undecorated Python. In the action, it can be just
: update_list
The wxpywf then scans the entire UI definition for action tags; for each, it builds the Python code, wraps the whole thing up as a Python class inheriting from wftk.cli, and returns it to the caller, which can then exec it to define the class.
The error handling leaves much to be desired; in case of a syntax error, it's almost understandable where the error is, but in case of a runtime error, Python's stack trace is essentially useless (file <string> line 90 doesn't help much with invisibly generated code.)
But it is cool, and on-the-fly code generation is always something I wanted to find a legitimate use for. And now I have!
At any rate, I obviously need to get a lot of code uploaded at some point and documentation caught up, but now is not the time. Work beckons, frantically.
I never really officially released wftk 1.0, of course (the magnitude of the task simply grew and grew and I became less and less certain of my approach -- and then the recession happened.) But I've been thinking a lot of a more reasoned approach lately, and maybe it's time to reboot the wftk project and start more or less "from scratch".
I see the modules in this new approach more or less as the following:
- Data management
This is the basic list-and-record aspect that the repository manager started out addressing. Now, of course, there is SQLite. So a principled workflow toolkit would start by using SQLite for local tables, and add "external tables" (for which the new SQLite has an API) defined in what SAP now calls the "system landscape". It's amazing, by the way, how much of my thinking over the past few years I see reflected in what SAP is doing lately in their NetWeaver stuff. - Document management
Document management, as I see it, consists of: (1) actual central storage and versioning of unstructured data; (2) storage of metadata about documents; (3) parsing and indexing of unstructured data to produce structured data elsewhere in the system. The document manager should be able to work well in either situations where it controls storage (and thus can initiate action whenever anything is changed) or when it merely indexes a storage which can be changed externally -- that latter might be, for instance, management of a Website's files in the file system. Or just your system files on a Windows machine. Periodically, the document manager could check in and see whether things had been changed, and if so, trigger arbitrary action. - "Action" management
A central script and code repository defines the actions that can be taken by a system. I consider this to include versioning and some kind of change management and documentation system, including literate programming and indexing of the code snippets. The build process should also be managed here, and should be capable, for instance, of taking algorithms written in C, compiling them into DLLs or .so dynamic load libraries, and calling them from Perl, say. Ultimately.Actions, documents, and data would have a nested structure, by the way; there would be global actions, application actions (a given case or project could be an instance of an application), and project/instance actions, and the same applies to data and documents, perhaps. Originally I'd thought of doing the same for users or organizational units, but I really think that if you're defining a common language of actions and data, it should be organized into applications and, perhaps, subapplications or something. But not differ by user! (I might be wrong, of course.)
The above three modules together allow a data-flow-oriented processing system, but we're still missing:
- Outgoing interfaces
This includes publishing of HTML pages, outgoing mail notifications, other notifications such as SMS or ... whatever. Logged, all of it. It includes report generation into the document management system or the file system, generation of PDFs, etc. - Incoming interfaces
Given the parsing power of the document management module, this is more an organizational module. The system should be able to receive email, parse it, and take action. Conversational interfaces are covered here as well, from SMTP- and IMAP-like state machines to chatbot NLP interfaces. And of course form submission from Websites also falls into this bucket. - Scheduling
Whether running on Unix with cron and at, or Windows with ... whatever the hell Windows offers, the system should have a single unified way of dealing with time in a list of scheduled tasks. - Users, groups, roles, and permissions
This module would be in charge of keeping track of who is performing a given action and whether they're allowed to do so. The original wftk already provided a really nice mechanism which would still be nice here: when judging permissions, any action can get the answers "yes, it's allowed", "no, it's not allowed," and "it's allowed subject to approval." That last invokes workflow for any arbitrary action and that would be a powerful abstraction for nearly any system. It's essentially transaction management on a much more abstract scale.And finally, the icing on the cake,
- Workflow
The two components which make workflow workflow are a task list (tasks are hierarchical in nature and so a task can have subtasks as a separate project) and a workflow process definition language. The new wftk should be able to work with any workflow formalism -- after all, the process definitions are considered scripts in the versioned script document repository. The existing wftk engine will almost certainly fit in here with little modification.The primary benefit of workflow is that it allows dissociation over time. A running workflow process isn't active on the machine for the weeks or months it might require -- it's simply a construct in the database that gets resurrected as required. There are a boatload of applications in general programming, but nobody sees them as workflow because everybody "knows" workflow is a business application. The wftk was to have changed that, and I think the potential's still there.
There's also a case to be made for a module for
- Knowledge management
This portion of my thinking is a little less organized. I'd kind of like to lump some kind of concept database in here, perhaps a semantic parser or something. Originally I'd thought that AI would go in here, but I actually think that Prolog might just be another action script language. This is definitely a blurry line in its native habitat, and crikey, he's not happy to see me here!But the point of a blog is to write this stuff down as it occurs. So there you have it, this would sit on top of the workflow. Think of it as a way to build smart agents into your data/document/action/workflow management system.
And there you have it -- my plan to wrap up the thought and work of eight years. Oh, and this time I'm not bothering with licensing requirements. Like SQLite, wftk 2.0 will be in the public domain. I don't really care if I get credit or not for every little thing, because frankly, anybody who counts will figure it out. And have you noticed how everything these days uses SQLite? It's because -- well, primarily because it works, but also because you don't have to worry about legal repercussions of using the code.
That's where wftk document management should be, where wftk workflow should be. Simple, easy to use, and ubiquitous.
Something that a lot of my project ideas have in common lately is a kind of generalized document management framework.
This isn't as impressive as it sounds, actually. But it's kind of a key notion for Web 2.0 stuff -- if you want collaboration, you have to have a place to store that collaborated content. That place is the document management system.
Let's consider this for a moment, in the context of the fantasy name generator from last week. That fascinating little thing takes a simple document -- the language definition -- and runs a Perl script on it, yielding some interesting results. The Toon-o-Matic does the same thing; it takes a simple XML document and runs a whole sheaf of Perl on it to generate an image. A Wiki for my general site content, or a forum, or even a simple Web form post, can all be seen as doing the same thing. An online programming tool; same thing. All these systems share a component -- the user can submit a large (ish) text object, often based on an existing one, for arbitrary processing, which usually has some visible effect on the system.
If you just look at that little unit of functionality, you can imagine lots of attractive ways to extend it, too. As I mentioned in my initial post on the fantasy namer, you can suddenly imagine allowing people to name a particular definition. You can imagine a page devoted to it, perhaps including all the results it's generated -- maybe in ranked order. All that's a lot of different features, but the central one is simply being able to store and manipulate that central document. It provides a hook on which you can start hanging interaction; without the hook you can't even conceive of where to start.
So this notion's been in my head lately. Oh, I'm sure all this was done in much more diligent detail ten years ago. (Well. Seven or eight years ago, anyway.) In fact, I can think of a couple of systems -- but they're all too damned complicated. What I'm after is the ability to boil these things down to their essence, to provide a language of thought about these systems. For myself, anyway. Assuming you exist, you may or may not benefit. (But I'll bet you would.)
