Why I love Units of Work

Not many lesson planning packages use Unit/s of Work (UoW) to organise their lessons. Classmaker uses them for everything! What is a UoW? A UoW is a container that holds a lesson collection. The lesson collection comprises individual lessons and supporting files. UoW are similar to tags, but unlike tags, a lesson cannot exist without belonging to a UoW and a lesson can only belong to one UoW.  Conversely, multiple tags can be assigned to an individual lesson and by default a lesson has no tags.

This graphic shows how the major components of Classmaker relate to each other using a Venn diagram.

At face value, UoW seem too restrictive..., and tags would appear to be a much better bet, but as I will show, this is not the case.

Pros

1.  Because a lesson must belong to a UoW, a UoW can be much more descriptive than a tag and combine operational information with data. For instance, in Classmaker, the UoW has a title, a short description, the lesson type (non-contact, routine, unit plan, rotating), whether the UoW has just been imported or if is marked for export and a list of all the supporting files attached to it. The subject of the lesson is listed in the Long Term Plan (a superset UoW) along with whether you want to display it's lessons on the Weekly Calendar or not. All of this is information that you don't want to have attached directly to individual lesson plans as some of it is operational information rather than data. Even the data may not be required on some lesson reports. But this is information that you want to be able to refer to easily if you need to, which is discussed next in 2.

2.  UoW's are reciprocal.  By this I mean that if you click on a UoW you can easily view all the child records that belong to it.  But this also works the other way around.  If you click on a child record you instantly know the parent UoW that it belongs to as well. Classmaker uses reciprocity extensively in it's user interface. Whenever you click on a lesson plan on the Weekly Calendar, the lesson detail brings up the UoW that lesson plan belongs to, all the other related lessons in the same UoW, the Long Term Plan the UoW belongs to and all the other related UoW in that Long Term Plan. This visual overview of your planning, Classmaker retrieves for you automatically, every time you click on a lesson plan.

3.  It is much easier to work with groups of records rather than one record at a time. UoW's facilitate this since a lesson can only belong to one UoW. When you delete a UoW, everything inside it goes too. Conversely, to create a clone of a UoW, export and import the entire UoW (1, 1a or 1b) via the hard disk.  The UoW keeps track of whether it is flagged to be exported, is waiting to be imported from disk or has recently been imported, rather than having to concern yourself with the state of the individual lessons involved.

4.  Batch jobs have been used in computing for a long time. The advantage of using batches is that small groups of records can be manipulated together, without exposing the database to the risk of all it's structure being changed beyond reversal by a single SQL statement. UoW's are analogous to a batch job. They can be cloned, deleted and their lesson plan dates changed using Shove. If the result is not what is expected, the cloned UoW is deleted.  Classmaker users will discover that they often use UoW's clone, delete and edit functionality to create new records from existing records as it is much easier to clone, delete and edit records than it is to create them from scratch, particularly if you use Rotating Schedules.

5.  A Classmaker user will clone and edit whenever they can.  For instance, say you need to create a new record in the same time slot as today, but for tomorrow in a different subject.  In all the other packages I've tried, your only solution would be to create a new record from scratch.  In Classmaker, you take today's record, push it forward one day (+Day) and then add it into a different Long Term Plan and Unit Plan (2c.), just four mouse-clicks with no typing of dates and times. This works due to UoW's reciprocity, mentioned above in 2.

6.  Hierarchical data structures are simple to export to disk as that is how the file system works. The parent child relationship implicit in UoW mean their existing structure can be written to disk intact. All other packages I've tried, export their lesson plans as flat files destroying their user interface relationships in the process (that's if they've even tried to export their lessons. Many vendors don't bother).

Of course, UoW's aren't all beer and skittles!

 

Cons

1.  A lesson cannot belong to multiple UoW.  With tags this is a no-brainer and their most useful attribute.
2.  You do not know at a glance all the different UoW you've created, but it's easy to see all your tags.
3.  The tags associated with a lesson can easily be changed at any time. It's more difficult to move a lesson plan from one UoW to another.

 

Summary

I prefer UoW to tags because their hierarchical structure readily translates to data hiding, reciprocity in user interfaces, easy exporting and importing, batching of work, cloning and editing records instead of creating new ones and database exporting to disk without destroying parent child relationships.

The Internet is Dead - Long Live the Portable App!

Recently, I converted Classmaker from a standard Windows app to PocketClassmaker, a portable Windows app.  To transform Classmaker into PocketClassmaker, I had to do the following:

1.  Make sure that all directory references inside Classmaker were relative rather than absolute.
2.  Write user settings to a configuration file rather than the Windows Registry.
3.  Change the database from a Windows Service to an embedded database.

Luckily for me, these three changes were quite easy to do.  In particular, the FirebirdSQL database comes in both flavours, server and embedded, and very little modification is needed to move from one to the other.  In my case, the backup batch file had to be altered, but not much else.

The advantages of a portable app over a standard app are:

1.  No installation is necessary. Just extract the application's directory tree somewhere, perhaps onto the user's Desktop initially.
2.  As the portable app name suggests you can carry around the application on a USB stick or use it from Google Drive or Dropbox.
3.  Portable apps don't alter the host PC's environment in any way.  To get rid of them you simply delete the application's directory tree. 

The disadvantages of a portable app over a standard app are:

1.  You have to start them directly yourself instead of Windows automatically starting them for you as a Windows service.  This means the user has to be comfortable using batch files.
2.  As you move between machines, environment settings such as the printer may change.  As you are no longer using the Windows Registry, you have to reset these every time.
3.  Not suitable for a multi-user environment.

Using a portable app confers the following advantages over a Web app:

1.  Much MUCH faster to use.
2.  Always available regardless of whether the Internet is up or down.
3.  No need for user names and passwords.
4.  Much more flexible use of screen real estate.
5.  Exclusive rights to your own data with full access to the local file system.
6.  Updates are simple.  Just copy the latest version over the top of the old one.

Microsoft Word Mail Merge

I no longer actively teach, but I continue to use Classmaker for all sorts of things.  It's been used as a rudimentary cashbook, a holiday diary, for Cub and Scout planning and lately as a document management program in my electrical business. Today it's been successfully implemented as a data source for Microsoft Word's mail merge.

As an electrician, I'm required to supply every customer with a official document called a Certificate of Compliance (CoC) when doing electrical work.  Until recently, these forms have been sold in triplicate books of twenty.  A form was completed manually, whenever new electrical wiring was done on a job and a copy posted to the customer.  The yellow copy of the triplicate had to be retained for three years.

The world moves on and now we are allowed to create electronic versions of this form, but they must be stored for seven years rather than three years. Furthermore the CoC has been renamed to a Compliance and Electrical Safety Certificate (CESC) and one must be supplied for all electrical work.  The Electrical Workers Registration Board (EWRB) created a new version of the CESC using Adobe PDF, but their PDF made completing a CESC more onerous than the old manual method, because I had no way of automatically populating the PDF's fields and typing is much slower than writing.

I felt that unless this new electronic method could deliver the following benefits there was no point having it:

1.  The form had to be much quicker to fill out than before and avoid any re-typing of existing information.
2.  The form had to be easy to store and retrieve in future with no chance of it being misplaced or the information originally supplied on the form being changed accidentally.
3. The form had to be capable of being both emailed and printed.
4. The form template had to be able to be edited in future as the legislation and/or my business changed.
5. The form had to be able to go multi-page if the certified design for the installation was large.

The EWRB PDF form delivered on points 2 and 3, but failed miserably on points 1, 4 and 5.

Based on my previous efforts with Classmaker and Microsoft Word mail merge, it seemed to me that I could deliver on all five points above by doing the following:

1.  Recreate the EWRB CESC form in Microsoft Word and pre-populate all the fields that will never change in the Word document, including my signature.
2.  Use some of the fields in Classmaker as fields on the CESC form.
3.  Print the Lesson Plan in Classmaker to HTML file.
4.  Use mail merge to populate the CESC form with variable data from the HTML file.
5.  Print the CESC form as a PDF.
6.  Save the PDF back into Classmaker as a file attachment.

Once I decided to recreate the EWRB CESC form from scratch in Microsoft Word, the rest of the process proceeded quite smoothly and after a morning's work I had a presentable CESC.

Instead of typing out a CESC all I have to do is cut and paste a customer's address and contact details from GNUCash, my accounting package, into Classmaker. Nothing else is duplicated and due to my CESC form's ability to go multi-page the certified design description can be much more detailed than is possible on the EWRB PDF.

Once again Classmaker has proved to be a decent jack of all trades due to:

1.  It's ability to write out it's reports underlying recordsets to HTML tables, one table per report.  The HTML table retains CRLF's but removes all other formatting enabling me to cleanly separate data from presentation.
2.  It's ability to store file attachments inside the database. I've only got to worry about backing up one Classmaker database file for my business.  Everything I need is stored in there.
3.  In the future, if I have to retrieve a CESC, it will be no problem.  Either I type in any part of the customer's address as a search key inside Classmaker, or I export the whole lot to disk and use Windows Search to find what I want.
4.  Having multiple users inside the same database with different subjects for each.  Decided to create a new user from today for my electronic CESC's.  Having many different users is no hardship, because Classmaker remembers the id of the last one you used and opens up with those settings when you restart it.
5.  Classmaker's automatic information aging using it's calendar screens, you only see the current information you are working on - older information automatically disappears.

Lost in Database

We have a problem Houston!  I want to find a record inside my database, but I'm not exactly sure what that record is. The record I'm looking for might be plain text inside a table field, but it could also be in some other format eg. a spreadsheet. I'm afraid the best I can do to begin my search is to provide you with a two consecutive letters eg. "sa".  From here we will narrow the search iteratively.

Faced with a problem like this most database systems can't cope.  That's because all database types, regardless of their design, trade-off searchability vs maintainability.  Let's look at this question for the three main database types.

Navigational (hierarchical and network)

Navigational databases comprise linked lists written to disk.  They generally have a root entry point and from here searches are conducted down branches and a list of addresses is returned that meet the criteria.  Used with indexes, searching navigational databases is very fast.  Their major downfall is that they have no pre-defined structure, so cannot be searched using a computationally complete query language.  Instead a domain specific query language must be created for each database.

Object-oriented

Object-oriented databases are really navigational databases dressed in sheep's clothing. Their main point of difference from navigational databases, is that they can store any type of data because the instructions required to interpret that data are stored in the same object as the data. Consequently, object-oriented databases are very good at storing multi-media files.  However, just like navigational databases, object-oriented databases cannot be searched using a computationally complete query language. They also grow very large quickly as the instructions required to interpret their data is replicated unnecessarily many times over in each object instance.

Relational

Relational databases use a quite different approach to data storage than navigational and object-oriented databases.  Instead of linked lists, relational databases use a collection of tables. The tables are not connected together directly as a linked list is.  Instead, the linkage between tables is an indirect one, where the id field of one table is stored as a field in another table and referred to as a foreign key.  The advantage that relational databases have over the other databases is that the query language used to join these tables (relations) together is computationally complete. That's why it's called SQL (STRUCTURED query language) folks! SQL is not a domain specific. It can be used on any relational database, regardless of it's design. Unfortunately for us, the table structure found in relational databases is not suited to binary data such as our spreadsheet, since SQL is optimised to traverse tables with fixed length fields.

Summary

While navigational databases and object-oriented databases can return search results quickly, to do so they use domain specific query languages.  In other words they trade-off maintainability for searchability.  Conversely, relational databases are slower, but anyone can query them using SQL, their universal query language.  Having SQL is a real bonus for maintainability, but it comes at the cost of not being suitable for manipulating binary data.

Solution

How does any of this help me to find my lost record?  Neither navigational databases nor relational databases are very promising. At face value, object-oriented databases look like they'll do the job, providing they have the correct code available to search through the spreadsheets stored inside them. The truth is they don't. It can be done, but we are back to that maintainability problem again.

BUT, through a happy coincidence with Classmaker, we get the best of both worlds, because when I implemented Classmaker I chose to structure the data hierarchically inside a relational database.  Most planning packages use tags to aggregate similar records together. This produces a similar effect to a network database where there are multiple routes to the same record.  I chose instead, to arrange my records hierarchically with Long Term plans above Unit plans above Lesson plans and Binary file attachments. This hierarchical structure means that while Classmaker can't slice and dice it's database like one with tags could, it was not a hard job to EXPORT the entire database structure to disk, because the directory structure on disk is hierarchical too. There's a further problem with tags. If you don't attach the correct tags to your records, you're NEVER going to locate them!

Windows Search

Windows Search is an interesting facility.  It comes with the IFilter plug-in. This is an API that lets third party binary file formats expose themselves to Windows Search for indexing. Using Windows Search we can look for all spreadsheets, or in fact, any file that contains the phrase "sa".

Conclusion

Classmaker uses Windows Search indirectly to let you examine every record in it's database whether plain text  or binary format for the phrase "sa". Simply export all Classmaker's data to your local Documents directory.  Windows Search automatically indexes all files written there.

I have used Windows Search many times since I discovered it.  It's great knowing that no matter what I load into Classmaker, I will always be able to retrieve it again.  No more lost records.  No more calls to Houston!

IsectMP

IsectMP is a middleware multiplexer written in ANSI C that uses blocking RPC's in a single threaded select loop. The goal of IsectMP is to solve the client/server concurrency problem using collections of identical processes rather than threads with the following constraints:

1.    Keep the API as simple as possible to promote language agnosity.
2.    Impose no record structure on the data being exchanged.
3.    Have all data transfer move through a central broker (Isectd).
4.    Be able to stop and start remote processes via Isectd.

1.    IsectMP only wants to solve the Request/Response model.  This is the classic client/server problem where you have a single static server and many clients that come and go.
2.    IsectMP deliberately uses blocking I/O on a single thread in Isectd.  It solves concurrency by breaking down responses from workers into a stream of discrete chunks which must be concatenated at their destination. Requests from clients can be chopped up too, but this is unusual when dealing with databases as requests are usually small while responses can be very large.
3.    IsectMP deliberately forces all communication to pass through Isectd.  While some may argue this is creating a bottleneck, the advantage of having a central broker is that administrators know the health of the entire application by querying Isectd's registers.
4.    Isectd comes supplied with a command language that lets you adjust its state dynamically.
5.    IsectMP comes supplied with a parent daemon (Execd) that can start child processes locally from an instruction sent remotely via Isectd.

It turns out there are lots of other open source libraries out there doing similar things, but they all have different goals in mind from IsectMP.  I freely admit to only knowing IsectMP well, but from what I can see, if you stick with open source:

1.    IsectMP is the only one that uses ANSI C and compiles on Windows without issue.
2.    IsectMP deliberately uses a central broker.  The others frequently try to avoid using a central broker.
3.    IsectMP has the smallest API of any, just five function calls.
4.    IsectMP comes supplied with its own console client that can configure and control the central broker remotely.
5.    IsectMP's two daemons are tiny, the largest daemon weighs in at < 250kB including DLL's.

Certainly IsectMP has it's failings:

1.    It is not asynchronous.
2.    It will only cope with hundreds of clients, since each client gets it's own dedicated socket.
3.    The only transport mechanism it can use is TCP.
4.    It is single threaded.
5.    It's API relies on a C struct which causes headaches when creating new language bindings (2021-08-22  This is no longer true. A lot of reworking to the internals of Isectd has been done since this post was written).

I would not attempt to use IsectMP for an enterprise size application, but for a department size application, IsectMP works very well indeed (2021-08-22 It also does a grand job acting as the middleware behind web sites).

Spreadsheet madness

Today, a large business (X) accidentally emailed me all their clients' data inside a spreadsheet. This is not the first time this has happened in New Zealand.  Recently EQC, a government department, emailed a spreadsheet containing thousands of clients claim records to one client.

Apparently, X sends spreadsheets routinely to it's staff for further manipulation. Why?

For X, I believe, it's because they've purchased a number of other smaller businesses recently and the small business units' systems are not integrated with X's centralised accounting system. The small businesses units aggregate their monthly figures into a spreadsheet which they then email to X's receivables department.

The problem X has, is that its small business units are aggregating their data too early in the process.  As soon as data is aggregated it is at risk of being divulged.  I contend that the solution to their problem is that the data aggregation should be delayed until just before it is posted to the centralised accounting system.  In fact, I suspect there is no value to aggregating it at all.  Most accounting systems require data to be keyed into them manually and I imagine X's accounting system is no different.

What X really needs, is for its small business units to be posting their individual transactions as they are created onto a searchable queue that can automatically be purged once the data is committed inside the accounting system.

X is partway there already since it's email system is already a queue.  Problem is, that queue isn't secure and isn't visible to all its relevant staff.  What they need is a single email address that multiple internal users can subscribe to, but not a message list, rather an issue list.  There are plenty of these about - we usually call them bug tracking software.  The one that I'm most familiar with is Roundup Issue Tracker. There are many others.

And what content gets posted to Roundup? The spreadsheet sent to me would be fine as long as it contained just one client's details, but I still don't like spreadsheets much, because they bundle data, business rules and presentation together in a single file. I would prefer instead to use a static Web browser application for the reasons outlined in my post called Real world Web browser development.

Using Roundup to avoid information overload

Roundup Issue Tracker helps prevent information overload by:

1.    Stopping ALL spam.  If an addressee is not known to Roundup the email is silently dropped.
2.    Only privileged users can create a new issue, so if an existing external user's email account is hacked, they still cannot spam you with a new issues.
3.    All superfluous email signatures and graphics are purged, just the plain text component and file attachments are retained.
4.    If a message gets through that does contain superfluous material, you can edit the message directly on disk or retire the message so it is no longer visible.
5.    As soon as you decide that a message thread is no longer current you simply retire it.  While it remains visible to searches, it no longer appears on your list of current conversations.
6.    A number of different views of your issues can be saved.
7.    File attachments aren't retained with the individual message, they are retained at the email conversation level.

These Roundup features mean that:

1.    The excessive broadcaster.
2.    The quote everything replier.
3.    The spammer.

can be stopped in their tracks.  The one problem that Roundup can't solve is the file attachment abuser.  Frequently the excessive broadcaster and the file attachment abuser are one and the same, so by preventing excessive broadcasting, hopefully file attachment abuse will decline too.