FORTH and the Rosetta Code Challenge

The other day I came across this Rosetta Code challenge and thought that seems simple enough for me to have a go at... using RobsForth! Perhaps, I misunderstood what the challenge was really about? As far as I understood it, most languages don't have (COBOL does) an IF conditional that can be fed two truth tests and will output:

1. Both tests are True
2. The first test is True 
3. The second test is True
4. Neither test is True

The challenge was to reformulate the typical solution using nested IF's  given below into a function called IF2 which can be used instead of the nested IF's anywhere in your code.

There were lots of languages on display, quite a few of them esoteric one-off's like RobsForth, as well as the usual suspects like C++, Java and Python. I was surprised how complex some of the solutions were. Some submitters provided their function without a test suite, but we all know how easy it is to introduce a logic error, even on the simplest code.

Rosetta Code's introduction assured me that Lisp and Forth automatically support language extension, so I figured this shouldn't be a problem 😆 and away I went. My first effort was too complex, but eventually I pared everything down and ended up with:

# cond1 cond2 -- int
: if2
	IF IF 3 exit THEN 2 exit THEN
	IF 1 exit THEN
	0
;

a Forth word that eschews the ELSE clause.  Forth, of course, is a stack based language and while RobsForth does have global variables, it doesn't have local variables and so everything you see below relies on the stack to store state.

"""
	if2 
	Rosetta Code challenge 
	https://rosettacode.org/wiki/Extend_your_language

if (condition1isTrue) {
     if (condition2isTrue)
        bothConditionsAreTrue();
     else
        firstConditionIsTrue();
  }
  else if (condition2isTrue)
     secondConditionIsTrue();
  else
     noConditionIsTrue();

if2 (condition1isTrue) (condition2isTrue)
     bothConditionsAreTrue();
  else1
     firstConditionIsTrue();
  else2
     secondConditionIsTrue();
  else
     noConditionIsTrue();
"""

• Anything between """ and """ is a multi-line comment
• # is a single line comment

# ----------------- if2 Definition ------------------------

# cond1 cond2 -- int
: if2
	IF IF 3 exit THEN 2 exit THEN
	IF 1 exit THEN
	0
;

# ----------------- if2 Test ------------------------------

# int -- stdout
: branch
	dup
	3 = IF drop |+ ' is divisible by 2 & 3 ' +| exit THEN
	dup
	2 = IF drop |+ ' is divisible by 3 ' +| exit THEN
	dup
	1 = IF drop |+ ' is divisible by 2 ' +| exit THEN
	drop
	|+ ' is not divisible by 2 or 3 ' +|
;

The above word, called branch, demonstrates much of what RobsForth is about:

• IF consumes a truth value, where 0 is false and anything else is true.  The truth value is supplied by the stack.
• Words between |+ & +| are prepended to the words queue and executed immediately, before returning to continue executing the compiled word.
• Words between ' & ' are printed to std out.  Inside a compiled word they must be surrounded by |+ & +| otherwise ' & '  would consume the words queue AFTER this compiled word.
• exit jumps out of a compiled word.  It turns a sequence of IF statements into a Case statement, rather than the deep nesting of IF ELSE's that would otherwise be required e.g.
  IF ELSE
  IF ELSE
  IF ELSE
  THEN
  THEN
  THEN

# --
14 0 BEGIN
   dup dup dup dup dup . space . 
   3 / swap 3 / int - 0 = pob
   2 / swap 2 / int - 0 = pob
   top top if2 branch /n
next WHILE drop drop

RobsForth doesn't have to compile words before using them. Code can just be a script as above. The only native conditional words in RobsForth are:

• IF ELSE THEN where IF is really Jump==0 while ELSE is Jump and THEN is a jump marker for IF or ELSE.
  
• BEGIN WHILE where WHILE is really Jump!=0 and BEGIN is its jump marker.

RobsForth uses a deque rather than a stack and pob and top are moving items from the top of the deque to the bottom and vice versa. Consequently, RobsForth has four easily accessible local variable slots rather than a typical Forth's two.  RobsForth does not implement complicated stack manipulators like pick, roll, rot, nip, tuck, etc, etc... just:

• dup, swap, over, drop at the top of the deque.
  
• dob, sop, oob, bop, pob, top, beep on the bottom of the deque. beep is equivalent to peek, but there is no peek in Forth, because Forth words automatically consume the stack when they are invoked and may leave items on the stack when they exit. This implied use of the stack for local variables removes the need to explicitly set and get local variables. The stack is an interesting structure:
• Variables on it are transient globals whose lifespan is determined by their depth on the stack. In most languages local variables must be explicitly passed by value to another function. In Forth, variable creation, passing and destruction is implied.
• The state of the program can be inferred by looking at the stack because it contains all local variables. RobsForth also stores true global variables on it's heap which live for the duration of the script, but whenever possible, I try to avoid using heap variables.
• Because RobsForth has no local variables, refactoring is straight forward.  Simply extract a sequence of words from a word and call them a new word. The key to refactoring is to have word names that accurately describe what that word is doing. If this is done then RobsForth becomes a domain specific language and understanding code means comprehending the major words in the task's context.

# ----------------- if2 Results ----------------------------

<!-- Forth --> @test.fr
0 is divisible by 2 & 3
1 is not divisible by 2 or 3
2 is divisible by 2
3 is divisible by 3
4 is divisible by 2
5 is not divisible by 2 or 3
6 is divisible by 2 & 3
7 is not divisible by 2 or 3
8 is divisible by 2
9 is divisible by 3
10 is divisible by 2
11 is not divisible by 2 or 3
12 is divisible by 2 & 3
13 is not divisible by 2 or 3
<!-- Forth --> 

I hope this gives you an idea of what RobsForth looks like. It's just a stream of white space separated tokens, which I have formatted, so that they are easy to look at.

A key objective with RobsForth is to keep software development simple. Below are three acronyms the industry uses in an attempt to keep software simple.

• KISS: Keep it simple stupid (”Write the code in the simplest way you can think of. Refactor constantly.”)
• YAGNI: You ain’t gonna need it (”Don’t write the code if you don’t need it now”)
• DRY: Don’t repeat yourself (”Do not copy-paste”)
  

Chuck Moore, the inventor of Forth, was a firm believer in KISS and YAGNI (DRY was probably so obvious to him, that he never bothered to mention it!) and I think the following acronym too.

• MS: Minimise syntax (”Avoid languages with excessive syntax”)
• Since using RobsForth, I no longer like languages which expect syntax. They're too busy which makes it harder to concentrate on the problem at hand.  White space syntax languages are the worst, which is ironic, since RobsForth is implemented using Python!
  

Go FORTH and build your own language!

FORTH is a wonderfully simple and compact programming language. Take a look at Rosetta Code Language Comparison.  Nearly every column on their table for FORTH is either... N/A, None or No!  I laughed at the column about the language being standardised (ANSI, ISO/IEC 15145:1997). Nothing could be further from the truth. At the beginning, a vanilla FORTH doesn't even have variables! You make them yourself by coding the word var (or whatever you think a variable should be called) using

 : var create 1 allot ; which is a compile time word 

every time the word var is invoked at run-time, create looks at the next word in the word list, let's call it counter for this example, assigns it a dictionary key, saves the current heap slot along with the instruction to push the heap slot to the data stack as a dictionary value and finally, advances to a new slot on the heap. So next time you call the word counter (not var. It's done what it needed to do), FORTH looks up the dictionary, and pushes it's heap slot to the data stack. Now you can retrieve the contents of the heap slot, using the built-in word @ or write a new value to the heap slot using the built-in word !

You're probably lost already, but this isn't a lesson in how to use FORTH. I cut my teeth on FORTH written in Python and highly recommend it. Instead I want to posit my pearl of FORTH. Somebody said somewhere that the pearl of Forth is it's create does> word pair, which in brief, let's you pair a compile time action (create) with a run time action (does>). This word pairing let's a user create their own language on top of FORTH. In the past people have created Object Oriented FORTHS and even BASIC. However, this is just syntactic sugar. I've discovered quite early in my FORTH journey that it's better to stay as close to the machine (Python virtual machine in my case) as you can.

No... for me the pearl of FORTH is that anybody can write their own FORTH and consequently there are hundreds of them out there. Few of them follow the FORTH standard. 

The reason you can write your own FORTH is due to it's very simple compile / interpret loop. FORTH doesn't have a compiler in the standard sense. Instead it has compiling words like var above. In FORTH there is no syntax, just a stream of white space separated tokens called words which are consumed in a single pass. The compile / interpret loop works as follows:

1.    We begin in interpret mode. If the word presented is a : we enter compile mode.  Otherwise consume the word and execute it immediately.

2.    If we are in compile mode, consume each word, compile it, but don't execute it.  Instead once ; is reached, store the compiled pcode in the dictionary for future execution, when the word we have just made (var above) is called again.

3.    Go back to 1.

A number of implications flow from this simple pcode compiler:

because the the words themselves act as switches to turn the compiler on and off, compiled words can be redefined further down the word list. Word (function) redefinition is the way a FORTH program deals with a function needing to do slightly different things based on it's context (function overloading in C++).

anybody can change how a FORTH compiler behaves by creating new built-in words that interact directly with the compile / interpret loop in a similar way as : and ; do.

a compiled word is a list of built-in words and perhaps other compiled words. At run-time those words call each other in sequence. My FORTH, like a lot of them, uses the concept of indirect threaded code where the compiled words are a list of function calls in a Python list stored in a dictionary. To execute each underlying Python function the FORTH word (key) calls the dictionary, which then calls the Python built-in function's memory address (value) which it looked up and stored during the compile process. In direct threaded code each function would call the next, similar to a linked list. The advantage of indirect threading, is that the built-in function doesn't need to know anything about calling or being called and so:

Compiled words can be defined that contain words that haven't been defined yet, as long as all words have been defined by execution time. 

Recursion is possible. 

All of FORTH's built-in tokens including the compiler control tokens : and ; can be renamed by storing them as constants. 

the words list is consumed as it is compiled / interpreted. This makes it easy to add new words to the words list at run-time, just create a word that reads in a new file and adds it's contents to the top of the words list. This lessens the need for namespaces as words are only introduced when needed rather than being present when the program loads.

you can increase the performance of your code by treating the word list as your string storage, something you would never do in a compiled language. This works particularly well when you want to print out large swaths of HTML which, just like FORTH, has no syntax. I put it all in my FORTH file (which is actually a collection of database records) and use the FORTH interpreter to print it directly to standard output (std out), bypassing the stack and heap. Want to store the string on the data stack instead (SQL queries come to mind for that)? Same approach as for std out except the word is different that puts it on the stack.

being able to poke around in the compiler gives you a very good understanding of how your FORTH works and the result is that you work to it's strengths and avoid it's weaknesses. The strengths I have noticed so far:

• the code base is very malleable (the refactoring everybody talks about) and it is easy to change things so that you can re-use code e.g. printing a table is commonplace so I have standard table printing code where I redefine what can happen before, in and after a table cell. Now it's easy to print out HTML, tab delimited format, CSV or whatever.
•  no types. If you want a type you have to explicitly cast it. Using HTML its all strings, so most of the time I don't want to be fooling around with types anyway. The only ones I've implemented so far are str and int. The other day I wanted a currency type. I found the appropriate Javascript and dropped my number between it. A quick and dirty solution that I wouldn't want to use for a table with hundreds of values, but formatting variables by surrounding them with FORTH string constants is standard practice for me.
• avoid local variables because they make it difficult to refactor. A web app is a collection of independent pages, so the chances of global variables standing on each other is reduced. I have a number of predefined global variables which I use over and over. Then I came across the concept of shadowing. A FORTH variable is really an array of one (1 allot).

          var counter

1 allot

You now have a variable with two slots.  To refer to the first slot you simply call it by its name:

counter

counter 1 +

gets you the second slot.

This shadowing concept means you can store a backup of your variable in the same variable name. 

The Weaknesses 

• being able to have local variables would be nice occasionally 😀. Stack twiddling is not useful work. My solution has been to turn the stack into a deque and to use the bottom of the stack to store one or two local variables. Many FORTHS use the top of the control stack to store a local variable. I haven't tried that, preferring the deque concept instead because you don't have to monitor it so closely. If I'm iterating through a table there's going to be a lot of values on the stack, so there's no chance the bottom value will be accessed from above. I've added a bottom peek instruction which works well, but would be a disaster at the top, since FORTH words almost without exception pop from and push to the stack.
• it's very easy to write inscrutable code. To address this, I indent my code following the typical conventions e.g. nested if statements. Plastering the code with comments is standard practice in other languages. Instead I try to use words that make sense when being read from left to right. Done right you end up with statements that intuitively make sense e.g. empty counter ! means put a '' into the counter variable and store it on the heap. Traditional FORTHS have a lot of one and two character words like @ ! . , : ; etc. I've tried to avoid those except for the most common ones that everyone knows. If I'm reimplementing a Python function I use the same name that Python has. I don't eschew comments completely. A longer explanation of what the code does is placed in the first record, although I've found the need for commenting is not that critical because the unit name and database record name combination gives you plenty of clues as to what the code is doing. Despite these things, it is not easy to read FORTH code. You have to be in the "zone" which requires time. C code is much easier to understand when you first see it.

A FORTH written in C would remain a toy without many many man hours, but that's not what I'm doing here. I'm using FORTH as a way to simplify Python down to doing one thing... emitting scripts to std out or to file. Creating those scripts is faster than doing them directly in Python, because FORTH doesn't do any syntax checking (Python's tab/space syntax checking drives me mad!) and FORTH never crashes. It just emits the Python error and remains in it's REPL (Read Evaluate Print Loop). The scripts are typically SQL, HTML, CSS and Javascript, but I could  target any tool that is scriptable. LaTEX and Python are likely to be targets in future.

 

Storing Your Code in a Database 2

Now that I've been working on my Web application for a while I've realised a few things. 

• I've essentially created an Access database, but using a Web front-end. In Access the code is divided into collections e.g. forms, queries, reports etc. When you create an application you create the individual components and then stitch them all together. I'm doing the same thing, but have yet to determine the most effective collections to have.
• I'm using my own traditional PC app (Classmaker) as my IDE. This is much faster to use than a Web IDE. It also means I can keep the same Web page up and simply refresh it to see my changes. I could code directly against a Cloud based code database using SSH tunneling as Classmaker uses Isectd to communicate with it's database. 
• Because I'm using a code database via Classmaker, its relatively easy to move code components around and I'm doing that a lot at the moment. I think once the app is mature, I will have a number of collections each containing scores of components that I call from a few Web forms. The Web forms themselves probably won't have much code in them.
• Rolling back changes to code is easy to do in a database. I just have a flag attached to the record. To make changes in a live environment I would duplicate the record, but with the flag set. If the flag is set that record is not selected for parsing.
• I'm using FORTH as my application language. It's my own implementation written on top of Python and it's optimised for working with disconnected recordsets and strings. FORTH is a tricky language which requires you to be in the "zone" to be productive. Because I'm architecting my development environment as I go (eating my own dog food!), not a lot of progress towards the app is happening yet. I'm hopeful the gains on my investment will occur later.
• FORTH does have several advantages though:
• It has no syntax. This is nice. I can just write the code and format it any way I want to.
• My FORTH doesn't have local variables (a discussion about the need for local variables deserves it's own blog post 😏), just the stack and global variables. This means it is a great language, maybe the best there is... for code refactoring. Combine this with a database and it's a simple task to bundle some repetitive code into a word and shift it to it's own record.
• FORTH is fast. You wouldn't think so, being an interpreted language written on top of an interpreted language, but...
•  The language core (both Python and FORTH) is compiled into pcode (assembly language for the Python virtual machine) and stored in memory, when the web server is started, so when you call a core compiled FORTH word you're calling a memory resident subroutine. The best analogy for what I'm doing is Apache's modPy, except I'm using Isectd to do it, not Apache and it's modFORTH rather than modPy. Apache thinks it's calling a CGI program, so I could use any CGI web server. The result would be the same.
• In FORTH, code is compiled/executed in a single pass. The word list is examined and if a subroutine is to be compiled that happens and the compiled code branch memory address is inserted into the dictionary. Further down the word list if that word appears the dictionary is looked up and the compiled word is executed. Whenever I branch to a database record it's word list is inserted into the trunk word list at that point and execution continues, so the database code is parsed only as it is called. JIT compilation/execution? Whatever you call it, no code is prematurely compiled apart from the language core.
• When using C++ or even Python it can be very difficult to work out just where an error has come from and what it is. In a database this can be narrowed down to a specific record. The smaller the record the less code to go through. With FORTH it's easier as you can locate in the word list the exact word the code failed on. This means that syntax errors can be immediately identified. Logic errors are more tricky, since they may originate earlier in the code base. I've found that most of the Python errors that are raised from a FORTH logic error are meaningless. It gets worse! If you've created a word that comprises many other words and that word should fail, you know it failed, but you don't know which subword caused the problem, because the subwords were compiled much earlier in the program. The lesson I'm learning is.. don't prematurely refactor. At least you can dump the contents of the stack (essentially all your local variables) at any point in your code to see what's going on. I tend to work back from the bug, inserting dump statements at random points in the code.

Storing Your Code in a Database 1

I've always been concerned about how to handle different versions of the same code for different customers. I've seen several different approaches over the years. 

The one that seemed to work best involved code stored in a Pick database. At my first job, we used Advanced Revelation, a DOS version of Pick. The Pick system mixed code files written in Pick Basic and multi-value data files together in a hashed database. We used to take a global source code file and append an asterisk and the customers name to the end of it when a customisation was required e.g. accounts*customerA, accounts*customerB etc. Code management modules were written that compiled accounts*customerA into the global accounts executable and then bundled all the executable records together into an archive that the customer extracted onto their system. Another department at that same work place were developing a C++ app, storing the code in standard text files. They got into all sorts of trouble handling variations, mainly because, I think, when developing using text files, to avoid clutter you tend to let the text files grow too large and then it becomes very difficult to refactor them. 

I've seen a FoxPro app where most of the app is standard, but you were able to request minor modifications e.g. customised reports. How this was handled at the vendor end, I'm not sure, but I did notice fields in the database that my customer didn't use.

Another approach is to have a single code base, but include an ini file that let's different customers switch on and off the functionality they need. Ini files used like this can grow to be very large e.g. Apache Web Server. I think the ini file approach is going to become unworkable as the code base becomes increasingly convoluted trying to account for every possibility.

I have no experience doing this, but you could build a conventional app using text files and then store these inside a versioning database e.g. Fossil

Today we have hosted web apps. I don't think much has changed except that you no longer have the upgrade pain that came with distributed apps. I'm guessing the standard approach is to build an app in a CMS and to associate code files and database records with registered users. So you end up with one huge CMS app and one large relational database. The database is a concern, because over time it is going to end up with lots of redundant fields from customisations. You must be passing some kind of unique token between the server and the client to ensure that the client can't trespass onto someone else's data and if that token gets hacked, the hacker might be able to gain access to your entire customer base! To avoid that database engineers resort to using GUIDs to identify records, so it is virtually impossible to pull up a database record using a random key.

I'm working on a different approach.

• Each customer has their own database for data. Consequently you can keep it simple with integer id's and integer foreign keys instead of a mess of GUIDs which as well as being confusing massively impact on performance as a index key. The database server manages multiple small databases.
• Their code, though, resides in a SINGLE database for code. The code database is organised like a library with shelves, books and chapters (pages exist too but these are called indirectly). A shelf is an entire application which comprises many books, but the customer directly accesses just four books, a private book (administration), a public book (customer web pages that can be used by anyone), a protected book (customer web pages that can only be accessed by their registered users) and a global book. The customer books contain only the chapters which are customised for them. Every time a client requests a web page, the customer books are browsed first. If the book's chapter is missing then the chapter is sourced from the global book instead.
• It's impossible for a customer to request another customer's book because to read their book you have to be standing right in front of it. This is pushing the analogy a bit far! I have just ONE very simple cgi file (all it does is collect cgi parameters and cookies, sends them to the code database and returns the reply to the web browser) that calls the code database, but there are MANY COPIES of that file in a shallow directory tree. Each copy has it's own ini file which specifies the name of the book it can access and which database. None of that information passes over the Internet. Access to the cgi files is password protected where necessary (using SSL encrypted basic authentication).

My 64bit Windows 7 Software Ecosystem

The tools listed below are where I spend most of my work time, ordered from most used to least:

Daily for time sheets, checking bank statements and general emails
Accounting - GNUCash
Web browsing - Mozilla Firefox
Email - Mozilla Thunderbird

Twice weekly for invoicing
Planning - Classmaker
Mail Merge reports - Microsoft Word 2003 (the data source is an HTML file emitted by Classmaker)
PDF Creation - Print to CutePDF

Once a week
Backup - Classmaker
Off-site backup - Google Drive

Once a fortnight
Scanning - Windows Paint
Creating multipage PDF's from scanned images - IrfanView
Spreadsheet - LibreOffice Calc
Word Processor - LibreOffice Writer

Once in a blue moon!
Chart creation - LibreOffice Draw
Desktop publishing - Scribus

When things go horribly wrong!
Puppy Linux Tahr 6.0 on a USB stick

•  Puppy has saved me several times over the years when Windows 7 hasn't booted or file corruptions have occurred.
• The most recent incident involved OLE embedding a LibreOffice Draw file inside a LibreOffice Writer document.  Don't do this!  That technology is unstable and you will find yourself locked out of the Writer document as somehow ALL the file permissions get removed.  The only way to retrieve the file is to boot into Linux, move the file into a Linux partition and then copy it back.
• Instead of OLE embedding, export your Draw file to disk as a PNG image and then insert that into your Writer document.
  

Safe Python Eval

Many of us want to use a safe simple scripting language to extend our Internet applications and I have seen many discussions on the Internet on how to implement Python so that it's eval function can't be exploited. The problem is, nobody can be sure that they've covered every scenario, because Python has become complex over time, with multiple ways to achieve the same objective.

My scenario is a typical one. I have a web based database application hosted on a remote server. Now I want it's users to access it's database to produce ad hoc reports, without compromising the database or anything else! I could do the typical thing of providing a screen where users can submit Select queries, but SQL by itself isn't enough to get a report that is useful except in the most simple scenarios. To begin with you can only work with one query result at a time. Reports frequently require multiple query results to be combined. Tables of course can be joined using SQL, but the result usually has lots of redundant data fields that must be omitted. A declarative language like SQL just can't do it alone. It needs to be combined with a procedural language. After all, that's how my web application works. The procedural capability of Python is combined with SQL to produce web pages that make sense to users.

After surveying the scripting languages out there, Python came in at number one, BUT, it has an eval function that can access the file system or carry out other security exploits. Vanilla Python isn't a sensible choice as an extension scripting language.

I have looked at FORTH in the past, ATLAST and Ficl were front runners, but they are traditional FORTHs with weak string manipulating capability. Strong string capability is what I need, because my database queries return as nested Python string lists. I came up with that format years ago before JSON became popular, using Python eval to parse the database query, returning a table that Python can iterate over using it's index functions.

Then I came across FORTH written in Python, a toy project for educational purposes. I realised it wasn't going to be fast, because FORTH is an interpreted language written in an interpreted language! What it did have going for it was very strong string capability and the ability to cherry pick the Python functions required, leaving everything else behind. Suddenly, eval wasn't a problem as I could overlay it with some checks to make sure eval could only eval a list. FORTH is a simple typeless stack based language with the entire environment comprising two stacks and a heap (Python lists), a dictionary (Python dictionary) and a heap pointer (Python integer). FORTH has no syntax! The language is just a stream of white space delimited tokens called words. Traditional FORTH has no error checking and crashes are common place. But having Python underneath FORTH  makes it uncrashable. Sure your FORTH code will frequently bomb, but the FORTH REPL (Read Evaluate Print Loop) just displays the Python error and awaits your next command.

I made some alterations to the Python code to get a FORTH that isn't quite FORTH anymore (as the saying goes 'when you've seen one FORTH - you've seen one FORTH')! Everything became a string, both the stack and the heap able to handle strings or lists of any size in addition to numbers. Actually Python is cleverer than that. Lists can store variables of any type, so my database lists stay as lists (under the hood, they're probably C arrays) which are passed by reference rather than value. Numbers stay as numbers and dates as dates unless they are cast explicitly, something I've brought over from Python. But apart from that the toy project has stayed largely untouched. All it can do is step, branch, loop, create variables, read variables and print to the console. But at just over 300 lines of Python, it's Turing complete and this is all you need to create a report. Already I have written a complex report utilising two tables that cannot be joined, the report output is styled HTML.

Perhaps the most interesting thing I've done is to create a persistant library of Forth definitions. The way I've done that is to store the FORTH environment after loading forth.py (the REPL) and forth.fr (the persistant library) as Python command line arguments. The working code FORTH file is then called manually in the REPL. Each time it is called the FORTH environment is restored to the finish of forth.fr. This means you can call the working code file over and over without having to restart the REPL.

But there's still a problem!  FORTH over Python might be safe, but a user could write a function that never returns. Endless loops are easy to create inadvertently in procedural languages. Code needs to be added to Isectd that checks how long a worker has been busy for. If it has been busy for longer than 20 seconds and has no clients attached then Isectd sends a kill command to the process which will automatically restart after it has been killed.

The working FORTH code is included in PocketClassmaker install file from the downloads area of my website.

Fox-Toolkit

I have been using Fox-Toolkit version 1.3.15 as my development environment for 15 years now. Prior to that I used Visual Basic 5, which I loved. Ever since moving to Fox-Toolkit I've been searching for a modern equivalent to VB5. I've never found it. Over and over, following these programming language journeys, I've concluded that Fox-Toolkit really is a brilliant piece of software engineering, all of it done by one man, Dutch born, Jeroen van der Zijp, beginning in 1997.

Here's why I think it's brilliant:

1. It's just a library written in C++, but Jeroen dislikes the STL and doesn't use it. The resulting language resembles C with inheritance rather than C++. Most of time I don't even use it like you would C or C++ as I have no need to concern myself with advanced concepts like inheritance, pointers and all the rest. Instead, I use it the way you used VB... without an IDE. In VB, you created forms and modules. Most of the code was written as event code associated with a widget and shared code was placed in modules. In Fox it's the same except that the form is just a standard *.cpp text file with Fox's window creation boilerplate and a module is a standard *.cpp text file without Fox's window creation boilerplate.

2. Because it's written in C++:

• it's runs extremely fast
• providing a project is set up correctly, compile times are quick, because most of the code is already compiled previously and only needs to be linked
• any external library written in C or C++ is easily integrated
• calls to the Windows API are a breeze
• I have multiple free compilers at my disposal
• the C syntax is terse and easy to read
• there are many code indenters available to tidy your code
• if I need to, I can program at a systems level, rather than at application level without having to move to another environment.

3. It doesn't use native widgets. Instead it emulates the look and feel of Windows 95, by using operating system primitives for drawing lines, filling spaces and obtaining mouse and keyboard input. This is the best part, because, Fox cannot be broken by a third party, since those primitive API calls are extremely unlikely to change. Also... I like Windows 95! Microsoft keeps tinkering with it's look and feel for marketing reasons, but in truth, nothing substantial has appeared since Windows 95 in terms of GUI usability.

4. It has a very powerful string class called FXString. I use this constantly. Most of my programming revolves around relational databases which I query using my own disconnected recordset object, written in Fox, which parses a JSON like string into linked lists of FXString. Very simple, very fast and rock solid performance.

5. The window geometry manager avoids fixed window sizes, so Fox applications resize themselves sanely. VB didn't come close in this regard.

6. Fox comes with it's own code editor called Adie. Of course, any code editor is viable. Notepad++ is popular. But I like Adie because it's so simple.

7. The widgets collection is huge and with a bit of digging in the code you discover things are available that you don't expect. For example, recently I found that the FXTable widget lets you copy it's contents to the clipboard in tab delimited format. Spreadsheets parse this into columns and rows directly without the usual dialog palaver that comes with a CSV file. Other time savers include INI file creation which comes baked into Fox allowing window positions to be saved on exiting your application.

8.  Being open source with a liberal licence, you have access to all the source code which is reasonably easy to follow, so you can see exactly how things work. The situation has yet to arise, but if a widget doesn't have a feature I want, I can inherit a new widget from it that does exactly what I want.

9. Cross-platform development is the goal of many tools, Fox included. But, I have come to see that real world application programming is about being able to call into an operating system's API to achieve outcomes that aren't clunky workarounds. For instance, I needed to convert rtf to pdf sanely and programmatically. The best way of doing this was to call LibreOffice in headless mode, but that resulted in console windows appearing. I found the appropriate Windows API call to suppress the console windows and substituted it for the normal system() call. While I could have done something similar in other languages it would have been much more difficult to implement. With Fox it was just a bit of cutting and pasting.

Perhaps the biggest benefit of all from using Fox, is that you cannot code yourself into a corner.  So many times you start to use a tool only to find that there is some aspect of it that constrains your development. In 15 years of developing with Fox that is yet to happen. I've produced desktop applications and web sites, just using Fox-Toolkit. Check it out.

Disconnected Recordsets

If you Google the term 'disconnected recordset' the results returned refer to old style Microsoft ADO recordsets which are able to use database cursors, but Microsoft has stopped using that terminology, because their newer ADO.NET recordsets are always disconnected. Database cursors have dropped out of favour because they maintain state on a database, which isn't scalable, particularly on web servers.

Reading some of Microsoft's documentation today surprised me, because ADO.NET's approach to database access is nearly identical to my own hand-crafted disconnected Recordset object, which I came up with nearly 20 years ago, roughly the same time ADO.NET 1.0 was released.

What does my Recordset object do?

1. It parses a tokenised string from the database driver or from a file into a double linked list which is then referred to as a recordset.
2. It contains methods to iterate forwards and backwards through a recordset.
3. It contains methods to retrieve the contents of a field by name or position from the current row in a recordset.
4. It contains date and time methods.
5. It can sort a recordset.
6. It can emit a recordset as a tokenised string.
7. It can write a recordset back to the database in one round trip.
8. All data access is carried out under transaction management.
9. Additional methods can be added to the object as needed, without impacting earlier code.

Why did Microsoft move to disconnected recordsets from cursors?  I have found the following benefits:

1. Multiple recordset objects share the same database connection which is opened automatically when the first query is run.
2. A recordset object can act as a record datatype without using a database.
3. The recordset object hides a lot of the repetitive insert, update, delete queries that come with SQL. SQL queries are still used a lot though. Without in depth knowledge of SQL, you can't use relational databases effectively.
4. As the recordset is a tokenised string it can be written to disk and retrieved again.
5. Transaction management can span multiple recordset objects.

OK, what's this continual banging on about tokenised strings!  It turns out that these are the key feature that makes the whole disconnected recordset thing work.  Here's a tokenised string:

[[{'name':'ID','type':'i','null':0,'width':4},{'name':'DESCRIPTION','type':'s','null':1,'width':30},{'name':'REORDER','type':'i','null':0,'width':4}],
[[          5,'Moved',          5],
[         10,'Critical',         10],
[         20,'Urgent',         20],
[         30,'Normal',         30],
[         40,'Wish',         40]]]

Through a historical accident, it so happens that this tokenised string is also a Python list. Perhaps if I was to reimplement it today I would use JSON which is more widely used. Nevertheless, this format works well. It can be evaluated directly using Python and I have written a parsing routine in C++ that converts it to a double linked list in a single parse.

JSON can exceed two dimensions and use schemas, but the column and row format is what makes my tokenised string so efficient to parse and to manipulate once parsed. Columns and rows are easy to understand and column numbers rather than column names can be used for faster data retrieval. Not that this makes much difference, since a linked list is very fast to iterate over, so I usually use column names instead of column numbers to maintain code legibility.

Looking at the string you can observe that it is restricted to columns and rows, but it isn't a flat file. Instead, this format lies somewhere between a flat file format like dBase which has fixed column widths and is very fast to query using file offsets and CSV files which don't have quite enough information to be useful as a recordset.

Here's another tokenized string:

[[{'name':'ID','type':'i','null':0,'width':4},{'name':'MSGID','type':'i','null':0,'width':4},{'name':'FKISSUES','type':'i','null':0,'width':4},{'name':'FKPEOPLE','type':'i','null':0,'width':4},{'name':'BODY','type':'s','null':1,'width':32765},{'name':'STATUS','type':'i','null':0,'width':4},{'name':'LAST_ACTIVITY','type':'d','null':0,'width':8}],
[[          4,          4,          3,          6,'What would we talk about?',         30,'2018-06-02 15:10:07.0000'],
[          6,          6,          3,          6,'I need to keep going with this software.',         30,'2018-06-07 21:01:40.0000'],
[          7,          7,          3,          6,'Another note...',         30,'2018-06-19 11:16:40.0000']]]

Notice that the field descriptors contain the field type. This is vital for sensible parsing of the recordset. Type inference which is what you have with CSV files can only get you so far.  Knowing the data type of the field beforehand allows for clever data massaging e.g. my recordset object has an optional flag for elapsed time which represents dates as duration from now rather than the raw format you see here.

There's a few things that aren't so great with disconnected recordsets:

1. They can't store binary data. I get around this by converting to Base64 which is slow. Over time, I've come to realise that binary data is much easier to use if it is kept out of a database anyway.
2. How do you deal with deeply nested data? Relational databases don't store data hierarchically.  Instead they use relations between tables. It is possible to store data hierarchically using table relations, although a little tricky (a whole blog post could be devoted to that subject). Recently, I have found the need to store data hierarchically, but I do that using the database tables themselves, so it has no effect on my recordsets.
3. What about shallow nested data? I use Base64 encoding to store recordsets inside recordsets, since none of the Base64 characters match the recordset's delimiters, but I've only encountered one instance where shallow nesting was needed.
4. The parsing routines must escape delimiting characters. I use octals to avoid the escaped representation containing letters which hexadecimal requires.

Technical Details 

• disconnected Recordset client (data view)
• C++ using Fox-Toolkit which has an excellent string class that I use heavily in my projects.
• eval in Python is all you need.
• disconnected Recordset driver (data adapter)
• native driver written in C for the FirebirdSQL relational database.
• disconnected Middleware (connection)
• both the client and server depend on IsectMP to work.  IsectMP is a tiny RPC multi-plexer written in C that I maintain.  It contains the FirebirdSQL driver, but not the C++ client.

What's wrong with CGI?

The accepted answer is that CGI is an outdated concept that no longer has a place in modern web applications.  I'm not so sure.  Let's look at how CGI works.

1.  The user requests a page.
2.  The web server starts up a new process, serves the content and the process dies.
3.  If the page is doing anything interesting that invariably means the CGI process must open a database connection and destroy it again.

In the past, operating systems weren't smart and starting a new process mean't reading the binary image into memory from disk. That's not true now, particularly for Windows (I'm a Windows developer, so I had better stick with what I know); it caches all recently read files in memory. So, if your CGI file is a compiled executable then the overhead of CGI has gone. If your CGI uses a scripting language such as Python, Perl, Tcl etc then their virtual machines must parse your script every time which will slow things down, but not by as much as you'd think, since both the virtual machine and the script are cached in memory and in Python's case at least, the script gets compiled to byte code on the first parse and thereafter runs much quicker.

Opening and destroying database connections is expensive, mainly because the database server must authenticate your connection. But if you insert some middleware in between the database and your CGI file and pool your database connections, the authentication overhead is removed. Ensuring that your database recordsets are disconnected recordsets also makes things run much faster.

So to recap, if you have CGI executables using disconnected recordsets from pooled database connections what's wrong with CGI?  Let's count the advantages:

1.  The CGI executables are tiny.  I have a single EXE for each page, all of them sharing a 2 Mb dll.  The EXE's never exceed 200 Kb. They compile in a flash.
2.  Because each CGI is stand-alone, I can keep the web site running, while just working on one aspect of it. And if a CGI should crash in production, the whole site isn't brought down, just that page.
3.  The site runs exceptionally fast, because the CGI files only do the bare minimum to create a page.  No lumbering template libraries here. In fact, no template libraries at all!  HTTP, when you remove all the towering library abstractions built on top of it, is actually very basic. Just GET and POST calls passing around variables, HTML Forms that collect the data into those variables and cookies to maintain state. It also helps to have a little bit of Javascript sprinkled about so that your Forms don't need to return to the server for every action the user takes. A typical CGI file begins by collecting any variables sent to it, reading cookies and opening a number of disconnected database recordsets.  The variables and cookies are transformed and saved as appropriate and the file finishes by writing out a new page. In my files less than a third of the code is devoted to writing the new page. Most of it revolves around updating the database.
4.  Secure. You can't have a scripting attack, if all the code is compiled into an EXE.

The disadvantages?

1.  My code uses combination of C and C++ supplied in three libraries. First, cgihtml-1.69, a minute CGI library written in C, that I scrounged off the web and which was last updated in 1998. Second, fox-1.3.15 written in C++ and released in 2002 (Fox-Toolkit is still current. The latest release is 1.7.61, but why update when the existing functionality surpasses anything I'll ever need?) and third, isectMP-1.0.3, a database pooling library written in C which I maintain. Fox-Toolkit comes with an excellent String manipulation class which I use constantly, but sometimes I yearn for the simple syntax of Python. That yearning soon disappears when I have to interface with a third party library or interact with Windows directly! And what about Python deployment? Let's not go there!

2.  Writing CGI's in C and C++, just isn't fashionable.  I constantly have to remind myself that it's results that matter, not that I should be using the latest language du jour.

And so, I find myself asking that question again, what is the problem with CGI?  Answer:  Nothing!

PC based Lesson Planning Software Shootout

It seems the writing is on the wall for PC based software.  Just three years ago there were at least five actively maintained lesson planning packages available for Windows users.  Now there are just two, my own PocketClassmaker and Jeff Hellman's Planbook 4. After looking at his Facebook page I'm not even sure that Jeff is still in business, since there are users requesting support and they have not heard from him since July 2015. To be fair, I haven't posted a software update since June 2015, so perhaps we're in the same boat!

Most teachers have probably moved to Web based solutions by now and they do have their advantages, but if you like me, prefer to have control over your own data and aren't constrained by school policy to use their mandated Web solution, then perhaps PC based software might still work for you.

I compared both packages exhaustively for usability, but omitted testing Planbook's PlanbookConnect which is some kind of Web based synchronisation service, since PocketClassmaker can already do that using Google Drive and I didn't want to sign up to PlanbookConnect.

If you can't be bothered to look through my comparison table below here's the Executive Summary:

Both packages cover similar territory, but from different angles which results in a different user experience:

1. Planbook is optimised for rotating lesson plans (create a rotating schedule by subject first and then fill in the detail later) for a single user and cannot be used as a normal calendar. It is relatively painless to get going, but has a number of features which I found very frustrating. These include entering lesson times which is fiddly at best, the startup screen position cannot be saved and  perhaps the biggest frustration, to save your work you must use the Ctrl S key combination constantly or risk losing your planning. There is also a bug that crashes the program when trying to access the Lesson Library. To allow the software to be used on several machines Planbook has a synchronisation function called PlanbookConnect which I didn't test.
2. PocketClassmaker is optimised for ad hoc planning (plan as you go by copying existing lessons forward and amending them to suit) with multiple users and thus can be used for any task that is calendar related, but, the learning curve for installing and using it is steeper than for Planbook. PocketClassmaker is a portable app and is designed to be installed on a USB stick.

 

	PlanBook	PocketClassmaker	Comments
Website	www.hellmansoft.com	www.creditscore.co.nz
Windows	64 bit Windows 10 Home	64 bit Windows 10 Home	PlanBook versions exist for Mac, Windows XP and Ipad. Classmaker only works on Windows XP and up.
Screen size	1366 * 768 on 14 inch screen	1366 * 768 on 14 inch screen
Version	4.0.51	2.0.6	Date downloaded 07 May 2016
Cost	USD30 for one user	Free for one user
Installation	Windows MSI. Only allows installation to C:\Program Files (x86)\Hellmansoft\PlanBook	Extract Zip archive to any location	Classmaker is a portable app.
Google Drive	No	Yes	Only one instance can be active at any time for Google Drive replication to work.
Invoking	Normal menu item	Two batch files located in the Scripts directory.
Multiple users	No	Yes
First invocation	Wizard	Blank screen	Classmaker comes with an example database.
Remembers window positions	No	Yes
Default dates	September 2013 to June 2014	Have to create a new user and a default non-contact lesson plan before the screen displays. Defaults to today’s date.
Subjects	Created using Wizard	Defined at Syndicate Leader level
Time entry	Fiddly input fields that require am/pm	Uses 24 hour clock	PlanBook's approach of using option boxes is disastrous from an inputting perspective. The a.m / p.m selection often fails to take correctly and the time reverts to the initial setting.
Date entry	A lesson can only belong to one day	Lessons can cross multiple days using the same start and end times	It is almost impossible to use PlanBook for add hoc planning as white space is not allowed
Fast Date Entry	Not needed because lessons are always prepopulated using some kind of rotating schedule	Just enter the day for the current month	PlanBook only works with rotating schedules
Database type	Individual files for user defined periods	One database file for all users and forever
Autosave to Database	No	Yes, but required to explicitly add, edit or delete a record	Losing your work after some time doing data entry in PlanBook is VERY frustrating.
Long Term Plan	No. Month display shows Subject name only.	Yes. Term display shows Subject and Unit Name.
Unit Plan	Optional. Used for printing and exporting / importing lessons.	Mandatory.
Permanent Lesson Storage	No. PlanBook crashes!	Yes	PlanBook developer’s main focus appears to be on Apple Mac’s.
Weekly Report	Attempts to include detail	Just prints lesson names
Daily Report	Ugly cross page splits. Includes empty lessons.	Nice table display
Future and Past Lessons	Can’t tell if you’ve created future lessons	Long term display tells you immediately if you have done future planning
Attachments	Attached to Lesson Plans using a link	Attached to Unit Plans and stored in database file
Website links in lesson plans	Yes	No	In Classmaker would have to use HTML tags eg. <a href=http://www.stuff.co.nz>http://www.stuff.co.nz</a> and print to HTML
Report Types	Daily, Weekly, Unit	Daily, Weekly, Term, Unit
Report Formats	Print only	Print, Postscript, HTML	In Classmaker HTML reports are similar to CSV files and can be imported into MS Word or formatted using CSS for display in a Web Browser
Database Export	Yes, but empty days are exported too. Assumed that database will be published to a web site between dates.	Yes. All date and time relationships have been removed. Just the data and file attachments are exported. Assumed data will be globally searched including file attachments.
Search	All fields in database, but not attachments.	Only lesson titles.	Both packages require database export to search attachments.
Lesson Types	Unit	Unit, Routine and Non-Contact
Schedule Types	Same subjects repeated daily, same subjects repeated weekly, rotating schedules up to 14 days	Ad hoc, subjects repeat daily, subjects repeat weekly, rotating schedules up to 30 days
Lesson Movements	Bump, Yank, Lock	+Day, +Week, Bump, Shove, Up, Down
Unit Export	Attachments aren't exported only links	Multiple units can be exported as a batch including attachments
Rotating Lessons	Set up lessons beforehand and then use directly	Use a placeholder unit that must be copied forward again and again
Field Input	On failure reverts to initial figure	On failure input figure retained but not written to database

Does your office have Issues?

1. If your office responds to repeat customer requests that require various staff in your office to carry out a number of time consuming tasks sequentially,  you have issues.
2. If on the other hand your office generates and disseminates information to multiple contacts, you don't have issues.

Many offices have issues with their issues, primarily because they are a type 1 office trying to use type 2 office tools!  A type 1 office needs to have a central email repository within which to store their issues.  Instead, many type 1 offices use standard email clients which are designed for type 2 offices.

As an electrician my office is a type 1 office.  Retailers and marketers have type 2 offices.  For my centralised email repository, I use Roundup Issue Tracker.

In a corporate environment Roundup does three things particularly well:

1. Acts as a email repository that can be easily searched by all staff.  Issues don't get lost in Roundup, because they can never be deleted, just retired.
2. Makes an excellent workflow tool that tracks issues and amends their status as they move through their lifecycle in the organisation.
3. Ensures that customers can easily communicate with staff, but in such a way that inappropriate information never crosses the boundary between the small office and the outside world.

But, to do these things I had to make some changes to Roundup, because Roundup's original target audience is software developers not corporate office environments. To make Roundup a happy camper in your typical small office environment, my version of Roundup now looks like this:

• I distinguish between internal users and external users, since external users only interact with Roundup from their email client while internal users favour its Web interface.  For this distinction we have four types of user:

1. Anonymous - Roundup does not know their email address
2. User - External user
3. Staff - Internal user
4. Admin - Administrator

• The Staff role is a new role that I created.  It is appended to the other roles to obtain the required behaviour eg. (Admin, Staff) or (User, Staff)

• I send out HTML emails which include corporate signatures to external users.  Internal emails remain as plain text.

• By default Roundup only sends emails internally to Staff. To send an email to an external client, Staff have to set a flag on their message.

• If Roundup is to act as an effective repository, it's database must not be polluted with spam. To prevent spam I wrote the following business rules:
• If an email sender is Anonymous their email is bounced to an Administrator.
• If a User (not Staff) is known to Roundup, but they are attempting to create a new issue, their email is bounced to an Administrator.
• In both cases, if the Administrator determines that the email is not spam, they forward it back into Roundup, where it appears as a new issue.

Why I love Units of Work

Not many lesson planning packages use Unit/s of Work (UoW) to organise their lessons. Classmaker uses them for everything! What is a UoW? A UoW is a container that holds a lesson collection. The lesson collection comprises individual lessons and supporting files. UoW are similar to tags, but unlike tags, a lesson cannot exist without belonging to a UoW and a lesson can only belong to one UoW.  Conversely, multiple tags can be assigned to an individual lesson and by default a lesson has no tags.

This graphic shows how the major components of Classmaker relate to each other using a Venn diagram.

At face value, UoW seem too restrictive..., and tags would appear to be a much better bet, but as I will show, this is not the case.

Pros

1.  Because a lesson must belong to a UoW, a UoW can be much more descriptive than a tag and combine operational information with data. For instance, in Classmaker, the UoW has a title, a short description, the lesson type (non-contact, routine, unit plan, rotating), whether the UoW has just been imported or if is marked for export and a list of all the supporting files attached to it. The subject of the lesson is listed in the Long Term Plan (a superset UoW) along with whether you want to display it's lessons on the Weekly Calendar or not. All of this is information that you don't want to have attached directly to individual lesson plans as some of it is operational information rather than data. Even the data may not be required on some lesson reports. But this is information that you want to be able to refer to easily if you need to, which is discussed next in 2.

2.  UoW's are reciprocal.  By this I mean that if you click on a UoW you can easily view all the child records that belong to it.  But this also works the other way around.  If you click on a child record you instantly know the parent UoW that it belongs to as well. Classmaker uses reciprocity extensively in it's user interface. Whenever you click on a lesson plan on the Weekly Calendar, the lesson detail brings up the UoW that lesson plan belongs to, all the other related lessons in the same UoW, the Long Term Plan the UoW belongs to and all the other related UoW in that Long Term Plan. This visual overview of your planning, Classmaker retrieves for you automatically, every time you click on a lesson plan.

3.  It is much easier to work with groups of records rather than one record at a time. UoW's facilitate this since a lesson can only belong to one UoW. When you delete a UoW, everything inside it goes too. Conversely, to create a clone of a UoW, export and import the entire UoW (1, 1a or 1b) via the hard disk.  The UoW keeps track of whether it is flagged to be exported, is waiting to be imported from disk or has recently been imported, rather than having to concern yourself with the state of the individual lessons involved.

4.  Batch jobs have been used in computing for a long time. The advantage of using batches is that small groups of records can be manipulated together, without exposing the database to the risk of all it's structure being changed beyond reversal by a single SQL statement. UoW's are analogous to a batch job. They can be cloned, deleted and their lesson plan dates changed using Shove. If the result is not what is expected, the cloned UoW is deleted.  Classmaker users will discover that they often use UoW's clone, delete and edit functionality to create new records from existing records as it is much easier to clone, delete and edit records than it is to create them from scratch, particularly if you use Rotating Schedules.

5.  A Classmaker user will clone and edit whenever they can.  For instance, say you need to create a new record in the same time slot as today, but for tomorrow in a different subject.  In all the other packages I've tried, your only solution would be to create a new record from scratch.  In Classmaker, you take today's record, push it forward one day (+Day) and then add it into a different Long Term Plan and Unit Plan (2c.), just four mouse-clicks with no typing of dates and times. This works due to UoW's reciprocity, mentioned above in 2.

6.  Hierarchical data structures are simple to export to disk as that is how the file system works. The parent child relationship implicit in UoW mean their existing structure can be written to disk intact. All other packages I've tried, export their lesson plans as flat files destroying their user interface relationships in the process (that's if they've even tried to export their lessons. Many vendors don't bother).

Of course, UoW's aren't all beer and skittles!

 

Cons

1.  A lesson cannot belong to multiple UoW.  With tags this is a no-brainer and their most useful attribute.
2.  You do not know at a glance all the different UoW you've created, but it's easy to see all your tags.
3.  The tags associated with a lesson can easily be changed at any time. It's more difficult to move a lesson plan from one UoW to another.

 

Summary

I prefer UoW to tags because their hierarchical structure readily translates to data hiding, reciprocity in user interfaces, easy exporting and importing, batching of work, cloning and editing records instead of creating new ones and database exporting to disk without destroying parent child relationships.

The Internet is Dead - Long Live the Portable App!

Recently, I converted Classmaker from a standard Windows app to PocketClassmaker, a portable Windows app.  To transform Classmaker into PocketClassmaker, I had to do the following:

1.  Make sure that all directory references inside Classmaker were relative rather than absolute.
2.  Write user settings to a configuration file rather than the Windows Registry.
3.  Change the database from a Windows Service to an embedded database.

Luckily for me, these three changes were quite easy to do.  In particular, the FirebirdSQL database comes in both flavours, server and embedded, and very little modification is needed to move from one to the other.  In my case, the backup batch file had to be altered, but not much else.

The advantages of a portable app over a standard app are:

1.  No installation is necessary. Just extract the application's directory tree somewhere, perhaps onto the user's Desktop initially.
2.  As the portable app name suggests you can carry around the application on a USB stick or use it from Google Drive or Dropbox.
3.  Portable apps don't alter the host PC's environment in any way.  To get rid of them you simply delete the application's directory tree. 

The disadvantages of a portable app over a standard app are:

1.  You have to start them directly yourself instead of Windows automatically starting them for you as a Windows service.  This means the user has to be comfortable using batch files.
2.  As you move between machines, environment settings such as the printer may change.  As you are no longer using the Windows Registry, you have to reset these every time.
3.  Not suitable for a multi-user environment.

Using a portable app confers the following advantages over a Web app:

1.  Much MUCH faster to use.
2.  Always available regardless of whether the Internet is up or down.
3.  No need for user names and passwords.
4.  Much more flexible use of screen real estate.
5.  Exclusive rights to your own data with full access to the local file system.
6.  Updates are simple.  Just copy the latest version over the top of the old one.

Microsoft Word Mail Merge

I no longer actively teach, but I continue to use Classmaker for all sorts of things.  It's been used as a rudimentary cashbook, a holiday diary, for Cub and Scout planning and lately as a document management program in my electrical business. Today it's been successfully implemented as a data source for Microsoft Word's mail merge.

As an electrician, I'm required to supply every customer with a official document called a Certificate of Compliance (CoC) when doing electrical work.  Until recently, these forms have been sold in triplicate books of twenty.  A form was completed manually, whenever new electrical wiring was done on a job and a copy posted to the customer.  The yellow copy of the triplicate had to be retained for three years.

The world moves on and now we are allowed to create electronic versions of this form, but they must be stored for seven years rather than three years. Furthermore the CoC has been renamed to a Compliance and Electrical Safety Certificate (CESC) and one must be supplied for all electrical work.  The Electrical Workers Registration Board (EWRB) created a new version of the CESC using Adobe PDF, but their PDF made completing a CESC more onerous than the old manual method, because I had no way of automatically populating the PDF's fields and typing is much slower than writing.

I felt that unless this new electronic method could deliver the following benefits there was no point having it:

1.  The form had to be much quicker to fill out than before and avoid any re-typing of existing information.
2.  The form had to be easy to store and retrieve in future with no chance of it being misplaced or the information originally supplied on the form being changed accidentally.
3. The form had to be capable of being both emailed and printed.
4. The form template had to be able to be edited in future as the legislation and/or my business changed.
5. The form had to be able to go multi-page if the certified design for the installation was large.

The EWRB PDF form delivered on points 2 and 3, but failed miserably on points 1, 4 and 5.

Based on my previous efforts with Classmaker and Microsoft Word mail merge, it seemed to me that I could deliver on all five points above by doing the following:

1.  Recreate the EWRB CESC form in Microsoft Word and pre-populate all the fields that will never change in the Word document, including my signature.
2.  Use some of the fields in Classmaker as fields on the CESC form.
3.  Print the Lesson Plan in Classmaker to HTML file.
4.  Use mail merge to populate the CESC form with variable data from the HTML file.
5.  Print the CESC form as a PDF.
6.  Save the PDF back into Classmaker as a file attachment.

Once I decided to recreate the EWRB CESC form from scratch in Microsoft Word, the rest of the process proceeded quite smoothly and after a morning's work I had a presentable CESC.

Instead of typing out a CESC all I have to do is cut and paste a customer's address and contact details from GNUCash, my accounting package, into Classmaker. Nothing else is duplicated and due to my CESC form's ability to go multi-page the certified design description can be much more detailed than is possible on the EWRB PDF.

Once again Classmaker has proved to be a decent jack of all trades due to:

1.  It's ability to write out it's reports underlying recordsets to HTML tables, one table per report.  The HTML table retains CRLF's but removes all other formatting enabling me to cleanly separate data from presentation.
2.  It's ability to store file attachments inside the database. I've only got to worry about backing up one Classmaker database file for my business.  Everything I need is stored in there.
3.  In the future, if I have to retrieve a CESC, it will be no problem.  Either I type in any part of the customer's address as a search key inside Classmaker, or I export the whole lot to disk and use Windows Search to find what I want.
4.  Having multiple users inside the same database with different subjects for each.  Decided to create a new user from today for my electronic CESC's.  Having many different users is no hardship, because Classmaker remembers the id of the last one you used and opens up with those settings when you restart it.
5.  Classmaker's automatic information aging using it's calendar screens, you only see the current information you are working on - older information automatically disappears.

Lost in Database

We have a problem Houston!  I want to find a record inside my database, but I'm not exactly sure what that record is. The record I'm looking for might be plain text inside a table field, but it could also be in some other format eg. a spreadsheet. I'm afraid the best I can do to begin my search is to provide you with a two consecutive letters eg. "sa".  From here we will narrow the search iteratively.

Faced with a problem like this most database systems can't cope.  That's because all database types, regardless of their design, trade-off searchability vs maintainability.  Let's look at this question for the three main database types.

Navigational (hierarchical and network)

Navigational databases comprise linked lists written to disk.  They generally have a root entry point and from here searches are conducted down branches and a list of addresses is returned that meet the criteria.  Used with indexes, searching navigational databases is very fast.  Their major downfall is that they have no pre-defined structure, so cannot be searched using a computationally complete query language.  Instead a domain specific query language must be created for each database.

Object-oriented

Object-oriented databases are really navigational databases dressed in sheep's clothing. Their main point of difference from navigational databases, is that they can store any type of data because the instructions required to interpret that data are stored in the same object as the data. Consequently, object-oriented databases are very good at storing multi-media files.  However, just like navigational databases, object-oriented databases cannot be searched using a computationally complete query language. They also grow very large quickly as the instructions required to interpret their data is replicated unnecessarily many times over in each object instance.

Relational

Relational databases use a quite different approach to data storage than navigational and object-oriented databases.  Instead of linked lists, relational databases use a collection of tables. The tables are not connected together directly as a linked list is.  Instead, the linkage between tables is an indirect one, where the id field of one table is stored as a field in another table and referred to as a foreign key.  The advantage that relational databases have over the other databases is that the query language used to join these tables (relations) together is computationally complete. That's why it's called SQL (STRUCTURED query language) folks! SQL is not a domain specific. It can be used on any relational database, regardless of it's design. Unfortunately for us, the table structure found in relational databases is not suited to binary data such as our spreadsheet, since SQL is optimised to traverse tables with fixed length fields.

Summary

While navigational databases and object-oriented databases can return search results quickly, to do so they use domain specific query languages.  In other words they trade-off maintainability for searchability.  Conversely, relational databases are slower, but anyone can query them using SQL, their universal query language.  Having SQL is a real bonus for maintainability, but it comes at the cost of not being suitable for manipulating binary data.

Solution

How does any of this help me to find my lost record?  Neither navigational databases nor relational databases are very promising. At face value, object-oriented databases look like they'll do the job, providing they have the correct code available to search through the spreadsheets stored inside them. The truth is they don't. It can be done, but we are back to that maintainability problem again.

BUT, through a happy coincidence with Classmaker, we get the best of both worlds, because when I implemented Classmaker I chose to structure the data hierarchically inside a relational database.  Most planning packages use tags to aggregate similar records together. This produces a similar effect to a network database where there are multiple routes to the same record.  I chose instead, to arrange my records hierarchically with Long Term plans above Unit plans above Lesson plans and Binary file attachments. This hierarchical structure means that while Classmaker can't slice and dice it's database like one with tags could, it was not a hard job to EXPORT the entire database structure to disk, because the directory structure on disk is hierarchical too. There's a further problem with tags. If you don't attach the correct tags to your records, you're NEVER going to locate them!

Windows Search

Windows Search is an interesting facility.  It comes with the IFilter plug-in. This is an API that lets third party binary file formats expose themselves to Windows Search for indexing. Using Windows Search we can look for all spreadsheets, or in fact, any file that contains the phrase "sa".

Conclusion

Classmaker uses Windows Search indirectly to let you examine every record in it's database whether plain text  or binary format for the phrase "sa". Simply export all Classmaker's data to your local Documents directory.  Windows Search automatically indexes all files written there.

I have used Windows Search many times since I discovered it.  It's great knowing that no matter what I load into Classmaker, I will always be able to retrieve it again.  No more lost records.  No more calls to Houston!

IsectMP

IsectMP is a middleware multiplexer written in ANSI C that uses blocking RPC's in a single threaded select loop. The goal of IsectMP is to solve the client/server concurrency problem using collections of identical processes rather than threads with the following constraints:

1.    Keep the API as simple as possible to promote language agnosity.
2.    Impose no record structure on the data being exchanged.
3.    Have all data transfer move through a central broker (Isectd).
4.    Be able to stop and start remote processes via Isectd.

1.    IsectMP only wants to solve the Request/Response model.  This is the classic client/server problem where you have a single static server and many clients that come and go.
2.    IsectMP deliberately uses blocking I/O on a single thread in Isectd.  It solves concurrency by breaking down responses from workers into a stream of discrete chunks which must be concatenated at their destination. Requests from clients can be chopped up too, but this is unusual when dealing with databases as requests are usually small while responses can be very large.
3.    IsectMP deliberately forces all communication to pass through Isectd.  While some may argue this is creating a bottleneck, the advantage of having a central broker is that administrators know the health of the entire application by querying Isectd's registers.
4.    Isectd comes supplied with a command language that lets you adjust its state dynamically.
5.    IsectMP comes supplied with a parent daemon (Execd) that can start child processes locally from an instruction sent remotely via Isectd.

It turns out there are lots of other open source libraries out there doing similar things, but they all have different goals in mind from IsectMP.  I freely admit to only knowing IsectMP well, but from what I can see, if you stick with open source:

1.    IsectMP is the only one that uses ANSI C and compiles on Windows without issue.
2.    IsectMP deliberately uses a central broker.  The others frequently try to avoid using a central broker.
3.    IsectMP has the smallest API of any, just five function calls.
4.    IsectMP comes supplied with its own console client that can configure and control the central broker remotely.
5.    IsectMP's two daemons are tiny, the largest daemon weighs in at < 250kB including DLL's.

Certainly IsectMP has it's failings:

1.    It is not asynchronous.
2.    It will only cope with hundreds of clients, since each client gets it's own dedicated socket.
3.    The only transport mechanism it can use is TCP.
4.    It is single threaded.
5.    It's API relies on a C struct which causes headaches when creating new language bindings (2021-08-22  This is no longer true. A lot of reworking to the internals of Isectd has been done since this post was written).

I would not attempt to use IsectMP for an enterprise size application, but for a department size application, IsectMP works very well indeed (2021-08-22 It also does a grand job acting as the middleware behind web sites).

Spreadsheet madness

Today, a large business (X) accidentally emailed me all their clients' data inside a spreadsheet. This is not the first time this has happened in New Zealand.  Recently EQC, a government department, emailed a spreadsheet containing thousands of clients claim records to one client.

Apparently, X sends spreadsheets routinely to it's staff for further manipulation. Why?

For X, I believe, it's because they've purchased a number of other smaller businesses recently and the small business units' systems are not integrated with X's centralised accounting system. The small businesses units aggregate their monthly figures into a spreadsheet which they then email to X's receivables department.

The problem X has, is that its small business units are aggregating their data too early in the process.  As soon as data is aggregated it is at risk of being divulged.  I contend that the solution to their problem is that the data aggregation should be delayed until just before it is posted to the centralised accounting system.  In fact, I suspect there is no value to aggregating it at all.  Most accounting systems require data to be keyed into them manually and I imagine X's accounting system is no different.

What X really needs, is for its small business units to be posting their individual transactions as they are created onto a searchable queue that can automatically be purged once the data is committed inside the accounting system.

X is partway there already since it's email system is already a queue.  Problem is, that queue isn't secure and isn't visible to all its relevant staff.  What they need is a single email address that multiple internal users can subscribe to, but not a message list, rather an issue list.  There are plenty of these about - we usually call them bug tracking software.  The one that I'm most familiar with is Roundup Issue Tracker. There are many others.

And what content gets posted to Roundup? The spreadsheet sent to me would be fine as long as it contained just one client's details, but I still don't like spreadsheets much, because they bundle data, business rules and presentation together in a single file. I would prefer instead to use a static Web browser application for the reasons outlined in my post called Real world Web browser development.

Using Roundup to avoid information overload

Roundup Issue Tracker helps prevent information overload by:

1.    Stopping ALL spam.  If an addressee is not known to Roundup the email is silently dropped.
2.    Only privileged users can create a new issue, so if an existing external user's email account is hacked, they still cannot spam you with a new issues.
3.    All superfluous email signatures and graphics are purged, just the plain text component and file attachments are retained.
4.    If a message gets through that does contain superfluous material, you can edit the message directly on disk or retire the message so it is no longer visible.
5.    As soon as you decide that a message thread is no longer current you simply retire it.  While it remains visible to searches, it no longer appears on your list of current conversations.
6.    A number of different views of your issues can be saved.
7.    File attachments aren't retained with the individual message, they are retained at the email conversation level.

These Roundup features mean that:

1.    The excessive broadcaster.
2.    The quote everything replier.
3.    The spammer.

can be stopped in their tracks.  The one problem that Roundup can't solve is the file attachment abuser.  Frequently the excessive broadcaster and the file attachment abuser are one and the same, so by preventing excessive broadcasting, hopefully file attachment abuse will decline too.