- Each customer has their own database for data. Consequently you can keep it simple with integer id's and integer foreign keys instead of a mess of GUIDs which as well as being confusing massively impact on performance as a index key. The database server manages multiple small databases.
- Their code, though, resides in a SINGLE database for code. The code database is organised like a library with shelves, books and chapters (pages exist too but these are called indirectly). A shelf is an entire application which comprises many books, but the customer directly accesses just four books, a private book (administration), a public book (customer web pages that can be used by anyone), a protected book (customer web pages that can only be accessed by their registered users) and a global book. The customer books contain only the chapters which are customised for them. Every time a client requests a web page, the customer books are browsed first. If the book's chapter is missing then the chapter is sourced from the global book instead.
- It's impossible for a customer to request another customer's book because to read their book you have to be standing right in front of it. This is pushing the analogy a bit far! I have just ONE very simple cgi file (all it does is collect cgi parameters and cookies, sends them to the code database and returns the reply to the web browser) that calls the code database, but there are MANY COPIES of that file in a shallow directory tree. Each copy has it's own ini file which specifies the name of the book it can access and which database. None of that information passes over the Internet. Access to the cgi files is password protected where necessary (using SSL encrypted basic authentication).
Monday, 11 April 2022
Storing Your Code in a Database 1
Sunday, 22 August 2021
My 64bit Windows 7 Software Ecosystem
The tools listed below are where I spend most of my work time, ordered from most used to least:
Daily for time sheets, checking bank statements and general emails
Accounting - GNUCash
Web browsing - Mozilla Firefox
Email - Mozilla Thunderbird
Twice weekly for invoicing
Planning - Classmaker
Mail Merge reports - Microsoft Word 2003 (the data source is an HTML file emitted by Classmaker)
PDF Creation - Print to CutePDF
Once a week
Backup - Classmaker
Off-site backup - Google Drive
Once a fortnight
Scanning - Windows Paint
Creating multipage PDF's from scanned images - IrfanView
Spreadsheet - LibreOffice Calc
Word Processor - LibreOffice Writer
Once in a blue moon!
Chart creation - LibreOffice Draw
Desktop publishing - Scribus
When things go horribly wrong!
Puppy Linux Tahr 6.0 on a USB stick
- Puppy has saved me several times over the years when Windows 7 hasn't booted or file corruptions have occurred.
- The most recent incident involved OLE embedding a LibreOffice Draw file inside a LibreOffice Writer document. Don't do this! That technology is unstable and you will find yourself locked out of the Writer document as somehow ALL the file permissions get removed. The only way to retrieve the file is to boot into Linux, move the file into a Linux partition and then copy it back.
- Instead of OLE embedding, export your Draw file to disk as a PNG image and then insert that into your Writer document.
Friday, 18 June 2021
Safe Python Eval
Many of us want to use a safe simple scripting language to extend our Internet applications and I have seen many discussions on the Internet on how to implement Python so that it's eval function can't be exploited. The problem is, nobody can be sure that they've covered every scenario, because Python has become complex over time, with multiple ways to achieve the same objective.
My scenario is a typical one. I have a web based database application hosted on a remote server. Now I want it's users to access it's database to produce ad hoc reports, without compromising the database or anything else! I could do the typical thing of providing a screen where users can submit Select queries, but SQL by itself isn't enough to get a report that is useful except in the most simple scenarios. To begin with you can only work with one query result at a time. Reports frequently require multiple query results to be combined. Tables of course can be joined using SQL, but the result usually has lots of redundant data fields that must be omitted. A declarative language like SQL just can't do it alone. It needs to be combined with a procedural language. After all, that's how my web application works. The procedural capability of Python is combined with SQL to produce web pages that make sense to users.
After surveying the scripting languages out there, Python came in at number one, BUT, it has an eval function that can access the file system or carry out other security exploits. Vanilla Python isn't a sensible choice as an extension scripting language.
I have looked at FORTH in the past, ATLAST and Ficl were front runners, but they are traditional FORTHs with weak string manipulating capability. Strong string capability is what I need, because my database queries return as nested Python string lists. I came up with that format years ago before JSON became popular, using Python eval to parse the database query, returning a table that Python can iterate over using it's index functions.
Then I came across FORTH written in Python, a toy project for educational purposes. I realised it wasn't going to be fast, because FORTH is an interpreted language written in an interpreted language! What it did have going for it was very strong string capability and the ability to cherry pick the Python functions required, leaving everything else behind. Suddenly, eval wasn't a problem as I could overlay it with some checks to make sure eval could only eval a list. FORTH is a simple typeless stack based language with the entire environment comprising two stacks and a heap (Python lists), a dictionary (Python dictionary) and a heap pointer (Python integer). FORTH has no syntax! The language is just a stream of white space delimited tokens called words. Traditional FORTH has no error checking and crashes are common place. But having Python underneath FORTH makes it uncrashable. Sure your FORTH code will frequently bomb, but the FORTH REPL (Read Evaluate Print Loop) just displays the Python error and awaits your next command.
I made some alterations to the Python code to get a FORTH that isn't quite FORTH anymore (as the saying goes 'when you've seen one FORTH - you've seen one FORTH')! Everything became a string, both the stack and the heap able to handle strings or lists of any size in addition to numbers. Actually Python is cleverer than that. Lists can store variables of any type, so my database lists stay as lists (under the hood, they're probably C arrays) which are passed by reference rather than value. Numbers stay as numbers and dates as dates unless they are cast explicitly, something I've brought over from Python. But apart from that the toy project has stayed largely untouched. All it can do is step, branch, loop, create variables, read variables and print to the console. But at just over 300 lines of Python, it's Turing complete and this is all you need to create a report. Already I have written a complex report utilising two tables that cannot be joined, the report output is styled HTML.
Perhaps the most interesting thing I've done is to create a persistant library of Forth definitions. The way I've done that is to store the FORTH environment after loading forth.py (the REPL) and forth.fr (the persistant library) as Python command line arguments. The working code FORTH file is then called manually in the REPL. Each time it is called the FORTH environment is restored to the finish of forth.fr. This means you can call the working code file over and over without having to restart the REPL.
But there's still a problem! FORTH over Python might be safe, but a user could write a function that never returns. Endless loops are easy to create inadvertently in procedural languages. Code needs to be added to Isectd that checks how long a worker has been busy for. If it has been busy for longer than 20 seconds and has no clients attached then Isectd sends a kill command to the process which will automatically restart after it has been killed.
The working FORTH code is included in PocketClassmaker install file from the downloads area of my website.
Monday, 16 December 2019
Fox-Toolkit
Here's why I think it's brilliant:
1. It's just a library written in C++, but Jeroen dislikes the STL and doesn't use it. The resulting language resembles C with inheritance rather than C++. Most of time I don't even use it like you would C or C++ as I have no need to concern myself with advanced concepts like inheritance, pointers and all the rest. Instead, I use it the way you used VB... without an IDE. In VB, you created forms and modules. Most of the code was written as event code associated with a widget and shared code was placed in modules. In Fox it's the same except that the form is just a standard *.cpp text file with Fox's window creation boilerplate and a module is a standard *.cpp text file without Fox's window creation boilerplate.
2. Because it's written in C++:
- it's runs extremely fast
- providing a project is set up correctly, compile times are quick, because most of the code is already compiled previously and only needs to be linked
- any external library written in C or C++ is easily integrated
- calls to the Windows API are a breeze
- I have multiple free compilers at my disposal
- the C syntax is terse and easy to read
- there are many code indenters available to tidy your code
- if I need to, I can program at a systems level, rather than at application level without having to move to another environment.
4. It has a very powerful string class called FXString. I use this constantly. Most of my programming revolves around relational databases which I query using my own disconnected recordset object, written in Fox, which parses a JSON like string into linked lists of FXString. Very simple, very fast and rock solid performance.
5. The window geometry manager avoids fixed window sizes, so Fox applications resize themselves sanely. VB didn't come close in this regard.
6. Fox comes with it's own code editor called Adie. Of course, any code editor is viable. Notepad++ is popular. But I like Adie because it's so simple.
7. The widgets collection is huge and with a bit of digging in the code you discover things are available that you don't expect. For example, recently I found that the FXTable widget lets you copy it's contents to the clipboard in tab delimited format. Spreadsheets parse this into columns and rows directly without the usual dialog palaver that comes with a CSV file. Other time savers include INI file creation which comes baked into Fox allowing window positions to be saved on exiting your application.
8. Being open source with a liberal licence, you have access to all the source code which is reasonably easy to follow, so you can see exactly how things work. The situation has yet to arise, but if a widget doesn't have a feature I want, I can inherit a new widget from it that does exactly what I want.
9. Cross-platform development is the goal of many tools, Fox included. But, I have come to see that real world application programming is about being able to call into an operating system's API to achieve outcomes that aren't clunky workarounds. For instance, I needed to convert rtf to pdf sanely and programmatically. The best way of doing this was to call LibreOffice in headless mode, but that resulted in console windows appearing. I found the appropriate Windows API call to suppress the console windows and substituted it for the normal system() call. While I could have done something similar in other languages it would have been much more difficult to implement. With Fox it was just a bit of cutting and pasting.
Perhaps the biggest benefit of all from using Fox, is that you cannot code yourself into a corner. So many times you start to use a tool only to find that there is some aspect of it that constrains your development. In 15 years of developing with Fox that is yet to happen. I've produced desktop applications and web sites, just using Fox-Toolkit. Check it out.
Friday, 29 June 2018
Disconnected Recordsets
Reading some of Microsoft's documentation today surprised me, because ADO.NET's approach to database access is nearly identical to my own hand-crafted disconnected Recordset object, which I came up with nearly 20 years ago, roughly the same time ADO.NET 1.0 was released.
What does my Recordset object do?
- It parses a tokenised string from the database driver or from a file into a double linked list which is then referred to as a recordset.
- It contains methods to iterate forwards and backwards through a recordset.
- It contains methods to retrieve the contents of a field by name or position from the current row in a recordset.
- It contains date and time methods.
- It can sort a recordset.
- It can emit a recordset as a tokenised string.
- It can write a recordset back to the database in one round trip.
- All data access is carried out under transaction management.
- Additional methods can be added to the object as needed, without impacting earlier code.
- Multiple recordset objects share the same database connection which is opened automatically when the first query is run.
- A recordset object can act as a record datatype without using a database.
- The recordset object hides a lot of the repetitive insert, update, delete queries that come with SQL. SQL queries are still used a lot though. Without in depth knowledge of SQL, you can't use relational databases effectively.
- As the recordset is a tokenised string it can be written to disk and retrieved again.
- Transaction management can span multiple recordset objects.
[[{'name':'ID','type':'i','null':0,'width':4},{'name':'DESCRIPTION','type':'s','null':1,'width':30},{'name':'REORDER','type':'i','null':0,'width':4}],
[[ 5,'Moved', 5],
[ 10,'Critical', 10],
[ 20,'Urgent', 20],
[ 30,'Normal', 30],
[ 40,'Wish', 40]]]
Through a historical accident, it so happens that this tokenised string is also a Python list. Perhaps if I was to reimplement it today I would use JSON which is more widely used. Nevertheless, this format works well. It can be evaluated directly using Python and I have written a parsing routine in C++ that converts it to a double linked list in a single parse.
JSON can exceed two dimensions and use schemas, but the column and row format is what makes my tokenised string so efficient to parse and to manipulate once parsed. Columns and rows are easy to understand and column numbers rather than column names can be used for faster data retrieval. Not that this makes much difference, since a linked list is very fast to iterate over, so I usually use column names instead of column numbers to maintain code legibility.
Looking at the string you can observe that it is restricted to columns and rows, but it isn't a flat file. Instead, this format lies somewhere between a flat file format like dBase which has fixed column widths and is very fast to query using file offsets and CSV files which don't have quite enough information to be useful as a recordset.
Here's another tokenized string:
[[{'name':'ID','type':'i','null':0,'width':4},{'name':'MSGID','type':'i','null':0,'width':4},{'name':'FKISSUES','type':'i','null':0,'width':4},{'name':'FKPEOPLE','type':'i','null':0,'width':4},{'name':'BODY','type':'s','null':1,'width':32765},{'name':'STATUS','type':'i','null':0,'width':4},{'name':'LAST_ACTIVITY','type':'d','null':0,'width':8}],
[[ 4, 4, 3, 6,'What would we talk about?', 30,'2018-06-02 15:10:07.0000'],
[ 6, 6, 3, 6,'I need to keep going with this software.', 30,'2018-06-07 21:01:40.0000'],
[ 7, 7, 3, 6,'Another note...', 30,'2018-06-19 11:16:40.0000']]]
Notice that the field descriptors contain the field type. This is vital for sensible parsing of the recordset. Type inference which is what you have with CSV files can only get you so far. Knowing the data type of the field beforehand allows for clever data massaging e.g. my recordset object has an optional flag for elapsed time which represents dates as duration from now rather than the raw format you see here.
There's a few things that aren't so great with disconnected recordsets:
- They can't store binary data. I get around this by converting to Base64 which is slow. Over time, I've come to realise that binary data is much easier to use if it is kept out of a database anyway.
- How do you deal with deeply nested data? Relational databases don't store data hierarchically. Instead they use relations between tables. It is possible to store data hierarchically using table relations, although a little tricky (a whole blog post could be devoted to that subject). Recently, I have found the need to store data hierarchically, but I do that using the database tables themselves, so it has no effect on my recordsets.
- What about shallow nested data? I use Base64 encoding to store recordsets inside recordsets, since none of the Base64 characters match the recordset's delimiters, but I've only encountered one instance where shallow nesting was needed.
- The parsing routines must escape delimiting characters. I use octals to avoid the escaped representation containing letters which hexadecimal requires.
Technical Details
- disconnected Recordset client (data view)
- C++ using Fox-Toolkit which has an excellent string class that I use heavily in my projects.
- eval in Python is all you need.
- disconnected Recordset driver (data adapter)
- native driver written in C for the FirebirdSQL relational database.
- disconnected Middleware (connection)
- both the client and server depend on IsectMP to work. IsectMP is a tiny RPC multi-plexer written in C that I maintain. It contains the FirebirdSQL driver, but not the C++ client.
Friday, 6 October 2017
What's wrong with CGI?
1. The user requests a page.
2. The web server starts up a new process, serves the content and the process dies.
3. If the page is doing anything interesting that invariably means the CGI process must open a database connection and destroy it again.
In the past, operating systems weren't smart and starting a new process mean't reading the binary image into memory from disk. That's not true now, particularly for Windows (I'm a Windows developer, so I had better stick with what I know); it caches all recently read files in memory. So, if your CGI file is a compiled executable then the overhead of CGI has gone. If your CGI uses a scripting language such as Python, Perl, Tcl etc then their virtual machines must parse your script every time which will slow things down, but not by as much as you'd think, since both the virtual machine and the script are cached in memory and in Python's case at least, the script gets compiled to byte code on the first parse and thereafter runs much quicker.
Opening and destroying database connections is expensive, mainly because the database server must authenticate your connection. But if you insert some middleware in between the database and your CGI file and pool your database connections, the authentication overhead is removed. Ensuring that your database recordsets are disconnected recordsets also makes things run much faster.
So to recap, if you have CGI executables using disconnected recordsets from pooled database connections what's wrong with CGI? Let's count the advantages:
1. The CGI executables are tiny. I have a single EXE for each page, all of them sharing a 2 Mb dll. The EXE's never exceed 200 Kb. They compile in a flash.
2. Because each CGI is stand-alone, I can keep the web site running, while just working on one aspect of it. And if a CGI should crash in production, the whole site isn't brought down, just that page.
3. The site runs exceptionally fast, because the CGI files only do the bare minimum to create a page. No lumbering template libraries here. In fact, no template libraries at all! HTTP, when you remove all the towering library abstractions built on top of it, is actually very basic. Just GET and POST calls passing around variables, HTML Forms that collect the data into those variables and cookies to maintain state. It also helps to have a little bit of Javascript sprinkled about so that your Forms don't need to return to the server for every action the user takes. A typical CGI file begins by collecting any variables sent to it, reading cookies and opening a number of disconnected database recordsets. The variables and cookies are transformed and saved as appropriate and the file finishes by writing out a new page. In my files less than a third of the code is devoted to writing the new page. Most of it revolves around updating the database.
4. Secure. You can't have a scripting attack, if all the code is compiled into an EXE.
The disadvantages?
1. My code uses combination of C and C++ supplied in three libraries. First, cgihtml-1.69, a minute CGI library written in C, that I scrounged off the web and which was last updated in 1998. Second, fox-1.3.15 written in C++ and released in 2002 (Fox-Toolkit is still current. The latest release is 1.7.61, but why update when the existing functionality surpasses anything I'll ever need?) and third, isectMP-1.0.3, a database pooling library written in C which I maintain. Fox-Toolkit comes with an excellent String manipulation class which I use constantly, but sometimes I yearn for the simple syntax of Python. That yearning soon disappears when I have to interface with a third party library or interact with Windows directly! And what about Python deployment? Let's not go there!
2. Writing CGI's in C and C++, just isn't fashionable. I constantly have to remind myself that it's results that matter, not that I should be using the latest language du jour.
And so, I find myself asking that question again, what is the problem with CGI? Answer: Nothing!
Sunday, 15 May 2016
PC based Lesson Planning Software Shootout
Most teachers have probably moved to Web based solutions by now and they do have their advantages, but if you like me, prefer to have control over your own data and aren't constrained by school policy to use their mandated Web solution, then perhaps PC based software might still work for you.
I compared both packages exhaustively for usability, but omitted testing Planbook's PlanbookConnect which is some kind of Web based synchronisation service, since PocketClassmaker can already do that using Google Drive and I didn't want to sign up to PlanbookConnect.
If you can't be bothered to look through my comparison table below here's the Executive Summary:
Both packages cover similar territory, but from different angles which results in a different user experience:
- Planbook is optimised for rotating lesson plans (create a rotating schedule by subject first and then fill in the detail later) for a single user and cannot be used as a normal calendar. It is relatively painless to get going, but has a number of features which I found very frustrating. These include entering lesson times which is fiddly at best, the startup screen position cannot be saved and perhaps the biggest frustration, to save your work you must use the Ctrl S key combination constantly or risk losing your planning. There is also a bug that crashes the program when trying to access the Lesson Library. To allow the software to be used on several machines Planbook has a synchronisation function called PlanbookConnect which I didn't test.
- PocketClassmaker is optimised for ad hoc planning (plan as you go by copying existing lessons forward and amending them to suit) with multiple users and thus can be used for any task that is calendar related, but, the learning curve for installing and using it is steeper than for Planbook. PocketClassmaker is a portable app and is designed to be installed on a USB stick.
| PlanBook | PocketClassmaker | Comments | |
|---|---|---|---|
| Website | www.hellmansoft.com | www.creditscore.co.nz | |
| Windows | 64 bit Windows 10 Home | 64 bit Windows 10 Home | PlanBook versions exist for Mac, Windows XP and Ipad. Classmaker only works on Windows XP and up. |
| Screen size | 1366 * 768 on 14 inch screen | 1366 * 768 on 14 inch screen | |
| Version | 4.0.51 | 2.0.6 | Date downloaded 07 May 2016 |
| Cost | USD30 for one user | Free for one user | |
| Installation | Windows MSI. Only allows installation to C:\Program Files (x86)\Hellmansoft\PlanBook |
Extract Zip archive to any location | Classmaker is a portable app. |
| Google Drive | No | Yes | Only one instance can be active at any time for Google Drive replication to work. |
| Invoking | Normal menu item | Two batch files located in the Scripts directory. | |
| Multiple users | No | Yes | |
| First invocation | Wizard | Blank screen | Classmaker comes with an example database. |
| Remembers window positions | No | Yes | |
| Default dates | September 2013 to June 2014 | Have to create a new user and a default non-contact lesson plan before the screen displays. Defaults to today’s date. | |
| Subjects | Created using Wizard | Defined at Syndicate Leader level | |
| Time entry | Fiddly input fields that require am/pm | Uses 24 hour clock | PlanBook's approach of using option boxes is disastrous from an inputting perspective. The a.m / p.m selection often fails to take correctly and the time reverts to the initial setting. |
| Date entry | A lesson can only belong to one day | Lessons can cross multiple days using the same start and end times | It is almost impossible to use PlanBook for add hoc planning as white space is not allowed |
| Fast Date Entry | Not needed because lessons are always prepopulated using some kind of rotating schedule | Just enter the day for the current month | PlanBook only works with rotating schedules |
| Database type | Individual files for user defined periods | One database file for all users and forever | |
| Autosave to Database | No | Yes, but required to explicitly add, edit or delete a record | Losing your work after some time doing data entry in PlanBook is VERY frustrating. |
| Long Term Plan | No. Month display shows Subject name only. | Yes. Term display shows Subject and Unit Name. | |
| Unit Plan | Optional. Used for printing and exporting / importing lessons. | Mandatory. | |
| Permanent Lesson Storage | No. PlanBook crashes! | Yes | PlanBook developer’s main focus appears to be on Apple Mac’s. |
| Weekly Report | Attempts to include detail | Just prints lesson names | |
| Daily Report | Ugly cross page splits. Includes empty lessons. | Nice table display | |
| Future and Past Lessons | Can’t tell if you’ve created future lessons | Long term display tells you immediately if you have done future planning | |
| Attachments | Attached to Lesson Plans using a link | Attached to Unit Plans and stored in database file | |
| Website links in lesson plans | Yes | No | In Classmaker would have to use HTML tags eg. <a href=http://www.stuff.co.nz>http://www.stuff.co.nz</a> and print to HTML |
| Report Types | Daily, Weekly, Unit | Daily, Weekly, Term, Unit | |
| Report Formats | Print only | Print, Postscript, HTML | In Classmaker HTML reports are similar to CSV files and can be imported into MS Word or formatted using CSS for display in a Web Browser |
| Database Export | Yes, but empty days are exported too. Assumed that database will be published to a web site between dates. | Yes. All date and time relationships have been removed. Just the data and file attachments are exported. Assumed data will be globally searched including file attachments. | |
| Search | All fields in database, but not attachments. | Only lesson titles. | Both packages require database export to search attachments. |
| Lesson Types | Unit | Unit, Routine and Non-Contact | |
| Schedule Types | Same subjects repeated daily, same subjects repeated weekly, rotating schedules up to 14 days | Ad hoc, subjects repeat daily, subjects repeat weekly, rotating schedules up to 30 days | |
| Lesson Movements | Bump, Yank, Lock | +Day, +Week, Bump, Shove, Up, Down | |
| Unit Export | Attachments aren't exported only links | Multiple units can be exported as a batch including attachments | |
| Rotating Lessons | Set up lessons beforehand and then use directly | Use a placeholder unit that must be copied forward again and again | |
| Field Input | On failure reverts to initial figure | On failure input figure retained but not written to database |
