Seamless Forth Scripting Engine

When I was young and reckless I wrote my first Forth system entirely in x86 Assembly and it has generated native code (i.e. machine code instead of threaded code). It ran on close to bare metal on PC hardware.

Soon afterwards I had to rewrite the whole darn thing in C++ to run it on Linux (which was something quite new back then). That taught me a lesson never to write huge programs in Assembly ever again.

I was still in school at that time, working on my Ph.D. thesis. My first real job afterwards was in Silicon Valley working for a company that wrote a lot of system code in Forth, trying to build a marketable product based on an i21 stack processor.

I never did that again either, i.e. I have never worked again on a project where large amount of code was written in Forth.

In fact I have spent the better part of my career writing embedded software in C, often controlling custom hardware (IP blocks that did not exist in any other product).

Eventually I ended up using Forth or a Forth-like command line in each and every job I had because it is just so useful when trying to get some piece of hardware to work (and work properly).

These weren't Forth projects, often minimal or partial Forth implementations were used as a debugging environment, or as a means of controlling the device through scripts to allow development and easy exploration, manufacturing, calibration, testing, product long term maintenance, etc.

Simply having a command line, with a real programming language behind it, to use over a serial line, or whatever interface one might have is way more powerful than some arbitrary user interface regardless if it is menu based or has some arbitrary syntax and commands.

Also having the ability to run scripts on an embedded device can be very useful for some production features, e.g. device configuration can sit in human readable text files.

Or e.g. any sort of non-trivial machine to machine communication with e.g. PC based software can be simplified if the device understands a standardized set of commands.

Just think about having symbolic names for registers in custom hardware blocks compiled into the command interpreter in the firmware instead of some other piece of software trying to figure out at what address a certain register is located at in a certain revision of the firmware that is now obsolete but still needs to be supported.

Of course one could use other languages for this, some with a more human friendly syntax, but Forth implementations tend to be small (so actually fit into the resources on a microcontroller). Forth is also relatively friendly to hardware operations (default numeric format is really native integers, and commands are executed in the order they appear in the input so one can actually tell just by looking at the source code when e.g. a hardware register is being accessed and exactly how). Also bit manipulation using logic operations is supported.

I ended up writing a custom Forth engine for each major project I worked on, but after some time it gets repetitive.

A few years ago I have implemented a general purpose Forth engine that is embeddable inside a C program or firmware -- I have later on started to refer to it as EFCI (Embeddable Forth Command Interpreter).

It is a token threaded system which is a very proper Forth in many ways, delightful to look at as a Forth programmer.

It is also not the easiest to work with in a C environment. Most of the functionality is implemented as Forth threaded code, compiled into a dictionary which needs to be generated by a program, thereafter it can be just compiled into the executable so long as nothing major changes. But it is a hassle if one wants to fiddle with features.

Also, because everything is a token arbitrary C functions in the system code can't just be exposed, they need to be registered with the token interpreter, which is kind of a clumsy way to interface inside the same executable. It is somewhat a hassle to call C and Forth code from each other.

I have been using this system heavily in my own "hobby" projects (it never went into a commercial product -- I was working on a project ever since its inception that already had its own custom Forth interpreter).

But recently I have decided that I needed a Forth scripting engine that merges with C code in a more seamless fashion, so I have created one, mainly by cannibalizing EFCI components, and applying what I have learned from using Forth scripting in actual products.

I have just made the sources available on GitHub https://github.com/azsoter/Seamless-Forth-Script, it is licensed under the MIT license, so anyone can use it pretty much any way they see fit.

This is still a fairly new system, so it does not have a lot of mileage on it, I am looking forward to discovering how much it can make my life and work easier when I need access to various functions in my firmware either for testing or even for some production features (I have only used it so far in a hobby project, so production as such remains to be seen).