23 KiB
RETRO: a Modern, Pragmatic Forth
Welcome to RETRO, my personal take on the Forth language. This is a modern system primarily targetting desktop, mobile, and servers, though it can also be used on some larger (ARM, MIPS32) embedded systems.
The language is Forth. It is untyped, uses a stack to pass data between functions called words, and a dictionary which tracks the word names and data structures.
But it's not a traditional Forth. RETRO draws influences from many sources and takes a unique approach to the language.
RETRO has a large vocabulary of words. Keeping a copy of the Glossary on hand is highly recommended as you learn to use RETRO.
This book is structured into topics.
Quick Start
Get RETRO
RETRO can be obtained at:
Build From Source
Run make to compile RETRO and its toolchain. The binaries will
be placed in the bin directory.
If the build fails, try make -f Makefile.alternate.
There are various binaries. The main one is retro. Copy this
to a place in your $PATH and you'll be set to run it.
Use A Prebuilt Binary
Go to forthworks.com/retro and download the binaries collection. Copy the appropriate binary to a location in your path.
Use An OS Package
RETRO is packaged for a few systems.
TODO: add details here.
Commercial Versions
iOS
macOS
Run RETRO
RETRO can be run for scripting or interactive use. To start it
interactively, run: retro -i or retro -c.
For a summary of the full command line arguments available:
Scripting Usage:
retro filename [script arguments...]
Interactive Usage:
retro [-h] [-i] [-c] [-s] [-f filename] [-t]
-h Display this help text
-i Interactive mode (line buffered)
-c Interactive mode (character buffered)
-s Suppress the 'ok' prompt and keyboard
echo in interactive mode
-f filename Run the contents of the specified file
-t Run tests (in ``` blocks) in any loaded files
The Virtual Machine
RETRO runs on a virtual machine called Nga. This emulates a MISC (minimal instruction set computer), with 30 instructions and two stacks.
Memory is provided as a linear array of signed 32-bit values. There is no direct addressing of other values.
Instruction Set
00 NOP
NOP does nothing. It's used for padding and reserving space.
01 LIT
LIT pushes the value in the following cell to the data stack. This is the only instruction which takes a value from a source other than the stack.
02 DUP
DUP duplicates the top value on the stack.
03 DROP
DROP discards the top item on the stack.
04 SWAP
SWAP switches the positions of the top two items on the stack.
05 PUSH
PUSH moves a value from the data stack to the address stack.
06 POP
POP moves a value from the address stack to the data stack.
07 JUMP
JUMP takes an address from the stack and transfers control to the code at the specified address.
08 CALL
CALL calls a subroutine at the address on the top of the stack.
09 CCALL
CCALL is a conditional call. It takes two values: a flag and a pointer for an address to jump to if the flag is true.
At the instruction level, a false flag is zero and any other value is true.
10 RETURN
RETURN ends a subroutine and returns flow to the instruction following the last CALL or CCALL.
11 EQ
EQ compares two values for equality and returns a flag.
12 NEQ
NEQ compares two values for inequality and returns a flag.
13 LT
LT compares two values for less than and returns a flag.
14 GT
GT compares two values for greater than and returns a flag.
15 FETCH
FETCH takes an address and returns the value stored there.
This doubles as a means of introspection into the VM state. Negative addresses correspond to VM queries:
| Address | Returns |
| ------- | ------------------- |
| -1 | Data stack depth |
| -2 | Address stack depth |
| -3 | Maximum Image Size |
An implementation may use negative values below -100 for implementation specific inquiries.
16 STORE
STORE stores a value into an address.
17 ADD
ADD adds two numbers together.
18 SUB
SUB subtracts two numbers.
19 MUL
MUL multiplies two numbers.
20 DIVMOD
DIVMOD divides and returns the quotient and remainder.
21 AND
AND performs a bitwise AND operation.
+----------------+
| Before | After |
| ------ | ----- |
| -1 | -1 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | 0 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| -1 | 0 |
| 0 | |
+----------------+
22 OR
OR performs a bitwise OR operation.
+----------------+
| Before | After |
| ------ | ----- |
| -1 | -1 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| -1 | -1 |
| 0 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | -1 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | 0 |
| 0 | |
+----------------+
23 XOR
XOR performs a bitwise XOR operation.
+----------------+
| Before | After |
| ------ | ----- |
| -1 | 0 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | -1 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| -1 | -1 |
| 0 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | 0 |
| 0 | |
+----------------+
24 SHIFT
SHIFT performs a bitwise arithmetic SHIFT operation.
This takes two values:
xy
And returns a single one:
z
If y is positive, this shifts right. If negative, it shifts
left.
25 ZRET
ZRET returns from a subroutine if the top item on the stack is zero. If not, it acts like a NOP instead.
26 END
END tells the VM to shut down.
27 IENUMERATE
IENUMERATE pushes the number of simulated I/O devices to the stack.
28 IQUERY
IQUERY takes a device number from the stack and queries it, returning two values: a device version number and a device type identifier.
29 IINTERACT
IINTERACT takes a device number from the stack and triggers an interaction with it.
Instruction Encoding
Nga allows up to four instructions to be packed into a single memory cell. These are decoded and executed sequentially.
first second third fourth
00000000 00000000 0000000 0000000
Any unused instructions should be replaced with NOP (00000000). If an instruction modifies the instruction pointer all later instructions should be NOP. This is because the later slots will be executed prior to the IP change. Consider:
LIT CALL LIT CALL
In this case only the second CALL actually gets control. The instructions which modify IP are:
JUMP
CALL
CCALL
RETURN
ZRET
The LIT instruction takes a value from the following cell and pushes it to the stack. Multiple LIT's will use sequential cells. E.g.,
LIT LIT ADD NOP
100
200
Muri Assembly
The initial kernel is written in Nga assembly and is built using an assembler named Muri. This is a simple, multipass model that's not fancy, but suffices for RETRO's needs.
A small example:
~~~
i liju....
r main
: c:put
i liiire..
i 0
: main
i lilica..
d 97
i liju....
r main
~~~
So breaking down: Muri extracts the assembly code blocks to assemble, then proceeds to do the assembly. Each source line starts with a directive, followed by a space, and then ending with a value.
The directives are:
: value is a label
i value is an instruction bundle
d value is a numeric value
r value is a reference
s value is a string to inline
Instructions for Nga are provided as bundles. Each memory location can store up to four instructions. And each instruction gets a two character identifier.
From the list of instructions:
0 nop 7 jump 14 gt 21 and 28 io query
1 lit 8 call 15 fetch 22 or 29 io interact
2 dup 9 ccall 16 store 23 xor
3 drop 10 return 17 add 24 shift
4 swap 11 eq 18 sub 25 zret
5 push 12 neq 19 mul 26 end
6 pop 13 lt 20 divmod 27 io enumerate
This boils down to:
0 .. 7 ju 14 gt 21 an 28 iq
1 li 8 ca 15 fe 22 or 29 ii
2 du 9 cc 16 st 23 xo
3 dr 10 re 17 ad 24 sh
4 sw 11 eq 18 su 25 zr
5 pu 12 ne 19 mu 26 en
6 po 13 lt 20 di 27 ie
Most are just the first two letters of the instruction name. I
use .. instead of no for NOP, and the first letter of
each I/O instruction name. So a bundle may look like:
dumure..
(This would correspond to dup multiply return nop).
RETRO also has a runtime variation of Muri that can be used when you need to generate more optimal code. So one can write:
:n:square dup * ;
Or:
:n:square as{ 'dumure.. i }as ;
The second one will be faster, as the entire definition is one bundle, which reduces memory reads and decoding by 2/3.
Doing this is less readable, so I only recommend doing so after you have finalized working RETRO level code and determined the best places to optimize.
Source Format
RETRO is literate. Most of the sources are in a format called Unu. This allows easy mixing of commentary and code blocks, making it simple to document the code.
As an example,
# Determine The Average Word Name Length
To determine the average length of a word name two values
are needed. First, the total length of all names in the
Dictionary:
~~~
#0 [ d:name s:length + ] d:for-each
~~~
And then the number of words in the Dictionary:
~~~
#0 [ drop n:inc ] d:for-each
~~~
With these, a simple division is all that's left.
~~~
/
~~~
Finally, display the results:
~~~
'Average_name_length:_%n\n s:format s:put
~~~
This illustrates the format. Only code in the fenced blocks (between ~~~ pairs) get extracted and run.
(Note: this only applies to source files; fences are not used when entering code interactively).
Syntax
Forth is largely freeform, and RETRO generally follows this. But there are a few things that impact this.
First, RETRO processes code as encountered, token by token. A standard system does not expose the parser to the user.
Secondly, RETRO uses single character prefixes to guide the processing of a token.
The interpreter takes a look at the first character of a token.
If this matches a known prefix, the rest of the token is
passed to a function which handles that group of tokens. If no
valid prefix is found, RETRO tries to find the word in the
dictionary. If successful, the information in the dictionary
header is used to carry out the actions specified in the name's
definition. If this also fails, RETRO calls err:notfound to
report the error.
A flowchart of the interpreter process:
+-------------+
| Get a Token |
+-------------+
|
+------------------------------------+
| Is first character a valid prefix? |
+------------------------------------+
| |
YES NO
| |
+-----------------------+ +------------------------+
| Pass rest of token to | | Is entire token a name |
| prefix handler | | in the dictionary? |
+-----------------------+ +------------------------+
| |
YES NO
| |
+------------------+ +-------------------+
| Push XT to stack | | Call err:notfound |
+------------------+ +-------------------+
|
+--------------------+
| Call class handler |
+--------------------+
In RETRO, prefix handlers and class handlers are responsible for dealing with the tokens and words. This includes interpret and compilation as necessary.
RETRO permits names to contain any characters other than space, tab, cr, dnd lf. Names are case sensitive, so the following are three different names:
foo Foo FOO
I recommend that names not start with a prefix as prefixes
are checked prior to dictionary lookups. So a word named @foo
would be treated as a fetch from a variable named foo instead
of running the @foo word.
Tip: Run 'prefix: d:words-with to get a list of prefix
handlers in your system.
The Compiler
To create new functions, you use the compiler. This is generally
started by using the : (pronounced colon) prefix. A simple
example:
:foo #1 #2 + n:put ;
Breaking this apart:
:foo
RETRO sees that : is a prefix and calls the handler for it.
The handler creates a new word named foo and starts the
compiler.
#1
RETRO sees that # is a prefix and calls the handler for it.
The handler converts the token to a number and then pushes the
value to the stack. Since the Compiler is now active, it then
pops the value off the stack and compiles it into the current
definition.
#2
And again, but compile a 2 instead of a 1.
RETRO does not find a + prefix, so it searches the dictionary.
Upon finding +, it pushes the address (xt) to the stack and
calls the corresponding class handler. The class handler for
normal words calls the code at the address if interpreting, or
compiles a call to it if the Compiler is active.
n:put
The process is repeated for n:put.
;
The last word has a slight difference. Like + and n:put,
this is a word, not a prefixed token. But the class handler
for this one always calls the associated code. In this case,
; is the word which ends a definition and turns off the
Compiler.
Hyperstatic Global Environment
This now brings up an interesting subpoint. RETRO provides what is sometimes called a hyper-static global environment. This can be difficult to explain, so let's take a quick look at how it works:
:scale (x-y) A fetch * ;
>>> A ?
#1000 'A var<n>
:scale (x-y) A fetch * ;
#3 scale n:put
>>> 3000
#100 A store
#3 scale n:put
>>> 300
#5 'A var<n>
#3 scale n:put
>>> 300
A fetch n:put
>>> 5
Output is marked with >>>.
Note that we create two variables with the same name (A). The definition for scale still refers to the old variable, even though we can no longer directly manipulate it.
In a hyper-static global environment, functions continue to refer to the variables and functions that existed when they were defined. If you create a new variable or function with the same name as an existing one, it only affects future code.
Take advantage of this to reuse short names whenever it helps to make your code easier to understand.
Classes
The interpreter is unaware of how to handle a dictionary entry and has no concept of the difference between compiling and interpreting.
The actual work is handled by something I call class handlers.
Each dictionary header contains a variety of information:
+--------+------------------+
| Offset | Description |
+========+==================+
| 0 | link to previous |
+--------+------------------+
| 1 | class handler |
+--------+------------------+
| 2 | xt |
+--------+------------------+
| 3+ | name of function |
+--------+------------------+
When a token is found, the listener pushes the contents of the xt field to the stack and then calls the function that the class handler field points to.
This model differs from traditional Forth since the interpreter is unaware of how tokens are handled. All actions are performed by the class handlers, which allows for easy addition of new categories of words and functionality.
Data Structures
You can create named data structures using d:create, var,
var<n>, and const.
Constants
These are the simplest data structure. The xt is set to a value, which is either left on the stack or compiled into a definition.
#100 'ONE-HUNDRED const
By convention, constants in RETRO should have names in all uppercase.
Variables
A variable is a named pointer to a memory location holding a value that may change over time. RETRO provides two ways to create a variable:
'A var
The first, using var, creates a name and allocates one cell
for storage. The memory is initialized to zero.
#10 'B var<n>
The second, var<n>, takes a value from the stack, and creates
a name, allocates one cell for storage, and then initializes it
to the value specified.
This is cleaner than doing:
'A var
#10 &A store
Custom Structures
You can also create custom data structures by creating a name, and allocating space yourself. For instance:
'Test d:create
#10 , #20 , #30 ,
This would create a data structure named Test, with three
values, initialized to 10, 20, and 30. The values would be
stored in consecutive memory locations.
If you want to allocate a specific amount of space for future
use, you could use allot here:
'Buffer d:create
#2048 allot
The use of allot reserves space, but does not initialize the
space.
Tip: If you want to deallocate space, pass a negative value to
allot. Be careful though: if you have added new words or
data after the allocation this will cause them to be
overwritten, which can lead to crashes or bugs.
Strings
In addition to the basic data structures above, RETRO also provides support for string data.
Creating a string simply requires using the ' prefix:
'this_is_a_string
'__this_string_has_leading_and_trailing_spaces__
When creating strings, RETRO uses a floating, rotating buffer for temporary strings. Strings created in a definition are permanent, but mutable.
Note the use of underscores in place of spaces. Since strings are handled by a prefix handler, the token can not contain whitespace characters. RETRO will convert the underscores into spaces by default.
Length
You can obtain the length of a string using s:length:
'this_is_a_string s:length
Comparisons
Strings can be compared using s:eq?:
'test_1 'test_2 s:eq? n:put
>>> 0
'test_3 'test_3 s:eq? n:put
>>> -1
The comparisons are case sensitive.
============= To Be Continued ...
Quotes and Combinators
RETRO leverages two concepts that need some explanation. These are quotes and combinators.
A quote is simply an anonymous, nestable function. They can be created at any time.
Example:
#12 [ dup * ] call
In this, the code stating with [ and ending with ] is the
quote. Here it's just called immediately, but you can also
pass it to other words.
We use the word combinator to refer to words that operate on quotes.
You'll use quotes and combinators extensively for controlling the flow of execution. This begins with conditionals.
Assuming that we have a flag on the stack, you can run a quote
if the flag is TRUE:
#1 #2 eq?
[ 'True! s:put ] if
Or if it's FALSE:
#1 #2 eq?
[ 'Not_true! s:put ] -if
There's also a choose combinator:
#1 #2 eq?
[ 'True! s:put ]
[ 'Not_true! s:put ] choose
RETRO also uses combinators for loops.
A counted loop takes a count and a quote:
#0 #100 [ dup n:put sp n:inc ] times
You can also loop while a quote returns a flag of TRUE:
#0 [ n:inc dup #100 lt? ] while
Or while it returns FALSE:
#100 [ n:dec dup n:zero? ] until
Combinators are also used to iterate over data structures. For
instance, many structures provide a for-each combinator which
can be run once for each item in the structure. E.g., with a
string:
'Hello [ c:put ] s:for-each
Moving further, combinators are also used for filters and operations on data. Again with strings:
'Hello_World! [ c:-vowel? ] s:filter
This runs s:filter which takes a quote returning a flag. For
each TRUE it appends the character into a new string, while
FALSE results are discarded.
You might also use a mapping combinator to update a data set:
'Hello_World [ c:to-upper ] s:map
This takes a quote that modifies a value which is then used to build a new string.
There are many more combinators. Look in the Glossary to find them. Some notable ones include:
bi
bi*
bi@
tri
tri*
tri@
dip
sip
I/O
RETRO provides three words for interacting with I/O. These are:
io:enumerate returns the number of attached devices
io:query returns information about a device
io:invoke invokes an interaction with a device
As an example, with an implementation providing an output source, a block storage system, and keyboard:
io:enumerate will return `3` since there are three
i/o devices
#0 io:query will return 0 0, since the first device
is a screen (type 0) with a version of 0
#1 io:query will return 1 3, since the second device is
block storage (type 3), with a version of 1
#2 io:query will return 0 1, since the last device is a
keyboard (type 1), with a version of 0
In this case, some interactions can be defined:
:c:put #0 io:invoke ;
:c:get #2 io:invoke ;
Setup the stack, push the device ID, and then use io:invoke
to invoke the interaction.
A RETRO system requires one I/O device (a generic output for a single character). This must be the first device, and must have a device ID of 0.
All other devices are optional and can be specified in any order.
Q & A
Q: Is RETRO Actually Useful?
A: Yes. I use it daily for handling a large number of small tasks. The HTTP and Gopher servers in the examples host both my personal sites and a half dozen other small websites. Most programs I write start off as prototypes in RETRO, even though I may need to rewrite them in other languages due to project requirements.
On a larger scale, RETRO code is used in an order managment system used by an electrial wholesale company, where it gets used daily.
It's also been used in data analysis, in an embedded prototype for vehicle tracking, and as part of an energy auditing tool.
Q: What Systems Are Supported?
A: Many. RETRO has a flexible I/O model that allows it to be tailored to a variety of hosts. The most full featured systems are BSD Unix, Linux, macOS, and iOS. It also runs on Windows, Haiku, FreeDOS, and directly on raw x86 hardware. It's been built and reported to work on x86, x86-64, MIPS32, ARM, and PowerPC.