retroforth/Handbook.md
crc 7e90cdfb61 add current draft of handbook
FossilOrigin-Name: e79065349f63b7dc873eba77060386eef743d6db98f6f758d3635f55ac35c84c
2018-12-07 16:41:14 +00:00

23 KiB

RETRO: a Modern, Pragmatic Forth

Welcome to RETRO, my personal take on the Forth language. This is a modern system primarily targetting desktop, mobile, and servers, though it can also be used on some larger (ARM, MIPS32) embedded systems.

The language is Forth. It is untyped, uses a stack to pass data between functions called words, and a dictionary which tracks the word names and data structures.

But it's not a traditional Forth. RETRO draws influences from many sources and takes a unique approach to the language.

RETRO has a large vocabulary of words. Keeping a copy of the Glossary on hand is highly recommended as you learn to use RETRO.

This book is structured into topics.


Quick Start

Get RETRO

RETRO can be obtained at:

Build From Source

Run make to compile RETRO and its toolchain. The binaries will be placed in the bin directory.

If the build fails, try make -f Makefile.alternate.

There are various binaries. The main one is retro. Copy this to a place in your $PATH and you'll be set to run it.

Use A Prebuilt Binary

Go to forthworks.com/retro and download the binaries collection. Copy the appropriate binary to a location in your path.

Use An OS Package

RETRO is packaged for a few systems.

TODO: add details here.

Commercial Versions

iOS

macOS

Run RETRO

RETRO can be run for scripting or interactive use. To start it interactively, run: retro -i or retro -c.

For a summary of the full command line arguments available:

Scripting Usage:

    retro filename [script arguments...]

Interactive Usage:

    retro [-h] [-i] [-c] [-s] [-f filename] [-t]

  -h           Display this help text
  -i           Interactive mode (line buffered)
  -c           Interactive mode (character buffered)
  -s           Suppress the 'ok' prompt and keyboard
               echo in interactive mode
  -f filename  Run the contents of the specified file
  -t           Run tests (in ``` blocks) in any loaded files

The Virtual Machine

RETRO runs on a virtual machine called Nga. This emulates a MISC (minimal instruction set computer), with 30 instructions and two stacks.

Memory is provided as a linear array of signed 32-bit values. There is no direct addressing of other values.

Instruction Set

00 NOP

NOP does nothing. It's used for padding and reserving space.

01 LIT

LIT pushes the value in the following cell to the data stack. This is the only instruction which takes a value from a source other than the stack.

02 DUP

DUP duplicates the top value on the stack.

03 DROP

DROP discards the top item on the stack.

04 SWAP

SWAP switches the positions of the top two items on the stack.

05 PUSH

PUSH moves a value from the data stack to the address stack.

06 POP

POP moves a value from the address stack to the data stack.

07 JUMP

JUMP takes an address from the stack and transfers control to the code at the specified address.

08 CALL

CALL calls a subroutine at the address on the top of the stack.

09 CCALL

CCALL is a conditional call. It takes two values: a flag and a pointer for an address to jump to if the flag is true.

At the instruction level, a false flag is zero and any other value is true.

10 RETURN

RETURN ends a subroutine and returns flow to the instruction following the last CALL or CCALL.

11 EQ

EQ compares two values for equality and returns a flag.

12 NEQ

NEQ compares two values for inequality and returns a flag.

13 LT

LT compares two values for less than and returns a flag.

14 GT

GT compares two values for greater than and returns a flag.

15 FETCH

FETCH takes an address and returns the value stored there.

This doubles as a means of introspection into the VM state. Negative addresses correspond to VM queries:

| Address | Returns             |
| ------- | ------------------- |
| -1      | Data stack depth    |
| -2      | Address stack depth |
| -3      | Maximum Image Size  |

An implementation may use negative values below -100 for implementation specific inquiries.

16 STORE

STORE stores a value into an address.

17 ADD

ADD adds two numbers together.

18 SUB

SUB subtracts two numbers.

19 MUL

MUL multiplies two numbers.

20 DIVMOD

DIVMOD divides and returns the quotient and remainder.

21 AND

AND performs a bitwise AND operation.

+----------------+
| Before | After |
| ------ | ----- |
| -1     | -1    |
| -1     |       |
+----------------+

+----------------+
| Before | After |
| ------ | ----- |
|  0     |  0    |
| -1     |       |
+----------------+

+----------------+
| Before | After |
| ------ | ----- |
| -1     |  0    |
|  0     |       |
+----------------+

22 OR

OR performs a bitwise OR operation.

+----------------+
| Before | After |
| ------ | ----- |
| -1     | -1    |
| -1     |       |
+----------------+

+----------------+
| Before | After |
| ------ | ----- |
| -1     | -1    |
|  0     |       |
+----------------+

+----------------+
| Before | After |
| ------ | ----- |
|  0     | -1    |
| -1     |       |
+----------------+

+----------------+
| Before | After |
| ------ | ----- |
|  0     |  0    |
|  0     |       |
+----------------+

23 XOR

XOR performs a bitwise XOR operation.

+----------------+
| Before | After |
| ------ | ----- |
| -1     |  0    |
| -1     |       |
+----------------+

+----------------+
| Before | After |
| ------ | ----- |
|  0     | -1    |
| -1     |       |
+----------------+

+----------------+
| Before | After |
| ------ | ----- |
| -1     | -1    |
|  0     |       |
+----------------+

+----------------+
| Before | After |
| ------ | ----- |
|  0     |  0    |
|  0     |       |
+----------------+

24 SHIFT

SHIFT performs a bitwise arithmetic SHIFT operation.

This takes two values:

xy

And returns a single one:

z

If y is positive, this shifts right. If negative, it shifts left.

25 ZRET

ZRET returns from a subroutine if the top item on the stack is zero. If not, it acts like a NOP instead.

26 END

END tells the VM to shut down.

27 IENUMERATE

IENUMERATE pushes the number of simulated I/O devices to the stack.

28 IQUERY

IQUERY takes a device number from the stack and queries it, returning two values: a device version number and a device type identifier.

29 IINTERACT

IINTERACT takes a device number from the stack and triggers an interaction with it.

Instruction Encoding

Nga allows up to four instructions to be packed into a single memory cell. These are decoded and executed sequentially.

first    second   third   fourth
00000000 00000000 0000000 0000000

Any unused instructions should be replaced with NOP (00000000). If an instruction modifies the instruction pointer all later instructions should be NOP. This is because the later slots will be executed prior to the IP change. Consider:

LIT CALL LIT CALL

In this case only the second CALL actually gets control. The instructions which modify IP are:

JUMP
CALL
CCALL
RETURN
ZRET

The LIT instruction takes a value from the following cell and pushes it to the stack. Multiple LIT's will use sequential cells. E.g.,

LIT LIT ADD NOP
100
200

Muri Assembly

The initial kernel is written in Nga assembly and is built using an assembler named Muri. This is a simple, multipass model that's not fancy, but suffices for RETRO's needs.

A small example:

~~~
i liju....
r main
: c:put
i liiire..
i 0
: main
i lilica..
d 97
i liju....
r main
~~~

So breaking down: Muri extracts the assembly code blocks to assemble, then proceeds to do the assembly. Each source line starts with a directive, followed by a space, and then ending with a value.

The directives are:

:    value is a label
i    value is an instruction bundle
d    value is a numeric value
r    value is a reference
s    value is a string to inline

Instructions for Nga are provided as bundles. Each memory location can store up to four instructions. And each instruction gets a two character identifier.

From the list of instructions:

0  nop     7  jump    14  gt      21  and    28  io query
1  lit     8  call    15  fetch   22  or     29  io interact
2  dup     9  ccall   16  store   23  xor
3  drop   10  return  17  add     24  shift
4  swap   11  eq      18  sub     25  zret
5  push   12  neq     19  mul     26  end
6  pop    13  lt      20  divmod  27  io enumerate

This boils down to:

0  ..    7  ju   14  gt   21  an   28  iq
1  li    8  ca   15  fe   22  or   29  ii
2  du    9  cc   16  st   23  xo
3  dr   10  re   17  ad   24  sh
4  sw   11  eq   18  su   25  zr
5  pu   12  ne   19  mu   26  en
6  po   13  lt   20  di   27  ie

Most are just the first two letters of the instruction name. I use .. instead of no for NOP, and the first letter of each I/O instruction name. So a bundle may look like:

dumure..

(This would correspond to dup multiply return nop).

RETRO also has a runtime variation of Muri that can be used when you need to generate more optimal code. So one can write:

:n:square dup * ;

Or:

:n:square as{ 'dumure.. i }as ;

The second one will be faster, as the entire definition is one bundle, which reduces memory reads and decoding by 2/3.

Doing this is less readable, so I only recommend doing so after you have finalized working RETRO level code and determined the best places to optimize.


Source Format

RETRO is literate. Most of the sources are in a format called Unu. This allows easy mixing of commentary and code blocks, making it simple to document the code.

As an example,

# Determine The Average Word Name Length

To determine the average length of a word name two values
are needed. First, the total length of all names in the
Dictionary:

~~~
#0 [ d:name s:length + ] d:for-each
~~~

And then the number of words in the Dictionary:

~~~
#0 [ drop n:inc ] d:for-each
~~~

With these, a simple division is all that's left.

~~~
/
~~~

Finally, display the results:

~~~
'Average_name_length:_%n\n s:format s:put
~~~

This illustrates the format. Only code in the fenced blocks (between ~~~ pairs) get extracted and run.

(Note: this only applies to source files; fences are not used when entering code interactively).


Syntax

Forth is largely freeform, and RETRO generally follows this. But there are a few things that impact this.

First, RETRO processes code as encountered, token by token. A standard system does not expose the parser to the user.

Secondly, RETRO uses single character prefixes to guide the processing of a token.

The interpreter takes a look at the first character of a token. If this matches a known prefix, the rest of the token is passed to a function which handles that group of tokens. If no valid prefix is found, RETRO tries to find the word in the dictionary. If successful, the information in the dictionary header is used to carry out the actions specified in the name's definition. If this also fails, RETRO calls err:notfound to report the error.

A flowchart of the interpreter process:

+-------------+
| Get a Token |
+-------------+
       |
+------------------------------------+
| Is first character a valid prefix? |
+------------------------------------+
            |                     |
           YES                   NO
            |                    |
+-----------------------+   +------------------------+
| Pass rest of token to |   | Is entire token a name |
| prefix handler        |   | in the dictionary?     |
+-----------------------+   +------------------------+
                                |                |
                               YES              NO
                                |               |
                +------------------+   +-------------------+
                | Push XT to stack |   | Call err:notfound |
                +------------------+   +-------------------+
                          |
               +--------------------+
               | Call class handler |
               +--------------------+

In RETRO, prefix handlers and class handlers are responsible for dealing with the tokens and words. This includes interpret and compilation as necessary.

RETRO permits names to contain any characters other than space, tab, cr, dnd lf. Names are case sensitive, so the following are three different names:

foo Foo FOO

I recommend that names not start with a prefix as prefixes are checked prior to dictionary lookups. So a word named @foo would be treated as a fetch from a variable named foo instead of running the @foo word.

Tip: Run 'prefix: d:words-with to get a list of prefix handlers in your system.


The Compiler

To create new functions, you use the compiler. This is generally started by using the : (pronounced colon) prefix. A simple example:

:foo #1 #2 + n:put ;

Breaking this apart:

:foo

RETRO sees that : is a prefix and calls the handler for it. The handler creates a new word named foo and starts the compiler.

#1

RETRO sees that # is a prefix and calls the handler for it. The handler converts the token to a number and then pushes the value to the stack. Since the Compiler is now active, it then pops the value off the stack and compiles it into the current definition.

#2

And again, but compile a 2 instead of a 1.

RETRO does not find a + prefix, so it searches the dictionary. Upon finding +, it pushes the address (xt) to the stack and calls the corresponding class handler. The class handler for normal words calls the code at the address if interpreting, or compiles a call to it if the Compiler is active.

n:put

The process is repeated for n:put.

;

The last word has a slight difference. Like + and n:put, this is a word, not a prefixed token. But the class handler for this one always calls the associated code. In this case, ; is the word which ends a definition and turns off the Compiler.


Hyperstatic Global Environment

This now brings up an interesting subpoint. RETRO provides what is sometimes called a hyper-static global environment. This can be difficult to explain, so let's take a quick look at how it works:

:scale (x-y) A fetch * ;
>>> A ?
#1000 'A var<n>
:scale (x-y) A fetch * ;
#3 scale n:put
>>> 3000
#100 A store
#3 scale n:put
>>> 300
#5 'A var<n>
#3 scale n:put
>>> 300
A fetch n:put
>>> 5

Output is marked with >>>.

Note that we create two variables with the same name (A). The definition for scale still refers to the old variable, even though we can no longer directly manipulate it.

In a hyper-static global environment, functions continue to refer to the variables and functions that existed when they were defined. If you create a new variable or function with the same name as an existing one, it only affects future code.

Take advantage of this to reuse short names whenever it helps to make your code easier to understand.


Classes

The interpreter is unaware of how to handle a dictionary entry and has no concept of the difference between compiling and interpreting.

The actual work is handled by something I call class handlers.

Each dictionary header contains a variety of information:

+--------+------------------+
| Offset | Description      |
+========+==================+
| 0      | link to previous |
+--------+------------------+
| 1      | class handler    |
+--------+------------------+
| 2      | xt               |
+--------+------------------+
| 3+     | name of function |
+--------+------------------+

When a token is found, the listener pushes the contents of the xt field to the stack and then calls the function that the class handler field points to.

This model differs from traditional Forth since the interpreter is unaware of how tokens are handled. All actions are performed by the class handlers, which allows for easy addition of new categories of words and functionality.


Data Structures

You can create named data structures using d:create, var, var<n>, and const.

Constants

These are the simplest data structure. The xt is set to a value, which is either left on the stack or compiled into a definition.

#100 'ONE-HUNDRED const

By convention, constants in RETRO should have names in all uppercase.

Variables

A variable is a named pointer to a memory location holding a value that may change over time. RETRO provides two ways to create a variable:

'A var

The first, using var, creates a name and allocates one cell for storage. The memory is initialized to zero.

#10 'B var<n>

The second, var<n>, takes a value from the stack, and creates a name, allocates one cell for storage, and then initializes it to the value specified.

This is cleaner than doing:

'A var
#10 &A store

Custom Structures

You can also create custom data structures by creating a name, and allocating space yourself. For instance:

'Test d:create
  #10 , #20 , #30 ,

This would create a data structure named Test, with three values, initialized to 10, 20, and 30. The values would be stored in consecutive memory locations.

If you want to allocate a specific amount of space for future use, you could use allot here:

'Buffer d:create
  #2048 allot

The use of allot reserves space, but does not initialize the space.

Tip: If you want to deallocate space, pass a negative value to allot. Be careful though: if you have added new words or data after the allocation this will cause them to be overwritten, which can lead to crashes or bugs.

Strings

In addition to the basic data structures above, RETRO also provides support for string data.

Creating a string simply requires using the ' prefix:

'this_is_a_string
'__this_string_has_leading_and_trailing_spaces__

When creating strings, RETRO uses a floating, rotating buffer for temporary strings. Strings created in a definition are permanent, but mutable.

Note the use of underscores in place of spaces. Since strings are handled by a prefix handler, the token can not contain whitespace characters. RETRO will convert the underscores into spaces by default.

Length

You can obtain the length of a string using s:length:

'this_is_a_string s:length

Comparisons

Strings can be compared using s:eq?:

'test_1  'test_2  s:eq? n:put
>>> 0
'test_3  'test_3  s:eq? n:put
>>> -1

The comparisons are case sensitive.

============= To Be Continued ...

Quotes and Combinators

RETRO leverages two concepts that need some explanation. These are quotes and combinators.

A quote is simply an anonymous, nestable function. They can be created at any time.

Example:

#12 [ dup * ] call

In this, the code stating with [ and ending with ] is the quote. Here it's just called immediately, but you can also pass it to other words.

We use the word combinator to refer to words that operate on quotes.

You'll use quotes and combinators extensively for controlling the flow of execution. This begins with conditionals.

Assuming that we have a flag on the stack, you can run a quote if the flag is TRUE:

#1 #2 eq?
[ 'True! s:put ] if

Or if it's FALSE:

#1 #2 eq?
[ 'Not_true! s:put ] -if

There's also a choose combinator:

#1 #2 eq?
[ 'True! s:put     ]
[ 'Not_true! s:put ] choose

RETRO also uses combinators for loops.

A counted loop takes a count and a quote:

#0 #100 [ dup n:put sp n:inc ] times

You can also loop while a quote returns a flag of TRUE:

#0 [ n:inc dup #100 lt? ] while

Or while it returns FALSE:

#100 [ n:dec dup n:zero? ] until

Combinators are also used to iterate over data structures. For instance, many structures provide a for-each combinator which can be run once for each item in the structure. E.g., with a string:

'Hello [ c:put ] s:for-each

Moving further, combinators are also used for filters and operations on data. Again with strings:

'Hello_World! [ c:-vowel? ] s:filter

This runs s:filter which takes a quote returning a flag. For each TRUE it appends the character into a new string, while FALSE results are discarded.

You might also use a mapping combinator to update a data set:

'Hello_World [ c:to-upper ] s:map

This takes a quote that modifies a value which is then used to build a new string.

There are many more combinators. Look in the Glossary to find them. Some notable ones include:

bi
bi*
bi@
tri
tri*
tri@
dip
sip

I/O

RETRO provides three words for interacting with I/O. These are:

io:enumerate    returns the number of attached devices
io:query        returns information about a device
io:invoke       invokes an interaction with a device

As an example, with an implementation providing an output source, a block storage system, and keyboard:

io:enumerate    will return `3` since there are three
                i/o devices
#0 io:query     will return 0 0, since the first device
                is a screen (type 0) with a version of 0
#1 io:query     will return 1 3, since the second device is
                block storage (type 3), with a version of 1
#2 io:query     will return 0 1, since the last device is a
                keyboard (type 1), with a version of 0

In this case, some interactions can be defined:

:c:put #0 io:invoke ;
:c:get #2 io:invoke ;

Setup the stack, push the device ID, and then use io:invoke to invoke the interaction.

A RETRO system requires one I/O device (a generic output for a single character). This must be the first device, and must have a device ID of 0.

All other devices are optional and can be specified in any order.


Q & A

Q: Is RETRO Actually Useful?

A: Yes. I use it daily for handling a large number of small tasks. The HTTP and Gopher servers in the examples host both my personal sites and a half dozen other small websites. Most programs I write start off as prototypes in RETRO, even though I may need to rewrite them in other languages due to project requirements.

On a larger scale, RETRO code is used in an order managment system used by an electrial wholesale company, where it gets used daily.

It's also been used in data analysis, in an embedded prototype for vehicle tracking, and as part of an energy auditing tool.

Q: What Systems Are Supported?

A: Many. RETRO has a flexible I/O model that allows it to be tailored to a variety of hosts. The most full featured systems are BSD Unix, Linux, macOS, and iOS. It also runs on Windows, Haiku, FreeDOS, and directly on raw x86 hardware. It's been built and reported to work on x86, x86-64, MIPS32, ARM, and PowerPC.