retroforth/Handbook.md
crc 7e90cdfb61 add current draft of handbook
FossilOrigin-Name: e79065349f63b7dc873eba77060386eef743d6db98f6f758d3635f55ac35c84c
2018-12-07 16:41:14 +00:00

938 lines
23 KiB
Markdown

# RETRO: a Modern, Pragmatic Forth
Welcome to RETRO, my personal take on the Forth language. This
is a modern system primarily targetting desktop, mobile, and
servers, though it can also be used on some larger (ARM, MIPS32)
embedded systems.
The language is Forth. It is untyped, uses a stack to pass data
between functions called words, and a dictionary which tracks
the word names and data structures.
But it's not a traditional Forth. RETRO draws influences from
many sources and takes a unique approach to the language.
RETRO has a large vocabulary of words. Keeping a copy of the
Glossary on hand is highly recommended as you learn to use RETRO.
This book is structured into topics.
---
# Quick Start
## Get RETRO
RETRO can be obtained at:
- [https://forthworks.com/retro](https://forthworks.com/retro)
- [gopher://forthworks.com/1/retro](gopher://forthworks.com/1/retro)
### Build From Source
Run `make` to compile RETRO and its toolchain. The binaries will
be placed in the `bin` directory.
If the build fails, try `make -f Makefile.alternate`.
There are various binaries. The main one is `retro`. Copy this
to a place in your $PATH and you'll be set to run it.
### Use A Prebuilt Binary
Go to forthworks.com/retro and download the binaries collection.
Copy the appropriate binary to a location in your path.
### Use An OS Package
RETRO is packaged for a few systems.
TODO: add details here.
### Commercial Versions
#### iOS
#### macOS
## Run RETRO
RETRO can be run for scripting or interactive use. To start it
interactively, run: `retro -i` or `retro -c`.
For a summary of the full command line arguments available:
Scripting Usage:
retro filename [script arguments...]
Interactive Usage:
retro [-h] [-i] [-c] [-s] [-f filename] [-t]
-h Display this help text
-i Interactive mode (line buffered)
-c Interactive mode (character buffered)
-s Suppress the 'ok' prompt and keyboard
echo in interactive mode
-f filename Run the contents of the specified file
-t Run tests (in ``` blocks) in any loaded files
---
# The Virtual Machine
RETRO runs on a virtual machine called Nga. This emulates a MISC
(minimal instruction set computer), with 30 instructions and two
stacks.
Memory is provided as a linear array of signed 32-bit values.
There is no direct addressing of other values.
## Instruction Set
### 00 NOP
**NOP** does nothing. It's used for padding and reserving space.
### 01 LIT
**LIT** pushes the value in the following cell to the data
stack. This is the only instruction which takes a value from
a source other than the stack.
### 02 DUP
**DUP** duplicates the top value on the stack.
### 03 DROP
**DROP** discards the top item on the stack.
### 04 SWAP
**SWAP** switches the positions of the top two items on the
stack.
### 05 PUSH
**PUSH** moves a value from the data stack to the address stack.
### 06 POP
**POP** moves a value from the address stack to the data stack.
### 07 JUMP
**JUMP** takes an address from the stack and transfers control
to the code at the specified address.
### 08 CALL
**CALL** calls a subroutine at the address on the top of the
stack.
### 09 CCALL
**CCALL** is a conditional call. It takes two values: a flag and
a pointer for an address to jump to if the flag is true.
At the instruction level, a false flag is zero and any other
value is true.
### 10 RETURN
**RETURN** ends a subroutine and returns flow to the instruction
following the last **CALL** or **CCALL**.
### 11 EQ
**EQ** compares two values for equality and returns a flag.
### 12 NEQ
**NEQ** compares two values for inequality and returns a flag.
### 13 LT
**LT** compares two values for less than and returns a flag.
### 14 GT
**GT** compares two values for greater than and returns a flag.
### 15 FETCH
**FETCH** takes an address and returns the value stored there.
This doubles as a means of introspection into the VM state.
Negative addresses correspond to VM queries:
| Address | Returns |
| ------- | ------------------- |
| -1 | Data stack depth |
| -2 | Address stack depth |
| -3 | Maximum Image Size |
An implementation may use negative values below -100 for
implementation specific inquiries.
### 16 STORE
**STORE** stores a value into an address.
### 17 ADD
**ADD** adds two numbers together.
### 18 SUB
**SUB** subtracts two numbers.
### 19 MUL
**MUL** multiplies two numbers.
### 20 DIVMOD
**DIVMOD** divides and returns the quotient and remainder.
### 21 AND
**AND** performs a bitwise AND operation.
+----------------+
| Before | After |
| ------ | ----- |
| -1 | -1 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | 0 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| -1 | 0 |
| 0 | |
+----------------+
### 22 OR
**OR** performs a bitwise OR operation.
+----------------+
| Before | After |
| ------ | ----- |
| -1 | -1 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| -1 | -1 |
| 0 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | -1 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | 0 |
| 0 | |
+----------------+
### 23 XOR
**XOR** performs a bitwise XOR operation.
+----------------+
| Before | After |
| ------ | ----- |
| -1 | 0 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | -1 |
| -1 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| -1 | -1 |
| 0 | |
+----------------+
+----------------+
| Before | After |
| ------ | ----- |
| 0 | 0 |
| 0 | |
+----------------+
### 24 SHIFT
**SHIFT** performs a bitwise arithmetic SHIFT operation.
This takes two values:
xy
And returns a single one:
z
If `y` is positive, this shifts right. If negative, it shifts
left.
### 25 ZRET
**ZRET** returns from a subroutine if the top item on the stack
is zero. If not, it acts like a **NOP** instead.
### 26 END
**END** tells the VM to shut down.
### 27 IENUMERATE
**IENUMERATE** pushes the number of simulated I/O devices to the
stack.
### 28 IQUERY
**IQUERY** takes a device number from the stack and queries it,
returning two values: a device version number and a device type
identifier.
### 29 IINTERACT
**IINTERACT** takes a device number from the stack and triggers
an interaction with it.
## Instruction Encoding
Nga allows up to four instructions to be packed into a single
memory cell. These are decoded and executed sequentially.
first second third fourth
00000000 00000000 0000000 0000000
Any unused instructions should be replaced with NOP (00000000).
If an instruction modifies the instruction pointer all later
instructions should be NOP. This is because the later slots
will be executed prior to the IP change. Consider:
LIT CALL LIT CALL
In this case only the second CALL actually gets control. The
instructions which modify IP are:
JUMP
CALL
CCALL
RETURN
ZRET
The LIT instruction takes a value from the following cell
and pushes it to the stack. Multiple LIT's will use sequential
cells. E.g.,
LIT LIT ADD NOP
100
200
---
# Muri Assembly
The initial kernel is written in Nga assembly and is built using
an assembler named **Muri**. This is a simple, multipass model
that's not fancy, but suffices for RETRO's needs.
A small example:
~~~
i liju....
r main
: c:put
i liiire..
i 0
: main
i lilica..
d 97
i liju....
r main
~~~
So breaking down: Muri extracts the assembly code blocks to
assemble, then proceeds to do the assembly. Each source line
starts with a directive, followed by a space, and then ending
with a value.
The directives are:
: value is a label
i value is an instruction bundle
d value is a numeric value
r value is a reference
s value is a string to inline
Instructions for Nga are provided as bundles. Each memory
location can store up to four instructions. And each instruction
gets a two character identifier.
From the list of instructions:
0 nop 7 jump 14 gt 21 and 28 io query
1 lit 8 call 15 fetch 22 or 29 io interact
2 dup 9 ccall 16 store 23 xor
3 drop 10 return 17 add 24 shift
4 swap 11 eq 18 sub 25 zret
5 push 12 neq 19 mul 26 end
6 pop 13 lt 20 divmod 27 io enumerate
This boils down to:
0 .. 7 ju 14 gt 21 an 28 iq
1 li 8 ca 15 fe 22 or 29 ii
2 du 9 cc 16 st 23 xo
3 dr 10 re 17 ad 24 sh
4 sw 11 eq 18 su 25 zr
5 pu 12 ne 19 mu 26 en
6 po 13 lt 20 di 27 ie
Most are just the first two letters of the instruction name. I
use `..` instead of `no` for `NOP`, and the first letter of
each I/O instruction name. So a bundle may look like:
dumure..
(This would correspond to `dup multiply return nop`).
RETRO also has a runtime variation of Muri that can be used
when you need to generate more optimal code. So one can write:
:n:square dup * ;
Or:
:n:square as{ 'dumure.. i }as ;
The second one will be faster, as the entire definition is one
bundle, which reduces memory reads and decoding by 2/3.
Doing this is less readable, so I only recommend doing so after
you have finalized working RETRO level code and determined the
best places to optimize.
---
# Source Format
RETRO is literate. Most of the sources are in a format called
Unu. This allows easy mixing of commentary and code blocks,
making it simple to document the code.
As an example,
# Determine The Average Word Name Length
To determine the average length of a word name two values
are needed. First, the total length of all names in the
Dictionary:
~~~
#0 [ d:name s:length + ] d:for-each
~~~
And then the number of words in the Dictionary:
~~~
#0 [ drop n:inc ] d:for-each
~~~
With these, a simple division is all that's left.
~~~
/
~~~
Finally, display the results:
~~~
'Average_name_length:_%n\n s:format s:put
~~~
This illustrates the format. Only code in the fenced blocks
(between \~~~ pairs) get extracted and run.
(Note: this only applies to *source files*; fences are not used
when entering code interactively).
---
# Syntax
Forth is largely freeform, and RETRO generally follows this. But
there are a few things that impact this.
First, RETRO processes code as encountered, token by token. A
standard system does not expose the parser to the user.
Secondly, RETRO uses single character prefixes to guide the
processing of a token.
The interpreter takes a look at the first character of a token.
If this matches a known *prefix*, the rest of the token is
passed to a function which handles that group of tokens. If no
valid prefix is found, RETRO tries to find the word in the
dictionary. If successful, the information in the *dictionary
header* is used to carry out the actions specified in the name's
*definition*. If this also fails, RETRO calls `err:notfound` to
report the error.
A flowchart of the interpreter process:
+-------------+
| Get a Token |
+-------------+
|
+------------------------------------+
| Is first character a valid prefix? |
+------------------------------------+
| |
YES NO
| |
+-----------------------+ +------------------------+
| Pass rest of token to | | Is entire token a name |
| prefix handler | | in the dictionary? |
+-----------------------+ +------------------------+
| |
YES NO
| |
+------------------+ +-------------------+
| Push XT to stack | | Call err:notfound |
+------------------+ +-------------------+
|
+--------------------+
| Call class handler |
+--------------------+
In RETRO, *prefix handlers* and *class handlers* are responsible
for dealing with the tokens and words. This includes interpret
and compilation as necessary.
RETRO permits names to contain any characters other than space,
tab, cr, dnd lf. Names are *case sensitive*, so the following
are three *different* names:
foo Foo FOO
I recommend that names not start with a *prefix* as prefixes
are checked prior to dictionary lookups. So a word named `@foo`
would be treated as a fetch from a variable named `foo` instead
of running the `@foo` word.
Tip: Run `'prefix: d:words-with` to get a list of prefix
handlers in your system.
---
# The Compiler
To create new functions, you use the compiler. This is generally
started by using the `:` (pronounced *colon*) prefix. A simple
example:
:foo #1 #2 + n:put ;
Breaking this apart:
:foo
RETRO sees that `:` is a prefix and calls the handler for it.
The handler creates a new word named *foo* and starts the
compiler.
#1
RETRO sees that `#` is a prefix and calls the handler for it.
The handler converts the token to a number and then pushes the
value to the stack. Since the `Compiler` is now active, it then
pops the value off the stack and compiles it into the current
definition.
#2
And again, but compile a *2* instead of a *1*.
+
RETRO does not find a `+` prefix, so it searches the dictionary.
Upon finding `+`, it pushes the address (*xt*) to the stack and
calls the corresponding class handler. The class handler for
normal words calls the code at the address if interpreting, or
compiles a call to it if the `Compiler` is active.
n:put
The process is repeated for `n:put`.
;
The last word has a slight difference. Like `+` and `n:put`,
this is a word, not a prefixed token. But the class handler
for this one always calls the associated code. In this case,
`;` is the word which ends a definition and turns off the
`Compiler`.
---
# Hyperstatic Global Environment
This now brings up an interesting subpoint. RETRO provides
what is sometimes called a *hyper-static global environment.*
This can be difficult to explain, so let's take a quick look
at how it works:
:scale (x-y) A fetch * ;
>>> A ?
#1000 'A var<n>
:scale (x-y) A fetch * ;
#3 scale n:put
>>> 3000
#100 A store
#3 scale n:put
>>> 300
#5 'A var<n>
#3 scale n:put
>>> 300
A fetch n:put
>>> 5
Output is marked with **>>>**.
Note that we create two variables with the same name (*A*). The
definition for *scale* still refers to the old variable, even
though we can no longer directly manipulate it.
In a hyper-static global environment, functions continue to
refer to the variables and functions that existed when they were
defined. If you create a new variable or function with the same
name as an existing one, it only affects future code.
Take advantage of this to reuse short names whenever it helps to
make your code easier to understand.
---
# Classes
The interpreter is unaware of how to handle a dictionary entry
and has no concept of the difference between compiling and
interpreting.
The actual work is handled by something I call *class handlers*.
Each dictionary header contains a variety of information:
+--------+------------------+
| Offset | Description |
+========+==================+
| 0 | link to previous |
+--------+------------------+
| 1 | class handler |
+--------+------------------+
| 2 | xt |
+--------+------------------+
| 3+ | name of function |
+--------+------------------+
When a token is found, the listener pushes the contents of the
xt field to the stack and then calls the function that the class
handler field points to.
This model differs from traditional Forth since the interpreter
is unaware of how tokens are handled. All actions are performed
by the class handlers, which allows for easy addition of new
categories of words and functionality.
---
# Data Structures
You can create named data structures using `d:create`, `var`,
`var<n>`, and `const`.
## Constants
These are the simplest data structure. The *xt* is set to a
value, which is either left on the stack or compiled into a
definition.
#100 'ONE-HUNDRED const
By convention, constants in RETRO should have names in all
uppercase.
## Variables
A variable is a named pointer to a memory location holding a
value that may change over time. RETRO provides two ways to
create a variable:
'A var
The first, using `var`, creates a name and allocates one cell
for storage. The memory is initialized to zero.
#10 'B var<n>
The second, `var<n>`, takes a value from the stack, and creates
a name, allocates one cell for storage, and then initializes it
to the value specified.
This is cleaner than doing:
'A var
#10 &A store
## Custom Structures
You can also create custom data structures by creating a name,
and allocating space yourself. For instance:
'Test d:create
#10 , #20 , #30 ,
This would create a data structure named `Test`, with three
values, initialized to 10, 20, and 30. The values would be
stored in consecutive memory locations.
If you want to allocate a specific amount of space for future
use, you could use `allot` here:
'Buffer d:create
#2048 allot
The use of `allot` reserves space, but does not initialize the
space.
Tip: If you want to deallocate space, pass a negative value to
`allot`. Be careful though: if you have added new words or
data after the allocation this will cause them to be
overwritten, which can lead to crashes or bugs.
## Strings
In addition to the basic data structures above, RETRO also
provides support for string data.
Creating a string simply requires using the `'` prefix:
'this_is_a_string
'__this_string_has_leading_and_trailing_spaces__
When creating strings, RETRO uses a floating, rotating buffer
for temporary strings. Strings created in a definition are
permanent, but mutable.
Note the use of underscores in place of spaces. Since strings
are handled by a prefix handler, the token can not contain
whitespace characters. RETRO will convert the underscores into
spaces by default.
### Length
You can obtain the length of a string using `s:length`:
'this_is_a_string s:length
### Comparisons
Strings can be compared using `s:eq?`:
'test_1 'test_2 s:eq? n:put
>>> 0
'test_3 'test_3 s:eq? n:put
>>> -1
The comparisons are case sensitive.
============= To Be Continued ...
# Quotes and Combinators
RETRO leverages two concepts that need some explanation. These
are quotes and combinators.
A *quote* is simply an anonymous, nestable function. They can
be created at any time.
Example:
#12 [ dup * ] call
In this, the code stating with `[` and ending with `]` is the
quote. Here it's just `call`ed immediately, but you can also
pass it to other words.
We use the word *combinator* to refer to words that operate on
quotes.
You'll use quotes and combinators extensively for controlling
the flow of execution. This begins with conditionals.
Assuming that we have a flag on the stack, you can run a quote
if the flag is `TRUE`:
#1 #2 eq?
[ 'True! s:put ] if
Or if it's `FALSE`:
#1 #2 eq?
[ 'Not_true! s:put ] -if
There's also a `choose` combinator:
#1 #2 eq?
[ 'True! s:put ]
[ 'Not_true! s:put ] choose
RETRO also uses combinators for loops.
A counted loop takes a count and a quote:
#0 #100 [ dup n:put sp n:inc ] times
You can also loop while a quote returns a flag of `TRUE`:
#0 [ n:inc dup #100 lt? ] while
Or while it returns `FALSE`:
#100 [ n:dec dup n:zero? ] until
Combinators are also used to iterate over data structures. For
instance, many structures provide a `for-each` combinator which
can be run once for each item in the structure. E.g., with a
string:
'Hello [ c:put ] s:for-each
Moving further, combinators are also used for filters and
operations on data. Again with strings:
'Hello_World! [ c:-vowel? ] s:filter
This runs `s:filter` which takes a quote returning a flag. For
each `TRUE` it appends the character into a new string, while
`FALSE` results are discarded.
You might also use a `map`ping combinator to update a data set:
'Hello_World [ c:to-upper ] s:map
This takes a quote that modifies a value which is then used to
build a new string.
There are many more combinators. Look in the Glossary to find
them. Some notable ones include:
bi
bi*
bi@
tri
tri*
tri@
dip
sip
---
# I/O
RETRO provides three words for interacting with I/O. These are:
io:enumerate returns the number of attached devices
io:query returns information about a device
io:invoke invokes an interaction with a device
As an example, with an implementation providing an output source,
a block storage system, and keyboard:
io:enumerate will return `3` since there are three
i/o devices
#0 io:query will return 0 0, since the first device
is a screen (type 0) with a version of 0
#1 io:query will return 1 3, since the second device is
block storage (type 3), with a version of 1
#2 io:query will return 0 1, since the last device is a
keyboard (type 1), with a version of 0
In this case, some interactions can be defined:
:c:put #0 io:invoke ;
:c:get #2 io:invoke ;
Setup the stack, push the device ID, and then use `io:invoke`
to invoke the interaction.
A RETRO system requires one I/O device (a generic output for a
single character). This must be the first device, and must have
a device ID of 0.
All other devices are optional and can be specified in any
order.
---
# Q & A
Q: Is RETRO Actually Useful?
A: Yes. I use it daily for handling a large number of small
tasks. The HTTP and Gopher servers in the examples host
both my personal sites and a half dozen other small
websites. Most programs I write start off as prototypes
in RETRO, even though I may need to rewrite them in other
languages due to project requirements.
On a larger scale, RETRO code is used in an order managment
system used by an electrial wholesale company, where it gets
used daily.
It's also been used in data analysis, in an embedded
prototype for vehicle tracking, and as part of an energy
auditing tool.
Q: What Systems Are Supported?
A: Many. RETRO has a flexible I/O model that allows it to be
tailored to a variety of hosts. The most full featured
systems are BSD Unix, Linux, macOS, and iOS. It also runs
on Windows, Haiku, FreeDOS, and directly on raw x86
hardware. It's been built and reported to work on x86,
x86-64, MIPS32, ARM, and PowerPC.
---