add s:tokenize to stdlib

FossilOrigin-Name: 1c4918bebdacb9e724b29f51f5a80751116c1afbd28606e8a8dfac2a9a71c489

doc/Glossary.txt

@@ -3421,6 +3421,18 @@ Class Handler: class:word | Namespace: s | Interface Layer: all
 
 ----------------------------------------------------------------
 
+s:tokenize
+
+  Data:  sc-a
+  Addr:  -
+  Float: -
+
+Takes a string and a character to use as a separator. It splits the string into a set of substrings and returns a set containing pointers to each of them.
+
+Class Handler: class:word | Namespace: {n/a} | Interface Layer: {n/a}
+
+----------------------------------------------------------------
+
 s:trim
 
   Data:  s-s

literate/RetroForth.md

@@ -907,6 +907,24 @@ located.
 }}
 ~~~
 
+`s:tokenize` takes a string and a character to use as a separator. It
+splits the string into a set of substrings and returns a set containing
+pointers to each of them.
+
+~~~
+{{
+  'Split-On var
+  :match?    (c-f) @Split-On eq? ;
+  :terminate (s-s) #0 over n:dec store ;
+  :step      (ss-s) [ n:inc ] dip match? [ dup , terminate ] if ;
+---reveal---
+  :s:tokenize (sc-a)
+    !Split-On s:keep
+    here #0 , [ dup , dup [ step ] s:for-each drop ] dip
+    here over - n:dec over store ;
+}}
+~~~
+
 Ok, This is a bit of a hack, but very useful at times.
 
 Assume you have a bunch of values:

words.tsv

@@ -276,6 +276,7 @@ s:to-float	s-	-	-F	Convert a string representation into a floating point value.
 s:to-lower	s-s	-	-	Convert uppercase ASCII characters in a string to lowercase.			class:word	{n/a}	{n/a}	s	all	
 s:to-number	s-n	-	-	Convert a string to a number.			class:word	{n/a}	{n/a}	s	all	
 s:to-upper	s-s	-	-	Convert lowercase ASCII characters in a string to uppercase.			class:word	{n/a}	{n/a}	s	all	
+s:tokenize	sc-a	-	-	Takes a string and a character to use as a separator. It splits the string into a set of substrings and returns a set containing pointers to each of them.			class:word	{n/a}	{n/a}	{n/a}	{n/a}	
 s:trim	s-s	-	-	Trim leading and trailing whitespace from a string.			class:word	{n/a}	{n/a}	s	all	
 s:trim-left	s-s	-	-	Trim leading whitespace from a string.			class:word	{n/a}	{n/a}	s	all	
 s:trim-right	s-s	-	-	Trim trailing whitespace from a string.			class:word	{n/a}	{n/a}	s	all