Re: Applesoft tokenization phases?

Martin Ward <martin@gkc.org.uk>
Sat, 21 Mar 2020 19:42:17 +0000

          From comp.compilers
Related articles
Applesoft tokenization phases? drikosev@gmail.com (Ev. Drikos) (2020-03-12)
Re: Applesoft tokenization phases? gneuner2@comcast.net (George Neuner) (2020-03-13)
Re: Applesoft tokenization phases? awanderin@gmail.com (awanderin) (2020-03-16)
Re: Applesoft tokenization phases? drikosev@gmail.com (Ev. Drikos) (2020-03-18)
Re: Applesoft tokenization phases? christopher.f.clark@compiler-resources.com (Christopher F Clark) (2020-03-20)
Re: Applesoft tokenization phases? martin@gkc.org.uk (Martin Ward) (2020-03-21)
| List of all articles for this month |
From: Martin Ward <martin@gkc.org.uk>
Newsgroups: comp.compilers
Date: Sat, 21 Mar 2020 19:42:17 +0000
Organization: Compilers Central
References: 20-03-013 20-03-016 20-03-017 20-03-022
Injection-Info: gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="66018"; mail-complaints-to="abuse@iecc.com"
Keywords: Basic, lex, history
Posted-Date: 21 Mar 2020 16:50:24 EDT

On 17/03/20 22:14, Ev. Drikos wrote:
> Another vague point or simply a point where I'm not really sure that I
> translate properly the manual are the reserved keywords before a certain
> delimiter. Likely an Applesoft parser must reject this valid UK101 code:
>
> 10 X=SHIMEM:
> 20 END


For those less familiar with UK101 BASIC: this code is only
valid because "HIMEM" is not a keyword in UK101 BASIC.
UK101 does not allow keywords as part of a variable.


> There is a reason, BASIC compilers that remove spaces often restrict
> variables (like the original Basic version did) to a letter optionally
> followed by a digit (and a dollar sign to indicate strings). It makes
> the lexing much simpler, even without spaces.


Lexing is not the issue: the real reason for the restriction
is to make the symbol table much simpler. Each variable in the table
takes up a fixed amount of space. For the UK101, only the first two
characters of a variable are significant, so only two characters
are needed in the symbol table. "Lexing" a statement just consists
of replacing all keywords outside strings by special characters.
When the interpreted parses it, each statement either starts
with a keyword or is an assignment and starts with a variable.


--
Martin


Dr Martin Ward | Email: martin@gkc.org.uk | http://www.gkc.org.uk
G.K.Chesterton site: http://www.gkc.org.uk/gkc | Erdos number: 4


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.