Skip to content

Commit

Permalink
Stop saying tokenisation is "like C"
Browse files Browse the repository at this point in the history
There are definitely some differences (e.g. Snowball tokenises `3or` as
integer literal `3` followed by token `or`, which seems questionable but
does match a literal interpretation of what the manual says - in C you
can't do the equivalent).
  • Loading branch information
ojwb committed Oct 11, 2024
1 parent 2f41ffb commit 25b8515
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions compiler/snowman.tt
Original file line number Diff line number Diff line change
Expand Up @@ -1257,8 +1257,7 @@ A name, reserved word or number is terminated by the first character that
cannot form part of it. A symbol is recognised as the longest sequence of
characters that forms a valid symbol. So &nbsp;<code>+=-</code>&nbsp; is two symbols, &nbsp;<code>+=</code>&nbsp; and
<code>-</code>, because &nbsp;<code>+=</code>&nbsp; is a valid symbol in the language while &nbsp;<code>+=-</code>&nbsp; is not.
Whitespace separates tokens but is otherwise ignored. This of course is
like C.
Whitespace separates tokens but is otherwise ignored.
</p>

<p>
Expand Down

0 comments on commit 25b8515

Please sign in to comment.