FAQ
Hi.
I've run into this little problem using the text/scanner package, and
wanted to ask if it's a bug or not.
Basically, since this package parses ints and floats, I figured negative
ints and floats are fine. Turns out the minus sign is read as a separate
token before the number.
Example:

func main() {
var s scanner.Scanner
s.Init(strings.NewReader("Foo 1 2.5 -3 -0.54"))
s.Mode = scanner.GoTokens
for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() {
fmt.Println("Tok: ", s.TokenText())
}
}

Outputs:
Tok: Foo
Tok: 1
Tok: 2.5
Tok: -
Tok: 3
Tok: -
Tok: 0.54


If it's not a bug it should be documented IMHO.

Thanks!

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Rob Pike at Aug 3, 2013 at 1:17 pm
    That package is intended as a scanner for Go tokens ("literals as defined
    by the Go language"), and in Go the minus sign is a separate token.

    -rob

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dvirsky at Aug 3, 2013 at 1:58 pm
    Thanks Rob, I saw that the default was the "go tokens", but figured it was
    just a default in a generic library.

    So maybe it's worth making this behavior configurable, among the other
    flags of the scanner?

    I found it very very useful and easy for parsing a tiny DSL to query some
    data structures (yay Greenspun's 10th rule! ;-) ), so I assure you it's a
    nice utility beyond parsing go text.

    BTW It resides under the text/ package, and there *is *a go/scanner
    package, this makes it a bit confusing.
    On Saturday, August 3, 2013 4:17:11 PM UTC+3, Rob Pike wrote:

    That package is intended as a scanner for Go tokens ("literals as defined
    by the Go language"), and in Go the minus sign is a separate token.

    -rob
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jan Mercl at Aug 3, 2013 at 2:42 pm

    On Sat, Aug 3, 2013 at 3:58 PM, wrote:
    I found it very very useful and easy for parsing a tiny DSL to query some
    data structures (yay Greenspun's 10th rule! ;-) ), so I assure you it's a
    nice utility beyond parsing go text.
    Design the tokenizer of your DSL to expect ]unary] '+'/'-' as a
    separate token as well, which is pretty standard anyway[0]; and you're
    good to go.

       [0] Eg.: http://www.lysator.liu.se/c/ANSI-C-grammar-l.html

    -j

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dvir Volk at Aug 3, 2013 at 4:14 pm

    On Saturday, August 3, 2013 5:42:28 PM UTC+3, Jan Mercl wrote:
    On Sat, Aug 3, 2013 at 3:58 PM, <dvi...@gmail.com <javascript:>> wrote:
    Design the tokenizer of your DSL to expect ]unary] '+'/'-' as a
    separate token as well, which is pretty standard anyway[0]; and you're
    good to go.
    that is of course what I did, I just found the behavior of the package not
    to be what I expected intuitively, and thought it might be a bug.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Rob Pike at Aug 3, 2013 at 9:49 pm
    It's a bit of an moribund package, not used much and showing its age.

    -rob


    On Sun, Aug 4, 2013 at 12:54 AM, Dvir Volk wrote:


    On Saturday, August 3, 2013 5:42:28 PM UTC+3, Jan Mercl wrote:

    On Sat, Aug 3, 2013 at 3:58 PM, wrote:
    Design the tokenizer of your DSL to expect ]unary] '+'/'-' as a
    separate token as well, which is pretty standard anyway[0]; and you're
    good to go.
    that is of course what I did, I just found the behavior of the package not
    to be what I expected intuitively, and thought it might be a bug.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jim Robinson at Aug 3, 2013 at 2:45 pm
    I think Rob's point was that the implementation really was specific
    to parsing Go tokens, that it was not intended to be generic, so I
    am guessing that it's unlikely he's going to want to move it.
    On Saturday, August 3, 2013 6:58:03 AM UTC-7, Dvir Volk wrote:

    Thanks Rob, I saw that the default was the "go tokens", but figured it was
    just a default in a generic library.

    So maybe it's worth making this behavior configurable, among the other
    flags of the scanner?

    I found it very very useful and easy for parsing a tiny DSL to query some
    data structures (yay Greenspun's 10th rule! ;-) ), so I assure you it's a
    nice utility beyond parsing go text.

    BTW It resides under the text/ package, and there *is *a go/scanner
    package, this makes it a bit confusing.
    On Saturday, August 3, 2013 4:17:11 PM UTC+3, Rob Pike wrote:

    That package is intended as a scanner for Go tokens ("literals as defined
    by the Go language"), and in Go the minus sign is a separate token.

    -rob
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedAug 3, '13 at 12:37p
activeAug 3, '13 at 9:49p
posts7
users4
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase