FAQ
Is there any interest in bouncing around syntax extensibility as a possible
Go 2.0 feature?

One thing I like about Rust and Ruby is how you can enhance these languages
to be more useful for a specific problem domain. It's called Domain
Specific Language - or DSL now days. I think this ability in Ruby is one
of the main reasons it gained popularity as an HTML generation language.
I've toyed with syntax extensibility before. My first one was a Scheme
interpreter with rewriting rules, which allowed a lot of Scheme to be
written in itself. I believe the same could be done for Go. For example,
the parenthesized version of import (and other) statements could be
auto-rewritten to the non-parenthesized form. Go could have a "core"
simpler syntax, with all the syntactic sugar we want written in Go
rewriting rules, and for improved speed, implementations could be free to
directly parse some or all of the larger syntax.

This was easy in Scheme because everything was just a list, and there was
little concern about speed. My first approach for applying this to a
modern high performance language was to write a GLR parser, which made
everything easy, but it was way too slow. Last year, I wrote a parser that
assumes the input is made of statements and expressions, where the
statements are newline or semicolon separated, and grouped into blocks with
curly braces or by indentation. I looked up statements in constant time in
a hash table once the keyword signature was found (a linear pass over the
input tokens). An enhanced precedence parser then parsed the expressions
in the statement. It was very fast, but not quite powerful enough. In
particular, it could parse Python, but not Javascript, because it could not
handle expressions containing statements.

I haven't finished writing the new parser yet, but I think I've finally got
the spec about right. I dropped statements all together, and now
everything is just an expression. Conceptually, it operates in two passes
(but mostly will combined into one):

1) Parse the input into a node tree with the precedence parser
2) Match the node tree to yacc-like matching rules to find syntax errors
not found by the precedence parser

To be able to parse languages like Go, I have to allow the operator
precedence tables to change while parsing sub-expressions. For example, in
parameter declarations, comma is just a separator - the lowest precedence
operator forming the root of the parameter expression tree. In block
statements, it has a different precedence, and in the parenthesized form of
import statements, commas aren't allowed at all. Clearly, a fixed
operator-table based precedence parser wont work. Instead, sub-expression
preceded by an operator keyword can switch the parser to a different
operator table (the keyword is "(" in the parameter parsing example). It
switches back once the largest possible sub-expression has been matched.

So far, it this to be powerful enough to parse everything I've had trouble
with before, including Python, Javascript, and and Go.

Also, the matching phase is quite a bit more powerful than bison/yacc.
Most expression parsers I've written in bison force me to make everything a
generic expression, and then I have to check that boolean expressions are
not being improperly mixed with integer expressions, and I have to verify
constant expressions are actually constant in the semantic checker. This
is because bison can't have multiple legal interpretations for a set of
input tokens. If I'm trying to verify that the expression is valid where
we require a Boolean in a bison parser, I would try to write a rule called
boolExpr which matches only Boolean expressions, and arithExpr which
matches only arithmetic expressions. Similarly, I might want to have a
constExpr rule for matching constant expressions. However, when matching
an identifier, bison doesn't what it is. It has to pick, and the best it
can do is call it a generic expression, leaving it up to the programmer to
write code to check all the cases for all the types. I just checked the Go
bison parser, and sure enough, it calls everything a generic expression...

Is this a good place to bounce around dumb ideas like this?

Thanks,
Bill

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Axel Wagner at May 18, 2015 at 9:32 pm

    'Bill Cox' via golang-nuts writes:

    Is there any interest in bouncing around syntax extensibility as a possible
    Go 2.0 feature?
    This is in direct conflict with on of the major design goals of go: Code
    written by any one developer should be immediately readable and
    maintainable by any other developer. This is necessary to build large
    systems.

    There is no way this would ever happen and even if there were, it would
    be far to soon to think about go 2.0 at this point in time.

    Best,

    Axel

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Bill Cox at May 18, 2015 at 11:15 pm

    On Mon, May 18, 2015 at 2:32 PM, Axel Wagner wrote:

    'Bill Cox' via golang-nuts <golang-nuts@googlegroups.com> writes:
    Is there any interest in bouncing around syntax extensibility as a possible
    Go 2.0 feature?
    This is in direct conflict with on of the major design goals of go: Code
    written by any one developer should be immediately readable and
    maintainable by any other developer. This is necessary to build large
    systems.
    I don't agree. I've worked on large systems with significant domain
    specific extensions. For example, the Qt GUI toolkit for C++ adds it's own
    C++ pre-processor, and it adds tremendous value. Similarly, Ruby has some
    standard extensions use with Ruby-on-Rails. The real-world examples of DSL
    show that they aid large projects, and make code more readable as well as
    maintainable. Plain old C is an excellent language when measured by a
    average programmer's ability to pick it up and understand the syntax.

    Bill

    Bill

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dan Kortschak at May 19, 2015 at 12:08 am
    You can add a DSL now with go generate. Commit the generated code and
    everyone else will just see the Go. You don't need to wait.
    On Mon, 2015-05-18 at 16:15 -0700, 'Bill Cox' via golang-nuts wrote:
    I don't agree. I've worked on large systems with significant domain
    specific extensions. For example, the Qt GUI toolkit for C++ adds
    it's own
    C++ pre-processor, and it adds tremendous value. Similarly, Ruby has
    some
    standard extensions use with Ruby-on-Rails. The real-world examples
    of DSL
    show that they aid large projects, and make code more readable as well
    as
    maintainable. Plain old C is an excellent language when measured by a
    average programmer's ability to pick it up and understand the syntax.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roberto Zanotto at May 19, 2015 at 12:18 am
    The problem with features is that the nanosecond you enable them, dumb
    programmers (= 99.999% of programmers, me included) start abusing them in
    all the possible ways. Sure operator overloading is a nice thing to have in
    C++ for a certain type of problems. Classes are handy in some cases.
    Templates are a good thing to have for certain tasks. The Qt preprocessor
    is probably good for Qt applications. Put everything together and you get
    an ugly and over-complicated language that programmers abuse to create
    monster codebases impossible to debug and maintain (and even compile). Go
    simplicity and the lack of features are considered a great value by this
    community :)
    On Tuesday, May 19, 2015 at 1:15:43 AM UTC+2, Bill Cox wrote:

    On Mon, May 18, 2015 at 2:32 PM, Axel Wagner <axel.wa...@googlemail.com
    <javascript:>> wrote:
    'Bill Cox' via golang-nuts <golan...@googlegroups.com <javascript:>>
    writes:
    Is there any interest in bouncing around syntax extensibility as a possible
    Go 2.0 feature?
    This is in direct conflict with on of the major design goals of go: Code
    written by any one developer should be immediately readable and
    maintainable by any other developer. This is necessary to build large
    systems.
    I don't agree. I've worked on large systems with significant domain
    specific extensions. For example, the Qt GUI toolkit for C++ adds it's own
    C++ pre-processor, and it adds tremendous value. Similarly, Ruby has some
    standard extensions use with Ruby-on-Rails. The real-world examples of DSL
    show that they aid large projects, and make code more readable as well as
    maintainable. Plain old C is an excellent language when measured by a
    average programmer's ability to pick it up and understand the syntax.

    Bill

    Bill
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Axel Wagner at May 19, 2015 at 12:43 am
    tl;dr: Brevity ≠ Readability. Readability is a measure of how easy it is
    to hit the ground running, brevity is a measure of the most concise way
    to express a problem.

    Hi,

    Bill Cox <waywardgeek@google.com> writes:
    I don't agree.
    Feel free :)

    I don't quite understand though, why people are trying to change go into
    rust, if there is already a perfectly good rust in existence. The fact
    that go makes it hard to write unreadable programs whereas rust makes it
    easy is the only thing preventing me from jumping ship :)
    For example, the Qt GUI toolkit for C++ adds it's own C++
    pre-processor, and it adds tremendous value.
    I find this specifically (I too am currently working on a large system
    written in C++ using Qt) an example of how horrific these things
    are. From what I can tell, the Qt approach is a mixture of Macros,
    artificially added keywords and syntax and dynamic-ish stringy-typing (I
    had at least one bug, which was due to a typo in a type-name, which was
    only discovered at runtime and buried in the logs then). I might be
    wrong about anything of this (I did not spend a lot of time dissecting
    the inner workings of Qt, after all), but that's pretty much my
    point. The code becomes more readable in the sense of "understanding the
    authors intent by skimming the code" *maybe*, but it becomes infinitely
    harder to understand what the code is actually doing, which is necessary
    to find and fix bugs, maintain the code and add to it. Instead of using
    the (already too powerfull and complex) language features of C++, Qt is
    forcing me to basically learn an entirely new language with new features
    to work on this. And as I don't spend a lot of time on this project, I
    find that very frustrating.

    Brevity is not equal to readability. The critical question for
    readability is the tradeoff between "knowledge you need to have
    generally available" and "time it takes you to get comfortable enough
    with a codebase to contribute". DSLs worsen this tradeoff: You need to
    know more coming in or take additional time to get to know the DSL
    used. And the point of pages like [0] is to provide some standard, that
    every go programmer should more or less adhere to and to improve this
    tradeoff: Give you some pool of knowledge that is as broadly usefull as
    possible to let you hit the ground running. There are many more examples
    in the design of go that make this tradeoff.
    Similarly, Ruby has some standard extensions use with Ruby-on-Rails.
    I don't know a lot of ruby, but I do know Perl, which I perceive as the
    gold-standard in the category "languages which enable DSLs". And the
    main reason why I don't use it anymore (in favor of Python) is, that
    there are (for example) a gazillion different DSLs adding object
    orientation to it. It is horrendous to the point, that to contribute to
    a single project, you often need to learn two or three new object
    oriented programming modules, because every subpart and imported library
    uses it's own flavor of "readable" DSL and module system.
    Plain old C is an excellent language when measured by a average
    programmer's ability to pick it up and understand the syntax.
    I disagree with this assessment. Plain ANSI C maybe, but even
    then. Even in C, every single project thinks it needs to redefine
    datatypes for their own project. uint64_t is not good enough, so we need
    FOOBAR_ULONG or whatever. And honestly, imagine you know C, nothing more
    and nothing less. What would your reaction be to Duff's device [1], or
    coroutines in C [2]? Imagine facing a bug, because some kind of
    macro-syntactical hack like this coroutine-library shadows some local
    variable somewhere, in a project you are not familiar with, only knowing
    C? Or some kind of faulty macro, leading to double-evaluation of an
    expression with side-effects or wrong order of evaluation *somewhere* in
    the program after a seemingly unrelated change. I don't know about you,
    but I cringe at the thought.

    [0] https://code.google.com/p/go-wiki/wiki/CodeReviewComments
    [1] http://www.lysator.liu.se/c/duffs-device.html
    [2] http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Simon place at May 18, 2015 at 11:35 pm
    i don't care that much about the syntax, for me its all about a well
    written standard lib. which is a hard thing to do.

    anyway aren’t standard lib. packages quite close on a spectrum to syntax
    specialisation, so much so that having different versions of the standard
    libs, tailored to specific domains, is effectively the same thing.


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Minux at May 19, 2015 at 2:14 am

    On Mon, May 18, 2015 at 5:23 PM, 'Bill Cox' via golang-nuts wrote:

    Is there any interest in bouncing around syntax extensibility as a
    possible Go 2.0 feature?
    It's too early to talk about Go 2 now.
    If you want, you probably could start a Go2 mailing list.

    Personally, I highly doubt that Go will ever have user extensible
    syntax.

    DSL makes sense if the domain of the code matches the readers',
    otherwise it will be a disaster.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Bill Cox at May 19, 2015 at 4:22 pm
    I just looked a bit deeper into Rust's plugin capability, and whoa... it is
    a potential long-term mess. I also looked at Ruby's, and I have the same
    concern. These extension capabilities are very unlike Scheme's simple
    rewriting rules, which make writing new compatible implementations
    _simpler_. If syntax extension capability significantly complicates
    writing a compiler, then forget it.

    The right way to build a new standard is to build two independent
    implementations, rather than just one, so you avoid baking in
    implementation specific cruft (like Python's one-thread problem). Having
    the original go compiler and gccgo is the right way to Go :-p

    For syntax extensibility, I prefer Scheme's rewriting rules, where if a
    list matches a pattern, it is replaced with another list that can
    instantiate portions of the original list, and which can have sub-lists
    that further match patterns recursively.

    Syntax extension by itself is tricky, especially if speed is important.
    That's the piece I've been playing with while. After creating the initial
    parse tree, the rewriting is still needed. I think there is considerable
    room for improvement in this area vs Rust and Ruby. Anyway, it's something
    I'll continue to play with. I was mostly interested in seeing if there
    were already people out there working on ideas like this for Go. I'd be
    interested in collaborating if possible.

    Thanks,
    Bill

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andrewchamberss at May 20, 2015 at 5:36 am
    I once played with modifying a C compiler to embed a scheme like
    interpreter, and change the macro system to use lisp style macros. A large
    downside is it needed two different ways to represent the same syntax tree.
    On Wednesday, May 20, 2015 at 4:22:32 AM UTC+12, Bill Cox wrote:

    I just looked a bit deeper into Rust's plugin capability, and whoa... it
    is a potential long-term mess. I also looked at Ruby's, and I have the
    same concern. These extension capabilities are very unlike Scheme's simple
    rewriting rules, which make writing new compatible implementations
    _simpler_. If syntax extension capability significantly complicates
    writing a compiler, then forget it.

    The right way to build a new standard is to build two independent
    implementations, rather than just one, so you avoid baking in
    implementation specific cruft (like Python's one-thread problem). Having
    the original go compiler and gccgo is the right way to Go :-p

    For syntax extensibility, I prefer Scheme's rewriting rules, where if a
    list matches a pattern, it is replaced with another list that can
    instantiate portions of the original list, and which can have sub-lists
    that further match patterns recursively.

    Syntax extension by itself is tricky, especially if speed is important.
    That's the piece I've been playing with while. After creating the initial
    parse tree, the rewriting is still needed. I think there is considerable
    room for improvement in this area vs Rust and Ruby. Anyway, it's something
    I'll continue to play with. I was mostly interested in seeing if there
    were already people out there working on ideas like this for Go. I'd be
    interested in collaborating if possible.

    Thanks,
    Bill
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andrewchamberss at May 19, 2015 at 2:22 am
    The Go compiler already uses the DSL YACC, and in the dev.ssa branch it is
    using a DSL to describe IR code transformations:
    https://github.com/golang/go/blob/083a646f63055427c203d5600ef65f05f55783bf/src/cmd/internal/ssa/rulegen/lower_amd64.rules
    .

    It is relatively easy to generate code yourself.
    On Tuesday, May 19, 2015 at 9:24:07 AM UTC+12, Bill Cox wrote:

    Is there any interest in bouncing around syntax extensibility as a
    possible Go 2.0 feature?

    One thing I like about Rust and Ruby is how you can enhance these
    languages to be more useful for a specific problem domain. It's called
    Domain Specific Language - or DSL now days. I think this ability in Ruby
    is one of the main reasons it gained popularity as an HTML generation
    language. I've toyed with syntax extensibility before. My first one was a
    Scheme interpreter with rewriting rules, which allowed a lot of Scheme to
    be written in itself. I believe the same could be done for Go. For
    example, the parenthesized version of import (and other) statements could
    be auto-rewritten to the non-parenthesized form. Go could have a "core"
    simpler syntax, with all the syntactic sugar we want written in Go
    rewriting rules, and for improved speed, implementations could be free to
    directly parse some or all of the larger syntax.

    This was easy in Scheme because everything was just a list, and there was
    little concern about speed. My first approach for applying this to a
    modern high performance language was to write a GLR parser, which made
    everything easy, but it was way too slow. Last year, I wrote a parser that
    assumes the input is made of statements and expressions, where the
    statements are newline or semicolon separated, and grouped into blocks with
    curly braces or by indentation. I looked up statements in constant time in
    a hash table once the keyword signature was found (a linear pass over the
    input tokens). An enhanced precedence parser then parsed the expressions
    in the statement. It was very fast, but not quite powerful enough. In
    particular, it could parse Python, but not Javascript, because it could not
    handle expressions containing statements.

    I haven't finished writing the new parser yet, but I think I've finally
    got the spec about right. I dropped statements all together, and now
    everything is just an expression. Conceptually, it operates in two passes
    (but mostly will combined into one):

    1) Parse the input into a node tree with the precedence parser
    2) Match the node tree to yacc-like matching rules to find syntax errors
    not found by the precedence parser

    To be able to parse languages like Go, I have to allow the operator
    precedence tables to change while parsing sub-expressions. For example, in
    parameter declarations, comma is just a separator - the lowest precedence
    operator forming the root of the parameter expression tree. In block
    statements, it has a different precedence, and in the parenthesized form of
    import statements, commas aren't allowed at all. Clearly, a fixed
    operator-table based precedence parser wont work. Instead, sub-expression
    preceded by an operator keyword can switch the parser to a different
    operator table (the keyword is "(" in the parameter parsing example). It
    switches back once the largest possible sub-expression has been matched.

    So far, it this to be powerful enough to parse everything I've had trouble
    with before, including Python, Javascript, and and Go.

    Also, the matching phase is quite a bit more powerful than bison/yacc.
    Most expression parsers I've written in bison force me to make everything a
    generic expression, and then I have to check that boolean expressions are
    not being improperly mixed with integer expressions, and I have to verify
    constant expressions are actually constant in the semantic checker. This
    is because bison can't have multiple legal interpretations for a set of
    input tokens. If I'm trying to verify that the expression is valid where
    we require a Boolean in a bison parser, I would try to write a rule called
    boolExpr which matches only Boolean expressions, and arithExpr which
    matches only arithmetic expressions. Similarly, I might want to have a
    constExpr rule for matching constant expressions. However, when matching
    an identifier, bison doesn't what it is. It has to pick, and the best it
    can do is call it a generic expression, leaving it up to the programmer to
    write code to check all the cases for all the types. I just checked the Go
    bison parser, and sure enough, it calls everything a generic expression...

    Is this a good place to bounce around dumb ideas like this?

    Thanks,
    Bill
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Minux at May 20, 2015 at 5:23 am

    On Mon, May 18, 2015 at 10:22 PM, wrote:

    The Go compiler already uses the DSL YACC, and in the dev.ssa branch it is
    using a DSL to describe IR code transformations:
    https://github.com/golang/go/blob/083a646f63055427c203d5600ef65f05f55783bf/src/cmd/internal/ssa/rulegen/lower_amd64.rules
    .
    Not supporting DSL as Go builtin syntax is very important.
    It makes the author resort to DSL only when it's truly necessary.

    Should Go has extensive syntax support, DSLs will be much more
    common, and that will surely hurt readability.

    IMHO, Go is a rare language that is optimized for the reader, not for
    the writer, and I hope it can keep the trait.

    Another reason why Go won't ever have extensible syntax is that
    we need to be able to gofmt it. For example, modifying go.y is more
    difficult than modifying other parts of the toolchain because gofmt
    doesn't work there.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at May 19, 2015 at 8:18 am
    Everything has trade-offs it makes and a cost.

    Making something difficult means that you will think several times before
    doing it. Do you really need the DSL extension? It complicates the building
    process, can make things more complicated, etc. Of course if it outweighs
    the cost of building a parser/language tooling, then maybe indeed you need
    it. I.e. by making harder to make extensions you make it less likely to
    find such extensions. If something is simple - people will do it, just
    because they can, not because they need to.

    Also, if you try to fit other syntax into an existing language, it will
    probably look much worse compared to a specifically designed DSL. So I
    think it's better to have a separate parser/transpiler, if needed, it
    allows better to match the Domain rather than being forced to deal with the
    base-language.

    + Egon

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Henry Adi Sumarto at May 19, 2015 at 9:52 am
    I don't like the idea of Go becoming like other languages. Other languages
    may give 50 words to swear at a person, while Go gives only one. When it
    comes to insulting business, the Go dude spits out the word and be done
    with it, while others are still contemplating which of the 50 words suit
    the situation.

    To be truthful, I had my doubts about Go. However, I was intrigued because
    philosophically speaking, I am very much in line with Go's simplicity
    ethos. While other languages are more expressive, I feel that today's
    programming languages are becoming increasingly bureaucratic and we often
    spend much time deciding which of the blues is the correct blue. Go is so
    much different from all other languages that it actually challenges the
    way I normally think in other languages. After giving Go a try in my pet
    project, I realize that I get things done faster with Go. I think the
    primary reason is that I have less design concerns to consider. I also
    write less code with Go. Some people say that Go insults the intelligent
    programmers by giving them a dumbed down, uncreative language. I say I am
    not interested in seeing wicked code, cleverly designed around the language
    semantics. I am more interested in getting things done and the software
    performs reasonably fast.

    I am pretty sure that my experience isn't unique. I hope the Go team
    continues to resist any impulsive attempt by us to complicate the language.


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMay 18, '15 at 9:23p
activeMay 20, '15 at 5:36a
posts14
users9
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase