FAQ
So the runtime "master" scheduler Sched maintains a monotonically
incrementing counter int32 goidgen which value is assigned to the goid
member for each process spawned. Now, what happens when goidgen overflows?
New processes might get assigned the same goid as existing processes, which
seems like a potential problem. Am I just crazy or missing something?

--

Search Discussions

  • Minux at Oct 19, 2012 at 8:29 am

    On Oct 19, 2012 1:32 PM, wrote:
    So the runtime "master" scheduler Sched maintains a monotonically
    incrementing counter int32 goidgen which value is assigned to the goid
    member for each process spawned. Now, what happens when goidgen overflows?
    New processes might get assigned the same goid as existing processes, which
    seems like a potential problem. Am I just crazy or missing something?
    this is a valid concern (note about terminology,
    it is goroutine, not process). but i think it's not fatal.

    1. goid is not exposed, so no user level damage if it has some duplicates.
    2. goid is maintained mainly for traceback and race detector.
    for the first usage, i don't think it has any impact.
    for the race detector, before int32 overflows, you'd
    probably lose your patience first ;-)

    if we want to fix it, we need to maintain a bitmap of allocated
    goids, which seems overkill. if we use int64, the problem of
    overflow still remains but 64-bit atomic operation will slow
    some 32-bit hosts down with no clear profit.

    --
  • Devon H. O'Dell at Oct 19, 2012 at 7:14 am
    2012/10/19 minux <minux.ma@gmail.com>:
    On Oct 19, 2012 1:32 PM, wrote:

    So the runtime "master" scheduler Sched maintains a monotonically
    incrementing counter int32 goidgen which value is assigned to the goid
    member for each process spawned. Now, what happens when goidgen overflows?
    New processes might get assigned the same goid as existing processes, which
    seems like a potential problem. Am I just crazy or missing something?
    this is a valid concern (note about terminology,
    it is goroutine, not process). but i think it's not fatal.

    1. goid is not exposed, so no user level damage if it has some duplicates.
    2. goid is maintained mainly for traceback and race detector.
    for the first usage, i don't think it has any impact.
    for the race detector, before int32 overflows, you'd
    probably lose your patience first ;-)

    if we want to fix it, we need to maintain a bitmap of allocated
    goids, which seems overkill. if we use int64, the problem of
    overflow still remains but 64-bit atomic operation will slow
    some 32-bit hosts down with no clear profit.
    If we use int64 on 64-bit hosts, it will take longer than the life of
    the machine to cycle through 2^63-1 ids. On 32-bit hosts, I think it's
    not too much of a worry to use 32-bit go routines, because those (I'm
    thinking ARM mostly) hosts don't tend to have a long uptime.

    --dho
    --
    --
  • Minux at Oct 19, 2012 at 7:28 am

    On Oct 19, 2012 3:07 PM, "Devon H. O&apos;Dell" wrote:
    If we use int64 on 64-bit hosts, it will take longer than the life of
    the machine to cycle through 2^63-1 ids. On 32-bit hosts, I think it's
    yeah, true.
    not too much of a worry to use 32-bit go routines, because those (I'm
    thinking ARM mostly) hosts don't tend to have a long uptime.
    i prefer we use the same definition for all
    arch. if we choose to fix this.

    i'm just not convinced this is a real issue need
    to addressed with such a "heavy" solution.
    goid is mostly invisible imho.

    --
  • Rasmus at Oct 19, 2012 at 9:04 am
    Fair point on 64-bit size (18 446 744 073 709 551 615 processes is probably
    enough). I could easily see a system running for a year or two causing a
    32-bit integer to overflow though. At the same time, the high-level cost of
    introducing something like a free list is probably way higher than the
    benefits it buys.

    I was reading through the source code of go and was baffled over the (to me
    at the time) crude solution of a monotonic counter.

    Thanks for chiming in.
    On Friday, October 19, 2012 12:07:53 AM UTC-7, Devon H. O'Dell wrote:

    2012/10/19 minux <minu...@gmail.com <javascript:>>:
    On Oct 19, 2012 1:32 PM, <ras...@notion.se <javascript:>> wrote:

    So the runtime "master" scheduler Sched maintains a monotonically
    incrementing counter int32 goidgen which value is assigned to the goid
    member for each process spawned. Now, what happens when goidgen
    overflows?
    New processes might get assigned the same goid as existing processes,
    which
    seems like a potential problem. Am I just crazy or missing something?
    this is a valid concern (note about terminology,
    it is goroutine, not process). but i think it's not fatal.

    1. goid is not exposed, so no user level damage if it has some
    duplicates.
    2. goid is maintained mainly for traceback and race detector.
    for the first usage, i don't think it has any impact.
    for the race detector, before int32 overflows, you'd
    probably lose your patience first ;-)

    if we want to fix it, we need to maintain a bitmap of allocated
    goids, which seems overkill. if we use int64, the problem of
    overflow still remains but 64-bit atomic operation will slow
    some 32-bit hosts down with no clear profit.
    If we use int64 on 64-bit hosts, it will take longer than the life of
    the machine to cycle through 2^63-1 ids. On 32-bit hosts, I think it's
    not too much of a worry to use 32-bit go routines, because those (I'm
    thinking ARM mostly) hosts don't tend to have a long uptime.

    --dho
    --
    --
  • Minux at Oct 19, 2012 at 9:09 am

    On Oct 19, 2012 4:58 PM, wrote:
    Fair point on 64-bit size (18 446 744 073 709 551 615 processes is
    probably enough). I could easily see a system running for a year or two
    causing a 32-bit integer to overflow though. At the same time, the
    high-level cost of introducing something like a free list is probably way
    higher than the benefits it buys.
    note again that is called goroutine, not process.

    --
  • Rasmus at Oct 19, 2012 at 9:07 pm
    On Friday, October 19, 2012 2:09:37 AM UTC-7, minux wrote:
    On Oct 19, 2012 4:58 PM, <ras...@notion.se <javascript:>> wrote:
    Fair point on 64-bit size (18 446 744 073 709 551 615 processes is
    probably enough). I could easily see a system running for a year or two
    causing a 32-bit integer to overflow though. At the same time, the
    high-level cost of introducing something like a free list is probably way
    higher than the benefits it buys.
    note again that is called goroutine, not process.
    I know. But historically they have been called both goroutines and
    processes. A large part of the goroutine implementation still lives in a
    file called "proc.c" (see http://golang.org/src/pkg/runtime/proc.c) wherein
    comments use both the term "process", "routine" and "goroutine". "Process"
    is just a more generic name and more people (i.e. not familiar with go)
    will have a better idea of what we are talking about.

    --
  • Jan Mercl at Oct 19, 2012 at 9:02 pm

    On Fri, Oct 19, 2012 at 10:06 PM, wrote:
    I know. But historically they have been called both goroutines and
    processes. A large part of the goroutine implementation still lives in a
    file called "proc.c" (see http://golang.org/src/pkg/runtime/proc.c) wherein
    comments use both the term "process", "routine" and "goroutine".
    The sub string "process" (case insensitive) occurs in the linked file
    exactly once: http://golang.org/src/pkg/runtime/proc.c#L41

    And that occurrence in no way mixes goroutines and processes.
    "Process"
    is just a more generic name and more people (i.e. not familiar with go) will
    have a better idea of what we are talking about.
    What? E.g. processes virtual memory address spaces are isolated from
    other processes. Goroutines work in the exactly opposite way.
    Processes are typically preemptively scheduled, goroutines
    cooperatively. Processes are exposed by the OS, goroutines are not.
    And so on and so on...

    -j

    --
  • Rasmus at Oct 19, 2012 at 10:38 pm

    On Friday, October 19, 2012 1:57:45 PM UTC-7, Jan Mercl wrote:

    On Fri, Oct 19, 2012 at 10:06 PM, <ras...@notion.se <javascript:>>
    wrote:
    I know. But historically they have been called both goroutines and
    processes. A large part of the goroutine implementation still lives in a
    file called "proc.c" (see http://golang.org/src/pkg/runtime/proc.c) wherein
    comments use both the term "process", "routine" and "goroutine".
    The sub string "process" (case insensitive) occurs in the linked file
    exactly once: http://golang.org/src/pkg/runtime/proc.c#L41

    And that occurrence in no way mixes goroutines and processes.
    "Process"
    is just a more generic name and more people (i.e. not familiar with go) will
    have a better idea of what we are talking about.
    What? E.g. processes virtual memory address spaces are isolated from
    other processes. Goroutines work in the exactly opposite way.
    Processes are typically preemptively scheduled, goroutines
    cooperatively. Processes are exposed by the OS, goroutines are not.
    And so on and so on...

    -j
    Wow, you guys really care about this. In CS, the concept of a "process" is
    a generic term for the logical unit of a program. In an operating system
    with preemptive scheduling, the term "process" might be used to describe a
    program. Memory address might or might not be separate. Some operating
    systems use cooperative scheduling (common in real-time systems.) The term
    "goroutine" is simply a "trademark name" used to say "a process in the go
    system". Since this group is all about go, these seems to be little need to
    explicitly state the term "goroutine" since "process" is equally
    describing, except in the case the discussion is about different types of
    processes (host system vs go, for instance.) I'm very well aware what a
    goroutine is and the various types of scheduling approaches and processes
    that are or have been around.

    My initial question has nothing to do with terminology, but with an
    interesting design choice in the go scheduler.

    Let's be adults and not start some kind of "flame war" over different
    opinions of what a common term means.

    --
  • Joubin Houshyar at Oct 19, 2012 at 10:56 pm

    On Friday, October 19, 2012 6:38:02 PM UTC-4, ras...@notion.se wrote:
    On Friday, October 19, 2012 1:57:45 PM UTC-7, Jan Mercl wrote:
    On Fri, Oct 19, 2012 at 10:06 PM, wrote:
    I know. But historically they have been called both goroutines and
    processes. A large part of the goroutine implementation still lives in a
    file called "proc.c" (see http://golang.org/src/pkg/runtime/proc.c) wherein
    comments use both the term "process", "routine" and "goroutine".
    The sub string "process" (case insensitive) occurs in the linked file
    exactly once: http://golang.org/src/pkg/runtime/proc.c#L41

    And that occurrence in no way mixes goroutines and processes.
    "Process"
    is just a more generic name and more people (i.e. not familiar with go) will
    have a better idea of what we are talking about.
    What? E.g. processes virtual memory address spaces are isolated from
    other processes. Goroutines work in the exactly opposite way.
    Processes are typically preemptively scheduled, goroutines
    cooperatively. Processes are exposed by the OS, goroutines are not.
    And so on and so on...

    -j
    Wow, you guys really care about this. In CS, the concept of a "process" is
    a generic term for the logical unit of a program. In an operating system
    with preemptive scheduling, the term "process" might be used to describe a
    program. Memory address might or might not be separate. Some operating
    systems use cooperative scheduling (common in real-time systems.) The term
    "goroutine" is simply a "trademark name"
    http://en.wikipedia.org/wiki/Fiber_(computer_science)
    http://en.wikipedia.org/wiki/Thread_(computer_science)
    http://en.wikipedia.org/wiki/Process_(computer_science)

    used to say "a process in the go system". Since this group is all about
    go, these seems to be little need to explicitly state the term "goroutine"
    since "process" is equally describing, except in the case the discussion is
    about different types of processes (host system vs go, for instance.) I'm
    very well aware what a goroutine is and the various types of scheduling
    approaches and processes that are or have been around.

    My initial question has nothing to do with terminology, but with an
    interesting design choice in the go scheduler.

    Let's be adults and not start some kind of "flame war" over different
    opinions of what a common term means.
    --
  • Andrew Gerrand at Oct 22, 2012 at 2:05 am

    On 20 October 2012 09:38, wrote:
    Wow, you guys really care about this.
    These are people that work on this code. They're just trying to
    encourage you to use the terminology they use. More generally, in the
    Go community "process" usually refers to an operation system process,
    "thread" refers to an operating system thread, and "goroutine" refers
    to a goroutine.

    FWIW, proc.c contains a single mention of the word "process" and it is
    in reference to an operating system process.

    Andrew

    --
  • Russ Cox at Oct 21, 2012 at 5:18 pm

    member for each process spawned. Now, what happens when goidgen overflows?
    New processes might get assigned the same goid as existing processes, which
    seems like a potential problem. Am I just crazy or missing something?
    If goidgen overflows then it will wrap around and goroutine ids will
    be reused. However, the goid is only used when printing tracebacks, so
    this hardly matters. The real "id" of an active goroutine, as far as
    the runtime is concerned, is the G* pointer corresponding to its
    metadata.

    We should probably make the identifiers 64 bits, and then they will
    never run out.

    Russ

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 19, '12 at 5:32a
activeOct 22, '12 at 2:05a
posts12
users7
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase