[go-nuts] How build an easy kernel in Go ?

Go depends on system calls. You currently have to have a kernel running in
order for a go program to work. There used to be a "tiny" runtime that did
not have this restriction, but it was removed (I believe because it was too
hard to keep it up-to-date with the regular runtime). I know of no way to
get around this restriction currently.


On Thu, Dec 20, 2012 at 11:28 AM, Guillaume Lescure
wrote:

You could boot the linux kernel and run a go program as your init.
That seems the easiest most elegant solution seeing as you would not have
to write drivers for any hardware/filesystems/basic network protocols/etc...

--

Should also add it is only easy to build a kernel after you have built a
kernel the hard way a number of times.
You have to know nearly everything about the hardware you want to use top
to bottom.


--

Yeah but all the project is to write a new kernel. I don't want to create
THE new kernel, I just want to build A new kernel for my own purpose, for
fun. I already write one in C and I know the Go team said the Go is really
suitable for building operative systems so I try :D

But the runtime is a big issue so I don't understand why the Go is so
suitable for operating systems.
Maybe I misunderstand something during that conference.


Le jeudi 20 décembre 2012 19:04:07 UTC+1, bryanturley a écrit :

I believe the Go team has said Go is appropriate for building "Systems
Software" not "Operating Systems".

But I don't speak for them.

-joe


On Thu, Dec 20, 2012 at 1:10 PM, Guillaume Lescure
wrote:

Well the go runtime has some very operating system-ish capabilities.
Writing a new runtime that works without a kernel would require you to
write the missing pieces of the kernel which is what I assume your class
assignment is?

Might be to big of a project just to get a grade though.

--

My goals are to work on kernel not to build a new runtime. Can I just copy
it ? Same architecture, same PC.
I have no assignment, just working on kernel programming. Choose whatever
technology I want. I did it in C, now I try in Go :)
Any piece of advice to make it work is welcome ;)


Le jeudi 20 décembre 2012 19:26:14 UTC+1, bryanturley a écrit :

you can dig out the tiny runtime from project history (but unfortunately,
adapting it
to newer Go requires a lot of hard work, so good luck!).
I can give you a pointer here: https://codereview.appspot.com/186144/

--

You could potentially sync back in the repository to when the "tiny"
runtime still worked. Go was a lot different back then, but the
documentation should be there to run a local godoc server that should have
most of what you'd need.


On Thu, Dec 20, 2012 at 1:41 PM, Guillaume Lescure
wrote:

As I repeat every time this issue comes up, have a look at Native
Oberon
and Blue Bottle.

These are desktop operating systems written in GC enabled systems
programming
languages Oberon and Active Oberon respectively, which were in use for
several years
at ETHZ (Zürich Technical University).

The only assembly parts are the boot loader and GC, everything else
was written in corresponding
system language.

http://en.wikipedia.org/wiki/Oberon_%28operating_system%29
http://www.ocp.inf.ethz.ch/wiki/Documentation/Front

Specially the book where the first version of Native Oberon is
described,
http://www.ethoberon.ethz.ch/WirthPubl/ProjectOberon.pdf

A few others are also available from http://www.ethoberon.ethz.ch/books.html

I really find sad that this information is not well known outside
Europe, as Native Oberon shown a
way to have a working desktop workstation based in a GC enabled
systems language. Who knows
when a similar system will ever reach mainstream.

Put all the information that you want to go into .grub_sig into one
package, and don't put anything else into that package. Use a linker
script to put that object file into the .grub_sig section.
Those functions are defined in libgo.

I don't know why you aren't picking up the definitions. If you are
using explicit -l options (-lgcc, -lgo) make sure you put them at the
end of the link line, and put -lgo before -lgcc. The ordering of -l
options and object files matters.

Ian

--

I don't know how modify my linker to do that. I found it on a tutorial but
I don't understand all of it.
I know grub_sig is a symbole in the begining of the programme like a
header, to allow Grub2 to boot on.
But I don't know how fill that symbole with data. I understand it's the
goal of `__attribute__((section (".grub_sig")))` but I can't use it in Go.
My linker is enclosed.

I don't know either ^^
I don't use -l and if I add -lgo then -lgcc :

ld: cannot find -lgo
ld: cannot find -lgcc

My Makefile is also enclosed.

I share my main.go too if that can help.

Thanks a lot

--

Off topic : that's weird, I had an error code 340 when I used the
GoogleGroups web interface to send my files. I had to use my gMail and even
that, the Makefile is missing on GoogleGroups :/

--

Change

.grub_sig 0xC0100000 : AT(0x100000)
{
*(.grub_sig)
}

to

.grub_sig 0xC0100000 : AT(0x100000)
{
grub_sig.o(*)
}

This isn't black magic, it's all documented at
http://sourceware.org/binutils/docs-2.23.1/ld/Scripts.html . This is
the kind of thing you are going to need to understand in order to
succeed with this project.
Don't link by running the program "ld". That is rarely correct.
Instead, run the program "gccgo". That is, set LD=gccgo in your
Makefile. Then change LDFLAGS to be -Wl,-T,linker.ld,-n. You may
also want to use -nostdlib, I don't know. See the GCC documentation
on -nostdlib and -nostartfiles.

Ian

--

Le jeudi 20 décembre 2012 22:07:29 UTC+1, Ian Lance Taylor a écrit :
Thanks.

I didn't know that. I have a hard time finding documentation about Kernel
Programming.
I will look that closely ;)

Don't link by running the program "ld". That is rarely correct.

I keep you update after I read that documentation if I have some troubles.
Thanks again for all your help, your pieces of advices and your links :)

--

If you haven't already seen it, http://wiki.osdev.org/Main_Page has
lots of reading material.

--

Very interresting links, thanks a lot :)


Le vendredi 21 décembre 2012 02:35:36 UTC+1, Nigel Tao a écrit :

This fellow have some attitude! well done.. thats the spirit of
hacking :) (i dont know why people are so sensitive these days)

A kernel in go? i would love to play with something like that some
time.. if you get somewhere with this..
please share your code.

the guy who could take linux drivers of the linux kernel, and create a
modular and reusable/kerneless piece of code with it..
would be a hero.. i´ve always dreamed about a exokernel

i just know of freebsd folks who have this linux driver layer to load
up with the freebsd kernel

On Fri, Dec 21, 2012 at 11:40 AM, Guillaume Lescure
wrote:

Um, yes.

Writing a kernel in a garbage-collected language requires rather
fancy machinery in the garbage collector, or some way to allocate
memory that never gets garbage collected. It's been done, but the
whole OS architecture then tends to revolve around garbage collection
issues. See Symbolics Lisp Machine.

A related goal would be to develop a minimal runtime system for
Go that allowed one Go program to run on bare hardware. The
runtime system might not be in Go; it's below it. That
would be worth having for ARM CPUs, for embedded applications.
The runtime would need to be able to turn interrupts into channel
events. You'd need a CPU scheduler. Go is still using a garbage
collector of the "stop the world" variety, so hard real-time, and
some device drivers, won't work during GC. You might have to
do a no-GC system, where all allocation happens at startup and
you never allocate again. Many packages may be incompatible with
that, though.

John Nagle



--

As described o a previous post of mine, have a look at Native Oberon
and
AOS (aka BlueBottle).

Those systems have a minimal assembly runtime, with everything else
written
it the corresponding systems language (GC enabled).

http://www.ocp.inf.ethz.ch/wiki/Documentation/Kernel
http://e-collection.library.ethz.ch/eserv/eth:26082/eth-26082-02.pdf

The Oberon's language family system module is similar to Go's unsafe +
syscall.

Thank a lot to minux for the "-fno-split-stack" flag that resolve my
"__more-stack" problem (
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/y2RIy0XLJ24
).

And I solve the "undefined reference to `__go_type_hash_identity'" deleting
my structure in the grubSig file.
The previous error :

main.8g:(.rodata+0xcc): undefined reference to `__go_type_hash_identity'

If I understand well, the compiler can't create structures because it
doesn't know how to handle them especially the access and the modification
of value. These functions are defined in the standard library that I don't
link with. So no structure in kernel ? But does that mean I can't use
structures in my kernel ? Sounds weird becase I can do it in C and without
it will be far more difficult :/ Any idea ?

And just for curiosity, did the garbage-collecting is disable when I don't
link with standard library ? How manage memory ?

--

Write a memory manager then write an GC ontop of it of course. You are
writing an OS.

--

Oh yeah sorry, even in C, malloc and free are not avaible directly before
handling memory management. Silly question, my bad ;)


Le lundi 14 janvier 2013 23:45:54 UTC+1, bryanturley a écrit :

You more or less have to link with the Go library in order to link Go
code. It's not really optional. You can avoid specific parts of the
library, but you can't avoid the whole thing.

The function __go_type_hash_identity will be used as the equality
operator for a struct with no padding, each of whose fields can be
compared for equality using memcmp. This function will appear in the
type reflection structure for the struct. There is nothing similar
this concept in C.
You will have to provide your own functions to allocate memory and to
free it, somehow.

Ian

--

2013/1/15 Ian Lance Taylor <iant@google.com>

Ok so I take out the "-nostdlib" and "-nodefaultlibs" flags and no problems
without structures. But if I add one, now I have :

/usr/bin/ld: kernel: could not find output section .gnu.hash

/usr/bin/ld: final link failed: Nonrepresentable section on output


I tried to add the ".hash" and ".gnu.hash" section in my script and I added
the "--hash-style" flag with "both" or "gnu" but the error stays (it
changes ".gnu.hash" to ".hash"). I run out of ideas to make it work but I'm
pretty sure it's almost good.

I shared my Makefile and my linker script.


On other hand, I really thought Go is not bad to kernel programming. I
completely agree with everyone before me, without runtime it's lost a lot
of good part of the language but it's normal it's kernel programming, you
can't have everything. But if we compare it with C, I think Go has some
advantages. I even think it could be kind of better than C if we make
kernel programming easier than with C. So, it's just an idea but what do
you think about including that in the go compiler to make it easy for
everyone else? Add a new keyword for the go toolchain to make kernel ? I
think "kernel" is simple and clear about what the command do so it would
look like :

$go kernel .


And it do the same thing than the "install" command but it take at least 2
files :

kernel.go :

---------------



void main() { // as usual, so the entry symbole is kernel.main

// your kernel here

}


-------------



// here your signature


And if we add some packages about kernel programming, we could statically
link every kernel with theses packages and make a kernel in few lignes. For
example, the grub signature :

-------------



const (

GrubMagic = 0x1BADB002

GrubFlags = 0x0

GrubChecksum = (-1 * (grubMagic + grubFlags))

)



type Signature struct {

Magic uint

Flags uint

Checksum int

}

return &Signature{GrubMagic, GrubFlags, GrubChecksum}

}


So after :

-------------



import "boot/grub"

sig = *grub.NewSignature()

)


And in the documentation, it would be great to mark wich package can be
statically linked for kernel. Like, for example, the "unsafe" package.

That's it ... all I've got ^^
It's raw but I really think that could be very good for Go and kernel
programmers.
What do you think ?

--

This error message implies that you are building a dynamically linked
executable or shared library. You don't want that. You need to use
-static, as you were earlier.

Ian

--

Le mardi 15 janvier 2013 15:05:53 UTC+1, Ian Lance Taylor a écrit :
Oh sorry yeah, I forgot to put it back because of the bug I highlighted
here :
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/y2RIy0XLJ24
So I add "-Wl,-u,pthread_create" and "-static" like you told me but now :

/usr/bin/ld: BFD (GNU Binutils for Ubuntu) 2.22.90.20120924 assertion fail

I'm so sorry to bother you with my errors but I really don't know how solve
them :/
Thanks a lot for your help Ian.

--

Sorry, I don't know what is causing that. Are you using a linker
script? If so, the problem is most likely that your linker script
isn't doing anything with the .ctors section.

Ian

--

script? If so, the problem is most likely that your linker script

isn't doing anything with the .ctors section.


Yeah I shared my linker script before. I added an .ctors and .dtors
sections and it's works but I have a warning :

/usr/bin/ld: kernel: section .tdata lma 0x1a0158 adjusted to 0x1a0214


So I try to add a .tdata section in my linker script but I have :

/usr/bin/ld: BFD (GNU Binutils for Ubuntu) 2.22.90.20120924 assertion fail

I also try to put it at 0x1a0214 with ". = 0x1a0214" but it's worst.

So I'm not sure what to do. I don't want to have a warning and I just want
to make my kernel right. I understand now the synthax of the script (thanks
to your link) but I have no idea wich section I have to put first or after,
wich size for the sections and wich one should be mention or discarded. I
understand what I do but I have no idea to what my binary should look like
and, at this point, my tutorials can't help me. I studied the C linker
script and now the C++ linker scripts (because he has .ctors and .dtors
sections) but I don't know what a Go linker script should be.

I share my new version of the linker script in this mail but feel free to
tell me what's wrong beacause I'm pretty sure there is a lot of mistakes.



2013/1/15 Ian Lance Taylor <iant@google.com>

If you run GNU ld with the --verbose option it will show you the
default linker script. I recommend starting from that and editing it.

Ian

--

2013/1/16 Ian Lance Taylor <iant@google.com>

Yeah thanks a lot, it works now :D
Like you said I started from the default script and I change some minor
things and voila ... a go kernel wih structures :)

The boot works great and the kernel is running. So I begin to write some
packages but I found a weird problem (yeah sorry again) :/
When I declare a pointer or an array, in a function scope or in the package
(even the main package), I have an error :

error: overlap detected.

So I think, there is a safety that kill my kernel when someting's wrong so
it's why the "no loaded kernel" is displayed but I don't get the "overlap"
thing ... ideas ?

PS : the Makefile and the new linker are enclosed in this email for who
want to know/try :)

Le jeudi 17 janvier 2013 00:43:56 UTC+1, Guillaume Lescure a écrit :
I think I partially found the solution, like said here :
http://wiki.osdev.org/Object_Files#Relocating_code, the data overlap in
some cases so I have to use the -q flag to the linker with -Wl but that
doesn't change anything :(

--

Salut Guillaume,

This is a very interesting project and you seem to be making good constant
progress. It would be awesome once you're (mostly) done if you could write
a blog post or some doc (wiki?) in one way or another about how you pulled
that off, what steps or tweaks or compiler flags were required, etc. I
think this would make for an interesting read and be a great contribution
to the community. Obviously some of this doc is right here in this thread,
but a cleaned-up, here's what I've done post would be much better. And this
has hacker news front page written all over it :) Anyway, that's your call,
just saying I'd +1 this.

Best of luck for the next steps, and sorry for the kinda meta-post.

--

Hi Guillaume,
I looked into this a few months back, and managed to get some really basic
code that will boot from grub and drop you into a Go function. As others
have mentioned, without the runtime it's pretty useless, but it might be a
fun project to try and get something useful running.

The code is up here if you're interested:
https://github.com/tgascoigne/goose

Tom

Hi Tom,

I try to clone and compile your project but I had the same problems that
with mine :

make: Dépendance circulaire video.go <- video.go.o abandonnée.
make: Dépendance circulaire goose.go <- goose.go.o abandonnée.
ld -T link.ld -o kernel.bin loader.o video.go.o video.gox goose.go.o
ld: architecture i386 du fichier d'entrée « loader.o » est incompatible
avec la sortie i386:x86-64
video.go.o: dans la fonction « go.video.Init »:
video.go:(.text+0x18): référence indéfinie vers « __morestack »
video.go.o: dans la fonction « go.video.Print »:
video.go:(.text+0x4c): référence indéfinie vers « __morestack »
video.go.o: dans la fonction « go.video.PutChar »:
video.go:(.text+0xcf): référence indéfinie vers « __morestack »
video.go.o: dans la fonction « go.video.Clear »:
video.go:(.text+0x369): référence indéfinie vers « __morestack »
video.go.o: dans la fonction « go.video..import »:
video.go:(.text+0x58a): référence indéfinie vers « __morestack »
goose.go.o:goose.go:(.text+0x18): encore plus de références indéfinies
suivent vers « __morestack »
make: *** [kernel.bin] Erreur 1

I think I have a big problem ^^

PS : your project is quite cool, it's a good start for what I look for :)


Le vendredi 21 décembre 2012 19:48:36 UTC+1, Tom Gascoigne a écrit :

Is that possible to build in golang something like http://xen.openmirage.org/ ?
What we need to port/rewrite to use full golang runtime?

2012/12/20 Guillaume Lescure <guil.lescure@gmail.com>:

Le mercredi 16 janvier 2013 11:27:41 UTC+1, Vasiliy Tolstov a écrit :
I'm not sure but I think memory management and syscalls are mandatory for
the runtime ... so you can use runtime when your system is already pretty
finished ^^

--

i think it's quite possible and this task will be very fun!

as xen provides a lot of hypercalls for the guest OS, I think targeting Go
to it
won't be as hard as targeting Go to bare mental as necessary drivers are
already complete.

For one thing, the Go runtime is already quite self-contained, it doesn't
depend
on libc, for example, whereas OCaml's runtime depends on libc.

As I understand it, Go's runtime only needs memory management help from
the exokernel, and it's capable of scheduling goroutines itself.

--

"...bare mental as..." Freudian slip?

--