[go-nuts] panic on arm cross compile

If your host has more than 128mb of ram can you try compiling locally. You
can grab precompiled tarballs
from http://dave.cheney.net/unofficial-arm-tarballs. The installation
instructions are identical to the official tarballs from the website.

Unfortunately our devices have virtually zero space left on their flash
drives where the OS is mounted. Any ideas on why the cross compile isn't
working? Or perhaps some more information I can gather that might help? I
am a bit clueless in this realm :(

Can you make the binary available somewhere, I'll test it on my machine.

What kernel is the machine running ?

Binary is available at http://temp-host.com/download.php?file=br91ln
cat /proc/version gives Linux version 3.2.40 (kman@kmachine) (gcc version
4.6.4 (Linaro GCC branch-4.6.4. Marvell GCC Dev 201310-2126.3d181f66 64K
MAXPAGESIZE ALIGN) ) #1 Wed Feb 11 16:59:39 CST 2015
These boxes are pretty custom so I wouldn't be surprised if there are
kernel changes.

Thanks I really appreciate the help! I love golang in my own projects and
there are some perfect use cases at the company I am with, so naturally
trying to get signoff. Hopefully I can add into the success stories and get
some open source work going too!

That file worked on an armv7 host.

I suggest downloading the 1.4.2 tarball mentioned above, extracting the
bin/go binary and running that on your host. If that doesn't work you may
have to provide more information about the customisations to your kernel.

Hey Dave, I tried both tarballs extracted to the device on some removable
storage, and both give not found errors. I'll see if I can't get more
information on the kernel, but I was hoping to run go on multiple devices
and they are all different.. most ARM though.
Most are some form of busybox on top of debian or some such thing. Simple C
programs compile with the toolchain and run without any issues.

From your original message it looked like you were trying to cross compile
with cgo enabled. There is a long answer to why this is difficult that
other may give, so i'll just simplify it to, please don't try.

I think if you build a new go install from source (it's literally 2
commands), the env GOARCH=arm GOOS=linux ./make.bash --no-clean to create
your cross compile env, you'll have better results.

Is there anyway I can do this on my machine and not the arm devices? The
devices have zero dev tools available and all we have available for them
are the toolchains. Perhaps I am missing something

Looks like I get further in the init on this particular device if I use
GOARM=5 as seen below:
unexpected fault address 0xaa2a0
fatal error: fault
[signal 0xb code=0x2 addr=0xaa2a0 pc=0xaa2a0]

goroutine 1 [running, locked to thread]:
runtime.gothrow(0xd29f8, 0x5)
/usr/local/go/src/runtime/panic.go:503 +0x84 fp=0x10335f1c sp=0x10335f10
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:29 +0x2a4 fp=0x10335f44
sp=0x10335f1c
time.init()
/usr/local/go/src/time/zoneinfo_unix.go:84 fp=0x10335f48 sp=0x10335f48
fmt.init()
/usr/local/go/src/fmt/scan.go:1169 +0x64 fp=0x10335f94 sp=0x10335f48
fmt.init()
/usr/local/go/src/fmt/scan.go:1169 +0x64 fp=0x10335fe0 sp=0x10335f94
runtime.main()
/usr/local/go/src/runtime/proc.go:46 +0x88 fp=0x10336004 sp=0x10335fe0

goroutine 2 [runnable]:
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:90
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:1322 +0x4

goroutine 3 [runnable]:
runtime.bgsweep()
/usr/local/go/src/runtime/mgc0.go:82
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:1322 +0x4



Does this have something to do with it being an ARM7 device with soft
floats?

according to cpuinfo, your device does have vfpv3, so you can use GOARM=7.
The fact that your OS is softfloat shouldn't affect this.
this segmentation fault is interesting, notice that addr == pc == 0xaa2a0.

what's the instruction at 0xaa2a0? Also, try running the program with
GOTRACEBACK=2.

When I use GOARM=7 I get the panic always in a different place than GOARM=5
which was the last I posted. GOARM=7 looks like the first post:
unexpected fault address 0xa8e24
fatal error: fault
[signal 0xb code=0x2 addr=0xa8e24 pc=0xa8e24]


goroutine 1 [running, locked to thread]:
runtime.gothrow(0xcc9f8, 0x5)
  /usr/local/go/src/runtime/panic.go:503 +0x84 fp=0x10335eb4 sp=0x10335ea8
runtime.sigpanic()
  /usr/local/go/src/runtime/sigpanic_unix.go:29 +0x2a4 fp=0x10335edc sp=
0x10335eb4
sync/atomic.LoadUint32(0x126b78, 0x0)
  ?:0 fp=0x10335ee0 sp=0x10335ee0
syscall.Getenv(0xd, 0x1030a060, 0x72ccc, 0xcecf0, 0x8)
  /usr/local/go/src/syscall/env_unix.go:72 +0x54 fp=0x10335f0c sp=0x10335ee0
syscall.Getenv(0x10335f60, 0x30315500, 0x1030a060, 0x5e064, 0x10338010)
  /usr/local/go/src/syscall/env_unix.go:72 +0x54 fp=0x10335f38 sp=0x10335f0c


goroutine 2 [runnable]:
runtime.forcegchelper()
  /usr/local/go/src/runtime/proc.go:90 fp=0x103187ec sp=0x103187ec
runtime.goexit()
  /usr/local/go/src/runtime/asm_arm.s:1322 +0x4 fp=0x103187ec sp=0x103187ec
created by runtime.init·4
  /usr/local/go/src/runtime/proc.go:87 +0x34


goroutine 3 [runnable]:
runtime.bgsweep()
  /usr/local/go/src/runtime/mgc0.go:82 fp=0x1031bfec sp=0x1031bfec
runtime.goexit()
  /usr/local/go/src/runtime/asm_arm.s:1322 +0x4 fp=0x1031bfec sp=0x1031bfec
created by gc
  /usr/local/go/src/runtime/mgc0.c:1386




And if I use objdump -m i386 then the relevant section around 0xa8e24 looks
like:
000a8dfc <sync/atomic.AddUint32>:
    a8dfc: 04 e0 add $0xe0,%al
    a8dfe: 2d e5 08 20 9d sub $0x9d2008e5,%eax
    a8e03: e5 0c in $0xc,%eax
    a8e05: 40 inc %eax
    a8e06: 9d popf
    a8e07: e5 00 in $0x0,%eax
    a8e09: 00 92 e5 00 10 a0 add %dl,-0x5fefff1b(%edx)
    a8e0f: e1 04 loope a8e15 <sync/atomic.AddUint32+0x19>
    a8e11: 10 81 e0 e1 ff ff adc %al,-0x1e20(%ecx)
    a8e17: eb fa jmp a8e13 <sync/atomic.AddUint32+0x17>
    a8e19: ff (bad)
    a8e1a: ff (bad)
    a8e1b: 3a 10 cmp (%eax),%dl
    a8e1d: 10 8d e5 04 f0 9d adc %cl,-0x620ffb1b(%ebp)
    a8e23: e4 04 in $0x4,%al


000a8e24 <sync/atomic.LoadUint32>:
    a8e24: 04 e0 add $0xe0,%al
    a8e26: 2d e5 08 20 9d sub $0x9d2008e5,%eax
    a8e2b: e5 00 in $0x0,%eax
    a8e2d: 00 92 e5 00 10 a0 add %dl,-0x5fefff1b(%edx)
    a8e33: e1 d9 loope a8e0e <sync/atomic.AddUint32+0x12>
    a8e35: ff (bad)
    a8e36: ff (bad)
    a8e37: eb fb jmp a8e34 <sync/atomic.LoadUint32+0x10>
    a8e39: ff (bad)
    a8e3a: ff (bad)
    a8e3b: 3a 0c 10 cmp (%eax,%edx,1),%cl
    a8e3e: 8d (bad)
    a8e3f: e5 04 in $0x4,%eax
    a8e41: f0 9d lock popf
    a8e43: e4 f6 in $0xf6,%al


000a8e44 <sync/atomic.LoadUintptr>:
    a8e44: f6 ff idiv %bh
    a8e46: ff (bad)
    a8e47: ea ff ff ff ea 04 e0 ljmp $0xe004,$0xeaffffff

What does that mean that that spot in memory doesn't contain that function?
(this is not my area of expertise :D)?

The crash has nothing to do with GOARM values. notice that addr is equal to
pc again.
what's your kernel version? Try running the program with $GOTRACEBACK=2,
e.g.

GOTRACEBACK=2 ./theprogram

the binary is for ARM, so you need an arm disassembler. If you don't have
one, you can
upload the binary somewhere, and I can take a look.

000a8dfc <sync/atomic.AddUint32>:
I really don't understand how that's possible (segfaulting in .text
section).
Something is weird with the environment.

Binary using GOARM=5 is at http://temp-host.com/download.php?file=hp61ll

When running with GOTRACEBACK=2 I don't see but a couple "created by" lines:
unexpected fault address 0xaa2a0
fatal error: fault
[signal 0xb code=0x2 addr=0xaa2a0 pc=0xaa2a0]

goroutine 1 [running, locked to thread]:
runtime.gothrow(0xd29f8, 0x5)
/usr/local/go/src/runtime/panic.go:503 +0x84 fp=0x10335f1c sp=0x10335f10
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:29 +0x2a4 fp=0x10335f44
sp=0x10335f1c
time.init()
/usr/local/go/src/time/zoneinfo_unix.go:84 fp=0x10335f48 sp=0x10335f48
fmt.init()
/usr/local/go/src/fmt/scan.go:1169 +0x64 fp=0x10335f94 sp=0x10335f48
fmt.init()
/usr/local/go/src/fmt/scan.go:1169 +0x64 fp=0x10335fe0 sp=0x10335f94
runtime.main()
/usr/local/go/src/runtime/proc.go:46 +0x88 fp=0x10336004 sp=0x10335fe0

goroutine 2 [runnable]:
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:90 fp=0x103187ec sp=0x103187ec
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:1322 +0x4 fp=0x103187ec sp=0x103187ec
created by runtime.init·4
/usr/local/go/src/runtime/proc.go:87 +0x34

goroutine 3 [runnable]:
runtime.bgsweep()
/usr/local/go/src/runtime/mgc0.go:82 fp=0x1031bfec sp=0x1031bfec
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:1322 +0x4 fp=0x1031bfec sp=0x1031bfec
created by gc
/usr/local/go/src/runtime/mgc0.c:1386

code=0x2 is SEGV_ACCERR, which means the address is not mapped for
execution.

   [ 1] .text PROGBITS 00010c00 000c00 09d7ec 00 AX 0
0 4

   Type Offset VirtAddr PhysAddr FileSiz
  MemSiz Flg Align
   LOAD 0x000000 0x00010000 0x00010000 0x9e3ec 0x9e3ec R E 0x1000

The address 0xaa2a0 should definitely be mapped for execution.

disassembly showed that 0xaa2a0 is the very first instruction of the
time.init function.
000aa2a0 <time.init>:
    aa2a0: e59a1008 ldr r1, [sl, #8]
    aa2a4: e15d0001 cmp sp, r1
    aa2a8: 91a0300e movls r3, lr
    aa2ac: 9bfe7d38 blls 49794 <runtime.morestack_noctxt>

I really don't understand what's happening here. Something is preventing
the OS from
mapping that page for execution.

I know these devices have page size changes. And the /proc/version file
shows:
Linux version 3.10.39 (kman@kmachine) (gcc version 4.6.4 (Linaro GCC
branch-4.6.4. Marvell GCC Dev 201310-2126.3d181f66 64K MAXPAGESIZE ALIGN) )
#26 SMP ...

which has the word align in it... :( I'll see if I can't get some details
from the vendors.

OK. That explains it. Assume the page size is 64KB, there is page
permission conflict
in the last page of .text segment and the start of .rodata segment.

   Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
   PHDR 0x000034 0x00010034 0x00010034 0x000c0 0x000c0 R 0x1000
   LOAD 0x000000 0x00010000 0x00010000 0x9e3ec 0x9e3ec R E 0x1000
   LOAD 0x09f000 0x000af000 0x000af000 0x75e11 0x75e11 R 0x1000
   LOAD 0x115000 0x00125000 0x00125000 0x05388 0x14ac8 RW 0x1000
   GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x4
   LOOS+5041580 0x000000 0x00000000 0x00000000 0x00000 0x00000 0x4

The .text segment ends at 0x9efff, and the next read-only segment starts at
0x9f000, and those are in the same 64KB page. Normally, the kernel should
either reject such binaries or set the permission of conflicting page to
the larger
set (R + E in this case), but somehow your kernel sets the permission to the
smaller one (R only).

I think it's a kernel bug, but beware that we've never tested linux/arm on
systems
non-4KB page size systems, there might be other problems after fixing the
kernel.

Ah thanks for looking, but this is bad news! :( Looking at some of the
C/C++ projects around it looks like on most devices they compile with
-fPIC. Would that help here? Could I add it into the cross compile?
I tried just using
CC_FOR_TARGET="<toolchain gcc> -fPIC" ... ./make.bash
but get the same results.

I am just shooting in the dark here but is it perhaps within go like it
maybe was for powerpc?
https://github.com/golang/go/commit/f9fdc887ae71ebcf26c980f2f15ace2efec94881

https://github.com/golang/go/blob/release-branch.go1.4/src/runtime/arch_arm.h

Reaaaally hoping I won't be going back to javaland !

Email me off list, I'll send you some patches to try

Just setting the PageSize in runtime won't help in this case. Some cmd/ld
changes
are needed to round the section sizes to multiples of 64K.

We really need builders that use 64k pages.

So is this a fix needed in go or in our kernel? If go, is anyone interested
in looking at this? If in the kernel, how do I get it to show up through a
C program?

In Go. You can work around it using a kernel with 4k pages, but
presumably you have a reason to use a kernel with 64k pages.

Yes we use 64k pages for other performance reasons and certainly can't be
changed.
So how do I wind up the interest and get more eyes on this? :D Are these
kind of things a priority for the go team or do I need to put some serious
time into brushing up with C?

I will deal with it as it's affecting arm64 as well.

Unfortunately I can't promise a timeframe, but it will get fixed.

Also, if you want to help you don't need to brush up with C, as the
runtime is all in Go and assembly now, no more C code in the tree.

Thanks Aram that's great news! I will start looking into the runtime once
1.5 is out for sure. Is there a ticket I can follow progress on for this
issue?

Hey Aram, just wanted to shoot you what one of our kernel guys said to see
if it jives with the changes that will be made.

The kernel is set to a 64K page size for performance reasons. In order to accomplish this, we had to change the linker so that the elf tables of all binaries are 64K aligned. So, binaries will run on a 4K kernel, but 4K aligned binaries from a standard distro (actually they are 32K aligned), will not run.
That doesn't put a wrench in the works does it?

Yes, quite a lot

I think we can just increase the section alignment to something even larger,
say 1MB. i will send a patch.

For the runtime, I'd prefer that _PhysPageSize be made a variable and
automatically detected. Non-4k page size systems are increasingly popular,
and Go's tradition is to the correct thing automatically.

I thanks minux for the offer of a patch. Jonathan, I can make you up a few test binaries to try.

Minux, I've been thinking that we can recover the page size from auxv. This bit isnt hard, but it would mean that _PageSize would become non constant on linux platforms and this may have some unexpected effects on the efficiency of the gc.

The simplest fix is to add -ldflags "-R 65536" to cmd/go command line
to build your binaries.

However, that will increase the size of the binary slightly. And I'm
looking for a solution that will set the INITRND value to a larger value
without impacting the size of the executable so that we can set that
to a much larger value (e.g. 4MB) and Go binary will be runnable on
machine with page size as large as 4MB.

Well that totally fixes everything! I even tried a little example that
utilizes cgo to make sure that works too.
I very much appreciate this workaround, and may I ask where if anywhere the
-R is documented so I can better arm myself in the future?

It's only documented in the go tool 5l output.... As I've said, for Go 1.5,
I want to have
a better solution for this problem so that the binary just works.

I've filed #10180 for this. the linker docs gets worse on the tip, and
that's #10179.

Hey minux,
Trying to call some of the image package functions, I am getting "cannot
map pages in arena address space". Does this have anything to do with
using -ldflags "-R 65536"? Granted the box doesn't have a lot of memory,
but shouldn't it start paging when it runs out?

Tried changing all ulimits to unlimited with no success. The program is
just decoding a jpeg, that's the failing line with jpeg.Decode().

I shouldn't have anything to do with -ldflags "-R 65536".

Please strace the process to see what does the failing mmap syscall
(should be the last one) returns, perhaps you need Dave's patch to
set _PhysPageSize to 65536 in the runtime (if the error is EINVAL).

strace does show mmap2 EINVAL, and a very long list of other stuff :D I'll
try with Dave's patch in just a bit.

Page size will be an issue, can you show that mmap strace results?

Minux, Dave, it does work with the -ldflags "-R 65536" and having done the
make.bash with the _PhysPageSize set to 65536. Thanks you guys rock! Will
this be included in 1.5?

Dave the patch also had one other thing which you said was disabling the
scavenger, what is that and what would it affect? (thinking potential other
failures cropping up here)

Great news. Is this with the other 64k patch I sent you?

This was with a vanilla checkout of go1.4.2, thanks for the patch though!

I have not had any more issues with our weird devices on ARM for go1.4,
that's good :) Sorry to beat a dead horse here but will this needing to
change PhysPageSize and give the linker option be fixed for go1.5? I don't
see any references to PhysPageSize in gotip or any PageSize reference that
has a meaningful number, so I am thinking the answer is yes.

Thanks!

that's good :) Sorry to beat a dead horse here but will this needing to
change PhysPageSize and give the linker option be fixed for go1.5? I don't
see any references to PhysPageSize in gotip or any PageSize reference that
has a meaningful number, so I am thinking the answer is yes.

There is issue 10180 for the non-4K page size problem. I intend to get it
fixed for Go 1.5.