Flops
Making go1.12 fly on OS X Mavericks (10.9.5)
15 May 2019
If you are like me and just cannot live without having go1.12 run on your trusty Mavericks, you need
to install libMacportsLegacySupport from project MacPorts. Check whether you already
have it installed:
$ ls -la /usr/local/lib/*cySu* -rw-r--r-- 1 root wheel 19848 May 11 07:47 /usr/local/lib/libMacportsLegacySupport.a -rwxr-xr-x 1 root wheel 19012 May 11 07:47 /usr/local/lib/libMacportsLegacySupport.dylib
I initially had not. As I am not letting gigabytes of useless crap to waste space on my hard disk,
I installed
macports-legacy-support directly:
$ mkdir macports $ ( cd macports/ && git clone https://github.com/macports/macports-legacy-support ) Cloning into 'macports-legacy-support'... remote: Enumerating objects: 32, done. remote: Counting objects: 100% (32/32), done. remote: Compressing objects: 100% (28/28), done. remote: Total 625 (delta 14), reused 8 (delta 3), pack-reused 593 Receiving objects: 100% (625/625), 156.30 KiB | 0 bytes/s, done. Resolving deltas: 100% (394/394), done. Checking connectivity... done. $ ( cd macports/macports-legacy-support/ && sudo make install ) >& macports-legacy-support.install.log $ ls -la /usr/local/lib/*cySu*
Alternatively, you can consult with project MacPorts on how to install legacy support for OS X.
You need a working version of Golang to bootstrap go1.12. I used /usr/local/go1.9.7,
however your version may be different.
Examples that follow work best if you begin in your home directory, your shell is /bin/bash
and your GOHOME is default $HOME/go.
Parte 1
Now that libMacportsLegacySupport was installed, I used
ticket and
patch from MacPorts to get working version of go1.12
for a starter. I created and downloaded a
fork of golang/go:
$ /usr/local/go1.9.7/bin/go get github.com/av86743/go package github.com/av86743/go: no Go files in /Users/ubuntu/go/src/github.com/av86743/go $
The patched code is now on the branch
go1.12.5-osx-legacy-take1 of this fork. After certain
tweaking of environment variables, suitable build incantation turned out to be this:
$ ( cd go/src/github.com/av86743/go/ && git checkout go1.12.5-osx-legacy-take1 )
Checking out files: 100% (10463/10463), done.
Previous HEAD position was 6174b5e... go1
Branch go1.12.5-osx-legacy-take1 set up to track remote branch go1.12.5-osx-legacy-take1 from origin.
Switched to a new branch 'go1.12.5-osx-legacy-take1'
$ ( cd go/src/github.com/av86743/go/ && cd src && configure_ldflags='-L/usr/local/lib -lMacportsLegacySupport' && env GOROOT_BOOTSTRAP=/usr/local/go1.9.7 GO_EXTLINK_ENABLED=1 GO_LDFLAGS='"-extldflags='"${configure_ldflags}"'"' BOOT_GO_LDFLAGS='-extldflags='"${configure_ldflags}"'' ./make.bash )
Building Go cmd/dist using /usr/local/go1.9.7.
Building Go toolchain1 using /usr/local/go1.9.7.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
clang: warning: argument unused during compilation: '-nopie'
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
---
Installed Go for darwin/amd64 in /Users/ubuntu/go/src/github.com/av86743/go
Installed commands in /Users/ubuntu/go/src/github.com/av86743/go/bin
$
Everything seemed to be in order, so far.
Parte 2
Original instructions to install golang from sources actually invoke
all.bash. all.bash
invokes make.bash and after that run.bash.
It is the latter that runs golang tests. I replaced make.bash with
all.bash, but this failed even before the tests started, because environment
variable BOOT_GO_LDFLAGS was still set after make.bash. This was one thing to fix.
MacPorts solution reqiures that golang environment variable GO_LDFLAGS is set to
(or contains) a very specific value in order to link golang executable externally with
libMacportsLegacySupport. This is not acceptable - consider the case when golang is invoked
from script beyond your control and already uses its own setting of GO_LDFLAGS.
To fix this, I added code to always include libMacportsLegacySupport when linking golang
executable externally. With this, GO_LDFLAGS need not be involved at all.
The patched code is now on the branch
go1.12.5-osx-legacy-take2 of the fork. Build
incantation became simpler:
$ ( cd go/src/github.com/av86743/go/ && git checkout go1.12.5-osx-legacy-take2 ) Branch go1.12.5-osx-legacy-take2 set up to track remote branch go1.12.5-osx-legacy-take2 from origin. Switched to a new branch 'go1.12.5-osx-legacy-take2' $ ( cd go/src/github.com/av86743/go/ && cd src && configure_mplsldflags='-L/usr/local/lib -lMacportsLegacySupport' && env GOROOT_BOOTSTRAP=/usr/local/go1.9.7 ./all.bash ) Building Go cmd/dist using /usr/local/go1.9.7. Building Go toolchain1 using /usr/local/go1.9.7. Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1. clang: warning: argument unused during compilation: '-nopie' Building Go toolchain2 using go_bootstrap and Go toolchain1. Building Go toolchain3 using go_bootstrap and Go toolchain2. Building packages and commands for darwin/amd64. ##### Testing packages. ok archive/tar 0.111s ok archive/zip 3.264s <...boring...details...omitted...> ok cmd/vendor/golang.org/x/arch/x86/x86asm 0.138s ok cmd/vendor/golang.org/x/crypto/ssh/terminal 0.015s ok cmd/vendor/golang.org/x/sys/unix 2.333s ok cmd/vet 18.499s 2019/05/16 12:04:10 Failed: exit status 1 $
With that, all.bash started to run tests. However, cross-compile tests were failing,
and on top of this several tests were giving segfault in FetchPEMRoots.
Segfaulting? In golang own crypto/x509? What the fkety-fk.
Intermezzo
I am a perfect tester. Whenever I give a try to another piece of some shitey half-baked software,
it instantly hits one of tripwires which I have naturally placed around. As luck had it this time,
"my problem" stemmed from user certificate present on my account. After a bit of boring bisecting,
I narrowed down segfault to this fragment of
isSSLPolicy():
CTypeRef value = NULL;
if (CFDictionaryGetValueIfPresent(properties, kSecPolicyOid, (const void **)&value) {
CFRelease(properties);
return CFEqual(value, kSecPolicyAppleSSL);
}
which obviously works much better when changed like this.
Here is a shapshot of the corresponding blame:
How very convenient. Evading detection for 9 months, and counting. So much for the most friendly code review.
Naturally, G shiteheads don't even bother to validate code which they write. And why would they - when they are so-o-o GOOGLE. Let's see how long does it take from them a) to notice that problem has been exposed and - if ever - b) to apply the fix.
In any case, if you are getting segfaults in FetchPEMRoots, apply patch from
this branch of the fork and see if this does the trick.
Parte 3
Cross-compile tests were failing because MacPorts solution has second requirement: it forces
golang to always use external link mode using environment variable setting
GO_EXTLINK_ENABLED=1. Conveniently, setting GO_EXTLINK_ENABLED=1
while building golang itself has an additional global effect of changing default value of
GO_EXTLINK_ENABLED to 1. This breaks cross-compile tests as latter naturally expect
default link mode to be internal.
Setting default value of GO_EXTLINK_ENABLED to 1 is of course a bad idea. But what is a
really bad idea, is to give an option to subtly change default value of GO_EXTLINK_ENABLED.
Not a single one official golang build uses default setting of 1 for GO_EXTLINK_ENABLED.
Consequently, golang tests were never ever run with default setting of 1. And I give you this
accurate prediction: no one at golang will ever bother to fix tests with regard to this. Indeed,
I will not be one to blame them for not doing it.
In order to verify that my understanding is correct, I additionally changed selection of link mode in
determineLinkMode() to be always internal for cross-compiles,
even when link mode is forced to be external by setting variable GO_EXTLINK_ENABLED=1
(explicitly or by default). This is not acceptable as a permanent solution as it changes the default
behavior of golang with regard to setting of GO_EXTLINK_ENABLED. However, at this
point I only wanted to see whether tests will pass after this modification.
The patched code is on the branch
go1.12.5-osx-legacy-take3 of the fork. Build
incantation remains the same:
$ ( cd go/src/github.com/av86743/go/ && git checkout go1.12.5-osx-legacy-take3 ) Branch go1.12.5-osx-legacy-take3 set up to track remote branch go1.12.5-osx-legacy-take3 from origin. Switched to a new branch 'go1.12.5-osx-legacy-take3' $ ( cd go/src/github.com/av86743/go/ && git show 2ce3919 | patch -p1 ) patching file src/crypto/x509/root_cgo_darwin.go $ ( cd go/src/github.com/av86743/go/ && cd src && configure_mplsldflags='-L/usr/local/lib -lMacportsLegacySupport' && env GOROOT_BOOTSTRAP=/usr/local/go1.9.7 ./all.bash ) Building Go cmd/dist using /usr/local/go1.9.7. Building Go toolchain1 using /usr/local/go1.9.7. Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1. clang: warning: argument unused during compilation: '-nopie' Building Go toolchain2 using go_bootstrap and Go toolchain1. Building Go toolchain3 using go_bootstrap and Go toolchain2. Building packages and commands for darwin/amd64. ##### Testing packages. ok archive/tar 0.107s ok archive/zip 2.884s <...boring...details...omitted...> ##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick ok runtime 13.517s ##### Testing without libgcc. dyld: Symbol not found: _unlinkat Referenced from: /var/folders/7c/vqbsk8sn48g90kh7fnmvpd080000gn/T/go-build075507559/b001/x509.test Expected in: flat namespace FAIL crypto/x509 1.361s 2019/05/16 15:18:35 Failed: exit status 1 dyld: Symbol not found: _unlinkat Referenced from: /var/folders/7c/vqbsk8sn48g90kh7fnmvpd080000gn/T/go-build124050274/b001/net.test Expected in: flat namespace FAIL net 5.617s 2019/05/16 15:18:35 Failed: exit status 1 ok os/user 0.012s ##### sync -cpu=10 ok sync 0.236s ##### Testing race detector skipped due to earlier error skipped due to earlier error skipped due to earlier error skipped due to earlier error skipped due to earlier error ##### ../misc/cgo/stdio skipped due to earlier error ##### ../misc/cgo/life skipped due to earlier error ##### ../misc/cgo/test skipped due to earlier error skipped due to earlier error skipped due to earlier error skipped due to earlier error 2019/05/16 15:18:42 FAILED $
Earlier tests have passed, but then more tests were run and some of them failed. This turned out to be end of the line for MacPorts solution as new failing tests require internal link mode.
Parte 4
At this point, the missing part was to figure out how to include libMacportsLegacySupport
for internal link mode. After looking under sufficient number of stones and flipping through
the content of some obscure tickets, I found a reference to the strange magic that I was looking
for - in the final lines of
runtime/sys_darwin.go:
// Magic incantation to get libSystem actually dynamically linked. // TODO: Why does the code require this? See cmd/link/internal/ld/go.go //go:cgo_import_dynamic _ _ "/usr/lib/libSystem.B.dylib"
I added similar incantation as
runtime/sys_mpls_darwin_amd64.go.
The final version of patched code is on the branch
go1.12.5-osx-legacy of the fork. Build
incantation now simplifies to an original one:
$ ( cd go/src/github.com/av86743/go/ && git checkout go1.12.5-osx-legacy ) M src/crypto/x509/root_cgo_darwin.go Branch go1.12.5-osx-legacy set up to track remote branch go1.12.5-osx-legacy from origin. Switched to a new branch 'go1.12.5-osx-legacy' $ ( cd go/src/github.com/av86743/go/ && cd src && env GOROOT_BOOTSTRAP=/usr/local/go1.9.7 ./all.bash ) Building Go cmd/dist using /usr/local/go1.9.7. Building Go toolchain1 using /usr/local/go1.9.7. Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1. Building Go toolchain2 using go_bootstrap and Go toolchain1. Building Go toolchain3 using go_bootstrap and Go toolchain2. Building packages and commands for darwin/amd64. ##### Testing packages. ok archive/tar 0.085s ok archive/zip 3.114s <...boring...details...omitted...> ##### ../test ##### API check Go version is "go1.12.5", ignoring -next /Users/ubuntu/go/src/github.com/av86743/go/api/next.txt ALL TESTS PASSED --- Installed Go for darwin/amd64 in /Users/ubuntu/go/src/github.com/av86743/go Installed commands in /Users/ubuntu/go/src/github.com/av86743/go/bin *** You need to add /Users/ubuntu/go/src/github.com/av86743/go/bin to your PATH. $
Now tests passed and all.bash completed as indicated in the
original instructions.
Finale
The final version of patch is specialized for darwin/amd64. It may work for
darwin/386 as well, however I do not have access to this platform in order to
verify this.
You may think that with libMacportsLegacySupport.a size being under 20kb, it
would make sense to always link this library statically. If you decide to go down this
route, you will hit two walls.
First, libMacportsLegacySupport.a is in
BSD ar format, which golang does not support.
This is fixable at an expense of a couple of dozen lines perhaps, and functionality
bloat that this will bring in.
Second and much more uglier obstacle is that all external symbols that are resolved through
libMacportsLegacySupport, are hardwired to be resolved through .dylib.
They simply do not belong to the list of external symbols which are resolved by internal
link. If you add libMacportsLegacySupport.a to the list of built-in packages,
its entry symbols will never actually be looked at.
In order to change this, a bunch of declarations for external functions has to be unwired from
being resolved strictly through .dylib. Such change would be really intrusive
and likely real pain to rebase for future releases of golang. I did not bother myself with even
evaluating it.
If you have read this far, you probably have now a nicely working installation of go1.12 for your legacy OS X. Perhaps it was not a totally worthless reading, after all?
Disclaimer: No pigsels were f... er-r, hm-m, harmed in the course of this undertaking.