Skip to content
This repository has been archived by the owner on Feb 7, 2022. It is now read-only.

Update ndk and gomobile #182

Open
wants to merge 31 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
b304b4f
Migrate to ubuntu 17.10 and gcc 6 and friends
tadovas Aug 27, 2018
6f1a398
Update iOS SDK to 10.3
tadovas Aug 29, 2018
902f17b
iOS cross compilation kit successful setup
tadovas Aug 29, 2018
c739ff2
Added extra environment variables for new golang
soffokl Sep 27, 2018
1b6c1b5
Add sample package for testing gomobile, setup ios/android tooling fo…
tadovas Oct 23, 2018
430127e
xcrun emulator moved to hacks, gomobile script added for simplified r…
tadovas Oct 23, 2018
33447f2
Gomobile injected into xgo container and is used for mobile framework…
tadovas Oct 25, 2018
8c8152d
Style fixes
tadovas Oct 25, 2018
278bee0
Fix formatting, comments, etc.
tadovas Oct 25, 2018
9c62298
gomobile is ready - no fork needed
tadovas Oct 26, 2018
4f564c8
Fix failing dockerhub builds. Remove gomobile fork
tadovas Oct 26, 2018
4596420
Add new line
Oct 26, 2018
180837c
Merge pull request #6 from mysteriumnetwork/hotfix/fix-dockerhub-builds
tadovas Oct 26, 2018
94aa95f
Cross compile clang with OSX min version 10.10 by default
tadovas Oct 26, 2018
789350d
Merge pull request #7 from mysteriumnetwork/improvement/go1.11-requir…
tadovas Oct 29, 2018
2bc182c
Add build support for android simulator
tadovas Oct 30, 2018
949451f
Merge pull request #8 from mysteriumnetwork/feature/amd64-sim-android…
tadovas Oct 30, 2018
2a9b190
Add x86 emulator support for android libs
tadovas Oct 31, 2018
791f48d
Merge pull request #9 from mysteriumnetwork/enable-386-android-emulator
tadovas Oct 31, 2018
e38f958
Android arm-a7v architecture support added
tadovas Nov 19, 2018
80ad8d6
Merge pull request #10 from mysteriumnetwork/feature/android-arm7-sup…
tadovas Nov 19, 2018
f8cce5a
Add passing ldflags arg to gomobile, fix arg passing issue
tadovas Dec 14, 2018
3587134
Merge pull request #11 from mysteriumnetwork/bugfix/fix-arg-passing-t…
tadovas Dec 14, 2018
f6d8225
Merge pull request #12 from karalabe/master
soffokl Jun 12, 2019
2294b27
Add go 1.13.1 support
anjmao Oct 28, 2019
1e937e6
Add go 1.13.6
anjmao Jan 22, 2020
5020308
Add Go runtime patches for time (#14)
anjmao Feb 27, 2020
db36438
Update ndk and gomobile
zolia Mar 4, 2020
992d3bf
Add support for go modules
anjmao Mar 4, 2020
9619dfd
Add test example with failing go-openvpn build
anjmao Mar 4, 2020
a7834c3
Update test example
anjmao May 12, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
.idea/
artifacts/
bin/
pkg/
build/
319 changes: 47 additions & 272 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,272 +1,47 @@
# xgo - Go CGO cross compiler

Although Go strives to be a cross platform language, cross compilation from one
platform to another is not as simple as it could be, as you need the Go sources
bootstrapped to each platform and architecture.

The first step towards cross compiling was Dave Cheney's [golang-crosscompile](https://github.com/davecheney/golang-crosscompile)
package, which automatically bootstrapped the necessary sources based on your
existing Go installation. Although this was enough for a lot of cases, certain
drawbacks became apparent where the official libraries used CGO internally: any
dependency to third party platform code is unavailable, hence those parts don't
cross compile nicely (native DNS resolution, system certificate access, etc).

A step forward in enabling cross compilation was Alan Shreve's [gonative](https://github.com/inconshreveable/gonative)
package, which instead of bootstrapping the different platforms based on the
existing Go installation, downloaded the official pre-compiled binaries from the
golang website and injected those into the local toolchain. Since the pre-built
binaries already contained the necessary platform specific code, the few missing
dependencies were resolved, and true cross compilation could commence... of pure
Go code.

However, there was still one feature missing: cross compiling Go code that used
CGO itself, which isn't trivial since you need access to OS specific headers and
libraries. This becomes very annoying when you need access only to some trivial
OS specific functionality (e.g. query the CPU load), but need to configure and
maintain separate build environments to do it.

## Enter xgo

My solution to the challenge of cross compiling Go code with embedded C/C++ snippets
(i.e. CGO_ENABLED=1) is based on the concept of [lightweight Linux containers](http://en.wikipedia.org/wiki/LXC).
All the necessary Go tool-chains, C cross compilers and platform headers/libraries
have been assembled into a single Docker container, which can then be called as if
a single command to compile a Go package to various platforms and architectures.

## Installation

Although you could build the container manually, it is available as an automatic
trusted build from Docker's container registry (not insignificant in size):

docker pull karalabe/xgo-latest

To prevent having to remember a potentially complex Docker command every time,
a lightweight Go wrapper was written on top of it.

go get github.com/karalabe/xgo

## Usage

Simply specify the import path you want to build, and xgo will do the rest:

$ xgo github.com/project-iris/iris
...

$ ls -al
-rwxr-xr-x 1 root root 9995000 Nov 24 16:44 iris-android-16-arm
-rwxr-xr-x 1 root root 6776500 Nov 24 16:44 iris-darwin-10.6-386
-rwxr-xr-x 1 root root 8755532 Nov 24 16:44 iris-darwin-10.6-amd64
-rwxr-xr-x 1 root root 7114176 Nov 24 16:45 iris-ios-5.0-arm
-rwxr-xr-x 1 root root 10135248 Nov 24 16:44 iris-linux-386
-rwxr-xr-x 1 root root 12598472 Nov 24 16:44 iris-linux-amd64
-rwxr-xr-x 1 root root 10040464 Nov 24 16:44 iris-linux-arm
-rwxr-xr-x 1 root root 7516368 Nov 24 16:44 iris-windows-4.0-386.exe
-rwxr-xr-x 1 root root 9549416 Nov 24 16:44 iris-windows-4.0-amd64.exe


If the path is not a canonical import path, but rather a local path (starts with
a dot `.` or a dash `/`), xgo will use the local GOPATH contents for the cross
compilation.


### Build flags

A handful of flags can be passed to `go build`. The currently supported ones are

- `-v`: prints the names of packages as they are compiled
- `-x`: prints the build commands as compilation progresses
- `-race`: enables data race detection (supported only on amd64, rest built without)
- `-tags='tag list'`: list of build tags to consider satisfied during the build
- `-ldflags='flag list'`: arguments to pass on each go tool link invocation
- `-buildmode=mode`: binary type to produce by the compiler


### Go releases

As newer versions of the language runtime, libraries and tools get released,
these will get incorporated into xgo too as extensions layers to the base cross
compilation image (only Go 1.3 and above will be supported).

You can select which Go release to work with through the `-go` command line flag
to xgo and if the specific release was already integrated, it will automatically
be retrieved and installed.

$ xgo -go 1.6.1 github.com/project-iris/iris

Additionally, a few wildcard release strings are also supported:

- `latest` will use the latest Go release (this is the default)
- `1.6.x` will use the latest point release of a specific Go version
- `1.6-develop` will use the develop branch of a specific Go version
- `develop` will use the develop branch of the entire Go repository

### Output prefixing

xgo by default uses the name of the package being cross compiled as the output
file prefix. This can be overridden with the `-out` flag.

$ xgo -out iris-v0.3.2 github.com/project-iris/iris
...

$ ls -al
-rwxr-xr-x 1 root root 9995000 Nov 24 16:44 iris-v0.3.2-android-16-arm
-rwxr-xr-x 1 root root 6776500 Nov 24 16:44 iris-v0.3.2-darwin-10.6-386
-rwxr-xr-x 1 root root 8755532 Nov 24 16:44 iris-v0.3.2-darwin-10.6-amd64
-rwxr-xr-x 1 root root 7114176 Nov 24 16:45 iris-v0.3.2-ios-5.0-arm
-rwxr-xr-x 1 root root 10135248 Nov 24 16:44 iris-v0.3.2-linux-386
-rwxr-xr-x 1 root root 12598472 Nov 24 16:44 iris-v0.3.2-linux-amd64
-rwxr-xr-x 1 root root 10040464 Nov 24 16:44 iris-v0.3.2-linux-arm
-rwxr-xr-x 1 root root 7516368 Nov 24 16:44 iris-v0.3.2-windows-4.0-386.exe
-rwxr-xr-x 1 root root 9549416 Nov 24 16:44 iris-v0.3.2-windows-4.0-amd64.exe


### Branch selection

Similarly to `go get`, xgo also uses the `master` branch of a repository during
source code retrieval. To switch to a different branch before compilation pass
the desired branch name through the `--branch` argument.

$ xgo --branch release-branch.go1.4 golang.org/x/tools/cmd/goimports
...

$ ls -al
-rwxr-xr-x 1 root root 4171248 Nov 24 16:40 goimports-android-16-arm
-rwxr-xr-x 1 root root 4139868 Nov 24 16:40 goimports-darwin-10.6-386
-rwxr-xr-x 1 root root 5186720 Nov 24 16:40 goimports-darwin-10.6-amd64
-rwxr-xr-x 1 root root 3202364 Nov 24 16:40 goimports-ios-5.0-arm
-rwxr-xr-x 1 root root 4189456 Nov 24 16:40 goimports-linux-386
-rwxr-xr-x 1 root root 5264136 Nov 24 16:40 goimports-linux-amd64
-rwxr-xr-x 1 root root 4209416 Nov 24 16:40 goimports-linux-arm
-rwxr-xr-x 1 root root 4348416 Nov 24 16:40 goimports-windows-4.0-386.exe
-rwxr-xr-x 1 root root 5415424 Nov 24 16:40 goimports-windows-4.0-amd64.exe


### Remote selection

Yet again similarly to `go get`, xgo uses the repository remote corresponding to
the import path being built. To switch to a different remote while preserving the
original import path, use the `--remote` argument.

$ xgo --remote github.com/golang/tools golang.org/x/tools/cmd/goimports
...

### Package selection

If you used the above *branch* or *remote* selection machanisms, it may happen
that the path you are trying to build is only present in the specific branch and
not the default repository, causing Go to fail at locating it. To circumvent this,
you may specify only the repository root for xgo, and use an additional `--pkg`
parameter to select the exact package within, honoring any prior *branch* and
*remote* selections.

$ xgo --pkg cmd/goimports golang.org/x/tools
...

$ ls -al
-rwxr-xr-x 1 root root 4194956 Nov 24 16:38 goimports-android-16-arm
-rwxr-xr-x 1 root root 4164448 Nov 24 16:38 goimports-darwin-10.6-386
-rwxr-xr-x 1 root root 5223584 Nov 24 16:38 goimports-darwin-10.6-amd64
-rwxr-xr-x 1 root root 3222848 Nov 24 16:39 goimports-ios-5.0-arm
-rwxr-xr-x 1 root root 4217184 Nov 24 16:38 goimports-linux-386
-rwxr-xr-x 1 root root 5295768 Nov 24 16:38 goimports-linux-amd64
-rwxr-xr-x 1 root root 4233120 Nov 24 16:38 goimports-linux-arm
-rwxr-xr-x 1 root root 4373504 Nov 24 16:38 goimports-windows-4.0-386.exe
-rwxr-xr-x 1 root root 5450240 Nov 24 16:38 goimports-windows-4.0-amd64.exe

This argument may at some point be integrated into the import path itself, but for
now it exists as an independent build parameter. Also, there is not possibility
for now to build mulitple commands in one go.

### Limit build targets

By default `xgo` will try and build the specified package to all platforms and
architectures supported by the underlying Go runtime. If you wish to restrict
the build to only a few target systems, use the comma separated `--targets` CLI
argument:

* `--targets=linux/arm`: builds only the ARMv5 Linux binaries (`arm-6`/`arm-7` allowed)
* `--targets=windows/*,darwin/*`: builds all Windows and OSX binaries
* `--targets=*/arm`: builds ARM binaries for all platforms
* `--targets=*/*`: builds all suppoted targets (default)

The supported targets are:

* Platforms: `android`, `darwin`, `ios`, `linux`, `windows`
* Achitectures: `386`, `amd64`, `arm-5`, `arm-6`, `arm-7`, `arm64`, `mips`, `mipsle`, `mips64`, `mips64le`

### Platform versions

By default `xgo` tries to cross compile to the lowest possible versions of every
supported platform, in order to produce binaries that are portable among various
versions of the same operating system. This however can lead to issues if a used
dependency is only supported by more recent systems. As such, `xgo` supports the
selection of specific platform versions by appending them to the OS target string.

* `--targets=ios-8.1/*`: cross compile to iOS 8.1
* `--targets=android-16/*`: cross compile to Android Jelly Bean
* `--targets=darwin-10.9/*`: cross compile to Mac OS X Mavericks
* `--targets=windows-6.0/*`: cross compile to Windows Vista

The supported platforms are:

* All Android APIs up to Android Lollipop 5.0 ([API level ids](https://source.android.com/source/build-numbers.html))
* All Windows APIs up to Windows 8.1 limited by `mingw-w64` ([API level ids](https://en.wikipedia.org/wiki/Windows_NT#Releases))
* OSX APIs in the range of 10.6 - 10.11
* All iOS APIs up to iOS 9.3

### Mobile libraries

Apart from the usual runnable binaries, `xgo` also supports building library
archives for Android (`android/aar`) and iOS (`ios/framework`). Opposed to
`gomobile` however `xgo` does not derive library APIs from the Go code, so
proper CGO C external methods must be defined within the package.

In the case of Android archives, all architectures will be bundled that are
supported by the requested Android platform version. For iOS frameworks `xgo`
will bundle armv7 and arm64 by default, and also the x86_64 simulator builds
if the iPhoneSimulator.sdk was injected by the user:

* Create a new docker image based on xgo: `FROM karalabe/xgo-latest`
* Inject the simulator SDK: `ADD iPhoneSimulator9.3.sdk.tar.xz /iPhoneSimulator9.3.sdk.tar.xz`
* Bootstrap the simulator SDK: `$UPDATE_IOS /iPhoneSimulator9.3.sdk.tar.xz`

### CGO dependencies

The main differentiator of xgo versus other cross compilers is support for basic
embedded C/C++ code and target-platform specific OS SDK availability. The current
xgo release introduces an experimental CGO *dependency* cross compilation, enabling
building Go programs that require external C/C++ libraries.

It is assumed that the dependent C/C++ library is `configure/make` based, was
properly prepared for cross compilation and is available as a tarball download
(`.tar`, `.tar.gz` or `.tar.bz2`). Further plans include extending this to cmake
based projects, if need arises (please open an issue if it's important to you).

Such dependencies can be added via the `--deps` argument. They will be retrieved
prior to starting the cross compilation and the packages cached to save bandwidth
on subsequent calls.

A complex sample for such a scenario is building the Ethereum CLI node, which has
the GNU Multiple Precision Arithmetic Library as it's dependency.

$ xgo --deps=https://gmplib.org/download/gmp/gmp-6.1.0.tar.bz2 \
--targets=windows/* github.com/ethereum/go-ethereum/cmd/geth
...

$ ls -al
-rwxr-xr-x 1 root root 16315679 Nov 24 16:39 geth-windows-4.0-386.exe
-rwxr-xr-x 1 root root 19452036 Nov 24 16:38 geth-windows-4.0-amd64.exe

Some trivial arguments may be passed to the dependencies' configure script via
`--depsargs`.

$ xgo --deps=https://gmplib.org/download/gmp/gmp-6.1.0.tar.bz2 \
--targets=ios/* --depsargs=--disable-assembly \
github.com/ethereum/go-ethereum/cmd/geth
...

$ ls -al
-rwxr-xr-x 1 root root 14804160 Nov 24 16:32 geth-ios-5.0-arm

Note, that since xgo needs to cross compile the dependencies for each platform
and architecture separately, build time can increase significantly.
# xgomobile - Go CGO cross compiler for gomobile

### Usage

```
docker run --rm \
-v "$PWD"/build:/build \
-v "$GOPATH"/.xgo-cache:/deps-cache:ro \
-v "$PWD"/src:/ext-go/1/src:ro \
-e OUT=Mysterium \
-e FLAG_V=false \
-e FLAG_X=false \
-e FLAG_RACE=false \
-e FLAG_BUILDMODE=default \
-e TARGETS=android/. \
-e EXT_GOPATH=/ext-go/1 \
-e GO111MODULE=off \
mysteriumnetwork/xgomobile:1.13.8 mobilepkg
```

Also see and run ./test.sh to build test examples.

### Building image

If you make changes in docker/base you need to rebuild base image.

```
docker build -t mysteriumnetwork/xgomobile:base -f ./docker/base/Dockerfile ./docker/base
docker push mysteriumnetwork/xgomobile:base
```

If you add new go version only when build and push.

Build new image.
```
docker build -t mysteriumnetwork/xgomobile:1.13.8 -f ./docker/go-1.13.8/Dockerfile .
```

Update and run tests.
```
./test.sh
```

Push image
```
docker push mysteriumnetwork/xgomobile:1.13.8
```
Loading