Build From Source¶
We recommend building EnvPool on Ubuntu 22.04 environment.
We use bazel to build EnvPool. Comparing with pip, using Bazel to build python package with C++ .so files has some advantages:
Bazel allows us to build from source code, but no need to directly includes the code in our repo;
no need to write complex CMake files, especially with multiple third-party dependencies;
using Bazel for CI test pipeline can only run the test with modified part, so that it can save a lot of time.
Install Bazelisk¶
Bazelisk is a version controller for Bazel. You can install it via npm
sudo apt install -y npm
npm install -g @bazel/bazelisk
or golang with version >= 1.16:
sudo apt install -y golang
export PATH=$HOME/go/bin:$PATH
go install github.com/bazelbuild/bazelisk@latest
ln -sf $HOME/go/bin/bazelisk $HOME/go/bin/bazel
Note
For users in mainland China, please do the following step to install go and bazel:
Install golang >= 1.16 from other sites, e.g., https://studygolang.com/dl
Change go proxy:
go env -w GOPROXY=https://goproxy.cn
Install bazel from https://mirrors.huaweicloud.com/bazel/
wget https://studygolang.com/dl/golang/go1.18.1.linux-amd64.tar.gz
# then follow the instructions on golang official website
go env -w GOPROXY=https://goproxy.cn
wget https://mirrors.huaweicloud.com/bazel/6.0.0/bazel-6.0.0-linux-x86_64
chmod +x bazel-6.0.0-linux-x86_64
mkdir -p $HOME/go/bin
mv bazel-6.0.0-linux-x86_64 $HOME/go/bin/bazel
export PATH=$PATH:$HOME/go/bin # or write to .bashrc / .zshrc
# check if successfully installed
bazel
See Issue #87.
Install Other Dependencies¶
EnvPool requires GCC/G++ version >= 9.0 to build the source code. To install:
# optional
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
# install
sudo apt install -y gcc-9 g++-9 build-essential
# to change the default cc to gcc-9:
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-9
It also requires Python version >= 3.7:
sudo apt install -y python3-dev python3-pip
sudo ln -sf /usr/bin/python3 /usr/bin/python
Install CUDA to enable XLA: see https://developer.nvidia.com/cuda-downloads
Install other dependencies: see Dockerfile.
Build Wheel¶
To build a release version, type:
cp third_party/pip_requirements/requirements-release.txt third_party/pip_requirements/requirements.txt
bazel run --config=release //:setup -- bdist_wheel
This creates a wheel under bazel-bin/setup.runfiles/envpool/dist
.
Note
For users in mainland China:
If you find
pip install
is quite slow to fetch 3rd-party libraries, the solution is to uncommentextra_args
inenvpool/pip.bzl
to switch the pip source.If you find
bazel build
is quite slow to fetch 3rd-party libraries, please refer https://docs.bazel.build/versions/main/external.html#using-proxiesexport HTTP_PROXY=http://... export HTTPS_PROXY=http://... # then run the command to build
See Issue #87.
Use Shortcut¶
We provide several shortcuts to make things easier.
# This will install bazelisk via golang, need sudo
make bazel-install
# This will verbose all compile commands to help debug
make bazel-debug
# This will build python wheel (.whl) file under `dist/` folder
make bazel-build
# This will automatically run the tests
make bazel-test
# This will build a wheel for release
make bazel-release
Use Docker to Create Develop Environment¶
We also provide dockerfile for building such a container. To create a docker develop environment, run
make docker-dev
The code is under /app
, and you can communicate with the host machine file
system via /host
.