The ODL release process¶
This document is intended to give precise instructions on the process of making a release. Its purpose is to avoid broken packages, broken documentation and many other things that can go wrong as a result of mistakes during the release process. Since this is not everyday work and may be done under the stress of a (self-imposed) deadline, it is clearly beneficial to have a checklist to hold on to.
Note
The instructions in this document are written from the perspective of Linux and may need adaption for other platforms.
1. Agree on a release schedule¶
This involves the “what” and “when” of the release process and fixes a feature set that is supposed to be included in the new version. The steps are:
- Open an issue on the issue tracker using the title Release X.Y.Z (insert numbers, of course).
- Discuss and agree on a set of open PRs that should be merged and issues that should be resolved before making a release.
2. Make sure tests succeed and docs are built properly¶
When all required PRs are merged, ensure that the latest master
branch is sane. Travis CI checks every PR, but certain things like CUDA cannot be tested there and must therefore undergo tests on a local machine, for at least Python 2.7 and one version of Python 3.
Make a new test conda environment and install all dependencies:
conda create -n release36 python=3.6 nomkl numpy scipy future matplotlib pytest source activate release36 pip install pyfftw pywavelets cd /path/to/odl_repo git checkout master git pull pip install -e .
Run the tests with
pytest
, including doctests, examples documentation and large-scale tests:pytest --doctest-modules --examples --doctest-doc --largescale
Do the same for Python 2.7.
Make sure the tests also run on the platforms you’re currently not testing on. Ask a buddy maintainer if necessary.
Build the documentation. This requires
sphinx
and thesphinxext
submodule:conda install sphinx sphinx_rtd_theme git submodule update --init --recursive cd doc && make clean cd source && python generate_doc.py cd .. make html 2>&1 |\ grep -E "SEVERE|ERROR|WARNING" |\ grep -E -v "more than one target found for|__eq__|document isn't included in any toctree"
The last command builds the documentation and filters from the output all irrelevant warnings, letting through only the “proper” warnings and errors. If possible, fix these remaining issues.
Glance the built documentation (usually in
doc/_build
) for obvious errors.
3. Make a release branch off master
¶
When all tests succeed and the docs are fine, start a release branch. Do not touch any actual code on this branch other than indicated below!
Create a branch off current
master
with the namerelease-X.Y.Z
, inserting the correct version number, of course.git checkout master git pull git checkout -b release-X.Y.Z git push -u my_fork release-X.Y.Z
Like any regular branch that should result in a PR, the release branch is pushed to a fork.
4. Bump the master
branch to the next development version¶
To ensure a higher version number for installations from the git master branch, the version number must be increased before merging the release branch.
On the
master
branch, change the version string inodl/__init__.py
to the next revision larger than the upcoming release version, plus'dev0'
. For example, if the release version string is'0.5.3'
, use'0.5.4.dev0'
.To make sure you don’t miss any other location (or the information here is outdated), perform a search:
cd doc && make clean && cd .. # remove the local HTML doc first grep -Ir "0\.5\.4" . | grep -E -v "\.git|release_notes\.rst|odl\.egg-info"
In the file
conda/meta.yaml
, change the version string afterversion:
to the same as above, but without thedev0
tag. In the example above, this would mean to change it from"0.5.3"
to"0.5.4"
.If necessary, change
git_rev
value tomaster
, although that should already be the case.Commit the changes, using a message like
REL: bump version to X.Y.Z.dev0
.Make a PR with just this change and merge it after review. It must be merged before the release branch.
5. Compile and publish the release¶
Back on the release branch with a git checkout release-X.Y.Z
, it is now time to prepare the release documents, increment the version number and make a release on GitHub.
Note
The release notes should actually be a running document where everybody who files a PR also makes an entry into the release notes file. If not, tough on you – it is your duty now to make up for all that missed work. Maybe you’ll remind your co-workers to do this in their next PR.
Compile the release notes. They should contain all user-visible changes, including performance improvements and other niceties – internal stuff like test modifications don’t belong here. The changes should be summarized in one or two sentences on top, perhaps mentioning the most notable ones in this release. Check the Release Notes file for details on sections, formatting etc.
Increment the version number in
odl/__init__.py
andconda/meta.yaml
. As in Section 4, perform a search to make sure you didn’t miss a version info location.Change the
git_rev
field inconda/meta.yaml
to'vX.Y.Z'
, using the upcoming version number. This is the git tag you will create when making the release on GitHub.Commit the changes, using a message like
REL: bump version to X.Y.Z
.Make a PR and fix review comments. When doing so, keep the
REL: bump version to X.Y.Z
commit last, for example by usinggit commit --amend
for fixes, or by squashing the commits on the release branch.Note
Getting a review is useful, but not mandatory since the only changes on the branch are the release notes and the version bump. It is important that the version information is correct, but errors in the release notes are less grave since they are not distributed with the package and can thus be fixed afterwards.
Push the release branch to the main repository so that it is possible to make a GitHub release from it:
git push origin release-X.Y.Z
Go to the Releases page on GitHub. Click on Draft a new release and select the
release-X.Y.Z
branch from the dropdown menu, not master. UsevX.Y.Z
as release tag (numbers inserted, of course).Paste the short summary from the release notes file (converting from RST to Markdown) but don’t insert the details.
Add a link to the release notes documentation page, as in earlier releases. Later on, when the documentation with the new release notes is online, you can edit this link to point to the exact section.
Note
If you encounter an issue (like a failing test) that needs immediate fix, stop at that point, fix the issue on a branch, make a PR and merge it after review. After that, rebase the release branch(es) on the new master and continue.
6. Create packages for PyPI and Conda¶
The packages should be built on the release branch to make sure that the version information is correct.
Making the packages for PyPI is straightforward. However, make sure you delete old
build
directories since they can pollute new builds:rm build/ -rf python setup.py sdist python setup.py bdist_wheel
The packages are by default stored in a
dist
folder.To build the conda packages, you should not work in a specific environment but rather exit to the root environment. There, install the
conda-build
tool for building packages:source deactivate conda install conda-build
Invoke the following command to build a package for your platform and all supported Python versions:
conda build conda/ --python 2.7 conda build conda/ --python 3.4 conda build conda/ --python 3.5 conda build conda/ --python 3.6 ...
Assuming this succeeds, enter the directory one above where the conda package was stored (as printed in the output). For example, if the package was stored as
$HOME/miniconda3/conda-bld/linux-64/odl-X.Y.Z-py36_0.bz2
, issue the commandcd $HOME/miniconda3/conda-bld/
In this directory, for each Python version “translate” the package to all platforms since ODL is actually platform-independent:
conda convert --platform all <package>
Replace
<package>
by the package file as built by the previousconda build
command.
7. Test installing the local packages and check them¶
Before actually uploading packages to “official” servers, first install the local packages and run the unit tests.
Install directly from the source package (
*.tar.gz
) or the wheel (*.whl
) into a new conda environment:source deactivate conda create -n pypi_install pytest python=X.Y # choose Python version source activate pypi_install cd /path/to/odl_repo cd dist pip install <pkg_filename> python -c "import odl; odl.test()"
Install and test the local conda packages in a new conda environment:
source deactivate conda create -n conda_install python=X.Y # choose Python version source activate conda_install conda install --use-local nomkl odl python -c "import odl; odl.test()"
8. Upload the packages to the official locations¶
Installing the packages works, now it’s time to put them out into the wild.
Install the
twine
package for uploading packages to PyPI in your working environment:source deactivate source activate release36 pip install twine
Upload the source package and the wheel to the PyPI server using
twine
:cd /path/to/odl_repo twine upload -u odlgorup dist/<pkg_filename>
This requires the access credentials for the
odlgroup
user on PyPI – the maintainers have them.Upload the conda packages to the
odlgroup
channel in the Anaconda cloud. The upload requires theanaconda-client
package:conda install anaconda-client cd $HOME/miniconda3/conda-bld anaconda upload -u odlgroup `find . -name "odl-X.Y.Z*"`
For this step, you need the access credentials for the
odlgroup
user on the Anaconda server. Talk to the maintainers to get them.
9. Merge the release branch¶
Now the release branch can finally be merged.
The sole purpose of this step is to update the release notes on master
and potentially get the last minute changes.
- The release branch will have conflicts with
master
since both have modified the version information. Resolve them in favor of the changes made onmaster
. In particular, make sure that the changes in Section 4 stay intact. - Merge the PR for the release.
Done!¶
Time to clean up, i.e., remove temporary conda environments, run conda build purge
, remove files in dist
and build
generated for the PyPI packages, etc.