Tutorial - start here

Well, this module does actually nothing.

It’s a valkka_live module that servers as a template for python3 projects:

_images/skull.png

Some features:

  • Autogenerated documentation
  • Python packaging
  • Development and production downloads from git

For the impatient

Set up your environment (you might have these directories already):

cd
mkdir python3
mkdir python3_packages

Include python3 into your PYTHONPATH (include this line into your .bashrc as well)

export PYTHONPATH=$HOME/python3

So, your development python packages are in ~/python3_packages. Your name is Janne Jantunen and you have decided to create a new package named valkka_live. Proceed like this:

cd ~/python3_packages
git clone https://github.com/elsampsa/valkka_live
mv valkka_live valkka_live
cd valkka_live
./reinit.bash
./setauthor.bash "Janne Jantunen"
./setver.bash "0.1"
ln -s $PWD/valkka_live $HOME/python3

The last line can be substituted with

python3 setup.py install -e .

The effect is the same - creating a link that allows python to find your package (in the latter case that link is the $HOME/.local/lib/python3.x/site-packages/valkka_live.egg-link file)

Mod the documentation by editing .rst files in the “docs/” directory. The idea is that you hide/delete this page (“tutorial.rst”) from your own module.

After this you can re-compile and check the documentation with

cd docs
./compile.bash
firefox index.html

If you want to push your newly minted python project into a git repo, do:

git init
git remote add origin https://[your-personal-git-repository]/valkka_live.git
git add *
git commit -m "initial commit"
git push -u origin master

Things to keep in mind:

  • The name of your package (that is also the name of your python module) should not contain “weird characters”, i.e. “.,-” etc.
  • Remember to add git tags with (here we create tag “v1.2”)
git tag v1.2 -m "cool version v1.2"
git push origin --tags
  • ..now production users can install with pip3 certain revisions of the software (see here)

Dependencies

Edit the setup.py and there, the “install_requires” field. This way, when installing your package with pip3, depencencies are automagically resolved. Keep them up to date also in the file docs/snippets/requirements.txt.

Files

This page has been produced with the file “docs/tutorial.rst”. Go ahead and open it in your editor. Open also “index.rst”. Edit them accordingly. Said that, let’s take a closer look at the directory structure:

  • valkka_live/ : This is a “scaffold” directory :

    README.md Readme file in online markdown format (nice for github). Play around with online markdown here
    LICENSE.* License text (say, LICENSE.LGPL3, LICENSE.MIT, etc.)
    CHANGELOG Recent changes in the code
    setup.py Python setup script for creating packages
    MANIFEST.in used by setup.py
    changestr.bash Helper script to change strings in your package
    setver.bash Helper script to change the version of your package
    setauthor.bash Helper script to change the author of the package
    reinit.bash Reinitializes the package name
    cleanpy.bash Removes __pycache__ directories and .pyc files; Do maybe before “git add .”
    upload.bash Uploads your python package into PyPi
    test_upload.bash Uploads your python package into PyPi test repository
    git_tag.bash Pushes version information into git. This script is modified by the setver.bash
    git_rm_tag.bash Removes the version information (if you made an error)

  • valkka_live/docs/ : Documentation and autogenerated documentation lives here

    index.html Redirects to the autogenerated html [don’t touch]
    index.rst The main index start from here
    tutorial.rst The file you are staring at the moment (or html version of it)
    intro.rst Introduction of the module
    requirements.rst What the user needs in order to use this module
    examples.rst Copy-pastable examples for the api user
    submodules.rst Documentation generated automatically from source code. Just add a new entry here for each submodule, don’t touch otherwise
    license.rst Copyright and License
    authors.rst Who is the maintainer & author
    compile.bash Run this always after modifying your documents / source code [don’t edit]
    clean.bash Clean autogenerated documentation [don’t edit]
    conf.py Sphinx configuration file. edit this and try different styles
    .nojekyll Dummy file. This is needed for the on-line documentation to work with GitHub [don’t touch]
    snippets/ Example snippet source files and scripts for generating pages from them. Edit also the file requirements.txt here.
    generated/ Auto-generated documents [don’t touch]

  • valkka_live/valkka_live : This is where the actual python module lives

    __init__.py Python module initialization
    greeters/ Submodule
    data/ Static data the module needs and that is included in the python package

The scaffolding “double-directory” valkka_live/valkka_live structure might seem inconvenient, but it is necessary for packaging. Here is what we just did in the For the impatient section:

  • Keep “scaffolded” python modules in, say “~/python3_packages”
  • Link the python module directories to “~/python3”

It’s time to start documenting! Edit the files “docs/*.rst”. Here are some nice tips for using Sphinx and here are some more

To create and to recreate the docs (after changing the code, etc.), do (in the “docs/” directory):

./compile.bash

To see your documentation, launch

firefox index.html

Online autodocumentation

Github users

After creating the git repo, create also a site for your project like this: Settings => GitHub pages => Source : master branch / docs folder => press Save. Now your documentation is online in github! (don’t forget to include the “docs/” folder into git).

Gitlab users

Using a generic service that uses a gitlab server? No problem. There is a simple hack to put the autodocs online.

  • In gitlab, create a wiki page for your project. Write there the words “Hello world”. Next, clone the wiki repository (not the repository, just the wiki repository) with that button on the right that says “clone repository”.
  • Edit your projects “docs/compile.bash”. Uncomment the options for Gitlab and set the directory of your wiki repository correctly. Run “./compile.bash”. Now the whole documentation tree has been copied to your wiki repo directory.
  • In the wiki repo directory, open “home.md” in an editor. Modify it to look like this:
For documentation
[click here](_build/html/index.html)
  • You still need to use git to add and push all the files to the wiki repository online (there are instructions in “compile.bash”)

Organizing large python projects

Let’s say we have a python project that consists of “entities” and those consist of submodules. Imagine a web-service of some sort with frontend, backend and some machine_learning behind them:

macro_project
rocket module
submodule1
submodule2
bicycle module
submodule1
submodule2
turbo module
submodule1
submodule2

Here are some possibilities:

  1. Clusterfuck
  • Keep everything in the same repository. Avoid defining API interfaces. Everybody can access all code.
  • We have all done this and it’s not a good idea..
  1. Each module in a separate repository
  • Each module should have a maintainer and a well-defined API interface
  • Development
    • Keep the cloned local git repositories at ~/python3_packages (as explained above)
  • Production
    • You can use pip3 to install and update the packages directly from the git repository - convenient if the development/production cycle is fast
    • When installed with pip3 there is no fiddling with paths and symlinks, it just works (if you’re interested in details, pip3 installs it under “~/.local/lib”)
  • See how this is managed in the Getting started section

Code Organization

So, let’s get back to this tutorial. This is a module that has a single submodule called “greeters” that is organized as follows:

greeters
__init__.py
base.py
fancy.py

The idea is that “base.py” has some base class definitions that are used bu “fancy.py” to create derived classes. __init__.py has been tweaked so that user can import with

from valkka_live.greeters import UberFancyHelloWorld

instead of the cumbersome

from valkka_live.fancy.greeters import UberFancyHelloWorld

(also, the api user should not be bothered with the internal inheritance, etc.)

So, I suggest you open in your editor all files of the “greeters” submodule and study them a bit.

Quick testing

Each module file under “greeters” works also as a stand-alone test. Go to the “greeters” directory and try (either ipython or ipython3):

ipython3
%run fancy.py

Or just (either python or python3):

python3 fancy.py

This is not proper testing, though. It’s quick’n’dirty testing during development. Separate testing programs for systematic testing are a good idea to include in your module package.

Packaging

If you have done everything as instructed here, creating a distributable python package can be done as follows:

cd ~/python3_packages/valkka_live
python3 setup.py sdist

Your distributable python package is now in directory “dist/”. You can install it with:

pip3 install --upgrade dist/valkka_live-version.tar.gz

The setup.py script automatically finds and includes python packages to the distribution package. In “MANIFEST.in” we also tell it to include the complete “docs/” directory and an auxiliary file from the “greeters” submodule. See “setup.py” for more instructions.

Polish your setup script with the following cycle:

rm dist/*
python3 setup.py sdist
pip3 install --user --upgrade --verbose dist/valkka_live-version.tar.gz

The verbose option is nice to see any problems, with say, your post-installing script defined in setup.py. Once you have got rid of all the errors, you can be sure that it works also when people install your python package directly from git using pip3.

Package testing

When testing the pip3 installation, use virtualenv to see that you got the dependencies right. First, create a virtualenv:

virtualenv --no-site-packages -p python3 test

Then let’s use that virtualenv (we also clean up PYTHONPATH) to test your production system:

cd test
source bin/activate
export PYTHONPATH=
pip3 install --user --upgrade git+git://[your-personal-git-repository]/valkka_live

See here how the end-user would be using your python module directly from git.

To exit from virtualenv, use:

deactivate

PyPi

By uploading your python package to PyPi (Python Package Index), end users can install it simply with

::
pip3 install –user packagename

To use the python package repositories, follow these steps:

  1. Create an account in `PyPi repository<https://pypi.org>`_ and in `PyPi test repository<https://testpypi.python.org>`_
  2. Create file $HOME/.pypirc with the following contents:
[distutils]
index-servers =
    pypi
    test

[pypi]
username: xxx
password: xxx

[test]
repository: https://test.pypi.org/legacy/
username: xxx
password: xxx
  1. Install twine
pip3 install --user twine

Now you can use the scripts test_upload.bash and upload.bash to send your package to the python (test) repository (first, edit those scripts).

To install from test repository (instead of the official one), use:

::
pip3 install -r test –user packagename

Example snippets

The following python code can be downloaded from [here]

from valkka_live.greeters import FancyHelloWorld
# let's create an instance
gr=FancyHelloWorld(person="Sampsa")
# print the result
print(gr)

Check out directory “docs/snippets/”. It serves as a collection for small example programs. Open and edit “form_snippets.bash” there. You can then run it. Remember to recompile the documentation once you’ve run that script.

Miscellaneous

  • Check out in the source code, how I init and check a large number of parameters in the constructor
  • To keep your code clean and PEP8 compliant, there’s a nice tool called “autopep8”, install with:
pip3 install --user autopep8

Now you can fix a python file like this:

::
autopep8 –aggressive –in-place yourfile.py