Significant white spacehttps://albertodonato.net/blog/2019-04-02T00:00:00+00:00Setting up a modern Python project2019-04-02T00:00:00+00:002019-04-02T00:00:00+00:00Alberto Donatotag:albertodonato.net,2019-04-02:/blog/posts/setting-up-a-modern-python-project<p>Recently I've been working on a new Python-based project, using Python 3.6.</p>
<p>Having the opportunity for a fresh start, I spent some time taking a look at
how to best make use of the the modern tools for project setup, testing, static
checking and so on, and how to …</p><p>Recently I've been working on a new Python-based project, using Python 3.6.</p>
<p>Having the opportunity for a fresh start, I spent some time taking a look at
how to best make use of the the modern tools for project setup, testing, static
checking and so on, and how to integrate them nicely into Continuous
Integration systems (e.g. <a class="reference external" href="https://travis-ci.com/">Travis</a>).</p>
<p>As a result I've also started updating my personal projects using the same
setup, and updated my <a class="reference external" href="https://github.com/albertodonato/python-skeleton">Python skeleton project</a> accordingly, so that it's easy
to apply it to new projects as well.</p>
<p>The main goals I've been pursuing in working on this setup are:</p>
<ul class="simple">
<li>dependencies should be declared in a single place</li>
<li>use standard Python tooling (e.g., avoid the need for a Makefile)</li>
<li>use the same setup in development and CI</li>
</ul>
<div class="section" id="the-basics-setup-py">
<h2>The basics: <tt class="docutils literal">setup.py</tt><a class="headerlink" href="#the-basics-setup-py" title="Permalink to this headline">¶</a></h2>
<p>The <tt class="docutils literal">setup.py</tt> file is the entry point for a Python project, so let's start
from there.</p>
<p>Aside from the usual boilerplate, the most importating thing here is to keep
track of install and test (and possibily development) dependencies
separately. It's quite common in projects to have an <tt class="docutils literal">extra_require</tt> entry
called <tt class="docutils literal">"testing"</tt>, so that test dependencies can be installed via</p>
<pre class="code shell literal-block">
pip<span class="w"> </span>install<span class="w"> </span>.<span class="o">[</span>testing<span class="o">]</span>
</pre>
<p>In the end the <tt class="docutils literal">setup.py</tt> will look like this:</p>
<pre class="code python literal-block">
<span class="kn">from</span> <span class="nn">setuptools</span> <span class="kn">import</span> <span class="p">(</span><span class="w">
</span> <span class="n">find_packages</span><span class="p">,</span><span class="w">
</span> <span class="n">setup</span><span class="p">,</span><span class="w">
</span><span class="p">)</span><span class="w">
</span><span class="n">tests_require</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'pytest'</span><span class="p">,</span> <span class="o">...</span><span class="p">]</span> <span class="c1"># ... other test-specific dependencies</span><span class="w">
</span><span class="n">config</span> <span class="o">=</span> <span class="p">{</span><span class="w">
</span> <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'myproject'</span><span class="p">,</span><span class="w">
</span> <span class="s1">'version'': '</span><span class="mf">0.0.1</span><span class="s1">',</span><span class="w">
</span> <span class="c1"># ... description, author details, and other metadata</span><span class="w">
</span> <span class="s1">'packages'</span><span class="p">:</span> <span class="n">find_packages</span><span class="p">(</span><span class="n">include</span><span class="o">=</span><span class="p">[</span><span class="s1">'myproject'</span><span class="p">,</span> <span class="s1">'myproject.*'</span><span class="p">]),</span><span class="w">
</span> <span class="s1">'install_requires'</span><span class="p">:</span> <span class="p">[</span><span class="o">...</span><span class="p">],</span> <span class="c1"># ... runtime dependencies</span><span class="w">
</span> <span class="s1">'tests_require'</span><span class="p">:</span> <span class="n">tests_require</span><span class="p">,</span><span class="w">
</span> <span class="s1">'extras_require'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'testing'</span><span class="p">:</span> <span class="n">tests_require</span><span class="p">},</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">setup</span><span class="p">(</span><span class="o">**</span><span class="n">config</span><span class="p">)</span>
</pre>
<p>Note that this also uses the standard <tt class="docutils literal">tests_require</tt> keyword, so that
<tt class="docutils literal">python setup.py test</tt> would also install test dependences.</p>
</div>
<div class="section" id="running-tox">
<h2>Running <tt class="docutils literal">tox</tt><a class="headerlink" href="#running-tox" title="Permalink to this headline">¶</a></h2>
<p><a class="reference external" href="https://tox.readthedocs.io/">Tox</a> is a very nice tool for setting up and running commands in separate
virtualenvs, with different set of dependencies.</p>
<p>In my setup, I use <tt class="docutils literal">tox</tt> for running everything, thus eliminating the need
for <tt class="docutils literal">make</tt> and makefiles.</p>
<p><tt class="docutils literal">tox</tt> is configured using a <tt class="docutils literal">tox.ini</tt> file at the root of the project.
It basically contains rules for the following stages:</p>
<ul class="simple">
<li>formatting</li>
<li>linting</li>
<li>static type checking</li>
<li>running unit tests (with coverage report)</li>
<li>(optionally) building documentation</li>
</ul>
<p>Each target tracks its dependencies independently, so that they're only
installed where really needed.</p>
<div class="section" id="formatting-and-linting">
<h3>Formatting and linting<a class="headerlink" href="#formatting-and-linting" title="Permalink to this headline">¶</a></h3>
<p>The formatting and linting stages use <a class="reference external" href="https://pypi.org/project/yapf">Yapf</a> and <a class="reference external" href="https://pypi.org/project/isort">isort</a> (for sorting imports);
finally, linting also runs <a class="reference external" href="https://pypi.org/project/flake8">Flake8</a>.</p>
<p>Both <tt class="docutils literal">yapf</tt> and <tt class="docutils literal">isort</tt> have their own config files which allows tweaking
the desired format.</p>
<p>The formatting stage actually updates files, while the linting one just ensure
source code conforms to the formatting conventions:</p>
<pre class="code ini literal-block">
<span class="k">[globals]</span><span class="w">
</span><span class="na">lint_files</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">setup.py myproject</span><span class="w">
</span><span class="k">[testenv:format]</span><span class="w">
</span><span class="na">deps</span><span class="w"> </span><span class="o">=</span><span class="w">
</span><span class="na">isort</span><span class="w">
</span><span class="na">yapf</span><span class="w">
</span><span class="na">commands</span><span class="w"> </span><span class="o">=</span><span class="w">
</span><span class="na">{envbindir}/yapf --in-place --recursive {[globals]lint_files}</span><span class="w">
</span><span class="na">{envbindir}/isort --recursive {[globals]lint_files}</span><span class="w">
</span><span class="k">[testenv:lint]</span><span class="w">
</span><span class="na">deps</span><span class="w"> </span><span class="o">=</span><span class="w">
</span><span class="na">flake8</span><span class="w">
</span><span class="na">isort</span><span class="w">
</span><span class="na">yapf</span><span class="w">
</span><span class="na">commands</span><span class="w"> </span><span class="o">=</span><span class="w">
</span><span class="na">{envbindir}/yapf --diff --recursive {[globals]lint_files}</span><span class="w">
</span><span class="na">{envbindir}/isort --check-only --diff --recursive {[globals]lint_files}</span><span class="w">
</span><span class="na">{envbindir}/flake8 {[globals]lint_files}</span>
</pre>
</div>
<div class="section" id="static-type-checking">
<h3>Static type checking<a class="headerlink" href="#static-type-checking" title="Permalink to this headline">¶</a></h3>
<p>Starting from version 3.6, Python supports variables annotations (<a class="reference external" href="https://www.python.org/dev/peps/pep-0526/">PEP-526</a>) in
addition to type annotations in function declarations.</p>
<p>This allows static type checking of code using these declarations, which can be
done with <a class="reference external" href="http://mypy-lang.org/">mypy</a>.</p>
<p>This can be easily run from <tt class="docutils literal">tox</tt> with the following stage:</p>
<pre class="code ini literal-block">
<span class="k">[testenv:check]</span><span class="w">
</span><span class="na">deps</span><span class="w"> </span><span class="o">=</span><span class="w">
</span><span class="na">mypy</span><span class="w">
</span><span class="na">commands</span><span class="w"> </span><span class="o">=</span><span class="w">
</span><span class="na">{envbindir}/mypy -p myproject {posargs}</span>
</pre>
</div>
<div class="section" id="running-tests">
<h3>Running tests<a class="headerlink" href="#running-tests" title="Permalink to this headline">¶</a></h3>
<p>Finally, but actually most importantly, we want to run tests on our code.</p>
<p>I use <a class="reference external" href="https://docs.pytest.org/en/latest/">pytest</a> in most my projects both as framework for writing tests and as
test runner. It is fully compatible with tests based on the <tt class="docutils literal">unittest</tt>
framework, so it can also be used just as a runner for existing test suites.</p>
<p>In addition to running tests, I also want to ensure the code 100%
test-covered. <tt class="docutils literal">pytest</tt> supports generating coverage reports through the
<tt class="docutils literal"><span class="pre">pytest-cov</span></tt> plugin.</p>
<pre class="code ini literal-block">
<span class="k">[testenv:coverage]</span><span class="w">
</span><span class="na">deps</span><span class="w"> </span><span class="o">=</span><span class="w">
</span><span class="na">.</span><span class="w">
</span><span class="na">.[testing]</span><span class="w">
</span><span class="na">pytest-cov</span><span class="w">
</span><span class="na">commands</span><span class="w"> </span><span class="o">=</span><span class="w">
</span><span class="na">{envbindir}/pytest --cov {posargs}</span>
</pre>
<p>Note that we're installing project test dependencies with <tt class="docutils literal">.[testing]</tt> as
mentioned above.</p>
<p>With this setup, we just need to run tox with the desired stages, such as:</p>
<pre class="code shell literal-block">
tox<span class="w"> </span>-e<span class="w"> </span>format,lint,check,coverage
</pre>
<p>Note that test stages pass <tt class="docutils literal">{posargs}</tt> to <tt class="docutils literal">pytest</tt>, which allows, for
instance, limiting runs to a set of files, or making the output verbose:</p>
<pre class="code shell literal-block">
tox<span class="w"> </span>-e<span class="w"> </span>coverage<span class="w"> </span>--<span class="w"> </span>-vs
</pre>
</div>
</div>
<div class="section" id="ci-setup">
<h2>CI setup<a class="headerlink" href="#ci-setup" title="Permalink to this headline">¶</a></h2>
<p>Using a public CI system (such as <a class="reference external" href="https://travis-ci.com/">Travis</a>), we can get test runs at every repository push.</p>
<p>With <tt class="docutils literal">tox</tt> properly set up with different targets, the setup is pretty straightforward.</p>
<p>We can use Travis' "stages" to run each step individually:</p>
<pre class="code yaml literal-block">
<span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">python</span><span class="w">
</span><span class="nt">python</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="s">"3.6"</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="s">"3.7-dev"</span><span class="w">
</span><span class="nt">matrix</span><span class="p">:</span><span class="w">
</span><span class="nt">fast_finish</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">true</span><span class="w">
</span><span class="nt">stages</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="l-Scalar-Plain">lint</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="l-Scalar-Plain">check</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="l-Scalar-Plain">test</span><span class="w">
</span><span class="nt">install</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">pip install tox codecov</span><span class="w">
</span><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span><span class="nt">include</span><span class="p">:</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">lint</span><span class="w">
</span><span class="nt">script</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">tox -e lint</span><span class="w">
</span><span class="nt">python</span><span class="p">:</span><span class="w"> </span><span class="s">"3.6"</span><span class="w">
</span><span class="p-Indicator">-</span><span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">check</span><span class="w">
</span><span class="nt">script</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">tox -e check</span><span class="w">
</span><span class="nt">python</span><span class="p">:</span><span class="w"> </span><span class="s">"3.6"</span><span class="w">
</span><span class="nt">script</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">tox -e coverage</span><span class="w">
</span><span class="nt">after_success</span><span class="p">:</span><span class="w"> </span><span class="l-Scalar-Plain">codecov</span>
</pre>
<p>This way, failures are reported nicely at the proper stage (lint, check or
test).</p>
<p>The configuration shown above also pushes coverage results to <a class="reference external" href="https://codecov.io/">Codecov</a> so that
coverage changes can also be tracked.</p>
<p>... and that's it!</p>
</div>
Full-disk encryption with Btrfs on Ubuntu Xenial2016-04-23T00:00:00+00:002016-04-23T00:00:00+00:00Alberto Donatotag:albertodonato.net,2016-04-23:/blog/posts/full-disk-encryption-with-btrfs-on-ubuntu-xenial<p><a class="reference external" href="https://btrfs.wiki.kernel.org/">Btrfs</a> is a copy-on-write (CoW) filesystem
available in the standard Linux kernel, which provides advanced features like
snapshotting and volume management, similarly to what <a class="reference external" href="https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)">LVM</a> provides.</p>
<p>I've been using <cite>LVM</cite> on my laptop for quite some time, to keep separate
volumes for filesystem root and <tt class="docutils literal">/home</tt>, and containers. <cite>LVM</cite> makes …</p><p><a class="reference external" href="https://btrfs.wiki.kernel.org/">Btrfs</a> is a copy-on-write (CoW) filesystem
available in the standard Linux kernel, which provides advanced features like
snapshotting and volume management, similarly to what <a class="reference external" href="https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)">LVM</a> provides.</p>
<p>I've been using <cite>LVM</cite> on my laptop for quite some time, to keep separate
volumes for filesystem root and <tt class="docutils literal">/home</tt>, and containers. <cite>LVM</cite> makes this
easy and also allows to resize partitions without needing to move data.</p>
<p>Even though way more flexible than physical partitions, <cite>LVM</cite> volumes too have
a preallocated size; this can be a problem with many volumes and not a lot of
disk size (such as on laptops with SSDs). The choice is to either create small
partitions and grown them as needed (leaving empty space in the volume group),
or create larger partitions and possibly waste space.</p>
<p>In this regard <cite>Btrfs</cite> is much more flexible, since subvolumes use a global
space pool: a single partition can host multiple subvolumes that can be
created, destroyed, copied and snapshotted without wasting any unused space.
On the other hand, <cite>Btrfs</cite> volumes are not generic space pools, they can't be
used to host other filesystem types, so for instance for swap, a separate
partition is needed.</p>
<p>I decided to reinstall my laptop with the latest beta of Ubuntu 16.04 (which
has now been finalized and released), replacing the root and <tt class="docutils literal">/home</tt> <cite>LVM</cite>
volumes with a single <cite>Btrfs</cite> partition and different subvolumes.</p>
<p>When reinstalling, I wanted to keep my full-disk encryption (FDE) setup, where
swap, root and <tt class="docutils literal">/home</tt> partitions were encrypted, which I had on my previous
Wily installation (using the installer encrypted setup choice).</p>
<p>Unfortunately <cite>FDE</cite> with <cite>Btrfs</cite> is not available out-of-the-box in the
installer, since the "Encrypt the new Ubuntu installation for security" option
uses a single <cite>Ext4</cite> partition, but with a few manual steps during
installation, it's possible to have the same result using <cite>Btrfs</cite>.</p>
<div class="section" id="partitioning-the-disk">
<h2>Partitioning the disk<a class="headerlink" href="#partitioning-the-disk" title="Permalink to this headline">¶</a></h2>
<p>The easiest way to do this is to use the disk configurator from the Ubuntu
installer:</p>
<ol class="arabic simple">
<li>boot the install media (DVD or USB stick), start the install and choose the
<em>"Try ubuntu without installing"</em> option at boot</li>
<li>at the "Installation type" page, choose <em>"Something else"</em></li>
<li>create 3 partitions of the following types:<ol class="arabic">
<li>EFI system partition (for <tt class="docutils literal">/boot/efi</tt>). It can be quite small, since
only a few MB will be used.</li>
<li>Ext2 filesystem (for <tt class="docutils literal">/boot</tt>). This also doesn't need to be very big, a
few hundred MB are enough.</li>
<li>physical volume for encryption (for the <cite>dm-crypt</cite> volume), using the
rest of the space. When selecting this option the installer will ask for
an encryption password, which will be needed at each boot for unlocking
the volume.</li>
</ol>
</li>
</ol>
<p>Now that we have physical partitions set up, we can quit the installer; we'll
restart after setting up volumes inside the last partition.</p>
<p>The next step is to create an LVM volume group in the encrypted partition, with
volumes for swap and root filesystem. Since the partition is <tt class="docutils literal">/dev/sda3</tt>
(we'll assume the target disk is <tt class="docutils literal">/dev/sda</tt>), the <cite>dm-crypt</cite> volume created
by the installer will be named <tt class="docutils literal">/dev/mapper/sda3_crypt</tt>.</p>
<p>Creating the LVM is just a few commands:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>vgcreate<span class="w"> </span>ubuntu<span class="w"> </span>/dev/mapper/sda3_crypt<span class="w">
</span><span class="gp">$ </span>sudo<span class="w"> </span>lvcreate<span class="w"> </span>--name<span class="w"> </span>swap<span class="w"> </span>-L<span class="w"> </span>16Gb<span class="w"> </span>ubuntu<span class="w"> </span><span class="c1"># adjust size as needed
</span><span class="gp">$ </span>sudo<span class="w"> </span>lvcreate<span class="w"> </span>--name<span class="w"> </span>root<span class="w"> </span>-l<span class="w"> </span><span class="m">100</span>%FREE<span class="w"> </span>ubuntu
</pre>
</div>
<div class="section" id="installing-ubuntu">
<h2>Installing Ubuntu<a class="headerlink" href="#installing-ubuntu" title="Permalink to this headline">¶</a></h2>
<p>At this point, we can launch the installer, selecting once again <em>"Something
else"</em> as installation type.</p>
<p>The disk partitioner will show the partitions we just created, we just need to
enable them with the proper mountpoint:</p>
<ul class="simple">
<li><tt class="docutils literal">/dev/sda1</tt>: "EFI system partition", mounted on <tt class="docutils literal">/boot/efi</tt></li>
<li><tt class="docutils literal">/dev/sda2</tt>: "Ext2 filesystem", mounted on <tt class="docutils literal">/boot</tt></li>
<li><tt class="docutils literal"><span class="pre">/dev/mapper/ubuntu-root</span></tt>: "Btrfs filesystem", mounted on <tt class="docutils literal">/</tt></li>
<li><tt class="docutils literal"><span class="pre">/dev/mapper/ubuntu-swap</span></tt>: "swap area"</li>
</ul>
<p>Make sure that for all partitions (except swap) the "format" checkbox is
selected.</p>
<p><em>IMPORTANT:</em> the physical device (<tt class="docutils literal">/dev/sda</tt>) must be selected as target for
bootloader install.</p>
<p>Now we can proceed with the normal Ubuntu install, answering all configuration
questions as desired.</p>
<p>At the end of the installation process, <em>DO NOT REBOOT YET</em> (select "continue
testing").</p>
</div>
<div class="section" id="post-install-setup">
<h2>Post-install setup<a class="headerlink" href="#post-install-setup" title="Permalink to this headline">¶</a></h2>
<p>A few more manual steps are required after installing the system to make it
aware of the full-disk encryption setup. Since the partitioning has been done
manually, the installer didn't do it for us. All the following commands need to
be run as <cite>root</cite>:</p>
<p>mount the target root filesystem, and all pseudo-filesystems under it (to be
able to enter a <tt class="docutils literal">chroot</tt> later)</p>
<pre class="code console literal-block">
<span class="gp"># </span>mount<span class="w"> </span>/dev/mapper/ubuntu-root<span class="w"> </span>/mnt<span class="w"> </span>-o<span class="w"> </span><span class="nv">subvol</span><span class="o">=</span>@<span class="w">
</span><span class="gp"># </span>mount<span class="w"> </span>-o<span class="w"> </span><span class="nb">bind</span><span class="w"> </span>/dev/<span class="w"> </span>/mnt/dev<span class="w">
</span><span class="gp"># </span>mount<span class="w"> </span>-t<span class="w"> </span>sysfs<span class="w"> </span>sysfs<span class="w"> </span>/mnt/sys<span class="w">
</span><span class="gp"># </span>mount<span class="w"> </span>-t<span class="w"> </span>proc<span class="w"> </span>procfs<span class="w"> </span>/mnt/proc
</pre>
<p>get the <tt class="docutils literal">UUID</tt> of the encrypted partition (<em>not</em> the <tt class="docutils literal">PARTUUID</tt>) and create
<tt class="docutils literal">/etc/crypttab</tt> in the target root</p>
<pre class="code console literal-block">
<span class="gp"># </span>blkid<span class="w"> </span>/dev/sda3<span class="w">
</span><span class="go">/dev/sda3: UUID="<YOUR-UUID>" TYPE="crypto_LUKS" PARTUUID="f25a9621-045f-4d79-b0a0-489c5f7c0562"
</span><span class="gp"># </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"sda3_crypt UUID=<YOUR-UUID> none luks,discard"</span><span class="w"> </span>><span class="w"> </span>/mnt/etc/crypttab
</pre>
<p><tt class="docutils literal">chroot</tt> into the target root directory, to rebuild the kernel initramfs and
grub config</p>
<pre class="code console literal-block">
<span class="gp"># </span>chroot<span class="w"> </span>/mnt<span class="w">
</span><span class="gp"># </span>mount<span class="w"> </span>/boot<span class="w">
</span><span class="gp"># </span>mount<span class="w"> </span>/boot/efi<span class="w">
</span><span class="gp"># </span>service<span class="w"> </span>lvm2-lvmetad<span class="w"> </span>start<span class="w"> </span><span class="c1"># needed for grub to find the LVM volumes
</span><span class="gp"># </span>update-initramfs<span class="w"> </span>-u<span class="w">
</span><span class="gp"># </span>update-grub
</pre>
<p>Now everything should be set up, so we can undo all mounts, including the
target root filesystem.</p>
<pre class="code console literal-block">
<span class="gp"># </span>service<span class="w"> </span>lvm2-lvmetad<span class="w"> </span>stop<span class="w">
</span><span class="gp"># </span>umount<span class="w"> </span>/boot/efi<span class="w">
</span><span class="gp"># </span>umount<span class="w"> </span>/boot<span class="w">
</span><span class="gp"># </span>umount<span class="w"> </span>/sys<span class="w">
</span><span class="gp"># </span>umount<span class="w"> </span>/proc<span class="w">
</span><span class="gp"># </span>umount<span class="w"> </span>/dev<span class="w">
</span><span class="gp"># </span><span class="nb">exit</span><span class="w"> </span><span class="c1"># from the chroot
</span><span class="gp"># </span>umount<span class="w"> </span>/mnt
</pre>
<p>Done! Now we can reboot into the new system.</p>
<p>Before actually booting, a splash screen will ask the password to unlock the
encrypted volume (the one chosen when creating the partition).</p>
</div>
<div class="section" id="recap-of-partition-setup">
<h2>Recap of partition setup<a class="headerlink" href="#recap-of-partition-setup" title="Permalink to this headline">¶</a></h2>
<p>The install uses three partitions, of which two get mounted directly:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>/dev/sda<span class="w">
</span><span class="go">/dev/sda2 on /boot type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)</span>
</pre>
<p>The encrypted <tt class="docutils literal">/dev/sda3</tt> partition will be visible through the <cite>dm-crypt</cite>
volume:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>cryptsetup<span class="w"> </span>status<span class="w"> </span>/dev/mapper/sda3_crypt<span class="w">
</span><span class="go">/dev/mapper/sda3_crypt is active and is in use.
type: LUKS1
cipher: aes-xts-plain64
keysize: 512 bits
device: /dev/sda3
offset: 4096 sectors
size: 311025664 sectors
mode: read/write
flags: discards</span>
</pre>
<p>Since the opened <tt class="docutils literal">/dev/mapper/sda3_crypt</tt> volume contains an LVM setup, the
kernel automatically makes volumes inside it available:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>vgs<span class="w">
</span><span class="go"> VG #PV #LV #SN Attr VSize VFree
ubuntu 1 2 0 wz--n- 148.30g 0
</span><span class="gp">$ </span>sudo<span class="w"> </span>lvs<span class="w">
</span><span class="go"> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root ubuntu -wi-ao---- 132.42g
swap ubuntu -wi-ao---- 15.88g</span>
</pre>
<p>Finally, volumes in the <cite>Btrfs</cite> partition, <tt class="docutils literal"><span class="pre">/dev/mapper/ubuntu-root</span></tt>, are
mounted. The installer automatically creates two subvolumes for <tt class="docutils literal">/</tt> and
<tt class="docutils literal">/home</tt>.</p>
<pre class="code console literal-block">
<span class="gp">$ </span>mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>/dev/mapper/ubuntu-root<span class="w">
</span><span class="go">/dev/mapper/ubuntu-root on / type btrfs (rw,relatime,ssd,space_cache,subvolid=257,subvol=/@)
/dev/mapper/ubuntu-root on /home type btrfs (rw,relatime,ssd,space_cache,subvolid=258,subvol=/@home)
</span><span class="gp">$ </span>sudo<span class="w"> </span>btrfs<span class="w"> </span>subvolume<span class="w"> </span>list<span class="w"> </span>/<span class="w">
</span><span class="go">ID 257 gen 16908 top level 5 path @
ID 258 gen 16908 top level 5 path @home</span>
</pre>
<p>Note that since the device is an <cite>SSD</cite>, <cite>Btrfs</cite> enables optimizations for it
(visible in the <tt class="docutils literal">ssd</tt> mount option).</p>
</div>
<div class="section" id="additional-subvolumes">
<h2>Additional subvolumes<a class="headerlink" href="#additional-subvolumes" title="Permalink to this headline">¶</a></h2>
<p>Arbitrary additional subvolumes can be created in the filesystem, even under
the root one. For example, tools like <a class="reference external" href="https://linuxcontainers.org/lxc/">LXC</a>, <a class="reference external" href="https://linuxcontainers.org/lxd/">LXD</a>, and <a class="reference external" href="https://www.docker.com/">Docker</a> take advantage of
the Btrfs capabilities to store container filesystems and images in subvolumes,
so that they can be copied and snapshotted very quickly, without needing actual
data copy.</p>
<p>These are be listed among other subvolumes:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>btrfs<span class="w"> </span>subvolume<span class="w"> </span>list<span class="w"> </span>/<span class="w">
</span><span class="go">ID 257 gen 16908 top level 5 path @
ID 258 gen 16908 top level 5 path @home
ID 343 gen 3691 top level 257 path var/cache/lxc/trusty/rootfs-amd64
ID 348 gen 3811 top level 257 path var/lib/lxc/trusty/rootfs
ID 562 gen 12995 top level 257 path var/lib/docker/btrfs/subvolumes/a1723918aa603a5c9d63bff2fc623ccbcc5ad1cbeb8c048929c65237ce61bebc
ID 563 gen 12996 top level 257 path var/lib/docker/btrfs/subvolumes/e8eb5e7f51f415678c3126ca447e2df32d74fe041d0782bfb39357ae6cf28cec
ID 581 gen 16882 top level 257 path var/lib/lxd/images/6cb0ba80a5fe32357568a473cbaf69f14d26da0ba6b08f5b1bcde7053fc73757.btrfs</span>
</pre>
</div>
Install all needed packages in one go2016-04-09T00:00:00+00:002016-04-09T00:00:00+00:00Alberto Donatotag:albertodonato.net,2016-04-09:/blog/posts/install-all-needed-packages-in-one-go<p>Installing a new workstation (or reinstalling one) is quite a boring task; the
OS setup itself is pretty quick, but then you have to install all the tools
and applications you use every day.</p>
<p>I end up reinstalling both my laptop and desktop quite often, almost every time
a new …</p><p>Installing a new workstation (or reinstalling one) is quite a boring task; the
OS setup itself is pretty quick, but then you have to install all the tools
and applications you use every day.</p>
<p>I end up reinstalling both my laptop and desktop quite often, almost every time
a new Ubuntu release comes out (both to check out the news in the installer and
to start with a fresh system); so at some point I started putting the list of
the packages I needed in a script, which would <tt class="docutils literal">apt install</tt> them. This was
a quick and simple solution that would allow me to install everything with a
single command.</p>
<p>But wouldn't it be nice to also have an automatic way to keep this list in sync
across mulitple machines and be able to also remove packages if they become
unneeded?</p>
<p>To do that, <cite>metapackages</cite> are the perfect tool. They're just empty packages
that depend on other packages. Our list of packages just becomes a <tt class="docutils literal">Depends</tt>
directive in the metapackage.</p>
<div class="section" id="metapackages-to-the-rescue">
<h2>Metapackages to the rescue<a class="headerlink" href="#metapackages-to-the-rescue" title="Permalink to this headline">¶</a></h2>
<p>Building a metapackage is very easy, all you need is a tool named <tt class="docutils literal">equivs</tt>
and a few steps:</p>
<ol class="arabic">
<li><p class="first">Install it</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>apt<span class="w"> </span>install<span class="w"> </span>equivs
</pre>
</li>
<li><p class="first">Create a template control file for the package</p>
<pre class="code console literal-block">
<span class="gp">$ </span>equivs-control<span class="w"> </span>ack-desktop.control
</pre>
</li>
<li><p class="first">Edit the control file. Only a few lines are required, but at least at least
<tt class="docutils literal">Package</tt>, <tt class="docutils literal">Version</tt>, <tt class="docutils literal">Maintainer</tt>, and <tt class="docutils literal">Description</tt> should be
filled.</p>
<p>But the most important directive, as said, is <tt class="docutils literal">Depends</tt>. Here we must list
all the packages we want to install. The result is something like this:</p>
<pre class="code text literal-block">
Package: ack-desktop
Version: 1.0
Maintainer: Alberto Donato <alberto.donato@gmail.com>
Architecture: all
Section: misc
Priority: optional
Depends:
emacs-snapshot, emacs-snapshot-el,
bpython, ipython, ipython3, python-pip, python-virtualenv,
...
Description: metapackage for installation of my package selection.
This installs a whole set of packages that I currently use on Ubuntu desktops.
</pre>
</li>
<li><p class="first">Build the <tt class="docutils literal">.deb</tt> package</p>
<pre class="code console literal-block">
<span class="gp">$ </span>equivs-build<span class="w"> </span>ack-desktop.control
</pre>
<p>which creates <tt class="docutils literal"><span class="pre">ack-desktop_1.0_all.deb</span></tt> in the current directory.</p>
</li>
<li><p class="first">Install package and dependencies</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>dpkg<span class="w"> </span>-i<span class="w"> </span>ack-desktop_1.0_all.deb<span class="w">
</span><span class="gp">$ </span>sudo<span class="w"> </span>apt<span class="w"> </span>install<span class="w"> </span>-f<span class="w"> </span><span class="c1"># pull in all dependencies</span>
</pre>
</li>
<li><p class="first">Profit!</p>
</li>
</ol>
<p>Whenever the list of needed packages changes, it's enough to update the control
file, rebuild and install the new package.</p>
<p><em>IMPORTANT:</em> you need to increment the package version at every change.</p>
<p>A nice thing of this approach is that all dependencies will be marked as
automatically installed. So, if packages are removed from the list, <tt class="docutils literal">apt</tt>
will show them as "automatically installed and no longer needed", and <tt class="docutils literal">sudo
apt autoremove</tt> will uninstall them.</p>
</div>
<div class="section" id="setting-up-a-repository">
<h2>Setting up a repository<a class="headerlink" href="#setting-up-a-repository" title="Permalink to this headline">¶</a></h2>
<p>Now, what about making these package available on multiple computers?</p>
<p>The easiest way is to create a Debian repository hosting the metapackages, and
make it available to all computers. This way, just adding an <cite>APT</cite> source on
each machine will packages available everywhere, and they'll be updatable as
any other package in the distribution.</p>
<p>Setting up a repository for just a couple of packages might seem overkill, but
it's actually so simple that it's worth the effort.</p>
<p>A repository can be build and maintained with just few steps, using
<tt class="docutils literal">reprepro</tt>.</p>
<ol class="arabic">
<li><p class="first">Install it</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>apt<span class="w"> </span>install<span class="w"> </span>reprepro
</pre>
</li>
<li><p class="first">Create the base directory of your repository (let's call it <tt class="docutils literal">ubuntu/</tt>),
and a <tt class="docutils literal">conf/</tt> directory in it. Then create a <tt class="docutils literal">distributions</tt> file in
this directory, which will contain the configuration for reprepro:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>tree<span class="w"> </span>ubuntu/<span class="w">
</span><span class="go">ubuntu/
└── conf
└── distributions
1 directory, 1 file</span>
</pre>
</li>
<li><p class="first">Edit the <tt class="docutils literal">distributions</tt> file like this:</p>
<pre class="code text literal-block">
Codename: unstable
Architectures: source i386 amd64
Origin: local-unstable
Label: Local repository
Components: main
</pre>
<p>The reason for the "unstable" codename is that equivs will use it by default
when building packages.</p>
</li>
<li><p class="first">From the <tt class="docutils literal">ubuntu/</tt> directory, import the package created before</p>
<pre class="code console literal-block">
<span class="gp">$ </span>reprepro<span class="w"> </span>includedeb<span class="w"> </span>unstable<span class="w"> </span>../ack-desktop_1.0_all.deb
</pre>
<p>At the first import, <tt class="docutils literal">reprepro</tt> will create the repository tree.</p>
<p>Once the repository is set up, this is the only command that needs to be run
again, when importing a new package or a new version for an existing package.</p>
</li>
</ol>
<p>The repository is now ready to be used.</p>
<p>To make it available to all the machines, you can put host the <tt class="docutils literal">ubuntu/</tt> tree
on on a machine running a web server, but if you already sync your files using
a sevice like Dropbox or OwnCloud, an easy solution is to just have the tree
synced by it. Since metapackages have no real content, they're very small.</p>
<p>All you need is to create a repository entry</p>
<pre class="code console literal-block">
<span class="gp">$ </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"deb file:///home/ack/repo/ubuntu/ unstable main"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>tee<span class="w"> </span>/etc/apt/sources.list.d/local.list<span class="w">
</span><span class="go">deb file:///home/ack/repo/ubuntu/ unstable main</span>
</pre>
<p>with the correct path to the <tt class="docutils literal">ubuntu/</tt> directory.</p>
<p>At this point, you can just install your package with <tt class="docutils literal">apt</tt>, and it will
automatically pull dependencies:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>apt<span class="w"> </span>update<span class="w">
</span><span class="go">...
</span><span class="gp">$ </span>sudo<span class="w"> </span>apt<span class="w"> </span>install<span class="w"> </span>ack-desktop
</pre>
</div>
Tracking Launchpad karma2015-10-16T00:00:00+00:002015-10-16T00:00:00+00:00Alberto Donatotag:albertodonato.net,2015-10-16:/blog/posts/tracking-launchpad-karma<p><a class="reference external" href="https://launchpad.net/">Launchpad</a> tracks how active a user is by assigning points to each activity
that is performed, such as working with branches, managing bugs, packaging
(uploading to PPAs), merge proposals. This resulting score constitutes the user
"<a class="reference external" href="https://help.launchpad.net/YourAccount/Karma">Karma</a>".</p>
<p>Since I use Launchpad on a daily basis, I was curious about how my …</p><p><a class="reference external" href="https://launchpad.net/">Launchpad</a> tracks how active a user is by assigning points to each activity
that is performed, such as working with branches, managing bugs, packaging
(uploading to PPAs), merge proposals. This resulting score constitutes the user
"<a class="reference external" href="https://help.launchpad.net/YourAccount/Karma">Karma</a>".</p>
<p>Since I use Launchpad on a daily basis, I was curious about how my Karma varies
over time; unfortunately Launchpad doesn't have a way of doing this, since it
only reports the current value.</p>
<p>So I thought it'd be nice to have a simple script to collect the Karma value on
a daily basis (the value is not updated in real time in Launchpad so there's no
value in collecting it more frequently).</p>
<p>Of course getting the data is only half of the story, the fun part is actually
building some nice graphs. So, after having a first version of the script
running for a while and just collecting data in a SQLite database, I finally
got some time to extend it to also generate a page with graphs. You can see the
result <a class="reference external" href="https://albertodonato.net/launchpad-karma">here</a>.</p>
<p>If you want to track your own Karma, the script is available on <a class="reference external" href="https://bitbucket.org/ack/launchpad-karma">Bitbucket</a>; it
is trivial to set up and run daily via <tt class="docutils literal">cron</tt>. By default, it both fetches
the current value (storing it into the database) and regenerate the graph
page. Just make the target directory available via static file serve on your
web server and you're all set.</p>
HTTP/2-enabled2015-09-29T00:00:00+00:002015-09-29T00:00:00+00:00Alberto Donatotag:albertodonato.net,2015-09-29:/blog/posts/http2-enabled<p class="first last">This blog is now served on HTTP/2...</p>
<p>This blog is now served on HTTP/2. It's powered by <a class="reference external" href="https://nginx.org/">Nginx</a> 1.9.5, which includes
experimental support for the protocol.</p>
<p>Enabling HTTP/2 support in nginx is just a matter of adding the <tt class="docutils literal">http2</tt> option
to the <tt class="docutils literal">listen</tt> directive:</p>
<pre class="code nginx literal-block">
<span class="k">listen</span><span class="w"> </span><span class="mi">443</span><span class="w"> </span><span class="s">ssl</span><span class="w"> </span><span class="s">http2</span><span class="p">;</span><span class="w">
</span><span class="k">listen</span><span class="w"> </span><span class="s">[::]:443</span><span class="w"> </span><span class="s">ssl</span><span class="w"> </span><span class="s">http2</span><span class="p">;</span>
</pre>
<p>Previously, I've been using the <tt class="docutils literal">spdy</tt> option to enable the (also
experimental) <a class="reference external" href="https://www.chromium.org/spdy/spdy-whitepaper">SPDY/3.1</a> support. From version 1.9.5, this option is
superseded in favor of the <tt class="docutils literal">http2</tt> one.</p>
Twisted-like unit testing for asyncio2015-08-25T00:00:00+00:002015-08-25T00:00:00+00:00Alberto Donatotag:albertodonato.net,2015-08-25:/blog/posts/twisted-like-unit-testing-for-asyncio<p>Recently I've been porting some code for a personal project from <a class="reference external" href="https://twistedmatrix.com/">Twisted</a> to
the python3 builtin <a class="reference external" href="https://docs.python.org/3/library/asyncio.html">asyncio</a> library. The change is not as drastic as it might
seem, since the two libraries share the same basic concepts (using just
different names, e.g. Future vs Deferred, loop vs reactor) and …</p><p>Recently I've been porting some code for a personal project from <a class="reference external" href="https://twistedmatrix.com/">Twisted</a> to
the python3 builtin <a class="reference external" href="https://docs.python.org/3/library/asyncio.html">asyncio</a> library. The change is not as drastic as it might
seem, since the two libraries share the same basic concepts (using just
different names, e.g. Future vs Deferred, loop vs reactor) and the asyncio
design is quite inspired by Twisted.</p>
<p>One thing I found is that adapting the code was actually easier and quicker
than changing unit tests.</p>
<p>Testing asynchronous code requires some setup, like managing the event loop,
creating callback chains to assert on results from asynchronous methods.
Twisted provides very nice facilities for this, which eliminate some of the
setup boilerplate and make test clearer to read and more concise.</p>
<p>Its <tt class="docutils literal">trial</tt> test runner automatically manages setting up, starting and
stopping the reactor, and handles waiting for a <tt class="docutils literal">Deferred</tt> when a test method
returns one. This makes it possible to add callbacks to it, that assert on the
result of the asynchronous call. For example:</p>
<pre class="code python literal-block">
<span class="c1"># Twisted</span><span class="w">
</span><span class="k">def</span> <span class="nf">test_async_call</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span><span class="w">
</span> <span class="n">deferred</span> <span class="o">=</span> <span class="n">call_under_test</span><span class="p">()</span><span class="w">
</span> <span class="k">def</span> <span class="nf">check</span><span class="p">(</span><span class="n">result</span><span class="p">):</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="mi">19</span><span class="p">)</span><span class="w">
</span> <span class="k">return</span> <span class="n">deferred</span><span class="o">.</span><span class="n">addCallback</span><span class="p">(</span><span class="n">check</span><span class="p">)</span>
</pre>
<p>will run the <tt class="docutils literal">check</tt> callback when the deferred yields a result.
The same can be written more concisely using <tt class="docutils literal">@inlineCallbacks</tt>:</p>
<pre class="code python literal-block">
<span class="c1"># Twisted</span><span class="w">
</span><span class="nd">@inlineCallbacks</span><span class="w">
</span><span class="k">def</span> <span class="nf">test_async_call</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span><span class="w">
</span> <span class="n">result</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">call_under_test</span><span class="p">()</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="mi">19</span><span class="p">)</span>
</pre>
<p>which comes especially handy when multiple asynchronous calls are unsed in a
single test (so you don't need to chain a lot of callbacks).</p>
<p>While moving my code to asyncio, I wanted to keep the same unit test style, but
unfortunately I didn't find much builtin support for it. Looking at tests for
libraries using asyncio, I noticed most of them deal with the event loop
directly, running the method under test under it:</p>
<pre class="code python literal-block">
<span class="c1"># asyncio</span><span class="w">
</span><span class="k">def</span> <span class="nf">test_async_call</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span><span class="w">
</span> <span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span><span class="w">
</span> <span class="n">result</span> <span class="o">=</span> <span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">call_under_test</span><span class="p">())</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="mi">19</span><span class="p">)</span>
</pre>
<p>That's fine if you're testing a single call, but it gets cumbersome when you
have multiple calls, possibly with depending on the result of the previous one,
or when an asynchronous function calls others. In this case there might not be
a single <tt class="docutils literal">Future</tt> to wait for, and tests can easily become intricated.</p>
<p>Luckily, it wasn't to hard to implement a test behavior like the one provided
by Twisted with asyncio, so I created a <a class="reference external" href="https://github.com/albertodonato/toolrack/blob/master/toolrack/testing/async.py">LoopTestCase</a> which provides an event
loop (derived from <tt class="docutils literal">asyncio.test_utils.TestLoop</tt>). The testcase class
automatically wraps test methods so that if they return a <tt class="docutils literal">Future</tt> or are
coroutines, the loop is run until until a result is returned.</p>
<p>In essence, the <tt class="docutils literal">TestCase</tt> wraps the test method in its <tt class="docutils literal">run()</tt> with a
method that run the original method, and makes the event loop wait for it if
it's asynchronous:</p>
<pre class="code python literal-block">
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">result</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span><span class="w">
</span> <span class="n">test_method</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_testMethodName</span><span class="p">)</span><span class="w">
</span> <span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_testMethodName</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_wrap_async</span><span class="p">(</span><span class="n">test_method</span><span class="p">))</span><span class="w">
</span> <span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">result</span><span class="o">=</span><span class="n">result</span><span class="p">)</span><span class="w">
</span><span class="k">def</span> <span class="nf">_wrap_async</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method</span><span class="p">):</span><span class="w">
</span> <span class="nd">@wraps</span><span class="p">(</span><span class="n">method</span><span class="p">)</span><span class="w">
</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span><span class="w">
</span> <span class="n">result</span> <span class="o">=</span> <span class="n">method</span><span class="p">()</span><span class="w">
</span> <span class="k">if</span> <span class="n">iscoroutine</span><span class="p">(</span><span class="n">result</span><span class="p">)</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">Future</span><span class="p">):</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="k">async</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">))</span><span class="w">
</span> <span class="k">return</span> <span class="n">wrapper</span>
</pre>
<p>So the test case above simply becomes something like:</p>
<pre class="code python literal-block">
<span class="c1"># asyncio</span><span class="w">
</span><span class="k">def</span> <span class="nf">test_async_call</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span><span class="w">
</span> <span class="n">result</span> <span class="o">=</span> <span class="k">yield from</span> <span class="n">call_under_test</span><span class="p">()</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="mi">19</span><span class="p">)</span>
</pre>
<p>Notice that there's no need to decorate the method as <tt class="docutils literal">@coroutine</tt> (like with
Twisted's <tt class="docutils literal">@inlineCallbacks</tt>).</p>
<div class="section" id="controlling-time">
<h2>Controlling time<a class="headerlink" href="#controlling-time" title="Permalink to this headline">¶</a></h2>
<p>Both Twisted and asyncio provide methods scheduling function calls at a certain
time, or after a time delta. Testing code that use these functionalities
requires to be able to manipulate the event loop time manually, otherwise tests
would have to actually wait for time to pass, which could make them slow, and
possibly flaky. Twisted provides <tt class="docutils literal">twisted.internet.task.Clock</tt> which behaves
like the reactor, but provides an <tt class="docutils literal">advance()</tt> method to move the time
forward.</p>
<pre class="code python literal-block">
<span class="c1"># Twisted</span><span class="w">
</span><span class="k">def</span> <span class="nf">test_call_later</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span><span class="w">
</span> <span class="n">calls</span> <span class="o">=</span> <span class="p">[]</span><span class="w">
</span> <span class="n">clock</span> <span class="o">=</span> <span class="n">Clock</span><span class="p">()</span><span class="w">
</span> <span class="n">clock</span><span class="o">.</span><span class="n">callLater</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="n">calls</span><span class="o">.</span><span class="n">append</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">calls</span><span class="p">,</span> <span class="p">[])</span><span class="w">
</span> <span class="n">clock</span><span class="o">.</span><span class="n">advance</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">calls</span><span class="p">,</span> <span class="p">[</span><span class="kc">True</span><span class="p">])</span>
</pre>
<p>The <tt class="docutils literal">asyncio.test_utils.TestLoop</tt> also provides an <tt class="docutils literal">advance_time()</tt> method,
but this just moves the time forwards, so test code still needs to manually
schedule an event loop run to cause the scheduled function to be called.</p>
<p>To make this kind of test more concise, I enhanced the <tt class="docutils literal">TestLoop</tt> used by
<tt class="docutils literal">LoopTestCase</tt> to do this automatically, via an <tt class="docutils literal">advance()</tt> method.
The change is actually pretty simple:</p>
<pre class="code python literal-block">
<span class="k">def</span> <span class="nf">advance</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">advance</span><span class="p">):</span><span class="w">
</span><span class="sd">'''Advance the loop time and schedule a run.'''</span><span class="w">
</span> <span class="k">assert</span> <span class="n">advance</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">,</span> <span class="s1">'Time advance must not be negative'</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">advance_time</span><span class="p">(</span><span class="n">advance</span><span class="p">)</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">_run_once</span><span class="p">()</span>
</pre>
<p>With this addition, the previous test looks pretty much the same with asyncio:</p>
<pre class="code python literal-block">
<span class="c1"># asyncio</span><span class="w">
</span><span class="k">def</span> <span class="nf">test_call_later</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span><span class="w">
</span> <span class="n">calls</span> <span class="o">=</span> <span class="p">[]</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">call_later</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="n">calls</span><span class="o">.</span><span class="n">append</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">calls</span><span class="p">,</span> <span class="p">[])</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">advance</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">calls</span><span class="p">,</span> <span class="p">[</span><span class="kc">True</span><span class="p">])</span>
</pre>
<p>This becomes handier when dealing, for instance, with async code that is called
periodically, since in this case there isn't a single <tt class="docutils literal">Future</tt> that can be waited for.</p>
<p>Let's consider, as an example, a class that executes a given function at periodic time intervals:</p>
<pre class="code python literal-block">
<span class="c1"># asyncio</span><span class="w">
</span><span class="k">def</span> <span class="nf">test_periodic</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span><span class="w">
</span> <span class="n">calls</span> <span class="o">=</span> <span class="p">[]</span><span class="w">
</span> <span class="n">call</span> <span class="o">=</span> <span class="n">PeriodicCall</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">,</span> <span class="n">calls</span><span class="o">.</span><span class="n">append</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span><span class="w">
</span> <span class="n">call</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">calls</span><span class="p">,</span> <span class="p">[</span><span class="kc">True</span><span class="p">])</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">advance</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">calls</span><span class="p">,</span> <span class="p">[</span><span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">])</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">advance</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="w">
</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">calls</span><span class="p">,</span> <span class="p">[</span><span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">])</span>
</pre>
<p><tt class="docutils literal">PeriodicCall</tt> (again from my <a class="reference external" href="https://github.com/albertodonato/toolrack">ToolRack</a> library) is basically a port of
Twisted's <tt class="docutils literal">LoopingCall</tt> to asyncio. The <tt class="docutils literal">start()</tt> method calls the function
and schedules the next execution after the specified time.</p>
</div>
Zero-setup VPN2015-06-15T00:00:00+00:002015-06-15T00:00:00+00:00Alberto Donatotag:albertodonato.net,2015-06-15:/blog/posts/zero-setup-vpn<p><a class="reference external" href="https://en.wikipedia.org/wiki/Virtual_private_network">VPN</a>s are a great tool to access resources on a remote network as if they were
local.</p>
<p>Working remotely, I use them a lot, and even to access my home network when I'm
away.</p>
<p>Even though open-source projects like <a class="reference external" href="https://openvpn.net/">OpenVPN</a> have made it quite easy to set up
your …</p><p><a class="reference external" href="https://en.wikipedia.org/wiki/Virtual_private_network">VPN</a>s are a great tool to access resources on a remote network as if they were
local.</p>
<p>Working remotely, I use them a lot, and even to access my home network when I'm
away.</p>
<p>Even though open-source projects like <a class="reference external" href="https://openvpn.net/">OpenVPN</a> have made it quite easy to set up
your own VPN server and configure clients to connect to it, they still require
some setup, and an additional opened port for the VPN service.</p>
<p>A pretty handy VPN tool I've been using quite a lot lately is <a class="reference external" href="https://github.com/apenwarr/sshuttle">sshuttle</a>.</p>
<p><tt class="docutils literal">sshuttle</tt> is basically VPN-over-SSH. SSH is usually available out of the box
on any server machine (physical machines, cloud instances, containers), and can
be easily installed where it is not (like desktop machines) with a single
command.</p>
<p><tt class="docutils literal">sshuttle</tt> requires no server-side setup or special privileges on the remote
host: if you can SSH into a machine, you can use <tt class="docutils literal">sshuttle</tt> too. It creates
an SSH tunnel and forwards traffic for specific networks from the local machine
to the remote server.</p>
<div class="section" id="creating-a-vpn-connection">
<h2>Creating a VPN connection<a class="headerlink" href="#creating-a-vpn-connection" title="Permalink to this headline">¶</a></h2>
<p><tt class="docutils literal">sshuttle</tt> is pretty simple to use, for instance:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sshuttle<span class="w"> </span><span class="m">10</span>.0.1.0/24<span class="w"> </span><span class="m">10</span>.0.2.0/24<span class="w"> </span>--remote<span class="o">=</span>remote.example.com<span class="w"> </span>--auto-hosts
</pre>
<p>will forward traffic for the <tt class="docutils literal">10.0.1.0/24</tt> and <tt class="docutils literal">10.0.2.0/24</tt> subnets
through <tt class="docutils literal">remote.example.com</tt>, so that all machines on that network that are
accessible from the server machine, will be accessible from the remote host
too.</p>
<p>The <tt class="docutils literal"><span class="pre">--auto-hosts</span></tt> option is a pretty handy one: it automatically adds
discovered hostnames to the machine's <tt class="docutils literal">/etc/hosts</tt>, so that they can be used
in place of IP addresses.</p>
<p>Since <tt class="docutils literal">sshuttle</tt> uses SSH under the hood, any user configuration from
<tt class="docutils literal"><span class="pre">~/.ssh/config</span></tt> is respected.</p>
</div>
<div class="section" id="using-a-lot-of-sshuttles">
<h2>Using a lot of <tt class="docutils literal">sshuttle</tt>s?<a class="headerlink" href="#using-a-lot-of-sshuttles" title="Permalink to this headline">¶</a></h2>
<p>After ending up with various shell scripts to start/stop <tt class="docutils literal">sshuttle</tt>
connections to different networks (with different configurations), I thought
I'd write a simple tool to manage all <tt class="docutils literal">shuttle</tt> connections, and easily check
out which ones are connected: it's called <a class="reference external" href="https://github.com/albertodonato/sshoot">sshoot</a>.</p>
<p><tt class="docutils literal">sshoot</tt> is basically a connection manager for <tt class="docutils literal">sshuttle</tt>: it lets you
define profiles, using the same command line options that would be passed to
<tt class="docutils literal">sshuttle</tt>:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sshoot<span class="w"> </span>create<span class="w"> </span>--remote<span class="o">=</span>remote.example.com<span class="w"> </span>--auto-hosts<span class="w"> </span>vpn1<span class="w"> </span><span class="m">10</span>.0.1.0/24<span class="w"> </span><span class="m">10</span>.0.2.0/24
</pre>
<p>and start/stop the connection using the profile name:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sshoot<span class="w"> </span>start<span class="w"> </span>vpn1<span class="w">
</span><span class="go">Profile started
</span><span class="gp">$ </span>sshoot<span class="w"> </span>stop<span class="w"> </span>vpn1<span class="w">
</span><span class="go">Profile stopped</span>
</pre>
<p>It's also possible to check which profiles are defined and connected</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sshoot<span class="w"> </span>list<span class="w">
</span><span class="go"> Profile Remote host Subnets
----------------------------------------------------------
* vpn1 remote.example.com 10.0.1.0/24 10.0.2.0/24
vpn2 remote2.example.com 192.168.9.0/24</span>
</pre>
<p>In this case, the first profile is currently connected.</p>
</div>
<div class="section" id="installing-sshoot">
<h2>Installing <tt class="docutils literal">sshoot</tt><a class="headerlink" href="#installing-sshoot" title="Permalink to this headline">¶</a></h2>
<p><tt class="docutils literal">sshoot</tt> can be easily installed from source:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>git<span class="w"> </span>clone<span class="w"> </span>https://bitbucket.org/ack/sshoot.git<span class="w">
</span><span class="gp">$ </span><span class="nb">cd</span><span class="w"> </span>sshoot<span class="w">
</span><span class="gp">$ </span>python3<span class="w"> </span>setup.py<span class="w"> </span>install
</pre>
<p>For latest Ubuntu releases, packages are also available from the <a class="reference external" href="https://launchpad.net/~sshoot/+archive/ubuntu/stable">PPA</a>.
To install from packages:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo<span class="w"> </span>apt-add-repository<span class="w"> </span>ppa:sshoot/stable<span class="w">
</span><span class="gp">$ </span>sudo<span class="w"> </span>apt-get<span class="w"> </span>update<span class="w">
</span><span class="gp">$ </span>sudo<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>sshoot
</pre>
<p>That's it!</p>
</div>
Blog is live2015-06-07T00:00:00+00:002015-06-07T00:00:00+00:00Alberto Donatotag:albertodonato.net,2015-06-07:/blog/posts/blog-is-live<p class="first last">Check .... 1, 2, 3 ...</p>
<p>Ok, it works!</p>
<p>Cool, I have a blog now. It's time to put something on it. It'll mostly be tech
posts about software, stuff I do or I find interesting. Despite the title, it's
not going to be just about Python.</p>
<p>Stay tuned.</p>