blog.g3rt.nlhttps://blog.g3rt.nl/2023-04-14T00:00:00+02:00First release of Purepythonmilter: 0.0.12023-02-07T23:30:00+01:002023-02-07T23:30:00+01:00Gert van Dijktag:blog.g3rt.nl,2023-02-07:/purepythonmilter-first-release.html<p>Purepythonmilter aims to be a modern, Postfix-first, high-quality, strictly typed framework and library. And then all of that with an easy to use API and a high-performance asynchronous embedded server.</p><h1 id="its-my-first">It's my first...<a class="headerlink" href="#its-my-first" title="Permanent link"> </a></h1>
<p>I've just released my first βseriousβ open source software project into the public! π</p>
<p>It's <a href="https://github.com/gertvdijk/purepythonmilter"><strong>Purepythonmilter</strong></a> β a pure-Python Milter framework.
If you're operating your own mail server or developing software in the field of email,
you may be interested in this. π¨</p>
<p><em>Purepythonmilter</em> aims to be a modern, Postfix-first, high-quality, strictly typed
framework and library.
And then all of that with an easy to use API and a high-performance asynchronous
embedded server to point Postfix to.</p>
<h1 id="what-is-a-milter">What is a Milter? π€<a class="headerlink" href="#what-is-a-milter" title="Permanent link"> </a></h1>
<p>Mail servers (<a href="https://en.wikipedia.org/wiki/Message_transfer_agent">MTA</a>s) like <a href="https://www.postfix.org/">Postfix</a> and
<a href="https://www.sendmail.org/">Sendmail</a> can connect to an external filter process, called a
'Milter', for actions to take during an incoming SMTP transaction.
You may consider it like a plugin on the mail server software using callbacks over a TCP
or UNIX socket.</p>
<p>An very much simplified example using Purepythonmilter:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #ababab; font-style: italic"># pip install purepythonmilter</span>
<span style="color: #6ebf26; font-weight: bold">import</span> <span style="color: #71adff; text-decoration: underline">purepythonmilter</span> <span style="color: #6ebf26; font-weight: bold">as</span> <span style="color: #71adff; text-decoration: underline">ppm</span>
<span style="color: #6ebf26; font-weight: bold">async</span> <span style="color: #6ebf26; font-weight: bold">def</span> <span style="color: #71adff">on_mail_from</span><span style="color: #d0d0d0">(cmd:</span> <span style="color: #d0d0d0">ppm.MailFrom)</span> <span style="color: #d0d0d0">-></span> <span style="color: #d0d0d0">ppm.VerdictOrContinue:</span>
<span style="color: #666666"> </span><span style="color: #ed9d13">"""</span>
<span style="color: #ed9d13"> Block all mail with a sender address (envelope-from) on the example.com domain.</span>
<span style="color: #ed9d13"> """</span>
<span style="color: #6ebf26; font-weight: bold">if</span> <span style="color: #d0d0d0">cmd.address.lower().endswith(</span><span style="color: #ed9d13">"@example.com"</span><span style="color: #d0d0d0">):</span>
<span style="color: #6ebf26; font-weight: bold">return</span> <span style="color: #d0d0d0">ppm.RejectWithCode(primary_code=(</span><span style="color: #51b2fd">5</span><span style="color: #d0d0d0">,</span> <span style="color: #51b2fd">7</span><span style="color: #d0d0d0">,</span> <span style="color: #51b2fd">1</span><span style="color: #d0d0d0">),</span> <span style="color: #d0d0d0">text=</span><span style="color: #ed9d13">"not allowed here!"</span><span style="color: #d0d0d0">)</span>
<span style="color: #6ebf26; font-weight: bold">return</span> <span style="color: #d0d0d0">ppm.Continue()</span>
<span style="color: #d0d0d0">mymilter</span> <span style="color: #d0d0d0">=</span> <span style="color: #d0d0d0">ppm.PurePythonMilter(name=</span><span style="color: #ed9d13">"mymilter"</span><span style="color: #d0d0d0">,</span> <span style="color: #d0d0d0">hook_on_mail_from=on_mail_from)</span>
<span style="color: #ababab; font-style: italic"># Configure Postfix with 'smtpd_milters = inet:127.0.0.1:9000'</span>
<span style="color: #d0d0d0">mymilter.run_server(host=</span><span style="color: #ed9d13">"127.0.0.1"</span><span style="color: #d0d0d0">,</span> <span style="color: #d0d0d0">port=</span><span style="color: #51b2fd">9000</span><span style="color: #d0d0d0">)</span>
</code></pre></div>
<p>Just run this as Python file and <em>it just works</em> β press Ctrl+C to shut it down
gracefully. β </p>
<p>More <a href="https://github.com/gertvdijk/purepythonmilter/tree/develop/src/purepythonmilter/examples">examples</a> are provided in the repository to get started
and include instructions to test your app with Postfix in Docker. π³</p>
<h1 id="why-this-project">Why this project?<a class="headerlink" href="#why-this-project" title="Permanent link"> </a></h1>
<p>Purepythonmilter was written as an alternative to, and, out of frustration with
<a href="https://pythonhosted.org/pymilter/">PyMilter</a>.
PyMilter is not type annotated (mypy), has signal handling issues (for me), the
dependency on a <a href="https://github.com/sdgathman/pymilter/blob/master/miltermodule.c">hand-crafted Python-C-extension</a> linking to
Sendmail's libmilter and no offering of a binary package (<a href="https://peps.python.org/pep-0427/">wheel</a>) to ease
installation. π₯</p>
<p>The aforementioned example above takes a lot more lines of code to get running using
PyMilter with its subclassing-style API as well.</p>
<p>Apart from these issues I have some concerns on the overall software quality of existing
projects and I want to gain more experience in writing async Python applications. π</p>
<p>Purepythonmilter may be far from a stable release, but it has ~ 95% test coverage and
validates 100% with mypy 1.0.0 in strict mode to deserve a solid start. π</p>
<h1 id="feedback-requested">Feedback requested π¬<a class="headerlink" href="#feedback-requested" title="Permanent link"> </a></h1>
<p>Please check out the <a href="https://github.com/gertvdijk/purepythonmilter">GitHub page</a>, install it
<a href="https://pypi.org/project/purepythonmilter/">from PyPI</a> and let me know what you think in the
<a href="https://github.com/gertvdijk/purepythonmilter/discussions/">discussions</a>, via
<a href="https://twitter.com/gertvdijk">Twitter</a> or email!</p>
<p>It's Apache-2.0-licensed allowing you to use it in a commercial setting as well.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">My first release of Purepythonmilter: 0.0.1 π<br><br>A pure-<a href="https://twitter.com/hashtag/Python?src=hash&ref_src=twsrc%5Etfw">#Python</a> Milter framework to quickly build your custom <a href="https://twitter.com/hashtag/Postfix?src=hash&ref_src=twsrc%5Etfw">#Postfix</a> smtpd 'plugin'. <a href="https://twitter.com/hashtag/opensource?src=hash&ref_src=twsrc%5Etfw">#opensource</a> <a href="https://twitter.com/hashtag/email?src=hash&ref_src=twsrc%5Etfw">#email</a><a href="https://t.co/vChUHJKb4C">https://t.co/vChUHJKb4C</a></p>— Gert van Dijk (@gertvdijk) <a href="https://twitter.com/gertvdijk/status/1623087511223271425?ref_src=twsrc%5Etfw">February 7, 2023</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>Generate QR-codes for Wi-Fi and Wireguard directly from your 'pass' passwordstore2021-10-29T00:00:00+02:002021-10-29T00:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2021-10-29:/generate-qr-code-wifi-wireguard-passwordstore.html<p>Conveniently and quickly generate QR codes for Wi-Fi access or Wireguard profiles with these shell scripts.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>genqr-wifi.sh<span style="color: #666666"> </span>mynetworkname
<span style="color: #cccccc">Wi-Fi network: mynetworkname</span>
<span style="color: #cccccc">βββββββββββββββββββββββββββββββββββββββββ</span>
<span style="color: #cccccc">βββββββββββββββββββββββββββββββββββββββββ</span>
<span style="color: #cccccc">ββββ βββββ β ββββ βββ ββ ββ βββββ ββββ</span>
<span style="color: #cccccc">ββββ β β β ββ β ββββββββ βββ β β ββββ</span>
<span style="color: #cccccc">ββββ βββββ ββββββββ βββ ββββββ βββββ ββββ</span>
<span style="color: #cccccc">ββββββββββββββββ βββββββ β β ββββββββββββ</span>
<span style="color: #cccccc">βββββββββ βββββββ βββββ ββββββ β βββββββ</span>
<span style="color: #cccccc">ββββ ββ ββββββ β ββββββ ββββ ββββ ββββββ</span>
<span style="color: #cccccc">ββββββββ ββββββββ ββββ ββββββ β ββββββββ</span>
<span style="color: #cccccc">ββββ βββββββ βββββββ ββββββββ βββββ ββββ</span>
<span style="color: #cccccc">βββββ β βββββ βββ βββββββββββββββββ βββββ</span>
<span style="color: #cccccc">ββββββββββββββ ββ βββββ ββββββββββ βββββ</span>
<span style="color: #cccccc">βββββββββ ββ βββββ ββββ β ββββ ββ ββββββ</span>
<span style="color: #cccccc">ββββββ β ββ βββββββ ββββββββ ββββ ββββββ</span>
<span style="color: #cccccc">βββββββββββββ ββ ββ β ββ β βββ β ββββββ</span>
<span style="color: #cccccc">ββββ βββββ ββββ ββββββββ β β βββ β βββββ</span>
<span style="color: #cccccc">ββββ β β βββ ββ ββββ ββββββ β ββ ββββ</span>
<span style="color: #cccccc">ββββ βββββ ββ βββ βββββ βββββββββββββββββ</span>
<span style="color: #cccccc">βββββββββββββββββββββββββββββββββββββββββ</span>
<span style="color: #cccccc">βββββββββββββββββββββββββββββββββββββββββ</span>
<span style="color: #cccccc">βββββββββββββββββββββββββββββββββββββββββ</span>
</code></pre></div>
<p>Cool, huh!? π</p>
<h1 id="concepts-and-security-concerns">Concepts and security concerns<a class="headerlink" href="#concepts-and-security-concerns" title="Permanent link"> </a></h1>
<h2 id="credentials-management">Credentials management<a class="headerlink" href="#credentials-management" title="Permanent link"> </a></h2>
<p>Firstly, I'd like to highlight the value of generating something locally on your machine rather than a random app or
site.
Entering credentials in online QR code generators come with significant risks as you basically enter them on an
untrusted website.
Even if they promise generating them in a local Javascript-only fashion, I consider it a bad practice to enter these
credentials in a browser window.
This post is about generating the QR code locally (with <code>qrencode</code>) and credentials won't leave your system.</p>
<p>Secondly, the use of a local password manager application that allows for integration in shell scripts.
I use <a href="https://passwordstore.org/">pass</a>, and I have the passwords for Wi-Fi networks and keys for Wireguard tunnels in there.
The use of this as the password manager allows for easy integration in shell scripts in its design.
But technically you could use any password store that allows you to obtain the secrets needed in shell scripts.</p>
<p>In this post I'll demo two example applications for this: Wi-Fi passwords (phones typically support that nowadays) and
setting up a Wireguard tunnel (as supported by the <a href="https://play.google.com/store/apps/details?id=com.wireguard.android">Wireguard app for Android</a>).</p>
<div class="admonition important">
<p class="admonition-title">At work</p>
<p>This could also be very useful in the office to quickly on-board a new team member for example!</p>
</div>
<h2 id="qr-codes-specification">QR codes specification<a class="headerlink" href="#qr-codes-specification" title="Permanent link"> </a></h2>
<p>Technically, it's possible to encode anything as a QR code, it's basically just freeform with respect to QR.
What's more important, though, is <em>how</em> to encode the text you feed the QR-encoder.</p>
<p><strong>For Wi-Fi</strong> access there's a standard on how it's structured; it is available in the Wi-Fi Alliance
<a href="https://www.wi-fi.org/download.php?file=/sites/default/files/private/WPA3_Specification_v3.0.pdf">specification document</a> section 7.</p>
<blockquote>
<p>Some examples of the WIFI URI format are as follows:</p>
<ol>
<li>
<p><code>WIFI:T:WPA;S:MyNet;P:MyPassword;;</code></p>
<ul>
<li>STA that supports WPA3-Personal might use SAE or PSK (WPA3-Personal transition mode)</li>
<li>STA that does not support WPA3-Personal uses PSK (WPA2-Personal)</li>
</ul>
</li>
</ol>
</blockquote>
<p><strong>For Wireguard</strong> tunnels in the app it's just the whole configuration file (that you'd normally use with <code>wg setconf</code>)
contents itself.</p>
<p>With the text you want to encode ready, you can then use any QR code generator.
One example is LibreOffice Writer (Menu <em>Insert</em> β <em>Object</em> β <em>QR Code</em>).</p>
<h1 id="scripts">Scripts<a class="headerlink" href="#scripts" title="Permanent link"> </a></h1>
<p>These scripts should be pretty much self-descriptive.
They just format a text entry and just pipe it to <code>qrencode --type=ANSIUTF8</code>, so you'll have to install <code>qrencode</code> on
your system.</p>
<p>Put the scripts somewhere on your <code>PATH</code>, make them executable (e.g. <code>chmod +x genqr-wifi.sh</code>) and make sure to adjust
the logic to your needs and situation to obtain the credentials.</p>
<h2 id="wi-fi">Wi-Fi<a class="headerlink" href="#wi-fi" title="Permanent link"> </a></h2>
<p><code>genqr-wifi.sh</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #ababab; font-style: italic">#!/usr/bin/env bash</span>
<span style="color: #2fbccd">set</span><span style="color: #666666"> </span>-e<span style="color: #666666"> </span>-u
<span style="color: #2fbccd">set</span><span style="color: #666666"> </span>-o<span style="color: #666666"> </span>pipefail
<span style="color: #ababab; font-style: italic"># Obtain Wi-Fi password from `wifi/<SSID>` entry in passwordstore</span>
<span style="color: #ababab; font-style: italic"># and print its QR-code for access on the commandline using ANSI</span>
<span style="color: #ababab; font-style: italic"># terminal format.</span>
<span style="color: #6ebf26; font-weight: bold">function</span><span style="color: #666666"> </span>show_wifi_qr<span style="color: #d0d0d0">()</span><span style="color: #666666"> </span><span style="color: #d0d0d0">{</span>
<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># TODO: figure out how to properly escape SSID and password</span>
<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># special characters.</span>
<span style="color: #666666"> </span><span style="color: #40ffff">WPA3_FLAG</span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># set "R:1;" to indicate WPA-3 compatibility</span>
<span style="color: #666666"> </span><span style="color: #40ffff">WIFI_SSID</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"</span><span style="color: #40ffff">$1</span><span style="color: #ed9d13">"</span>
<span style="color: #666666"> </span><span style="color: #40ffff">WIFI_PASS</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"</span><span style="color: #6ebf26; font-weight: bold">$(</span>pass<span style="color: #666666"> </span>show<span style="color: #666666"> </span><span style="color: #ed9d13">"wifi/${</span><span style="color: #40ffff">WIFI_SSID</span><span style="color: #ed9d13">}"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>head<span style="color: #666666"> </span>-n1<span style="color: #6ebf26; font-weight: bold">)</span><span style="color: #ed9d13">"</span>
<span style="color: #666666"> </span><span style="color: #40ffff">QR_CODE_TEXT</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"WIFI:T:WPA;${</span><span style="color: #40ffff">WPA3_FLAG</span><span style="color: #ed9d13">}S:${</span><span style="color: #40ffff">WIFI_SSID</span><span style="color: #ed9d13">};P:${</span><span style="color: #40ffff">WIFI_PASS</span><span style="color: #ed9d13">};;"</span>
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #ed9d13">"Wi-Fi network: ${</span><span style="color: #40ffff">WIFI_SSID</span><span style="color: #ed9d13">}"</span>
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span>-n<span style="color: #666666"> </span><span style="color: #ed9d13">"${</span><span style="color: #40ffff">QR_CODE_TEXT</span><span style="color: #ed9d13">}"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>qrencode<span style="color: #666666"> </span>--type<span style="color: #d0d0d0">=</span>ANSIUTF8
<span style="color: #d0d0d0">}</span>
show_wifi_qr<span style="color: #666666"> </span><span style="color: #ed9d13">"</span><span style="color: #40ffff">$1</span><span style="color: #ed9d13">"</span>
</code></pre></div>
<h2 id="wireguard">Wireguard<a class="headerlink" href="#wireguard" title="Permanent link"> </a></h2>
<p>This assumes a <code>pass</code> layout like this, but should be trivial to adjust to your needs.
The entry should contain the private key as printed with <code>wg genkey</code> and the <code>psk</code> shared secret I use for additional
security, contents as printed by <code>wg genpsk</code>.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>pass<span style="color: #666666"> </span>ls
<span style="color: #cccccc">Password Store</span>
<span style="color: #cccccc">βββ wireguard</span>
<span style="color: #cccccc"> βββ myinstance</span>
<span style="color: #cccccc"> βΒ Β βββ mydevice</span>
<span style="color: #cccccc"> βΒ Β βββ myotherdevice</span>
<span style="color: #cccccc"> βΒ Β βββ myserver</span>
<span style="color: #cccccc"> βΒ Β βββ psk</span>
<span style="color: #cccccc">[...]</span>
</code></pre></div>
<p><code>genqr-wireguard.sh</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #ababab; font-style: italic">#!/usr/bin/env bash</span>
<span style="color: #2fbccd">set</span><span style="color: #666666"> </span>-e<span style="color: #666666"> </span>-u
<span style="color: #2fbccd">set</span><span style="color: #666666"> </span>-o<span style="color: #666666"> </span>pipefail
<span style="color: #6ebf26; font-weight: bold">function</span><span style="color: #666666"> </span>prepare_wireguard_credentials<span style="color: #d0d0d0">()</span><span style="color: #666666"> </span><span style="color: #d0d0d0">{</span>
<span style="color: #666666"> </span><span style="color: #40ffff">INSTANCE_NAME</span><span style="color: #d0d0d0">=</span><span style="color: #40ffff">$1</span>
<span style="color: #666666"> </span><span style="color: #40ffff">DEVICE_NAME</span><span style="color: #d0d0d0">=</span><span style="color: #40ffff">$2</span>
<span style="color: #666666"> </span><span style="color: #40ffff">DEVICE_IP_ADDRESS</span><span style="color: #d0d0d0">=</span><span style="color: #40ffff">$3</span>
<span style="color: #666666"> </span><span style="color: #40ffff">DEVICE_DNS_SERVERS</span><span style="color: #d0d0d0">=</span><span style="color: #40ffff">$4</span>
<span style="color: #666666"> </span><span style="color: #40ffff">PEER_INVENTORY_NAME</span><span style="color: #d0d0d0">=</span><span style="color: #40ffff">$5</span>
<span style="color: #666666"> </span><span style="color: #40ffff">PEER_ENDPOINT</span><span style="color: #d0d0d0">=</span><span style="color: #40ffff">$6</span>
<span style="color: #666666"> </span><span style="color: #40ffff">PEER_ALLOWED_IPS</span><span style="color: #d0d0d0">=</span><span style="color: #40ffff">$7</span>
<span style="color: #666666"> </span><span style="color: #40ffff">DEVICE_PRIVATE_KEY</span><span style="color: #d0d0d0">=</span><span style="color: #6ebf26; font-weight: bold">$(</span>pass<span style="color: #666666"> </span>show<span style="color: #666666"> </span><span style="color: #ed9d13">"wireguard/${</span><span style="color: #40ffff">INSTANCE_NAME</span><span style="color: #ed9d13">}/${</span><span style="color: #40ffff">DEVICE_NAME</span><span style="color: #ed9d13">}"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>head<span style="color: #666666"> </span>-n1<span style="color: #6ebf26; font-weight: bold">)</span>
<span style="color: #666666"> </span><span style="color: #40ffff">PEER_PUBLIC_KEY</span><span style="color: #d0d0d0">=</span><span style="color: #6ebf26; font-weight: bold">$(</span>pass<span style="color: #666666"> </span>show<span style="color: #666666"> </span><span style="color: #ed9d13">"wireguard/${</span><span style="color: #40ffff">INSTANCE_NAME</span><span style="color: #ed9d13">}/${</span><span style="color: #40ffff">PEER_INVENTORY_NAME</span><span style="color: #ed9d13">}"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>head<span style="color: #666666"> </span>-n1<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>wg<span style="color: #666666"> </span>pubkey<span style="color: #6ebf26; font-weight: bold">)</span>
<span style="color: #666666"> </span><span style="color: #40ffff">PEER_PRESHAREDKEY</span><span style="color: #d0d0d0">=</span><span style="color: #6ebf26; font-weight: bold">$(</span>pass<span style="color: #666666"> </span>show<span style="color: #666666"> </span><span style="color: #ed9d13">"wireguard/${</span><span style="color: #40ffff">INSTANCE_NAME</span><span style="color: #ed9d13">}/psk"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>head<span style="color: #666666"> </span>-n1<span style="color: #6ebf26; font-weight: bold">)</span>
<span style="color: #d0d0d0">}</span>
<span style="color: #6ebf26; font-weight: bold">function</span><span style="color: #666666"> </span>make_wireguard_config<span style="color: #d0d0d0">()</span><span style="color: #666666"> </span><span style="color: #d0d0d0">{</span>
<span style="color: #666666"> </span><span style="color: #40ffff">WIREGUARD_CONFIG</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"# ${</span><span style="color: #40ffff">INSTANCE_NAME</span><span style="color: #ed9d13">} for device ${</span><span style="color: #40ffff">DEVICE_NAME</span><span style="color: #ed9d13">}</span>
<span style="color: #ed9d13">[Interface]</span>
<span style="color: #ed9d13">PrivateKey = ${</span><span style="color: #40ffff">DEVICE_PRIVATE_KEY</span><span style="color: #ed9d13">}</span>
<span style="color: #ed9d13">Address = ${</span><span style="color: #40ffff">DEVICE_IP_ADDRESS</span><span style="color: #ed9d13">}</span>
<span style="color: #ed9d13">DNS = ${</span><span style="color: #40ffff">DEVICE_DNS_SERVERS</span><span style="color: #ed9d13">}</span>
<span style="color: #ed9d13">[Peer]</span>
<span style="color: #ed9d13">PublicKey = ${</span><span style="color: #40ffff">PEER_PUBLIC_KEY</span><span style="color: #ed9d13">}</span>
<span style="color: #ed9d13">PresharedKey = ${</span><span style="color: #40ffff">PEER_PRESHAREDKEY</span><span style="color: #ed9d13">}</span>
<span style="color: #ed9d13">Endpoint = ${</span><span style="color: #40ffff">PEER_ENDPOINT</span><span style="color: #ed9d13">}</span>
<span style="color: #ed9d13">AllowedIPs = ${</span><span style="color: #40ffff">PEER_ALLOWED_IPS</span><span style="color: #ed9d13">}</span>
<span style="color: #ed9d13">"</span>
<span style="color: #d0d0d0">}</span>
<span style="color: #6ebf26; font-weight: bold">function</span><span style="color: #666666"> </span>show_wireguard_qr<span style="color: #d0d0d0">()</span><span style="color: #666666"> </span><span style="color: #d0d0d0">{</span>
<span style="color: #666666"> </span>prepare_wireguard_credentials<span style="color: #666666"> </span><span style="color: #ed9d13">"</span><span style="color: #40ffff">$@</span><span style="color: #ed9d13">"</span>
<span style="color: #666666"> </span>make_wireguard_config
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #ed9d13">"Wireguard instance: ${</span><span style="color: #40ffff">INSTANCE_NAME</span><span style="color: #ed9d13">} for device ${</span><span style="color: #40ffff">DEVICE_NAME</span><span style="color: #ed9d13">}"</span>
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span>-n<span style="color: #666666"> </span><span style="color: #ed9d13">"${</span><span style="color: #40ffff">WIREGUARD_CONFIG</span><span style="color: #ed9d13">}"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>qrencode<span style="color: #666666"> </span>--type<span style="color: #d0d0d0">=</span>ANSIUTF8
<span style="color: #d0d0d0">}</span>
show_wireguard_qr<span style="color: #666666"> </span>myinstance<span style="color: #666666"> </span>mydevice<span style="color: #666666"> </span><span style="color: #51b2fd">10</span>.1.2.3/24<span style="color: #666666"> </span><span style="color: #ed9d13">"8.8.8.8 9.9.9.9"</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>myserver<span style="color: #666666"> </span><span style="color: #51b2fd">11</span>.22.33.44:51820<span style="color: #666666"> </span><span style="color: #51b2fd">0</span>.0.0.0/0
show_wireguard_qr<span style="color: #666666"> </span>myinstance<span style="color: #666666"> </span>myotherdevice<span style="color: #666666"> </span><span style="color: #51b2fd">10</span>.1.2.4/24<span style="color: #666666"> </span><span style="color: #ed9d13">"8.8.8.8 9.9.9.9"</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>myserver<span style="color: #666666"> </span><span style="color: #51b2fd">11</span>.22.33.44:51820<span style="color: #666666"> </span><span style="color: #51b2fd">0</span>.0.0.0/0
</code></pre></div>
<h1 id="results">Results<a class="headerlink" href="#results" title="Permanent link"> </a></h1>
<p>Now whenever someone asks me for access on my Wi-Fi with a phone π±, I can just type the following and it will always
include the current password. β </p>
<p>For Wireguard I like it in particular just for my own phone.
Whenever I change the configuration or rotate keys, I can just delete it and scan the new one with ease. π</p>
<p>At work, I create a git repository that includes the script and password store files, add the use <a href="https://direnv.net/">direnv</a> to
adjust the <code>PASSWORD_STORE_DIR</code> and <code>PATH</code> and the possibilities are endless!</p>SSH host key validation done right β strict yet user-friendly2021-10-27T22:15:00+02:002021-10-27T22:15:00+02:00Gert van Dijktag:blog.g3rt.nl,2021-10-27:/ssh-host-key-validation-strict-yet-user-friendly.html<p>Validating the SSH host key is equally important as validating HTTPS certificates in a browser; but you have to do it yourself. In this post I will try to convince you to manage a "known hosts" file yourself and you'll should never see the prompt again to accept a host key.</p>
<h1 id="what-is-an-ssh-host-key">What is an SSH <em>host</em> key?<a class="headerlink" href="#what-is-an-ssh-host-key" title="Permanent link"> </a></h1>
<p>For visiting a website over HTTPS securely, it's crucial that an HTTPS client application validates the server
certificate.
This is to ensure the integrity of the connection (not being a victim of a <a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">man-in-the-middle attack</a>,
for example).</p>
<p>The same goes for SSH connections, but the difference is that SSH does not typically use a certificate authority with
certificate chains and trust anchor stores.
Instead, in most scenarios the public key as a whole is checked for equality for a "known" one on your system that
matches for this hostname on an earlier connection.</p>
<div class="admonition important">
<p class="admonition-title">Important</p>
<p>Do not confuse SSH <em>host</em> key verification (target machine) with <i>client</i> key authentication "authorized keys",
nor with <i>host based</i> authentication; these three different keys each have their own purpose in the protocol.</p>
<p>It's very common to talk about "SSH keys" referring to authentication keys for you as a user, but in order to ensure
integrity and security, the importance of validating SSH host keys should be understood.</p>
</div>
<h1 id="aim-of-this-post">Aim of this post<a class="headerlink" href="#aim-of-this-post" title="Permanent link"> </a></h1>
<p>In this post I explain how to manage a typical scenario in which you manage a moderate amount of hosts:
Manage them manually in a file you put in version control, share it with your team and use it with your favourite
automation tool like Ansible.</p>
<h1 id="the-tofu-model-problem">The TOFU-model problem<a class="headerlink" href="#the-tofu-model-problem" title="Permanent link"> </a></h1>
<p>The typical SSH client configuration is to store the public key of the host key on the first connection.
By default, it stores the keys seen in <code>~/.ssh/known_hosts</code>, via OpenSSH's <code>UserKnownHostsFile</code> setting.
Have a look and open that file; it may include <em>a lot</em> of lines and you probably don't understand what line is for which
host. π</p>
<p>Whenever you connect to a host for the first time, it asks you to accept the key by its fingerprint.
This looks like this, typically:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>ssh<span style="color: #666666"> </span>user@host.tld
<span style="color: #cccccc">The authenticity of host 'host.tld (1.2.3.4)' can't be established.</span>
<span style="color: #cccccc">ED25519 key fingerprint is SHA256:eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8.</span>
<span style="color: #cccccc">Are you sure you want to continue connecting (yes/no)?</span>
</code></pre></div>
<p>A major problem with this <a href="https://en.wikipedia.org/wiki/Trust_on_first_use">"Trust On First Use" (TOFU)</a> model is that you will just type <em>yes</em> and
'be done with it', instead of going through the pain of validating it. π
I mean, who wouldn't be annoyed by this message and reluctant to carefully check, right?
This problem grows with the size of your team, as every member has to validate the key fingerprint for himself in that
prompt!</p>
<p>Even more so when you reinstall a server or change the IP of a server, you (and each member in your team) will be
presented with this scary message:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>ssh<span style="color: #666666"> </span>user@host.tld
<span style="color: #cccccc">@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@</span>
<span style="color: #cccccc">@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @</span>
<span style="color: #cccccc">@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@</span>
<span style="color: #cccccc">IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!</span>
<span style="color: #cccccc">Someone could be eavesdropping on you right now (man-in-the-middle attack)!</span>
<span style="color: #cccccc">It is also possible that a host key has just been changed.</span>
<span style="color: #cccccc">The fingerprint for the ED25519 key sent by the remote host is</span>
<span style="color: #cccccc">SHA256:PqpHHVHz+guHaNZMoQIhTKJsE4ByH1xkkM9qfTJOheI.</span>
<span style="color: #cccccc">Please contact your system administrator.</span>
<span style="color: #cccccc">Add correct host key in /home/myusername/.ssh/known_hosts to get rid of this message.</span>
<span style="color: #cccccc">Offending ED25519 key in /home/myusername/.ssh/known_hosts:3</span>
<span style="color: #cccccc"> remove with:</span>
<span style="color: #cccccc"> ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "10.1.2.3"</span>
<span style="color: #cccccc">ED25519 host key for 10.1.2.3 has changed and you have requested strict checking.</span>
<span style="color: #cccccc">Host key verification failed.</span>
</code></pre></div>
<p>The thing you will probably do is just follow that suggested command to remove the key stored, no? π
Well, that may provide an actual attacker his way in... π€¦</p>
<p>I strongly believe that this important warning should <em>never</em> become something that will be "annoying" to go ignored in
practice and undermine the security properties of the SSH protocol.
With more careful management of the "known hosts" file one can ensure that such a message will only be displayed with a
true positive for an attack or network misconfiguration.</p>
<h1 id="opensshs-known-hosts-files-purpose-and-format">OpenSSH's "known hosts" files purpose and format<a class="headerlink" href="#opensshs-known-hosts-files-purpose-and-format" title="Permanent link"> </a></h1>
<p>By default, OpenSSH would check several "known hosts" files on your filesystem to see if a key matches the one the
target machine is using.
Both the <code>UserKnownHostsFile</code> and <code>GlobalKnownHostsFile</code> SSH client settings are used to specify paths for "known hosts"
files.
The "user" one is written to and read from, the "global" one is considered read-only by the SSH client.
Each setting can also take multiple files, but unlike the <code>Include</code> setting, it does not let you specify a ".d"-style
directory with separate files, unfortunately. π</p>
<p>The contents of a "known hosts" file is straightforward; each line consists of the machine's host name or IP address,
the key and some optional comments, each of these fields space-separated.
A full explanation on the format can be found in the <a href="https://man.openbsd.org/sshd#SSH_KNOWN_HOSTS_FILE_FORMAT">sshd(8) manpage</a>.
Why this is explained in the <em>SSHD</em> (server) manpage rather than the SSH client ssh(1) manpage is a mystery to me.
It's SSH <em>clients</em> using this logic, after all, and you generally do not have an SSH server installed on client
machines. π€·</p>
<p>An excerpt from the manpage:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>SSH_KNOWN_HOSTS FILE FORMAT
The /etc/ssh/ssh_known_hosts and ~/.ssh/known_hosts files contain host pubβ
lic keys for all known hosts. [...]
Each line in these files contains the following fields: markers (optional),
hostnames, keytype, base64-encoded key, comment. The fields are separated
by spaces.
[...]
Hostnames is a comma-separated list of patterns (β*β and β?β act as wildβ
cards); each pattern in turn is matched against the host name. [...]
When ssh(1) is authenticating a server, this will be the host name given by
the user, the value of the ssh(1) HostkeyAlias if it was specified, or the
canonical server hostname if the ssh(1) CanonicalizeHostname option was
used.
[...]
Alternately, hostnames may be stored in a hashed form which hides host
names and addresses should the file's contents be disclosed. Hashed hostβ
names start with a β|β character. Only one hashed hostname may appear on a
single line and none of the above negation or wildcard operators may be
applied.
The keytype and base64-encoded key are taken directly from the host key;
they can be obtained, for example, from /etc/ssh/ssh_host_rsa_key.pub. The
optional comment field continues to the end of the line, and is not used.
Lines starting with β#β and empty lines are ignored as comments.
When performing host authentication, authentication is accepted if any
matching line has the proper key; [...]
</code></pre></div>
<p>For privacy concerns over a compromised "known hosts" file, the SSH client may store the <em>hash</em> of host names instead of
its actual host name.
I generally don't really like hashed host names as I consider it a disadvantage to be unable to inspect what's in this
"trust store". π
Also, it does not really make sense to obfuscate what hosts you manage over SSH when you also have your inventory of
systems in plain text (your Ansible inventory for example). π </p>
<p>The use of plain text hostnames gives you the following benefits:</p>
<ul>
<li>It's a little easier to manually craft the file, and, a hidden gem...</li>
<li>it automagically gives you tab-completion in your shell for all hosts. π<br/>
<em>Shell completion scripts included in most distributions.</em></li>
</ul>
<p>Given the large size of (secure) RSA type of public keys, I would definitely recommend taking only EC keysβof which the
shortest is an Ed25519 key.
Any modern OpenSSH installation now should be equipped with <code>ssh-ed25519</code> keys.</p>
<div class="admonition hint">
<p class="admonition-title">Question</p>
<p>I'm a bit surprised to see that the <code>known_hosts</code> file only takes full public keys and does not allow you to specify
the much shorter (SHA256) hash of itβas it does in the prompt.
Let me know if you know a reason why this is or if there's any plans out there to include that.</p>
</div>
<h1 id="creating-your-own-openssh-known_hosts-file">Creating your own OpenSSH <code>known_hosts</code> file<a class="headerlink" href="#creating-your-own-openssh-known_hosts-file" title="Permanent link"> </a></h1>
<p>An easy way to get started is using the <code>ssh-keyscan</code> utility.
For example, here's how to create your own "known hosts" file that contains the line for the host on IP
<code>srv1.mydomain.tld</code> and default SSH port 22:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">user@myhost:~$ </span>ssh-keyscan<span style="color: #666666"> </span>-t<span style="color: #666666"> </span>ed25519<span style="color: #666666"> </span>srv1.mydomain.tld<span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>tee<span style="color: #666666"> </span>-a<span style="color: #666666"> </span>~/teamrepos/ansible/files/ssh_known_hosts
</code></pre></div>
<div class="admonition danger">
<p class="admonition-title">No validation with <code>ssh-keyscan</code></p>
<p>This key is obtained without any validation.
You may want to validate it by comparing the contents on the server over another secure channel, remote location
<code>/etc/ssh/ssh_host_ed25519_key.pub</code>.
Do this once for every host, it's an investment for the benefit of security.</p>
</div>
<p>Repeat this for all your hosts and it should look like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>srv1.mydomain.tld ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFsS76WSPEm8JbUTt6hSFs3iVQlNZp4oJYLmCPylr2ry
srv1.mydomain.tld ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFpnbOdxKmTdgmLnW/0lHSOVfQSmS6Ob4+jjKKSzoFe6
</code></pre></div>
<p>You can add aliases or IP addresses separated by commas so that it will work for those as well, e.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>srv1.mydomain.tld,10.1.2.3,[2a05:f480:1400:246:5400:2ff:fe35:52ff] ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFsS76WSPEm8JbUTt6hSFs3iVQlNZp4oJYLmCPylr2ry
</code></pre></div>
<p>I would recommend only setting the fully qualified domain here, as shortened aliases may match more than once which
impacts security and may complicate troubleshooting.
Note that you will have tab-completion for your ease and it's also still possible to create Host aliases in the SSH
configuration (see below).</p>
<h1 id="configure-openssh-client-for-strict-checking">Configure OpenSSH client for strict checking<a class="headerlink" href="#configure-openssh-client-for-strict-checking" title="Permanent link"> </a></h1>
<p>Now let's configure the OpenSSH client for strict checking host keys against this "known hosts" file.</p>
<p>In <code>~/.ssh/config</code> put this, follow the comments inline:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Host *
# This points to the manually managed "known hosts" file.
GlobalKnownHostsFile ~/teamrepos/ansible/files/ssh_known_hosts
# Instead of 'ask', always be strict!
StrictHostKeyChecking yes
# This disables the default behaviour of writing to a local
# ~/.ssh/known_hosts file.
UserKnownHostsFile /dev/null
# I like to disable this setting or else OpenSSH attempts to
# write to the UserKnownHostsFile with an entry based on the IP.
# It would also show a tedious warning on every connection:
# Warning: Permanently added the ED25519 host key for IP address [...]
# However, that would be written to /dev/null... π
CheckHostIP no
</code></pre></div>
<p>In case you already have a local SSH configuration in <code>~/.ssh/config</code>, I'd suggest to add the stanza at the very bottom
of that file.</p>
<p>Now try to use the tab-completion!
It should work with just <code>srv1</code>-<TAB>. π</p>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>System administrators of workstations may consider adding such a configuration also system-wide, in
<code>/etc/ssh/ssh_config</code>.</p>
</div>
<h1 id="caveats-exceptions">Caveats & exceptions<a class="headerlink" href="#caveats-exceptions" title="Permanent link"> </a></h1>
<h2 id="ssh-host-aliases">SSH Host aliases<a class="headerlink" href="#ssh-host-aliases" title="Permanent link"> </a></h2>
<p>In case you use Host aliases for short-hands or connecting over an alternative IP address like below, but you don't like
to add its alias in the "known hosts" file for everyone, the <code>HostKeyAlias</code> option can point to the host key this host
should match with, e.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Host srv1-out-of-band
# Special out of band IP in case internet connection is down!
Hostname 10.99.99.99
<span style="background-color: #404040"> HostKeyAlias srv1.mydomain.tld
</span></code></pre></div>
<h2 id="additional-hosts-eg-github">Additional hosts, e.g. GitHub<a class="headerlink" href="#additional-hosts-eg-github" title="Permanent link"> </a></h2>
<p>In case you need to fetch/clone/pull/push to GitHub over SSH, or any other public server for that matter that you don't
really manage yourself, you may want to add these to a separate "known hosts" file like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">user@myhost:~$ </span>ssh-keyscan<span style="color: #666666"> </span>github.com<span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>tee<span style="color: #666666"> </span>-a<span style="color: #666666"> </span>~/.ssh/known_hosts_personal
</code></pre></div>
<p>And then configure this as additional "known hosts" file like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Host *
# This points to the manually managed "known hosts" file.
<span style="background-color: #404040"> GlobalKnownHostsFile ~/.ssh/known_hosts_personal ~/teamrepos/ansible/files/ssh_known_hosts
</span></code></pre></div>
<h2 id="relaxed-checks-for-local-development">Relaxed checks for local development<a class="headerlink" href="#relaxed-checks-for-local-development" title="Permanent link"> </a></h2>
<p>You can configure a pattern of hosts in OpenSSH client with relaxed checks, or even none.
This avoids having to accept a key every time on automated re-installation of hosts or reprovisioning scripts
regenerating the host key on purpose.</p>
<p>Add something like this to your SSH configuration:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Host *.localdomain
# No host key checks for anything on .localdomain!
StrictHostKeyChecking no
</code></pre></div>
<h1 id="bonus-empower-your-ansible-ssh-connections">Bonus: empower your Ansible-SSH connections<a class="headerlink" href="#bonus-empower-your-ansible-ssh-connections" title="Permanent link"> </a></h1>
<p>With a typical case of an Ansible repository shared with your team, you could do the following:</p>
<ol>
<li>Add the managed "known hosts" file in the repository.<br/>
For example in <code><ansibleroot>/files/ssh_known_hosts</code>.</li>
<li>
<p>In <code><ansibleroot>/ansible.cfg</code> add this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #6ebf26; font-weight: bold">[ssh_connection]</span>
<span style="color: #ababab; font-style: italic"># Default ssh_args if not specified are:</span>
<span style="color: #ababab; font-style: italic"># "-C -o ControlMaster=auto -o ControlPersist=60s"</span>
<span style="color: #ababab; font-style: italic"># Add the SSH configuration inline to use strict host key</span>
<span style="color: #ababab; font-style: italic"># checking against the in-repository ssh_known_hosts file.</span>
<span style="color: #bbbbbb">ssh_args</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ed9d13">-C -o ControlMaster=auto -o ControlPersist=60s</span>
<span style="color: #666666"> </span><span style="color: #bbbbbb">-o UserKnownHostsFile</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">/dev/null</span>
<span style="color: #666666"> </span><span style="color: #bbbbbb">-o GlobalKnownHostsFile</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">./files/ssh_known_hosts</span>
<span style="color: #666666"> </span><span style="color: #bbbbbb">-o StrictHostKeyChecking</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yes</span>
</code></pre></div>
</li>
<li>
<p>Now profit from out-of-the-box secure SSH connections!</p>
</li>
</ol>
<p>Benefits for this setup in a small team include a) that changes to SSH host keys can happen through regular change
processes like version control / code review / merge requests.
b) Everyone will be "in sync" with the actual host keys.
Whenever a new host has been deployed or an existing one has been reinstalled by a team member, you just pull in the
(git) changes.
c) No need for every team member to perform an interactive "dance" with SSH on the commandline to validate and store the
host key locally. β </p>
<h1 id="scaling-up-with-many-more-hosts">Scaling up with many more hosts<a class="headerlink" href="#scaling-up-with-many-more-hosts" title="Permanent link"> </a></h1>
<p>Traditionally, several solutions are widely described and available to scaling up this in a larger organization.
They do come with some pain and effort, though.</p>
<ul>
<li>OpenSSH supports a limited form of X.509 TLS-like 'certificates'.<br/>
By creating an SSH certificate authority you could sign all individual host keys and install the trusted CA public
key on the clients to validate the host keys.
Problems that remain with this: revocation, additional work with private keys to be accessible when (re)installing
hosts and limited support for this in SSH clients.
If you're interested in a solution in this direction, look at
<a href="https://engineering.fb.com/security/scalable-and-secure-access-with-ssh/">this blog by Facebook Engineering</a> with a more specific use case or
<a href="https://roumenpetrov.info/secsh/">this fork by Roumen Petrov</a> of OpenSSH that properly support X.509 v3 with CRL and even
<a href="https://en.wikipedia.org/wiki/OCSP_stapling">OCSP stapling</a>.</li>
<li>Leverage DNSSEC-enabled DNS and publish SSH host key fingerprints with <em>SSHFP</em> records.
This is truly a very elegant olution, but requires a proper DNSSEC deployment and support on all clients.
It also only accounts for accessing hosts with their FQDN, where a plain <code>known_hosts</code> file allows you to add aliases
in any form with tab completion as a bonus.</li>
<li>Centralized deployments with a database like LDAP.
This could be cool, but also fragile.
When this fails, you won't be able to access any machine, and it could be a single point of failure in terms of
security as well.</li>
</ul>systemd-networkd to the rescue for DHCPv6 Prefix Delegation with my ISP2021-10-27T00:00:00+02:002023-04-14T00:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2021-10-27:/systemd-networkd-dhcpv6-pd-configuration.html<p>Magic is done by systemd-networkd to configure IPv6 on your Linux router with DHCPv6 Prefix Delegation from your ISP. The example configuration used is Ziggo in The Netherlands (part of VodafoneZiggo / Liberty Global).</p><div class="admonition warning">
<p class="admonition-title">This article has become outdated</p>
<p>First of all, the systemd-networkd settings have become outdated; some settings have been renamed in newer versions.</p>
<p>Moreover, this setup has broken later in 2022, likely due to firmware changes by Ziggo on their ConnectBox modem.
For some reason, the packets returned do not arrive at my own router anymore after a otherwise successful prefix
delegation.
It's likely that an error has been introduced in the firmware of at least the <i>Compal</i> ConnectBox causing this.
For more information see this forum post on the Dutch forum of the ISP:
<a href="https://community.ziggo.nl/t5/Internet/ConnectBox-heeft-geen-route-naar-mijn-eigen-router-voor-IPv6-via/m-p/1173127">
community.ziggo.nl: ConnectBox heeft geen route [...]
</a>.</p>
<p>A similar setup as described in this article may still be possible by changing the operation mode of your modem to
'bridge mode'.
Support for IPv6 with DHCPv6 prefix delegation in bridge mode has been introduced after this article was published
initially and thus not considered at the time of publication.</p>
<p>In the meantime, Ziggo also enabled support for using your own equipment as modem in 2022.
And so I have acquired a Fritz!Box 6660 Cable modem and may publish a new and up-to-date article on how to configure
a systemd-networkd enabled host as downstream IPv6 router behind it, in the near future.</p>
</div>
<p>This post is an attempt to explain what are the challenges and magic needed for a regular DIY IPv6 router at home.
Most ISPs will require you to use DHCPv6 Prefix Delegation and this poses some challenges.
I will use my own ISP Ziggo and systemd-networkd as the tool here as an example.</p>
<h1 id="ipv6-deployment-at-ziggo-in-the-netherlands">IPv6 deployment at Ziggo in The Netherlands<a class="headerlink" href="#ipv6-deployment-at-ziggo-in-the-netherlands" title="Permanent link"> </a></h1>
<p>My ISP at home is <a href="https://www.ziggo.nl/">Ziggo</a> in The Netherlands (part of VodafoneZiggo / Liberty Global).
The situation on their IPv6 roll-out could be qualified as messy at best, in my opinion.
Many other ISPs follow a similar IPv6 deployment strategy here, though.
My situation is as follows:</p>
<ul>
<li>Full <a href="https://en.wikipedia.org/wiki/IPv6#Dual-stack_IP_implementation">Dual-Stack</a> (IPv4 + IPv6) connection.<br/>
<em>The post should also apply for <a href="https://en.wikipedia.org/wiki/IPv6_transition_mechanism#Dual-Stack_Lite_(DS-Lite)">DS-Lite</a> connections, which I
<a href="https://twitter.com/gertvdijk/status/1124386797096701962">used to have</a>.</em><br/>
After having spent <a href="https://twitter.com/gertvdijk/status/1124386501914243074">some serious effort</a> back in 2019 convincing the customer service to remove
the IPv4-only flag on my account, I was able to get on the IPv6-internet too. ππ€</li>
<li>Using the ISP-provided modem/router (ConnectBox, Compal variant).<br/>
From now on, I'll call this a <em>CPE</em> (<a href="https://en.wikipedia.org/wiki/Customer-premises_equipment">customer-premises equipment</a>).</li>
<li>My own router (a Linux server) connected on a LAN-port of the CPE.</li>
</ul>
<p>While the connection may be Dual-Stack, Ziggo does not offer a configuration to get Dual-Stack <em>to your own router</em>,
sadly.
The CPE can be configured in a bridge setup to get the public IPv4 address on your own router, Ziggo will then disable
IPv6. π€·
The sole option for native IPv6 on your own router is to wire it up behind the CPE with double-NAT (stacked router),
like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>INTERNET
.
. βββββββββ βββββββββββββββ
. β ZIGGO β eth0β β
. β βLAN1ββββββββββββββββββββ‘ my β
.. β CPE βLAN2 β Linux β
......β βLAN3 β server β
COAXβ βLAN4 eth1β & β
β β βββββββββ‘ router β
β "Con- β β β β
β nect- β β βββββββββββββββ
β Box" β β
βββββββββ β
β
β
β
β HOME NETWORK
ββββββββββββββββββ΄βββββββββββββββββ
. . .
. . .
βββββββ . .
β β . .
β β . .
βββββββ ββββββββ .
β β ββββββββ
β β β β
β β β β
ββββββββ β β
ββββββββ
</code></pre></div>
<p>There's literally zero documentation, specification or support on the level of IPv6 service provided by Ziggo at the
time of writing.
Every question asked to support regarding IPv6 is simply directed to their <a href="https://community.ziggo.nl/">'online community'</a>
(discussion forum). π
Most content in this post is based on observations by other customers and 'common' industry standards.</p>
<p>It appears Ziggo provisions a /56 IPv6 prefix to your CPE (undocumented), of which the first /64 is used for devices on
the home LAN.
This first /64 prefix is configured ready out-of-the-box for devices and uses <em>stateless address autoconfiguration</em>
(<a href="https://en.wikipedia.org/wiki/IPv6#Stateless_address_autoconfiguration_(SLAAC)">SLAAC</a>).
The rest of the address space can be delegated to your router using
<a href="https://en.wikipedia.org/wiki/Prefix_delegation#:~:text=DHCPv6%20Prefix%20Delegation%20is%20supported,64%20prefix%20to%20the%20subscriber.">DHCPv6 Prefix Delegation</a>.
Unfortunately, this requirement by Ziggo results in dynamic (global) prefixes in your home network and complicates
things, hence, this blog post.</p>
<p>Actually, all major service providers in The Netherlands use DHCPv6-PD on residential connections.
So, the majority of the post also applies to connections with ISPs like KPN or even premium operators like XS4ALL and
Freedom Internet.</p>
<h1 id="background-on-what-needs-to-happen-on-your-router">Background on what needs to happen on your router<a class="headerlink" href="#background-on-what-needs-to-happen-on-your-router" title="Permanent link"> </a></h1>
<p>What needs to happen, basically, is the negotiation between your own router and the CPE to get your prefix delegated.
This is a DHCPv6 client sending a request with an option set for prefix delegation (IA_PD; Identity Association for
Prefix Delegation, number 25).
Typically, the CPE just relays the DHCP request to the ISP server, and it replies with a prefix together with a
lifetime, and then it relays it back to you.
It's kind of similar to traditional DHCPv4 in terms of assigning you some address for a certain amount of time.</p>
<p>Note that DHCPv6 can be used in both a stateless and stateful way in IPv6 deployments and these should not be confused.
Older devices that perform stateless address autoconfiguration without support for extensions to configure DNS servers
for example, will then use stateless DHCPv6 to obtain this "other information".
In order to use DHCPv6-Prefix Delegation (that includes a lease time for the prefix(es) delegated), it does make sense
that a <em>stateful</em> request β a <em>solicit</em> message in DHCP-speak β is required.</p>
<p>Another important thing to get right is the local routing of any delegated prefix, in both directions.
As this IPv6-deployment does not include any routing protocols, things tend to be implicit and easily omitted.
After completing the delegation 'dance' the ingress routing part is done automatically; packets with the delegated
prefix as destination should reach the host having requested it (based on its <a href="https://en.wikipedia.org/wiki/Link-local_address#IPv6">link-local</a>
address).
So that part should be handled as magic by the CPE.
<strong>You may need to disable the IPv6 firewall of the CPE in its web configuration, though, before you will observe the
traffic on your router's interface.</strong></p>
<div class="admonition danger">
<p class="admonition-title">Danger</p>
<p>Be sure to understand what are the consequenses of having the IPv6 firewall on your CPE disabled.
All devices connected to it will be directly exposed to the internet, as well as devices connected to your own
router if you don't run a firewall there.</p>
</div>
<p>The second part about routing is to have a "return route" back to your CPE.
Coming from an IPv4-world of things where DHCPv4 would hand you a default route, I fell for this caveat big time:
DHCPv6-PD does not cover handing the requester a return route (gateway) for the prefix(es) delegated.
It appears common to just (re-)use the default route obtained for the first /64 global prefix via SLAAC.
Setting up SLAAC next to the prefix delegation seems redundant, though.
Technically it could have been sufficient to not configure any global address for your router on the CPE-LAN (in that
single /64).
But oh well, there's no harm in it, and it does the job: providing me a default route that works.</p>
<p>What's most tricky, is the step <em>after</em> having a (new) prefix assigned: it needs to be configured on all your
'downstream' interfaces.
That means it has to 'follow' whatever comes from the service provider and start advertising a /64 from it, for each
downstream interface it is operating as a router for.
It has to make sure the lifetimes advertised have to match the lease time, but also, on a prefix change, it has to send
unsolicited Router Advertisements to invalidate the prefix advertised earlier.
How is one going to deal with that? π€</p>
<h1 id="configuration-challenges-of-dhcpv6-pd-on-linux">Configuration challenges of DHCPv6-PD on Linux<a class="headerlink" href="#configuration-challenges-of-dhcpv6-pd-on-linux" title="Permanent link"> </a></h1>
<p>To do all this people share all kinds of complex setups with scripts parsing the response of the DHCPv6 client to assign
the address to the downstream interfaces.
One of these examples is this on the Debian wiki page <a href="https://wiki.debian.org/IPv6PrefixDelegation#Assigning_the_prefix_to_another_interface">IPv6PrefixDelegation</a>.
Yuck.
Then you may have to do some kung-fu with radvd to have it pick up any new global prefix and correct lifetime (and yes
this had an error <a href="https://github.com/radvd-project/radvd/commit/fdacad4a5e76c90022348ec38b85471662bb28cb">now fixed</a> on master branch still unreleased at the time of writing,
sigh!).</p>
<p>Commercially available routers for (smaller) enterprises have all the convenience functionality built-in.
It comes down to configuration of a dynamic pool of prefixes you can refer to in the configuration of 'downstream'
interfaces.
However, I'm just using a Linux PC as router here β a more elegant way to handle this in software <em>should</em> exist, right?</p>
<h2 id="meet-systemd-networkd">Meet systemd-networkd<a class="headerlink" href="#meet-systemd-networkd" title="Permanent link"> </a></h2>
<p>In order to get things running smooth and ready for dynamic changes on my router, I chose systemd-networkd to operate
this.
Why systemd-networkd?
It has all the functionality built-in for such a situation.
With only a single tool to configure...:</p>
<ul>
<li>systemd-networkd can request prefixes to be delegated using DHCPv6-PD. β</li>
<li>systemd-networkd will maintain a local pool of prefixes it was delegated. πΎ</li>
<li>systemd-networkd can assign prefixes (/64) from this pool to networks it manages dynamically. π€</li>
<li>systemd-networkd will also 'announce' the current up-to-date routable prefix assigned via
<a href="https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol#Functions">Router Advertisements</a> to hosts in a 'downstream' network. π</li>
</ul>
<p>Just perfect! π</p>
<h2 id="the-systemd-networkd-configuration">The systemd-networkd configuration<a class="headerlink" href="#the-systemd-networkd-configuration" title="Permanent link"> </a></h2>
<p>The <code>.network</code> file for the connection to the CPE, comments inline:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #6ebf26; font-weight: bold">[Match]</span>
<span style="color: #bbbbbb">Name</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">eth0</span>
<span style="color: #6ebf26; font-weight: bold">[Network]</span>
<span style="color: #bbbbbb">Description</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">Link to Ziggo CPE LAN port</span>
<span style="color: #ababab; font-style: italic"># We only need a link-local address for IPv6 (required to run IPv6),</span>
<span style="color: #ababab; font-style: italic"># but not for IPv4 (using DHCP).</span>
<span style="color: #bbbbbb">LinkLocalAddressing</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">ipv6</span>
<span style="color: #ababab; font-style: italic"># SLAAC IPv6 for obtaining the default route.</span>
<span style="color: #ababab; font-style: italic"># This is needed, because the DHCPv6 response does not include an</span>
<span style="color: #ababab; font-style: italic"># address with a gateway for the prefixes. We're supposed to use</span>
<span style="color: #ababab; font-style: italic"># the SLAAC-announced default route one as the 'return route'.</span>
<span style="color: #bbbbbb">IPv6AcceptRA</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yes</span>
<span style="color: #ababab; font-style: italic"># Boolean true enables both DHCPv4 and DHCPv6 client. See also the</span>
<span style="color: #ababab; font-style: italic"># IPv6AcceptRA.DHCPv6Client setting.</span>
<span style="color: #bbbbbb">DHCP</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yes</span>
<span style="color: #6ebf26; font-weight: bold">[DHCPv4]</span>
<span style="color: #ababab; font-style: italic"># Hostname sent in the DHCPv4 solicit (request). I like to set this to</span>
<span style="color: #ababab; font-style: italic"># some bogus name. π </span>
<span style="color: #bbbbbb">Hostname</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">myhostname</span>
<span style="color: #ababab; font-style: italic"># Ignore the hostname to set in the reply.</span>
<span style="color: #ababab; font-style: italic"># I don't want anything to control my server's hostname! π€¨</span>
<span style="color: #bbbbbb">UseHostname</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #ababab; font-style: italic"># I handle all this myself, not using Ziggo's DNS/NTP servers</span>
<span style="color: #ababab; font-style: italic"># mentioned in the reply.</span>
<span style="color: #ababab; font-style: italic"># The only thing I need for IPv4 to work is an address and a gateway.</span>
<span style="color: #bbbbbb">UseDNS</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">UseNTP</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">UseSIP</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">UseRoutes</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">UseGateway</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yes</span>
<span style="color: #6ebf26; font-weight: bold">[IPv6AcceptRA]</span>
<span style="color: #ababab; font-style: italic"># Similar for IPv6 via Router Advertisements; I'll handle DNS myself,</span>
<span style="color: #ababab; font-style: italic"># please.</span>
<span style="color: #bbbbbb">UseDNS</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #ababab; font-style: italic"># Force starting the DHCPv6 client even if the Router Advertisement</span>
<span style="color: #ababab; font-style: italic"># indicates it's not required.</span>
<span style="color: #bbbbbb">DHCPv6Client</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yes</span>
<span style="color: #6ebf26; font-weight: bold">[DHCPv6]</span>
<span style="color: #ababab; font-style: italic"># The Ziggo CPE does not advertise DHCPv6 stateful mode availability</span>
<span style="color: #ababab; font-style: italic"># in the Router Advertisements. Without this override, the DHCPv6</span>
<span style="color: #ababab; font-style: italic"># solicit (request) won't result in a reply with a prefix (IA_PD).</span>
<span style="color: #bbbbbb">ForceDHCPv6PDOtherInformation</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yes</span>
<span style="color: #ababab; font-style: italic"># Similar as for DHCPv4, I dislike systemd-networkd to use any other</span>
<span style="color: #ababab; font-style: italic"># information in the reply.</span>
<span style="color: #bbbbbb">UseHostname</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">UseDNS</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">UseNTP</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
</code></pre></div>
<p>The <code>.network</code> file for my client network (leaving the IPv4 DHCPv4 server out of scope here, I may move to
systemd-networkd for that as well):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #6ebf26; font-weight: bold">[Match]</span>
<span style="color: #bbbbbb">Name</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">eth1</span>
<span style="color: #6ebf26; font-weight: bold">[Network]</span>
<span style="color: #bbbbbb">Description</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">Link to the LAN for my clients</span>
<span style="color: #bbbbbb">LinkLocalAddressing</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">ipv6</span>
<span style="color: #bbbbbb">IPv6AcceptRA</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #ababab; font-style: italic"># Announce a prefix here and act as a router.</span>
<span style="color: #bbbbbb">IPv6SendRA</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yes</span>
<span style="color: #ababab; font-style: italic"># Use a DHCPv6-PD delegated prefix (DHCPv6PrefixDelegation.SubnetId)</span>
<span style="color: #ababab; font-style: italic"># from the pool and assigns one /64 to this network.</span>
<span style="color: #bbbbbb">DHCPv6PrefixDelegation</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yes</span>
<span style="color: #6ebf26; font-weight: bold">[Address]</span>
<span style="color: #ababab; font-style: italic"># Simple static IPv4 configuration.</span>
<span style="color: #bbbbbb">Address</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">10.3.2.1/24</span>
<span style="color: #6ebf26; font-weight: bold">[IPv6SendRA]</span>
<span style="color: #ababab; font-style: italic"># Currently my DHCPv4 server configures a DNS server already.</span>
<span style="color: #bbbbbb">EmitDNS</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">EmitDomains</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #6ebf26; font-weight: bold">[DHCPv6PrefixDelegation]</span>
<span style="color: #ababab; font-style: italic"># This assigns the second prefix from the pool.</span>
<span style="color: #bbbbbb">SubnetId</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">0x1</span>
</code></pre></div>
<p>With this configuration in place, restart systemd-networkd to activate it:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">root@myrouter:~# </span>systemctl<span style="color: #666666"> </span>restart<span style="color: #666666"> </span>systemd-networkd
</code></pre></div>
<p>Now, let's have a look at the logs:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>root@myrouter:~# journalctl -u systemd-networkd.service
[...]
systemd-networkd[1337]: eth0: DHCP6: received PD Prefix 2001:1c00:1234:ff50::/60
systemd-networkd[1337]: eth1: DHCPv6-PD address 2001:1c00:1234:ff51:36e6:d7ff:fe1b:48fe/64 (valid for 16h, preferred for 8h)
</code></pre></div>
<p><em>The actual addresses have been obfuscated.</em></p>
<p>It appears that the CPE or the ISP DHCP server has decided to delegate a /60 to me (for 16 /64-subnets).
A more visual view on what I've got:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>sipcalc<span style="color: #666666"> </span><span style="color: #51b2fd">2001</span>:1c00:1234:ff50::/60
<span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">Network range - 2001:1c00:1234:ff50:0000:0000:0000:0000 -</span>
<span style="color: #cccccc"> 2001:1c00:1234:ff5f:ffff:ffff:ffff:ffff</span>
</code></pre></div>
<p>Then systemd-networkd successfully assigned the second subnet (ID 1, <code>ff51</code>) to my eth1 interface and the hosts in this
network picked up global addresses from this prefix! π₯³</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">root@myrouter:~# </span>ip<span style="color: #666666"> </span>addr<span style="color: #666666"> </span>show<span style="color: #666666"> </span>eth0
<span style="color: #cccccc">1: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000</span>
<span style="color: #cccccc"> link/ether [...]</span>
<span style="color: #cccccc"> inet 192.168.178.115/24 metric 1024 brd 192.168.178.255 scope global dynamic eth0</span>
<span style="color: #cccccc"> valid_lft 2509sec preferred_lft 2509sec</span>
<span style="background-color: #404040"><span style="color: #cccccc"> inet6 2001:1c00:1234:ff00:629d:30c1:4adc:57e6/64 scope global dynamic mngtmpaddr noprefixroute</span>
</span><span style="color: #cccccc"> valid_lft 45277sec preferred_lft 16477sec</span>
<span style="color: #cccccc"> inet6 fe80::629d:30c1:4adc:57e6/64 scope link</span>
<span style="color: #cccccc"> valid_lft forever preferred_lft forever</span>
<span style="color: #aaaaaa">root@myrouter:~# </span>ip<span style="color: #666666"> </span>addr<span style="color: #666666"> </span>show<span style="color: #666666"> </span>eth1
<span style="color: #cccccc">2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000</span>
<span style="color: #cccccc"> link/ether [...]</span>
<span style="color: #cccccc"> inet 10.50.0.1/24 brd 10.50.0.255 scope global eth1</span>
<span style="color: #cccccc"> valid_lft forever preferred_lft forever</span>
<span style="background-color: #404040"><span style="color: #cccccc"> inet6 2001:1c00:1234:ff51:629d:30c1:4adc:57e6/64 scope global dynamic mngtmpaddr</span>
</span><span style="color: #cccccc"> valid_lft 45882sec preferred_lft 17082sec</span>
<span style="color: #cccccc"> inet6 fe80::629d:30c1:4adc:57e6/64 scope link</span>
<span style="color: #cccccc"> valid_lft forever preferred_lft forever</span>
<span style="color: #aaaaaa">root@myrouter:~# </span>ip<span style="color: #666666"> </span>-6<span style="color: #666666"> </span>route
<span style="color: #cccccc">::1 dev lo proto kernel metric 256 pref medium</span>
<span style="color: #cccccc">2001:1c00:1234:ff00::/64 dev eth0 proto ra metric 1024 expires 44796sec pref medium</span>
<span style="color: #cccccc">2001:1c00:1234:ff00::/64 via fe80::7ffb:b602:2423:a1a6 dev eth0 proto ra metric 1024 expires 57527sec pref medium</span>
<span style="color: #cccccc">2001:1c00:1234:ff51::/64 dev eth1 proto kernel metric 256 expires 45401sec pref medium</span>
<span style="color: #cccccc">2001:1c00:1234:ff51::/64 dev eth1 proto dhcp metric 1024 pref medium</span>
<span style="color: #cccccc">unreachable 2001:1c00:1234:ff50::/60 dev lo proto dhcp metric 1024 pref medium</span>
<span style="color: #cccccc">fe80::/64 dev eth1 proto kernel metric 256 pref medium</span>
<span style="color: #cccccc">fe80::/64 dev eth0 proto kernel metric 256 pref medium</span>
<span style="background-color: #404040"><span style="color: #cccccc">default via fe80::7ffb:b602:2423:a1a6 dev eth0 proto ra metric 1024 expires 1727sec mtu 1500 pref high</span>
</span></code></pre></div>
<p>I'm now enjoying full Dual Stack future-proof modern internet! π₯³</p>
<p><img alt="IPv6 test site https://ipv6-test.com/" src="https://blog.g3rt.nl/images/20211027_ipv6test2.png" style="width: 46%;"/>
<img alt="IPv6 test site https://internet.nl/" src="https://blog.g3rt.nl/images/20211027_ipv6test1.png" style="float: right; width: 46%;"/></p>
<h1 id="results-and-further-thoughts">Results and further thoughts<a class="headerlink" href="#results-and-further-thoughts" title="Permanent link"> </a></h1>
<p>Just a /60 is a smaller space (longer prefix) than I expected, but okay. π
Quite possibly, Ziggo does this on purpose to leave room for more routers receiving different /60 prefixes delegated
from the /56 pool available per subscription.
It would be a bit of a challenge in case you need more than 16 subnets, though.
Others have <a href="https://community.ziggo.nl/t5/Internet/ConnectBox-heeft-geen-route-naar-eigen-router-met-IPv6-via/m-p/858078/highlight/true#M91493">reported</a> different prefix sizes with Ziggo β told you; it's messy.
Either way, just a /60 for your own router is very greedy given the IPv6 standards and the
<a href="https://www.ripe.net/publications/docs/ripe-690#4-2-1---48-for-everybody">recommendation by RIPE</a> to hand everyone a /48.</p>
<p>I've been quite happy to just use a single tool β systemd-networkd β for this complex job.
Also, I like the fact of having a declarative way to configure my network.
With just a few files and a service restart it is able to bring the network up in the desired state and keep it that
way.</p>
<p>To me it seems DHCPv6-PD is more of a convenience to the ISP's IPv6 deployment than it is for end-users to configure on
their own routers.
Ideally, I would have liked to be able to choose an ISP that statically routes the whole prefix to my connection over
plain Ethernet and a link-local address, or SLAAC.
None of them seem to do so here.
If any of you have more thoughts about why, let me know, I'd like to learn about their reasoning.</p>
<h1 id="configuration-for-other-isps">Configuration for other ISPs<a class="headerlink" href="#configuration-for-other-isps" title="Permanent link"> </a></h1>
<p>While my setup with Ziggo should be similar to many other configurations with DHCPv6-PD involved (aside differently
sized prefixes), there's one complicating factor for many other: PPPoE.
Many residential connections on DSL and FTTH in The Netherlands are connected via access brokers using PPP.</p>
<p>The bad news here: systemd-networkd does not support PPP/PPPoE natively, yet (see RFE <a href="https://github.com/systemd/systemd/issues/481">#481</a>).
π’</p>How to do VLANs with systemd-networkd on Linux2021-10-24T23:45:00+02:002021-10-24T23:45:00+02:00Gert van Dijktag:blog.g3rt.nl,2021-10-24:/systemd-networkd-vlans-howto.html<p>How to configure VLANs with systemd-networkd on Linux hosts is not very straightforward. Here's a short how-to on getting started.</p>
<h1 id="vlans-the-systemd-networkd-way">VLANs the systemd-networkd-way<a class="headerlink" href="#vlans-the-systemd-networkd-way" title="Permanent link"> </a></h1>
<p>I have spent way too much time to configure VLAN ID X on underlying ethernet device Y with systemd-networkd.
The documentation appears to be lacking a full example, so I decided to write a blog post after I finally succeeded.</p>
<p>To me the systemd-networkd-way seems a bit complex with quite some redundancy. π
The benefit for using it, however, is the declarative form and many features it offers which would take quite some
scripting effort with plain 'ifupdown'. Those features are out of scope for this post, though.</p>
<h2 id="register-the-vlan-ids-as-a-netdev">Register the VLAN IDs as a 'netdev'<a class="headerlink" href="#register-the-vlan-ids-as-a-netdev" title="Permanent link"> </a></h2>
<p>The first step is to just register the VLAN ID β yes, just its number β as network device (<a href="https://www.freedesktop.org/software/systemd/man/systemd.netdev.html">netdev</a>).
Despite it being named a 'dev' configured in a <code>NetDev</code> section in a <code>.netdev</code> file, it does not actually add a
(virtual) device <em>yet</em>. π</p>
<p>Example (in e.g. <code>/etc/systemd/network/00-myvlan.netdev</code>)</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #6ebf26; font-weight: bold">[NetDev]</span>
<span style="color: #bbbbbb">Name</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">myvlan</span>
<span style="color: #bbbbbb">Kind</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">vlan</span>
<span style="color: #6ebf26; font-weight: bold">[VLAN]</span>
<span style="color: #bbbbbb">Id</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">123</span>
</code></pre></div>
<p>Repeat this for as many VLANs that you need, each in a separate file.
I did this for VLAN IDs 1234 and 1337 too.</p>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>The name of the file does not really matter; the files in <code>/etc/systemd/network/</code> are being processed in lexicographical
order and the VLANs are matched by the name in the NetDev.Name setting.</p>
</div>
<h2 id="assign-it-to-an-underlying-ethernet-device">Assign it to an underlying ethernet device<a class="headerlink" href="#assign-it-to-an-underlying-ethernet-device" title="Permanent link"> </a></h2>
<p>The second step is making the underlying ethernet device a member of VLAN(s) in a <a href="https://www.freedesktop.org/software/systemd/man/systemd.network.html">'network' file</a>.
This will create virtual interfaces for each VLAN on all matched devices.</p>
<p>By example (in e.g. <code>/etc/systemd/network/10-myethernet.network</code>):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #6ebf26; font-weight: bold">[Match]</span>
<span style="color: #bbbbbb">Name</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">eth1</span>
<span style="color: #bbbbbb">Type</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">ether</span>
<span style="color: #6ebf26; font-weight: bold">[Network]</span>
<span style="color: #bbbbbb">Description</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">The unconfigured physical ethernet device</span>
<span style="color: #ababab; font-style: italic"># Make eth1 member of these three VLANs and create virtual</span>
<span style="color: #ababab; font-style: italic"># interfaces on it:</span>
<span style="color: #bbbbbb">VLAN</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">myvlan</span>
<span style="color: #bbbbbb">VLAN</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">othervlan</span>
<span style="color: #bbbbbb">VLAN</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">yetanother</span><span style="color: #666666"> </span>
<span style="color: #ababab; font-style: italic"># In case of 'tagged only' setups, you probably don't need any IP</span>
<span style="color: #ababab; font-style: italic"># configuration on the link without VLAN (or: default VLAN).</span>
<span style="color: #ababab; font-style: italic"># For that just omit an [Address] section and disable all the</span>
<span style="color: #ababab; font-style: italic"># autoconfiguration magic like this:</span>
<span style="color: #bbbbbb">LinkLocalAddressing</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">LLDP</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">EmitLLDP</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">IPv6AcceptRA</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
<span style="color: #bbbbbb">IPv6SendRA</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">no</span>
</code></pre></div>
<p>I expected this to happen in a <a href="https://www.freedesktop.org/software/systemd/man/systemd.link.html">link-configuration file</a> at first.</p>
<h2 id="configure-the-vlan-interface-with-another-network-file">Configure the VLAN interface with another network file<a class="headerlink" href="#configure-the-vlan-interface-with-another-network-file" title="Permanent link"> </a></h2>
<p>Final configuration step three is the configuration of the (virtual) interface for the VLAN in another 'network' file
matching the VLAN. </p>
<p>By example (in e.g. <code>/etc/systemd/network/20-myvlan.network</code>):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #6ebf26; font-weight: bold">[Match]</span>
<span style="color: #bbbbbb">Name</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">myvlan</span>
<span style="color: #bbbbbb">Type</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">vlan</span>
<span style="color: #6ebf26; font-weight: bold">[Network]</span>
<span style="color: #bbbbbb">Description</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">The interface for myvlan</span>
<span style="color: #ababab; font-style: italic"># Very simple static IPv4-only address configuration.</span>
<span style="color: #ababab; font-style: italic"># Well, an IPv6-link-local address as well, by default.</span>
<span style="color: #6ebf26; font-weight: bold">[Address]</span>
<span style="color: #bbbbbb">Address</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">10.1.2.3/24</span>
</code></pre></div>
<h2 id="apply-the-configuration-and-inspect">Apply the configuration and inspect<a class="headerlink" href="#apply-the-configuration-and-inspect" title="Permanent link"> </a></h2>
<p>Now that the configuration is done, restart the 'systemd-networkd' service to apply it and inspect the result.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>systemctl<span style="color: #666666"> </span>restart<span style="color: #666666"> </span>systemd-networkd
</code></pre></div>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">root@myhost:~# </span>networkctl<span style="color: #666666"> </span>list
<span style="color: #cccccc">IDX LINK TYPE OPERATIONAL SETUP </span>
<span style="color: #cccccc"> 1 lo loopback routable configured</span>
<span style="color: #cccccc"> 2 eth1 ether carrier configured</span>
<span style="color: #cccccc"> 3 wlp2s0 wlan off unmanaged</span>
<span style="color: #cccccc"> 4 myvlan vlan routable configured</span>
<span style="color: #cccccc"> 5 othervlan vlan routable configured</span>
<span style="color: #cccccc"> 6 yetanother vlan routable configured</span>
</code></pre></div>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">root@myhost:~# </span>cat<span style="color: #666666"> </span>/proc/net/vlan/config<span style="color: #666666"> </span>
<span style="color: #cccccc">VLAN Dev name | VLAN ID</span>
<span style="color: #cccccc">Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD</span>
<span style="color: #cccccc">myvlan | 123 | eth1</span>
<span style="color: #cccccc">othervlan | 1234 | eth1</span>
<span style="color: #cccccc">yetanother | 1337 | eth1</span>
</code></pre></div>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">root@myhost:~# </span>ip<span style="color: #666666"> </span>addr<span style="color: #666666"> </span>show<span style="color: #666666"> </span>dev<span style="color: #666666"> </span>eth1<span style="color: #666666"> </span>
<span style="color: #cccccc">2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000</span>
<span style="color: #cccccc"> link/ether [...]</span>
<span style="color: #aaaaaa">root@myhost:~# </span>ip<span style="color: #666666"> </span>addr<span style="color: #666666"> </span>show<span style="color: #666666"> </span>dev<span style="color: #666666"> </span>myvlan
<span style="color: #cccccc">4: myvlan@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000</span>
<span style="color: #cccccc"> link/ether [...]</span>
<span style="color: #cccccc"> inet 10.1.2.3/24 brd 10.1.2.255 scope global myvlan</span>
<span style="color: #cccccc"> valid_lft forever preferred_lft forever</span>
<span style="color: #cccccc"> inet6 fe80::fd5b:7cd4:d4c6:f123/64 scope link </span>
<span style="color: #cccccc"> valid_lft forever preferred_lft forever</span>
</code></pre></div>
<h1 id="vlans-the-old-school-way">VLANs the old-school way<a class="headerlink" href="#vlans-the-old-school-way" title="Permanent link"> </a></h1>
<p>Adding a <code>.123</code> to the interface name would already configure VLAN 123 on the interface before, e.g. <code>eth1.123</code>.</p>
<p>Alternatively, a name with <code>vlan123</code> would configure it using <code>/etc/network/interfaces</code> like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code># myvlan
auto vlan123
iface vlan123 inet static
vlan-raw-device eth1
address 10.1.2.3/24
</code></pre></div>
<p>Very short compared to systemd-networkd π¬, but also quite implicit.</p>Misleading news on false positives with PCR tests (COVID-19)2020-07-24T17:30:00+02:002020-07-24T17:30:00+02:00Gert van Dijktag:blog.g3rt.nl,2020-07-24:/misleading-news-false-positives-pcr-tests-covid-19.html<p>PCR testing in general is very accurate and also for the novel coronavirus this is the case. Every now and then I see articles with conclusions the false positive rates are high for low prevalence areas and we should not trust them. However, that seems false information. In this article I will explain why.</p>
<p><em>This is an experiment to publish both an article and a Twitter thread. The individual tweets are numbered and linked after each paragraph/section.</em></p>
<h1 id="a-recent-misleading-article">A recent misleading article<a class="headerlink" href="#a-recent-misleading-article" title="Permanent link"> </a></h1>
<p>I believe <a href="https://www.spectator.co.uk/article/how-many-covid-diagnoses-are-false-positives-">this article</a> on false positives with PCR tests is misleading.
It seems to only apply simple math to a problem with some wrong assumptions.
It baffles me, as Carl Heneghan is a medical professor and I'm not, yet I spot some obvious errors. π <a href="https://twitter.com/gertvdijk/status/1286685155567050752">π¦1</a></p>
<h1 id="specificity-sensitivity-with-multi-target-tests">Specificity & Sensitivity with multi-target tests<a class="headerlink" href="#specificity-sensitivity-with-multi-target-tests" title="Permanent link"> </a></h1>
<p>Most important of all, what seems to be lacking in the article is the huge difference in <em>specificity</em> when it comes to the number of targets testing for.
PCR aims for finding a certain sequence in the genetic material (gene, the target). <a href="https://twitter.com/gertvdijk/status/1286685158171652097">π¦2</a></p>
<p>If we would be testing for a single target, this would result in quite a lot false positivesβin the order of magnitude ~ 1-3 per cent.
Reasons include a gene that's not specific to SARS-CoV-2 or a tiny viral fragment that somehow ended up in the sample as noise/contamination. <a href="https://twitter.com/gertvdijk/status/1286685160726044673">π¦3</a></p>
<p><img alt="excerpt from German PCR testing paper HCoV positive" src="https://blog.g3rt.nl/images/20200724_german_paper_hcov.png" style="float:right; margin-left: 3em; height: 100px;"/></p>
<p><img alt="excerpt from German PCR testing paper known-negative" src="https://blog.g3rt.nl/images/20200724_german_paper_neg.png" style="float:right; margin-left: 3em; height: 100px;"/></p>
<p><a href="https://www.instand-ev.de/aktuelles/detail/news/extra-instand-ringversuch-340-virusgenom-nachweis-sars-cov-2-april-2020-teilnahmedokumente-sin/">A German test report on PCR tests</a> shows the characteristics on different genes/targets. It was used by Dutch skeptics as proof that PCR tests are unreliable, but in fact it shows a completely different result. Some picks shown on the right. <a href="https://twitter.com/gertvdijk/status/1286685162638540800">π¦4</a></p>
<p>A single sample run with multiple targets is basically running separate tests. More than one 'test error' to occur is mathematically very unlikely to happen as well as a false positive to an error in specificity.</p>
<p>Running with multiple targets leads to a much lower false <em>positive</em> rate at the expense of an increased false <em>negative</em> rate. <a href="https://twitter.com/gertvdijk/status/1286685165339713537">π¦5</a></p>
<p>This should lead to a much lower false <em>positive</em> rate at the expense of an increased false <em>negative</em> rate. <a href="https://twitter.com/gertvdijk/status/1286685167432740864">π¦6</a></p>
<p>Here's an example calculation with testing for both E and N gene. Assuming respectively 99.5% and 98.2% correctness on a known-negative sample (from that German paper). This means 0.5% multiplied by 1.8% = 0.009% chance that both genes show (false) positive. <a href="https://twitter.com/gertvdijk/status/1286685169601183744">π¦7</a></p>
<p><img alt="excerpt from WHO guidance document" src="https://blog.g3rt.nl/images/20200724_who_pcr_testing_prevalence.png" style="float:right; margin-left: 3em; width: 300px;"/></p>
<p>Of course, single-target testing is probably cheaper and faster, but should never happen in low-prevalence areas for the reasons described in the Spectator article. This is just part of the guidance/procedures in testing! <a href="https://apps.who.int/iris/handle/10665/331501">WHO guidance document from March</a>. <a href="https://twitter.com/gertvdijk/status/1286685171643711488">π¦8</a></p>
<h1 id="pcr-test-results-arent-binary">PCR test results aren't binary<a class="headerlink" href="#pcr-test-results-arent-binary" title="Permanent link"> </a></h1>
<p>Also please don't mistake PCR testing for a test with a binary outcome. The testing machines produce a response curve over time of amplification. Sometimes these curves aren't proving a very clear outcome and a human should interpret the raw results. <a href="https://twitter.com/gertvdijk/status/1286685174449819648">π¦9</a></p>
<p>In case of doubt, they may supply additional notes with the diagnostics outcome. This really happens. For example, this case in The Netherlands. A sample of people at a meat factory mostly indicated curves that fit the "found a case too late" picture. <a href="https://www.bd.nl/meierij/steekproef-vion-in-boxtel-een-op-drie-medewerkers-besmet-met-coronavirus~a58a3417/">BD: Steekproef Vion in Boxtel</a> <a href="https://twitter.com/gertvdijk/status/1286685176525983745">π¦10</a></p>
<p>That nuance can be part of the test result! With this notes on the results, the persons were informed that they are likely not infectious any longer. I think that's a great result and not to be considered a false positive. <a href="https://twitter.com/gertvdijk/status/1286685178904170497">π¦11</a></p>
<p>It also demonstrates that if the curves are showing inconclusive, the test result can come back as inconclusive as well. Perhaps you will be asked to test again. <a href="https://twitter.com/gertvdijk/status/1286685181408161792">π¦12</a></p>
<h1 id="reality-in-mass-testing">Reality in mass testing<a class="headerlink" href="#reality-in-mass-testing" title="Permanent link"> </a></h1>
<p>Now, here's a well-documented example that proves the point from the reality. <a href="https://www.voanews.com/covid-19-pandemic/chinas-wuhan-tests-11-million-covid-19">11 million people were mass-tested in China</a>. How many positives? 206. <em>Even if</em> zero were infected, it's only 0,00187% false positive and professor Heneghan is off by about factor 50. <a href="https://twitter.com/gertvdijk/status/1286685183769546752">π¦13</a></p>
<p>Then, finally, it's typically <em>not</em> a random group of people being tested, so the factor of 50 is likely <em>even bigger</em>. <a href="https://twitter.com/regordane/status/1285472304496082944">Valid point by @regordane</a>. <a href="https://twitter.com/gertvdijk/status/1286685185917046784">π¦14</a></p>
<h1 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link"> </a></h1>
<p>To summarize: I think given these examples, the article by Carl is misleading. In practice health authorities and labs are perfectly aware of the shortcomings of PCR and act accordingly to avoid the pitfall he's describing. π€· <a href="https://twitter.com/gertvdijk/status/1286685188752384001">π¦15</a></p>
<p>Applying <em>just</em> math to a binary problem that's not binary to begin with, ignoring multi-target testing in the probability calculation, is plain wrong in my opinion. <a href="https://twitter.com/gertvdijk/status/1286685191407308801">π¦16</a></p>
<p><em>Comments on this blog post are disabled; please use Twitter to reply with your commentsβthanks.</em></p>How to create a backport of a package2017-08-08T00:00:00+02:002017-08-08T00:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2017-08-08:/create-backport-package-ppa.html<p>This is how to create a Debian/Ubuntu package from a distribution release newer than your current one (backport) and optionally publish it in your Launchpad PPA.</p>
<h2 id="situation">Situation<a class="headerlink" href="#situation" title="Permanent link"> </a></h2>
<p>Suppose you are running Ubuntu LTS or Debian (Old)Stable, but you want the newer version of a package that's available in a newer distribution release (e.g. Ubuntu non-LTS or Debian testing).
Then this how-to is for you.</p>
<p>This type of backport will be a no-change-of-sources rebuild.
While it's not always trivial, it turns out to be working quite well usually.</p>
<p>First of all, you might want to check if the package is already backported for you in Debian/Ubuntu.
Debian offers quite some backports already.
For example in stretch-backports, at the timme of writing there are <a href="https://packages.debian.org/stretch-backports/allpackages">more than 1400 packages available</a>.</p>
<h2 id="requirements">Requirements<a class="headerlink" href="#requirements" title="Permanent link"> </a></h2>
<ul>
<li>Launchpad account.</li>
<li>An SSH key.</li>
<li>A GPG key. Not covered in this post.</li>
</ul>
<h2 id="example-in-this-blog-with-ccid">Example in this blog with CCID<a class="headerlink" href="#example-in-this-blog-with-ccid" title="Permanent link"> </a></h2>
<p>For this article I'll be using the <a href="https://pcsclite.alioth.debian.org/ccid.html">CCID library</a>, a piece of software for using smartcard readers.
At the time of writing, Debian testing has <a href="https://packages.debian.org/testing/libccid">version 1.4.27</a> and Ubuntu Xenial (16.04) has <a href="https://packages.ubuntu.com/xenial-updates/libccid">version 1.4.22</a> and there's <a href="https://packages.ubuntu.com/xenial-backports/libccid">no newer package available in xenial-backports</a>.</p>
<p>At the time of writing Ubuntu Zesty (17.04) is the next non-LTS stable distribution release.
This distribution offers <a href="https://packages.ubuntu.com/zesty/libccid">version 1.4.26</a>.
That's nice, but we want the newer version for improved hardware support in our stable software stack rather than upgrading to a newer Ubuntu version just for this.
That's what we're going to do here.</p>
<p>In this full example I'll use Ubuntu Xenial as local system to download/build the backport and upload the sources to Launchpad to get them built for many Ubuntu distributions.</p>
<h2 id="install-dependencies">Install Dependencies<a class="headerlink" href="#install-dependencies" title="Permanent link"> </a></h2>
<p>In order to verify the downloaded sources, I'll advice to download the GPG keys of the Debian maintainers from a trusted source.
Ubuntu has packaged them nicely for you in the <code>debian-keyring</code> package.
Apart from that we need <code>devscripts</code> for the general package tooling.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>apt-get<span style="color: #666666"> </span>install<span style="color: #666666"> </span>debian-keyring<span style="color: #666666"> </span>devscripts
</code></pre></div>
<p>You don't need build dependencies here if you want to have Launchpad build your package in the cloud.</p>
<p>In case you want to build it on your local machine, you can install it with <code>sudo apt-get build-dep <package></code> if present already in your current distribution.
Please note that if new requirements are required for new releases these are not included yet since your APT installation doesn't know that.</p>
<h2 id="configure-devscripts">Configure devscripts<a class="headerlink" href="#configure-devscripts" title="Permanent link"> </a></h2>
<p>Configure the keyring to use the installed Debian developers keyring.
For example, in <code>~/.devscripts</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>DSCVERIFY_KEYRINGS="/etc/apt/trusted.gpg:/usr/share/keyrings/debian-maintainers.gpg:~/.gnupg/pubring.gpg"
</code></pre></div>
<p>The syntax is like <code>PATH</code>, colon-separated.
In the above example it's a 'sane' list of keyrings on your system that you probably want to check against.</p>
<p>Set your username/email, e.g.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span><span style="color: #2fbccd">export</span><span style="color: #666666"> </span><span style="color: #40ffff">DEBEMAIL</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"gertvdijk@gmail.com"</span>
<span style="color: #aaaaaa">$ </span><span style="color: #2fbccd">export</span><span style="color: #666666"> </span><span style="color: #40ffff">DEBFULLNAME</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"Gert van Dijk"</span>
</code></pre></div>
<p>Or, as a permanent solution to the above, put them in your environment variables and get a new login shell.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #ed9d13">'export DEBEMAIL="gertvdijk@gmail.com"'</span><span style="color: #666666"> </span>>><span style="color: #666666"> </span>~./profile
<span style="color: #aaaaaa">$ </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #ed9d13">'export DEBFULLNAME="Gert van Dijk"'</span><span style="color: #666666"> </span>>><span style="color: #666666"> </span>~./profile
</code></pre></div>
<h2 id="create-the-ppa-on-launchpad">Create the PPA on Launchpad<a class="headerlink" href="#create-the-ppa-on-launchpad" title="Permanent link"> </a></h2>
<p>If you don't have a Launchpad account yet, create one.</p>
<p>Configure an SSH key for authentication on your profile page.
If you don't have an SSH key yet, see this <a href="https://blog.g3rt.nl/upgrade-your-ssh-keys.html">other blog post of me</a>.
Create an RSA key for Launchpad, because it does not support ed25519 keys at the time of writing (error: "Invalid key").
So, for this purpose, we'll be creating an extra RSA key just for Launchpad.
E.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>ssh-keygen<span style="color: #666666"> </span>-o<span style="color: #666666"> </span>-a<span style="color: #666666"> </span><span style="color: #51b2fd">100</span><span style="color: #666666"> </span>-t<span style="color: #666666"> </span>rsa<span style="color: #666666"> </span>-b<span style="color: #666666"> </span><span style="color: #51b2fd">2048</span><span style="color: #666666"> </span>-f<span style="color: #666666"> </span>~/.ssh/id_rsa_launchpad
</code></pre></div>
<p>Use the the <em>public key</em> for your Launchpad profile:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>cat<span style="color: #666666"> </span>~/.ssh/id_rsa_launchpad.pub
</code></pre></div>
<p>Add the output of the command above in the box for "Public key line".</p>
<p>Also upload your GPG key to your Launchpad profile in the "OpenPGP keys" section.
If you don't have one yet, I can recommend <a href="https://alexcabal.com/creating-the-perfect-gpg-keypair/">the article by Alex Cabal</a>.</p>
<p>After configuring your SSH and GPG keys, you should see a button "Create a new PPA" on your profile in the "Personal package archives" section.
Follow those steps online and keep the URL in mind for the next step.</p>
<h2 id="configure-dput">Configure dput<a class="headerlink" href="#configure-dput" title="Permanent link"> </a></h2>
<p>From the <a href="https://help.launchpad.net/Packaging/PPA/Uploading">Launchpad PPA Upload page</a>, configure your <code>~/.dput.cf</code>.
E.g. for my <code>ccid-backports</code> PPA I chose SFTP (using SSH):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #6ebf26; font-weight: bold">[ccid-backports]</span>
<span style="color: #bbbbbb">fqdn</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ed9d13">ppa.launchpad.net</span>
<span style="color: #bbbbbb">incoming</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ed9d13">~gertvdijk/ubuntu/ccid-backports/</span>
<span style="color: #bbbbbb">method</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ed9d13">sftp</span>
<span style="color: #bbbbbb">login</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ed9d13">gertvdijk</span>
<span style="color: #bbbbbb">ssh_config_options</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ed9d13">IdentityFile ~/.ssh/id_rsa_launchpad</span>
</code></pre></div>
<h2 id="get-sources">Get sources<a class="headerlink" href="#get-sources" title="Permanent link"> </a></h2>
<p>Find the link to the .dsc file on <a href="https://packages.debian.org/testing/libccid">the Debian packages website for libccid in the testing distribution</a>.
On the right hand side you shoul see that link.</p>
<p>And use that URL for <code>dget</code>, e.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>dget<span style="color: #666666"> </span>http://http.debian.net/debian/pool/main/c/ccid/ccid_1.4.27-1.dsc
</code></pre></div>
<p>The above should now "just work" without errors/warnings in the output, because the sources are signed by a Debian developer whose key is present in the debian-keyring package.</p>
<p>Sources should be available unpacked now.
Change the working directory to there:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span><span style="color: #2fbccd">cd</span><span style="color: #666666"> </span>ccid-1.4.27
</code></pre></div>
<h2 id="edit-the-changelog-file">Edit the changelog file<a class="headerlink" href="#edit-the-changelog-file" title="Permanent link"> </a></h2>
<p>Set the version in the Debian changelog with the appropriate target distribution and a locally suffixed version string.
In this case using Xenial:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>dch<span style="color: #666666"> </span>--local<span style="color: #666666"> </span>ppa~xenial<span style="color: #666666"> </span>--distribution<span style="color: #666666"> </span>xenial<span style="color: #666666"> </span><span style="color: #ed9d13">"No-change rebuild from current Debian testing version for Ubuntu xenial."</span>
</code></pre></div>
<p>The above is setting all required information on the command line.
Alternatively, go full interactive and edit the file using <code>dch -i</code>.
This opens a text editor in a way similar to what you might be used to when editing the <code>sudoers</code> file with <code>visudo</code>.</p>
<p>I'm not using the <code>--rebuild</code> option, since I'm not able to control the suffix label (<code>buildX</code> where <code>X</code> is a counter).
Instead, I want to be explicit about this being a version in a PPA.
I believe this could help users identifying possible issues later on.</p>
<h2 id="build-the-source-package">Build the source package<a class="headerlink" href="#build-the-source-package" title="Permanent link"> </a></h2>
<p>Meet <code>debuild</code>.
It's the universal way to build a package from its sources, provided it's 'packaged' by your distribution.</p>
<p>As we're going to build the binary package in the Launchpad cloud, we will build a signed source package and have Launchpad build the binaries.</p>
<p>Without the original sources, only diff-upload:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>debuild<span style="color: #666666"> </span>-S
</code></pre></div>
<p>This approach is very efficient, but will only be accepted after upload if Launchpad can access the original upstream tarball of the sources itself.
This is the case usually, but in some cases for packages not synced into Ubuntu yet or for unreleased versions you can run:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>debuild<span style="color: #666666"> </span>-S<span style="color: #666666"> </span>-sa
</code></pre></div>
<p>The above command will ask for your GPG keyring to be unlocked for signing the changes file.</p>
<p>Output will be in the <em>parent directory</em>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>ls<span style="color: #666666"> </span>-al<span style="color: #666666"> </span>../*.changes
<span style="color: #cccccc">-rw-r--r-- 1 gert gert 1933 Aug 7 17:37 ../ccid_1.4.27-1ppa~xenial1_source.changes</span>
</code></pre></div>
<h2 id="upload-the-sources-to-launchpad">Upload the sources to Launchpad<a class="headerlink" href="#upload-the-sources-to-launchpad" title="Permanent link"> </a></h2>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>dput<span style="color: #666666"> </span><ppa-name><span style="color: #666666"> </span><changes-file>
</code></pre></div>
<p>E.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>dput<span style="color: #666666"> </span>ccid-backports<span style="color: #666666"> </span>../ccid_1.4.27-1ppa~xenial1_source.changes
</code></pre></div>
<p>This should upload the packages.
Reporting of the acceptance will be communicated over email.
Keep an eye on your email inbox.</p>
<p>In the report by email you'll get either:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Accepted:
OK: ccid_1.4.27.orig.tar.bz2
OK: ccid_1.4.27-1ppaΛxenial1.debian.tar.xz
OK: ccid_1.4.27-1ppaΛxenial1.dsc
-> Component: main Section: libs
[...]
</code></pre></div>
<p>or:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Rejected:
Unable to find ccid_1.4.27.orig.tar.gz in upload or distribution.
Files specified in DSC are broken or missing, skipping package unpack verification.
[...]
</code></pre></div>
<p>For the latter case, rebuild the package with the full sources included (<code>-sa</code>) in the step above and increase the version in the Changelog because Launchpad will complain the version has already been uploaded before:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>dput<span style="color: #666666"> </span>ccid-backports<span style="color: #666666"> </span>../ccid_1.4.27-1ppa~xenial1_source.changes
</span><span style="color: #cccccc">Package has already been uploaded to ccid-backports on ppa.launchpad.net</span>
<span style="color: #cccccc">Nothing more to do for ccid_1.4.27-1ppa~xenial1_source.changes</span>
</code></pre></div>
<p>Go to the Launchpad PPA page to see the build progress.</p>
<p>You're done now, hopefully, if the package compiles!</p>
<p>I've published the <code>libccid</code> packages from this example in <a href="https://launchpad.net/~gertvdijk/+archive/ubuntu/ccid-backports">my PPA</a>.</p>
<h2 id="how-users-can-use-your-ppa">How users can use your PPA<a class="headerlink" href="#how-users-can-use-your-ppa" title="Permanent link"> </a></h2>
<p>Follow the instructions on the PPA page.
Usually, using the <code>add-apt-repository</code> tool approach is the easiest.</p>
<p>E.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>add-apt-repository<span style="color: #666666"> </span>ppa:gertvdijk/ccid-backports
<span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>apt-get<span style="color: #666666"> </span>update
</code></pre></div>
<h2 id="security-updates-and-ppas">Security updates and PPAs<a class="headerlink" href="#security-updates-and-ppas" title="Permanent link"> </a></h2>
<p>In case Ubuntu publishes a security update, it's usually done without upgrading the upstream version of the package.
However, by installing your PPA, the newer version available in your PPA will get precendence over the one available in the security repository as the version compare wins for the newer major version.</p>
<p>This can be tweaked by APT 'pinning', but the real solution would be to stay up-to-date and publish new releases regularly for your users.</p>
<h2 id="build-for-more-distributions">Build for more distributions<a class="headerlink" href="#build-for-more-distributions" title="Permanent link"> </a></h2>
<p>In case you want to build for more Ubuntu distributions, first remove the newly created changelog entry in <code>debian/changelog</code>.
After that, repeat the process of the changelog entry above.
Then rebuild the source package and upload the same way.
Launchpad simply builds the package for the target distribution specified in the Changelog.</p>
<h2 id="buildinstall-the-binary-locally">Build/Install the binary locally<a class="headerlink" href="#buildinstall-the-binary-locally" title="Permanent link"> </a></h2>
<p>Before uploading and publishing the package you may want to test it locally.
Install the build dependencies and run this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>apt-get<span style="color: #666666"> </span>build-dep<span style="color: #666666"> </span>libccid
<span style="color: #aaaaaa">$ </span>debuild<span style="color: #666666"> </span>-uc<span style="color: #666666"> </span>-us<span style="color: #666666"> </span>-b
</code></pre></div>
<p>The <code>-uc</code>/<code>-us</code> and <code>-b</code> options are for "unsigned changes/sources file" and "binary only" respectively.</p>
<p>It will create a binary package in the parent directory.
Install with <code>dpkg -i</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>dpkg<span style="color: #666666"> </span>-i<span style="color: #666666"> </span>../libccid_1.4.27-1ppa~xenial1_amd64.deb
<span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>apt-get<span style="color: #666666"> </span>install<span style="color: #666666"> </span>-f<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># in case of dependency errors to be resolved</span>
</code></pre></div>
<h2 id="other-useful-tools">Other useful tools<a class="headerlink" href="#other-useful-tools" title="Permanent link"> </a></h2>
<h3 id="apt-cache-policy"><code>apt-cache policy</code><a class="headerlink" href="#apt-cache-policy" title="Permanent link"> </a></h3>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>apt-cache<span style="color: #666666"> </span>policy<span style="color: #666666"> </span><package>
</code></pre></div>
<p>Shows the options the APT package management sees and what version is chosen for installation.</p>
<h3 id="apt-get-source"><code>apt-get source</code><a class="headerlink" href="#apt-get-source" title="Permanent link"> </a></h3>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>apt-get<span style="color: #666666"> </span><span style="color: #2fbccd">source</span><span style="color: #666666"> </span><package>
</code></pre></div>
<p>Downloads the sources of a given binary package in the current working directory.
This assumes the presence of <code>deb-src</code> lines in your <code>/etc/apt/sources.list</code>.</p>
<h3 id="docker">Docker<a class="headerlink" href="#docker" title="Permanent link"> </a></h3>
<p>I find Docker very powerful to experiment with packages in other distribution releases.
Very quickly I can check the presence of packages, have a peek at the sources and try to build them.</p>
<p>It's too broad to go into detail in this post, though.</p>
<h2 id="another-example-openvpn">Another example: OpenVPN<a class="headerlink" href="#another-example-openvpn" title="Permanent link"> </a></h2>
<p>OpenVPN 2.4.0 appeared to be buggy for me and I noticed <a href="https://packages.debian.org/stretch-backports/openvpn">2.4.3 available in stretch-backports</a> worked for me instead.
Building it for Ubuntu Xenial <a href="https://launchpad.net/~gertvdijk/+archive/ubuntu/openvpn-backports/+build/13199614">failed for me</a> at first.
Fortunately, it turns out that some build dependencies adjustments were all that was required to get it built.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #d22323">--- upstream/openvpn-2.4.3/debian/control 2017-06-30 15:39:56.000000000 +0200</span>
<span style="color: #589819">+++ my/openvpn-2.4.3/debian/control 2017-08-07 23:06:35.852621110 +0200</span>
<span style="color: #ffffff; text-decoration: underline">@@ -4,7 +4,7 @@</span>
<span style="color: #666666"> </span>Maintainer: Bernhard Schmidt <berni@debian.org>
<span style="color: #666666"> </span>Uploaders: JΓΆrg Frings-FΓΌrst <debian@jff-webhosting.net>
<span style="color: #666666"> </span>Build-Depends:
<span style="color: #d22323">- debhelper (>= 10),</span>
<span style="color: #589819">+ debhelper (>= 9),</span>
<span style="color: #666666"> </span> dh-systemd (>= 1.5),
<span style="color: #666666"> </span> dpkg-dev (>= 1.16.1),
<span style="color: #666666"> </span> iproute2 [linux-any],
<span style="color: #ffffff; text-decoration: underline">@@ -12,7 +12,7 @@</span>
<span style="color: #666666"> </span> liblzo2-dev,
<span style="color: #666666"> </span> libpam0g-dev,
<span style="color: #666666"> </span> libpkcs11-helper1-dev,
<span style="color: #d22323">- libssl1.0-dev,</span>
<span style="color: #589819">+ libssl-dev,</span>
<span style="color: #666666"> </span> libsystemd-dev [linux-any],
<span style="color: #666666"> </span> net-tools [!linux-any],
<span style="color: #666666"> </span> pkg-config
</code></pre></div>
<p>You can see the results in <a href="https://launchpad.net/~gertvdijk/+archive/ubuntu/openvpn-backports">my OpenVPN-backports PPA</a>.</p>Upgrade your SSH keys!2016-09-23T09:00:00+02:002022-04-01T22:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2016-09-23:/upgrade-your-ssh-keys.html<p>Upgrade to faster and more secure SSH keys. I promise a smooth transition to the use of your new keys.</p>
<p>Whether you're a software developer or a sysadmin, I bet you're using SSH keys.
Pushing your commits to GitHub or managing your Unix systems, it's best practice to do this over SSH with public key authentication rather than passwords.
However, as time flies, many of you are using older keys and not aware of the need to generate fresh ones to protect your privates much better.
In this post I'll demonstrate how to transition to an Ed25519 type of key smoothly, why you would want this and show some tips and tricks on the way there.</p>
<div class="admonition note">
<p class="admonition-title">Tl;dr:</p>
<p>Generate any new key with <code>ssh-keygen -o -a 100 -t ed25519</code>, specify a strong passphrase and read further if you need a smooth transition.</p>
</div>
<h2 id="dsa-and-rsa-1024-bit-are-deprecated-now">DSA and RSA 1024 bit are deprecated now<a class="headerlink" href="#dsa-and-rsa-1024-bit-are-deprecated-now" title="Permanent link"> </a></h2>
<p>If you've created your key using software released before 2013 with the default options it's probably insecure (RSA < 2048 bits).
Even worse, I've seen tweeps, colleagues and friends still using DSA keys (<code>ssh-dss</code> in OpenSSH format) recently.
That's a key type similar to RSA, but limited to 1024 bits size and therefore <a href="https://security.stackexchange.com/a/5100/12948">recommended against</a> for a long time.
It's plainly insecure and refused for valid reasons in recent OpenSSH versions (see also the <a href="http://www.openssh.com/txt/release-7.0">changelog for 7.0</a>). π¬</p>
<p>The sad thing about it is that I see posts on how to re-enable DSA key support rather than moving to a more secure type of key.
Really, it's unwise to follow instructions to change the configuration for <code>PubkeyAcceptedKeyTypes</code> or <code>HostKeyAlgorithms</code> (host keys are for a later post).
Instead, upgrade your keys!</p>
<p><img alt="Picture of an ancient key" src="https://blog.g3rt.nl/images/20160923_old_key_picture.jpg" style="float:right; margin-left: 3em;"/></p>
<p>Compare DSA with the technology of locks using keys like this one.
You wouldn't want this type of key to unlock your front door, right?</p>
<div class="spacer" style="clear: both;"></div>
<h2 id="determine-your-current-situation">Determine your current situation<a class="headerlink" href="#determine-your-current-situation" title="Permanent link"> </a></h2>
<p>List all your keys:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span><span style="color: #6ebf26; font-weight: bold">for</span><span style="color: #666666"> </span>keyfile<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">in</span><span style="color: #666666"> </span>~/.ssh/id_*<span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">do</span><span style="color: #666666"> </span>ssh-keygen<span style="color: #666666"> </span>-l<span style="color: #666666"> </span>-f<span style="color: #666666"> </span><span style="color: #ed9d13">"${</span><span style="color: #40ffff">keyfile</span><span style="color: #ed9d13">}"</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">done</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>uniq
</code></pre></div>
<ul>
<li>DSA or RSA 1024 bits: red flag. Unsafe.</li>
<li>RSA 2048: yellow recommended to change</li>
<li>RSA 3072/4096: great, but Ed25519 has some benefits!</li>
<li>ECDSA: depends. Recommended to change</li>
<li>Ed25519: wow cool, but are you brute-force safe?</li>
</ul>
<h2 id="a-smooth-transition-i-promise">A smooth transition, I promise.<a class="headerlink" href="#a-smooth-transition-i-promise" title="Permanent link"> </a></h2>
<p>You're probably thinking⦠"I'm using my key for a long time, I don't want to change them everywhere now."
Valid point, but you don't have to! It's good to know you can have multiple keys on your system and your SSH client will pick the right one for the right system automatically.</p>
<p>It's part of the SSH protocol that it can offer multiple keys and the server picks the one your client will have to prove it has possession of the private key by a challenge.
See it in action adding some verbosity to the SSH connect command (<code>-vvv</code>).
Also if you're using an SSH agent you can load multiple keys and it will discover them all.
Easy as that.</p>
<h2 id="youll-like-the-twisted-edwards-curve">You'll like the Twisted Edwards curve<a class="headerlink" href="#youll-like-the-twisted-edwards-curve" title="Permanent link"> </a></h2>
<p>Most common is the RSA type of key, also known as <code>ssh-rsa</code> with SSH.
It's very compatible, but also slow and potentially insecure if created with a small amount of bits (< 2048).
We just learned that your SSH client can handle multiple keys, so enable yourself with the newest faster elliptic curve cryptography and enjoy the very compact key format it provides!</p>
<p>Ed25519 keys are short. Very short. If you're used to copy multiple lines of characters from system to system you'll be happily surprised with the size. The public key is just about 68 characters. It's also much faster in authentication compared to secure RSA (3072+ bits).</p>
<p>Generating an Ed25519 key is done using the <code>-t ed25519</code> option to the ssh-keygen command.</p>
<p>Ed25519 is a reference implementation for EdDSA using Twisted Edward curves (<a href="https://en.wikipedia.org/wiki/Twisted_Edwards_curve">Wikipedia link</a>).</p>
<h2 id="increase-resistance-to-brute-force-password-cracking">Increase resistance to brute-force password cracking<a class="headerlink" href="#increase-resistance-to-brute-force-password-cracking" title="Permanent link"> </a></h2>
<p>When generating the keypair, you're asked for a passphrase to encrypt the private key with.
If you will ever lose your private key it should protect others from impersonating you because it will be encrypted with the passphrase.
To actually prevent this, one should make sure to prevent easy brute-forcing of the passphrase.</p>
<p>OpenSSH key generator offers two options to resistance to brute-force password cracking: using the new OpenSSH key format and increasing the amount of key derivation function rounds.
It slows down the process of unlocking the key, but this is what prevents efficient brute-forcing by a malicious user too.
I'd say experiment with the amount of rounds on your system.
Start at about 100 rounds.
On my system it takes about one second to decrypt and load the key once per day using an agent.
Very much acceptable, imo.</p>
<p>With <code>ssh-keygen</code> use the <code>-o</code> option for the new RFC4716 key format and the use of a modern key derivation function powered by bcrypt.
Use the <code>-a <num></code> option for <code><num></code> amount of rounds.</p>
<p>Actually, it appears that when creating a Ed25519 key the <code>-o</code> option is implied.</p>
<p>The OpenSSH manpages are not really explanatory about the 'new' format.
I found this article pretty useful: <a href="http://www.tedunangst.com/flak/post/new-openssh-key-format-and-bcrypt-pbkdf">"new openssh key format and bcrypt pbkdf" on www.tedunangst.com</a>.</p>
<h2 id="generate-your-new-sexy-ed25519-key">Generate your new sexy Ed25519 key<a class="headerlink" href="#generate-your-new-sexy-ed25519-key" title="Permanent link"> </a></h2>
<div class="admonition tip">
<p class="admonition-title">Protip</p>
<p>Use the same passphrase on all of your key types and profit with more convenience.
(See also <a href="#multi-key-aware-ssh-client">Multi-key aware SSH client</a>.)</p>
</div>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #d0d0d0">::</span><span style="color: #a61717; background-color: #e3d2d2">:</span><span style="color: #d0d0d0">console</span><span style="color: #666666"> </span><span style="color: #d0d0d0">hl_lines=</span><span style="color: #ed9d13">"1"</span>
<span style="color: #a61717; background-color: #e3d2d2">$</span><span style="color: #666666"> </span><span style="color: #d0d0d0">ssh-keygen</span><span style="color: #666666"> </span><span style="color: #d0d0d0">-o</span><span style="color: #666666"> </span><span style="color: #d0d0d0">-a</span><span style="color: #666666"> </span><span style="color: #51b2fd">100</span><span style="color: #666666"> </span><span style="color: #d0d0d0">-t</span><span style="color: #666666"> </span><span style="color: #d0d0d0">ed25519</span>
<span style="color: #d0d0d0">Generating</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">public</span><span style="color: #d0d0d0">/private</span><span style="color: #666666"> </span><span style="color: #d0d0d0">ed25519</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">key</span><span style="color: #666666"> </span><span style="color: #d0d0d0">pair.</span>
<span style="color: #d0d0d0">Enter</span><span style="color: #666666"> </span><span style="color: #d0d0d0">passphrase</span><span style="color: #666666"> </span><span style="color: #d0d0d0">(empty</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">for</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">no</span><span style="color: #666666"> </span><span style="color: #d0d0d0">passphrase)</span><span style="color: #a61717; background-color: #e3d2d2">:</span>
<span style="color: #d0d0d0">Enter</span><span style="color: #666666"> </span><span style="color: #d0d0d0">same</span><span style="color: #666666"> </span><span style="color: #d0d0d0">passphrase</span><span style="color: #666666"> </span><span style="color: #d0d0d0">again:</span>
<span style="color: #d0d0d0">Your</span><span style="color: #666666"> </span><span style="color: #d0d0d0">identification</span><span style="color: #666666"> </span><span style="color: #d0d0d0">has</span><span style="color: #666666"> </span><span style="color: #d0d0d0">been</span><span style="color: #666666"> </span><span style="color: #d0d0d0">saved</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">in</span><span style="color: #666666"> </span><span style="color: #d0d0d0">/home/gert/.ssh/id_ed25519.</span>
<span style="color: #d0d0d0">Your</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">public</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">key</span><span style="color: #666666"> </span><span style="color: #d0d0d0">has</span><span style="color: #666666"> </span><span style="color: #d0d0d0">been</span><span style="color: #666666"> </span><span style="color: #d0d0d0">saved</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">in</span><span style="color: #666666"> </span><span style="color: #d0d0d0">/home/gert/.ssh/id_ed25519.pub.</span>
<span style="color: #d0d0d0">The</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">key</span><span style="color: #666666"> </span><span style="color: #d0d0d0">fingerprint</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">is</span><span style="color: #a61717; background-color: #e3d2d2">:</span>
<span style="color: #d0d0d0">SHA256:</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[...]</span><span style="color: #666666"> </span><span style="color: #d0d0d0">gert</span><span style="color: #40ffff">@hostname</span>
<span style="color: #d0d0d0">The</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">key</span><span style="color: #a61717; background-color: #e3d2d2">'</span><span style="color: #d0d0d0">s</span><span style="color: #666666"> </span><span style="color: #d0d0d0">randomart</span><span style="color: #666666"> </span><span style="color: #71adff; text-decoration: underline">image</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">is</span><span style="color: #a61717; background-color: #e3d2d2">:</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[...]</span>
</code></pre></div>
<p>Note the line 'Your identification has been saved in /home/gert/.ssh/id_ed25519'.
Your current RSA/DSA keys are next to it in the same <code>~/.ssh</code> folder.
As with any other key you can copy the public key in <code>~/.ssh/id_ed25519.pub</code> to target hosts for authentication.</p>
<h2 id="multi-key-aware-ssh-client">Multi-key aware SSH client<a class="headerlink" href="#multi-key-aware-ssh-client" title="Permanent link"> </a></h2>
<p>All keys available on default paths will be autodetected by SSH client applications, including the SSH agent via ssh-add.
So, if you were using an application like ssh/scp/rsync before like...</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>ssh<span style="color: #666666"> </span>user@host
</code></pre></div>
<p>it will now offer multiple public keys to the server and the server will request proof of possession for a matching entry for authentication.
And your daily use of the <code>ssh-add</code> command will not change and autodiscover the Ed25519 key:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>ssh-add
</span><span style="color: #cccccc">Enter passphrase for /home/gert/.ssh/id_rsa:</span>
<span style="color: #cccccc">Identity added: /home/gert/.ssh/id_rsa (gert@hostname)</span>
<span style="color: #cccccc">Identity added: /home/gert/.ssh/id_ed25519 (gert@hostname)</span>
</code></pre></div>
<p>It not only discovered both keys, it also loaded them by entering a single passphrase (because it's the same)!</p>
<p>We've reached a very important goal now.
Without any change to your daily routine we can slowly change the existing configuration on remote hosts to accept the Ed25519 key.
In the meantime the RSA key will still work.
Great, right!?</p>
<h2 id="change-or-set-a-passphrase">Change or set a passphrase<a class="headerlink" href="#change-or-set-a-passphrase" title="Permanent link"> </a></h2>
<p>If you're afraid this will change your key, don't worry.
The private part of your keypair is encrypted with a passphrase which only exists locally on your machine.
Change it as often as you like.
This is recommended to prevent abuse in case the key file gets into the wrong hands.
Repeat for all your key files to ensure a new key format with 100 bcrypt KDF rounds:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>ssh-keygen<span style="color: #666666"> </span>-f<span style="color: #666666"> </span>~/.ssh/id_rsa<span style="color: #666666"> </span>-p<span style="color: #666666"> </span>-o<span style="color: #666666"> </span>-a<span style="color: #666666"> </span><span style="color: #51b2fd">100</span>
</code></pre></div>
<h2 id="upgrade-your-current-rsa-key">Upgrade your current RSA key<a class="headerlink" href="#upgrade-your-current-rsa-key" title="Permanent link"> </a></h2>
<p>Using Ed25519 will (and should) work in most situations by now, but legacy systems may not support them as of yet.
The best fallback is a strong RSA keypair for this.</p>
<p>While the OpenSSH client supports multiple RSA keys, it requires configuration/command line options to specify the path so it's rather error-prone.
Instead, I'd recommend upgrading your existing key in-place to keep things simple once this is done.
Depending on the strength (key size) of your current RSA key you can migrate urgently or comfortably.</p>
<p>In case you have a weak RSA key still, move it out of the way from the standard path and generate a new one of 4096 bits size:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>mv<span style="color: #666666"> </span>~/.ssh/id_rsa<span style="color: #666666"> </span>~/.ssh/id_rsa_legacy
<span style="color: #aaaaaa">$ </span>mv<span style="color: #666666"> </span>~/.ssh/id_rsa.pub<span style="color: #666666"> </span>~/.ssh/id_rsa_legacy.pub
<span style="color: #aaaaaa">$ </span>ssh-keygen<span style="color: #666666"> </span>-t<span style="color: #666666"> </span>rsa<span style="color: #666666"> </span>-b<span style="color: #666666"> </span><span style="color: #51b2fd">4096</span><span style="color: #666666"> </span>-o<span style="color: #666666"> </span>-a<span style="color: #666666"> </span><span style="color: #51b2fd">100</span>
</code></pre></div>
<p>If you are using an agent, manually point it to all your keys:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>ssh-add<span style="color: #666666"> </span>~/.ssh/id_rsa<span style="color: #666666"> </span>~/.ssh/id_rsa_legacy<span style="color: #666666"> </span>~/.ssh/id_ed25519
</code></pre></div>
<p>Once you are finished the transition on all remote targets you can go back to convenience and let it autodiscover your new RSA and Ed25519 keys; simply omit the keyfile arguments.</p>
<h2 id="software-support-for-ed25519">Software support for Ed25519<a class="headerlink" href="#software-support-for-ed25519" title="Permanent link"> </a></h2>
<p>Support is available since OpenSSH 6.5 and well adopted in the Unix world OSs for workstations.
Ubuntu 14.04+, Debian 8+, CentOS/RedHat 7+ etc. all support it already.
(If you have details about Mac OS X please drop a line, couldn't find it with a quick search).
Some software like custom desktop key agents may not like the new keys for several reasons (see below <a href="#my-gnome-keyring-doesnt-work-anymore">about the Gnome-keyring</a> for example).</p>
<p>GitHub works pretty well!</p>
<p>Launchpad gained support for Ed25519 keys <a href="https://bugs.launchpad.net/launchpad/+bug/907675/comments/55">since February 16, 2022</a>. Thanks Colin Watson for pointing this out in the comments!</p>
<p>Gerrit Code Review gained support <a href="https://gerrit-review.googlesource.com/c/gerrit/+/100998/">since 2017</a> with the 2.14 release.</p>
<p>PuTTY on Windows? See below.</p>
<h3 id="my-gnome-keyring-doesnt-work-anymore">My Gnome-keyring doesn't work anymore<a class="headerlink" href="#my-gnome-keyring-doesnt-work-anymore" title="Permanent link"> </a></h3>
<p>The Gnome-keyring, as used in Ubuntu Unity at least, fails to read the new RFC4716 format keys but reports success.
It's bugged.
More details here in <a href="https://askubuntu.com/q/564821/88802">my AskUbuntu Q&A post</a>.
I'd recommend disabling the Gnome keyring for SSH agent use and use the plain OpenSSH agent instead.</p>
<h3 id="im-using-windows-with-putty">I'm using Windows with PuTTY<a class="headerlink" href="#im-using-windows-with-putty" title="Permanent link"> </a></h3>
<p>Sorry, I'm not using PuTTY, but make sure to upgrade first.
This page suggests Ed25519 support since a late-2015 version according to a <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/ed25519.html">wishlist item</a>.
Generally speaking, I'm not too excited with the speed of implementation of security features in it.</p>
<h2 id="is-this-the-ultimate-secure-ssh-keypair">Is this the ultimate secure SSH keypair?<a class="headerlink" href="#is-this-the-ultimate-secure-ssh-keypair" title="Permanent link"> </a></h2>
<p>We've taken some steps, important ones, but it's far from ultimate security.
When dealing with high assurance environments I would strongly discourage key usage like described in this post as this holds the unencrypted private key in memory.
Instead, use hardware security (smart cards) to avoid leaking keys even from memory dumps.
It's not covered in this post, mainly because it requires a hardware device you need to buy and secondly because the limitations are device dependent.
A nice cute solution would be to make use of your TPM already built-in your PC probably, but that would definitely deserve another post.</p>
<h2 id="follow-up-posts">Follow-up posts<a class="headerlink" href="#follow-up-posts" title="Permanent link"> </a></h2>
<p>I'm planning on writing some more on how to harden SSH a bit more; custom host keys, custom DH moduli, strong ciphers (e.g. <code>chacha20-poly1305</code>) and secure KeyExchange/MACs.
For now this is a great resource already: <a href="https://stribika.github.io/2015/01/04/secure-secure-shell.html">https://stribika.github.io/2015/01/04/secure-secure-shell.html</a></p>
<p>For SSH <em>host</em> key validation, please see my other article: <a href="https://blog.g3rt.nl/ssh-host-key-validation-strict-yet-user-friendly.html">SSH host key validation done right β strict yet user-friendly</a>.</p>
<h2 id="your-thoughts">Your thoughts<a class="headerlink" href="#your-thoughts" title="Permanent link"> </a></h2>
<p>Want to share some ideas?
Post it below in the comments.</p>
<p>Love my post? Please share it.</p>
<blockquote class="twitter-tweet" data-lang="en"><p dir="ltr" lang="en">π Upgrade your SSH keys! (blog)<br/>Use Ed25519, about a transition and other tips & tricks. <a href="https://t.co/KDY2Ufh5FC">https://t.co/KDY2Ufh5FC</a></p>β Gert van Dijk β οΈ (@gertvdijk) <a href="https://twitter.com/gertvdijk/status/779202957988466688">September 23, 2016</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>Nginx add_header configuration pitfall2016-02-17T23:00:00+01:002016-02-17T23:00:00+01:00Gert van Dijktag:blog.g3rt.nl,2016-02-17:/nginx-add_header-pitfall.html<p>Nginx's add_header configuration directive does not inherit from parent scopes. This can lead to headers intended to be added are omitted. A common case is HSTS, Clickjacking protection and caching headers.</p><h2 id="security-by-http-headers">Security by HTTP headers<a class="headerlink" href="#security-by-http-headers" title="Permanent link"> </a></h2>
<p>Some HTTP security policies are as simple as adding a header to the response.
It's common to just add it to your configuration, but did you actually check if this is working as you expected?
For Nginx, a relatively popular and commonly used web server, this might seem surprisingly easy, but there's a huge pitfall.</p>
<p>I'll show you in this blog post it's easy to end up with an insecure configuration which you may look good from looking at the server configuration. It's about the <code>add_header</code> Nginx configuration directive that handles scoping completely different from what you may expect.</p>
<p>While I'm not the only one running into it and there are plenty of troubleshooting topics for this directive indicating this pitfall, I still see bad examples online.
Lately when I made the same mistake, a colleague noticed we weren't doing HSTS (<a href="https://nl.wikipedia.org/wiki/HTTP_Strict_Transport_Security">HTTP Strict Transport Security</a>) anymore after deploying a change involving caching headers.
This made me write this up to raise some attention.</p>
<h2 id="by-example-clickjacking-protection-and-hsts">By example; Clickjacking protection and HSTS.<a class="headerlink" href="#by-example-clickjacking-protection-and-hsts" title="Permanent link"> </a></h2>
<p>Let's go over this by example.
Your site is TLS-enabled (HTTPS), it is clickjacking protected, it is HSTS enabled and you're confident it will pass the security scan.
Below is the basics of the Nginx configuration for such, as you may consider sensible.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"># IMPORTANT! BELOW IS UNSAFE. DON'T COPY-PASTE ME. READ THE BLOG POST.
</span>
http {
# Clickjacking protection, see:
# https://developer.mozilla.org/en-US/docs/HTTP/X-Frame-Options
<span style="background-color: #404040"> add_header X-Frame-Options SAMEORIGIN;
</span>
# Serve HTTP non-TLS
server {
listen 80;
server_name www.example.com;
...
}
# Serve HTTPS
server {
listen 443 ssl;
server_name www.example.com;
ssl_certificate www.example.com.crt;
# Enable HSTS, only for HTTPS!
<span style="background-color: #404040"> add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
</span>
root /srv/web/www.example.com/http_root;
# To my dynamic web application (e.g. fcgi, uwsgi, ...)
...
location /api/sensitive {
# Responses contain sensitive data; browsers and proxy servers should
# not cache any of this.
<span style="background-color: #404040"> add_header Pragma "no-cache";
</span><span style="background-color: #404040"> add_header Cache-Control "private, max-age=0, no-cache, no-store";
</span> }
location /static {
alias /srv/web/www.example.com/static;
# This content never changes; aggressive caching enabled.
<span style="background-color: #404040"> add_header Pragma "cache";
</span><span style="background-color: #404040"> add_header Cache-Control "public";
</span> }
...
}
}
</code></pre></div>
<p>Clickjacking protection header applied globally in the configuration, check.
HSTS header present and only on HTTPS, check.
And sensitive data is not cached or stored, check.</p>
<p>Now, you're going to test the output in the browser.
Does it actually respond with all the headers you would expect?
Let's test this with <code>curl</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>HTTP
<span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>curl<span style="color: #666666"> </span>-Is<span style="color: #666666"> </span>http://www.example.com/<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>grep<span style="color: #666666"> </span>-F<span style="color: #666666"> </span>X-Frame-Options
</span><span style="color: #cccccc">X-Frame-Options: SAMEORIGIN</span>
<span style="color: #aaaaaa"># </span>HTTPS
<span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>curl<span style="color: #666666"> </span>-Is<span style="color: #666666"> </span>https://www.example.com/<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>grep<span style="color: #666666"> </span>-F<span style="color: #666666"> </span>X-Frame-Options
</span></code></pre></div>
<p>What?
We've defined the <code>X-Frame-Options</code> on the <code>http</code> scope that covers both the HTTP and HTTPS <code>server</code> scopes, right?</p>
<p>The answer is, yes, but the <code>add_header</code> for HSTS in the <code>server</code> scope has cleared the <code>X-Frame-Options</code> header in its parent scope.</p>
<p>But, really?
It's a totally unrelated header!</p>
<p>Yep.
It's behaviour as <a href="http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header">documented</a>:</p>
<blockquote>
There could be several <tt>add_header</tt> directives. These directives are inherited from the previous level if and only if there are no <tt>add_header</tt> directives defined on the current level.
</blockquote>
<p>Same goes for the caching headers in the HTTPS <code>server</code> block:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>HSTS<span style="color: #666666"> </span>works<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">for</span><span style="color: #666666"> </span>HTTPS,<span style="color: #666666"> </span>yey!
<span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>curl<span style="color: #666666"> </span>-Is<span style="color: #666666"> </span>https://www.example.com/<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>grep<span style="color: #666666"> </span>-F<span style="color: #666666"> </span>Strict-Transport-Security
</span><span style="color: #cccccc">Strict-Transport-Security: max-age=31536000; includeSubDomains</span>
<span style="color: #aaaaaa"># </span>Also<span style="color: #666666"> </span>when<span style="color: #666666"> </span>accessing<span style="color: #666666"> </span>any<span style="color: #666666"> </span>actual<span style="color: #666666"> </span>content?
<span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>curl<span style="color: #666666"> </span>-Is<span style="color: #666666"> </span>https://www.example.com/static/main.js<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>grep<span style="color: #666666"> </span>-F<span style="color: #666666"> </span>Strict-Transport-Security
</span></code></pre></div>
<p>This could mean that if the user is not actually accessing any content outside the unprotected URIs, he will effectively not see any HSTS protection.
For this example configuration it could be a low impact, but for a scope breaking it that covers most of the requests, it could be very harmful!</p>
<h2 id="possible-solutions">Possible solutions<a class="headerlink" href="#possible-solutions" title="Permanent link"> </a></h2>
<h3 id="alternative-module-for-setting-headers">Alternative module for setting headers<a class="headerlink" href="#alternative-module-for-setting-headers" title="Permanent link"> </a></h3>
<p>The <a href="https://github.com/openresty/headers-more-nginx-module#readme">ngx_headers_more</a> plugin will by default preserve headers added in the parent scope.</p>
<p>Procedures to install this unofficial plugin may not be a solution for everyone, although it is available in Debian/Ubuntu via the <a href="https://packages.debian.org/jessie/nginx-extras">nginx-extras</a> package.
It also requires to change existing configurations.</p>
<h3 id="define-a-common-config-snippet">Define a common config snippet<a class="headerlink" href="#define-a-common-config-snippet" title="Permanent link"> </a></h3>
<p>Create files to include always when fiddling with headers. For example:</p>
<p>In <code>http_headers.conf</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>add_header X-Frame-Options SAMEORIGIN;
</code></pre></div>
<p>In <code>https_headers.conf</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>include http_headers.conf
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
</code></pre></div>
<p>In your site configuration:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>http {
<span style="background-color: #404040"> include http_headers.conf;
</span>
# Serve HTTP non-TLS
server {
listen 80;
...
}
# Serve HTTPS
server {
listen 443 ssl;
...
<span style="background-color: #404040"> include https_headers.conf;
</span> ...
location /api/sensitive {
# Responses contain sensitive data; browsers and proxy servers should
# not cache any of this.
<span style="background-color: #404040"> add_header Pragma "no-cache";
</span><span style="background-color: #404040"> add_header Cache-Control "private, max-age=0, no-cache, no-store";
</span><span style="background-color: #404040"> include https_headers.conf;
</span> }
...
}
}
</code></pre></div>
<p>It will work, it's "copy-paste safe", I'd say, but it has some drawbacks:</p>
<ul>
<li>It suddenly breaks when someone adds a <code>add_header</code> statement in the first <code>server</code> scope.</li>
<li>Quite some extra configuration overhead.</li>
</ul>
<h2 id="bad-examples-in-the-public">Bad examples in the public<a class="headerlink" href="#bad-examples-in-the-public" title="Permanent link"> </a></h2>
<ul>
<li>GitHub Gist <a href="https://gist.github.com/plentz/6737338">Best nginx configuration for improved security(and performance)</a>
All headers in the parent scope are not effective by the HSTS header, as also noted in <a href="https://gist.github.com/plentz/6737338#gistcomment-1415691">a comment</a> there.</li>
<li>Misleading and wrong, but <a href="http://stackoverflow.com/a/19155090/1254292">accepted answer</a> on StackOverflow on the question why the <code>add_header</code> isn't working in the child scope.</li>
<li><a href="http://wiki.horde.org/webserver/nginx">Horde's nginx example configuration</a>.</li>
</ul>
<h2 id="share-your-thoughts">Share your thoughts<a class="headerlink" href="#share-your-thoughts" title="Permanent link"> </a></h2>
<p>Have a better solution?
Please share it below in the comments!</p>
<p>Also shocked?
Feel free to retweet. π</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Using Nginx? You should know about the add_header directive pitfall.<a href="https://t.co/7MMyuJ6YOn">https://t.co/7MMyuJ6YOn</a></p>— Gert van Dijk β οΈ (@gertvdijk) <a href="https://twitter.com/gertvdijk/status/700077735960563713">February 17, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>Android device encryption user interface flaw2015-08-18T09:00:00+02:002015-08-18T09:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2015-08-18:/android-lollipop-encryption-user-interface-flaw.html<p>Android's user interface insufficiently warns users on the impact of disabling of what is presented as an "enhancement" for encryption. Many Android owners may be unprotected unknowingly while their Android settings menu would suggest they are perfectly fine. If the device is affected, data on devices lost can be easily accessed without the PIN/password/pattern.</p>
<h1 id="tldr">TL;DR:<a class="headerlink" href="#tldr" title="Permanent link"> </a></h1>
<p>Don't get tempted by any app asking you to enable its Accessibility Service.
It will change your encryption password to the Android default one allowing everyone to decrypt the "encrypted" data.
The PIN you enter at powering on your device may not be used for encryption at all - without a warning given.</p>
<div class="admonition attention">
<p class="admonition-title">If you use encryption on your Android device, follow these steps now:</p>
<ol>
<li>Open up the Accessibility settings menu.</li>
<li>If a Accessibility Service is listed there at the top, disable it.</li>
<li>Change your screen lock PIN/password/pattern whatever to reset the encryption password.
Make sure to tick the "Require PIN to start device" option.</li>
</ol>
</div>
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link"> </a></h1>
<p>Every few weeks I feel like rebooting my Nexus 5 because it get's a little slow for some reason.
This morning I noticed it booted up without asking my encryption PIN.
I was expecting this early boot PIN entry dialog:</p>
<p><img alt="Early boot PIN entry dialog for decryption" src="https://blog.g3rt.nl/images/20150818_android_earlyboot_pin_entry.jpg"/></p>
<p>It did ask for a PIN though, but it was clearly already at the regular lock screen, with the launcher background already visible and notifications sounding.
For a moment I thought the PIN entry at boot for decrypting the user data had improved so much with the latest Cyanogenmod 12.1 nightly, but no... also TWRP recovery was able to access all my files without a PIN!</p>
<p>An encrypted filesystem doesn't get unencrypted overnight, so, what the hack happened?
I had to get to the bottom of this!</p>
<p>I took some dive into how Android encryption is implemented and found what I believe is a relatively serious security issue as a result of the user interface implementation flaw.
I'm publishing this because I think people should be aware of until this is improved by Google.</p>
<p>There may be many people out there like me until today, thinking they are protected by their "Encrypt phone" state set to "Encrypted" while in fact they are not.</p>
<p><img alt="Encryption status shown in Android settings" src="https://blog.g3rt.nl/images/20150818_android_encrypted_status.png"/></p>
<h1 id="background-information-on-android-encryption">Background information on Android encryption<a class="headerlink" href="#background-information-on-android-encryption" title="Permanent link"> </a></h1>
<p>The implementation of encryption in Android comprises of the Linux dm-crypt module.
It's the same as what is used by LUKS on your favourite Linux distribution, but Android developers have chosen to implement their own key management.</p>
<p>Custom recoveries like TWRP have built-in support for managing the encrypted partitions and allows the user to input the password to decrypt.</p>
<p>In Android versions 4.x, you might have noticed that when enabling encryption, Android forces you to set a PIN or password for the lock screen to use that as the encryption password as well.
During power up of your phone you would then enter your device PIN/password/pattern early in the boot process, in order the user specific data to be loaded.</p>
<p>With the release of Android Lollipop, encryption security has been improved greatly on a technical level I won't be touching much.
However, it has also "improved" on how tightly the lock screen PIN/password/pattern was connected to the device encryption key.</p>
<p>This article is more about a user interface flaw for that improvement.
It can lead to insufficient awareness of the user being raised with regard to the impact it has to the protection level of the encrypted data.</p>
<h1 id="default-password-in-lollipop">Default password in Lollipop<a class="headerlink" href="#default-password-in-lollipop" title="Permanent link"> </a></h1>
<p>For the Lollipop release, Google is motivating vendors to enable encryption by default with the following feature.
Devices should be encrypted in the box, using a default password.
This default password is hidden for the user as it is not asked for during boot.
The user is asked to set a lock screen PIN/password/pattern in the first time setup wizard, which then also updates the password for the device encryption.</p>
<p>This approach also improves the user experience by not having to go through the long process of encrypting a currently unencrypted data partition, but instead, offers a near-instant encryption protection by just changing the passphrase for the master key of the encrypted partitions.</p>
<p>The default password is <code>default_password</code> according to the <a href="https://source.android.com/devices/tech/security/encryption/">Android Documentation on encryption</a>:</p>
<blockquote>
<p>The default password is: "default_password". However, the resultant hash is also signed through a TEE (such as TrustZone), which uses a hash of the signature to encrypt the master key.</p>
</blockquote>
<p>Furthermore, the documentation mentions in quite some detail on how proper security is handled:</p>
<blockquote>
<p>Hardware backing is implemented by using Trusted Execution Environmentβs (TEE) signing capability. Previously, we encrypted the master key with a key generated by applying scrypt to the user's password and the stored salt. In order to make the key resilient against off-box attacks, we extend this algorithm by signing the resultant key with a stored TEE key. The resultant signature is then turned into an appropriate length key by one more application of scrypt. This key is then used to encrypt and decrypt the master key.</p>
</blockquote>
<p>This means that even with the default password the storage is safe for "off-box attacks".
Sounds good, right?
Well... how likely would it be to to find phone built-in flash memory without the actual device?
So, in practice the use of this default password doesn't offer any data protection for the simple case of losing a device.</p>
<p>Given that users reinitialize this default password on first use... could it still go wrong?
Yes.</p>
<h1 id="accessibility-services">Accessibility Services<a class="headerlink" href="#accessibility-services" title="Permanent link"> </a></h1>
<p>Before an Accessibility Service is turned on, a user is warned about the data encryption not being "enhanced" anymore as shown below:</p>
<p><img alt="Accessibility service warning message" src="https://blog.g3rt.nl/images/20150818_android_accessibility_service_warning_message.png"/></p>
<p>First of all, I don't get what accessibility services have to deal with the device encryption to start with.
Please enlighten me if you do.
Secondly and more importantly, it's not just some "enhancement" of a screen lock that gets disabled; it's the whole data protection that's being reduced to nothing, because <strong>the encryption password is being reset to the default!</strong></p>
<p>Moreover, I didn't even spot that part of the warning message.
It looks like a "this apps needs access to..." dialog, don't you agree?</p>
<p>Upon proceeding, your PIN/password/pattern is unchanged for the lock screen and the Encryption status will still say "Encrypted".
This effectively hides the disabled data protection and the warning is dismissed without ever reminding you.</p>
<p>Anyone having physical access to your Android device will be able to decrypt all your data without knowledge of your PIN/password/pattern.
It's defeating the main purpose of the device encryption to begin with, so how does that align with the text in the warning?</p>
<h1 id="am-i-the-only-one-to-notice">Am I the only one to notice?<a class="headerlink" href="#am-i-the-only-one-to-notice" title="Permanent link"> </a></h1>
<p>No, I'm not the first.
When googling I'm bumping into a lot of confused users shown the incorrect warning for devices not encrypted at all and for which it's not even applicable.
In <a href="https://code.google.com/p/android/issues/detail?id=79309">Android issue 79309</a> a user mentions:</p>
<blockquote>
<p>I have a Nexus 6 on T-Mobile. Enabling app permissions in the Accessibly settings DISABLES the requirement for a pin during the boot process. This is a pretty big security issue.</p>
</blockquote>
<p>And another one:</p>
<blockquote>
<p>This one is probably slightly higher than Priority-Small, as it's likely a deployment blocker for enterprises, where it presents a serious security issue.
[...]
I confirmed that providing the disk encryption credentials then turns off PIN or password requirement at boot, meaning the disk is in practice unencrypted per access requirements.</p>
</blockquote>
<p>Yet the issue isn't given priority by Google.</p>
<p>In the meantime I see instructions posted to just hit OK to be able to use the app involved.
Like this one on the <a href="https://lastpass.com/support.php?cmd=showfaq&id=8936">official LastPass Q&A site</a>:</p>
<blockquote>
<p>When enabling LastPass Accessibility Service, you may see the following warning: "Because you've turned on an accessibility service, your device won't use your screen lock to enhance data encryption" OR your "require pin at boot" option is disabled. Please note that it is a Google bug and not specific to LastPass but all other Accessibility Services. For more information, please see the post here: https://code.google.com/p/android/issues/detail?id=79309.</p>
<p>You need to disable "require pin at boot" option, enable Accessibility Service, and re-enable the option.</p>
</blockquote>
<p>LastPass just refers to this bug report as if the warning being shown by Android is a bug and advices users to just proceed, yet severely impacting the encryption protection level! I'm quite sure LastPass isn't the only example.</p>
<h1 id="how-to-reproduce">How to reproduce<a class="headerlink" href="#how-to-reproduce" title="Permanent link"> </a></h1>
<p>Here's how I reproduced it on a Nexus 5 with both stock ROM 5.1.1 build LMY48I as well as Cyanogenmod 12.1 nightlies.</p>
<ol>
<li>Set up device encryption on your Android device using any option that requires the password/pattern at power on time.</li>
<li>Notice that you'll get the early PIN/password/pattern input interface which looks different from the lock screen.</li>
<li>Install an app that installs an Accessibility Service, e.g. <a href="https://play.google.com/store/apps/details?id=com.lukekorth.screennotifications">Screen Notifications by Luke Korth</a>. <em>There's nothing wrong with this particular app, it's just an example for demonstration purposes.</em></li>
<li>Open the app. For the example application above, allow access to the notifications.</li>
<li>Enable the Accessibility service it requires from the app or, ...</li>
<li>Verify that the service is indeed enabled by checking the Accessibility section in the Android system settings. <img alt="Accessibility service enabled in settings" src="https://blog.g3rt.nl/images/20150818_android_accessibility_service_enabled.png"/></li>
<li>Notice that when enabling the service the warning message shown does not mention clearly that the protection level is seriously compromised. Instead, it lets you focus on the other single item shown in a list, which is totally sane for the purpose of this app. Proceed by tapping OK.</li>
<li>Power off your Android device.</li>
<li>Power it back on. Notice that you won't be asked for user input until you already see your regular lock screen.</li>
</ol>
<h1 id="the-attacker-victim-scenario">The attacker-victim scenario<a class="headerlink" href="#the-attacker-victim-scenario" title="Permanent link"> </a></h1>
<p>Identifying the critical steps from the section above, all an attacked needs to do is persuade a victim to install a perfectly legitimate app.
The single requirement for the app chosen is that it needs to be convincing enough for a victim to get him to enable the app's Accessibility Service on his device.</p>
<p>Once the victim has noticed the app may have caused some harm, he uninstalls it, but it is still affected by the default password being used for encryption.
Unless the victim changes his lock screen password, the default encryption password will remain in place.</p>
<h1 id="what-i-think-should-be-improved">What I think should be improved<a class="headerlink" href="#what-i-think-should-be-improved" title="Permanent link"> </a></h1>
<p>If and only if there's a good reason for Accessibility Services to deal with data encryption settings, I think Android should be improved to handle the use of the default password with a lot more noise and clear warnings.</p>
<ol>
<li>Change wording from "disable enhanced encryption" to "use unsafe default password for encryption".</li>
<li>A sticky notification should be present as long as the device encryption password is the default one.</li>
<li>Suggest the user to re-enable the password if all Accessibility Services are disabled.
Also when the app providing the service is removed, this should trigger.</li>
</ol>
<p>The above should avoid users being tricked into enabling Accessibility Services installed by apps and unknowingly disabling the protection offered by device encryption.</p>
<p>However, most effective is just have Accessibility Services not interfere with data encryption settings.</p>
<h1 id="conclusions">Conclusions<a class="headerlink" href="#conclusions" title="Permanent link"> </a></h1>
<p>Android's user interface insufficiently warns users on the impact of disabling of what is presented as an "enhancement" for encryption.
Many Android owners may be unprotected unknowingly while their Android settings menu would suggest they are perfectly fine. If the device is affected, data on devices lost can be easily accessed without the PIN/password/pattern.</p>
<p>Additionally, it opens up ways for attackers to get hold of user's data stored on encrypted devices through social engineering.</p>
<p>I think this deserves some attention by end users, Android developers at Google and those managing mobile devices in enterprises.
I'd be grateful for anyone forwarding this to some Google/AOSP developers to get this sorted in a future Android version.
Thanks!</p>
<p>Do you agree, do you have suggestions or remarks?
Please share your thoughts in a comment below!</p>Enabling Dell Latitude RFID/NFC (Broadcom 5880)2015-07-17T09:00:00+02:002015-07-17T09:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2015-07-17:/enable-dell-nfc-contactless-reader.html<p>Dell Latitude laptops with a Broadcom 5880 security device usually have an integrated NFC/RFID reader too. The device is, however, not recognized by the operating system. Dell ships these laptops in the ControlVault-only mode, preventing access to it on the CCID level. This article shows how to enable the device for use in the OS.</p>
<h1 id="about-the-dell-security-solution">About the Dell security solution<a class="headerlink" href="#about-the-dell-security-solution" title="Permanent link"> </a></h1>
<p><img alt="Dell's RFID icon at the Latitude E7240 palm rest" src="https://blog.g3rt.nl/images/20150717_dell_rfid_palmrest.jpg"/></p>
<p>The Dell Latitude machines I use, E7240 and an E6530, show an icon on the palm rest indicating there's an contactless reader (NFC/RFID) device present.
However, in the OS there's no sign of this is.
PCSC does recognize the 'Contacted' reader, yet not the 'Contactless'.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa"># </span>pcsc_scan
</span><span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">Scanning present readers...</span>
<span style="color: #cccccc">0: Broadcom Corp 5880 [Contacted SmartCard] (0123456789ABCD) 00 00</span>
</code></pre></div>
<p>In this article I will show the steps on how to include...</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">1: Broadcom Corp 5880 [Contactless SmartCard] (0123456789ABCD) 01 00</span>
</code></pre></div>
<p>It appears that Dell cooperated with Broadcom and created a security solution known as <a href="http://www.dell.com/learn/us/en/25/campaigns/dell-data-protection-solutions">ControlVault</a> for the Broadcom Unified Security Hub (USH).
It may offer some nice pre-boot authentication solutions I can't seem to enable in the BIOS setup anyway.
Also online not much is telling me how to actually use it.
As I don't find any official Broadcom documentation and all points to Dell, I think this is a Dell-only offered OEM solution.</p>
<h1 id="some-success-on-older-latitudes">Some success on older Latitudes<a class="headerlink" href="#some-success-on-older-latitudes" title="Permanent link"> </a></h1>
<p>Some first googling brought me to <a href="http://bibuweb.de/?p=101">the blog of bibuweb</a>.
It shows how to disable the "CV-only" mode to enable CCID support.
Nice!
But... downloading a very old - 2008 - random binary from a Dell website still work on my 2014 E7240 machine?
No, it didn't.
Damn.
But at least it gave a some hope to research on this further.</p>
<h1 id="controlvault-support-package-dos-tools">ControlVault support package DOS tools<a class="headerlink" href="#controlvault-support-package-dos-tools" title="Permanent link"> </a></h1>
<p>Luckily, in some Dell ControlVault 'firmware upgrade' Windows-only support packages provided on the support page of the Latitude model I spotted some interesting document watermarked 'confidential' - about the use of this USH diagnostics tool, as well as a much newer <code>ushdiag.exe</code> tool itself!</p>
<p>Some first preparation steps:</p>
<ol>
<li>Download the ControlVault upgrade package, e.g. <code>ControlVault_Setup_09J7J_A33_ZPE.exe</code>.</li>
<li>Rename the file to have it the <code>.zip</code> 'extention'.</li>
<li>Unzip it.</li>
<li>Notice a <code>DOS</code> folder there.<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>DOS
βββ DOS4GW.EXE
βββ dosushdiag.pdf <-- "Broadcom USH Keyboard Architecture Document"
βββ errlvl.exe
βββ release.txt
βββ sleep.exe
βββ ushdiag.exe <-- there!
βββ ushfwumg.bat
</code></pre></div>
</li>
</ol>
<h1 id="broadcom-ush-diagnostics-tool">Broadcom USH diagnostics tool<a class="headerlink" href="#broadcom-ush-diagnostics-tool" title="Permanent link"> </a></h1>
<p>This PDF by Broadcom explains the usage of the <code>ushdiag.exe</code> tool.
Most important is this part about the options to provide.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>5.22 Device Enable (-de <devMask>)
This command will enable the specified devices.
<devMask>:
0: Smart Card:
1: Fingerprint:
2: RFID radio
3: CV Only Radio
5.23 Device Disable (-dd <devMask>)
This command will disable the specified devices.
<devMask>:
0: Smart Card:
1: Fingerprint:
2: RFID radio
3: CV Only Radio
</code></pre></div>
<h1 id="running-it-off-a-dos-usb-flash-drive">Running it off a DOS USB flash drive<a class="headerlink" href="#running-it-off-a-dos-usb-flash-drive" title="Permanent link"> </a></h1>
<p>Let's continue on getting this DOS executable going.</p>
<ol>
<li>Create a DOS bootable USB flash drive.
The easiest method I know is the <a href="http://www.sysresccd.org/Sysresccd-manual-en_How_to_install_SystemRescueCd_on_an_USB-stick">SystemRescueCd for USB flash drives</a>.</li>
<li>Copy the whole <code>DOS</code> folder onto the drive.</li>
<li>Boot your Latitude from the SystemRescueCd drive, in legacy (non-UEFI) mode. For my E7240 this means enabling Legacy Option ROMs and using F12 during POST to get the one-time boot menu to boot off the USB flash drive.</li>
<li>In the boot choice option menu of SystemRescueCd, choose the floppy disk images, option FreeDOS.</li>
<li>During boot of FreeDOS it will ask about which memory manager; I chose none.</li>
<li>Go to the 'C' drive, so type: <code>C:</code> and press <enter></enter></li>
<li>Go to the DOS folder: <code>cd DOS</code>.</li>
<li>
<p>Check the current status of the USH device:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #cccccc">FreeDOS C:\DOS>ushdiag.exe -u -stat</span>
</span><span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">Smart Card: Present; Enabled</span>
<span style="color: #cccccc">Fingerprint: Present; Enabled</span>
<span style="color: #cccccc">RFID Radio: Present; Enabled</span>
<span style="color: #cccccc">RFID Lock: Disabled</span>
<span style="background-color: #404040"><span style="color: #cccccc">CV Only Radio: Enabled</span>
</span><span style="color: #cccccc">RFID AutoDetect Set</span>
<span style="color: #cccccc">RFID Present Not Forced</span>
<span style="color: #cccccc">WBDI: Enabled</span>
<span style="background-color: #404040"><span style="color: #cccccc">RFID Block Mode: Unknown (CV Only Radio Mode Enabled)</span>
</span></code></pre></div>
<p>As you can see the CV-only mode is enabled.</p>
</li>
<li>
<p>Now <em>disable</em> the CV-only device to actually <em>enable</em> regular CCID RFID access by providing hexadecimal mask <code>8</code>. This comes from the document found earlier by sending a disable command on bit field 3.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #cccccc">FreeDOS C:\DOS>ushdiag.exe -u -dd 8</span>
</span><span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">Disabled CV Only Radio Mode. waiting for USH to reset</span>
<span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">RFID Lock: Disabled</span>
<span style="color: #cccccc">CV Only Radio: Disabled</span>
<span style="color: #cccccc">[...]</span>
</code></pre></div>
</li>
<li>
<p>Reboot your system and enjoy your RFID/NFC device!</p>
</li>
</ol>
<p>Confirmed working is my Yubikey NEO (<a href="https://developers.yubico.com/yubico-piv-tool/YubiKey_NEO_PIV_introduction.html">PIV applet</a>).
It appears the device lacks support in libnfc, so I haven't successfully managed to talk with MiFare cards for example.</p>
<p>Any thoughts?
Or issues?
Confirmed working for another Dell machine?
Leave a comment below!</p>16 tips on OpenVPN security2014-10-30T12:00:00+01:002017-08-03T00:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2014-10-30:/openvpn-security-tips.html<p>16 tips in securing your OpenVPN configuration.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Information here applies to OpenVPN version 2.3.x. For 2.4.x, more secure options are available. This post will be updated soon to include more on 2.4!</p>
</div>
<div class="admonition tip">
<p class="admonition-title">Updates</p>
<p><strong>August 3, 2017:</strong> Added in #4: note about DH parameters and added the <code>-check</code> option to the <code>openssl</code> command.</p>
<p><strong>March 24, 2015:</strong> Included <code>tls-version-min</code> (now #12) and updated #16 to include a note about IANA TLS cipher suite names.</p>
</div>
<h2 id="when-default-settings-arent-sufficient">When default settings aren't sufficient<a class="headerlink" href="#when-default-settings-arent-sufficient" title="Permanent link"> </a></h2>
<p>OpenVPN developers tend to prioritize backward compatibility over security.
This is not a general bad practise, but the current OpenVPN defaults aren't <em>that</em> well from a security perspective, in my opnion.
In this post I hope to help you with 16 practical tips to a more secure OpenVPN setup.</p>
<p>By following the tips in this post it will help minimizing (or even full mitigation of) risks like:</p>
<ul>
<li>A man-in-the-middle attack.</li>
<li>Future decryption of data in case of private key compromise (by enabling forward secrecy).</li>
<li>Client key compromise allowing an attacker to impersonate a server.</li>
</ul>
<p>The post may seem very large and overwhelming, I'm aware of that possibility.
It just totally depends on what your're trying to protect with your OpenVPN installation.
For protecting really secure networks (confidentiality) and the need for data integrity, please read up on all of it.
For just bypassing a simple firewall without the specific needs of data encryption, this post is a probably a lot less relevant to you as by default OpenVPN is secure enough for you probably.</p>
<h2 id="do-it-right-the-first-time">Do it right the first time<a class="headerlink" href="#do-it-right-the-first-time" title="Permanent link"> </a></h2>
<p>Unless mentioned otherwise, all configuration changes apply to both client <em>and</em> server configuration.
Most changes require the client to update his configuration as well if changed later.
If you're currently running on defaults with a large amount of clients you may want to apply all changes at once and plan the transition in a maintenance window.</p>
<p>In this post I'm assuming you're running OpenVPN version 2.3.x with TLS authentication (certificates, opposed to pre-shared keys).</p>
<h2 id="the-16-tips">The 16 tips<a class="headerlink" href="#the-16-tips" title="Permanent link"> </a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The 16 items below aren't ordered in a specific way. The first may be as important as the last in your case.</p>
</div>
<h3 id="1-verify-the-x509-subject-name">1: Verify the X.509 subject name<a class="headerlink" href="#1-verify-the-x509-subject-name" title="Permanent link"> </a></h3>
<p>In the client configuration, verify the server certificate subject string.
For example:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>verify-x509-name 'C=NL, O=Gert van Dijk, CN=xlsvps.gertvandijk.net' subject
</code></pre></div>
<h3 id="2-check-the-extended-key-usage-on-the-certificates">2: Check the Extended Key Usage on the certificates<a class="headerlink" href="#2-check-the-extended-key-usage-on-the-certificates" title="Permanent link"> </a></h3>
<p>Take this measure to prevent a client using his certificate to impersonate a server.</p>
<p>Certificates using the X509v3 format have key usage flags set.
Clients should use certificates with the "TLS Web Client Authentication" set and servers should be sending a certificate with "TLS Web Server Authentication" set.
Now configure OpenVPN to check for this:</p>
<p>For client configurations, connecting to servers:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>remote-cert-eku "TLS Web Server Authentication"
</code></pre></div>
<p>For server configurations, accepting client connections:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>remote-cert-eku "TLS Web Client Authentication"
</code></pre></div>
<h3 id="3-use-a-tls-authentication-secret">3: Use a TLS authentication secret<a class="headerlink" href="#3-use-a-tls-authentication-secret" title="Permanent link"> </a></h3>
<p>Use an additional shared secret for authenticating the TLS handshake, minimizing (D)DoS attacks.
The server and all clients should share the same secret to pass the initial TLS handshake.</p>
<p>Generate the shared secret:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openvpn --genkey --secret tls-auth.key</span>
</code></pre></div>
<p>For client configurations, connecting to servers:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>tls-auth /path/to/tls-auth.key 1
</code></pre></div>
<p>For servers, accepting client connections:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>tls-auth /path/to/tls-auth.key 0
</code></pre></div>
<p>Alternatively, embed the key and specify the key direction separately:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><tls-auth>
...
</tls-auth>
key-direction 0
</code></pre></div>
<p>The latter approach appears to be incompatible with OpenVPN-NL.</p>
<h3 id="4-generate-diffie-hellman-parameters">4: Generate Diffie-Hellman parameters<a class="headerlink" href="#4-generate-diffie-hellman-parameters" title="Permanent link"> </a></h3>
<p>With pregenerated Diffie-Hellman parameters the TLS session will be enabled (but not limited to) for use with TLS ciphersuites providing <a href="https://en.wikipedia.org/wiki/Forward_secrecy">forward secrecy</a>.
This means that even if a malicious user got hold of the secret keys of any of the peers, he can still not decrypt the encrypted data intercepted.
More on that in the last item.</p>
<p>DH parameters are considered public and it's hard to verify that those generated are actually strong.
Unfortunately, OpenVPN does not ship with proven strong built-in DH parameters.</p>
<p>Only required on a server.</p>
<p>Generate a 2048 bits DH parameters file:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openssl dhparam 2048 -out dh2048.pem -check</span>
</code></pre></div>
<p>In the configuration:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>dh /path/to/dh2048.pem
</code></pre></div>
<h3 id="5-increase-rsa-key-sizes">5: Increase RSA key sizes<a class="headerlink" href="#5-increase-rsa-key-sizes" title="Permanent link"> </a></h3>
<p>For all certificates/keys, please use <em>at least</em> 2048 bits (RSA) as it's the minimum considered key size.
By default older easy-rsa helper scripts generated all 1024 bits keys.
Revoke and reissue all certificates/keys with a size lower than 2048 bits in size.
If your situation allows you, use 4096 bits RSA key size.</p>
<h3 id="6-check-for-revoked-certificates">6: Check for revoked certificates<a class="headerlink" href="#6-check-for-revoked-certificates" title="Permanent link"> </a></h3>
<p>A list of revoked certificates should be checked against to deny access with such a certificate.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>crl-verify /path/to/crl.pem
</code></pre></div>
<h3 id="7-use-sha-2-certificate-signatures">7: Use SHA-2 certificate signatures<a class="headerlink" href="#7-use-sha-2-certificate-signatures" title="Permanent link"> </a></h3>
<p>When signing certificates the digital signature of the certificate to be signed is calculated.
This hash (digest message) is the actual data being signed by the CA key for trusting it.
It is of great importance you don't use MD5, SHA-1 or even weaker digests, because it is considered unsafe and someone could impersonate your server's or client's identity.</p>
<p>To check your certificate (or intermediate CA) signature, do</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openssl x509 -in /path/to/cert.crt -text</span>
</code></pre></div>
<p>And it should output lines like these:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Signature Algorithm: sha256WithRSAEncryption
</code></pre></div>
<p>Any algorithm of the SHA-2 family (SHA-256, SHA-384, SHA-512) should be fine.</p>
<h3 id="8-use-a-security-device">8: Use a security device<a class="headerlink" href="#8-use-a-security-device" title="Permanent link"> </a></h3>
<p>In typical use of certificates, clients' private keys are everywhere in your organisation and being used on many machines.
The risk of them being compromised could be mitigated with revocation, but that won't help if keys have gone stolen unnoticed.</p>
<p>Instead of storing private keys on regular storage, it's a lot better to use a security device like a token or smartcard.
Private keys on such devices will never leave it and cryptographic operations are performed on the card itself.
This minimizes the risk in a private key compromise significantly.</p>
<p>OpenVPN can talk to PKCS#11 compatible devices. For setting up such a device, have a look at another post: <a href="https://blog.g3rt.nl/getting-started-smartcard-hsm.html">Getting started with the SmartCard-HSM</a>.</p>
<p>Select the right certificate by listing all on the token:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openvpn --show-pkcs11-ids my-pkcs11-middleware.so</span>
</code></pre></div>
<p>Where <code>my-pkcs11-middleware.so</code> is the path to the manufacturer provided middleware (shared library).
For OpenSC cards using the OpenSC provided PKCS#11 middleware, use:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openvpn --show-pkcs11-ids opensc-pkcs11.so</span>
</code></pre></div>
<p>It will output something like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Certificate
DN: C=NL, O=Gert van Dijk, CN=Gert van Dijk, description=ePass2003 #2 Q4 2014 VPN
Serial: 15
<span style="background-color: #404040"> Serialized id: EnterSafe/PKCS\x2315/7528531617051201/ePass2003\x20\x232\x20\x28User\x20PIN\x29/D08DD75984CC577F
</span></code></pre></div>
<p>Then use it like this in your OpenVPN client configuration:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>pkcs11-id 'EnterSafe/PKCS\x2315/7528531617051201/ePass2003\x20\x232\x20\x28User\x20PIN\x29/D08DD75984CC577F'
pkcs11-providers opensc-pkcs11.so
</code></pre></div>
<p>During the connection initiation the PKCS#11 middleware will ask for the security token passphrase.</p>
<h3 id="9-persistent-tuntap-device">9: Persistent tun/tap device<a class="headerlink" href="#9-persistent-tuntap-device" title="Permanent link"> </a></h3>
<p>While your connetion might be interrupted and OpenVPN is trying to reconnect, you may be using the default network routes again, bypassing the tunnel.
For accessing private networks this might not be a big issue as the network addresses may not be reachable from outside the tunnel, but it may expose information you'd rather keep private like an HTTP request containing cookies.</p>
<p>To tell OpenVPN to keep the device open and to hold traffic until the connection restored, simply set the <code>persist-tun</code> option.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>persist-tun
</code></pre></div>
<h3 id="10-dont-pull-configurations">10: Don't pull configurations<a class="headerlink" href="#10-dont-pull-configurations" title="Permanent link"> </a></h3>
<p>It's quite common in OpenVPN usage to have a server to send some client configuration parameters (push) over and the client to apply these (pull).
If you just want to be sure your client configuration is exactly as you configured it, then consider to not include the <code>pull</code> option.
This will require you to set up everything explicitly in the client configuration, and it may be severely impacting the ease of OpenVPN client deployments.</p>
<h3 id="11-set-a-script-security-level">11: Set a script security level<a class="headerlink" href="#11-set-a-script-security-level" title="Permanent link"> </a></h3>
<p>In normal configurations you'd want the OpenVPN daemon to configure the tun/tap device for you.
(E.g.: configure the interface with the IP address assigned by the server.)
However, you might want to consider doing it yourself, just to be sure in avoiding any command injection.
Shellshock, anyone? ;-)</p>
<p>From the manpage:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>script-security level
This directive offers policy-level control over OpenVPN's usage of exterβ
nal programs and scripts. Lower level values are more restrictive, higher
values are more permissive. Settings for level:
0 -- Strictly no calling of external programs.
1 -- (Default) Only call built-in executables such as ifconfig, ip, route,
or netsh.
2 -- Allow calling of built-in executables and user-defined scripts.
3 -- Allow passwords to be passed to scripts via environmental variables
(potentially unsafe).
</code></pre></div>
<h3 id="12-set-a-minimum-tls-protocol-version">12: Set a minimum TLS protocol version<a class="headerlink" href="#12-set-a-minimum-tls-protocol-version" title="Permanent link"> </a></h3>
<p>In order to prevent any form of downgrade attack on the TLS protocol level, set on both clients and server the minimum version.
If your clients and servers are modern (2.3.3+), they should support TLSv1.2 just fine, so you can configure it like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>tls-version-min 1.2
</code></pre></div>
<p>Note that this will break OpenVPN versions 2.3.2 and earlier, which only expect TLSv1.0 handshake signatures.</p>
<h3 id="13-migrate-to-openvpn-nl">13: Migrate to OpenVPN-NL<a class="headerlink" href="#13-migrate-to-openvpn-nl" title="Permanent link"> </a></h3>
<p>All items listed above are general TLS configuration options for your PKI, basically.
For further security enhancements one could use <a href="https://openvpn.fox-it.com/">OpenVPN-NL</a>, a fork off OpenVPN using PolarSSL instead of OpenSSL as cryptography library and having more secure defaults.
The items in below don't apply to OpenVPN-NL, because it already incorporates settings strong enough, or doesn't even offer other.
I recommend at least considering using OpenVPN-NL instead of OpenVPN on your server and/or clients.
In the meantime, apply the remaining items to a regular OpenVPN installation; they're chosen to be compatible with OpenVPN-NL.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>When using OpenVPN-NL, shell commands in this post should use <code>openvpn-nl</code> rather than <code>openvpn</code>.</p>
</div>
<p><em>Disclaimer: I'm employed by <a href="https://www.fox-it.com/">Fox-IT</a>, the company developing <a href="https://openvpn.fox-it.com/">OpenVPN-NL</a> in collaboration with the OpenVPN community.</em></p>
<h3 id="14-set-a-stronger-cipher">14: Set a stronger cipher<a class="headerlink" href="#14-set-a-stronger-cipher" title="Permanent link"> </a></h3>
<p><em>Not required for OpenVPN-NL.</em></p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>cipher AES-256-CBC
</code></pre></div>
<p>OpenVPN's default encryption algorithm <code>BF-CBC</code> (Blowfish, block-cipher) with a 128-bit (variable) key size.
While it's certainly not a terrible or 'broken' cipher like <a href="https://community.qualys.com/blogs/securitylabs/2013/03/19/rc4-in-tls-is-broken-now-what">RC4</a> or single-DES, I prefer a more modern and widely used cipher like AES.
Out of all other strong options, I've chosen AES-256-CBC for interoperability with OpenVPN-NL.</p>
<p>To see which other ciphers your version of OpenVPN supports, run</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openvpn --show-ciphers</span>
</code></pre></div>
<h3 id="15-use-sha-2-for-message-authentication">15: Use SHA-2 for message authentication<a class="headerlink" href="#15-use-sha-2-for-message-authentication" title="Permanent link"> </a></h3>
<p><em>Not required for OpenVPN-NL.</em></p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>auth SHA-256
</code></pre></div>
<p>Message authentication is what's referred to as HMAC.
Using a HMAC is to ensure the encrypted data hasn't been altered in transit.
OpenVPN's default setting is <code>SHA-1</code>.
<a href="http://en.wikipedia.org/wiki/SHA-1">SHA-1</a> is <a href="https://www.schneier.com/blog/archives/2005/02/sha1_broken.html">considered weak</a> since 2005 and Microsoft has announced their <a href="http://blogs.technet.com/b/pki/archive/2013/11/12/sha1-deprecation-policy.aspx">deprecation policy</a> for it.
The <a href="http://en.wikipedia.org/wiki/SHA-2">SHA-2</a> set of hashing algorithms are considered stronger and one should use those in favour of SHA-1 whenever possible.
Out of the other strong options, I've chosen SHA-256 for interoperability with OpenVPN-NL.</p>
<p>To see which other HMACs are supported by your OpenVPN, run</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openvpn --show-digests</span>
</code></pre></div>
<h3 id="16-limit-the-list-of-supported-tls-ciphersuites">16: Limit the list of supported TLS ciphersuites<a class="headerlink" href="#16-limit-the-list-of-supported-tls-ciphersuites" title="Permanent link"> </a></h3>
<p><em>Not required for OpenVPN-NL.</em></p>
<p>Limiting the list of TLS ciphers is recommended, because you want to enforce a secure cipher suite for the connection.
Basically, you want to strip down the list OpenVPN offers a client to the ones you think are secure.
This eliminates downgrade attacks or security issues in client configurations as well as the use of plain RSA key exchange.</p>
<p>From the OpenVPN manpage:</p>
<blockquote>
<p>A list <code>l</code> of allowable TLS ciphers delimited by a colon (":").
If you require a high level of security, you may want to set this parameter manually, to prevent a version rollback attack where a man-in-the-middle attacker tries to force two peers to negotiate to the lowest level of security they both support.</p>
</blockquote>
<p>To see which TLS ciphers are supported by your OpenVPN, run</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openvpn --show-tls</span>
</code></pre></div>
<p>I'd recommend setting a small list of ciphers matching a <a href="https://wiki.mozilla.org/Security/Server_Side_TLS">commonly recommended set</a>, for example:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
</code></pre></div>
<p>The list above is basically a combination of the two strongest ciphers with regular OpenVPN (OpenSSL 1.0.1) and the two strongest offered by OpenVPN-NL, included for interoperability reasons.
All of these are DHE or ECDHE enabled ciphersuites which means key exchange is done with Diffie-Hellman enabled, providing forward secrecy.
ECDHE is preferred, because it is faster by the use of elliptic curve cryptography rather than the much slower plain Diffie-Hellman.</p>
<p>Cipher suite names have to be specified in IANA format, rather than OpenSSL format as you would normally find on the Internet.
A mapping table from IANA to/from OpenSSL cipher suite names is available in the OpenVPN source code <code>src/openvpn/ssl.c</code>, for currently stable version 2.3.6 that is from <a href="http://sourceforge.net/p/openvpn/openvpn-testing/ci/v2.3.6/tree/src/openvpn/ssl.c#l116">line 116</a>.
Thanks to 'sosbar' in the comments to point that out to me.</p>
<h2 id="suggestions-or-remarks">Suggestions or remarks?<a class="headerlink" href="#suggestions-or-remarks" title="Permanent link"> </a></h2>
<p>Is something unclear?
Do you have a remark on this post?
Did I miss out on an important thing?
Please let me know!
Use the comments below or send me a <a href="https://twitter.com/gertvdijk">tweet</a>.</p>
<p>Do you like the post?
Feel free to retweet or share in other ways!</p>How to take down SSLv3 in your network using iptables firewall? (POODLE)2014-10-16T00:00:00+02:002014-10-16T23:20:00+02:00Gert van Dijktag:blog.g3rt.nl,2014-10-16:/take-down-sslv3-using-iptables.html<p>Here's how to take down SSLv3 down using iptables if you can't shut it down in your application.</p>
<blockquote class="twitter-tweet" lang="en"><p>Is there a way to drop/stop <a href="https://twitter.com/hashtag/SSLv3?src=hash">#SSLv3</a> handshake packets in iptables on routers?
<a href="https://twitter.com/hashtag/POODLE?src=hash">#POODLE</a> <a href="https://twitter.com/hashtag/Linux?src=hash">#Linux</a> <a href="https://twitter.com/hashtag/security?src=hash">#security</a></p>β Gert van Dijk β οΈ (@gertvdijk) <a href="https://twitter.com/gertvdijk/status/522443873080324096">October 15, 2014</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<h2 id="tldr">TL;DR:<a class="headerlink" href="#tldr" title="Permanent link"> </a></h2>
<p>Only do this if you can't change your application/device in your network and you want to kill SSLv3 on network level.</p>
<p>Example for logging SSLv3 outbound connections on your host. Run:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">iptables -I OUTPUT 1 \</span>
<span style="color: #cccccc"> -p tcp \! -f --dport 443 \</span>
<span style="color: #cccccc"> -m state --state ESTABLISHED -m u32 --u32 \</span>
<span style="color: #cccccc"> "0>>22&0x3C@ 12>>26&0x3C@ 0 & 0xFFFFFF00=0x16030000 && \</span>
<span style="color: #cccccc"> 0>>22&0x3C@ 12>>26&0x3C@ 2 & 0xFF=0x01 && \</span>
<span style="color: #cccccc"> 0>>22&0x3C@ 12>>26&0x3C@ 7 & 0xFFFF=0x0300" \</span>
<span style="color: #cccccc"> -j LOG --log-prefix "SSLv3 Client Hello detected: " # or -j DROP ...</span>
</code></pre></div>
<p>The above will <strong>only log</strong> them and show up in your kernel messages like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>[11318.883379] SSLv3 Client Hello detected: IN= OUT=eth0 SRC=192.168.31.10 DST=74.208.162.100 LEN=154 [...]
[11318.889486] SSLv3 Client Hello detected: IN= OUT=eth0 SRC=192.168.31.10 DST=74.208.162.100 LEN=154 [...]
</code></pre></div>
<p>Adjust this <code>OUTPUT</code> chain to <code>INPUT</code> for an SSL server and to the <code>FORWARD</code> chain on a gateway. And change <code>-j LOG --log-prefix ...</code> to <code>-j DROP</code> or <code>-j REJECT</code> to actually block the traffic.</p>
<div class="admonition danger">
<p class="admonition-title">Danger</p>
<p>Don't use this in production to mitigate POODLE now. That would be silly. I haven't tested this on an actual downgrade attack. But let me know what you think.</p>
</div>
<h2 id="do-i-need-this">Do I need this?<a class="headerlink" href="#do-i-need-this" title="Permanent link"> </a></h2>
<p>No, you should be disabling SSLv3 properly on application level. However, there could be use cases for blocking a protocol on network level. For example:</p>
<ul>
<li>An appliance or server application that does not offer a way to disable SSLv3 (or at least CBC ciphersuites in SSLv3).</li>
<li>You can't change all browser client configurations in your network, but you abolutely need to turn off SSLv3.</li>
</ul>
<p>Then a firewall block could be your last resort.</p>
<h2 id="iptables-and-ssl-level">iptables and SSL-level?<a class="headerlink" href="#iptables-and-ssl-level" title="Permanent link"> </a></h2>
<p>How can one determine the SSLv3 packets?
iptables can't work at that level, right?
Well, it can.
I remembered from the Heartbleed attack, I saw some magic iptables commands mitigating Heartbleed.
Will it work for killing SSLv3 too?
Let's find out!</p>
<h2 id="sslv3-versus-tlsv1">SSLv3 versus TLSv1+<a class="headerlink" href="#sslv3-versus-tlsv1" title="Permanent link"> </a></h2>
<p>First, let's take a look at how an SSLv3 handshake looks in Wireshark.</p>
<p><img alt="SSLv3 handshake in Wireshark" src="https://blog.g3rt.nl/images/20141016_sslv3_wireshark.png"/></p>
<p>And below the handshake for TLSv1 (TLSv1.2 in this case).</p>
<p><img alt="TLSv1.2 handshake in Wireshark" src="https://blog.g3rt.nl/images/20141016_tlsv1_wireshark.png"/></p>
<p>Based on pure assumption I'm deciding that this selection based on those four conditions is quite reliable for machting a true SSLv3 Client Hello packet.</p>
<h2 id="selecting-the-right-data-in-iptables">Selecting the right data in iptables<a class="headerlink" href="#selecting-the-right-data-in-iptables" title="Permanent link"> </a></h2>
<p>In order to quickly detect the packet based on the highlighted items, one can use the <a href="http://www.netfilter.org/documentation/HOWTO/netfilter-extensions-HOWTO-3.html#ss3.21"><code>u32</code> iptables extension</a>.
It handles matches per 4 bytes (32 bits) and here's a quickly drawn scheme on the offsets and masks in order to match the right stuff in the payload.</p>
<p><img alt="Data and offsets for fields in SSLv3 Client Hello payload" src="https://blog.g3rt.nl/images/20141016_sslv3_payload_scheme.png"/></p>
<p>The hard part was selecting the right offsets for the TCP payload.
I've found this tutorial on the u32 iptables extension which explains how <code>0>>22&0x3C@ 12>>26&0x3C@</code> sets the offset right on the TCP payload: <a href="http://www.stearns.org/doc/iptables-u32.current.html">http://www.stearns.org/doc/iptables-u32.current.html</a>.
Nice!</p>
<p>Then it's just doing the work using the documentation, resulting in the following u32 'selector' for SSLv3 Client Hello.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>0>>22&0x3C@ 12>>26&0x3C@ 0 & 0xFFFFFF00=0x16030000 && \
0>>22&0x3C@ 12>>26&0x3C@ 2 & 0xFF=0x01 && \
0>>22&0x3C@ 12>>26&0x3C@ 7 & 0xFFFF=0x0300
</code></pre></div>
<h2 id="inserting-it-in-your-iptables-chain">Inserting it in your iptables chain<a class="headerlink" href="#inserting-it-in-your-iptables-chain" title="Permanent link"> </a></h2>
<p>If you want to protect your host running SSL clients (e.g. your workstation with a browser), you'll want to block SSLv3 Client Hello <em>outbound</em> packets (<code>OUTPUT</code> chain) and SSLv3 Server Hello <em>inbound</em> packets (<code>INPUT</code> chain).
I haven't constructed nor tested any rule for Server Hello messages yet, but see below in 'Future work' for more on that.</p>
<p>You probably want to insert the rule <em>above</em> the regular rules.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>iptables<span style="color: #666666"> </span>-L<span style="color: #666666"> </span>-n<span style="color: #666666"> </span>-v
</span><span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">Chain OUTPUT (policy DROP 280 packets, 118K bytes)</span>
<span style="color: #cccccc"> pkts bytes target prot opt in out source destination </span>
<span style="color: #cccccc"> 2990 265K ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0 /* 001 accept all outbound to lo interface */</span>
<span style="color: #cccccc"> 841K 138M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* 003 accept related outbound established rules */ ctstate RELATED,ESTABLISHED</span>
<span style="color: #cccccc"> 1449 93561 ACCEPT udp -- * * 0.0.0.0/0 192.168.30.2 multiport dports 53 /* 100 accept outbound DNS to local recursor */</span>
<span style="color: #cccccc">[...]</span>
</code></pre></div>
<p>If the above is like what you have, then insert a rule on top like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>iptables<span style="color: #666666"> </span>-I<span style="color: #666666"> </span>OUTPUT<span style="color: #666666"> </span><span style="color: #51b2fd">1</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span>...<span style="color: #d0d0d0">]</span>
</code></pre></div>
<p>Before we actually construct the full example we'll have to look at performance.</p>
<h2 id="performance">Performance<a class="headerlink" href="#performance" title="Permanent link"> </a></h2>
<p>Using solely the above u32 rule would be VERY slow as it parses all traffic on that chain.
How can we optimize this?
Be more specific!</p>
<p>Prepend the following extra selectors.</p>
<ul>
<li><code>-p tcp \! -f --dport 443</code> <br/> Selects only TCP with destination port 443 which are not fragments.</li>
<li><code>-m state --state ESTABLISHED</code> <br/> Only look at packets which have obtained the ESTABLISHED state (TCP).</li>
</ul>
<p>This then finally leads up to the example at the top of the page which should result in an output like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>sudo<span style="color: #666666"> </span>iptables<span style="color: #666666"> </span>-L<span style="color: #666666"> </span>-n<span style="color: #666666"> </span>-v
</span><span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">Chain OUTPUT (policy DROP 280 packets, 118K bytes)</span>
<span style="color: #cccccc"> pkts bytes target prot opt in out source destination </span>
<span style="color: #cccccc"> 0 0 LOG tcp !f * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 state ESTABLISHED u32 "0x0>>0x16&0x3c@0xc>>0x1a&0x3c@0x0&0xffffff00=0x16030000&&0x0>>0x16&0x3c@0xc>>0x1a&0x3c@0x2&0xff=0x1&&0x0>>0x16&0x3c@0xc>>0x1a&0x3c@0x7&0xffff=0x300" LOG flags 0 level 4 prefix "SSLv3 Client Hello detected: "</span>
<span style="color: #cccccc"> 2990 265K ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0 /* 001 accept all outbound to lo interface */</span>
<span style="color: #cccccc"> 841K 138M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* 003 accept related outbound established rules */ ctstate RELATED,ESTABLISHED</span>
<span style="color: #cccccc"> 1449 93561 ACCEPT udp -- * * 0.0.0.0/0 192.168.30.2 multiport dports 53 /* 100 accept outbound DNS to local recursor */</span>
<span style="color: #cccccc">[...]</span>
</code></pre></div>
<h2 id="testing">Testing<a class="headerlink" href="#testing" title="Permanent link"> </a></h2>
<p>You can use the SSLv3 server from https://www.poodletest.com/ like this in an OpenSSL command line:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>openssl<span style="color: #666666"> </span>s_client<span style="color: #666666"> </span>-connect<span style="color: #666666"> </span>sslv3.dshield.org:443
</span><span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">SSL-Session:</span>
<span style="background-color: #404040"><span style="color: #cccccc"> Protocol : SSLv3</span>
</span><span style="color: #cccccc">[...]</span>
</code></pre></div>
<p>Verify the session is established using SSLv3. Then watch your logs.</p>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>Use <code>iptables -Z</code> to clear the counters and check the <code>iptables -L -n -v</code> output repeatedly for increasing counters.</p>
</div>
<h2 id="further-work-todo">Further work, TODO<a class="headerlink" href="#further-work-todo" title="Permanent link"> </a></h2>
<ul>
<li>Improving the conditionals for starting on u32 matching to improve performance. <br/> Maybe using the <code>-m connbytes --connbytes 0:<nbytes></code> to grab only a certain amount of data from a stream. See also <a href="https://gist.github.com/lhaagsma/f2de4647e841f4696175">this discussion</a>.</li>
<li>Include blocking SSLv3 Sever Hello. A downgrade attack or even an alternative way to connect with SSLv3 may not always involve a Client Hello as pointed out in the above discussion.</li>
<li>Ensuring/testing on a gateway (<code>FORWARD</code> chain) and how to configure this on OpenWRT for example.</li>
</ul>
<p>More ideas? Or suggestions? Let me know via Twitter or in the Disqus comments below!</p>
<blockquote class="twitter-tweet" lang="en"><p>Blog post: How to block <a href="https://twitter.com/hashtag/SSLv3?src=hash">#SSLv3</a> using iptables for <a href="https://twitter.com/hashtag/POODLE?src=hash">#POODLE</a>. <a href="https://twitter.com/hashtag/Linux?src=hash">#Linux</a>
<a href="https://t.co/Xv4oua53Ql">https://t.co/Xv4oua53Ql</a></p>β Gert van Dijk β οΈ (@gertvdijk) <a href="https://twitter.com/gertvdijk/status/522505699884945408">October 15, 2014</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>How to allow a Google Chromecast in your host firewall?2014-09-30T21:45:00+02:002014-09-30T21:45:00+02:00Gert van Dijktag:blog.g3rt.nl,2014-09-30:/allow-google-chromecast-host-firewall-iptables.html<p>Here's how to enable your iptables configuration for the Google Chromecast.</p><h2 id="tldr">TL;DR:<a class="headerlink" href="#tldr" title="Permanent link"> </a></h2>
<p>Provided you've allowed 'established' traffic.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">export CHROMECAST_IP=10.1.2.3 # Adjust to the Chromecast IP in your local network</span>
<span style="color: #cccccc">iptables -A INPUT -s ${CHROMECAST_IP}/32 -p udp -m multiport --sports 32768:61000 -m multiport --dports 32768:61000 -m comment --comment "Allow Chromecast UDP data (inbound)" -j ACCEPT</span>
<span style="color: #cccccc">iptables -A OUTPUT -d ${CHROMECAST_IP}/32 -p udp -m multiport --sports 32768:61000 -m multiport --dports 32768:61000 -m comment --comment "Allow Chromecast UDP data (outbound)" -j ACCEPT</span>
<span style="color: #cccccc">iptables -A OUTPUT -d ${CHROMECAST_IP}/32 -p tcp -m multiport --dports 8008:8009 -m comment --comment "Allow Chromecast TCP data (outbound)" -j ACCEPT</span>
<span style="color: #cccccc">iptables -A OUTPUT -d 239.255.255.250/32 -p udp --dport 1900 -m comment --comment "Allow Chromecast SSDP" -j ACCEPT</span>
</code></pre></div>
<h2 id="when-do-you-need-this">When do you need this?<a class="headerlink" href="#when-do-you-need-this" title="Permanent link"> </a></h2>
<p>Well, I really like to tighten my hosts' firewall configuration very strict, not just on my routers, but also on the clients/workstations.
For instance, on my laptop I like to have all firewall chains to have a <code>DROP</code> <em>policy</em>.
E.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">iptables -P INPUT DROP</span>
<span style="color: #cccccc">iptables -P OUTPUT DROP</span>
<span style="color: #cccccc">iptables -P FORWARD DROP</span>
</code></pre></div>
<p>Then I'll be making the regular exemptions in my firewall for whatever doesn't work anymore.
For example, I'll see my DHCP is blocked, then my DNS requests, then HTTP, etc., and at some point I came to using the Chromecast from Chrome.</p>
<h2 id="finding-the-right-firewall-exemptions">Finding the right firewall exemptions<a class="headerlink" href="#finding-the-right-firewall-exemptions" title="Permanent link"> </a></h2>
<p>At the point I noticed I couldn't access my Chromecast with this firewall policy, I found out using <code>tcpdump</code> and Wireshark that it was quite complicated, using mDNS, some other multicast UDP, random high UDP ports, TCP ports 8008 and 8009, etc.</p>
<p>With the help of some extra googling I stumbled upon a document from Cisco: <a href="http://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/7-6/chromecastDG76/ChromecastDG76.html">Chromecast Deployment Guide, Release 7.6</a>.
While it was quite useful, it was a lot of <em>bla bla</em> and also about wireless network tweaking which I wasn't really interested in.</p>
<p>To cut to the chase, here's what you need:</p>
<ul>
<li>
<p>Allow high UDP ports both incoming and outgoing.<br />
"High ports" are the local ports usually ranging 32768-61000 on most Linux systems.</p>
</li>
<li>
<p>Allow both TCP ports 8008 and 8009 outbound to the Chromecast device.<br />
I've noticed most reference only use 8008, but that didn't do it for me and saw outbound connection to port 8009 being blocked.</p>
</li>
<li>
<p>Allow the special SSDP packets outbound (which is UDP traffic to the multicast IP <code>239.255.255.250</code>, destination port 1900).<br />
As far as I understand, a Chromecast app should send information over SSDP if it wants to discover the Chromecasts in the network. The Chromecast should then reply to the source IP it was given.</p>
</li>
<li>
<p>In the <code>INPUT</code> chain, allow <code>ESTABLISHED</code> traffic.<br />
This is very common to have in your firewall, usually even together with <code>RELATED</code>, e.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT</span>
</code></pre></div>
</li>
</ul>Enable your Debian Squeeze LTS security support now!2014-06-05T22:00:00+02:002014-06-05T22:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2014-06-05:/how-to-enable-debian-squeeze-lts-security-support.html<p>Debian Sqeeze support ended May 31st, 2014. Further updates are received through the Squeeze-LTS distribution and requires manual configuration on your server!</p>
<h2 id="tldr">TL;DR<a class="headerlink" href="#tldr" title="Permanent link"> </a></h2>
<p>The extended Squeeze security support is opt-in. One has to enable the <code>squeeze-lts</code> distribution in the APT sources; it is <em>not</em> provided through <code>security.debian.org</code>. Don't forget to do that if you're still running Squeeze, or you will not receive important security updates!</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #ed9d13">"deb http://ftp.nl.debian.org/debian squeeze-lts main non-free contrib"</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>>><span style="color: #666666"> </span>sudo<span style="color: #666666"> </span>tee<span style="color: #666666"> </span>-a<span style="color: #666666"> </span>/etc/apt/sources.list<span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">&&</span><span style="color: #666666"> </span>sudo<span style="color: #666666"> </span>apt-get<span style="color: #666666"> </span>update<span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">&&</span><span style="color: #666666"> </span>sudo<span style="color: #666666"> </span>apt-get<span style="color: #666666"> </span>dist-upgrade
</code></pre></div>
<h2 id="about-debian-lts">About Debian LTS<a class="headerlink" href="#about-debian-lts" title="Permanent link"> </a></h2>
<p>Standard security support for Debian GNU/Linux release 6.0 (codename Squeeze) has been terminated May 31st, 2014.
While users should have been putting effort in upgrading to Wheezy, a separate team of volunteers <a href="https://www.debian.org/News/2014/20140424">announced</a> prolonged security support on April 24, 2014 for at least until February 2016.
Cool to see a Debian release to be supported for such a longer time, but I'm just not too excited about it, because I really think this should be announced from the initial release on.
Now this is just discouraging users to upgrade to Wheezy and the user base of Squeeze-LTS is just only going to become smaller and smaller.</p>
<h2 id="not-enabled-by-default">Not enabled by default<a class="headerlink" href="#not-enabled-by-default" title="Permanent link"> </a></h2>
<p>What's not mentioned in the announcement of Squeeze-LTS, is that one needs to enable the <code>squeeze-lts</code> release channel in order to receive the updates.
I believe a lot of users will not notice it and their systems will likely not be receiving critical security updates released this month like the <a href="https://lists.debian.org/debian-lts-announce/2014/06/msg00000.html">gnutls26 update</a> and the <a href="https://lists.debian.org/debian-lts-announce/2014/06/msg00002.html">openssl update</a>.</p>
<p>With a fully updates Squeeze machine, I can confirm the updates are <em>not</em> pushed through <code>security.debian.org</code>.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>apt-cache<span style="color: #666666"> </span>policy<span style="color: #666666"> </span>libgnutls26
<span style="color: #cccccc">libgnutls26:</span>
<span style="color: #cccccc">Installed: 2.8.6-1+squeeze3</span>
<span style="color: #cccccc">Candidate: 2.8.6-1+squeeze3</span>
<span style="color: #cccccc">Version table:</span>
<span style="color: #cccccc">*** 2.8.6-1+squeeze3 0</span>
<span style="color: #cccccc"> 500 http://security.debian.org/ squeeze/updates/main amd64 Packages</span>
<span style="color: #cccccc"> 100 /var/lib/dpkg/status</span>
<span style="color: #cccccc"> 2.8.6-1+squeeze2 0</span>
<span style="color: #cccccc"> 500 http://ftp.nl.debian.org/debian/ squeeze/main amd64 Packages</span>
</code></pre></div>
<p>GnuTLS version <code>2.8.6-1+squeeze3</code> is from March 2014 (<a href="http://metadata.ftp-master.debian.org/changelogs//main/g/gnutls26/gnutls26_2.8.6-1+squeeze3_changelog">changelog</a>) and clearly does not include fixes for those like the 'Client Hello' vulnerability (CVE-2014-3466) fixed in the <a href="https://lists.debian.org/debian-lts-announce/2014/06/msg00000.html">announced 2.8.6-1+squeeze4 version</a> by the Squeeze-LTS team.</p>
<h2 id="how-to-enable-squeeze-lts">How to enable Squeeze LTS?<a class="headerlink" href="#how-to-enable-squeeze-lts" title="Permanent link"> </a></h2>
<p>Simply put, add the <code>squeeze-lts</code> distribution from your regular Debian mirror to your APT's <code>sources.list</code> configuration. E.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>deb http://ftp.nl.debian.org/debian squeeze main non-free contrib
deb http://ftp.nl.debian.org/debian squeeze-updates main non-free contrib
deb http://security.debian.org squeeze/updates main non-free contrib
# Squeeze LTS
deb http://ftp.nl.debian.org/debian squeeze-lts main non-free contrib
</code></pre></div>
<p>Then run</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>apt-get<span style="color: #666666"> </span>update
</code></pre></div>
<p>to update the lists.</p>
<p>Now you can enjoy new security updates pending:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>apt-cache<span style="color: #666666"> </span>policy<span style="color: #666666"> </span>libgnutls26
<span style="color: #cccccc">libgnutls26:</span>
<span style="color: #cccccc">Installed: 2.8.6-1+squeeze3</span>
<span style="color: #cccccc">Candidate: 2.8.6-1+squeeze4</span>
<span style="color: #cccccc">Version table:</span>
<span style="color: #cccccc"> 2.8.6-1+squeeze4 0</span>
<span style="color: #cccccc"> 500 http://ftp.nl.debian.org/debian/ squeeze-lts/main amd64 Packages</span>
<span style="color: #cccccc">*** 2.8.6-1+squeeze3 0</span>
<span style="color: #cccccc"> 500 http://security.debian.org/ squeeze/updates/main amd64 Packages</span>
<span style="color: #cccccc"> 100 /var/lib/dpkg/status</span>
<span style="color: #cccccc"> 2.8.6-1+squeeze2 0</span>
<span style="color: #cccccc"> 500 http://ftp.nl.debian.org/debian/ squeeze/main amd64 Packages</span>
</code></pre></div>
<p>Don't forget to actually upgrade and to restart all related services!</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>apt-get<span style="color: #666666"> </span>dist-upgrade
<span style="color: #cccccc">Reading package lists... Done</span>
<span style="color: #cccccc">Building dependency tree</span>
<span style="color: #cccccc">Reading state information... Done</span>
<span style="color: #cccccc">Calculating upgrade... Done</span>
<span style="color: #cccccc">The following packages will be upgraded:</span>
<span style="color: #cccccc">libgnutls26</span>
<span style="color: #cccccc">[...]</span>
<span style="color: #aaaaaa"># </span>/etc/init.d/exim4<span style="color: #666666"> </span>restart
</code></pre></div>Getting started with the SmartCard-HSM2014-04-23T23:33:00+02:002014-12-10T22:30:00+01:00Gert van Dijktag:blog.g3rt.nl,2014-04-23:/getting-started-smartcard-hsm.html<p>The SmartCard-HSM smart card is one of the few cards offering Elliptic Curve cryptography with OpenSC support. Here's how to get started (RSA-only for now, though).</p>
<h2 id="about-the-smartcard-hsm">About the SmartCard-HSM<a class="headerlink" href="#about-the-smartcard-hsm" title="Permanent link"> </a></h2>
<p>Most smart cards compatible with OpenSC will offer only RSA 1024 bits key usage.
Some will offer also RSA 2048 bits, but only very few do more.
The <a href="http://g10code.com/p-card.html">OpenPGP card</a> and the Cryptostick are known to be able to handle 4096 bits keys.
However, the storage capacity of the OpenPGP is quite limited and the Cryptostick isn't available anymore (a new version is <a href="http://shop.crypto-stick.com/de/home/11-crypto-stick-14-beta.html">in beta</a>, though).</p>
<p>Elliptic Curve Cryptography done on smart cards is very rare.
And this is where the <a href="http://www.smartcard-hsm.com/">SmartCard-HSM</a> stands out and it even works with OpenSC!</p>
<p>In this article I will only cover on initializing the tokend and usage of simple RSA keys, the usage on ECC is for a future post. </p>
<div class="admonition tip">
<p class="admonition-title">Update</p>
<p>OpenSC 0.14 release fixed the use of EC keys with the SmartCard-HSM.
Grab it in my PPA for Ubuntu: <a href="https://launchpad.net/~gertvdijk/+archive/ubuntu/opensc-backports">opensc-backports PPA</a>
Expect some update on this how-to soon.</p>
</div>
<h2 id="opensc-support-in-ubuntudebian">OpenSC support in Ubuntu/Debian<a class="headerlink" href="#opensc-support-in-ubuntudebian" title="Permanent link"> </a></h2>
<p>Unfortunately not all of the patches have made it in OpenSC 0.13.0 for properly supporting the SmartCard-HSM.
The first issue I've encountered was the inability to work with RSA keys of 2048 bits.</p>
<p>Here's my bug report and fix taken from upstream for that: <a href="https://bugs.launchpad.net/ubuntu/+source/opensc/+bug/1311921">LP Bug #1311921</a>.
Ubuntu Trusty (14.04) and up includes the fix for this in <a href="https://launchpad.net/ubuntu/+source/opensc/0.13.0-3ubuntu4.1"><code>0.13.0-3ubuntu4.1</code></a>.</p>
<p>A similar issue is present for EC keys and requires OpenSC 0.14 for it to work.
Ubuntu Vivid (15.04) and Debian Jessie will include 0.14, but for any current Ubuntu version you can install the backports I've compiled in my <a href="https://launchpad.net/~gertvdijk/+archive/ubuntu/opensc-backports">opensc-backports PPA</a>.</p>
<p>For Debian support, the maintainer decided not to backport the fixes to Wheezy (7.x).
The Jessie (8.x) release includes 0.14 and should be all fine.</p>
<h2 id="initializing-the-card">Initializing the card<a class="headerlink" href="#initializing-the-card" title="Permanent link"> </a></h2>
<p>First, choose a SO-PIN of exactly 16 hexadecimal characters.
It will be stored internally as an 8-byte key.</p>
<p>My recommendation is to store the SO PIN in a very safe place.</p>
<p>Secondly, make up a (user) PIN.
It can be of any length from four up to sixteen ASCII characters with an important limitation listed below.</p>
<div class="admonition danger">
<p class="admonition-title">Danger</p>
<p>Some really big warnings here!
Don't say I didn't warn you.
Read them.</p>
</div>
<ul>
<li>
<p>Any further reinitialization of the card will require you to enter the SO PIN.
Unlike other smart cards or tokens, you cannot erase the card without the knowledge of the SO PIN!</p>
</li>
<li>
<p>The counter for wrong-entered SO PINs is 15 and is not unblockable.
So, yes, entering the SO PIN wrong for 15 times renders the card unusable forever.</p>
</li>
<li>
<p>The initialization tool only takes SO PIN and PIN using command line arguments (opposed to reading from keyboard interactively).
Any user on the system can see these secrets by doing a full process listing on the machine.
You may want to run the initialization in an offline Live CD environment.
Also, the full PINs will end up in your shell history.
Bash allows you to omit this by prepending the command with a space.</p>
</li>
<li>
<p>Card versions up to 1.0 will not allow you to change this SO PIN ever again.
To check your card version, run <code>pkcs11-tool -L</code>, e.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>pkcs11-tool<span style="color: #666666"> </span>--module<span style="color: #666666"> </span>opensc-pkcs11.so<span style="color: #666666"> </span>-L
</code></pre></div>
</li>
<li>
<p>While the (User) PIN can be changed, it's not allowed to change it to another size (length).
Andreas Schwier of CardContact confirmed this unfortunate restriction in private communication while noting that this holds for cards at least up to and including card version 1.2.
Changing the size of the PIN currently requires reinitialization of the card (wiping its contents!).</p>
</li>
</ul>
<p>Initialize using the <code>sc-hsm-tool</code> command shipped with OpenSC:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>DO<span style="color: #666666"> </span>NOT<span style="color: #666666"> </span>CONTINUE<span style="color: #666666"> </span>UNLESS<span style="color: #666666"> </span>YOU<span style="color: #a61717; background-color: #e3d2d2">'</span>VE<span style="color: #666666"> </span>READ<span style="color: #666666"> </span>THE<span style="color: #666666"> </span>WARNINGS<span style="color: #666666"> </span>LISTED<span style="color: #666666"> </span>ABOVE
<span style="color: #aaaaaa"># </span>prepend<span style="color: #666666"> </span>the<span style="color: #666666"> </span><span style="color: #2fbccd">command</span><span style="color: #666666"> </span>with<span style="color: #666666"> </span>a<span style="color: #666666"> </span>space<span style="color: #666666"> </span>to<span style="color: #666666"> </span>avoid<span style="color: #666666"> </span>saving<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">in</span><span style="color: #666666"> </span>bash<span style="color: #666666"> </span><span style="color: #2fbccd">history</span>
<span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>sc-hsm-tool<span style="color: #666666"> </span>--initialize<span style="color: #666666"> </span>--so-pin<span style="color: #666666"> </span><span style="color: #51b2fd">0123012301230123</span><span style="color: #666666"> </span>--pin<span style="color: #666666"> </span>abcdABCD12349876
</span></code></pre></div>
<p>Unfortunately, it's not (yet) possible to initialize the card using the more generic <code>pkcs15-init</code> tool.</p>
<h2 id="generating-first-keypair">Generating first keypair<a class="headerlink" href="#generating-first-keypair" title="Permanent link"> </a></h2>
<p>To create an RSA keypair of 2048 bits (maximum size for SmartCard-HSM) use <code>pkcs11-tool</code> like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>pkcs11-tool<span style="color: #666666"> </span>--module<span style="color: #666666"> </span>opensc-pkcs11.so<span style="color: #666666"> </span>-l<span style="color: #666666"> </span>--keypairgen<span style="color: #666666"> </span>--key-type<span style="color: #666666"> </span>rsa:2048<span style="color: #666666"> </span>--id<span style="color: #666666"> </span><span style="color: #51b2fd">10</span><span style="color: #666666"> </span>--label<span style="color: #666666"> </span><span style="color: #ed9d13">"My first RSA keypair"</span>
</code></pre></div>
<p>It will ask for your PIN and the keypair will be generated on-card.
I've made up an ID and label here; choose your own.</p>
<p>To see the keypair listed, you could use</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>pkcs11-tool<span style="color: #666666"> </span>--module<span style="color: #666666"> </span>opensc-pkcs11.so<span style="color: #666666"> </span>--list-objects
</span><span style="color: #cccccc">Using slot 1 with a present token (0x1)</span>
<span style="color: #cccccc">Public Key Object; RSA 2048 bits</span>
<span style="color: #cccccc">label: My first RSA keypair</span>
<span style="color: #cccccc">ID: 10</span>
<span style="color: #cccccc">Usage: none</span>
</code></pre></div>
<p>Note it lists <code>Usage: none</code>.
This really should be something like <code>Usage: encrypt, verify, wrap</code>.
I'm not sure what's going on here, but I am able to use the public key to encrypt data.</p>
<h3 id="obtaining-the-public-key">Obtaining the public key<a class="headerlink" href="#obtaining-the-public-key" title="Permanent link"> </a></h3>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>pkcs15-tool<span style="color: #666666"> </span>--read-public-key<span style="color: #666666"> </span><span style="color: #51b2fd">10</span><span style="color: #666666"> </span>><span style="color: #666666"> </span>/tmp/publickey.pem
</code></pre></div>
<p>where <code>10</code> is the ID of the object.</p>
<p>In case you see</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>error: object not found
</code></pre></div>
<p>you're probably hit by <a href="https://bugs.launchpad.net/ubuntu/+source/opensc/+bug/1311921">LP Bug #1311921</a>.
Install your system updates or if you're still running anything older than Trusty, scroll up and install the package from <a href="https://launchpad.net/~gertvdijk/+archive/ubuntu/opensc-backports">my PPA</a>.</p>
<h2 id="encrypting-and-decrypting-data">Encrypting and Decrypting data<a class="headerlink" href="#encrypting-and-decrypting-data" title="Permanent link"> </a></h2>
<p>RSA is not meant to encrypt/decrypt large amounts of data.
245 bytes is the maximum message length for a 2048 bits key as described in an earlier article: <a href="https://blog.g3rt.nl/luks-smartcard-or-token.html">How to encrypt a LUKS container using a smart card or token</a>.</p>
<ol>
<li>
<p>Create a random piece of data and hash it so we can identify it's the same later:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>dd<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #d0d0d0">=</span>/dev/urandom<span style="color: #666666"> </span><span style="color: #40ffff">of</span><span style="color: #d0d0d0">=</span>/tmp/mydata<span style="color: #666666"> </span><span style="color: #40ffff">bs</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">1</span><span style="color: #666666"> </span><span style="color: #40ffff">count</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">245</span>
<span style="color: #aaaaaa">$ </span>sha256sum<span style="color: #666666"> </span>/tmp/mydata
</code></pre></div>
</li>
<li>
<p>Encrypt the data using the public key, in a PKCS#1 padding:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>openssl<span style="color: #666666"> </span>rsautl<span style="color: #666666"> </span>-inkey<span style="color: #666666"> </span>/tmp/publickey.pem<span style="color: #666666"> </span>-pubin<span style="color: #666666"> </span>-encrypt<span style="color: #666666"> </span>-pkcs<span style="color: #666666"> </span>-in<span style="color: #666666"> </span>/tmp/mydata<span style="color: #666666"> </span>-out<span style="color: #666666"> </span>/tmp/encrypteddata.pkcs1
</code></pre></div>
</li>
<li>
<p>Decrypt the data again using the smart card:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>pkcs15-crypt<span style="color: #666666"> </span>--decipher<span style="color: #666666"> </span>--key<span style="color: #666666"> </span><span style="color: #51b2fd">10</span><span style="color: #666666"> </span>--input<span style="color: #666666"> </span>/tmp/encrypteddata.pkcs1<span style="color: #666666"> </span>--pkcs1<span style="color: #666666"> </span>--raw<span style="color: #666666"> </span>><span style="color: #666666"> </span>/tmp/decrypteddata
<span style="color: #aaaaaa">$ </span>sha256sum<span style="color: #666666"> </span>/tmp/decrypteddata
</code></pre></div>
</li>
</ol>
<h2 id="more-applications">More applications<a class="headerlink" href="#more-applications" title="Permanent link"> </a></h2>
<p>Many many applications can be found for using any smart card or token.
On some of them I have written (or will write later):</p>
<ul>
<li><a href="https://blog.g3rt.nl/luks-smartcard-or-token.html">Disk encryption using LUKS</a></li>
<li>SSH publickey authentication</li>
<li>Set up your own PKI</li>
<li><a href="https://blog.g3rt.nl/openvpn-security-tips.html">OpenVPN</a></li>
<li>Second-factor in workstation login</li>
<li>E-mail encryption/signing</li>
<li>GnuPG</li>
<li>SSL Client authentication on web servers</li>
</ul>
<h2 id="references">References<a class="headerlink" href="#references" title="Permanent link"> </a></h2>
<ul>
<li><a href="https://github.com/OpenSC/OpenSC/wiki/SmartCardHSM">OpenSC wiki page on SmartCard-HSM</a></li>
<li><a href="https://github.com/OpenSC/OpenSC">OpenSC on GitHub</a> and the <a href="https://github.com/CardContact/OpenSC">fork by CardContact</a></li>
<li><a href="http://www.cardcontact.de/products/SmartCard-HSM_V1.1.pdf">SmartCard-HSM product flyer</a> (PDF)</li>
</ul>How to encrypt a LUKS container using a smart card or token2014-04-20T12:00:00+02:002014-04-20T12:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2014-04-20:/luks-smartcard-or-token.html<p>In this post I'll be uncovering how to utilize your smart card (or token) with LUKS encrypted volumes.</p>
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link"> </a></h2>
<p>Encrypting whole volumes is usually done using LUKS volumes, managed using <a href="https://code.google.com/p/cryptsetup/">cryptsetup</a>.
The LUKS format allow one to use several key slots, i.e. allowing multiple passphrases and/or keyfiles to unlock the volume.
Out of the box, it does not allow you to use a security device like a smart card or token to store the secret.
In this article I'll show you how enable a smart card or token device.</p>
<h2 id="prerequisites">Prerequisites<a class="headerlink" href="#prerequisites" title="Permanent link"> </a></h2>
<p>For this approach to work, I assume you have...</p>
<ul>
<li>your smart card or token working with OpenSC</li>
<li>initialized it and have at least one RSA keypair installed</li>
</ul>
<p>I.e. <code>pkcs15-tool --list-keys</code> shows list your keys installed.</p>
<p>I've tested this on Ubuntu 12.04 and 14.04 using a <a href="http://www.ftsafe.com/product/epass/epass2003">Feitian ePass2003</a>, but it should work with any OpenSC-compatible device.</p>
<h2 id="using-an-encrypted-keyfile">Using an encrypted keyfile<a class="headerlink" href="#using-an-encrypted-keyfile" title="Permanent link"> </a></h2>
<p>LUKS can't talk to your token directly and only knows about static key files, so here's the basic principle of using an encrypted keyfile.
Create a fully random keyfile, add that to a LUKS keyslot and encrypt the keyfile using your RSA public key.
To unlock the volume, one can then decrypt the keyfile using the token providing the decrypted keyfile to cryptsetup.</p>
<p>Here's how to do that in practice:</p>
<ol>
<li>
<p>Creating a random keyfile can be accomplished by using <code>dd</code> and the <code>/dev/urandom</code> random source.
The size of the key is chosen to be 245 bytes here.
Why 245 bytes?
See the section <em>Key size of 245 bytes</em> below.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">dd if=/dev/urandom of=/tmp/mykey bs=1 count=245</span>
</code></pre></div>
<p><em>Note that this approach temporarily stores the encryption key in plaintext on your system. To avoid this, use the script as mentioned below.</em></p>
</li>
<li>
<p>Add the key to the LUKS volume.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">cryptsetup luksAddKey /dev/sdb1 /tmp/mykey</span>
</code></pre></div>
<p>Replace <code>/dev/sdb1</code> with the path to your encrypted volume.</p>
</li>
<li>
<p>Obtain the public key of the keypair on your token, store it to a file.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">pkcs15-tool --read-public-key 1234 > /tmp/publickey.pem</span>
</code></pre></div>
<p>Replace the <code>1234</code> with your key ID as listed using <code>pkcs15-tool --list-public-keys</code>.</p>
</li>
<li>
<p>Encrypt the keyfile using the RSA public key.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">openssl rsautl -inkey /tmp/publickey.pem -pubin -encrypt -pkcs -in /tmp/mykey -out /tmp/encryptedkey.pkcs1</span>
</code></pre></div>
</li>
<li>
<p>Remove the plaintext key.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">shred /tmp/mykey</span>
</code></pre></div>
</li>
</ol>
<p>Then test the ability to unlock the volume by decrypting the encrypted encryption key:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">pkcs15-crypt --decipher --input /tmp/encryptedkey.pkcs1 --pkcs1 --raw | cryptsetup --key-file=- luksOpen /dev/sdb1</span>
</code></pre></div>
<p>Some more explanation on the command above:</p>
<ul>
<li>combination of <code>--pkcs1</code> and <code>--raw</code> ensures 'raw' output of an PKCS1 padded encrypted file.</li>
<li><code>--key-file=-</code> will have cryptsetup read the keyfile from <code>stdin</code> (output of the command left of the pipe).</li>
</ul>
<h2 id="enhance-security-avoid-temporary-key-storage">Enhance security: avoid temporary key storage<a class="headerlink" href="#enhance-security-avoid-temporary-key-storage" title="Permanent link"> </a></h2>
<p>By using a small script to generate the temporary key and feeding that to both OpenSSL and cryptsetup is more secure, because it avoids saving the key file to disk.
The trick is to use <code>tee</code> and some subshells to copy <code>stdout</code>.
In order to use the script without interaction this does require you to set up a temporary static key which will be replaced by this script.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">dd if=/dev/urandom of=/tmp/mykey bs=1 count=256</span>
<span style="color: #cccccc">cryptsetup luksAddKey /dev/sdb1 /tmp/mykey</span>
</code></pre></div>
<p>Then execute this script (save it to a file, <code>chmod +x file</code>, <code>./file</code>):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #ababab; font-style: italic">#!/bin/bash</span>
<span style="color: #40ffff">LUKS_TARGET_DEV</span><span style="color: #d0d0d0">=</span>/dev/sdb1
<span style="color: #40ffff">LUKS_CURRENTKEY_FILE</span><span style="color: #d0d0d0">=</span>/tmp/mykey
<span style="color: #40ffff">RSA_PUBKEY_FILE</span><span style="color: #d0d0d0">=</span>/tmp/publickey.pem
<span style="color: #40ffff">ENCRYPTED_KEYFILE</span><span style="color: #d0d0d0">=</span>/tmp/encryptedkey.pkcs1
<span style="color: #ababab; font-style: italic"># generate random key (unique, plaintext)</span>
<span style="color: #ababab; font-style: italic"># 245 bytes is based on 2048 bits RSA key minus 11 bytes PKCS1 padding</span>
dd<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #d0d0d0">=</span>/dev/urandom<span style="color: #666666"> </span><span style="color: #40ffff">of</span><span style="color: #d0d0d0">=</span>/dev/stdout<span style="color: #666666"> </span><span style="color: #40ffff">bs</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">1</span><span style="color: #666666"> </span><span style="color: #40ffff">count</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">245</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># Copy stdout pipe to multiple subshells, avoiding the need to save the \</span>
<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># secret key to a temporary file. \</span>
<span style="color: #666666"> </span>tee<span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>><span style="color: #d0d0d0">(</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># encrypt the key using the public key \</span>
<span style="color: #666666"> </span>openssl<span style="color: #666666"> </span>rsautl<span style="color: #666666"> </span>-inkey<span style="color: #666666"> </span><span style="color: #ed9d13">${</span><span style="color: #40ffff">RSA_PUBKEY_FILE</span><span style="color: #ed9d13">}</span><span style="color: #666666"> </span>-pubin<span style="color: #666666"> </span>-encrypt<span style="color: #666666"> </span>-pkcs<span style="color: #666666"> </span>-out<span style="color: #666666"> </span><span style="color: #ed9d13">${</span><span style="color: #40ffff">ENCRYPTED_KEYFILE</span><span style="color: #ed9d13">}</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">)</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>><span style="color: #d0d0d0">(</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># add the key to the LUKS container \</span>
<span style="color: #666666"> </span>cryptsetup<span style="color: #666666"> </span>--key-file<span style="color: #d0d0d0">=</span><span style="color: #ed9d13">${</span><span style="color: #40ffff">LUKS_CURRENTKEY_FILE</span><span style="color: #ed9d13">}</span><span style="color: #666666"> </span>luksChangeKey<span style="color: #666666"> </span><span style="color: #ed9d13">${</span><span style="color: #40ffff">LUKS_TARGET_DEV</span><span style="color: #ed9d13">}</span><span style="color: #666666"> </span>/dev/stdin<span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">)</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>><span style="color: #666666"> </span>/dev/null
</code></pre></div>
<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link"> </a></h2>
<h3 id="key-size-of-245-bytes">Key size of 245 bytes<a class="headerlink" href="#key-size-of-245-bytes" title="Permanent link"> </a></h3>
<p>An RSA key of 2048 bits (256 bytes) should in theory allow you to use a message of up to the same size.
However, by using a PKCS1 padding (which uses a minimum of 11 bytes for padding) the maximum message to encrypt/decrypt with the RSA key is 256 - 11 = 245 bytes.</p>
<h2 id="todo">TODO<a class="headerlink" href="#todo" title="Permanent link"> </a></h2>
<ul>
<li>Use of PKCS#1 v2.1 padding instead of v1.5.
At the time of writing both OpenSSL's <code>rsautl</code> and OpenSC's <code>pkcs15-crypt</code> don't support the use of v2.1 on the command line.</li>
<li>Explain a bit more on the PKCS#1 format, regarding padding.</li>
</ul>
<h2 id="more-later">More later...<a class="headerlink" href="#more-later" title="Permanent link"> </a></h2>
<p>In a later post I'll show how this applies to full disk encryption (root filesystem); unlocking the device from the initramfs during boot.</p>Announcing my blog2014-04-19T20:00:00+02:002014-04-19T20:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2014-04-19:/announcing-my-blog.html<p>Announcing this blog; to share my knowledge with the world.</p><h2 id="from-dokuwiki-to-pelican">From DokuWiki to Pelican<a class="headerlink" href="#from-dokuwiki-to-pelican" title="Permanent link"> </a></h2>
<p>I've found Pelican to be a very elegant way of publishing my site, so I decided to migrate from my DokuWuki installation to Pelican.</p>
<p>In the next few weeks I'll be migrating current posts with relevance on the old site to here.</p>Python virtualenv for development and deployment2013-03-27T00:28:00+01:002013-03-27T11:38:00+01:00Gert van Dijktag:blog.g3rt.nl,2013-03-27:/python-virtualenv-for-development-and-deployment.html<p>In this how-to I'd like to introduce you to using pip, virtualenv and virtualenvwrapper.</p>
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link"> </a></h2>
<p>This how-to is provided because I've spent several hours in order to accomplish a simple recommendation in the Django documentation to use virtual environments for developing.
I wasn't able to find a concise and simple way to do this and had to take a dive in several pieces of general Python tools: <code>pip</code>, <code>virtualenv</code> and <code>virtualenvwrapper</code>.</p>
<h2 id="setting-up-the-environment">Setting up the environment<a class="headerlink" href="#setting-up-the-environment" title="Permanent link"> </a></h2>
<p>I'm running Ubuntu 12.04 and the Python 2.7.3 default interpreter.
This is supposed to work on Debian Wheezy (7.0) too.</p>
<h3 id="install-pip">Install pip<a class="headerlink" href="#install-pip" title="Permanent link"> </a></h3>
<p>In order to be able to install <a href="https://pypi.python.org/">PyPi</a> packages, we need to use the <a href="http://www.pip-installer.org/en/latest/">pip</a> installer.</p>
<p>Install it:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">sudo apt-get install python-pip</span>
</code></pre></div>
<h3 id="initialize-a-virtualenv">Initialize a virtualenv<a class="headerlink" href="#initialize-a-virtualenv" title="Permanent link"> </a></h3>
<p>Using a <code>virtualenv</code> in Python to develop and run your application in is highly recommended in order to be able to satisfy the latest dependencies on the usually 'older' Ubuntu system.</p>
<p>I really like to use <a href="https://pypi.python.org/pypi/virtualenvwrapper">virtualenvwrapper</a> for this task.</p>
<p>Let's install it:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">sudo pip install virtualenvwrapper</span>
</code></pre></div>
<p>Above is the last global <code>pip</code> command we've used and the last requiring root privileges - everything from here on will be inside a virtual environment within your home directory!</p>
<div class="admonition warning">
<p class="admonition-title">Heads up!</p>
<p>Avoid the use of <code>pip</code> for installing Python packages system-wide.
I'm encouraging the use of <code>pip</code>, but only <em>inside</em> a virtualenv.
We need to install <a href="https://pypi.python.org/pypi/virtualenv">virtualenv</a> and <a href="https://pypi.python.org/pypi/virtualenvwrapper">virtualenvwrapper</a>, so those are the only ones we're installing system-wide.</p>
</div>
<p>The actual environments are to be located in a directory of your choice. I like to use the suggested default <code>~/Envs</code>.
Execute these commands (customize the directory if you like):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">export WORKON_HOME=~/Envs</span>
<span style="color: #cccccc">mkdir -p $WORKON_HOME</span>
<span style="color: #cccccc">source /usr/local/bin/virtualenvwrapper.sh</span>
</code></pre></div>
<p>Install the last line in your shell startup file as well, unless you want to type it in every newly opened shell:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.profile</span>
</code></pre></div>
<h2 id="using-virtualenvs-with-virtualenvwrapper">Using virtualenvs with virtualenvwrapper<a class="headerlink" href="#using-virtualenvs-with-virtualenvwrapper" title="Permanent link"> </a></h2>
<h3 id="creating-a-new-environment">Creating a new environment<a class="headerlink" href="#creating-a-new-environment" title="Permanent link"> </a></h3>
<p>To create a new environment with the name <code>env1</code>, do:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">mkvirtualenv env1</span>
</code></pre></div>
<p>Create as many as you like.</p>
<h3 id="managing-environments">Managing environments<a class="headerlink" href="#managing-environments" title="Permanent link"> </a></h3>
<p>The <code>/usr/local/bin/virtualenvwrapper.sh</code> script added to your shell startup provided a prompt prefix to list which environment you're currently in.
E.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">(env2)</span> <span style="color: #aaaaaa">gert@gert-laptop:~ $</span>
</code></pre></div>
<p>It also provides a whole set of commands to manage them:</p>
<ul>
<li>List environments: <code>lsvirtualenv</code> or <code>workon</code> (without arguments)</li>
<li>Copy: <code>cpvirtualenv env1 env2</code> (you'll now be on ''env2'')</li>
<li>Remove: <code>rmvirtualenv</code></li>
</ul>
<h3 id="enteringleaving-environments">Entering/Leaving environments<a class="headerlink" href="#enteringleaving-environments" title="Permanent link"> </a></h3>
<p>To enter another environment:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">workon env2</span>
</code></pre></div>
<p>to move to ''env2''.</p>
<p>To leave any environment and go back to the general system environment (this was hard to find - undocumented!):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">deactivate</span>
</code></pre></div>
<h3 id="environments-vs-projects">Environments vs Projects<a class="headerlink" href="#environments-vs-projects" title="Permanent link"> </a></h3>
<p>From the <a href="http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html#project-directory-management">Virtualenvwrapper documentation on Project management</a> you'll notice it has a feature to use in your projects.
What's this about?
Well, it's very simpleβsuppose you have a Python code project and you want to bind it one-to-one to a specific environment, then this is what projects are for.</p>
<p>Binding a virtualenv to a project directory will result in nothing more than changing to the project directory whenever you use the <code>workon</code> command.</p>
<p>Setting/changing a project directory to the environment is just this (run being inside the project dir):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">setvirtualenvproject</span>
</code></pre></div>
<h2 id="package-management">Package management<a class="headerlink" href="#package-management" title="Permanent link"> </a></h2>
<h3 id="using-pip">Using pip<a class="headerlink" href="#using-pip" title="Permanent link"> </a></h3>
<p>The version of <code>pip</code> we installed on the system is different from the one installed in the environments.
Please check this to have at least version 1.3 in your environments:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">gert@gert-laptop:~ $ </span>pip<span style="color: #666666"> </span>--version
</span><span style="color: #cccccc">pip 1.0 from /usr/lib/python2.7/dist-packages (python 2.7)</span>
<span style="background-color: #404040"><span style="color: #aaaaaa">gert@gert-laptop:~ $ </span>workon<span style="color: #666666"> </span>env2
</span><span style="background-color: #404040"><span style="color: #aaaaaa">(env2)</span> <span style="color: #aaaaaa">gert@gert-laptop:~ $ </span>pip<span style="color: #666666"> </span>--version
</span><span style="color: #cccccc">pip 1.3.1 from /home/gert/Envs/env1/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg (python 2.7)</span>
</code></pre></div>
<p><em>Within</em> a virtual environment...</p>
<ul>
<li><strong>Install</strong>: <code>pip install packagename</code></li>
<li><strong>Install</strong> specific version (e.g. Django 1.5): <code>pip install Django==1.5</code></li>
<li><strong>List</strong> all available packages in the environment: <code>pip list</code></li>
<li><strong>List</strong> packages only installed locally: <code>pip list --local</code></li>
<li><strong>List</strong> in the special Requirements format (see below): <code>pip freeze</code></li>
<li><strong>Uninstall</strong>: <code>pip uninstall packagename</code></li>
</ul>
<h3 id="requirements-file">Requirements file<a class="headerlink" href="#requirements-file" title="Permanent link"> </a></h3>
<p>Just like you installed the package using <code>pip</code>, you can create a specification for all the packages your project requires.
For each package put it on a new line.
Example:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>Django==1.5
ipython
</code></pre></div>
<p>Or list those using</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">pip freeze</span>
</code></pre></div>
<p>This file is usually called <code>requirements.txt</code> for a Python project.
Users are then able to satisfy these easily by issuing:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">pip install -r requirements.txt</span>
</code></pre></div>How to retain exit status codes through pipes in Bash scripts2012-08-10T12:00:00+02:002012-08-10T12:00:00+02:00Gert van Dijktag:blog.g3rt.nl,2012-08-10:/retain-exit-status-code-through-pipe.html<p>Piping the output of a critical process through a compressor? Then you're most likely not interested in the exit status code of the compressor.</p><p>Suppose you have a line</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>mysqldump<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>gzip
</code></pre></div>
<p>in your script.
Then the exit status code will be of <code>gzip</code>, rather than <code>mysqldump</code>, while the most likely process to fail here is <code>mysqldump</code>.</p>
<p>To fix this, add this at the top of your bash scripts:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #2fbccd">set</span><span style="color: #666666"> </span>-o<span style="color: #666666"> </span>pipefail
</code></pre></div>Arduino serial-over-TCP using your OpenWRT device2012-01-13T00:59:00+01:002012-01-13T00:59:00+01:00Gert van Dijktag:blog.g3rt.nl,2012-01-13:/arduino-serial-over-tcp-openwrt.html<p>Here's how to control your Arduino over the network. Connect it using a serial cable to an OpenWRT device and use a network socket (TCP) to connect to it remotely.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Tested using Arduino Mega (AtMega 2560) and OpenWRT backfire 10.03.1 on a TP-Link TL-WR1043ND.</p>
</div>
<h2 id="basics">Basics<a class="headerlink" href="#basics" title="Permanent link"> </a></h2>
<p>Install the following packages on your OpenWRT device:</p>
<ul>
<li><code>kmod-usb-acm</code></li>
<li><code>socat</code></li>
</ul>
<p>Connect your Arduino on the USB port and check the kernel messages.
It should mention a new <code>ttyACM0</code> device; that's our Arduino.</p>
<p>Now fire up <code>socat</code> to listen on a TCP port and connect it to <code>/dev/ttyACM0</code> like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>socat<span style="color: #666666"> </span>tcp-l:1234,reuseaddr,fork<span style="color: #666666"> </span>file:/dev/ttyACM0,nonblock,raw,echo<span style="color: #d0d0d0">=</span><span style="color: #51b2fd">0</span>,waitlock<span style="color: #d0d0d0">=</span>/var/run/tty,b9600
</code></pre></div>
<p>Change your baudrate (here <code>9600</code>) and port (here <code>1234</code>) accordingly.
Now you should be able to telnet to the device on another machine like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>telnet<span style="color: #666666"> </span><span style="color: #51b2fd">192</span>.168.1.1<span style="color: #666666"> </span><span style="color: #51b2fd">1234</span>
</code></pre></div>
<h2 id="background-service">Background service<a class="headerlink" href="#background-service" title="Permanent link"> </a></h2>
<p>In order to create it a always-running background service we should hook in to OpenWRT's services, but this will do as well:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #ababab; font-style: italic">#!/bin/ash</span>
<span style="color: #40ffff">DEV</span><span style="color: #d0d0d0">=</span>/dev/ttyACM0
<span style="color: #40ffff">PORT</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">50000</span>
<span style="color: #40ffff">BAUD</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">9600</span>
<span style="color: #6ebf26; font-weight: bold">while</span><span style="color: #666666"> </span>true<span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">do</span><span style="color: #666666"> </span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span>-e<span style="color: #666666"> </span><span style="color: #40ffff">$DEV</span><span style="color: #666666"> </span><span style="color: #d0d0d0">]</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span>
<span style="color: #666666"> </span>socat<span style="color: #666666"> </span>tcp-l:<span style="color: #40ffff">$PORT</span>,reuseaddr,fork<span style="color: #666666"> </span>file:<span style="color: #40ffff">$DEV</span>,nonblock,raw,echo<span style="color: #d0d0d0">=</span><span style="color: #51b2fd">0</span>,waitlock<span style="color: #d0d0d0">=</span>/var/run/tty,b<span style="color: #40ffff">$BAUD</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">else</span>
<span style="color: #666666"> </span>sleep<span style="color: #666666"> </span><span style="color: #51b2fd">2</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span>
<span style="color: #6ebf26; font-weight: bold">done</span>
</code></pre></div>
<p>Put it somewhere and make it executable (<code>chmod +x filename</code>).
Add it to boot time by editing <code>/etc/rc.local</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #ababab; font-style: italic"># make sure it's above 'exit 0'!</span>
/path/to/the-script_above.sh<span style="color: #666666"> </span><span style="color: #d0d0d0">&</span>
<span style="color: #2fbccd">exit</span><span style="color: #666666"> </span><span style="color: #51b2fd">0</span>
</code></pre></div>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>The appended <code>&</code> puts it in the background so <code>rc.local</code> will finish (and the boot of the device finishes) with the script in the background.</p>
</div>Creating a my.cnf for easy command line operations2011-11-09T11:15:00+01:002011-11-09T15:36:00+01:00Gert van Dijktag:blog.g3rt.nl,2011-11-09:/create-my-cnf-for-easy-mysql-cli.html<p>How to avoid typing your MySQL password every time.</p><p>Place in <code>~/.my.cnf</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #6ebf26; font-weight: bold">[client]</span>
<span style="color: #bbbbbb">user</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ed9d13">gert</span>
<span style="color: #bbbbbb">password</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span><span style="color: #ed9d13">mypassword</span>
</code></pre></div>
<p>Then you can use <code>mysql</code> and <code>mysqldump</code> operations without entering a password or specifying a username.</p>How to regenerate SSH host keys2011-11-08T15:17:00+01:002011-11-08T15:17:00+01:00Gert van Dijktag:blog.g3rt.nl,2011-11-08:/regenerate-ssh-host-keys.html<p>In some cases you'd want to regenerate SSH host keys, e.g. after an upgrade of OpenSSH with new features or when you put the machine in another network. Here's how.</p><h2 id="regenerate-ssh-host-keys">Regenerate SSH host keys<a class="headerlink" href="#regenerate-ssh-host-keys" title="Permanent link"> </a></h2>
<p>This applies to OpenSSH installation on both Debian and Ubuntu.</p>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Please note that while you could do issue the commands below over SSH maintaining the session, it is recommended to do this using the console.
You could get yourself isolated from SSH if the connection gets dropped.</p>
</div>
<p>First, remove the existing host keys:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">rm /etc/ssh/ssh_host_*</span>
</code></pre></div>
<p>Then we reconfigure OpenSSH, which will trigger the keys to be regenerated:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">dpkg-reconfigure openssh-server</span>
</code></pre></div>Patch for overheating ThinkPads2011-10-01T18:48:00+02:002012-04-23T02:16:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-10-01:/thinkfan-patch-overheating-bug.html<p>An issue with thinkfan in Ubuntu before the 12.04 release will cause your Thinkpad to overheat. Here's how to patch manually.</p>
<div class="admonition tip">
<p class="admonition-title">"Update April 2012</p>
<p>Ubuntu versions 12.04 and up includes a more recent version of thinkfan (<code>0.7.3-1</code>) which should not need this patch.</p>
</div>
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link"> </a></h2>
<p>Due to <a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/751689">Ubuntu bug #751689</a> and <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=610722">Debian bug 610722</a> your ThinkPad can easily overheat and do an emergency shutdown.
To fix this, you can use the <code>thinkfan</code> daemon to control your fan speeds.
Unfortunately, it comes with a bug (at the time of writing) and it actually slows down your fan when it should be at full speed.
Here's how to patch the Ubuntu package.</p>
<p>Ubuntu and Debian are known to be affected, including Debian Squeeze, Ubuntu 9.10, 10.04, 10.10, 11.04 and 11.10.
At the time of writing a fix was <a href="http://sourceforge.net/projects/thinkfan/files/">released upstream</a> <code>0.7.3</code> on 2011-09-20, but it wasn't yet incorporated in the stable release channel of Debian/Ubuntu.
Rather than upgrading to <code>0.7.3</code> which might introduce other bugs, this is how to patch the stable version shipped with your OS.</p>
<div class="admonition info">
<p class="admonition-title">Heads up!</p>
<p>In order to download, patch and build the new <code>thinkfan</code> package, you don't have to become root.
Only when installing the the required packages for building (and the resulting package), you need to be root.</p>
</div>
<h2 id="prerequisites">Prerequisites<a class="headerlink" href="#prerequisites" title="Permanent link"> </a></h2>
<p>First, install the packages we need to be able to compile the package.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">sudo apt-get install debhelper build-essential dpkg-dev</span>
<span style="color: #cccccc">sudo apt-get build-dep thinkfan</span>
</code></pre></div>
<p>In the last part we will build a Debian/Ubuntu package.</p>
<h2 id="download-source-and-patch">Download source and patch<a class="headerlink" href="#download-source-and-patch" title="Permanent link"> </a></h2>
<p>Now, somewere in your homedir, create a directory to work in.
I chose <code>~/thinkfan</code> and start working in here:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">mkdir ~/thinkfan; cd ~/thinkfan</span>
</code></pre></div>
<p>Download the original Ubuntu package (yes, we <code>apt-get</code> as user):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">apt-get source thinkfan</span>
</code></pre></div>
<p>You should end up with similar files:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>drwxrwxr-x 4 gert gert 4096 2011-10-01 18:26 thinkfan-0.7.1
-rw-rw-r-- 1 gert gert 8326 2010-08-12 23:05 thinkfan_0.7.1-2.diff.gz
-rw-rw-r-- 1 gert gert 1124 2010-08-12 23:05 thinkfan_0.7.1-2.dsc
-rw-rw-r-- 1 gert gert 21413 2010-08-12 23:05 thinkfan_0.7.1.orig.tar.gz
</code></pre></div>
<p>Now change the file <code>thinkfan-0.7.1/system.c</code> with this patch:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #d22323">--- thinkfan-0.7.1.orig/system.c</span>
<span style="color: #589819">+++ thinkfan-0.7.1/system.c</span>
<span style="color: #ffffff; text-decoration: underline">@@ -83,7 +83,7 @@</span>
<span style="color: #666666"> </span> else {
<span style="color: #666666"> </span> if (unlikely(cur_lvl == INT_MIN)) strcpy(buf, "level disengaged\n");
<span style="color: #666666"> </span> else snprintf(buf, 10, "level %d\n", cur_lvl);
<span style="color: #d22323">- if (unlikely(write(ibm_fan, buf, 8) != 8)) {</span>
<span style="color: #589819">+ if (unlikely(write(ibm_fan, buf, strlen(buf)) != strlen(buf))) {</span>
<span style="color: #666666"> </span> showerr(IBM_FAN);
<span style="color: #666666"> </span> message(LOG_ERR, MSG_ERR_FANCTRL);
<span style="color: #666666"> </span> errcnt++;
</code></pre></div>
<p>Basically, the hardcoded length of 8 of the string <code>buf</code> is now variable.
This fixes the issue that <code>level 127</code> is cut to <code>level 1</code> and actually slows down the fan instead of getting it to the highest level.</p>
<p>You can either apply this patch manually by editing the file on line 86, or apply it with <code>patch</code>.</p>
<h2 id="updating-the-changelog">Updating the changelog<a class="headerlink" href="#updating-the-changelog" title="Permanent link"> </a></h2>
<p>Before we can build a new package, it is mandatory to update the changelog with an entry explaining what's changed.
It will be used to update more metadata as well including the version number.
That is important for the package manager to identify this as a new version.</p>
<p>The changelog file is located in <code>thinkfan-0.7.1/debian/changelog</code>.
Prepend the following (example):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>thinkfan (0.7.1-2+gert) unstable; urgency=high
* patch from debian bug #610722 applied
-- Gert van Dijk <gert@gertvandijk.net> Wed, 25 May 2011 19:10:00 +0200
</code></pre></div>
<p>Note the exact formatting of the entry (the package builder is very strict in its checks) and the version number suffix.</p>
<h2 id="building-patched-version">Building patched version<a class="headerlink" href="#building-patched-version" title="Permanent link"> </a></h2>
<p>Now we move to the actual source directory:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">cd thinkfan-0.7.1</span>
</code></pre></div>
<p>And start building the package:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">dpkg-buildpackage</span>
</code></pre></div>
<p>After some time of compiling and packaging, we should have the following file in the original working directory (one level lower):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>drwxrwxr-x 4 gert gert 4096 2011-10-01 18:40 thinkfan-0.7.1
-rw-rw-r-- 1 gert gert 8326 2010-08-12 23:05 thinkfan_0.7.1-2.diff.gz
-rw-rw-r-- 1 gert gert 1124 2010-08-12 23:05 thinkfan_0.7.1-2.dsc
-rw-rw-r-- 1 gert gert 1250 2011-10-01 18:40 thinkfan_0.7.1-2+gert_amd64.changes
-rw-r--r-- 1 gert gert 34308 2011-10-01 18:40 thinkfan_0.7.1-2+gert_amd64.deb
-rw-rw-r-- 1 gert gert 8640 2011-10-01 18:40 thinkfan_0.7.1-2+gert.diff.gz
-rw-rw-r-- 1 gert gert 898 2011-10-01 18:40 thinkfan_0.7.1-2+gert.dsc
-rw-rw-r-- 1 gert gert 21413 2010-08-12 23:05 thinkfan_0.7.1.orig.tar.gz
</code></pre></div>
<p>Et voilΓ : we have <code>thinkfan_0.7.1-2+gert_amd64.deb</code>!</p>
<h2 id="installing-the-package">Installing the package<a class="headerlink" href="#installing-the-package" title="Permanent link"> </a></h2>
<p>Install it using <code>dpkg</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">sudo dpkg -i thinkfan_0.7.1-2+gert_amd64.deb</span>
</code></pre></div>
<p>Don't forget to <a href="https://blog.g3rt.nl/thinkfan-usage.html">configure thinkfan</a>.</p>
<h2 id="see-also">See also<a class="headerlink" href="#see-also" title="Permanent link"> </a></h2>
<ul>
<li><a href="https://blog.g3rt.nl/thinkfan-usage.html">Configuring and using thinkfan</a></li>
</ul>Building and using shared libraries in Linux2011-09-30T00:24:00+02:002011-09-30T01:12:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-09-30:/building-using-shared-libraries-linux.html<p>On this page I will use a simple example on how to use shared libraries on Linux in C++ and using the GNU g++ compiler. You will be able to compile your library and application separately.</p>
<h2 id="building-and-using-shared-libraries">Building and using shared libraries<a class="headerlink" href="#building-and-using-shared-libraries" title="Permanent link"> </a></h2>
<p>On this page I will use a simple example on how to use shared libraries on Linux in C++ and using the GNU <code>g++</code> compiler.
You will be able to compile your library and application separately.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>In Windows we have DLLs, in Linux/Unix we have shared libraries.
It's the same!</p>
</div>
<p>However, you can't specify your library on runtime easily.
It is possible though, using <code>dlopen()</code>, but that's not really easy.</p>
<h2 id="design">Design<a class="headerlink" href="#design" title="Permanent link"> </a></h2>
<p>A very simple UML model about the design in this example.</p>
<p><img alt="UML model" src="https://blog.g3rt.nl/images/20110930_building_and_using_shared_libraries_linux_gcc_diagram.png"/></p>
<p>Our file structure will look like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>foo/
foo.cpp
foo.h
libfoo.so
qux/
main.cpp
qux
</code></pre></div>
<h2 id="building-a-shared-library-libfoo">Building a shared library libfoo<a class="headerlink" href="#building-a-shared-library-libfoo" title="Permanent link"> </a></h2>
<p><code>foo/foo.h</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cd2828; font-weight: bold">#ifndef LIBFOO_H</span>
<span style="color: #cd2828; font-weight: bold">#define LIBFOO_H</span>
<span style="color: #6ebf26; font-weight: bold">class</span><span style="color: #666666"> </span><span style="color: #71adff; text-decoration: underline">Foo</span><span style="color: #666666"> </span><span style="color: #d0d0d0">{</span>
<span style="color: #6ebf26; font-weight: bold">public</span><span style="color: #d0d0d0">:</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">Foo();</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">int</span><span style="color: #666666"> </span><span style="color: #71adff">bar</span><span style="color: #d0d0d0">()</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">const</span><span style="color: #d0d0d0">;</span>
<span style="color: #d0d0d0">};</span>
<span style="color: #cd2828; font-weight: bold">#endif</span>
</code></pre></div>
<p><code>foo/foo.cpp</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cd2828; font-weight: bold">#include</span><span style="color: #666666"> </span><span style="color: #ababab; font-style: italic">"foo.h"</span>
<span style="color: #d0d0d0">Foo::Foo(){}</span>
<span style="color: #6ebf26; font-weight: bold">int</span><span style="color: #666666"> </span><span style="color: #d0d0d0">Foo::bar()</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">const</span><span style="color: #666666"> </span><span style="color: #d0d0d0">{</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">return</span><span style="color: #666666"> </span><span style="color: #51b2fd">10</span><span style="color: #d0d0d0">;</span>
<span style="color: #d0d0d0">}</span>
</code></pre></div>
<p>Now, in the folder <code>foo/</code> run the following command:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">g++ -shared -fPIC foo.cpp -o libfoo.so</span>
</code></pre></div>
<p>The options explained:</p>
<ul>
<li><code>-shared</code>βto create a shared object file (shared library)</li>
<li><code>-fPIC</code>βabout memory management.
In short: memory addresses are mapped independent of the location.</li>
<li><code>foo.cpp</code>βthe C++ file(s) to compile</li>
<li><code>-o libfoo.so</code>βinstead of outputting to ''a.out'', use a sensible name.</li>
</ul>
<p>Now check if you can see if all methods are included:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>nm<span style="color: #666666"> </span>foo/libfoo.so
</span><span style="color: #cccccc">0000000000200e20 a _DYNAMIC</span>
<span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">00000000000005ec T _ZN3FooC1Ev</span>
<span style="color: #cccccc">00000000000005ec T _ZN3FooC2Ev</span>
<span style="color: #cccccc">00000000000005f6 T _ZNK3Foo3barEv</span>
<span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">00000000000005c0 t frame_dummy</span>
</code></pre></div>
<p>These are all very cryptic names, called <em>symbols</em>.
To make it human readable, use <code>c++filt</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span>_ZN3FooC1Ev<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>c++filt<span style="color: #666666"> </span>
</span><span style="color: #cccccc">Foo::Foo()</span>
<span style="background-color: #404040"><span style="color: #aaaaaa">$ </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span>_ZNK3Foo3barEv<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>c++filt<span style="color: #666666"> </span>
</span><span style="color: #cccccc">Foo::bar() const</span>
</code></pre></div>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>In case you don't have the utilities <code>c++filt</code> or <code>nm</code> then install the package <code>binutils</code> on Debian/Ubuntu and -derivatives.</p>
</div>
<p>Obviously, our shared library is created successfully.
Now let's move on to how to use it.</p>
<h2 id="using-shared-libraries">Using shared libraries<a class="headerlink" href="#using-shared-libraries" title="Permanent link"> </a></h2>
<p><code>qux/main.cpp</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cd2828; font-weight: bold">#include</span><span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"><iostream></span>
<span style="color: #cd2828; font-weight: bold">#include</span><span style="color: #666666"> </span><span style="color: #ababab; font-style: italic">"../foo/foo.h"</span>
<span style="color: #6ebf26; font-weight: bold">int</span><span style="color: #666666"> </span><span style="color: #71adff">main</span><span style="color: #d0d0d0">(){</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">Foo</span><span style="color: #666666"> </span><span style="color: #d0d0d0">myfoo;</span>
<span style="color: #666666"> </span><span style="color: #d0d0d0">std::cout</span><span style="color: #666666"> </span><span style="color: #d0d0d0"><<</span><span style="color: #666666"> </span><span style="color: #ed9d13">"Output of Foo.bar()="</span><span style="color: #666666"> </span><span style="color: #d0d0d0"><<</span><span style="color: #666666"> </span><span style="color: #d0d0d0">myfoo.bar()</span><span style="color: #666666"> </span><span style="color: #d0d0d0"><<</span><span style="color: #666666"> </span><span style="color: #d0d0d0">std::endl;</span>
<span style="color: #d0d0d0">}</span>
</code></pre></div>
<p>Note that we use the <code>foo.h</code> file from the <code>foo/</code> folder as we need it to compile the <code>qux</code> application.</p>
<p>Now we can compile the <code>qux</code> application as follows:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">g++ -lfoo -L ../foo/ main.cpp -o qux</span>
</code></pre></div>
<p>The options explained:</p>
<ul>
<li><code>-lfoo</code>βuse library <code>foo</code>, means look for shared object files with the name <code>libfoo.so.*</code></li>
<li><code>-L ../foo/</code>βwhen trying to link object files look in the directory <code>../foo/</code>.</li>
<li><code>main.cpp</code>βthe source file to compile.</li>
<li><code>-o qux</code>βoutput to file <code>qux</code> instead of <code>a.out</code>.</li>
</ul>
<p>Just when you think you're done and you try to run the program...</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>./qux<span style="color: #666666"> </span>
</span><span style="color: #cccccc">./qux: error while loading shared libraries: libfoo.so: cannot open shared object file: No such file or directory</span>
</code></pre></div>
<p>Apparently it is not able to find our <code>libfoo.so</code> file on runtime!
To debug it, we can use <code>ldd</code> to find out what's going on.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>ldd<span style="color: #666666"> </span>qux
</span><span style="color: #cccccc"> linux-vdso.so.1 => (0x00007fffa9dff000)</span>
<span style="color: #cccccc"> libfoo.so => not found</span>
<span style="color: #cccccc"> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f007578d000)</span>
<span style="color: #cccccc"> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0075507000)</span>
<span style="color: #cccccc"> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f00752f1000)</span>
<span style="color: #cccccc"> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0074f5d000)</span>
<span style="color: #cccccc"> /lib64/ld-linux-x86-64.so.2 (0x00007f0075abd000)</span>
</code></pre></div>
<p>Apparently it doesn't know where it can find <code>libfoo.so</code>.
This can be solved by specifying the correct path for the linker as follows:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span><span style="color: #40ffff">LD_LIBRARY_PATH</span><span style="color: #d0d0d0">=</span>../foo/<span style="color: #666666"> </span>./qux
</span><span style="color: #cccccc">Output of Foo.bar()=10</span>
</code></pre></div>
<p>Party! We did it! :-D</p>
<p>Why doesn't it know the correct path?
We specified it on compile time, right?
Yes, that was on compile time, not runtime.
In real installations we put the shared library in system paths like <code>/usr/lib</code> and in that case we don't need to specify the <code>LD_LIBRARY_PATH</code> variable.</p>
<h2 id="things-to-keep-in-mind">Things to keep in mind<a class="headerlink" href="#things-to-keep-in-mind" title="Permanent link"> </a></h2>
<ul>
<li>In order to be able to build any application in Ubuntu/Debian, make sure you have installed the <code>build-essential</code> package.</li>
<li>When you want to use a shared library you'll need the corresponding header files <code>foo.h</code> to be able to use it.</li>
<li>Also, when distributing or sharing shared libraries with others, you'll need to share the header files as well.</li>
<li>If you want to compile against a system library, you will need to install the header files.
Usually you'll find them in the <code>-dev</code> packages on Debian/Ubuntu distrubutions.</li>
<li>With the method described on this page you can't specify the <code>.so</code> file in runtime.</li>
<li>None of the commands given need to be run as root.</li>
</ul>
<h2 id="see-also">See also<a class="headerlink" href="#see-also" title="Permanent link"> </a></h2>
<ul>
<li><a href="https://blog.g3rt.nl/building-cpp-shared-libraries-qt-creator.html">How to build and use shared libraries in Qt Creator</a></li>
</ul>SSL client authentication in Apache2011-09-22T14:35:00+02:002011-09-22T14:35:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-09-22:/client-certificate-authentication-apache.html<p>Avoid the use of passwords and authenticate to your (privately) secured website using SSL client certificate authentication.</p>
<h2 id="background-and-goal">Background and goal<a class="headerlink" href="#background-and-goal" title="Permanent link"> </a></h2>
<p>Suppose you want to secure some part of your website by authentication.
You could use HTTP Basic authentication in combination with (plain) SSL to do that, but that requires a username/password combination to be stored on the remote host as well as communicating these credentials.
The better way is to use SSL client authentication.</p>
<p>The main advantage of using SSL is that it uses asymmetrical encryption, which means your clients give you their <em>public</em> key, you can manage them as trusted/untrusted and they can simply use their private key.
This is all done without communicating any secret information.
It can be compared to SSH and public key authentication.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>In this how-to I will not deal with Certificate Authorities, but only with self-signed certificates which are trusted individually.
This method is not recommended for large number of certificates to maintain.
If you have lots of clients, you should set up your own Certificate Authority, sign the client certificates (using certificate signing requests) and trust the CAs explicitly everywhere.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Before you move on, make sure you have an Apache 2.x running with a server certificate and <code>mod_ssl</code> enabled. To test it, Make sure you can reach your server using <code>https://1.2.3.4/</code> (replace <code>1.2.3.4</code> with the IP address of your server).</p>
</div>
<h2 id="configuring-the-clients-browser">Configuring the client's browser<a class="headerlink" href="#configuring-the-clients-browser" title="Permanent link"> </a></h2>
<p>TODO, left for another post.</p>
<h2 id="generating-client-key-and-certificate">Generating client key and certificate<a class="headerlink" href="#generating-client-key-and-certificate" title="Permanent link"> </a></h2>
<p>Clients (with their browsers) need to have a private key along with a public certificate.
On Linux this can be generated as follows (as user):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>Generate<span style="color: #666666"> </span>a<span style="color: #666666"> </span><span style="color: #51b2fd">2048</span><span style="color: #666666"> </span>bits<span style="color: #666666"> </span>RSA<span style="color: #666666"> </span>private<span style="color: #666666"> </span>key<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">in</span><span style="color: #666666"> </span>PEM<span style="color: #666666"> </span>format
<span style="background-color: #404040"><span style="color: #cccccc">openssl genrsa -out myprivatekey.pem 2048</span>
</span><span style="color: #aaaaaa"># </span>Generate<span style="color: #666666"> </span>a<span style="color: #666666"> </span>certificate<span style="color: #666666"> </span>with<span style="color: #666666"> </span>a<span style="color: #666666"> </span>validity<span style="color: #666666"> </span>of<span style="color: #666666"> </span><span style="color: #51b2fd">1095</span><span style="color: #666666"> </span>days.
<span style="background-color: #404040"><span style="color: #cccccc">openssl req -new -x509 -key myprivatekey.pem -out mycertificate.pem -days 1095</span>
</span><span style="color: #aaaaaa"># </span>Answer<span style="color: #666666"> </span>the<span style="color: #666666"> </span>questions<span style="color: #666666"> </span>about<span style="color: #666666"> </span>the<span style="color: #666666"> </span>certificate<span style="color: #666666"> </span>details.
</code></pre></div>
<h2 id="configuring-the-apache-server">Configuring the Apache server<a class="headerlink" href="#configuring-the-apache-server" title="Permanent link"> </a></h2>
<p>Create a directory containing all the trusted certificates.
By default, Apache uses your system-wide trusted CAs, but we only want to trust some certificates explicitly.
Otherwise, anyone using <em>any</em> certificate signed by <em>any</em> trusted CA on the system can be validated, unless you set any checks on the contents of the certificate.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">mkdir /etc/apache2/mytrustedclientcerts</span>
</code></pre></div>
<p>Now copy over the client certificate you received from your client in that folder.
The filename does not matter, only the file format does; it has to be in PEM format.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa"># </span>ls<span style="color: #666666"> </span>-al<span style="color: #666666"> </span>/etc/apache2/mytrustedclientcerts
</span><span style="color: #cccccc">total 16</span>
<span style="color: #cccccc">drwxr-xr-x 2 root root 4096 Sep 22 13:07 .</span>
<span style="color: #cccccc">drwxr-xr-x 8 root root 4096 Sep 22 13:01 ..</span>
<span style="color: #cccccc">-rw-r--r-- 1 root root 1679 Sep 22 13:01 mycertificate.pem</span>
</code></pre></div>
<p>Suppose you have hundreds of these certificates, then Apache would need to open every certificate until it finds the right one.
You can imagine that would be very inefficient.
To speed that up, Apache looks for a file with the hash of the certificate it gets from the client.
For example, if my certificate would be hashed as <code>27e66395</code> then it would look for files with the name of <code>27e66395.X</code> where <code>X</code> is a number starting with <code>0</code>.
This avoids hashing collisions.</p>
<p>For administration it would be nice if not all files would have their file names based on hashes.
We'll use symbolic links in order to point to the real certificate file and generate the symbolic links with a script.
This can be done using a proposed makefile as shown on the bottom of this page.
Place the makefile as <code>Makefile</code> in the <code>/etc/apache2/mytrustedclientcerts</code> folder and run it with just calling </p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">make</span>
</code></pre></div>
<p>You now should have something like this:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa"># </span>ls<span style="color: #666666"> </span>-al<span style="color: #666666"> </span>/etc/apache2/mytrustedclientcerts
</span><span style="color: #cccccc">total 16</span>
<span style="color: #cccccc">drwxr-xr-x 2 root root 4096 Sep 22 13:07 .</span>
<span style="color: #cccccc">drwxr-xr-x 8 root root 4096 Sep 22 13:01 ..</span>
<span style="color: #cccccc">lrwxrwxrwx 1 root root 15 Sep 22 13:06 27e66395.0 -> mycertificate.pem</span>
<span style="color: #cccccc">-rw-r--r-- 1 root root 1679 Sep 22 13:01 mycertificate.pem</span>
<span style="color: #cccccc">-rw-r--r-- 1 root root 1537 Sep 22 13:06 Makefile</span>
</code></pre></div>
<p>Now you're ready to set the following Apache directives in the SSL-enabled site configuration:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>SSLCACertificatePath /etc/apache2/mytrustedclientcerts
SSLVerifyClient require
SSLVerifyDepth 1
</code></pre></div>
<p>The <code>SSLVerifyDepth</code> is set to <code>1</code>, so that it will not use any hierarchy in X.509, but only check against the certificates explicitly.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The directives <code>SSLVerifyClient</code> and <code>SSLVerifyDepth</code> could be used within other directives to match a specific location, files or anything that Apache allows you to match.</p>
</div>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>It is strongly recommended to disable SSLv2 using the directive <code>SSLProtocol all -SSLv2</code> in your site config.
This prevents the use of the old and deprecated SSL protocol version.</p>
</div>
<p>Check the syntax of your site configuration and restart Apache:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa"># </span>apache2ctl<span style="color: #666666"> </span>configtest
</span><span style="color: #cccccc">Syntax OK</span>
<span style="background-color: #404040"><span style="color: #aaaaaa"># </span>/etc/init.d/apache2<span style="color: #666666"> </span>restart
</span><span style="color: #cccccc">Restarting web server: apache2.</span>
</code></pre></div>
<h2 id="appendix-makefile-generating-symbolic-links">Appendix: Makefile generating symbolic links<a class="headerlink" href="#appendix-makefile-generating-symbolic-links" title="Permanent link"> </a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This script is based on <code>pkg.sslcfg/Makefile.crt</code> from an older <code>mod_ssl</code> release (http://www.modssl.org/source/mod_ssl-2.8.31-1.3.41.tar.gz).</p>
</div>
<p>Run it every time you add or remove a client certificate.</p>
<p>Adjusted are the <code>SSL_PROGRAM</code> variable and the files to match.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>for file in *.crt; do \
for file in *.pem; do \
</code></pre></div>
<p>The latter edit is in order to match client certificates (<code>.pem</code> files) instead of CA certificates (usually stored as <code>.crt</code>).</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #ababab; font-style: italic">##</span>
<span style="color: #ababab; font-style: italic">## Makefile to keep the hash symlinks in SSLCACertificatePath up to date</span>
<span style="color: #ababab; font-style: italic">## Copyright (c) 1998-2001 Ralf S. Engelschall, All Rights Reserved. </span>
<span style="color: #ababab; font-style: italic">##</span>
<span style="color: #40ffff">SSL_PROGRAM</span><span style="color: #d0d0d0">=</span>/usr/bin/openssl
update:<span style="color: #666666"> </span>clean
<span style="color: #666666"> </span>-@ssl_program<span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"</span><span style="color: #6ebf26; font-weight: bold">$(</span>SSL_PROGRAM<span style="color: #6ebf26; font-weight: bold">)</span><span style="color: #ed9d13">"</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span><span style="color: #ed9d13">".</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">ssl_program"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span>.<span style="color: #666666"> </span><span style="color: #d0d0d0">];</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">for</span><span style="color: #666666"> </span>dir<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">in</span><span style="color: #666666"> </span>.<span style="color: #666666"> </span><span style="color: #ed9d13">`</span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #40ffff">$$</span>PATH<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>sed<span style="color: #666666"> </span>-e<span style="color: #666666"> </span><span style="color: #ed9d13">'s/:/ /g'`</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">do</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">for</span><span style="color: #666666"> </span>program<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">in</span><span style="color: #666666"> </span>openssl<span style="color: #666666"> </span>ssleay<span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">do</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span>-f<span style="color: #666666"> </span><span style="color: #ed9d13">"</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">dir/</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">program"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">];</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span>-x<span style="color: #666666"> </span><span style="color: #ed9d13">"</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">dir/</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">program"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">];</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #40ffff">ssl_program</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">dir/</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">program"</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>break<span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">done</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span><span style="color: #ed9d13">".</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">ssl_program"</span><span style="color: #666666"> </span>!<span style="color: #d0d0d0">=</span><span style="color: #666666"> </span>.<span style="color: #666666"> </span><span style="color: #d0d0d0">];</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>break<span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">done</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span><span style="color: #ed9d13">".</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">ssl_program"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">=</span><span style="color: #666666"> </span>.<span style="color: #666666"> </span><span style="color: #d0d0d0">];</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #ed9d13">"Error: neither 'openssl' nor 'ssleay' program found"</span><span style="color: #666666"> </span><span style="color: #51b2fd">1</span>><span style="color: #d0d0d0">&</span><span style="color: #51b2fd">2</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #2fbccd">exit</span><span style="color: #666666"> </span><span style="color: #51b2fd">1</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">for</span><span style="color: #666666"> </span>file<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">in</span><span style="color: #666666"> </span>*.pem<span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">do</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span><span style="color: #ed9d13">".`grep SKIPME </span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">file`"</span><span style="color: #666666"> </span>!<span style="color: #d0d0d0">=</span><span style="color: #666666"> </span>.<span style="color: #666666"> </span><span style="color: #d0d0d0">];</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span>dummy<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>awk<span style="color: #666666"> </span><span style="color: #ed9d13">'{ printf("%-15s ... Skipped\n", file); }'</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #ed9d13">"file=</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">file"</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">else</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #40ffff">n</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">0</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">while</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span><span style="color: #51b2fd">1</span><span style="color: #666666"> </span><span style="color: #d0d0d0">];</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">do</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #40ffff">hash</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">"`</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">ssl_program x509 -noout -hash <</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">file`"</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span>-r<span style="color: #666666"> </span><span style="color: #ed9d13">"</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">hash.</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">n"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">];</span><span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #40ffff">n</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">`</span>expr<span style="color: #666666"> </span><span style="color: #40ffff">$$</span>n<span style="color: #666666"> </span>+<span style="color: #666666"> </span><span style="color: #51b2fd">1</span><span style="color: #ed9d13">`</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">else</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span>dummy<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>awk<span style="color: #666666"> </span><span style="color: #ed9d13">'{ printf("%-15s ... %s\n", file, hash); }'</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #ed9d13">"file=</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">file"</span><span style="color: #666666"> </span><span style="color: #ed9d13">"hash=</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">hash.</span><span style="color: #40ffff">$$</span><span style="color: #ed9d13">n"</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>ln<span style="color: #666666"> </span>-s<span style="color: #666666"> </span><span style="color: #40ffff">$$</span>file<span style="color: #666666"> </span><span style="color: #40ffff">$$</span>hash.<span style="color: #40ffff">$$</span>n<span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span>break<span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">done</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span><span style="color: #d0d0d0">;</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">done</span>
clean:
<span style="color: #666666"> </span>-@rm<span style="color: #666666"> </span>-f<span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #51b2fd">0</span>-9a-fA-F<span style="color: #d0d0d0">]</span>*.<span style="color: #d0d0d0">[</span><span style="color: #51b2fd">0</span>-9<span style="color: #d0d0d0">]</span>*
</code></pre></div>Building C++ shared libraries in Qt Creator (cross-platform)2011-09-20T23:59:00+02:002012-04-12T11:09:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-09-20:/building-cpp-shared-libraries-qt-creator.html<p>Here's how to use QMake in Qt Creator to build shared libraries in a cross-platform fashion during development.</p>
<h2 id="terminology">Terminology<a class="headerlink" href="#terminology" title="Permanent link"> </a></h2>
<p>The first hurdle to take was <em>terminology</em>.
Apparently I was the only one in the world building DLLs on Linux as Google didn't give me any hope.
But are Linux applications all statically built?
No.
I was wrong to make the assumption that terminology was the same across the OSs.</p>
<p>While in Windows world it's called DLLs, in Unix/Linux world the same thing is called 'shared libraries' and they usually have the <code>.so</code> file extension.
That was an important breakthrough as that really helped me finding useful results on Google.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>In Windows we have DLLs, in Linux/Unix we have shared libraries.
It's the same!</p>
</div>
<h2 id="creating-the-projects">Creating the projects<a class="headerlink" href="#creating-the-projects" title="Permanent link"> </a></h2>
<h3 id="step-1-using-the-wizard">Step 1: Using the wizard<a class="headerlink" href="#step-1-using-the-wizard" title="Permanent link"> </a></h3>
<p>We'll need at least two projects in Qt Creator; one will be the shared library, the other one the <code>main()</code> and maybe some other classes.
This, of course, depends on the model you're trying to put life to.</p>
<ol>
<li>Create a new project.</li>
<li>Choose 'Other Project' on the left and select <em>C++ Libary</em>.</li>
<li>A wizard will open and select 'Shared Library' as the type.
Fill in the name of your library and the (base) path to create it in.</li>
<li>Select 'QtCore' as only required Qt module.
(Don't know if this is really needed.)</li>
<li>Create your first class from the wizard.</li>
<li>Finish the wizard.</li>
<li>Open the <code>.pro</code> file with the editor and notice the following: <code>TEMPLATE</code> (should be <code>LIB</code>).</li>
</ol>
<p><em>Build</em> (don't run) the project.</p>
<p>Now you should get <code>.dll</code> along with <code>.a</code> files on Windows (G++/MinGW setup) or <code>.so</code> files on Linux (with a lot of symlinks) in the build directory (probably some shadow build folderβthis is fine).</p>
<h3 id="step-2-moving-the-shared-libraries-to-a-common-folder">Step 2: Moving the shared libraries to a common folder<a class="headerlink" href="#step-2-moving-the-shared-libraries-to-a-common-folder" title="Permanent link"> </a></h3>
<p>It's quite common to build libraries yourself and actually use them in another project at the same time.
One would need to <code>make install</code> all of the libraries in order to use them.
For developing your libraries and their uses at the same time, this is unwanted.
In order to move the shared libraries (and the needed header files to interface with it) we'll use some QMake magic.</p>
<p>Open the <code>.pro</code> file and <em>append</em> on the end of the file the following (example, customize to your own needs):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code># Set the destination directory of the shared libraries
# MYDLLDIR is a name something I made up myself, DESTDIR is a QMake variable.
# $$IN_PWD expands to the directory your .pro file is in.
# This example copies the final build output to ../dlls (one level above your project).
MYDLLDIR = $$IN_PWD/../dlls
# We need quotes around the path in order to support whitespaces in the path
# for Windows. e.g. (''C:\Users\Gert van Dijk\...'').
# For some strange reason $$quote(...) does not seem to work for Windows here.
DESTDIR = \"$$MYDLLDIR\"
#
# Here's some magic to move all of the project's header files to the same DLL directory.
#
DDIR = \"$$MYDLLDIR/\"
SDIR = \"$$IN_PWD/\"
# Replace slashes in paths with backslashes for Windows
win32:file ~= s,/,\\,g
win32:DDIR ~= s,/,\\,g
win32:SDIR ~= s,/,\\,g
# For-loop to copy all header files to DDIR.
for(file, HEADERS) {
QMAKE_POST_LINK += $$QMAKE_COPY $$quote($${SDIR}$${file}) $$quote($$DDIR) $$escape_expand(\\n\\t)
}
</code></pre></div>
<p>In the directory <code>dlls</code> (located one level above your project) you should now find the library along with the header files.</p>
<div class="admonition hint">
<p class="admonition-title">Hint</p>
<p>If you don't want to expose all of the <code>HEADERS</code>, just specify your on list of headers and let the for-loop run on that.
This can be useful in cases that you want your internal library method calls to be completely closed and only expose some classes to interface with others.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This whole magic could also be done using the 'Build Settings' and specifying custom build steps of your project, but this ends up in the <code>.pro.user</code> file and this is not desiged to be shared with other developers.</p>
</div>
<h3 id="step-3-creating-a-project-using-the-shared-library">Step 3: Creating a project using the shared library<a class="headerlink" href="#step-3-creating-a-project-using-the-shared-library" title="Permanent link"> </a></h3>
<p>Now that we output the library and their headers to some common folder, we can use that as the path for QMake to depend on when linking against the libraries in there.</p>
<p>You can leave the project of the shared library open or close it, it doesn't matter.</p>
<ol>
<li>Create a new project.</li>
<li>Choose 'Other Project' on the left and select 'Qt Console application' (just an example).</li>
<li>Fill in the name of your library and the (base) path to create it in.</li>
<li>Use default build set up (both Release and Debug, enable shadow building).</li>
<li>Create your first class from the wizard.</li>
<li>Finish the wizard.</li>
<li>Open the <code>.pro</code> file with the editor and notice the following: <code>TEMPLATE</code> (should be <code>app</code>).</li>
</ol>
<p>Now in order to make it dependent on the library, append the following to the <code>.pro</code> file:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>MYDLLDIR = $$IN_PWD/../dlls
# As our header files are in the same directory, we can make Qt Creator find it
# by specifying it as INCLUDEPATH.
INCLUDEPATH += $$MYDLLDIR
# Dependency to library domain (libdomain.so for Unices or domain.dll on Win32)
# Repeat this for more libraries if needed.
win32:LIBS += $$quote($$MYDLLDIR/domain.dll)
unix:LIBS += $$quote(-L$$MYDLLDIR) -ldomain
</code></pre></div>
<p>Now you should be able to <em>build</em> (not yet run) your project.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>To create libraries depending on other libraries, you simply specify the QMake <code>LIBS</code> also in you shared library project.
If you fail to do so, you'll get runtime errors.</p>
</div>
<h3 id="step-4-set-up-runtime-linker-paths">Step 4: Set up runtime linker paths<a class="headerlink" href="#step-4-set-up-runtime-linker-paths" title="Permanent link"> </a></h3>
<p>While it now builds, it needs to be able to find your shared libraries at runtime.
To make it search in the specified directory we will need to append it to the <code>PATH</code> environment variable (Windows) or <code>LD_LIBRARY_PATH</code> environment variable (Unices).</p>
<ol>
<li>Click on 'Projects' in the left toolbar in Qt Creator.</li>
<li>Select the leaf project (the one that builds to the executable to run) at the top.</li>
<li>Click on 'Run Settings'</li>
<li>Somewhat below, you'll find 'Run Environment'.
Expand it (click on 'Details').</li>
<li>For Unices:<ul>
<li>Click on the button 'Add' to the right.</li>
<li>Enter <code>LD_LIBRARY_PATH</code> as 'Variable' and <code>../dlls</code> as Value.</li>
</ul>
</li>
<li>For Windows:<ul>
<li>Find <code>PATH</code> In the list of Variables.</li>
<li>Click 'Edit'.</li>
<li>Add our path relative by appending <code>;..\dlls</code>.</li>
</ul>
</li>
</ol>
<p>Now you should be able to build and run your leaf project depending on a library!</p>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Build <strong>order does matter</strong>.
While you can specify build dependencies in Qt Creator in order to just hit "Build All", this didn't work out for me.
Just build the libraries in the order it works for their dependencies, and your leaf project as the last one.</p>
</div>Determine if the host will run filesystem check when reboot2011-08-25T12:31:00+02:002011-08-25T12:31:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-08-25:/determine-filesystem-check-reboot.html<p>Want to check if the reboot you planned is going to trigger a filesystem check? This is how to prepare for that.</p>
<h2 id="check-before-reboot">Check before reboot<a class="headerlink" href="#check-before-reboot" title="Permanent link"> </a></h2>
<p>Want to check if the reboot you planned is going to trigger a filesystem check?
This is how to prepare for that.
It is particularly useful on systems with large amounts of data on which a filesystem check can take a long time.
Planning in maintenance windows can be critical.</p>
<p>As root, run:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">dumpe2fs -h /dev/sda1</span>
</code></pre></div>
<p>To get information the filesystem on <code>/dev/sda1</code>.
If you don't know which partition or device your filesystem is on, check the output of the command <code>mount</code>.</p>
<p>Here's an example output (on August 25, 2011):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa"># </span>dumpe2fs<span style="color: #666666"> </span>-h<span style="color: #666666"> </span>/dev/cciss/c0d0p3
</span><span style="color: #cccccc">dumpe2fs 1.41.3 (12-Oct-2008)</span>
<span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">Filesystem state: clean</span>
<span style="color: #cccccc">[...]</span>
<span style="color: #cccccc">Mount count: 3</span>
<span style="color: #cccccc">Maximum mount count: 33</span>
<span style="color: #cccccc">Last checked: Tue Feb 15 10:53:25 2011</span>
<span style="color: #cccccc">Check interval: 15552000 (6 months)</span>
<span style="background-color: #404040"><span style="color: #cccccc">Next check after: Sun Aug 14 11:53:25 2011</span>
</span><span style="color: #cccccc">[...]</span>
</code></pre></div>
<p>In the above example you can see it will be checked on the next reboot because of the date <code>Aug 14</code> is passed.
Another reason for a trigger is when the maximum mount count is reached.</p>
<h2 id="change-the-triggers">Change the triggers<a class="headerlink" href="#change-the-triggers" title="Permanent link"> </a></h2>
<p>In order to change the values of the check interval or maximum mount count, please refer to the <code>tune2fs</code> tool.
See the manpage for an easy explanation on how to tune your filesystem.
It can be really useful to prevent a filesystem check the next reboot in an uplanned, during office hours, emergency maintenance to avoid additional downtime.</p>Rsyslog remote logging using RELP2011-08-10T01:33:00+02:002011-08-10T01:33:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-08-10:/remote-logging-rsyslog-relp.html<p>When a good hacker enters your network he can also erase all traces he leaves in your logs. By forwarding all messages sent to rsyslog to a remote host over network, you can prevent the bastard to clean his tracks.</p>
<h2 id="why-remote-logging">Why remote logging?<a class="headerlink" href="#why-remote-logging" title="Permanent link"> </a></h2>
<p>When a good hacker enters your network he can also erase all traces he leaves in your logs.
By forwarding all messages sent to rsyslog to a remote host over network, you can prevent the bastard to clean his tracks.</p>
<p>As I was unable to find a good up-to-date technical howto on this topic, I decided to write my own to share with you.</p>
<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link"> </a></h2>
<h3 id="about-relp">About RELP<a class="headerlink" href="#about-relp" title="Permanent link"> </a></h3>
<p>By default and on historical grounds syslog messages were sent as UDP fragments over the network.
Despite also TCP is possible and much more reliable, it will not cover network hickups completely.
See the blog article <a href="http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html">"On the (un)reliability of plain tcp syslog..."</a> for a very good explanation.</p>
<p>RELP is designed to overcome these shortcomings of TCP and UDP in application layer.
Whenever possible, it is advised to use RELP for reliability.</p>
<h3 id="other-syslog-daemons">other syslog daemons<a class="headerlink" href="#other-syslog-daemons" title="Permanent link"> </a></h3>
<p>Please check with your Linux distribution what kind of syslog daemon it is shipped with.
Recent Debian runs rsyslog, earlier versions than Lenny were shipped with sysklogd.
This article applies to the use of rsyslog only, both server and client.
Please check with your Linux distribution on how to move to rsyslog before proceeding here.</p>
<h2 id="configure-the-server">Configure the server<a class="headerlink" href="#configure-the-server" title="Permanent link"> </a></h2>
<p>First, make sure you have installed the RELP extensions for rsyslog.
For Debian and -derivatives you'll need to install the package <code>rsyslog-relp</code>.
That package includes support for RELP for both input and output.</p>
<p>Add the following two lines to the configuration of your rsyslog configuration.
Preferably, add it as a separate file in <code>/etc/rsyslog.d</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>$ModLoad imrelp # Load the input module ('im') 'relp'
$InputRELPServerRun 20514 # Set the port to 20514
</code></pre></div>
<p>This will enable rsyslog to listen on TCP port 20514 for RELP syslog messages.
Change the port to whatever you want it to listen on.</p>
<p>Don't forget to reload rsyslog to activate the change: <code>/etc/init.d/rsyslog restart</code> or <code>service rsyslog restart</code> for Ubuntu and confirm it is running on the specified port with for example <code>netstat -ntlp | grep rsyslog</code>.</p>
<p>Now you're all set for the server.</p>
<h2 id="configure-the-client">Configure the client<a class="headerlink" href="#configure-the-client" title="Permanent link"> </a></h2>
<p>First, install the same RELP support as on the server.</p>
<p>Add the following to the rsyslog configuration (like we did on the server) to forward all messages to the server:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>*.* :omrelp:loghost.example.com:20514;RSYSLOG_ForwardFormat
</code></pre></div>
<p>Explanation from left to right: match all, output module 'relp', to the host <code>loghost.example.com</code> on port 20514.
The template <code>RSYSLOG_ForwardFormat</code> is used to maintain some high-precision timestamp.
See also the statement on the <a href="http://www.rsyslog.com/doc/omrelp.html">omrelp module documentation</a>:</p>
<blockquote>
<p>Rsyslog's high-precision timestamp format is used, thus the special "RSYSLOG_ForwardFormat" (case sensitive!) template is used.</p>
</blockquote>
<p>Reload rsyslogβyou will now see the messages from the client appearing in the log files on the server.</p>
<h3 id="disable-local-logging">Disable local logging<a class="headerlink" href="#disable-local-logging" title="Permanent link"> </a></h3>
<p>While messages are forwarded over the network, it is still logged in local files.
I should cover this in a later post.</p>
<h2 id="other-valid-use-cases">Other valid use cases<a class="headerlink" href="#other-valid-use-cases" title="Permanent link"> </a></h2>
<p>Apart from security you can think of other reasons to enable remote logging:</p>
<ul>
<li>Embedded devices with insufficient storage (e.g. network devices).
However, most embedded devices will not support RELP.
Configuring rsyslog to listen also for regular syslog messages on the network is not covered in this article.</li>
<li>When you have <em>lots</em> of logging from <em>lots</em> of different virtual machines it is usually a lot of random I/O on the backend storage.
With remote logging that becomes sequential, more efficient writes.</li>
</ul>How to change the hostname of a current running Linux installation2011-08-06T22:00:00+02:002014-04-21T14:41:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-08-06:/change-hostname-current-installation.html<p>Here's how to properly change the hostname of a current installation without rebooting.</p>
<h2 id="change-the-hostname">Change the hostname<a class="headerlink" href="#change-the-hostname" title="Permanent link"> </a></h2>
<p>This is how to change the hostname as well as the domainname of a machine running Debian or a Debian-like OS.</p>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Changing the hostname on a running Linux system will break X on that machineβthat means you will probably get in trouble while running in a graphical interface.
It won't cause any trouble in a virtual terminal (Ctrl+Alt+F1) or remotely via SSH.</p>
</div>
<h3 id="temporary-immediate-change">Temporary, immediate change<a class="headerlink" href="#temporary-immediate-change" title="Permanent link"> </a></h3>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">gert@oldhostname:~# </span>hostname<span style="color: #666666"> </span>mynewhostname
</code></pre></div>
<p>Now close your shell (or log out), open a new one and you'll see the hostname is changed.</p>
<p>You can do the same thing for any of the domain names, just exchange the <code>hostname</code> command with any of the ones below (taken from the manpage of <code>hostname</code>):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>domainname - show or set the system's NIS/YP domain name
ypdomainname - show or set the system's NIS/YP domain name
nisdomainname - show or set the system's NIS/YP domain name
dnsdomainname - show the system's DNS domain name
</code></pre></div>
<div class="admonition warning">
<p>This change will not survive a reboot.
See below for a permanent way to change it.</p>
</div>
<h3 id="permanent-change">Permanent change<a class="headerlink" href="#permanent-change" title="Permanent link"> </a></h3>
<p>Edit the file containing your hostname, for example, just overwrite it with the new one:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #ed9d13">"mynewhostname"</span><span style="color: #666666"> </span>><span style="color: #666666"> </span>/etc/hostname
</code></pre></div>
<p>Then, change the entries accordingly in <code>/etc/hosts</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>127.0.1.1 oldhostname
</code></pre></div>
<p>Usually only the line starting with <code>127.0.1.1</code> has to be modified.</p>
<p>Now, run the <code>hostname</code> init-script to finish everything up:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>/etc/init.d/hostname.sh<span style="color: #666666"> </span>start
</code></pre></div>
<p>for Debian or</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>service<span style="color: #666666"> </span>hostname<span style="color: #666666"> </span>start
</code></pre></div>
<p>for Ubuntu.</p>
<p>Complaints after the last command about the service not running can be discarded; it's not a real service, rather it's just a script run at boot time.</p>
<div class="admonition tip">
<p>Also, don't forget to change the settings for other services like your mail server, usually in <code>/etc/mailname</code>.</p>
</div>
<h3 id="check">Check<a class="headerlink" href="#check" title="Permanent link"> </a></h3>
<p>To verify the system has picked up the new hostname, <code>hostname</code> can show you what it sees.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>hostname<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># just print the hostname</span>
<span style="color: #aaaaaa">$ </span>hostname<span style="color: #666666"> </span>-f<span style="color: #666666"> </span><span style="color: #ababab; font-style: italic"># print the fully qualified domain name</span>
</code></pre></div>Changing the username, user id (number), groupname and group id2011-07-22T15:23:00+02:002012-01-10T13:39:00+01:00Gert van Dijktag:blog.g3rt.nl,2011-07-22:/change-username-uid-groupname-gid.html<p>Want to change the user ID, username, group ID or group name? Here's how.</p>
<h2 id="by-example">By example<a class="headerlink" href="#by-example" title="Permanent link"> </a></h2>
<ul>
<li>
<p>Change the uid of <code>gert</code> (now <code>1000</code>) to <code>1002</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">usermod -u 1002 gert</span>
</code></pre></div>
<div class="admonition tip">
<p>Changing user IDs with <code>usermod</code> will also automagically change the ownership of files within the user's home directory.
For files on other locations, please refer to the section below.</p>
</div>
</li>
<li>
<p>Change the username of <code>gert</code> to <code>gvandijk</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">usermod -l gvandijk gert</span>
</code></pre></div>
</li>
<li>
<p>Change the gid (primary group) of <code>gert</code> (now <code>1000</code>) to <code>1002</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">usermod -g 1002 gert</code></span>
</code></pre></div>
</li>
<li>
<p>Change the gid of the group <code>thegroup</code> (now <code>1500</code>) to <code>1501</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">groupmod -g 1501 thegroup</span>
</code></pre></div>
</li>
<li>
<p>Change the groupname of <code>thegroup</code> to <code>mygroup</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">groupmod -n mygroup thegroup</span>
</code></pre></div>
</li>
<li>
<p>Changing the user ID and group ID can also be combined in one command:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">usermod -u 1002 -g 1002 gert</span>
</code></pre></div>
</li>
<li>
<p>Changing the user's home directory to the new location (move the directory manually!):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">usermod -d /newhomes/gvandijk gvandijk</span>
</code></pre></div>
</li>
</ul>
<div class="admonition note">
<p>Most <code>usermod</code> commands can only be executed when the user is not logged in.</p>
</div>
<h2 id="fix-ownership-of-files">Fix ownership of files<a class="headerlink" href="#fix-ownership-of-files" title="Permanent link"> </a></h2>
<p>Files on a Linux filesystem are stored with metadata like owner and group using numeric IDs.
When you change these IDs, you will have to fix the IDs on the files.</p>
<p>For files outside the home directory, you'll have to fix all the ownerships of the user's files:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">find / -user 1000 -exec chown -h 1002 {} \;</span>
</code></pre></div>
<p>Explanation: Find all kind of files/directories with user ID <code>1000</code> from the base <code>/</code>, then execute the command <code>chown -h 1002 <entry></code> for every entry found.</p>How to copy MySQL privileges from one server to another2011-07-20T10:50:00+02:002011-07-21T14:20:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-07-20:/copy-mysql-privileges-from-server.html<p>In this article I will describe on how to transfer the privileges (grants) from one MySQL server to another.</p>
<h2 id="my-approach">My approach<a class="headerlink" href="#my-approach" title="Permanent link"> </a></h2>
<p>In this article I will describe on how to transfer the privileges (grants) from one MySQL server to another.
While you <em>could</em> do it dumping and importing the <code>mysql</code> database from one to another, I prefer it to do using 'plain' SQL.</p>
<div class="admonition tip">
<p class="admonition-title">The base of this article is taken from the Serverfault discussion</p>
<p><a href="https://serverfault.com/q/8860/135437">How can I export the privileges from MySQL and then import to a new server?</a>.</p>
</div>
<p>Do not mess with the database <code>mysql</code>.
There is a lot more going on there than just the users table.
Your best bet is the ''SHOW GRANTS FOR'' command.</p>
<h2 id="bash-function-mygrants">Bash function mygrants()<a class="headerlink" href="#bash-function-mygrants" title="Permanent link"> </a></h2>
<p>You could use this Bash alias function as a helper in this maintenance.
Just add it in your <code>.bashrc</code> file or include it globally on your server.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>mygrants<span style="color: #d0d0d0">()</span>
<span style="color: #d0d0d0">{</span>
mysql<span style="color: #666666"> </span>-B<span style="color: #666666"> </span>-N<span style="color: #666666"> </span><span style="color: #40ffff">$@</span><span style="color: #666666"> </span>-e<span style="color: #666666"> </span><span style="color: #ed9d13">"SELECT DISTINCT CONCAT(</span>
<span style="color: #ed9d13"> 'SHOW GRANTS FOR ''', user, '''@''', host, ''';'</span>
<span style="color: #ed9d13"> ) AS query FROM mysql.user"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
mysql<span style="color: #666666"> </span><span style="color: #40ffff">$@</span><span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span><span style="color: #ed9d13">\</span>
sed<span style="color: #666666"> </span><span style="color: #ed9d13">'s/\(GRANT .*\)/\1;/;s/^\(Grants for .*\)/## \1 ##/;/##/{x;p;x;}'</span>
<span style="color: #d0d0d0">}</span>
</code></pre></div>
<p>The first <code>mysql</code> command uses SQL to generate valid query which is piped to the second <code>mysql</code> command.
The output is then piped through sed to add pretty comments.</p>
<p>The <code>$@</code> in the command will allow you to call it with the same options as you would call <code>mysql</code>, e.g.:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">mygrants --host=prod-db1 --user=admin --password=secret</span>
</code></pre></div>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>You might not want to disclose your MySQL password in the process list this way.</p>
<p>To have the script read the password from <code>stdin</code> (ask you for the password) it has to be adjusted, which I will do some time later.</p>
</div>
<h2 id="migrate-from-one-server-to-another">Migrate from one server to another<a class="headerlink" href="#migrate-from-one-server-to-another" title="Permanent link"> </a></h2>
<p>You can use your full Unix toolkit on this like so:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">mygrants --host=server1 --user=root --password=secret | grep phpbb_user | mysql --host=server2 --user=root --password=secret</span>
</code></pre></div>
<p>That is <em>the</em> right way to move users; your MySQL ACL is modified with pure SQL.</p>How to use Thinkfan to control your Thinkpad's fan speed2011-07-18T17:53:00+02:002011-10-01T19:21:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-07-18:/thinkfan-usage.html<p>How to install and configure the Thinkfan daemon to control your fan speed of your Thinkpad.</p>
<h2 id="obtain-thinkfan">Obtain Thinkfan<a class="headerlink" href="#obtain-thinkfan" title="Permanent link"> </a></h2>
<p>Download <code>thinkfan</code> from the repositories of your operating system, get it from the <a href="http://thinkfan.sourceforge.net/">upstream website</a> or <a href="https://blog.g3rt.nl/thinkfan-patch-overheating-bug.html">build your own</a> with the proposed fix for overheating issues.</p>
<p>Before you can use thinkfan, <a href="https://blog.g3rt.nl/thinkpad-acpi-kernel-module.html">enable fan control</a> of the <code>thinkpad_acpi</code> module.</p>
<p>After installation you'll have to configure it yourself and set your own thresholds and fan levels accordingly.</p>
<div class="admonition danger">
<p class="admonition-title">Danger ahead!</p>
<p>It will not check any settings and you could damage your machine in overheating!</p>
</div>
<p>This is what I use on my T61p in the <code>/etc/thinkfan.conf</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>sensor /proc/acpi/ibm/thermal (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
(0, 0, 55)
(4, 48, 60)
(5, 50, 61)
(6, 52, 63)
(7, 59, 76)
(127, 73, 32767)
</code></pre></div>
<p>To enable the daemon, first edit the file <code>/etc/default/thinkfan</code> so that it reads <code>START=yes</code> and start it using</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>/etc/init.d/thinkfan<span style="color: #666666"> </span>start
</code></pre></div>
<p>for Debian, or (Ubuntu)</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>service<span style="color: #666666"> </span>thinkfan<span style="color: #666666"> </span>start
</code></pre></div>
<h2 id="see-also">See also<a class="headerlink" href="#see-also" title="Permanent link"> </a></h2>
<ul>
<li><a href="https://blog.g3rt.nl/thinkfan-patch-overheating-bug.html">Workaround for a bug in Debian/Ubuntu with overheating</a></li>
</ul>How to use the Thinkpad ACPI kernel module2011-07-18T17:28:00+02:002011-07-18T18:05:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-07-18:/thinkpad-acpi-kernel-module.html<p>How to use the thinkpad-acpi kernel module for controlling fan speed.</p>
<h2 id="obtaining-the-kernel-module">Obtaining the kernel module<a class="headerlink" href="#obtaining-the-kernel-module" title="Permanent link"> </a></h2>
<p>The ThinkPad ACPI module is included in the Linux kernel (<code>thinkpad-acpi</code>).</p>
<h2 id="enable-fan_control-and-experimental-options">Enable fan_control and experimental options<a class="headerlink" href="#enable-fan_control-and-experimental-options" title="Permanent link"> </a></h2>
<p>To be able to control your fan and do more sophisticated stuff we need to set some parameters on load time of the module.</p>
<p>Set the following content in the file <code>/etc/modprobe.d/thinkpad-acpi.conf</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>options thinkpad-acpi experimental=1 fan_control=1
</code></pre></div>
<p>Reload the module:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span>rmmod<span style="color: #666666"> </span>thinkpad-acpi<span style="color: #666666"> </span><span style="color: #d0d0d0">&&</span><span style="color: #666666"> </span>modprobe<span style="color: #666666"> </span>thinkpad-acpi
</code></pre></div>
<p>Now you'll get more options in <code>/proc/acpi/ibm</code> to control ACPI.</p>
<h2 id="examples">Examples<a class="headerlink" href="#examples" title="Permanent link"> </a></h2>
<div class="admonition tip">
<p><strong>Note:</strong> All done in working directory <code>/proc/acpi/ibm</code>.</p>
</div>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>:::console hl_lines="1 4 5"
# cat fan
status: enabled
speed: 3051
level: 5
commands: level <level> (<level> is 0-7, auto, disengaged, full-speed)
commands: enable, disable
commands: watchdog <timeout> (<timeout> is 0 (off), 1-120 (seconds))
</code></pre></div>
<p>Set the level of the fan to full speed.</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa"># </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #ed9d13">"level full-speed"</span><span style="color: #666666"> </span>><span style="color: #666666"> </span>fan
</code></pre></div>
<h2 id="see-also">See also<a class="headerlink" href="#see-also" title="Permanent link"> </a></h2>
<ul>
<li><a href="https://blog.g3rt.nl/thinkpad_smapi_driver_battery_contol_and_more.html">SMAPI driver</a>βbattery information and more power control</li>
<li><a href="https://blog.g3rt.nl/thinkfan-usage.html">Thinkfan</a></li>
</ul>Thinkpad SMAPI driver: Battery control and more2011-07-18T12:38:00+02:002011-07-22T15:34:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-07-18:/thinkpad_smapi_driver_battery_contol_and_more.html<p>The SMAPI driver enables you to take more control of your Thinkpad's battery. Learn how to set charge and discharge tresholds to prolong your battery's life.</p>
<h2 id="smapi-driver-battery-control-and-more">SMAPI driver: battery control and more<a class="headerlink" href="#smapi-driver-battery-control-and-more" title="Permanent link"> </a></h2>
<p>In recent Debian and Ubuntu releases there's an interesting package for ThinkPad owners: <code>tp-smapi-dkms</code>.</p>
<p>By installing this package you'll get a way of controlling the battery charge stop/start thresholds, HDAPS driver and more.</p>
<h2 id="battery-control">Battery control<a class="headerlink" href="#battery-control" title="Permanent link"> </a></h2>
<p>See the contents of the directory <code>/sys/devices/platform/smapi/BAT0</code>.</p>
<h2 id="setting-the-start-and-stop-thresholds">Setting the start and stop thresholds<a class="headerlink" href="#setting-the-start-and-stop-thresholds" title="Permanent link"> </a></h2>
<p>I created a script to set those thresholds.
It just sets the values in the files in <code>/sys/devices/platform/smapi/BAT0/</code> and logs the actions to the logger daemon using the syslog.</p>
<p>Below is the script, a sensible location for it would be <code>/usr/local/bin/set_charge_tresholds.sh</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #ababab; font-style: italic">#!/bin/bash</span>
<span style="color: #ababab; font-style: italic">#Configure the three options below yourself. Only one battery is supported at the moment.</span>
<span style="color: #40ffff">BAT</span><span style="color: #d0d0d0">=</span>BAT0
<span style="color: #40ffff">START_TRESH</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">85</span>
<span style="color: #40ffff">STOP_TRESH</span><span style="color: #d0d0d0">=</span><span style="color: #51b2fd">90</span>
<span style="color: #40ffff">SMAPI_LOADED</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">`</span>awk<span style="color: #666666"> </span><span style="color: #ed9d13">'{ print $1 }'</span><span style="color: #666666"> </span>/proc/modules<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>grep<span style="color: #666666"> </span>tp_smapi<span style="color: #666666"> </span><span style="color: #d0d0d0">|</span><span style="color: #666666"> </span>wc<span style="color: #666666"> </span>-l<span style="color: #ed9d13">`</span>
<span style="color: #40ffff">SMAPI_DIR</span><span style="color: #d0d0d0">=</span>/sys/devices/platform/smapi
<span style="color: #40ffff">BAT_DIR</span><span style="color: #d0d0d0">=</span><span style="color: #40ffff">$SMAPI_DIR</span>/<span style="color: #40ffff">$BAT</span>
<span style="color: #40ffff">LOGGER</span><span style="color: #d0d0d0">=</span>/usr/bin/logger
<span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span><span style="color: #ed9d13">"</span><span style="color: #40ffff">$SMAPI_LOADED</span><span style="color: #ed9d13">"</span><span style="color: #666666"> </span>-gt<span style="color: #666666"> </span><span style="color: #ed9d13">"0"</span><span style="color: #666666"> </span><span style="color: #d0d0d0">]</span>
<span style="color: #6ebf26; font-weight: bold">then</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span>-e<span style="color: #666666"> </span><span style="color: #40ffff">$BAT_DIR</span>/start_charge_thresh<span style="color: #666666"> </span><span style="color: #d0d0d0">]</span><span style="color: #666666"> </span><span style="color: #d0d0d0">&&</span><span style="color: #666666"> </span><span style="color: #d0d0d0">[</span><span style="color: #666666"> </span>-e<span style="color: #666666"> </span><span style="color: #40ffff">$BAT_DIR</span>/stop_charge_thresh<span style="color: #666666"> </span><span style="color: #d0d0d0">]</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">then</span>
<span style="color: #666666"> </span><span style="color: #40ffff">BEFORE_START</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">`</span>cat<span style="color: #666666"> </span><span style="color: #40ffff">$BAT_DIR</span>/start_charge_thresh<span style="color: #ed9d13">`</span>
<span style="color: #666666"> </span><span style="color: #40ffff">BEFORE_STOP</span><span style="color: #d0d0d0">=</span><span style="color: #ed9d13">`</span>cat<span style="color: #666666"> </span><span style="color: #40ffff">$BAT_DIR</span>/stop_charge_thresh<span style="color: #ed9d13">`</span>
<span style="color: #666666"> </span><span style="color: #40ffff">$LOGGER</span><span style="color: #666666"> </span>-s<span style="color: #666666"> </span>-p<span style="color: #666666"> </span>syslog.info<span style="color: #666666"> </span><span style="color: #ed9d13">"About to change the start/stop charging tresholds for Thinkpad battery </span><span style="color: #40ffff">$BAT</span><span style="color: #ed9d13">. Values before change: start=</span><span style="color: #40ffff">$BEFORE_START</span><span style="color: #ed9d13">, stop=</span><span style="color: #40ffff">$BEFORE_STOP</span><span style="color: #ed9d13">"</span>
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #40ffff">$START_TRESH</span><span style="color: #666666"> </span>><span style="color: #666666"> </span><span style="color: #40ffff">$BAT_DIR</span>/start_charge_thresh
<span style="color: #666666"> </span><span style="color: #2fbccd">echo</span><span style="color: #666666"> </span><span style="color: #40ffff">$STOP_TRESH</span><span style="color: #666666"> </span>><span style="color: #666666"> </span><span style="color: #40ffff">$BAT_DIR</span>/stop_charge_thresh
<span style="color: #666666"> </span><span style="color: #40ffff">$LOGGER</span><span style="color: #666666"> </span>-s<span style="color: #666666"> </span>-p<span style="color: #666666"> </span>syslog.info<span style="color: #666666"> </span><span style="color: #ed9d13">"Successfully set start/stop charging tresholds for Thinkpad battery </span><span style="color: #40ffff">$BAT</span><span style="color: #ed9d13"> to start=</span><span style="color: #40ffff">$START_TRESH</span><span style="color: #ed9d13"> and stop=</span><span style="color: #40ffff">$STOP_TRESH</span><span style="color: #ed9d13">."</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">else</span>
<span style="color: #666666"> </span><span style="color: #40ffff">$LOGGER</span><span style="color: #666666"> </span>-s<span style="color: #666666"> </span>-p<span style="color: #666666"> </span>syslog.err<span style="color: #666666"> </span><span style="color: #ed9d13">"Unable to set start/stop charging tresholds for Thinkpad battery </span><span style="color: #40ffff">$BAT</span><span style="color: #ed9d13">; file not found."</span>
<span style="color: #666666"> </span><span style="color: #6ebf26; font-weight: bold">fi</span>
<span style="color: #6ebf26; font-weight: bold">else</span>
<span style="color: #666666"> </span><span style="color: #40ffff">$LOGGER</span><span style="color: #666666"> </span>-s<span style="color: #666666"> </span>-p<span style="color: #666666"> </span>syslog.err<span style="color: #666666"> </span><span style="color: #ed9d13">"Thinkpad SMAPI kernel module seems not loaded, skipping."</span>
<span style="color: #6ebf26; font-weight: bold">fi</span>
</code></pre></div>
<p>To call it at boot time, just add one line in your <code>/etc/rc.local</code> file (above the <code>exit 0</code> line!):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code>/usr/local/bin/set_charge_tresholds.sh
</code></pre></div>
<h2 id="see-also">See also<a class="headerlink" href="#see-also" title="Permanent link"> </a></h2>
<ul>
<li><a href="https://blog.g3rt.nl/thinkpad-acpi-kernel-module.html">ThinkPad ACPI driver</a>: fan control. </li>
</ul>Using wget or curl with HTTP authentication2011-07-18T12:08:00+02:002011-07-20T13:41:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-07-18:/wget-or-curl-with-http-authentication.html<p>In this post I'll show you how to enter a username/password on the command line for wget/curl if a website requires you to authenticate.</p><p>If you want to download a file directly on your server using <code>wget</code>, but it requires HTTP authentication, then you'll get this error:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="background-color: #404040"><span style="color: #aaaaaa">$ </span>wget<span style="color: #666666"> </span>http://to-some-domain.tld/path/to/file
</span><span style="color: #cccccc">--2011-06-09 18:02:36-- https://to-some-domain.tld/path/to/file</span>
<span style="color: #cccccc">Resolving to-some-domain.tld... 10.20.30.40</span>
<span style="color: #cccccc">Connecting to to-some-domain.tld|10.20.30.40|:80... connected.</span>
<span style="background-color: #404040"><span style="color: #cccccc">HTTP request sent, awaiting response... 401 Authorization Required</span>
</span><span style="color: #cccccc">Authorisation failed.</span>
</code></pre></div>
<p>And here's how to solve it:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>wget<span style="color: #666666"> </span>--user<span style="color: #d0d0d0">=</span>username<span style="color: #666666"> </span>--ask-password<span style="color: #666666"> </span>http://to-some-domain.tld/path/to/file
</code></pre></div>
<p>Alternatively, using <a href="https://curl.haxx.se/">curl</a>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #aaaaaa">$ </span>curl<span style="color: #666666"> </span>-u<span style="color: #666666"> </span>username<span style="color: #666666"> </span>http://to-some-domain.tld/path/to/file
</code></pre></div>How to use a custom backend for HTTP basic authentication in Apache2011-07-14T21:33:00+02:002011-07-18T12:09:00+02:00Gert van Dijktag:blog.g3rt.nl,2011-07-14:/custom-http-basic-authentication-apache.html<p>If you want to secure your webpage with a simple authentication you may want to use an external authentication backend. For example, you already have some authentication in PHP in an existing system, then here's how to extend Apache's HTTP Basic Auth with it.</p><h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link"> </a></h2>
<p>If you want to secure your webpage with a simple authentication you may want to use an external authentication backend.
For example, you already have some authentication in PHP in an existing system, then here's how to extend Apache's HTTP Basic Auth with it.</p>
<h2 id="getting-started">Getting started<a class="headerlink" href="#getting-started" title="Permanent link"> </a></h2>
<p>Install the <code>authnz-external</code> Apache module.
In Debian/Ubuntu:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">apt-get install libapache2-mod-authnz-external</span>
</code></pre></div>
<p>Enable the module.</p>
<p>In Debian/Ubuntu:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">a2enmod authnz_external</span>
</code></pre></div>
<p>then reload Apache's config</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">apache2ctl graceful</span>
</code></pre></div>
<p>An example script to test the credentials in PHP. (<code>myauth.php</code>):</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #d0d0d0">#!/usr/bin/php5</span>
<span style="color: #cd2828; font-weight: bold"><?php</span>
<span style="color: #ababab; font-style: italic">// Read from stdin. First line is the username, second line is the password.</span>
<span style="color: #40ffff">$handle</span> <span style="color: #d0d0d0">=</span> <span style="color: #2fbccd">fopen</span> <span style="color: #d0d0d0">(</span><span style="color: #ed9d13">"php://stdin"</span><span style="color: #d0d0d0">,</span><span style="color: #ed9d13">"r"</span><span style="color: #d0d0d0">);</span>
<span style="color: #40ffff">$username</span> <span style="color: #d0d0d0">=</span> <span style="color: #2fbccd">trim</span><span style="color: #d0d0d0">(</span><span style="color: #2fbccd">fgets</span><span style="color: #d0d0d0">(</span><span style="color: #40ffff">$handle</span><span style="color: #d0d0d0">));</span>
<span style="color: #40ffff">$password</span> <span style="color: #d0d0d0">=</span> <span style="color: #2fbccd">trim</span><span style="color: #d0d0d0">(</span><span style="color: #2fbccd">fgets</span><span style="color: #d0d0d0">(</span><span style="color: #40ffff">$handle</span><span style="color: #d0d0d0">));</span>
<span style="color: #ababab; font-style: italic">// Check the username/password. Below is a very simple example, write your own!</span>
<span style="color: #ababab; font-style: italic">// Probably you want to create a query to some database, add salts, etc.</span>
<span style="color: #6ebf26; font-weight: bold">if</span><span style="color: #d0d0d0">(</span><span style="color: #40ffff">$username</span> <span style="color: #d0d0d0">!=</span> <span style="color: #ed9d13">'gert'</span> <span style="color: #d0d0d0">||</span> <span style="color: #40ffff">$password</span> <span style="color: #d0d0d0">!=</span> <span style="color: #ed9d13">'mypassword'</span><span style="color: #d0d0d0">){</span>
<span style="color: #ababab; font-style: italic"># Output to stdout/stderr will be included in the Apache log for debugging purposes</span>
<span style="color: #6ebf26; font-weight: bold">echo</span> <span style="color: #ed9d13">"wrong username or password for user $username\n"</span><span style="color: #d0d0d0">;</span>
<span style="color: #ababab; font-style: italic"># In case of a failure, sleep a few seconds to slowdown bruteforce attacks.</span>
<span style="color: #2fbccd">sleep</span> <span style="color: #d0d0d0">(</span><span style="color: #51b2fd">3</span><span style="color: #d0d0d0">);</span>
<span style="color: #6ebf26; font-weight: bold">exit</span> <span style="color: #d0d0d0">(</span><span style="color: #51b2fd">1</span><span style="color: #d0d0d0">);</span>
<span style="color: #d0d0d0">}</span> <span style="color: #6ebf26; font-weight: bold">else</span> <span style="color: #d0d0d0">{</span>
<span style="color: #6ebf26; font-weight: bold">echo</span> <span style="color: #ed9d13">"username/password allowed for user $username\n"</span><span style="color: #d0d0d0">;</span>
<span style="color: #6ebf26; font-weight: bold">exit</span> <span style="color: #d0d0d0">(</span><span style="color: #51b2fd">0</span><span style="color: #d0d0d0">);</span>
<span style="color: #d0d0d0">}</span>
<span style="color: #cd2828; font-weight: bold">?></span>
</code></pre></div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This is an example of a PHP5 CLI script (for which you need the <code>php5-cli</code> package).</p>
</div>
<p>While this is a PHP5 script, it could actually be any kind of script or executable which integrates with your current authentication system, as long as it complies with the exit status codes; 0 means OK, anything else means NOT OK.</p>
<p>Important note: In order to slow bruteforce attacks down be sure to set up some sleep time for a failed attempt.
Also, I recommend to configure fail2ban for Apache to actually stop these attacks.</p>
<p>Don't forget to set the script as executable:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><span style="color: #cccccc">chmod +x myauth.php</span>
</code></pre></div>
<p>Define an <code>ExternalAuth</code> directive in for example <code>/etc/apache2/conf.d/authnz_external.conf</code>:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code># define phptest for authentication
DefineExternalAuth phptest pipe /path/to/script/myauth.php
</code></pre></div>
<p>In some site config you need to provide the <code>AuthBasicProvider</code> and <code>AuthExternal</code> directive.
For example, to protect the location <code>/secure</code> on your website:</p>
<div class="highlight" style="background: #202020"><pre style="line-height: 125%;"><span></span><code><Location /secure>
AuthType Basic
AuthName "Gert test"
AuthBasicProvider external
AuthExternal phptest
Require valid-user
</Location>
</code></pre></div>
<p>Finally, reload Apache again and test your configuration!</p>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>You can reuse the configured <code>AuthExternal</code> in any other site configuration on the server.</p>
</div>