<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://xai.sh/feed.xml" rel="self" type="application/atom+xml" /><link href="https://xai.sh/" rel="alternate" type="text/html" /><updated>2025-10-18T21:37:55+02:00</updated><id>https://xai.sh/feed.xml</id><title type="html">./xai.sh</title><entry><title type="html">Build a Signal Desktop AppImage from source</title><link href="https://xai.sh/2023/01/04/Signal-desktop.html" rel="alternate" type="text/html" title="Build a Signal Desktop AppImage from source" /><published>2023-01-04T00:00:00+01:00</published><updated>2023-01-04T00:00:00+01:00</updated><id>https://xai.sh/2023/01/04/Signal-desktop</id><content type="html" xml:base="https://xai.sh/2023/01/04/Signal-desktop.html"><![CDATA[<p>While <a href="https://signal.org/download/">Signal</a> provides packages for Debian-based Linux distributions, there are no official binaries for other distributions available, such as Gentoo or Fedora.</p>

<p>Sure, there are enough of third-party binaries around,
but if you are like me and you would like to avoid running binaries provided by random strangers on the internet, the best strategy is always to build from source.</p>

<p>And Signal Desktop is easy to build!
You can even create an AppImage that contains everything you need to run Signal on your favorite flavor of Linux.</p>

<ol>
  <li><code class="language-plaintext highlighter-rouge">git clone https://github.com/signalapp/Signal-Desktop &amp;&amp; cd Signal-Desktop</code></li>
  <li>Checkout the release you want to build, e.g., <code class="language-plaintext highlighter-rouge">git checkout v6.1.0</code></li>
  <li>Edit <code class="language-plaintext highlighter-rouge">package.json</code> and add <code class="language-plaintext highlighter-rouge">--linux AppImage</code> to the end of the <code class="language-plaintext highlighter-rouge">build:electron</code> target</li>
  <li>Run <code class="language-plaintext highlighter-rouge">git lfs install</code></li>
  <li>Make sure you have the correct nodejs version (requires <a href="https://github.com/nvm-sh/nvm"><code class="language-plaintext highlighter-rouge">nvm</code></a> to be installed): <code class="language-plaintext highlighter-rouge">nvm use</code></li>
  <li>Make sure you have yarn: <code class="language-plaintext highlighter-rouge">npm install --global yarn</code></li>
  <li><code class="language-plaintext highlighter-rouge">yarn install --frozen-lockfile</code></li>
  <li><code class="language-plaintext highlighter-rouge">yarn generate</code></li>
  <li><code class="language-plaintext highlighter-rouge">yarn build</code></li>
  <li>Copy the executable <code class="language-plaintext highlighter-rouge">./release/Signal-x.y.z.AppImage</code> to wherever you want</li>
  <li>Start with <code class="language-plaintext highlighter-rouge">--use-tray-icon</code> to have a nice tray indicator</li>
</ol>

<p>Make sure to regularly check for new releases and rebuild your AppImage to stay up to date and secure.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[How to build a Signal Desktop AppImage from source]]></summary></entry><entry><title type="html">Compile Mozilla VPN client from source</title><link href="https://xai.sh/2022/12/30/mozilla-vpn-from-source.html" rel="alternate" type="text/html" title="Compile Mozilla VPN client from source" /><published>2022-12-30T00:00:00+01:00</published><updated>2022-12-30T00:00:00+01:00</updated><id>https://xai.sh/2022/12/30/mozilla-vpn-from-source</id><content type="html" xml:base="https://xai.sh/2022/12/30/mozilla-vpn-from-source.html"><![CDATA[<h2 id="what-is-mozilla-vpn">What is Mozilla VPN</h2>

<p><a href="https://vpn.mozilla.org">Mozilla VPN</a> is a VPN client developed by Mozilla.
It is open source and uses <a href="https://www.wireguard.com/">WireGuard</a>.
It is available for Linux, Windows, Mac, Android, and iOS.
To use it, you need to buy a subscription from Mozilla.</p>

<h2 id="building-from-source">Building from source</h2>

<p>As usual, Linux binaries are only provided for Debian-based systems.
If you want to run <a href="https://vpn.mozilla.org">Mozilla’s VPN client</a> on e.g., Gentoo, you might want to build from source.</p>

<p>So cloning the <a href="https://github.com/mozilla-mobile/mozilla-vpn-client">repository</a> is the first step:</p>

<pre><code class="language-[sh]">git clone https://github.com/mozilla-mobile/mozilla-vpn-client
</code></pre>

<p>It uses git submodules, so we have to do</p>

<pre><code class="language-[sh]">git submodule init
git submodule update
</code></pre>

<p>The project uses <code class="language-plaintext highlighter-rouge">cmake</code>, so it should be easy enough. In theory!</p>

<p>If you do not have <code class="language-plaintext highlighter-rouge">rust</code> installed yet, that is the time!
Get the toolchain installer and run <code class="language-plaintext highlighter-rouge">rustup default stable</code> to get everything set up.
Also, you need <code class="language-plaintext highlighter-rouge">wireguard</code> and its <code class="language-plaintext highlighter-rouge">wireguard-tools</code> installed.</p>

<p>Surprisingly, Gentoo’s <code class="language-plaintext highlighter-rouge">Qt</code> is currently too old (<code class="language-plaintext highlighter-rouge">Qt6</code> is required by the vpn client) and even with the qt overlay and after unmasking a bunch of hard-masked stuff I was not able to produce a successful build.</p>

<p>So I figured that I had to compile <code class="language-plaintext highlighter-rouge">Qt6</code> from source first.
I was delighted when I found out that in the repository of the vpn client a <a href="https://github.com/mozilla-mobile/mozilla-vpn-client/blob/main/scripts/utils/qt6_compile.sh">script</a> is provided for that exact purpose!</p>

<h3 id="building-qt6">Building Qt6</h3>

<p>How nice! So downloading and unpacking the <code class="language-plaintext highlighter-rouge">qt6</code> sources and running</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>scripts/utils/qt6_compile.sh path/to/qt-everywhere-src-6.2.4 path/to/destination
</code></pre></div></div>

<p>should be all that is needed for that.</p>

<p>However, I learned that my cmake was not built with zstd support and I did not find a quick way to resolve that in Gentoo.
So, I added the line</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">set</span><span class="o">(</span>QT_AVOID_CMAKE_ARCHIVING_API ON<span class="o">)</span>
</code></pre></div></div>

<p>in <code class="language-plaintext highlighter-rouge">CMakeLists.txt</code> of the <code class="language-plaintext highlighter-rouge">qt6</code> sources and then everything was fine.</p>

<h4 id="update-2023-11">Update 2023-11</h4>

<p>It seems that in some cases, qt6.6 has issues building and you might need to add</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-skip qtspeech \
</code></pre></div></div>
<p>to <code class="language-plaintext highlighter-rouge">scripts/utils/qt6_compile.sh</code></p>

<h3 id="building-mozilla-vpn-client">Building Mozilla VPN Client</h3>

<p>First, we need to install the python dependencies:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install</span> <span class="nt">-r</span> requirements.txt <span class="nt">--user</span>
</code></pre></div></div>

<p>Now we can use the compiled <code class="language-plaintext highlighter-rouge">qt6</code> to configure the project:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>build <span class="o">&amp;&amp;</span> cmake <span class="nt">-S</span> <span class="nb">.</span> <span class="nt">-B</span> build <span class="nt">-DCMAKE_PREFIX_PATH</span><span class="o">=</span>path/to/qt6/lib/cmake
</code></pre></div></div>

<p>If everything went well, we compile the vpn client:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cmake <span class="nt">--build</span> build <span class="nt">-j</span><span class="si">$(</span><span class="nb">nproc</span><span class="si">)</span>
</code></pre></div></div>

<p>Finally,</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>cmake <span class="nt">--install</span> build
</code></pre></div></div>

<p>installs everything to <code class="language-plaintext highlighter-rouge">/usr/local</code> including a <code class="language-plaintext highlighter-rouge">.desktop</code> file in <code class="language-plaintext highlighter-rouge">/usr/local/share/applications</code>, so you can start the tool conveniently.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[How to compile Mozilla VPN client from source]]></summary></entry><entry><title type="html">Rate-limit userspace processes and virtual machines</title><link href="https://xai.sh/2021/05/30/Rate-limit-processes.html" rel="alternate" type="text/html" title="Rate-limit userspace processes and virtual machines" /><published>2021-05-30T00:00:00+02:00</published><updated>2021-05-30T00:00:00+02:00</updated><id>https://xai.sh/2021/05/30/Rate-limit-processes</id><content type="html" xml:base="https://xai.sh/2021/05/30/Rate-limit-processes.html"><![CDATA[<p>In a <a href="/2019/02/08/Rate-limiting-pipes.html">previous post</a>,
I mentioned how to rate-limit Unix pipes.<br />
But what can we do about other processes?</p>

<h2 id="trickle">Trickle</h2>

<p><a href="https://github.com/mariusae/trickle">Trickle</a> is a userland bandwith
shaper that can be used to manage the bandwith usage of any userspace
process (unless it is statically linked or setuid):</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>trickle <span class="nt">-d</span> 300 <span class="nt">-u</span> 300 firefox
</code></pre></div></div>

<p>starts firefox and limits it to 300 KB/s downstream and upstream.</p>

<h2 id="libvirt-virtual-machines">Libvirt Virtual Machines</h2>

<p>To limit the bandwidth usage of a whole virtual machine,
we can use <a href="https://libvirt.org/formatnetwork.html#elementQoS">Libvirt’s QoS capabilities</a>:</p>

<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;bandwidth&gt;</span>
  <span class="nt">&lt;inbound</span> <span class="na">average=</span><span class="s">"300"</span> <span class="na">peak=</span><span class="s">"800"</span><span class="nt">/&gt;</span>
  <span class="nt">&lt;outbound</span> <span class="na">average=</span><span class="s">"300"</span> <span class="na">peak=</span><span class="s">"800"</span><span class="nt">/&gt;</span>
<span class="nt">&lt;/bandwidth&gt;</span>
</code></pre></div></div>

<p>inside of the <code class="language-plaintext highlighter-rouge">&lt;interface&gt;</code> block of a virtual machine aims for an
average bandwith usage of 300 KB/s with a maximum peak of 800 KB/s,
each up- and downstream.</p>

<p>I mostly use this to upgrade one or more virtual machines in the
background without disturbing other applications or video calls too
much.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[How to save bandwidth by limiting the bandwith usage of userspace processes and virtual machines]]></summary></entry><entry><title type="html">Use git filter to prevent a change from being committed</title><link href="https://xai.sh/2020/11/03/git-filter.html" rel="alternate" type="text/html" title="Use git filter to prevent a change from being committed" /><published>2020-11-03T00:00:00+01:00</published><updated>2020-11-03T00:00:00+01:00</updated><id>https://xai.sh/2020/11/03/git-filter</id><content type="html" xml:base="https://xai.sh/2020/11/03/git-filter.html"><![CDATA[<p>Especially when working on a paper, a regular problem I end up with is
the following:<br />
Maybe I use a different version of texlive than a co-author, e.g., a more
recent one, and the document won’t compile unless I start making changes in the
preamble.
So I need to make those changes to compile and view the paper but I don’t want
to commit them along with my contributions to the repository — because if I
would do that, the other authors could probably no longer compile the document.</p>

<p>Two inconvenient solutions for this are:</p>

<ol>
  <li>Use interactive staging for each commit and make sure that this particular
change doesn’t end up in the commit.<br />
Until I accidentally stage it at some point in time and discover that it made it
into the upstream branch. Which, at least for me, is really just a matter of time.</li>
  <li>Commit the fix to a dedicated branch, make the contributions on top of it
and when ready, cherry-pick them into the target branch.
Variations of this are working with an integration branch.
This solution is a lot of juggling branches and commits around
and will in the end break my running latexmk build.</li>
</ol>

<p>Turns out this can be done elegantly with the filter functionality from git-attributes.
With git filter you can automatically run two scripts: a smudge script when a file is checked out, and clean script when a file is checked in.</p>

<p>So in my case, I can define in <code class="language-plaintext highlighter-rouge">.git/config</code> what I want to happen:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[filter "fix-texlive2019"]
 smudge = sed '/^\\\\usepackage{amssymb}.*/d'
 clean = sed '/^\\\\usepackage{booktabs}/i \\\\\\usepackage{amssymb}'
</code></pre></div></div>

<p>While the escaping-hell looks a bit odd, these sed commands achieve exactly what I need:
In my workspace, the <code class="language-plaintext highlighter-rouge">\includepackage{amssymb}</code> line will be removed
(because it created a definition conflict with another package in my texlive version),
whereas during a check-in, the line will be inserted right before the
<code class="language-plaintext highlighter-rouge">\usepackage{booktabs}</code> line, which is the place where it was before.</p>

<p>Now we just have to assign the filter to tex files, which is done in <code class="language-plaintext highlighter-rouge">.git/info/attributes</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>*.tex filter=fix-texlive2019
</code></pre></div></div>

<p>Using this approach allows me to magically have my necessary changes,
being able to compile the document at all times,
without having to worry about accidentally committing the required local fix.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[How to use git filter to prevent a change from being committed]]></summary></entry><entry><title type="html">Move a running process into a tmux session</title><link href="https://xai.sh/2020/10/16/Move-running-process-into-tmux-session.html" rel="alternate" type="text/html" title="Move a running process into a tmux session" /><published>2020-10-16T00:00:00+02:00</published><updated>2020-10-16T00:00:00+02:00</updated><id>https://xai.sh/2020/10/16/Move-running-process-into-tmux-session</id><content type="html" xml:base="https://xai.sh/2020/10/16/Move-running-process-into-tmux-session.html"><![CDATA[<p>It’s rare, but sometimes it still happens that I forget to open a tmux or
screen session when working with something that is supposed to be quickly done.
However, it also happens that “quickly done” turns into “tedious and ugly” and now
the process lives longer than it was supposed to and I become afraid of ssh
disconnects or something.</p>

<p>So an obvious solution is killing the process and running it in a newly
created tmux session — but what if the process ran for a while and I don’t want
to kill it because I either lose progress or end up in a mess?
Instead of killing and re-running a process, it would be much smoother to just
move it into a tmux session. This involves changing the parent of a process,
which is not exactly trivial, but thankfully @nelhage made a tool for that: <a href="https://github.com/nelhage/reptyr">reptyr</a>.
If you’re interested in how <code class="language-plaintext highlighter-rouge">reptyr</code> actually achieves its goal, check out his blog posts[^1]<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>!</p>

<p>As for usage, it is very easy:</p>

<ol>
  <li>Suspend the respective process with <code class="language-plaintext highlighter-rouge">Ctrl-Z</code></li>
  <li>Send the job to background using <code class="language-plaintext highlighter-rouge">bg</code></li>
  <li>Take away the ownership from the shell using <code class="language-plaintext highlighter-rouge">disown</code></li>
  <li>Start or enter your tmux/screen session</li>
  <li>Run <code class="language-plaintext highlighter-rouge">reptyr PID</code> to attach the process to the current shell</li>
</ol>

<p>It also has some additional useful features, such as TTY-stealing,
which is documented in the man page.</p>

<p>Before compiling <code class="language-plaintext highlighter-rouge">reptyr</code>, make sure to check whether it is in your distributions repository. At the time of writing, this was at least the case for Gentoo, Fedora, and Debian.</p>

<p><strong>Update (2023-01-05): ptrace scoping</strong><br />
A fellow Gentoo enthusiast noticed that on recent systems, the following error occurs when invoking <code class="language-plaintext highlighter-rouge">reptyr</code> as regular user:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Unable to attach to pid 1348999: Operation not permitted
The kernel denied permission while attaching. If your uid matches
the target's, check the value of /proc/sys/kernel/yama/ptrace_scope.
For more information, see /etc/sysctl.d/10-ptrace.conf
</code></pre></div></div>

<p>And this is how I learned about <a href="https://www.kernel.org/doc/Documentation/security/Yama.txt">ptrace scoping</a>,
a feature that was added to the Linux kernel in version 3.4.</p>

<p><em>The problem</em>:<br />
In order to attach to a process, <code class="language-plaintext highlighter-rouge">reptyr</code> uses the <code class="language-plaintext highlighter-rouge">ptrace</code> system call, which
is used to debug processes. In order to prevent unprivileged users from
attaching to processes, the kernel has a feature called “ptrace scoping” which
allows you to restrict which users and processes can attach to which processes.
The value of <code class="language-plaintext highlighter-rouge">/proc/sys/kernel/yama/ptrace_scope</code> determines the scope of
<code class="language-plaintext highlighter-rouge">ptrace</code>:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">0</code>: No restrictions, anything can be attached as long as the uids match.</li>
  <li><code class="language-plaintext highlighter-rouge">1</code>: Only the process owner and root can attach to a process. Also, some kind of relationship is required between the processes.</li>
  <li><code class="language-plaintext highlighter-rouge">2</code>: Only root can attach to a process.</li>
  <li><code class="language-plaintext highlighter-rouge">3</code>: No one can attach to a process.</li>
</ul>

<p>On my Gentoo, it was set to <code class="language-plaintext highlighter-rouge">1</code> by default, which seems reasonable.
As there is no relationship between the process I want to move and the <code class="language-plaintext highlighter-rouge">reptyr</code> process, <code class="language-plaintext highlighter-rouge">reptyr</code> is denied the permission to attach via <code class="language-plaintext highlighter-rouge">ptrace</code>.</p>

<p>So what can we do about this?
I assumed that I could use the <code class="language-plaintext highlighter-rouge">prctl</code> system call to set <code class="language-plaintext highlighter-rouge">PR_SET_PTRACER</code> to <code class="language-plaintext highlighter-rouge">PR_SET_PTRACER_ANY</code> for the process I want to move.
However, it seems that I can only set a tracer for the calling process, not for an arbitrary one, which I find rather annoying.</p>

<p>If somebody finds out how to do this, I would be glad to include it here instead of the following “dirty” workaround.</p>

<p><em>The workaround</em>:<br />
The only remaining option I found so far is temporarily setting <code class="language-plaintext highlighter-rouge">ptrace_scope</code> to <code class="language-plaintext highlighter-rouge">0</code> before invoking <code class="language-plaintext highlighter-rouge">reptyr</code>,
which can be done by running <code class="language-plaintext highlighter-rouge">sysctl -w kernel.yama.ptrace_scope=0</code>.
Note that this creates a security problem, as now <em>all</em> processes can be attached.
For security reasons, I prefer to reset it to <code class="language-plaintext highlighter-rouge">1</code> right after the process has been re-attached.</p>

<hr />

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:2">
      <p><a href="https://blog.nelhage.com/2011/02/changing-ctty/">https://blog.nelhage.com/2011/02/changing-ctty/</a> <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name></name></author><summary type="html"><![CDATA[How to move a running process into a tmux session]]></summary></entry><entry><title type="html">Finding files in git repositories using checksums</title><link href="https://xai.sh/2019/03/13/Find-file-in-git-by-hash.html" rel="alternate" type="text/html" title="Finding files in git repositories using checksums" /><published>2019-03-13T00:00:00+01:00</published><updated>2019-03-13T00:00:00+01:00</updated><id>https://xai.sh/2019/03/13/Find-file-in-git-by-hash</id><content type="html" xml:base="https://xai.sh/2019/03/13/Find-file-in-git-by-hash.html"><![CDATA[<p>Recently I wanted to replicate an evaluation of an older paper, which also required using an older version of one of my tools.
I knew exactly which code was used for the evaluation because I had a tar (jdime-src.tar.xz) of the source code.
What I did not know was whether this tar was the result of a specific release, a specific commit, or just something that was never put into git in this very state — So I wanted to find out and (if necessary) create a respective commit and tag it.</p>

<p>I started by extracting the tar and generating checksums for each relevant file:<br />
<code class="language-plaintext highlighter-rouge">tar -xJf jdime-src.tar.xz &amp;&amp; cd jdime-src</code><br />
<code class="language-plaintext highlighter-rouge">find src/ -type f -name \*.java -exec sha1sum {} \; &gt; /tmp/sha1sums</code></p>

<p><code class="language-plaintext highlighter-rouge">/tmp/sha1sums</code> now looked like this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>015bd1948789546e977266196fbea427b2eb64d1  src/de/fosd/jdime/merge/package-info.java
c8cdccd30d215fb955a9b0011a442d6a792cf91f  src/de/fosd/jdime/merge/MergeInterface.java
28722ec3fd2e280519cc4d9bdc97496f4ae57505  src/de/fosd/jdime/merge/UnorderedMerge.java
5c1b1082417790b853c9bb971cf22667ecc25274  src/de/fosd/jdime/merge/OrderedMerge.java
bc18227a8c63100bb5d97e9d5923e76df3d9eb46  src/de/fosd/jdime/merge/Merge.java
...
</code></pre></div></div>

<p>Next, I needed a way to look for the files in my git repository and find the right commit for each file (i.e., if there is one that matches the checksum).</p>

<p>I found <a href="https://gist.github.com/mloberg/3750653">this gist</a> on github by <a href="https://github.com/mloberg">mloberg</a>,
that I rewrote a little bit for my needs.<br />
The result was this (also available as <a href="https://gist.github.com/xai/3f6f3c0518e77bc87afd922519a9b370">gist</a>):</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="c"># find-by-hash.sh</span>

usage<span class="o">()</span> <span class="o">{</span>
 <span class="nb">echo</span> <span class="s2">"Usage: </span><span class="nv">$0</span><span class="s2"> [-m] [-s] hash file"</span>
 <span class="nb">echo</span> <span class="s2">"</span><span class="se">\t</span><span class="s2">-m use md5 for hashing"</span>
 <span class="nb">echo</span> <span class="s2">"</span><span class="se">\t</span><span class="s2">-s use sha1 for hashing (this is the default)"</span>
 <span class="nb">exit </span>1
<span class="o">}</span>

<span class="nv">HASHCMD</span><span class="o">=</span><span class="s2">"sha1sum"</span>

<span class="k">while </span><span class="nb">getopts</span> <span class="s2">":m:s"</span> opt<span class="p">;</span> <span class="k">do
 case</span> <span class="nv">$opt</span> <span class="k">in
  </span>m<span class="p">)</span>
   <span class="nv">HASHCMD</span><span class="o">=</span><span class="s2">"md5sum"</span>
   <span class="nb">shift</span>
   <span class="p">;;</span>
  s<span class="p">)</span>
   <span class="nv">HASHCMD</span><span class="o">=</span><span class="s2">"sha1sum"</span>
   <span class="nb">shift</span>
   <span class="p">;;</span>
 <span class="k">esac</span>
<span class="k">done

</span><span class="nv">CHECKSUM</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">FILE</span><span class="o">=</span><span class="nv">$2</span>

<span class="k">if</span> <span class="o">[</span> <span class="nt">-z</span> <span class="s2">"</span><span class="nv">$CHECKSUM</span><span class="s2">"</span> <span class="nt">-o</span> <span class="nt">-z</span> <span class="s2">"</span><span class="nv">$FILE</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
 </span>usage
<span class="k">fi</span>

<span class="c"># Check if valid git repo</span>
<span class="nv">ROOT</span><span class="o">=</span><span class="si">$(</span>git rev-parse <span class="nt">--show-toplevel</span><span class="si">)</span>

<span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> <span class="nt">-ne</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
 </span><span class="nb">echo</span> <span class="s2">"Not a valid git repo."</span>
 <span class="nb">exit </span>1
<span class="k">fi

</span><span class="nb">cd</span> <span class="nv">$ROOT</span>

<span class="c"># git revision for file</span>
<span class="nv">REVS</span><span class="o">=</span><span class="si">$(</span>git rev-list <span class="nt">--all</span> <span class="nt">--</span> <span class="nv">$FILE</span><span class="si">)</span>

<span class="c"># temp file</span>
<span class="nv">file_to_check</span><span class="o">=</span><span class="si">$(</span><span class="nb">mktemp</span><span class="si">)</span>

<span class="c"># check each revision for checksum</span>
<span class="nv">FOUND</span><span class="o">=</span><span class="s2">""</span>
<span class="k">for </span>rev <span class="k">in</span> <span class="nv">$REVS</span><span class="p">;</span> <span class="k">do
 </span>git show <span class="nv">$rev</span>:<span class="nv">$FILE</span> <span class="o">&gt;</span> <span class="nv">$file_to_check</span> 2&gt;/dev/null
 <span class="k">if</span> <span class="nv">$HASHCMD</span> <span class="nv">$file_to_check</span> | <span class="nb">grep</span> <span class="nt">-q</span> <span class="nv">$CHECKSUM</span><span class="p">;</span> <span class="k">then
  </span><span class="nv">FOUND</span><span class="o">=</span><span class="s2">"</span><span class="nv">$rev</span><span class="s2">"</span>
  <span class="c"># intentionally no break to see if we find an older revision</span>
  <span class="c"># insert a break if you want the most recent commit instead of the oldest</span>
 <span class="k">fi
done</span>

<span class="c"># cleanup</span>
<span class="nb">rm</span> <span class="nv">$file_to_check</span>

<span class="c"># output</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-n</span> <span class="s2">"</span><span class="nv">$FOUND</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
 </span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$FOUND</span><span class="s2">"</span>
 <span class="nb">exit </span>0
<span class="k">else
 </span><span class="nb">echo</span> <span class="s2">"Not found: </span><span class="nv">$CHECKSUM</span><span class="s2"> </span><span class="nv">$FILE</span><span class="s2">"</span>
 <span class="nb">exit </span>1
<span class="k">fi</span>

</code></pre></div></div>

<p>The rest was easy: Executing the script for each line in <code class="language-plaintext highlighter-rouge">/tmp/sha1sums</code> reveals whether there is a matching commit in the repository in the format <code class="language-plaintext highlighter-rouge">commit checksum filename</code>:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;</span> <span class="k">while </span><span class="nb">read</span> <span class="nt">-r</span> line<span class="p">;</span> <span class="k">do </span><span class="nb">echo</span> <span class="s2">"</span><span class="si">$(</span>./find-by-hash.sh <span class="nv">$line</span><span class="si">)</span><span class="s2"> </span><span class="nv">$line</span><span class="s2">"</span><span class="p">;</span> <span class="k">done</span> &lt; /tmp/sha1sums | <span class="nb">tee</span> /tmp/results
6b14e2bf8eeaba3176c4e310c055cd4480e06ebd 8f2ebe7afe4b540516388763688675d8  src/de/fosd/jdime/merge/package-info.java
6b14e2bf8eeaba3176c4e310c055cd4480e06ebd bb5d89779398697f956d7f6d5e7d16b6  src/de/fosd/jdime/merge/MergeInterface.java
...
</code></pre></div></div>

<p>Let’s see how many different commits we’re dealing with:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;</span> <span class="nb">awk</span> <span class="s1">'{print $1}'</span> /tmp/results | <span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span>
     50 6b14e2bf8eeaba3176c4e310c055cd4480e06ebd
      1 a0ba24e5c0237d6cb1c394cbfadb9d9df895a410
      2 de655106866164efccd060b36064c8c6db87cb7a
      1 f2f5265f1dd666948962cdeb63e90d08c1cf322c
</code></pre></div></div>

<p>Great, this means each file was found in the exact version that it is in the tar!
They’re just spread over 4 different commits, but that’s no big problem.
What I wanted to do next was to create a commit that resembles the state of the tar,
so I can put a tag on it and find it easily in the future.</p>

<p>The majority of files (50) is already in a single commit (<code class="language-plaintext highlighter-rouge">6b14e2bf</code>), so I used that as a base for the branch:<br />
<code class="language-plaintext highlighter-rouge">git checkout -b old-evaluation 6b14e2bf8eeaba3176c4e310c055cd4480e06ebd</code></p>

<p>So what about the 4 remaining files that are from different commits?
Let’s just put them on top of our new branch:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for </span>rev <span class="k">in</span> <span class="si">$(</span><span class="nb">awk</span> <span class="s1">'{print $1}'</span> /tmp/results | <span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span> | <span class="nb">tail</span> <span class="nt">-n</span>+2 | <span class="nb">awk</span> <span class="s1">'{print $2}'</span><span class="si">)</span><span class="p">;</span> <span class="k">do
 </span><span class="nv">file</span><span class="o">=</span><span class="si">$(</span><span class="nb">grep</span> <span class="nv">$rev</span> /tmp/results | <span class="nb">awk</span> <span class="s1">'{print $3}'</span><span class="si">)</span>
 git checkout <span class="nv">$rev</span> <span class="nt">--</span> <span class="nv">$file</span> <span class="o">&amp;&amp;</span> git add <span class="nv">$file</span>
<span class="k">done</span>
</code></pre></div></div>

<p>All there was left to do was committing these changes and tagging the revision:<br />
<code class="language-plaintext highlighter-rouge">git commit -m ... &amp;&amp; git tag -a ...</code></p>]]></content><author><name></name></author><summary type="html"><![CDATA[How to find files in git repositories using checksums]]></summary></entry><entry><title type="html">Set default python version in Debian stretch</title><link href="https://xai.sh/2019/03/07/Debian-stretch-default-python.html" rel="alternate" type="text/html" title="Set default python version in Debian stretch" /><published>2019-03-07T00:00:00+01:00</published><updated>2019-03-07T00:00:00+01:00</updated><id>https://xai.sh/2019/03/07/Debian-stretch-default-python</id><content type="html" xml:base="https://xai.sh/2019/03/07/Debian-stretch-default-python.html"><![CDATA[<p>With both <code class="language-plaintext highlighter-rouge">python</code> and <code class="language-plaintext highlighter-rouge">python3</code> packages installed,
my Debian stretch systems seem to use python2.7 when /usr/bin/python is executed.<br />
I prefer python3 as a default, as python2 is IMHO somewhat deprecated
and I try to avoid it whenever I can.</p>

<p>Therefore, I was surprised that <code class="language-plaintext highlighter-rouge">update-alternatives --config python</code>
resulted in an error message:<br />
<code class="language-plaintext highlighter-rouge">update-alternatives: error: no alternatives for python</code></p>

<p>However, this is easily fixed by manually installing alternative entries:
<code class="language-plaintext highlighter-rouge">update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1</code><br />
<code class="language-plaintext highlighter-rouge">update-alternatives --install /usr/bin/python python /usr/bin/python3.5 2</code></p>

<p>A higher number specifies a higher priority, which in this case results in <code class="language-plaintext highlighter-rouge">python3</code> being the default from now on.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[How to set the default python version in Debian stretch]]></summary></entry><entry><title type="html">Adding arbitrary files to a gist</title><link href="https://xai.sh/2019/02/25/Add-arbitrary-files-to-gists.html" rel="alternate" type="text/html" title="Adding arbitrary files to a gist" /><published>2019-02-25T00:00:00+01:00</published><updated>2019-02-25T00:00:00+01:00</updated><id>https://xai.sh/2019/02/25/Add-arbitrary-files-to-gists</id><content type="html" xml:base="https://xai.sh/2019/02/25/Add-arbitrary-files-to-gists.html"><![CDATA[<p>While <a href="https://gist.github.com">gist</a>s provide a nice alternative to pastebins,
especially when sharing source code, you might at first find them a bit limited.</p>

<p>Sometimes I want to include a pdf or an image, e.g., a plot or a screenshot in a gist.<br />
While this is not possible using the web interface,
it can easily be done by operating on the gist directly with git
(gists are git repositories):</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone git@gist.github.com:HASH.git /tmp/mygist
<span class="nb">cd</span> /tmp/mygist
<span class="nb">cp</span> /path/to/plot.pdf <span class="nb">.</span> <span class="o">&amp;&amp;</span> git add plot.pdf <span class="o">&amp;&amp;</span> git commit <span class="nt">-m</span> <span class="s2">"Add plot"</span>
git push
</code></pre></div></div>

<p>Done :)</p>]]></content><author><name></name></author><summary type="html"><![CDATA[How to add arbitrary files to a gist]]></summary></entry><entry><title type="html">Rate-limit pipes with pv</title><link href="https://xai.sh/2019/02/08/Rate-limiting-pipes.html" rel="alternate" type="text/html" title="Rate-limit pipes with pv" /><published>2019-02-08T00:00:00+01:00</published><updated>2019-02-08T00:00:00+01:00</updated><id>https://xai.sh/2019/02/08/Rate-limiting-pipes</id><content type="html" xml:base="https://xai.sh/2019/02/08/Rate-limiting-pipes.html"><![CDATA[<p>Here’s a short one: <a href="http://www.ivarch.com/programs/pv.shtml">pv</a>
can be used to rate-limit pipes.<br />
pv is often used to show the progress of operations, but it offers a lot more.</p>

<p>In my case, I had to limit the network usage of a zfs send/recv that was running over ssh,
because the traffic was disturbing other processes who needed the bandwidth as well.</p>

<p>Turns out it’s easy with pv:<br />
<code class="language-plaintext highlighter-rouge">zfs send ... | pv -L 5M | ssh remote zfs recv ...</code><br />
limits the pipe to 5M/s.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[How to rate-limit pipes with pv]]></summary></entry><entry><title type="html">Encrypted backups with restic</title><link href="https://xai.sh/2018/10/08/restic.html" rel="alternate" type="text/html" title="Encrypted backups with restic" /><published>2018-10-08T00:00:00+02:00</published><updated>2018-10-08T00:00:00+02:00</updated><id>https://xai.sh/2018/10/08/restic</id><content type="html" xml:base="https://xai.sh/2018/10/08/restic.html"><![CDATA[<p>I find <a href="https://restic.net/">restic</a> a convincing tool for encrypted incremental backups.<br />
It supports a huge number of backends - in the following sftp is used.</p>

<p>As an example, I show how I backup my ~/.mail (which is fetched by <a href="http://isync.sourceforge.net/mbsync.html">mbsync</a> in maildir format).</p>

<h2 id="initialization">Initialization</h2>

<p>Before we can store our data, we have to initialize a restic repository:<br />
<code class="language-plaintext highlighter-rouge">restic -r sftp:user@remotehost:/path/to/storage/mail init</code><br />
Restic will now let you set the encryption passphrase.</p>

<h2 id="backup">Backup</h2>

<p>Now we can start our first backup operation:<br />
<code class="language-plaintext highlighter-rouge">restic -r sftp:user@remotehost:/path/to/storage/mail backup ~/.mail --exclude=".notmuch/" --exclude="*.mbsyncstate"</code><br />
The exclude arguments in this example are used to avoid copying the state files of mbsync and the index created by notmuch.</p>

<p>Subsequent backups are issued the same way and are performed in an incremental way.<br />
Each backup is represented by a snapshot in the restic repository.</p>

<h2 id="list">List</h2>

<p>List all snapshots:<br />
<code class="language-plaintext highlighter-rouge">restic -r sftp:user@remotehost:/path/to/storage/mail snapshots</code></p>

<h2 id="restore">Restore</h2>

<p>If we want to restore a snapshot, we just retrieve the hash with the above command and issue a restore:<br />
<code class="language-plaintext highlighter-rouge">restic -r sftp:user@remotehost:/path/to/storage/mail restore HASH --target /tmp/restore-mail</code></p>

<p>Another convenient way to access the backups is the ability to use fuse mounts:<br />
<code class="language-plaintext highlighter-rouge">restic -r sftp:user@remotehost:/path/to/storage/mail mount /mnt/restic-mail</code></p>

<h2 id="read-password-from-gpg-encrypted-file">Read password from gpg-encrypted file</h2>

<p>I usually don’t want to remember/type/paste my encryption passphrase, so let’s store it in a file encrypted with our gpg key.<br />
Using bash process substitution (not /bin/sh compatible), we can then operate on our repository like this:<br />
<code class="language-plaintext highlighter-rouge">restic -p &lt;(gpg -dq ~/.restic-password.gpg) -r ...</code><br />
Now we just have to decrypt the password file using our gpg passphrase (or query a running gpg-agent process).<br />
Note that with this approach, the encryption passphrase appears in the process list - so maybe don’t do this in multi user environments.</p>

<h2 id="scripting">Scripting</h2>

<p>If you want to trigger your backups with cron or systemd-timers, you might be looking for a way to automatically provide the encryption passphrase.<br />
This can be done by storing the passphrase in the RESTIC_PASSWORD shell variable,
but please make sure that’s really what you want to do.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[A guide on how to use restic for encrypted incremental backups]]></summary></entry></feed>