tNew article about software packaging - monochromatic - monochromatic blog: http://blog.z3bra.org
HTML git clone git://z3bra.org/monochromatic
DIR Log
DIR Files
DIR Refs
---
DIR commit 251e9d348d825c5b45f0ecef5d19692bdc25ef01
DIR parent c324148a7e8621ec00c94808b1e52ff8fd0f2f1b
HTML Author: z3bra <willyatmailoodotorg>
Date: Fri, 16 Sep 2016 00:57:56 +0200
New article about software packaging
Diffstat:
A 2016/09/pack-it-up.txt | 220 +++++++++++++++++++++++++++++++
M config.mk | 3 ++-
M index.txt | 1 +
3 files changed, 223 insertions(+), 1 deletion(-)
---
DIR diff --git a/2016/09/pack-it-up.txt b/2016/09/pack-it-up.txt
t@@ -0,0 +1,220 @@
+# [Pack it up!](#)
+## — 16 September, 2016
+
+Today was a big day for me. It was the day all my software could play
+together to reach a single goal:
+
+*Maintaining and hosting software packages!*
+
+Here are the tools:
+
+* [pm](http://z3bra.org/pm) - manage local packs
+* [sick](http://z3bra.org/sick) - sign and check files
+* [synk](http://z3bra.org/synk) - synchronize files between hosts
+* [wendy](http://z3bra.org/wendy) - run a command on filesystem change
+
+Tied together with a fistful of scripts, and some easy preparation,
+they help medeploy and manage my own software accross multiple operating
+systems without efforts.
+
+This process includes three main tasks: packaging, deploying and
+installing. This whole process is still at and early stage, and will
+get stronger and sharper with time (and shell script will most likely
+turn into actual programs).
+Here is an explanation of how it works:
+
+0. pack an application and sign the tarball
+1. upload the tarball to an online repository
+2. get the latest tarball for a utility
+3. install/update it locally
+
+The process is straigh-forward, and multiple parts can be automated.
+
+### Packaging
+
+In order to build my softwares, I need either `make` or
+[`mk`](http://doc.cat-v.org/plan_9/4th_edition/papers/mk) (it's tending
+to be `mk` only, but for now I still need both). Then process is then
+fairly simple:
+
+ $ cd $utility
+ $ mk
+ # mk install
+
+I wrote a quick script to "pack" these utilities for me. It will build
+the software, install it to a temporary location, create a tarball out
+of it, and sign this tarball with my private key:
+
+ #!/bin/sh
+
+ # user specific variables
+ SICKKEY=$HOME/.sick.d/$USER.key
+ REPO=/var/www/htdocs/dl.z3bra.org/pack
+
+ # guess the pack name from current directory, or use given name
+ DIR=$(basename `pwd`)
+ PKG=${1:-$DIR}
+
+ # set version from latest git tag, if applicable
+ TAG=$(git tag | sed -n '$p' | tr -d a-z)
+ VER=${TAG:-0.0}
+
+ # this part should die...
+ test -f mkfile && MK=mk || MK=make
+
+ # build pack and install to ./rootfs/usr
+ $MK
+ $MK DESTDIR=$(pwd)/rootfs PREFIX=/usr MANDIR=/usr/share/man install
+ (
+ cd rootfs
+ mkdir -p $REPO
+ # pack and sign the installed utility
+ tar cvj * | sick -s -f ${SICKKEY} > $REPO/${PKG}#${VER}.tar.bz2
+ )
+ rm -rf rootfs
+
+ # simply ensure that the file has been created correctly
+ echo
+ ls $REPO/${PKG}#${VER}.tar.bz2
+
+At this point, to pack one of my utilities, all I need is:
+
+ $ git clone git://z3bra.org/skroll
+ $ cd skroll
+ $ pack
+
+And I'm done :)
+
+### Deploying
+
+This part require a bit of setup. My current repository is at
+http://dl.z3bra.org/pack, which is, locally in
+`/var/www/htdocs/dl.z3bra.org/pack`. My tool `synk` can get a file
+synchronized between two peers, but they will have the same path,
+which is why I also created this directory on my local machine.
+I also need to upload my public key (for `sick` checks) and a list
+of what's currently in the repo.
+
+First, here is the `repogen` script, which will list the content of the
+local repo, and write the pack names and version available to a file:
+
+ #!/bin/sh
+
+ REPO=/var/www/htdocs/dl.z3bra.org/pack
+ for tarball in $(find $REPO -name '*.tar.bz2'); do
+ pkg=$(basename $tarball | cut -d# -f1)
+ ver=$(basename $tarball | cut -d# -f2 | sed 's/.tar.bz2//')
+ printf '%s\t%s\n' "$pkg" "$ver"
+ done | sort | tee $REPO/.list
+
+I also copied the public key as `.key` in the directory.
+
+Now everything is ready for `synk`ronisation (over a VPN, in this case):
+
+ find /var/www/htdocs/dl.z3bra.org/pack -type f | synk -h apophis.2f30
+
+It can be automated using `wendy`, so that everytime the `.list` file
+is modified by `repogen`, everything is replicated on the remote repo:
+
+ wendy -m 8 -f $REPO/.list sh -c "find $REPO -type f | synk -h apophis.2f30"
+
+### Installing
+
+Now that we can create packs and upload them quickly to the repository,
+it's time to install them!
+
+Using the `.list` file, we can check what's available. With the `.key`
+file, we can ensure that no-one tampered with our pack during the
+retrieval process. Using `pm`, we can install and update our packs for
+daily use.
+
+All we need now, is a utility to fetch packs from the repo. I named this
+script "`repo`"!
+
+
+ #!/bin/sh
+
+ url="http://dl.z3bra.org/pack"
+ cache="$HOME/.cache/repo"
+ keyring="$HOME/.sick.d"
+
+ usage() {
+ echo "usage: $(basename $0) [-s] [-t reponame] [PKG..]" >&2
+ exit 1
+ }
+
+ reposync() {
+ mkdir -p ${cache}
+ curl -Ls ${url}/.list | tee ${cache}/.list
+ }
+
+ repocheck() {
+ sick -f $HOME/.sick.d/egull.pub
+ }
+
+ repotrust() {
+ curl -sL ${url}/.key > ${keyring}/${name}.pub
+ }
+
+ repoget() {
+ pkg="$1"
+ ver=$(grep -E "^${pkg} " ${cache}/.list | tac | sed 1q | cut -f2)
+
+ file="${pkg}#${ver}.tar.bz2"
+ html="${pkg}%23${ver}.tar.bz2"
+
+ curl -sL ${url}/${html} | repocheck | ifne sponge ${cache}/${file}
+ test -f ${cache}/${file} \
+ && readlink -f ${cache}/${file} \
+ || echo "$pkg: signature check failed" >&2
+ }
+
+ repolist() {
+ pg -e ${cache}/.list
+ }
+
+ test $# -eq 0 && { repolist && exit 0; }
+
+ case $1 in
+ -s) reposync; exit 0 ;;
+ -l) repolist; exit 0 ;;
+ -t) test -z "$2" && usage || { repotrust $2; exit 0; } ;;
+ esac
+
+ for n in $@; do
+ repoget $n
+ done
+
+ exit 0
+
+First, we retrieve the ed25519 public key from the repo:
+
+ $ repo -t z3bra
+
+Then, the package list:
+
+ $ repo -s
+ libwm 1.1
+ skroll 0.6
+
+And finally, install it!
+
+ $ ROOT=$HOME/.local
+ $ export ROOT
+ $ pm -a $(repo skroll)
+ $ pm -i
+ skroll 0.6
+ $ echo amazing! | skroll
+
+### Conclusion
+
+Ok, so this whole post was rather long, and not especially good at
+describing the actual workflow. So as usual. Here is a quick video to
+show off the whole process!
+
+<video controls>
+ <source src='/vid/20160916-skroll-install.webm' type='video/webm'>
+</video>
+<span class='caption'>Packagin, deploying and installing
+[`skroll`](http://z3bra.org/skroll) on my system</span>
+
DIR diff --git a/config.mk b/config.mk
t@@ -32,7 +32,8 @@ PAGES = index.html \
2015/08/install-alpine-at-onlinenet.html \
2016/01/make-your-own-distro.html \
2016/03/hand-crafted-containers.html \
- 2016/08/desktop-streaming.html
+ 2016/08/desktop-streaming.html \
+ 2016/09/pack-it-up.html
FEEDS = rss/feed.xml
EXTRA = css img vid data errors favicon.ico
DIR diff --git a/index.txt b/index.txt
t@@ -1,3 +1,4 @@
+* 0x001d - [Pack it up!](/2016/09/pack-it-up.html)
* 0x001c - [Desktop streaming](/2016/08/desktop-streaming.html)
* 0x001b - [Hand-crafted containers](/2016/03/hand-crafted-containers.html)
* 0x001a - [Make your own distro](/2016/01/make-your-own-distro.html)