tpack-it-up.txt - monochromatic - monochromatic blog: http://blog.z3bra.org
HTML git clone git://z3bra.org/monochromatic
DIR Log
DIR Files
DIR Refs
---
tpack-it-up.txt (5868B)
---
1 # Pack it up!
2
3 16 September, 2016
4
5 Today was a big day for me. It was the day all my software could play
6 together to reach a single goal:
7
8 *Maintaining and hosting software packages!*
9
10 Here are the tools:
11
12 * [pm](http://z3bra.org/pm) - manage local packs
13 * [sick](http://z3bra.org/sick) - sign and check files
14 * [synk](http://z3bra.org/synk) - synchronize files between hosts
15 * [wendy](http://z3bra.org/wendy) - run a command on filesystem change
16
17 Tied together with a fistful of scripts, and some easy preparation,
18 they help me deploy and manage my own utilities accross multiple operating
19 systems without efforts.
20
21 This process includes three main tasks: packaging, deploying and
22 installing. This whole process is still at and early stage, and will
23 get stronger and sharper with time (and shell script will most likely
24 turn into actual programs).
25 Here is an explanation of how it works:
26
27 0. pack an application and sign the tarball
28 1. upload the tarball to an online repository
29 2. get the latest tarball for a utility
30 3. install/update it locally
31
32 The process is straigh-forward, and multiple parts can be automated.
33
34 ## Packaging
35
36 In order to build my softwares, I need either `make` or
37 [`mk`](http://doc.cat-v.org/plan_9/4th_edition/papers/mk) (it's tending
38 to be `mk` only, but for now I still need both). Then process is then
39 fairly simple:
40
41 $ cd $utility
42 $ mk
43 # mk install
44
45 I wrote a quick script to "pack" these utilities for me. It will build
46 the software, install it to a temporary location, create a tarball out
47 of it, and sign this tarball with my private key:
48
49 #!/bin/sh
50
51 # user specific variables
52 SICKKEY=$HOME/.sick.d/$USER.key
53 REPO=/var/www/htdocs/dl.z3bra.org/pack
54
55 # guess the pack name from current directory, or use given name
56 DIR=$(basename `pwd`)
57 PKG=${1:-$DIR}
58
59 # set version from latest git tag, if applicable
60 TAG=$(git tag | sed -n '$p' | tr -d a-z)
61 VER=${TAG:-0.0}
62
63 # this part should die...
64 test -f mkfile && MK=mk || MK=make
65
66 # build pack and install to ./rootfs/usr
67 $MK
68 $MK DESTDIR=$(pwd)/rootfs PREFIX=/usr MANDIR=/usr/share/man install
69 (
70 cd rootfs
71 mkdir -p $REPO
72 # pack and sign the installed utility
73 tar cvj * | sick -s -f ${SICKKEY} > $REPO/${PKG}#${VER}.tar.bz2
74 )
75 rm -rf rootfs
76
77 # simply ensure that the file has been created correctly
78 echo
79 ls $REPO/${PKG}#${VER}.tar.bz2
80
81 At this point, to pack one of my utilities, all I need is:
82
83 $ git clone git://z3bra.org/skroll
84 $ cd skroll
85 $ pack
86
87 And I'm done :)
88
89 ## Deploying
90
91 This part require a bit of setup. My current repository is at
92 http://dl.z3bra.org/pack, which is, locally in
93 `/var/www/htdocs/dl.z3bra.org/pack`. My tool `synk` can get a file
94 synchronized between two peers, but they will have the same path,
95 which is why I also created this directory on my local machine.
96 I also need to upload my public key (for `sick` checks) and a list
97 of what's currently in the repo.
98
99 First, here is the `repogen` script, which will list the content of the
100 local repo, and write the pack names and version available to a file:
101
102 #!/bin/sh
103
104 REPO=/var/www/htdocs/dl.z3bra.org/pack
105 for tarball in $(find $REPO -name '*.tar.bz2'); do
106 pkg=$(basename $tarball | cut -d# -f1)
107 ver=$(basename $tarball | cut -d# -f2 | sed 's/.tar.bz2//')
108 printf '%s\t%s\n' "$pkg" "$ver"
109 done | sort | tee $REPO/.list
110
111 I also copied the public key as `.key` in the directory.
112
113 Now everything is ready for `synk`ronisation (over a VPN, in this case):
114
115 find /var/www/htdocs/dl.z3bra.org/pack -type f | synk -h apophis.2f30
116
117 It can be automated using `wendy`, so that everytime the `.list` file
118 is modified by `repogen`, everything is replicated on the remote repo:
119
120 wendy -m 8 -f $REPO/.list sh -c "find $REPO -type f | synk -h apophis.2f30"
121
122 ## Installing
123
124 Now that we can create packs and upload them quickly to the repository,
125 it's time to install them!
126
127 Using the `.list` file, we can check what's available. With the `.key`
128 file, we can ensure that no-one tampered with our pack during the
129 retrieval process. Using `pm`, we can install and update our packs for
130 daily use.
131
132 All we need now, is a utility to fetch packs from the repo. I named this
133 script "`repo`"!
134
135
136 #!/bin/sh
137
138 url="http://dl.z3bra.org/pack"
139 cache="$HOME/.cache/repo"
140 keyring="$HOME/.sick.d"
141
142 usage() {
143 echo "usage: $(basename $0) [-s] [-t reponame] [PKG..]" >&2
144 exit 1
145 }
146
147 reposync() {
148 mkdir -p ${cache}
149 curl -Ls ${url}/.list | tee ${cache}/.list
150 }
151
152 repocheck() {
153 sick -f $HOME/.sick.d/egull.pub
154 }
155
156 repotrust() {
157 curl -sL ${url}/.key > ${keyring}/${name}.pub
158 }
159
160 repoget() {
161 pkg="$1"
162 ver=$(grep -E "^${pkg} " ${cache}/.list | tac | sed 1q | cut -f2)
163
164 file="${pkg}#${ver}.tar.bz2"
165 html="${pkg}%23${ver}.tar.bz2"
166
167 curl -sL ${url}/${html} | repocheck | ifne sponge ${cache}/${file}
168 test -f ${cache}/${file} \
169 && readlink -f ${cache}/${file} \
170 || echo "$pkg: signature check failed" >&2
171 }
172
173 repolist() {
174 pg -e ${cache}/.list
175 }
176
177 test $# -eq 0 && { repolist && exit 0; }
178
179 case $1 in
180 -s) reposync; exit 0 ;;
181 -l) repolist; exit 0 ;;
182 -t) test -z "$2" && usage || { repotrust $2; exit 0; } ;;
183 esac
184
185 for n in $@; do
186 repoget $n
187 done
188
189 exit 0
190
191 First, we retrieve the ed25519 public key from the repo:
192
193 $ repo -t z3bra
194
195 Then, the package list:
196
197 $ repo -s
198 libwm 1.1
199 skroll 0.6
200
201 And finally, install it!
202
203 $ ROOT=$HOME/.local
204 $ export ROOT
205 $ pm -a $(repo skroll)
206 $ pm -i
207 skroll 0.6
208 $ echo amazing! | skroll
209
210 ## Conclusion
211
212 Ok, so this whole post was rather long, and not especially good at
213 describing the actual workflow. So as usual. Here is a quick video to
214 show off the whole process!
215
216 <video controls>
217 <source src="http://pub.z3bra.org/monochromatic/vid/20160916-skroll-install.webm" type="video/webm">
218 </video>
219 *Packaging, deploying and installing
220 [`skroll`](http://z3bra.org/skroll) on my system*