## Shoot Footgun

Delivering Server Failure

by

Josuah Demangeon <me@josuah.net>


## How to mess up a server

 * Delete a random binary in /bin
   * "command not found"
   * scripts that randomly crash


## How to mess up a server really bad

 * Move off /lib
   * "command not found" even though the binaries are there
   * dynamic linking <3
#pause
        This is still not what I did...


## How to comletely destroy a server

#pause
 * Upload /bin from Linux to an OpenBSD system
#pause
   * "^?ELF^B^A^A: command not found"
#pause
   * Upload the whole / while we are at it


## Self doom recipe

	#!/bin/sh -e
	# export all local objects to a remote system
	host=$1
	exec rsync -Pa --size-only "$snap/" "$host:$snap/"


## Self doom recipe

	#!/bin/sh -e
	# export all local objects to a remote system
	host=${1:?}
	exec rsync -Pa --size-only "$snap/" "$host:$snap/"


## Self doom recipe

	#!/bin/sh -e
	# export all local objects to a remote system
	host=${1:?}
	snap=${SNAPSHOT:-/var/lib/snapshot
	exec rsync -Pa --size-only "$snap/" "$host:$snap/"


## How to comletely destroy a server

 * Joyfully uploading / to the remote system
 * OpenBSD/musl-libc/busybox/Linux


## How to comletely destroy a server forever

#pause
 * The backup script uploading / on all of my servers (2 of them)
 * Everything down: smtpd, gopherd, gitd, httpd, dnsd, irc client...


## EMERGENCY REBOOTSTRAP (1/3)

 * Custom packages
 * Custom service ./run scripts
 * No server to fetch them from (both are down now)
#pause
 * Must get it from previous system


## EMERGENCY REBOOTSTRAP (2/3)

 * Boot into install .iso then (U) update
 * Only OpenBSD 6.4 .iso available on VPS
 * OpenBSD 6.6/6.4/musl-libc/busybox/Linux
 * Download 


## EMERGENCY REBOOTSTRAP (3/3)

 * Book another VPS
 * Download packages
 * Run services


## Facts

 * Things can still go wrong when you backup your backups
 * No "password recovery" possible when DNS and SMTP servers all down
 * The most important data to backup: registrar admin panel password