------------------------------------------------------------------------ Shredder Concept to Quest ------------------------------------------------------------------------ Date: 2026-05-20 Author: paulr ------------------------------------------------------------------------ Shredder Concept to Quest Shredder: Concept to Quest Laboratory Report ? Automated POSIX Markdown Publishing Pipeline Paul R. ? SDF.org ? paulr.sdf.org ? 2026-05-19 Public Domain ? MIT License --- Abstract This report documents the design, architecture, and implementation of shredder.sh, a POSIX-compliant shell script that automates the publication of Markdown documents to HTML, Gopher text, and PDF formats on SDF.org BSD servers. The script applies a wizard pattern ? prompting for missing information through structured shell-function interactions rather than failing silently ? and frames the entire pipeline as a Teenage Mutant Ninja Turtles quest narrated by Master Splinter via cowsay(1) speech bubbles. Each processing stage is assigned a named character; the main orchestrator function is called SHREDDER. The system requires no external runtimes, no package managers, and no configuration files beyond a single variable block at the top of the script. It targets the NetBSD-based ARPA shell environment at SDF.org. This document is itself the inaugural post of the Shredder Quest on paulr.sdf.org, and will be published by the tool it describes. --- I. Introduction Public-access Unix systems such as SDF.org provide shell accounts, Gopher hosting, and web hosting with nothing but a terminal and an SSH client. Publishing to all three formats from a single Markdown source file is a three-step process that is easy to forget and tedious to repeat. The goal of shredder.sh was to reduce that to one command ? or zero arguments, if the document does not yet exist. A secondary goal emerged during design: the tool should not be silent. Traditional shell pipelines communicate through exit codes and sparse log lines. This script takes the opposite approach: every stage is announced by its assigned character, and the transitions between stages are narrated by Master Splinter, rendered as cowsay(1) speech bubbles with a custom ASCII rat cow file generated at runtime. The tertiary goal ? which was not planned ? was that the whole thing should be fun to run. This report documents the design decisions, function architecture, control-flow branches, utility dependencies, and shell-portability rationale. It is simultaneously a technical record and, upon being published by the tool it describes, the first entry in the Shredder Quest on paulr.sdf.org. This mission is yours, if you choose to accept it. --- II. Equipment and Operating Environment 2.1 Host System | Item | Detail | | Service | SDF.org Public Access Unix | | Account tier | ARPA (ARPA-level shell access) | | Operating system | NetBSD (primary target); any POSIX BSD Unix | | Shell | /bin/sh ? POSIX sh; not bash, not zsh | | Access method | SSH to tty.sdf.org | | Gopher hosting | gopher.sdf.org/users/$USER | | Web hosting | sdf.org/~$USER or $USER.sdf.org | SDF.org is a public-access Unix system founded in 1987, operating continuously since. MetaARPA accounts include Gopher server space, personal web hosting, email, and access to the full NetBSD base system. The arpa designation is a reference to ARPANET, the precursor to the modern internet, and to the academic culture of public shell accounts that predates the web. 2.2 Required Utilities shredder.sh depends exclusively on tools present in the NetBSD base system or otherwise standard on POSIX platforms. No package manager invocation is required. | Utility | Role in Pipeline | | sh | POSIX shell interpreter; set -eu throughout | | awk | Block-level Markdown?HTML and Markdown?groff-ms state machines | | sed | Inline Markdown?HTML substitutions; Gopher text stripping | | grep | Structural validation ? heading counts and fence balance | | spell | BSD spell(1) for interactive spell checking | | groff | PDF typesetting via the -ms macro package | | ps2pdf | PostScript-to-PDF conversion via Ghostscript | | fold | Line-wrapping Gopher text to 72 columns | | mktemp | Isolated temp workspace; cleaned by trap on EXIT | | ed | Line editor for interactive document creation in no-arg mode | | cowsay | Splinter's speech bubbles; optional, graceful fallback | | printf | POSIX-portable output; echo is not used | | date | ISO 8601 datestamps for derived output filenames | | chmod | File permission verification in the final stage | | mkdir | Output directory creation | --- III. Design Philosophy 3.1 The Wizard Pattern Command-line tools fail in two modes: silently (producing wrong output) or noisily (printing a usage string and exiting with status 1). Neither is satisfying when the tool is used infrequently by one person. A third mode exists: the wizard. The wizard detects what information is missing and asks for it before proceeding. GUI installers use this pattern. Terminal tools rarely do. shredder.sh applies the wizard pattern at three points in the pipeline: 1. No input file: rather than print usage and exit, the script prompts for a title, derives a filename and slug, seeds the file with a # Title heading, displays a Markdown syntax reference and an ed(1) quick-reference card, and drops the user into ed for composition. Control returns to the pipeline when ed exits. 2. Spell check gate: if spell(1) finds misspellings, the script displays them and prompts [y/N] before continuing. This is Raphael's contribution. He disagrees with everything. 3. Gophermap subtitle: before inserting the gophermap entry, the script offers the author the opportunity to supply an extended display title. If the subtitle was already provided in the no-arg wizard flow, this prompt is skipped. The wizard is implemented entirely in shell functions. No external expect, no Python, no curses. Each prompt uses printf and read -r. The functions share state through script-level variables ? $SUBTITLE being the primary example of a variable set in one branch and consumed in a later stage. 3.2 The Narrative Framework The mapping of pipeline stages to Teenage Mutant Ninja Turtles characters is structural, not merely cosmetic. The franchise centres on a group of specialists directed by a wise sensei toward a common goal, repeatedly encountering named antagonists along the way before facing the main villain. The pipeline has exactly this shape. | Character | Stage | Task | | Splinter | Narrator | Introduces each stage; guides the user | | April O'Neil | Logging (APRIL_*) | Chronicles every step with ==> lines | | Splinter | Stage 1 | Structural validation ? a master checks personally | | Raphael | Stage 2 | Spell check ? the sceptic who questions every word | | Leonardo | Stage 3 | HTML assembly ? disciplined structure | | Donatello | Stages 3 & 5 | AWK converters ? the technologist behind the scenes | | Michelangelo | Stage 4 | Gopher ASCII ? strips all formality into plain text | | Krang | Stage 5 | PDF via groff ? the disembodied brain applies typesetting | | Bebop | Stage 6 | Gophermap update ? chaotic energy, correctly directed | | Rocksteady | Stage 7 | Sitemap regeneration ? methodical, one entry at a time | | Baxter Stockman | Stage 8 | Permission verification ? compound eyes miss nothing | | Shredder | Orchestrator | The villain who makes everything happen | Splinter is rendered as a 12-line ASCII rat in a custom cowsay(1) cow file. The cow file is generated at runtime into the script's temp directory ($TMP/rat.cow). No external file is required; the art is embedded as a shell heredoc with backslashes doubled for Perl heredoc interpolation rules. --- IV. Architectural Overview 4.1 Single-File Design shredder.sh is a single POSIX sh file. All logic ? validation, conversion, PDF generation, gophermap management, sitemap regeneration, permission checking, narration ? lives in one file. There are no sourced libraries, no helper scripts, no configuration files. Site-specific parameters (hostnames, paths, URLs) are declared as a single variable block near the top and are the only values a new user needs to edit before first use. 4.2 Preconditions The following conditions must hold before the pipeline will run to completion: 1. The input file exists and contains at least one # Title line. 2. The HTML template exists at ~/html/template.html (overrideable with the -t flag). 3. If ~/gopher/gophermap exists, it contains a line reading exactly #top. If the file does not exist, the script creates a skeleton. 4. All utilities listed in $2.2 are present on $PATH. 5. The user has write access to ~/html/ and ~/gopher/. The script validates preconditions 1 and 2 explicitly before any stage runs. Precondition 3 is checked and remedied inside BEBOP. Preconditions 4 and 5 are assumed ? the script targets a known environment. 4.3 Temporary Workspace A single temp directory is created by mktemp -d /tmp/shredder.XXXXXX early in the script ? before argument parsing ? so that the rat.cow file is available for the first Splinter narration. A trap on EXIT, INT, and TERM removes the entire directory on script termination. All intermediate files (HTML body fragments, groff source, gophermap scratch files) are written into this directory and cleaned up automatically. 4.4 Output Artefacts Given an input file with heading # My Post run on 2026-05-19: | Artefact | Path | | HTML page | ~/html/2026-05-19-My-Post.html | | Gopher text | ~/gopher/2026-05-19-My-Post.txt | | PDF (Gopher) | ~/gopher/2026-05-19-My-Post.pdf | | PDF (web) | ~/html/2026-05-19-My-Post.pdf | | Gophermap | ~/gopher/gophermap (updated in place) | | Sitemap | ~/html/index.html (fully regenerated) | --- V. The Pipeline: Functions in Sequence The main controller function SHREDDER() calls eight stage functions in fixed order. Each must complete before the next begins. SHREDDER() ? ?? 1. SPLINTER() structural validation ?? 2. RAPHAEL() interactive spell check ?? 3. LEONARDO() HTML assembly ? ?? DONATELLO_html_body() awk block-level converter ? ?? DONATELLO_inline_sed() sed inline converter ?? 4. MICHELANGELO() Gopher ASCII text ?? 5. KRANG() PDF via groff -ms ? ?? DONATELLO_ms_body() awk groff-ms converter ?? 6. BEBOP() gophermap entry insertion ?? 7. ROCKSTEADY() sitemap regeneration ?? 8. BAXTER_STOCKMAN() file permission verification Stage 1 ? SPLINTER: Structural Validation Counts # heading lines with grep -c. Counts ` code-fence lines; verifies the count is even (balanced open and close fences). Calls APRIL_die on failure, which prints to stderr and exits with status 1. Stage 2 ? RAPHAEL: Spell Check Strips Markdown syntax with a sed pass into a temp file, then pipes through BSD spell(1). If misspellings are found, they are displayed and the user is prompted to continue (y) or abort (N). This is the first interactive branch point and the only stage that can abort the pipeline with user consent. Stage 3 ? LEONARDO: HTML Assembly Calls DONATELLO_html_body (awk) then DONATELLO_inline_sed (sed) to produce a body fragment. Extracts
paulr.sdf.org [COURSE NUMBER] — IEEE Laboratory Report LABREPORT(7)
TMPL_EOF APRIL_note "Template created: $TEMPLATE" fi ?? Derive output names from date + first # heading ?????????????????????????? DATE=$(date '+%Y-%m-%d') YEAR=$(date '+%Y') TITLE_RAW=$(sed -n 's/^# //p' "$INPUT" | head -1) if [ -z "$TITLE_RAW" ]; then # No # heading found ? derive a title from the filename and prepend it. # This lets short posts (even a single paragraph) flow through unchanged. _fn="${INPUT##*/}"; _fn="${_fn%.md}" case "$_fn" in [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-*) _fn="${_fn#????-??-??-}" ;; esac TITLE_RAW=$(printf '%s' "$_fn" | sed 's/[-_]/ /g') APRIL_warn "no '# Title' heading ? using filename: '$TITLE_RAW'" { printf '# %s\n\n' "$TITLE_RAW"; cat "$INPUT"; } > "$TMP/titled.md" INPUT="$TMP/titled.md" fi SLUG=$(printf '%s' "$TITLE_RAW" \ | sed 's/[^A-Za-z0-9 ]//g; s/ */ /g; s/ /-/g' \ | tr '[:upper:]' '[:lower:]') BASE="${DATE}-${SLUG}" if [ -n "$CATEGORY" ]; then HTML_OUT="${HOME}/html/${CATEGORY}/${BASE}.html" PDF_HTML="${HOME}/html/${CATEGORY}/${BASE}.pdf" TXT_OUT="${HOME}/gopher/${CATEGORY}/${BASE}.txt" PDF_GOPHER="${HOME}/gopher/${CATEGORY}/${BASE}.pdf" mkdir -p "${HOME}/html/${CATEGORY}" "${HOME}/gopher/${CATEGORY}" else HTML_OUT="${HOME}/html/${BASE}.html" PDF_HTML="${HOME}/html/${BASE}.pdf" TXT_OUT="${HOME}/gopher/${BASE}.txt" PDF_GOPHER="${HOME}/gopher/${BASE}.pdf" mkdir -p "${HOME}/html" "${HOME}/gopher" fi SPLINTER_narrate "A markdown file has entered my dojo. Title: '$TITLE_RAW'. I have seen many such files. Most were... adequate. Let us see what becomes of this one." APRIL_say "SHREDDER activated." APRIL_note "Input : $INPUT" APRIL_note "Title : $TITLE_RAW" APRIL_note "Base : $BASE" APRIL_note "Category: ${CATEGORY:-uncategorized}" echo "" ==================================================================== SPLINTER ? Stage 1: Structural Validation "A ninja's greatest tool is patience... and awk." ==================================================================== SPLINTER() { APRIL_say "SPLINTER: Validate" # awk avoids the grep -c exit-code trap: grep exits 1 on zero matches, # so "grep -c ... || echo 0" can produce "0\n0" instead of "0". H1=$(awk '/^# /{n++} END{print n+0}' "$INPUT") [ "$H1" -ge 1 ] || APRIL_die "no '# Title' heading found" APRIL_note "$H1 heading(s) found" FENCES=$(awk '/^`/{n++} END{print n+0}' "$INPUT") if [ $(( FENCES % 2 )) -ne 0 ]; then echo "" APRIL_warn "odd number of code fences ($FENCES) ? one may be unclosed" echo " Fence locations:" awk '/^`/{printf " line %d: %s\n", NR, $0}' "$INPUT" echo "" printf " Continue anyway? [y/N] " read -r _ANS case "$_ANS" in [Yy]*) APRIL_note "Continuing past fence warning." ;; *) APRIL_die "Aborted. Fix the unclosed fence and re-run." ;; esac else APRIL_note "fences balanced ($FENCES)" fi } ==================================================================== RAPHAEL ? Stage 2: Interactive Spell Check "Maybe I like being wrong. ...No I don't. Fix it." ==================================================================== RAPHAEL() { APRIL_say "RAPHAEL: Spell check" STRIPPED="$TMP/stripped.txt" # Strip markdown syntax before handing off to spell(1). # Code blocks are removed entirely; inline markers are erased. sed \ -e '/^\\\/,/^\\\/d' \ -e 's/^#\{1,6\} *//g' \ -e 's/\\[^]\\//g' \ -e 's/\[^]\//g' \ -e 's/[^]*`//g' \ -e 's/\[[^]]\]([^)])//g' \ -e 's/^\|.*//g' \ -e 's/^> //' \ "$INPUT" > "$STRIPPED" # spell(1) may exit non-zero when it finds words; capture safely MISSPELLED=$(spell < "$STRIPPED" 2>/dev/null) || true if [ -z "$MISSPELLED" ]; then APRIL_note "OK ? no misspellings found" return 0 fi echo "" echo " Possible misspellings:" printf '%s\n' "$MISSPELLED" | sed 's/^/ /' echo "" printf " Continue anyway? [y/N] " read -r ANS case "$ANS" in [Yy]*) APRIL_note "Continuing. (Raphael disagrees.)" ;; *) APRIL_die "Aborted. Correct spelling and re-run." ;; esac } ==================================================================== DONATELLO ? AWK Block-Level Converters "I have a machine for this. Obviously." Two functions: html_body and ms_body. Both use the same state-machine pattern: escape first, detect structure second. A third function handles inline sed substitutions. ==================================================================== HTML block structure. Lines are HTML-escaped (&, <, >) before any structural check, so content and tag output cannot interfere. Paragraphs accumulate in 'para' and are flushed on blank lines or block-level elements. DONATELLO_html_body() { awk ' BEGIN { in_code=0; in_table=0; para="" } function flush() { if (para != "") { print "

" para "

"; para="" } } function escape() { gsub(/&/, "\\&") gsub(//, "\\>") } /^`/ { if (!in_code) { flush(); print "
"; in_code=1 }
        else          { print "
"; in_code=0 } next } in_code { escape(); print; next } /^\|/ { if (/^\|[-: |]+\|[[:space:]]*$/) next # skip separator rows if (!in_table) { flush(); print "
"; in_table=1 }
        escape(); print; next
    }
    in_table && !/^\|/ { print "
"; in_table=0 } /^#### / { flush(); escape(); sub(/^#### /,""); print "

"$0"

"; next } /^### / { flush(); escape(); sub(/^### /,""); print "

"$0"

"; next } /^## / { flush(); escape(); sub(/^## /,""); print "

"$0"

"; next } /^# / { flush(); escape(); sub(/^# /,""); print "

"$0"

"; next } /^---[[:space:]]*$/ { flush(); print "
"; next } /^> / { flush(); sub(/^> /,""); escape() print "

"$0"

"; next } /^[-*] / { flush(); sub(/^[-*] /,""); escape() print "
  • "$0"
    • "; next } /^[[:space:]]*$/ { flush(); next } { escape(); if (para!="") para=para" "; para=para$0 } END { flush() if (in_table) print "" if (in_code) print "" } ' "$1" } Inline markdown ? HTML. Runs after DONATELLO_html_body, so content is already HTML-escaped; only markdown punctuation remains. DONATELLO_inline_sed() { sed \ -e 's/\([^]*\)`/\1<\/code>/g' \ -e 's/\\\([^]\)\\/\1<\/strong>/g' \ -e 's/\\([^]\)\/\1<\/em>/g' \ -e 's/\[\([^]]\)\](\([^)]\))/\1<\/a>/g' } groff -ms block structure. Mirrors DONATELLO_html_body but emits ms(7) macros. Inline markup is applied by the ms_inline() awk function using match()/RSTART/RLENGTH loops. DONATELLO_ms_body() { awk ' BEGIN { in_code=0; in_table=0; para="" } function flush() { if (para!="") { print ".PP"; print para; para="" } } function ms_inline(s, r,pre,mid,post,bra) { r=s while(match(r,/\\[^]+\\*/)) { pre=substr(r,1,RSTART-1); mid=substr(r,RSTART+2,RLENGTH-4) post=substr(r,RSTART+RLENGTH); r=pre"\\fB"mid"\\fR"post } while(match(r,/\[^]+\*/)) { pre=substr(r,1,RSTART-1); mid=substr(r,RSTART+1,RLENGTH-2) post=substr(r,RSTART+RLENGTH); r=pre"\\fI"mid"\\fR"post } while(match(r,/\[[^]]+\]\([^)]+\)/)) { pre=substr(r,1,RSTART-1) bra=index(substr(r,RSTART),"]"); mid=substr(r,RSTART+1,bra-2) post=substr(r,RSTART+RLENGTH); r=pre mid post } while(match(r,/[^]+`/)) { pre=substr(r,1,RSTART-1); mid=substr(r,RSTART+1,RLENGTH-2) post=substr(r,RSTART+RLENGTH); r=pre"\\f(CW"mid"\\fR"post } return r } /^`/ { if(!in_code){flush();print".DS L\n.ft CR";in_code=1} else{print".ft\n.DE";in_code=0}; next } in_code { print; next } /^\|/ { if(/^\|[-: |]+\|[[:space:]]*$/) next if(!in_table){flush();print".DS L\n.ft CR";in_table=1} print; next } in_table&&!/^\|/ { print".ft\n.DE"; in_table=0 } /^#### /{flush();sub(/^#### /,"");print".IP \""ms_inline($0)"\" 4";next} /^### / {flush();sub(/^### /,""); print".SS "ms_inline($0);next} /^## / {flush();sub(/^## /,""); print".SH "ms_inline($0);next} /^# / {flush();sub(/^# /,""); print".SH "ms_inline($0);next} /^---[[:space:]]*$/ { flush(); print".br\n.sp 0.5v"; next } /^> / { flush(); sub(/^> /,"") print".RS\n.I\n"ms_inline($0)"\n.RE"; next } /^[-*] / { flush(); sub(/^[-*] /,"") print".IP \\(bu 2\n"ms_inline($0); next } /^[[:space:]]*$/ { flush(); next } { line=ms_inline($0); if(para!="")para=para" "; para=para line } END { flush() if(in_code) print".ft\n.DE" if(in_table) print".ft\n.DE" } ' "$1" } ?? Shared filesystem scanner (BEBOP + ROCKSTEADY) ??????????????????????????? Emits TSV: CAT\tDATE\tTITLE\tBASE\tHAS_HTML\tHAS_PDF Title is line 2 of the .txt file (MICHELANGELO's fixed format). PDF is considered present if it exists in either ~/html/ or the gopher dir. _scan_gopher_dir() { _sg_dir="$1" _sg_cat="$2" for _f in "$_sg_dir"/[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-*.txt; do [ -f "$_f" ] || continue _base="${_f##*/}"; _base="${_base%.txt}" _date=$(printf '%s' "$_base" | cut -c1-10) _title=$(sed -n '2p' "$_f" 2>/dev/null) || true [ -n "$_title" ] || _title="$_base" if [ -n "$_sg_cat" ]; then _html_dir="${HOME}/html/${_sg_cat}" else _html_dir="${HOME}/html" fi _hh=0; [ -f "${_html_dir}/${_base}.html" ] && _hh=1 _hp=0 { [ -f "${_html_dir}/${_base}.pdf" ] || \ [ -f "${_sg_dir}/${_base}.pdf" ]; } && _hp=1 printf '%s\t%s\t%s\t%s\t%s\t%s\n' \ "$_sg_cat" "$_date" "$_title" "$_base" "$_hh" "$_hp" done } ?? Returns 0 (true) if a category dir contains at least one dated post ?????? _cat_has_posts() { for _chp in "$1"/[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-*.txt; do [ -f "$_chp" ] && return 0 done return 1 } ?? BUILD_NAV: generate nav bar from ~/gopher/ category subfolders ??????????? BUILD_NAV() { printf '\n' } ==================================================================== LEONARDO ? Stage 3: HTML Assembly + Template Injection "We work as a team. I assemble the final output." ==================================================================== LEONARDO() { APRIL_say "LEONARDO: Generate HTML" BODY="$TMP/html_body.html" DONATELLO_html_body "$INPUT" | DONATELLO_inline_sed > "$BODY" # Extract reusable blocks from the template with awk range patterns, # then patch the placeholder strings with sed. T_STYLE=$(awk '/