Add mqtt hackathon - brcon2024-hackathons - Bitreichcon 2024 Hackathons
HTML git clone git://bitreich.org/brcon2024-hackathons git://enlrupgkhuxnvlhsf6lc3fziv5h2hhfrinws65d7roiv6bfj7d652fid.onion/brcon2024-hackathons
DIR Log
DIR Files
DIR Refs
DIR Tags
DIR Submodules
---
DIR commit 7ac09da5aedd5b5e3fbbdcc57adadb263f09d011
DIR parent 657b359aba0a48bc973a0cffe152a941297f6543
HTML Author: Christoph Lohmann <20h@r-36.net>
Date: Thu, 8 Aug 2024 23:01:46 +0200
Add mqtt hackathon
Diffstat:
A mqtt/README.md | 8 ++++++++
A mqtt/brcon2024_adc.md | 483 +++++++++++++++++++++++++++++++
A mqtt/mqtt-wrapper | 59 +++++++++++++++++++++++++++++++
A mqtt/tamagotchi | 1 +
4 files changed, 551 insertions(+), 0 deletions(-)
---
DIR diff --git a/mqtt/README.md b/mqtt/README.md
@@ -0,0 +1,8 @@
+# MQTT
+
+## Following the talk of Thursday
+
+brcon2024_adc.md has the challenges
+tamagotchi/ for the tamagotchi source
+mqtt-wrapper for running some command
+
DIR diff --git a/mqtt/brcon2024_adc.md b/mqtt/brcon2024_adc.md
@@ -0,0 +1,483 @@
+## brcon2024 - 2024-08-08
+
+ title: Deep C programming - UNIX philosophy in distributed offshore systems
+
+ author: Anders Damsgaard (adc)
+
+ contact: anders@adamsgaard.dk
+ gopher://adamsgaard.dk
+ https://adamsgaard.dk
+
+ slides: gopher://adamsgaard.dk/tmp/brcon2024_adc.md
+
+## brcon2024 - 2024-08-08
+
+ title: Deep C programming - UNIX philosophy in distributed offshore systems
+
+ author: Anders Damsgaard (adc)
+
+ contact: anders@adamsgaard.dk
+ gopher://adamsgaard.dk
+ https://adamsgaard.dk
+ <marquee><blink>
+ slides: gopher://adamsgaard.dk/tmp/brcon2024_adc.md
+ </blink></marquee>
+
+
+## Previously..
+* brcon2020: Energy efficient programming in science
+* brcon2021: Unix principles for science simulations
+* brcon2022: Est. Bitreich Arctic Vault
+* brcon2023: Using Unix and Bitreich tools for preparing scientific papers
+* Today: UNIX philosophy in distributed offshore systems
+#pause
+* Tomorrow: UNOX philosophy in Vitré
+
+## Outline
+* How we can benefit from hyped technology with true and tried principles
+* Overview of the offshore system
+* The talk transitions into hands-on exercises
+* Ask questions along the way in #bitreich-con
+
+## Danish Geotechnical Institute (geo.dk)
+* ~250 employees in Lyngby and Aarhus, Denmark
+* Geotechnical investigations (onshore/offshore)
+* Environmental consultancy
+* Laboratory tests
+* Geodata analysis
+* Geoscientific software development
+
+#pause
+-> New offshore rig built from the ground up
+
+## Offshore geotechnical investigations
+ .* +------------------+
+ \|\ | ,_____. |
+ _============. | | | |
+ \\# # # #/ | | ._______|--|--| |
+ | +.,| | Geo .--|--| |
+ _---------------------|-+-------|--|--|--|\
+ \ MS/Bitreich | +--|--+ | |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~~~~|~|~~~~~~~~~~~~
+ \ +------------------+'' ~
+ \_________________________________/--+ ~ . ~~
+ ~
+
+
+ ><>
+
+ <><
+
+
+
+----------------------------------------------------------------------
+
+
+## Offshore geotechnical investigations
+ .*
+ \|\ ,_____.
+ _============. | |
+ \\# # # #/ | ._______|--|--|
+ | +., | Geo .--|--|
+ _---------------------+-+-------|--|--|---\
+ \ MS/Bitreich +--|--+ |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~
+ \ .-'' ~
+ \_________________________________/--+ ~ . ~~
+ ~
+
+
+ ><>
+
+ <><
+
+
+
+----------------------------------------------------------------------
+
+
+
+## Offshore geotechnical investigations
+ .*
+ \|\ ,_____.
+ _============. | |
+ \\# # # #/ | ._______|--|--|
+ | +., | Geo | | |
+ _---------------------+-+-------|--|--|---\
+ \ MS/Bitreich | |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
+ \ | .-'' ~
+ \_____________________________|___/--x~ ~ . ~~
+ .--|--. ~
+ | v |
+ +-----+
+ ><>
+
+ <><
+
+
+
+----------------------------------------------------------------------
+
+
+
+
+## Offshore geotechnical investigations
+ .*
+ \|\ ,_____.
+ _============. | |
+ \\# # # #/ | ._______|--|--|
+ | +., | Geo | | |
+ _---------------------+-+-------|--|--|---\
+ \ MS/Bitreich | |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
+ \ | .-'~ ~ . ~~
+ \_____________________________|___/--x ~
+ |
+ |
+ |
+ ><> |
+ |
+ <>< |
+ |
+ .--|--.
+ | v |
+----------------------------------------------+-----+-----------------
+
+
+
+
+## Offshore geotechnical investigations
+ .*
+ \|\ ,_____.
+ _============. | |
+ \\# # # #/ | ._______|--|--|
+ | +., | Geo | | |
+ _---------------------+-+-------|--|--|---\
+ \ MS/Bitreich | |
+~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
+ \ | .-'' ~
+ \_____________________________|___/--x
+ |
+ |
+ |
+ ><> |
+ |
+ <>< |
+ |
+ .--|--.
+ | | |
+----------------------------------------------+--|--+-----------------
+ |
+ |
+ |
+ v
+
+## Offshore geotechnical investigations
+ | |
+ | |
+ | |
+ | |
+ | |
+ | |
+ | |
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~|~~~~~~~~~~~~
+ | |
+ | |
+ | |
+ | |
+ ? | |
+ ><> | |
+ | |
+ | | !
+ | | <><
+ | |
+ | |
+-----------------------------------------|---------------|------------
+ | |
+ | |
+ | |
+ | |
+ |_______________|
+
+https://www.mdpi.com/energies/energies-17-01964/article_deploy/html/images/energies-17-01964-g001.png
+
+## Offshore systems overview
+* Up to 800 sensors, usually 1-10 Hz
+* SCADA, PLCs, environmental sensors, navigation, scientific results,
+ calibration data, ...
+* Control messages
+* Asynchronous data streams
+* Event driven logic
+* Raw sensor output into physical units
+* Aggregation of time series and combination of sources
+* Time series databases
+* Development project with many involved partners
+
+## Approaches: Unix philosophy vs. Microsoft mentality
+* Does the client know what they want?
+* Do we know everything about what we want to solve?
+* Do we know if our assumptions are valid, in particular in complex systems?
+* Do we have full specifications?
+* Should we produce a big integrated C# monolith to handle computations,
+ flow, storage, logic?
+
+#pause
+ > yes no | head -n 5
+
+
+## Data routing system
+* MQTT: Message Queuing Telemetry Transport
+* Started as a proprietary protocol for SCADA systems in O&G
+* Lightweight, publish-subscribe network protocol
+* Constrained devices and low-bandwidth, high-latency networks
+* Publish/Subscribe Model: Decouples message producers from consumers
+* Data accessible via network, not living internally in an obscure program
+
+Specifications:
+- MQTT v5.0: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html
+- MQTT v3.1.1: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html
+- https://github.com/mqtt/mqtt.org/wiki/Differences-between-3.1.1-and-5.0
+- MQTT and TLS: https://www.rfc-editor.org/rfc/rfc9431.txt
+
+## Central concepts
+* Broker: Central server that routes messages
+* Clients: Devices that publish or subscribe to one or more topics
+* Messages: Data packets sent between clients via the broker
+* Topics: Hierarchical and dynamic name space for message
+
+Libs for many languages, e.g.,
+ - https://github.com/eclipse/paho.mqtt.c
+ - https://github.com/eclipse/paho.mqtt.golang
+
+## DISCLAIMER
+
+The coming exercises mainly to demonstrate fundamentals and principles
+
+Most are suboptimal or bad practice
+
+But, let's have some #fun ...
+
+## MQTT use cases
+* Home thermometers and weather stations (if you ask the web)
+* IoT Devices: Smart homes, wearables, and industrial automation
+* Mobile Applications: Real-time messaging and notifications
+* Remote Monitoring: Environmental sensors, health monitoring
+* Connected Vehicles: Telematics and fleet management
+
+## MQTT messages
+* TCP/IP
+* Limit to message size configurable
+
+* New subscribers receive only new messages (no backlog)
+* Retained Messages: Last known good value for new subscribers
+* Last Will and Testament (LWT): Notifies other clients about an ungraceful disconnection
+
+* Quality of Service (QoS): Three levels to ensure message delivery
+* QoS 0: At most once delivery (fire and forget)
+* QoS 1: At least once delivery (acknowledged delivery)
+* QoS 2: Exactly once delivery (assured delivery)
+
+## Why MQTT for the offshore project?
+* Hierarchial data structure through topics
+* A node in the topic tree exists as soon as someone publishes or
+ subscribes to it
+* Data accessible via network and all clients can read or publish
+ (by default)
+
+#pause
+ rig/sensor1/raw
+ rig/sensor1/phys
+ rig/sensor2/raw
+ rig/sensor2/phys
+ ...
+ ship/navigation/easting
+ ship/navigation/northing
+ ...
+
+## Topic examples
+Wildcards: #, + (subscribe only)
+
+Subscription examples:
+ 1) rig/sensor1/raw
+ 2) ship/#: Subscribe to everything from the ship
+ 3) rig/+/phys: Subscribe to all physical values
+
+## Example: Subscribe and publish
+$ cat bin/timestamp # in case you don't have ts(1)
+#!/bin/sh
+while read -r REPLY
+do
+ printf '%s %s\n' "$(date '+%Y-%m-%d %Z %H:%M:%S')" "$REPLY"
+done
+#pause
+$ export MQTTURI='mqtt://bitreich:bitreich@mx2.adamsgaard.dk:65431/'
+#pause
+$ mosquitto_sub -L "${MQTTURI}#" -v -t '#' | timestamp | grep -v hatesensor
+#pause
+
+$ date | mosquitto_pub -L "${MQTTURI}$(hostname)/$(whoami)" -l
+
+## Reporting online status via last will and retain, reporting some metrics
+
+gophers://duckless.org/0/tmp/mqtt-online
+
+#!/bin/sh
+online_topic="${ONLINE_TOPIC:-$(hostname)/online}"
+mqtt_broker="${MQTT_BROKER:-mx2.adamsgaard.dk}"
+mqtt_port="${MQTT_PORT:-65431}"
+mqtt_user="${MQTT_USER:-bitreich}"
+mqtt_password="${MQTT_PASSWORD:-bitreich}"
+args="-h ${mqtt_broker} -p ${mqtt_port} -u ${mqtt_user} -P ${mqtt_password}"
+mosquitto_pub ${args} -t "${online_topic}" -r -m "1"
+while :
+do
+ printf '%s\t%s\t%s\t%s\n' "$(date +%s)" "$(date)" "$(hostname)" "$(uptime)"
+ sleep 360
+done | mosquitto_pub ${args} --will-topic "${online_topic}" \
+ --will-retain --will-payload "0" -t "${online_topic}" -r -l
+
+## Data structure on the network
+Interact with data hierarchy via small modular clients.
+Brings all benefits of UNIX design into play.
+
+The tenents of the UNIX philosophy:
+
+ - Small is beautiful
+ - Make each program do one thing well
+ - Build a prototype as soon as possible
+ - Choose portability over efficiency
+ - Store numerical data in flat ASCII files
+ - Use software leverage to your advantage
+ - Use shell scripts to increase leverage and portability
+ - Avoid captive user interfaces
+ - Make every program a filter
+
+src: Make Gancarz 1995 "The UNIX Philosophy", ISBN 1-55558-123-4
+
+## Distributed computing
+Assign small modular programs to read stdin from topics and post their
+result to another topic
+
+* Rudimentary distributed computing
+* Each program acts as a text filter
+* Each program can contribute local data
+
+(disclaimer: for 1:1 connections, netcat or named pipes over SSH is simpler)
+
+Broker statistics: $SYS/broker
+ mosquitto_sub -L "${MQTTURI}\$SYS/broker/#" -v
+
+## Connecting stdin, stdout, and stderr with MQTT topics
+Full script: gophers://duckless.org/0/tmp/mqtt-wrapper
+
+ fifodir="$(mktemp -d)" || die 'mktemp'
+ fin="${fifodir}/in"
+ fout="${fifodir}/out"
+ ferr="${fifodir}/err"
+ mkfifo "$fin" "$fout" "$ferr" || die 'mkfifo'
+
+#pause
+ mosquitto_sub ${mosquitto_args} -t $subscribe_args >"${fin}" &
+ mosquitto_pub ${mosquitto_args} -t "${output_topic}" -l <"${fout}" &
+ mosquitto_pub ${mosquitto_args} -t "${error_topic}" -l <"${ferr}" &
+ trap -- 'cleanup' ERR INT HUP EXIT CHLD
+
+#pause
+ while :
+ do
+ $runtime $runtime_args <"${fin}" >"${fout}" 2>"${ferr}"
+ done
+
+## Connecting stdin, stdout, and stderr with MQTT topics (cont.)
+
+Works if input is continuously evaluated (i.e., does not wait for EOF)
+
+ RUNTIME=bc ARGS="-l" sh -x ./mqtt-wrapper
+ RUNTIME=sh ARGS="your-favorite-cgi-script" sh -x ./mqtt-wrapper
+ RUNTIME=sh ARGS="your-sd-program" sh -x ./mqtt-wrapper
+
+## Outlook
+* Lots of possibilities coupling pub/sub with text streams
+* Use existing client/broker implementations and libraries
+* Minimal C implementation would be nice, aiming at a subset of features
+* Represent topic structure with virtual file system representation
+ (i.e., plan9 or /proc in linux)
+
+## Challenge 0: Installing a MQTT broker/client
+Mosquitto (mostly C, BSD-like license).
+Pretty bloated, a minimal alternative would be nice.
+You just need the client for these exercises.
+
+ https://github.com/eclipse/mosquitto
+
+Package names in common OS:
+
+ OpenBSD mosquitto broker+client
+ Arch mosquitto broker+client
+ Gentoo app-misc/mosquitto broker+client
+ Debian mosquitto-clients client
+
+## Challenge agenda
+First one to post a working solution will get points towards a
+price.
+
+## Challenge 1: Let's write a chat
+Hints:
+
+ mosquitto_pub(1): Publish a message to a named topic
+ Useful flags:
+ -d: Show debug info
+ -l: Read messages from stdin (each line 1 message)
+ mosquitto_sub(1): Subscribe to a path in the topic hierarchy
+ -d: Show debug info
+ -v: Print topic for each received message as first field
+
+ Connection URI:
+ -L 'mqtt://bitreich:bitreich@mx2.adamsgaard.dk:65431/testtopic'
+
+Let's use the message format: "nick\tmessage"
+
+## Challenge 2: Write a script that records the chat
+Hint: The '#' wild card
+
+What is the best choice for a time stamp?
+* Time as the sender reports it?
+* Time as the subscriber sees it?
+
+## Challenge 3: Let the data flow
+Anyone has a funky data source?
+
+* USB body thermometer?
+* spoon -t
+* while :; do printf '%s\t%s' "$(hostname)" "$(uptime)"; sleep 10; done
+* ii/annna?
+* vmstat(8)
+
+Task: Publish some data and let the others guess what it is.
+The funkier, the better.
+
+Message format ideas:
+ TSV, JSON, XLSX? sfeed(5)?
+ VT100 control codes??
+
+## Challenge 4: Raise an alert!
+Set up a script that monitors one of the data sources, and triggers
+an alert upon a condition (email, xmessage, log line, ...).
+
+## Challenge 5: Make some gnuplot
+Monitor one of the sensors from the previous, and create a continuously
+updating gnuplot (-t dumb) with a rolling window.
+
+## Challenge 6: Health check reporting
+Write a script that monitors some gopher and reports their status to
+
+ gopherstatus/<uri>
+ up|down
+
+## Challenge 7: Tamagotchi
+Make script that imitates a virtual pet, expecting to be feed
+"cookies" as messages on stdin every 30 s. It should die if it gets to much or too little food.
+
+Report its happyness to a topic with ASCII smileys.
+
DIR diff --git a/mqtt/mqtt-wrapper b/mqtt/mqtt-wrapper
@@ -0,0 +1,59 @@
+#!/bin/sh
+set -x
+#
+# $input_topic | $runtime $runtime_args | $output_topic
+# 2>$error_topic
+#
+# application to run
+runtime="${RUNTIME:-cat}"
+# arguments to $runtime
+runtime_args="${ARGS}"
+# stdin for $runtime; where others send to
+input_topics="${INPUT_TOPICS:-$$/in}"
+# stdout of $runtime; where others need to subscribe
+output_topic="${OUTPUT_TOPIC:-$$/out}"
+# stderr of $runtime; where others can subscribe
+error_topic="${ERROR_TOPIC:-$$/err}"
+
+mqtt_broker="${MQTT_BROKER:-mx2.adamsgaard.dk}"
+mqtt_port="${MQTT_BROKER:-65431}"
+mqtt_user="${MQTT_USER:-bitreich}"
+mqtt_password="${MQTT_PASSWORD:-bitreich}"
+mosquitto_args="-h ${mqtt_broker} -p ${mqtt_port} -u ${mqtt_user} -P ${mqtt_password}"
+
+cleanup() {
+ kill -TERM -- -$$
+ rm -f "$fin" "$fout" "$ferr"
+ rmdir "$fifodir"
+ exit 1
+}
+
+die() {
+ printf '%s: %s\n' "${0##*/}" "${1}" >&2
+ cleanup
+ exit "${2:-1}"
+}
+
+fifodir="$(mktemp -d)" || die 'mktemp'
+fin="${fifodir}/in"
+fout="${fifodir}/out"
+ferr="${fifodir}/err"
+mkfifo "$fin" "$fout" "$ferr" || die 'mkfifo'
+
+# split multiple input topics separated by "," into separate args
+subscribe_args="$(printf '%s' "$input_topics" | sed 's/,/ -t /g')"
+
+# subscribe to input topic(s), add -v option if you want message topic as first field
+mosquitto_sub ${mosquitto_args} -t $subscribe_args >"${fin}" &
+
+# create named pipes for output streams
+mosquitto_pub ${mosquitto_args} -t "${output_topic}" -l <"${fout}" &
+mosquitto_pub ${mosquitto_args} -t "${error_topic}" -l <"${ferr}" &
+trap -- 'cleanup' ERR INT HUP EXIT CHLD
+
+# run the program (lossy restart if it fails), connecting to named pipes
+while :
+do
+ $runtime $runtime_args <"${fin}" >"${fout}" 2>"${ferr}"
+done
+
DIR diff --git a/mqtt/tamagotchi b/mqtt/tamagotchi
@@ -0,0 +1 @@
+Subproject commit 61808fed51f116f4c1b5ab622a9e3e76738b303b