## brcon2024 - 2024-08-08
title: Deep C programming - UNIX philosophy in distributed offshore systems
author: Anders Damsgaard (adc)
contact: anders@adamsgaard.dk
gopher://adamsgaard.dk
https://adamsgaard.dk
slides: gopher://adamsgaard.dk/0/tmp/brcon2024_adc.md
## brcon2024 - 2024-08-08
title: Deep C programming - UNIX philosophy in distributed offshore systems
author: Anders Damsgaard (adc)
contact: anders@adamsgaard.dk
gopher://adamsgaard.dk
https://adamsgaard.dk
## Previously..
* brcon2020: Energy efficient programming in science
* brcon2021: Unix principles for science simulations
* brcon2022: Est. Bitreich Arctic Vault
* brcon2023: Using Unix and Bitreich tools for preparing scientific papers
* Today: UNIX philosophy in distributed offshore systems
#pause
* Tomorrow: UNOX philosophy in Vitré
## Outline
* How we can benefit from hyped technology with true and tried principles
* Overview of the offshore system
* The talk transitions into hands-on exercises
* Ask questions along the way in #bitreich-con
## Danish Geotechnical Institute (geo.dk)
* ~250 employees in Lyngby and Aarhus, Denmark
* Geotechnical investigations (onshore/offshore)
* Environmental consultancy
* Laboratory tests
* Geodata analysis
* Geoscientific software development
#pause
-> New offshore rig built from the ground up
## Offshore geotechnical investigations
.* +------------------+
\|\ | ,_____. |
_============. | | | |
\\# # # #/ | | ._______|--|--| |
| +.,| | Geo .--|--| |
_---------------------|-+-------|--|--|--|\
\ MS/Bitreich | +--|--+ | |
~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~~~~|~|~~~~~~~~~~~~
\ +------------------+'' ~
\_________________________________/--+ ~ . ~~
~
><>
<><
----------------------------------------------------------------------
## Offshore geotechnical investigations
.*
\|\ ,_____.
_============. | |
\\# # # #/ | ._______|--|--|
| +., | Geo .--|--|
_---------------------+-+-------|--|--|---\
\ MS/Bitreich +--|--+ |
~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~
\ .-'' ~
\_________________________________/--+ ~ . ~~
~
><>
<><
----------------------------------------------------------------------
## Offshore geotechnical investigations
.*
\|\ ,_____.
_============. | |
\\# # # #/ | ._______|--|--|
| +., | Geo | | |
_---------------------+-+-------|--|--|---\
\ MS/Bitreich | |
~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
\ | .-'' ~
\_____________________________|___/--x~ ~ . ~~
.--|--. ~
| v |
+-----+
><>
<><
----------------------------------------------------------------------
## Offshore geotechnical investigations
.*
\|\ ,_____.
_============. | |
\\# # # #/ | ._______|--|--|
| +., | Geo | | |
_---------------------+-+-------|--|--|---\
\ MS/Bitreich | |
~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
\ | .-'~ ~ . ~~
\_____________________________|___/--x ~
|
|
|
><> |
|
<>< |
|
.--|--.
| v |
----------------------------------------------+-----+-----------------
## Offshore geotechnical investigations
.*
\|\ ,_____.
_============. | |
\\# # # #/ | ._______|--|--|
| +., | Geo | | |
_---------------------+-+-------|--|--|---\
\ MS/Bitreich | |
~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
\ | .-'' ~
\_____________________________|___/--x
|
|
|
><> |
|
<>< |
|
.--|--.
| | |
----------------------------------------------+--|--+-----------------
|
|
|
v
## Offshore geotechnical investigations
| |
| |
| |
| |
| |
| |
| |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~|~~~~~~~~~~~~
| |
| |
| |
| |
? | |
><> | |
| |
| | !
| | <><
| |
| |
-----------------------------------------|---------------|------------
| |
| |
| |
| |
|_______________|
https://www.mdpi.com/energies/energies-17-01964/article_deploy/html/images/energies-17-01964-g001.png
## Offshore systems overview
* Up to 800 sensors, usually 1-10 Hz
* SCADA, PLCs, environmental sensors, navigation, scientific results,
calibration data, ...
* Control messages
* Asynchronous data streams
* Event driven logic
* Raw sensor output into physical units
* Aggregation of time series and combination of sources
* Time series databases
* Development project with many involved partners
## Approaches: Unix philosophy vs. Microsoft mentality
* Does the client know what they want?
* Do we know everything about what we want to solve?
* Do we know if our assumptions are valid, in particular in complex systems?
* Do we have full specifications?
* Should we produce a big integrated C# monolith to handle computations,
flow, storage, logic?
#pause
> yes no | head -n 5
## Data routing system
* MQTT: Message Queuing Telemetry Transport
* Started as a proprietary protocol for SCADA systems in O&G
* Lightweight, publish-subscribe network protocol
* Constrained devices and low-bandwidth, high-latency networks
* Publish/Subscribe Model: Decouples message producers from consumers
* Data accessible via network, not living internally in an obscure program
Specifications:
- MQTT v5.0: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html
- MQTT v3.1.1: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html
- https://github.com/mqtt/mqtt.org/wiki/Differences-between-3.1.1-and-5.0
- MQTT and TLS: https://www.rfc-editor.org/rfc/rfc9431.txt
## Central concepts
* Broker: Central server that routes messages
* Clients: Devices that publish or subscribe to one or more topics
* Messages: Data packets sent between clients via the broker
* Topics: Hierarchical and dynamic name space for message
Libs for many languages, e.g.,
- https://github.com/eclipse/paho.mqtt.c
- https://github.com/eclipse/paho.mqtt.golang
## DISCLAIMER
The coming exercises mainly to demonstrate fundamentals and principles
Most are suboptimal or bad practice
But, let's have some #fun ...
## MQTT use cases
* Home thermometers and weather stations (if you ask the web)
* IoT Devices: Smart homes, wearables, and industrial automation
* Mobile Applications: Real-time messaging and notifications
* Remote Monitoring: Environmental sensors, health monitoring
* Connected Vehicles: Telematics and fleet management
## MQTT messages
* TCP/IP
* Limit to message size configurable
* New subscribers receive only new messages (no backlog)
* Retained Messages: Last known good value for new subscribers
* Last Will and Testament (LWT): Notifies other clients about an ungraceful disconnection
* Quality of Service (QoS): Three levels to ensure message delivery
* QoS 0: At most once delivery (fire and forget)
* QoS 1: At least once delivery (acknowledged delivery)
* QoS 2: Exactly once delivery (assured delivery)
## Why MQTT for the offshore project?
* Hierarchial data structure through topics
* A node in the topic tree exists as soon as someone publishes or
subscribes to it
* Data accessible via network and all clients can read or publish
(by default)
#pause
rig/sensor1/raw
rig/sensor1/phys
rig/sensor2/raw
rig/sensor2/phys
...
ship/navigation/easting
ship/navigation/northing
...
## Topic examples
Wildcards: #, + (subscribe only)
Subscription examples:
1) rig/sensor1/raw
2) ship/#: Subscribe to everything from the ship
3) rig/+/phys: Subscribe to all physical values
## Example: Subscribe and publish
$ cat bin/timestamp # in case you don't have ts(1)
#!/bin/sh
while read -r REPLY
do
printf '%s %s\n' "$(date '+%Y-%m-%d %Z %H:%M:%S')" "$REPLY"
done
#pause
$ export MQTTURI='mqtt://bitreich:bitreich@mx2.adamsgaard.dk:65431/'
#pause
$ mosquitto_sub -L "${MQTTURI}#" -v -t '#' | timestamp | grep -v hatesensor
#pause
$ date | mosquitto_pub -L "${MQTTURI}$(hostname)/$(whoami)" -l
## Reporting online status via last will and retain, reporting some metrics
gophers://duckless.org/0/tmp/mqtt-online
#!/bin/sh
online_topic="${ONLINE_TOPIC:-$(hostname)/online}"
mqtt_broker="${MQTT_BROKER:-mx2.adamsgaard.dk}"
mqtt_port="${MQTT_PORT:-65431}"
mqtt_user="${MQTT_USER:-bitreich}"
mqtt_password="${MQTT_PASSWORD:-bitreich}"
args="-h ${mqtt_broker} -p ${mqtt_port} -u ${mqtt_user} -P ${mqtt_password}"
mosquitto_pub ${args} -t "${online_topic}" -r -m "1"
while :
do
printf '%s\t%s\t%s\t%s\n' "$(date +%s)" "$(date)" "$(hostname)" "$(uptime)"
sleep 360
done | mosquitto_pub ${args} --will-topic "${online_topic}" \
--will-retain --will-payload "0" -t "${online_topic}" -r -l
## Data structure on the network
Interact with data hierarchy via small modular clients.
Brings all benefits of UNIX design into play.
The tenents of the UNIX philosophy:
- Small is beautiful
- Make each program do one thing well
- Build a prototype as soon as possible
- Choose portability over efficiency
- Store numerical data in flat ASCII files
- Use software leverage to your advantage
- Use shell scripts to increase leverage and portability
- Avoid captive user interfaces
- Make every program a filter
src: Make Gancarz 1995 "The UNIX Philosophy", ISBN 1-55558-123-4
## Distributed computing
Assign small modular programs to read stdin from topics and post their
result to another topic
* Rudimentary distributed computing
* Each program acts as a text filter
* Each program can contribute local data
(disclaimer: for 1:1 connections, netcat or named pipes over SSH is simpler)
Broker statistics: $SYS/broker
mosquitto_sub -L "${MQTTURI}\$SYS/broker/#" -v
## Connecting stdin, stdout, and stderr with MQTT topics
Full script: gophers://duckless.org/0/tmp/mqtt-wrapper
fifodir="$(mktemp -d)" || die 'mktemp'
fin="${fifodir}/in"
fout="${fifodir}/out"
ferr="${fifodir}/err"
mkfifo "$fin" "$fout" "$ferr" || die 'mkfifo'
#pause
mosquitto_sub ${mosquitto_args} -t $subscribe_args >"${fin}" &
mosquitto_pub ${mosquitto_args} -t "${output_topic}" -l <"${fout}" &
mosquitto_pub ${mosquitto_args} -t "${error_topic}" -l <"${ferr}" &
trap -- 'cleanup' ERR INT HUP EXIT CHLD
#pause
while :
do
$runtime $runtime_args <"${fin}" >"${fout}" 2>"${ferr}"
done
## Connecting stdin, stdout, and stderr with MQTT topics (cont.)
Works if input is continuously evaluated (i.e., does not wait for EOF)
RUNTIME=bc ARGS="-l" sh -x ./mqtt-wrapper
RUNTIME=sh ARGS="your-favorite-cgi-script" sh -x ./mqtt-wrapper
RUNTIME=sh ARGS="your-sd-program" sh -x ./mqtt-wrapper
## Outlook
* Lots of possibilities coupling pub/sub with text streams
* Use existing client/broker implementations and libraries
* Minimal C implementation would be nice, aiming at a subset of features
* Represent topic structure with virtual file system representation
(i.e., plan9 or /proc in linux)
## Challenge 0: Installing a MQTT broker/client
Mosquitto (mostly C, BSD-like license).
Pretty bloated, a minimal alternative would be nice.
You just need the client for these exercises.
https://github.com/eclipse/mosquitto
Package names in common OS:
OpenBSD mosquitto broker+client
Arch mosquitto broker+client
Gentoo app-misc/mosquitto broker+client
Debian mosquitto-clients client
## Challenge agenda
First one to post a working solution will get points towards a
price.
## Challenge 1: Let's write a chat
Hints:
mosquitto_pub(1): Publish a message to a named topic
Useful flags:
-d: Show debug info
-l: Read messages from stdin (each line 1 message)
mosquitto_sub(1): Subscribe to a path in the topic hierarchy
-d: Show debug info
-v: Print topic for each received message as first field
Connection URI:
-L 'mqtt://bitreich:bitreich@mx2.adamsgaard.dk:65431/testtopic'
Let's use the message format: "nick\tmessage"
## Challenge 2: Write a script that records the chat
Hint: The '#' wild card
What is the best choice for a time stamp?
* Time as the sender reports it?
* Time as the subscriber sees it?
## Challenge 3: Let the data flow
Anyone has a funky data source?
* USB body thermometer?
* spoon -t
* while :; do printf '%s\t%s' "$(hostname)" "$(uptime)"; sleep 10; done
* ii/annna?
* vmstat(8)
Task: Publish some data and let the others guess what it is.
The funkier, the better.
Message format ideas:
TSV, JSON, XLSX? sfeed(5)?
VT100 control codes??
## Challenge 4: Raise an alert!
Set up a script that monitors one of the data sources, and triggers
an alert upon a condition (email, xmessage, log line, ...).
## Challenge 5: Make some gnuplot
Monitor one of the sensors from the previous, and create a continuously
updating gnuplot (-t dumb) with a rolling window.
## Challenge 6: Health check reporting
Write a script that monitors some gopher and reports their status to
gopherstatus/
up|down
## Challenge 7: Tamagotchi
Make script that imitates a virtual pet, expecting to be feed
"cookies" as messages on stdin every 30 s.
It should die if it gets to much or too little food.
Report its happyness to a topic with ASCII smileys.