brcon2024_adc.md - brcon2024-hackathons - Bitreichcon 2024 Hackathons
HTML git clone git://bitreich.org/brcon2024-hackathons git://enlrupgkhuxnvlhsf6lc3fziv5h2hhfrinws65d7roiv6bfj7d652fid.onion/brcon2024-hackathons
DIR Log
DIR Files
DIR Refs
DIR Tags
DIR Submodules
---
brcon2024_adc.md (17244B)
---
1 ## brcon2024 - 2024-08-08
2
3 title: Deep C programming - UNIX philosophy in distributed offshore systems
4
5 author: Anders Damsgaard (adc)
6
7 contact: anders@adamsgaard.dk
8 gopher://adamsgaard.dk
9 https://adamsgaard.dk
10
11 slides: gopher://adamsgaard.dk/tmp/brcon2024_adc.md
12
13 ## brcon2024 - 2024-08-08
14
15 title: Deep C programming - UNIX philosophy in distributed offshore systems
16
17 author: Anders Damsgaard (adc)
18
19 contact: anders@adamsgaard.dk
20 gopher://adamsgaard.dk
21 https://adamsgaard.dk
22 <marquee><blink>
23 slides: gopher://adamsgaard.dk/tmp/brcon2024_adc.md
24 </blink></marquee>
25
26
27 ## Previously..
28 * brcon2020: Energy efficient programming in science
29 * brcon2021: Unix principles for science simulations
30 * brcon2022: Est. Bitreich Arctic Vault
31 * brcon2023: Using Unix and Bitreich tools for preparing scientific papers
32 * Today: UNIX philosophy in distributed offshore systems
33 #pause
34 * Tomorrow: UNOX philosophy in Vitré
35
36 ## Outline
37 * How we can benefit from hyped technology with true and tried principles
38 * Overview of the offshore system
39 * The talk transitions into hands-on exercises
40 * Ask questions along the way in #bitreich-con
41
42 ## Danish Geotechnical Institute (geo.dk)
43 * ~250 employees in Lyngby and Aarhus, Denmark
44 * Geotechnical investigations (onshore/offshore)
45 * Environmental consultancy
46 * Laboratory tests
47 * Geodata analysis
48 * Geoscientific software development
49
50 #pause
51 -> New offshore rig built from the ground up
52
53 ## Offshore geotechnical investigations
54 .* +------------------+
55 \|\ | ,_____. |
56 _============. | | | |
57 \\# # # #/ | | ._______|--|--| |
58 | +.,| | Geo .--|--| |
59 _---------------------|-+-------|--|--|--|\
60 \ MS/Bitreich | +--|--+ | |
61 ~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~~~~|~|~~~~~~~~~~~~
62 \ +------------------+'' ~
63 \_________________________________/--+ ~ . ~~
64 ~
65
66
67 ><>
68
69 <><
70
71
72
73 ----------------------------------------------------------------------
74
75
76 ## Offshore geotechnical investigations
77 .*
78 \|\ ,_____.
79 _============. | |
80 \\# # # #/ | ._______|--|--|
81 | +., | Geo .--|--|
82 _---------------------+-+-------|--|--|---\
83 \ MS/Bitreich +--|--+ |
84 ~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~
85 \ .-'' ~
86 \_________________________________/--+ ~ . ~~
87 ~
88
89
90 ><>
91
92 <><
93
94
95
96 ----------------------------------------------------------------------
97
98
99
100 ## Offshore geotechnical investigations
101 .*
102 \|\ ,_____.
103 _============. | |
104 \\# # # #/ | ._______|--|--|
105 | +., | Geo | | |
106 _---------------------+-+-------|--|--|---\
107 \ MS/Bitreich | |
108 ~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
109 \ | .-'' ~
110 \_____________________________|___/--x~ ~ . ~~
111 .--|--. ~
112 | v |
113 +-----+
114 ><>
115
116 <><
117
118
119
120 ----------------------------------------------------------------------
121
122
123
124
125 ## Offshore geotechnical investigations
126 .*
127 \|\ ,_____.
128 _============. | |
129 \\# # # #/ | ._______|--|--|
130 | +., | Geo | | |
131 _---------------------+-+-------|--|--|---\
132 \ MS/Bitreich | |
133 ~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
134 \ | .-'~ ~ . ~~
135 \_____________________________|___/--x ~
136 |
137 |
138 |
139 ><> |
140 |
141 <>< |
142 |
143 .--|--.
144 | v |
145 ----------------------------------------------+-----+-----------------
146
147
148
149
150 ## Offshore geotechnical investigations
151 .*
152 \|\ ,_____.
153 _============. | |
154 \\# # # #/ | ._______|--|--|
155 | +., | Geo | | |
156 _---------------------+-+-------|--|--|---\
157 \ MS/Bitreich | |
158 ~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~|~~~~~~~~~~~~
159 \ | .-'' ~
160 \_____________________________|___/--x
161 |
162 |
163 |
164 ><> |
165 |
166 <>< |
167 |
168 .--|--.
169 | | |
170 ----------------------------------------------+--|--+-----------------
171 |
172 |
173 |
174 v
175
176 ## Offshore geotechnical investigations
177 | |
178 | |
179 | |
180 | |
181 | |
182 | |
183 | |
184 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~|~~~~~~~~~~~~
185 | |
186 | |
187 | |
188 | |
189 ? | |
190 ><> | |
191 | |
192 | | !
193 | | <><
194 | |
195 | |
196 -----------------------------------------|---------------|------------
197 | |
198 | |
199 | |
200 | |
201 |_______________|
202
203 https://www.mdpi.com/energies/energies-17-01964/article_deploy/html/images/energies-17-01964-g001.png
204
205 ## Offshore systems overview
206 * Up to 800 sensors, usually 1-10 Hz
207 * SCADA, PLCs, environmental sensors, navigation, scientific results,
208 calibration data, ...
209 * Control messages
210 * Asynchronous data streams
211 * Event driven logic
212 * Raw sensor output into physical units
213 * Aggregation of time series and combination of sources
214 * Time series databases
215 * Development project with many involved partners
216
217 ## Approaches: Unix philosophy vs. Microsoft mentality
218 * Does the client know what they want?
219 * Do we know everything about what we want to solve?
220 * Do we know if our assumptions are valid, in particular in complex systems?
221 * Do we have full specifications?
222 * Should we produce a big integrated C# monolith to handle computations,
223 flow, storage, logic?
224
225 #pause
226 > yes no | head -n 5
227
228
229 ## Data routing system
230 * MQTT: Message Queuing Telemetry Transport
231 * Started as a proprietary protocol for SCADA systems in O&G
232 * Lightweight, publish-subscribe network protocol
233 * Constrained devices and low-bandwidth, high-latency networks
234 * Publish/Subscribe Model: Decouples message producers from consumers
235 * Data accessible via network, not living internally in an obscure program
236
237 Specifications:
238 - MQTT v5.0: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html
239 - MQTT v3.1.1: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html
240 - https://github.com/mqtt/mqtt.org/wiki/Differences-between-3.1.1-and-5.0
241 - MQTT and TLS: https://www.rfc-editor.org/rfc/rfc9431.txt
242
243 ## Central concepts
244 * Broker: Central server that routes messages
245 * Clients: Devices that publish or subscribe to one or more topics
246 * Messages: Data packets sent between clients via the broker
247 * Topics: Hierarchical and dynamic name space for message
248
249 Libs for many languages, e.g.,
250 - https://github.com/eclipse/paho.mqtt.c
251 - https://github.com/eclipse/paho.mqtt.golang
252
253 ## DISCLAIMER
254
255 The coming exercises mainly to demonstrate fundamentals and principles
256
257 Most are suboptimal or bad practice
258
259 But, let's have some #fun ...
260
261 ## MQTT use cases
262 * Home thermometers and weather stations (if you ask the web)
263 * IoT Devices: Smart homes, wearables, and industrial automation
264 * Mobile Applications: Real-time messaging and notifications
265 * Remote Monitoring: Environmental sensors, health monitoring
266 * Connected Vehicles: Telematics and fleet management
267
268 ## MQTT messages
269 * TCP/IP
270 * Limit to message size configurable
271
272 * New subscribers receive only new messages (no backlog)
273 * Retained Messages: Last known good value for new subscribers
274 * Last Will and Testament (LWT): Notifies other clients about an ungraceful disconnection
275
276 * Quality of Service (QoS): Three levels to ensure message delivery
277 * QoS 0: At most once delivery (fire and forget)
278 * QoS 1: At least once delivery (acknowledged delivery)
279 * QoS 2: Exactly once delivery (assured delivery)
280
281 ## Why MQTT for the offshore project?
282 * Hierarchial data structure through topics
283 * A node in the topic tree exists as soon as someone publishes or
284 subscribes to it
285 * Data accessible via network and all clients can read or publish
286 (by default)
287
288 #pause
289 rig/sensor1/raw
290 rig/sensor1/phys
291 rig/sensor2/raw
292 rig/sensor2/phys
293 ...
294 ship/navigation/easting
295 ship/navigation/northing
296 ...
297
298 ## Topic examples
299 Wildcards: #, + (subscribe only)
300
301 Subscription examples:
302 1) rig/sensor1/raw
303 2) ship/#: Subscribe to everything from the ship
304 3) rig/+/phys: Subscribe to all physical values
305
306 ## Example: Subscribe and publish
307 $ cat bin/timestamp # in case you don't have ts(1)
308 #!/bin/sh
309 while read -r REPLY
310 do
311 printf '%s %s\n' "$(date '+%Y-%m-%d %Z %H:%M:%S')" "$REPLY"
312 done
313 #pause
314 $ export MQTTURI='mqtt://bitreich:bitreich@mx2.adamsgaard.dk:65431/'
315 #pause
316 $ mosquitto_sub -L "${MQTTURI}#" -v -t '#' | timestamp | grep -v hatesensor
317 #pause
318
319 $ date | mosquitto_pub -L "${MQTTURI}$(hostname)/$(whoami)" -l
320
321 ## Reporting online status via last will and retain, reporting some metrics
322
323 gophers://duckless.org/0/tmp/mqtt-online
324
325 #!/bin/sh
326 online_topic="${ONLINE_TOPIC:-$(hostname)/online}"
327 mqtt_broker="${MQTT_BROKER:-mx2.adamsgaard.dk}"
328 mqtt_port="${MQTT_PORT:-65431}"
329 mqtt_user="${MQTT_USER:-bitreich}"
330 mqtt_password="${MQTT_PASSWORD:-bitreich}"
331 args="-h ${mqtt_broker} -p ${mqtt_port} -u ${mqtt_user} -P ${mqtt_password}"
332 mosquitto_pub ${args} -t "${online_topic}" -r -m "1"
333 while :
334 do
335 printf '%s\t%s\t%s\t%s\n' "$(date +%s)" "$(date)" "$(hostname)" "$(uptime)"
336 sleep 360
337 done | mosquitto_pub ${args} --will-topic "${online_topic}" \
338 --will-retain --will-payload "0" -t "${online_topic}" -r -l
339
340 ## Data structure on the network
341 Interact with data hierarchy via small modular clients.
342 Brings all benefits of UNIX design into play.
343
344 The tenents of the UNIX philosophy:
345
346 - Small is beautiful
347 - Make each program do one thing well
348 - Build a prototype as soon as possible
349 - Choose portability over efficiency
350 - Store numerical data in flat ASCII files
351 - Use software leverage to your advantage
352 - Use shell scripts to increase leverage and portability
353 - Avoid captive user interfaces
354 - Make every program a filter
355
356 src: Make Gancarz 1995 "The UNIX Philosophy", ISBN 1-55558-123-4
357
358 ## Distributed computing
359 Assign small modular programs to read stdin from topics and post their
360 result to another topic
361
362 * Rudimentary distributed computing
363 * Each program acts as a text filter
364 * Each program can contribute local data
365
366 (disclaimer: for 1:1 connections, netcat or named pipes over SSH is simpler)
367
368 Broker statistics: $SYS/broker
369 mosquitto_sub -L "${MQTTURI}\$SYS/broker/#" -v
370
371 ## Connecting stdin, stdout, and stderr with MQTT topics
372 Full script: gophers://duckless.org/0/tmp/mqtt-wrapper
373
374 fifodir="$(mktemp -d)" || die 'mktemp'
375 fin="${fifodir}/in"
376 fout="${fifodir}/out"
377 ferr="${fifodir}/err"
378 mkfifo "$fin" "$fout" "$ferr" || die 'mkfifo'
379
380 #pause
381 mosquitto_sub ${mosquitto_args} -t $subscribe_args >"${fin}" &
382 mosquitto_pub ${mosquitto_args} -t "${output_topic}" -l <"${fout}" &
383 mosquitto_pub ${mosquitto_args} -t "${error_topic}" -l <"${ferr}" &
384 trap -- 'cleanup' ERR INT HUP EXIT CHLD
385
386 #pause
387 while :
388 do
389 $runtime $runtime_args <"${fin}" >"${fout}" 2>"${ferr}"
390 done
391
392 ## Connecting stdin, stdout, and stderr with MQTT topics (cont.)
393
394 Works if input is continuously evaluated (i.e., does not wait for EOF)
395
396 RUNTIME=bc ARGS="-l" sh -x ./mqtt-wrapper
397 RUNTIME=sh ARGS="your-favorite-cgi-script" sh -x ./mqtt-wrapper
398 RUNTIME=sh ARGS="your-sd-program" sh -x ./mqtt-wrapper
399
400 ## Outlook
401 * Lots of possibilities coupling pub/sub with text streams
402 * Use existing client/broker implementations and libraries
403 * Minimal C implementation would be nice, aiming at a subset of features
404 * Represent topic structure with virtual file system representation
405 (i.e., plan9 or /proc in linux)
406
407 ## Challenge 0: Installing a MQTT broker/client
408 Mosquitto (mostly C, BSD-like license).
409 Pretty bloated, a minimal alternative would be nice.
410 You just need the client for these exercises.
411
412 https://github.com/eclipse/mosquitto
413
414 Package names in common OS:
415
416 OpenBSD mosquitto broker+client
417 Arch mosquitto broker+client
418 Gentoo app-misc/mosquitto broker+client
419 Debian mosquitto-clients client
420
421 ## Challenge agenda
422 First one to post a working solution will get points towards a
423 price.
424
425 ## Challenge 1: Let's write a chat
426 Hints:
427
428 mosquitto_pub(1): Publish a message to a named topic
429 Useful flags:
430 -d: Show debug info
431 -l: Read messages from stdin (each line 1 message)
432 mosquitto_sub(1): Subscribe to a path in the topic hierarchy
433 -d: Show debug info
434 -v: Print topic for each received message as first field
435
436 Connection URI:
437 -L 'mqtt://bitreich:bitreich@mx2.adamsgaard.dk:65431/testtopic'
438
439 Let's use the message format: "nick\tmessage"
440
441 ## Challenge 2: Write a script that records the chat
442 Hint: The '#' wild card
443
444 What is the best choice for a time stamp?
445 * Time as the sender reports it?
446 * Time as the subscriber sees it?
447
448 ## Challenge 3: Let the data flow
449 Anyone has a funky data source?
450
451 * USB body thermometer?
452 * spoon -t
453 * while :; do printf '%s\t%s' "$(hostname)" "$(uptime)"; sleep 10; done
454 * ii/annna?
455 * vmstat(8)
456
457 Task: Publish some data and let the others guess what it is.
458 The funkier, the better.
459
460 Message format ideas:
461 TSV, JSON, XLSX? sfeed(5)?
462 VT100 control codes??
463
464 ## Challenge 4: Raise an alert!
465 Set up a script that monitors one of the data sources, and triggers
466 an alert upon a condition (email, xmessage, log line, ...).
467
468 ## Challenge 5: Make some gnuplot
469 Monitor one of the sensors from the previous, and create a continuously
470 updating gnuplot (-t dumb) with a rolling window.
471
472 ## Challenge 6: Health check reporting
473 Write a script that monitors some gopher and reports their status to
474
475 gopherstatus/<uri>
476 up|down
477
478 ## Challenge 7: Tamagotchi
479 Make script that imitates a virtual pet, expecting to be feed
480 "cookies" as messages on stdin every 30 s. It should die if it gets to much or too little food.
481
482 Report its happyness to a topic with ASCII smileys.
483