Data-Flow Programming in Parts of Pieces I'd like to implement BitTorrent one day, so I occasionally mull over different parts of the design. I believe it to be feasible to implement a subset of BitTorrent with use of neither TCP nor HTTP and expect it to work well for the most part, perhaps even largely lacking DNS; that's for some time not today, however. I was lately mulling over the calculation for the pieces part of a BitTorrent info- hash and noticed the Common Lisp CONCATENATED-STREAM trivialized it: Rather than splitting the files manually, pouring all into such a stream and reading blocks out of it removes all thought needed for the task. I figured the retarded data-flow programming system that is POSIX sh could also handle it well enough but differently, so I chose to implement both approaches and compare them in an article. I wanted to have an Ada implementation as well, but currently lack that time and energy to write it. At some other time, I plan to update this article to include the manual approach implemented in Ada. The idea behind my Common Lisp is simple enough, but practical matters made it much more complicated than first expected. Disregarding edge cases, it's as easy as this chain of calls via the listener:  CL-USER> (map 'open (list ...)) (# ...) CL-USER> (apply 'make-concatenated-stream *) # CL-USER> (loop :with vector := (make-array block :element-type 'octet) ;Define BLOCK and OCTET here. :for count := (read-sequence vector *) :until (zerop count) :collect (hash vector :start 0 :end count)) ;These keyword options make this trivial. (# ...) CL-USER> (mapcar 'close (list* ** ***)) (T ...)  Most of the hardship comes from properly handling the calls to OPEN while accounting for errors that will prevent the corresponding calls to CLOSE; neither WITH-OPEN-FILE nor WITH-OPEN-STREAM will work because they handle cases with only one stream, so it's necessary to use UNWIND-PROTECT. The common macros and functions that make collecting lists easy must be avoided also, however, as an incomplete list of open streams has to be accessible to the form that will close them. Ignoring the errors and only afterwards handling them is poor style in Common Lisp too, since interactive restarts available as any issue rises are best. The last pain is accommodating any function for HASH, with a displaced array used in the final call, rather than assuming :START and :END parameters or using SUBSEQ. Only that interaction with the listener clearly reveals the idea, hence its presence; I feel the finished code to obscure its purpose. Over half of the function comprises restarts and documentation string. I lately struggle to see the point in writing such error handling code, however; it usually feels to me like wasted effort to write such restarts, I vehemently dislike the explicit loops required which resist a nicer approach, thus I lately find myself preferring code which fails in different ways. I notice there's usually room for another restart, if only someone would place it, a sign of something wrong with the basic approach. I like the way by which an APL function fails, if it fails; the hard nature of an Ada system that won't compile rather than raise errors; and even the approach toys like Decker's Lil use, where the program makes some sense of what's happening and carries on. I'm rather certain Common Lisp's approach works well in large systems, but I've never worked with such systems. Error handling is always nicer elsewhere, yet the Free Software Common Lisp systems are rather bare. The sh script is simple only due to the pains taken by the GNU project, as GNU is the one UNIX clone worth a fuck, because the program uses a GNU extension to the split command which makes it much more useful: the --filter parameter turns the command into a harness deferring to a recursive instance of sh, sparing one from managing a folder of individual files output. Without GNU's split command, I'd not have bothered to carry the idea to completion; this pattern is common to the pathetic sh system: One must find exactly what one needs, and had better hope he finds it, or else it's not worth doing. Two lines are used as a single line would be too long. That first separates the parameter providing the block size from the rest providing the files. The second concatenates the files via cat; splits them into blocks by split; and both latter commands, cut and xxd, are needed to coerce the output of sha1sum into its straightest form. All logic is unconditional, at least if one ignores the variable referencing method that causes an exit. There were three lines originally, the second a conditional exit to handle that case of no files given; the cat command will read from standard input otherwise. I realized the first file given can be /dev/null, which defeats this and has no effect on the output whatsoever; in exchange for this dependency on an irrelevant detail of the file system that can't be changed once enough uses it, I saved one line of code; when this is done long enough, one gets UNIX. In comparison, the sh script is certainly much smaller than the Common Lisp; I make no measures, but wouldn't be surprised if it were much faster too. The Common Lisp certainly does far less worthless work and provides better error handling, however. Both demonstrate the basic principle well enough: The given files are concatenated, split into chunks taken from said stream, and their digests taken. The sh produces the exact output found in bencoding, whereas the Common Lisp returns a list which is more useful, whose elements may be concatenated and bencoded easily. The sh script is littered with utter nonsense requiring worthless knowledge to understand at all, else it resembles the magic which delights UNIX weenies purely because of its obscurity. The sh script benefits purely from the great deal of work done to provide a wide library of common functions. The Common Lisp system can be pure and provide introspection for every part of the program. Both programs are ultimately unimpressive. .