Separating Data Streams into Flows

Not two weeks in the past, I'd an epiphany about formatting metadata and the like.  I prefer network
databases, and the obvious way to persist such state is an event log that can be followed to rebuild
the database from nothing, as needed.  It seems nicer to me if different kinds of events were stored
in different streams, and it clearly follows that streams are more pleasant homogeneous than not.  I
see in my mind's eye streams for each unique interaction with the database, each record therein with
the minimum information needed to be unambiguous.  I see no issue with placing an entire database in
working memory, as this is ostensibly that one true reason virtual memory exists in the first place.

I've implemented metadata formats for my Meta-Machine Code tool, and I've designed some whose intake
has never been finished.  It's painful, boring, and tedious to make such things, and I long for ways
to avoid this drudgery.  The easiest way seems to be to avoid this problem entirely by using a lower
system which represents such data in its own ways, but this leads to a lack of unique character, and
merely shifts some of the work to different places.  I believe the problem looks much nicer when the
single stream be split into many, as I'd do with persistence of a database, however.  The epiphany's
as follows: Instead of placing specialized metadata in a single stream and pretending such things as
headers with version numbers matter, cease with the charade.  Just as APL has a hacker splitting his
code into succinct and simple steps to be combined to solve his problem, this approach has it easier
to use loops and functions already designed and applicable for far more than but a single algorithm.

I believe this be distinct from how the big and bad databases put data in storage, but am uncertain.

I figure it to be always better to split streams into multiple flows, so long as some simplification
occurs afterwards.  This contrasts with that conventional way to handle such streams: To define many
little functions that take turns drinking from the stream, knowing how to let each other take turns.
Both approaches are clearly equivalent, and yet I too see it clearly easier to use separate streams.

The metadata format of the Meta-Machine Code tool is simple, but I've never found it to be pleasant:
A version header precedes everything; followed by a pair of addresses which reveal the length of the
program's main metadata, following in turn as a sequence of fixed-length records; and these followed
by names with variable lengths tacked onto the end.  To parse this metadata, I used a single, larger
function which also checked and bemoaned any extra data left over.  Though it comprises well-defined
steps, I still find the code to be clumsy and its best error handling strategy annoyingly vague.  To
split this metadata across two streams seems much cleaner: The header would be unnecessary and gone;
that first sequence of records could be replaced by a sequence of integers, all but the first two of
which correspond to records, as the first two would instead represent the addresses; and those names
would have the same representation, but entirely alone in their own stream.  This changes the intake
drinking from one stream carefully to drinking carelessly from two; the loops or patterns needed are
but taking until naught remains or, more defensively, imposing a limit to such taking, which becomes
a clear error whenever met.  Checking metadata formed into objects afterwards uses normal functions,
such as length verification, duplicate detection, and simple code that checks each item identically.

I've given a decent deal of thought, although no implementation, to the design of the dictionary for
the Elision system into which I want to breathe life sometime soon.  One aspect of its design is use
of a large pool holding letters with smaller representations than that common octet; this introduces
issues of padding and alignment, the best order to put this data with the rest, and how to represent
its length.  Drinking this data from its very own stream handles all of these problems so trivially.

Separation of data into multiple flows also makes unpleasant additions easier to consider.  My Meta-
Machine Code tool's current metadata format for CHIP-8 has a minor flaw: It can't represent a single
rare case, an octet with more than one name used in it, which occurs with a single instruction.  The
option of bloating the metadata for a single rarity sickened me, and I chose to ignore the matter as
I'm the only user.  It occurred to me that such specialized metadata, using a less efficient format,
could go behind the usual metadata; it's clear there's no end to the garbage that can be shoved into
the end of such a place.  It's best to have such in its own stream, where it will go mostly ignored.
Another matter is the inclusion of Meta-Machine Code metadata containing only names, which would use
a special version number, and whose existence is precluded entirely by this quite simple separation.

Separating streams into flows so requires a higher system to locate them, but this is rather simple,
and not greatly offensive.  Use of a higher system would pop up somewhere in all likelihood, anyway.

I likewise want to change how this HTTP server logs requests, and have long thought about the layers
of logging needed.  The first layer of storage must be fast to set, linear time processing at worst,
and would consist of nothing but the request's lines, preferably prefixed by their lengths, using an
empty line to terminate the request; storing time and address information somewhere, somehow, in the
first line works well enough.  The second layer of storage would transform the first into a sequence
of fixed-length records, the first of which holds the time and address, with those remaining holding
references to a second stream carrying lines in a relatively unstructured way; an empty record would
terminate the request.  Optimizations such as deduplication or elimination of substrings are obvious
and not particularly hard to realize.  The logging format would be small and hold all relevant data.

When I design formats in the future, I'll be preferring to split data across multiple flows like so.
.