Subj : Double postings
To   : Mark Lewis
From : Joe Martin
Date : Sun Sep 15 2019 08:31:21

-> JM> Just out of curiousity, what's the recommended way of dupe checki
-> JM> these days, MSGID, TID/PID, CRC, combination there of (including
-> JM> Header)?
->
-> MSGID is the main way but older software doesn't generate MSGID so
-> other methods need to be used...

My mailer/tosser uses a combined approach.  If the message contains a
MSGID then use its value, otherwise CRC the header and message body
including control lines but never the SEEN-BY/PATH lines (considering
they change all the time).  The tosser never duplicates an MSGID either
as it maintains a file with the last used value seeded upon creation by
the current date/time.  This prevents issues should that file get
deleted.

To provide speed and limit disk space, I also have an expiration
mechanism (user configurable) that will purge CRC entries after a given
amount of time (ie: 2 weeks but not more than 30 days).  So while it's
efficient catching dupes in that time period, if someone does a rescan
and dumps everything back into the echo a month later, it won't catch
them. It's a trade off, but back in the day when we had 40mb drives and
8088/80286 processors, it was extremely important.

-> instead of CRC... the problem then comes from those systems that
-> mistakenly reformat the messages as they process them and write the
-> reformatted messages to new PKTs... now the message body is

Yeah this is and always will be an issue.

-> is apparent on systems that only get, for example, one posting of an
-> echos rules each month and only accept new postings of those rules

It would seem to me, (me mind you) that if you're moderating an echo,
your software "should" be able to generate a MSGID to prevent this issue
entirely.  But hey...

-> what i would do would be to ask other tosser devs what they use in
-> their code...
->
-> listed in no particular order:
->
-> tobias burchhardt  - fastecho
-> rob swindell       - sbbsecho
-> nick andre         - d'bridge
-> vince coen         - mbse's tosser
-> kim heino          - bbbs' tosser
-> wilfred van velzen - fmail
-> james coyle        - mystic

Thanks Mark...

--- ViaMAIL!/WC v2.00
 * Origin: ViaSoft Support BBS - Back online at 303-953-0568  (1:104/57)

.