ListProc 8.1 Release Notes

Contents

• New Features
  • Performance Enhancements
    • Smart Configuration Loading
    • Initialize
    • Threading
    • Memory Usage
    • ILP Logins
    • String Handling
    • Well behaved Error processing
  • User Commands
    • WHICH-OWNED
    • SUBSCRIBE
  • List Options and Commands
    • Confirmation of Subscriber Commands
    • Alternate Address Commands
    • Subscribe with no name
    • REVIEW-BY-SUBSCRIBERS-SHORT
    • MTA-HOST
  • Config File Directives
    • max_file_length
• General Changes
• Bug Fixes

New Features

PERFORMANCE ENHANCEMENTS

• In past versions, the listproc command would load configuration and owner information for all lists at startup. Hence, processing any request - list related or otherwise - would require parsing all of the configuration information, and allocating memory for all of the lists. ListProc 8.1 reduces the extra processing by only loading a list's configuration information if and when it receives a request for that list.

• Owner information is not loaded when commands are sent by the manager, as no checks are required.

• Requests that require that all lists be loaded (like LISTS) do so after all syntax and semantics checks have been made.

• Remote lists loading is done on demand as well. Previously, all remote lists were loaded when a list was not local.

• In previous versions, an INIT LIST-NAME command would cause serverd to throw out all previously loaded list information, and reload everything. ListProc 8.1 only loads the new list information, but does not dump previously configured lists.

• When processing multiple INIT commands submitted in a single email, ListProc only initializes once - AFTER all commands have been processed.

• ListProc 8.0 would not spawn any new list processes while it processed incoming requests. Version 8.1 lifts this restriction.

• The memory footprint for served has been reduced by around 72% - for 1000 lists it used to consume 17MB; it now consumes 2.3MB.

• Special processing has been added to speed up owner logins.

• Subscription managers, errors-to recipients, moderators and subscribers can now specify a list name when logging in to ILP. This reduces the number of checks that must be made, and greatly speeds up the authentication process.

• Improvements have been made to some of the internal string handling routines, cutting their running time in half.

• The ListProc error processing thread now checks periodically to make sure it is not blocking any of the list threads. If it is, it allows the list threads to take precedence.

To top of page

USER COMMANDS

• A "WHICH-OWNED" command has been added, to allow owners to find out which lists they own.

• The SUBSCRIBE command now behaves differently when the user is already subscribed, but has her/his mail method set to POSTPONE. In the past, the user would simply be informed that they were already subscribed. In addition, ListProc 8.1 resets their mail method to the default for that list, and notifies the user that their mail method has been reset.

To top of page

LIST OPTIONS AND COMMANDS

• List owners now have the ability to force subscribers to confirm their intent to either subscribe OR unsubscribe from the list. If these options are turned on, subscribers who request to be (un)subscribed will be sent an additional message, that they must forward back to the server before their request will take effect. This can help prevent spoofed messages from adding bogus users, or removing legitimate ones.

  The following new options are defined for use with the CONFIG command:

    CONFIRM-ALL-SUBSCRIPTIONS
    DONT-CONFIRM-ALL-SUBSCRIPTIONS
  			
    CONFIRM-ALL-UNSUBSCRIPTIONS
    DONT-CONFIRM-ALL-UNSUBSCRIPTIONS
  				

  Both of these default to the 8.0 behavior of NOT requiring confirmation.

• ListProc 8.1 also introduces the ability for subscribers to issue commands on behalf of a different address. Since subscribers don't have permission to force actions for other users, these requests ALWAYS generate a confirmation message to the affected user.

  Among other things, this will allow users whose sending address has changed, but who can still receive at their old address, to unsubscribe themselves. For example, user "jrvb@info.cren.net" could send the following message to remove themselves from the FOO list:

    unsubscribe foo for jrvb@cren.net
  				

  A confirmation message would be sent to jrvb@cren.net, asking if he would really like to be removed from the list. If he forwarded this message back to the server, the request would go through. Otherwise, no action would be taken.

  The decision of whether or not to allow these ".... for user" commands is up to the list owners, and controlled by the following list config options:

    ALTERNATE-ADDRESS-COMMANDS
    NO-ALTERNATE-ADDRESS-COMMANDS
  				

  The default is to follow the previous behavior, and NOT allow subscribers to specify an alternate address.

• List owners now have the option of allowing users to subscribe without giving a name in the subscribe command. This is controlled by the following new liist config options:

    ALLOW-EMPTY-SUBSCRIBER-NAMES
    DONT-ALLOW-EMPTY-SUBSCRIBER-NAMES
  				

  If DONT-ALLOW-EMPTY-SUBSCRIBER-NAMES is set, ListProc will accept commands such as this for the list:

    SUBSCRIBE listname
  				

  In this case, the name field in the subscribers file will be filled with "No Name Given".

  The default is to use the 8.0 behavior, and require names for subscription.

• The new configuration option, REVIEW-BY-SUBSCRIBERS-SHORT, grants subscribers the ability to retrieve the list's current settings but NOT to retrieve the list of current subscribers.

• The MTA-HOST list config option allows the site manager to fine tune ListProc's performance by splitting the mail delivery load among multiple hosts. This way, lists that generate a particularly large amount of traffic won't slow down the delivery of other lists.

  The syntax is as follows:

    CONFIG <;list> <;password> MTA-HOST <;host> [port]
  				

  <;host> can be "" or '' to revert to the system default. This option works with the system mailmethod only. The default port is 25. A dash can be used as place holder for the hostname if only the port needs to be changed.

To top of page

CONFIG FILE DIRECTIVES

• The maximum length (in numbers of lines) for message ID and checksum files is now configurable with the max_file_length directive:

    max_file_length nlines
  				

To top of page

GENERAL CHANGES

• Made sysexec() fault-tolerant with fork() -- retry up to 1 minute when fork() fails. Usually fork failures are temporary.

• Modified the way listproc handles addresses to be ignored; before it would ignore them, but it would report so for every request sent in thus wasting cycles; it now ignores them after the first line of input and tosses the rest of the message.

• The REVIEW and STATS defaults are now -TO-OWNERS.

• Added the following regex to SUSP_SUBJECT: MAIL.*[ \t]FAIL Some weird address, "SJPL, SJPLPO/POSTMASTER" <;SJPLS@ci.sj.ca.us> generated a loop with Subject: Mail Failure that was not caught.

• As a result of the above we have reinstated mailer-daemon checks not only on the address, but the user name too, and we have expanded the check to the Resent-From address and user name.

• modified server_response() to read the MTA response all at once, rather than one character at a time. This requires that all systems support non-blocking I/O.

To top of page

BUG FIXES

• Fixed another core with handling of mmap(); it occurred when the file size was an exact multiple of the page size.

• Restricted matching of alternate addresses to 9; it is possible to define an alias like .+ which will match everything and listproc will match all addresses thus overflowing internal buffers.

• Fixed a logic bug with sensing requests in list mail (broken if no ignored_requests were defined in the config file).

• Fixed a long outstanding bug with zero-length temp files left around in tmp/. Still testing the fix on list.cren.net though (it's a timing issue)...

• Updated queued to require a semaphore id when using multiple threads; the semaphore id is passed onto pqueue.

• Fixed bug introduced in 8.0 beta where mail to multiple owners with any mailmethod except system was not working correctly (addresses were mangled). Turned out to be a wider problem with sending to more than one addresses with these mailmethods.

• Fixed bug with 7.2 and prior with mail sent to listname-request (meant for the owners); listproc failed to preserve MIME headers of incoming MIME messages thus sometimes rendering impossible the decoding of these message by the owners.

• Fixed core with "SET list FOR address"; this was broken in 7.2 too.

• Fixed core dump with fill_text() in list.c that Warren had written; apparently it did not work with words longer than 79 chars; an address (considered a word) longer than that would cause a seg fault.

• Fixed a serious bug with gexit() on most executables: it was calling internal routines, like tsprintf() and setup_gstring(); but when a error occurred in one of these gexit() would be called, which would in turn call these routines again; this internal loop would also fill up the .warning file and eat up all disk space eventually. gexit() is now a true exit routine.

• Fixed core dump with multi-line header lines; it was possible for header lines to get mixed up, or a seg fault to occur.

• Converted variable 'symptoms' to a dynamically allocated string in list.c is_mail_loop() and in similar places in listproc.c; we found cases of too many error reports that caused memory overflow and crash. We now report unique error findings only.

• Fixed a bug where requests spanning multiple lines separated by the & continuation character were not being properly saved in the batch queue.

• Fixed bug with sending a partial digest when a user changed mail mode from DIGEST to ACK or NOACK; when the same list was distributing its regular digest at the same time havoc could ensue.

• queued now requires both semaphore ids as arguments; it was a bug in 8.0 to require only one.

• Under 8.0 some pqueue processes would not place the outgoing mail in mqueue/ if delivery failed and would toss the file entirely; this was due to the missing gatesid that was not passed to them. This is fixed in 8.1 by fixing list.c and queued (see above).

• Fixed a long lasting bug with sorting of subscribers files; we were not handling subscriber names separated with tabs correctly; this resulted in internal sorting errors.

• Fixed bug with SET request where if more params were specified and 'option relaxed_syntax' was set in LPDIR/config, we neglected to extract only the information we were after and toss the remainder; we used the whole string with the bogus extra parameters, thus corrupting the subscribers file.

• Found and fixed file descriptor leak in mmap_owner_listed(). This leak could potentially bring a system down when the system limit of open files was reached.

• There are some cases in the code when a string initialized with strncpy() was not terminated with \0. That could cause core dumps.

• Fixed two more core-dumping problems with the SYSTEM request (new with 8.1) and a special error condition with list: while distributing the digest if an invalid address were encountered core was dumped.

• Fixed core dump with ARCHIVE ... REVIEW of non-list related archives, issued by archive owners.

• Fixed bug with zero-byte allocation in farch.c.

• Fixed core dump with list and listproc due to bad treatment of pointers after realloc() -- realloc() may move the block and we were not accounting for it in some cases.

• Fixed bug with the load-on-demand stuff -- unix_cmd directives were not properly loaded.

• Fixed bug in config_owner_prefs (was modified in 8.1) -- it caused a memory leak and a segmentation violation after repeated INIT requests.

• Revised signal handling for SIGCHLD in serverd.c and list.c after proving that signals can get lost; the result was zombie processes, and infinite wait by parents for processes to exit. This turned up as more threads were being defined by various sites.

• Fixed bug in ILP parser, to correctly handle extra spaces in archive requests.

• Now save unprocessed server messages when shutting down.

• Fixed core-dumping bug with mail delivery in 8.0 with all mail methods asside from system.

  To top of page