==== 20070914 ==== Bugs fixed: - namespace bug: lockset was not initialized when opening a maildir() file. - conformance change to sieve "deleteheader" Improvements: - Work to make mfl literal quoted strings able to contain NUL bytes. - mvmtr control system has been reworked, to great advantage. - Some of the recent advancements in mvmtr controls were removed. - Add RISETs to mvmtr, to aid in reporting/logging of factors contributing to decisions. - Add standard SMTP reply template, with some customization via controls. - Many other mvmtr improvements, some even noted in the ChangeLog :) ==== 20070405 ==== Bugs fixed: - MFL portability problem caused by misuse of tolower() et al; - sieve parsing bug when MFL function formal argument has same name as a sieve tagged option. Improvements: - A bunch of ongoing and un-noted stuff in mvmcas; - Improve the way that some checks in mvmtr are specified and controlled; - Change mvmtr "connect_check" to "client_check" and add a different sort of "connect check"; - Add mail_check checks in mvmtr; - In mvmtr, detect attempt to send stuff after DATA response; - Various other mvmtr improvements; - Rename the "docs" subdirectory to "notes"; - Various document/sample/reference updates. ==== 20070108 ==== Bugs fixed: - sieve language tokens are supposed to be case insensitive, but somehow were not implemented that way; - on the other hand, sieve capabilities names are supposed to be case sensitive, but were implemented case insensitive; - A fix to pre-decrement and pre-increment of string values (who wants that anyway??); - Portability issues with code that tried to find local addresses (for mvmtr, mainly); - Fixes to parsing of mailbox names in mvmtr, there were some minor issues; - Bug in the controls interface where one control name was an initial substring of another, and they were defined in a particular order; - Bug in $env_variable_assign() MFL built-in-function, it wasn't returning the right status when environment variable was empty. Improvements: - Improvement to interfaces of some of the hooks in mvmtr; - Added some built-in helo and connection checking in mvmtr; - In mvmda, attempt to notice if setgid, and add that group to the group list when becoming the real user (if started as root); - Better process sanitization when forking (needed if setgid); - other mvmtr/mvmcas-related stuff; - Add "tailhook" hook option when defining namespaces, to let you rewrite the tail portion of a foldername; - Rearrange the "files" subdirectory in the source tree, so as to better be able to find installion-related files and examples; - Add some things to the files subdirectory. ==== 20061113 ==== Some more big changes. With the big change to mvmda last time, where a lot of its flow was removed from the program itself and handed over to the system-wide MFL script, I had some concern that there would now be no standard and predictable way that mvmda operates from one system to another (each system administrator getting to decide various things about the control flow). I added a new mvmda option, "-b", to force the use of a built-in MFL script instead of the system-wide one. However, the administrator still has to allow this to be used, by using a new configuration switch --mvmda-builtin-sysrc-forceable . I think this is a reasonable compromise. The 'msgstore' module has been torn apart and message storing reworked. msgstore used to know about all aspects of message filing. Message storing is now implemented in terms of specific message store type modules, which in turn can make use of general types (though "general" types are only known to the specific types and have no further public interface). The two specific message types are "mbox" and "maildir" and they are internally related through a general type "file". The concept of message store namespaces has been added. When writing a message to a folder (e.g. via sieve's "fileinto") the folder name is decoded via a matching namespace. The namespace is determined by folder name's initial substring: the longest match against defined namespaces selects the namespace. So one might have a namespace "INBOX" that matches the specific /var/mail/user mailbox (if that's what you want), and another namespace "INBOX." that refers to maildir folders inside a user's home directory. Namespaces MUST be defined before use: I expect that for most sites this will simply be done inside the system-wide MFL script. This initial implementation of namespaces is reasonably functional; I have some other enhancements in mind that may or may not come to pass. mvmda command-line arguments -l, -r, and -s were formerly used to essentially hardwire a single namespace. These arguments now do nothing, as now you can have multiple namespaces. Similarly, the old controls "lock_mbox" and "lock_maildir" have been removed, as those are now attributes on storage types and namespaces derived from those storage types. mvmda now has a "-d" argument to specify a user to deliver to (option chosen to be compatible with other MDAs). If you specify a user with this option, mvmda will attempt to become that user if it's started as root, and will in any case refuse to run if it is not the user specified. Plus other things as mentioned in the ChangeLog. The major things mentioned above are likely to be enhanced; this release is to roll them out. Note well that there have been pretty big changes in philosophy and in implementation represented in both this release and the last. I believe it's all for the better, but watch out for changes in operation and, although I've done some testing on multiple systems, for bugs that may have crept in. ==== 20060821 ==== [ Lots of changes in this release. The most significant is with mvmda, and its delegation of much of its former built-in responsibility to the system-wide startup file. This puts a heavier burden on the installer (to create or at least install a system-wide rc script), but on the upside gives a lot more flexibility and control to the system administrator (which is, after all, one of the stated goals of mvmda). Since there are so many changes, please let me know of new problems. There are also documentation updates that go with the above change. As always, I'd like to know if there are documentation errors or things that could be readily improved. ] See ChangeLog for gory details, although this overview is gory enough. Bugs fixed: - sieve's "vacation" statement wasn't requiring a semicolon. - several built-in functions were found to misinterpret their string arguments; these broken functions were using a string thru their NUL-termination, rather than using the length of the string. While you wouldn't really have noticed unless you are using those functions in combination with substrings and string pointers, this was a serious set of errors. Significant changes: - In mvmda, much built-in logic has been removed and delegated to the system-wide startup file. This startup file is now responsible for things like opening the incoming message, setting up paths for @include and hook searces, executing a user's rc file (if wanted), and running scripts specified by the user on the command line. See 'files/' directory for sample (albeit rudimentary) startup and other files. - In mvmda, the default path to the system-wide rc file has changed from mvmf_libdir/mvmda-rc.mfl to mvmf-libdir/mvmda/mvmda-rc.mfl (i.e., down into the 'mvmda' subdirectory). This is where it really ought to be, plus moving it makes it a little easier to upgrade to the new style of rc file without disrupting the old. - In mvmda, do nothing with any command-line arguments beyond the options. Instead, provide them to the startup file via MFL variables. - In mvmda, remove all hard-coded default definitions for MFL hook functions. All hooks now have to be provided externally and won't be defaulted. - In mvmda, remove -b command-line argument, which specified bodycache size. I doubt that anybody is using this; it's now something that can be specified by the rc file when it opens the message. - In mvmda, remove the :rdmail command since we now have $msg_open_fname(). - Built-in function $mfl_exec_string has been renamed to $mfl_run_string, with a second argument. $mfl_exec_string_sieve has been removed altogether. If you make any use of $mfl_exec_string, you could temporarily use a @define somewhere in your startup processing to conceal the change, e.g.: @ifndef_func $mfl_exec_string @ define $mfl_exec_string(x) $mfl_run_string( x, "Mc" ) @endif Improvements & additions: - Added on-demand searching for MFL hook functions. If an MFL hook function is needed and not found, it will be looked for in files along an MFL hook directory path. - Added built-in functions $mfl_run_fname, $mfl_protect_admin, $mfl_hookdir_add, $msg_open_fname - Add MFL built-in function module for mvmda, with one MFL built-in function: $mvmda_run_fname. This is a hack to support very old script files that might use :xxx style commands. I doubt anybody but me has any users that do this, but I need it, so there you go. - In mvmda, add '-e' and '-E' options, whose arguments are given to the startup script via MFL variables. - In mvmda, provide the operating mode as the value of an MFL string variable $OPMODE. - In mvmda, provide environment variables MVMF_LIBDIR and MVMF_SYSDIR to be the build-time configured paths to the shared and system data directories. - Add mfl control "mfl_mode_sieve". If 0, and if not specifically overridden when calling script-running functions, the default parsing of scripts starts in C-like mode; if 1, default parsing starts in sieve mode. - In mvmda, -S option now just sets the mfl_mode_sieve control to 1. - Add configuration option --enable-mvmda-builtin-sysrc -- this can be enabled to hardwire a default system-wide rc script if no rc file is found. Not really recommended, but might help with the move to this new style. - Add mvmda-sysrc.5 man page for mvmda system-wide rc file discussion. - Make a separate INSTALL.mvmda file - In files/ directory, rewrite the sample system-mvmda-rc.mfl - In files/ directory, add a mvmda-hooks/ subdirectory with some sample hooks definitions. - Some generic code cleanup/changes. ==== 20060718 ==== Bugfix: an uninitialized pointer in the new(ish) controls code can cause application (like mvmda) coredumps on some systems- often upon startup. ==== 20060630 ==== Bugfix: bug in msg_bounce() introduced back in April, that would cause coredumps when 'reject' or other bounce invocations are done. ==== 20060629 ==== Bugs fixed: - @include would operate even if it was inside a false preprocessor conditional!; - bug picking up the ":addresses" tagged argument to 'vacation' statement. The bug would likely keep 'vacation' from working, or make it blow up, in many circumstances. Improvements & additions: - Better handling (and detection) of errors when reading mail message. Affected mainly SMTP message input (i.e. DATA command) but the improvements will help all. A lot of this involved changes in mml, rather than mvmf, but some mvmf code had to change to accomodate mml interface changes; - Added some rudimentary MFL trace functions, to help debugging scripts; - Add "C" keyword to C mode, and "sieve" keyword to sieve mode. With this, C {} and sieve {} blocks can be used anywhere, without having to know the invoking context. In fact I would recommend using these for at least library include files; - Various tinkering with mvmtr and mvmcas. ==== 20060510 ==== Bugs fixed: - Fix a test that would allow a "From " (i.e. From_, the mbox separator, not "From:") header field to appear anywhere in a header; - Make sure MIME type/subtype is defaulted correctly. Although this may not seem connected, this allows junk that doesn't look like a mail message to be handled without failing (i.e. coredumping); - If a header is required when writing a MIME part, make a fake one; - A fix that might cause dynamic execution of MFL scripts at runtime (such as MFL hooks) to confuse the parser. The problem was re-use of a pushed-back token from another context; - A variable, relating to peeking ahead at lines, should have been an int but was a char. This would cause errors in long lines in some contexts: previously unlikely, but the likelihood was significantly increased with the new handling of message/rfc822 parts. - Fix improper dot-unstuffing in mvmtr: a dot followed by any non-line- terminator should be removed, previously only a dot followed by another dot would be; - A couple of control-related functions were using int values instead of longs. New & Improved: - Some new man pages for clamdif, cdbgen, dnsblc, mvmdalogsum; - message/rfc822 MIME parts now treat their body as an implicit child; This simplifies access to the header section of the body, and mimics what IMAP does with message/rfc822; - Added an mvmtr control to delay after bad RCPT-TO; - Improved internal handling of recipient lists in mvmtr; - Improve some Makefiles for pthread inclusion or exclusion; - Improve the functions that handle controls (control strings); - Add '-V' command-line options to mvmda and mvmcas, to dump the control settings in an mfl-readable format; - In mvmtr, add some controls relating to whether to adhere to strict SMTP session rules (i.e., waiting for the client to QUIT instead of dropping the connection); - Finalize the initial greylisting support in mvmtr; - Add message size limit in mvmtr; - Considerable work on mvmcas, but we don't talk about that yet. And other minor things noted in the ChangeLog. ==== 20060420 ==== Mainly because of a fix of a critical MIME bug introduced last edit. Bugs fixed: - Problems with body and epilog MIME parsing in some circumstances. - Minor initialization fix in mvmtr - Sieve "size" had broken quite a while ago, now it's fixed. New stuff: - New built-in MFL functions $msgpart_epilog, $msgpart_cur_size_body, $msgpar_cur_size_hdr, $str_quote_regex, $str_quote_wild - relational sieve "header" test now returns the count of headers if doing so doesn't conflict with the true/false result, just for grins. ==== 20060410 ==== New kit today for two main reasons: the improvements in MIME handling, and the desire to make the kit match the web page. Bug fixed: Any MIME "epilog" part of a message was previously being ignored. The epilog is the part following any MIME children, following the final MIME boundary up to the logical end of the containing message part. You don't see epilogs very much, and when you do, it's usually some broken MIME. mvmf now records any epilog parts and emits them properly. This should also allow variously broken MIME to be accepted in some way. Another bug: in mvmda, qmail mode (the '-q' flag) would be forced on if maildir storage was being used. This was wrong. The only effect of qmail mode is to provide a default "$hook_exitcode()" MFL function if none was provided-- this too will likely be removed at some point, since you can just as easily put it in a startup file. Improvements: "From " line quoting is now only done for mbox storage, whereas formerly it was always done. Similarly, a final blank line is only added for mbox storage, whereas it too used to always be done. And some other minor stuff, see the ChangeLog. Addition: I now consider the experimental mvmtr program to be a keeper, and will start recording major changes and will attempt to add its level of documentation. Currently it's only of interest to qmail users; this could change at some point. ==== 20050329 ==== With the last release I had broken some of the DNS stuff (the dnsbl statement in sieve, the $dns_query() MFL function). So this is a needed release. There have also been a bunch of changes in some of the experimental server code (mvmtr, mvmcas) but most people will ignore that. ==== 20050317 ==== The main news is a fix in a bug that affected the parsing of the MIME boundary string in the mail message header. ==== 20050127 ==== Bugs fixed: Some of the C code executes MFL functions (or defines MFL hooks), and had the '#' preprocessor character hardwired in. If you didn't --enable-preprocessor-via-hashmark during configure, this wouldn't work at all. Change to use the primary preprocessor character as chosen during configure. Also, a fix for MFL functions that return complex types. There's a new hook, $hook_exitcode(code) which you can supply to translate the internal exit code to something you'd rather see. The -q option (qmail mode) now supplies a default $hook_exitcode() hook instead of using a hardwired translation. To go along with this, MFL defines some MFL preprocessor symbols for exit code values: $MEX_OK, $MEX_HARD, $MEX_SOFT, and $MEX_OKCONT. If qmail support is compiled in, it also defines the exit codes that qmail likes to see, as $QMEX_OK, $QMEX_STOP, $QMEX_HARD, and $QMEX_SOFT . The --enable-faster-tokback option has been removed, and the old tokback code along with it. "faster tokback" is now the default. Other enhancements include better internal handling of controls (the $control_xxx() suite of functions), addition of a "email_primary_address" control, and new controls that let you specify the type and order of locking to apply when writing mail messages. ==== 20050117 ==== faster-tokback (for mfl language tokenizing) is now the default; this option (and the non-faster-tokback code) will be removed after this. A very minor bugfix in parsing 'continue' and 'break' statements- the bug made the ';' optional, which could be problematic inside if/then/else . Several new extremely useful built-in functions, including the ability to execute mfl code out of a string (which in turn makes it feasible to load functions on demand at runtime, instead of always loading/compiling functions that might rarely be needed). An improvement to cdbgen. Minor changes to accomodate the addition of some pthreads support in mml. Improved handling of @include; both the "" and <> forms now have a directory search list. ==== 20050103 ==== MFL now has a rudimentary concept of external plug-in programs, which are called CUSPs (commonly used service programs). The CUSP idea has been brewing for a long time, and this is an initial stab at the idea. This first-pass will allow some testing and feedback and more thought about how the cusps can be administered and configured. Also included is a sample CUSP called "clamdif" which is an interface to the clamav antivirus clamd daemon. (see the "cusp" subdirectory.) This is also a first-pass effort that is likely to evolve; I have, however, been using it privately for a short while with success. The CUSP stuff is new, certainly in need of embellishment, and probably in need of further debugging. The main reason to push this out now is that misc/msgpart.c didn't compile well on some systems as it was missing an #include of ; lesser motivations are the rolling in of the new year, and the fact that I've already documented $cusp functions on the web site and need to package up the code that matches the documentation -- not to mention that I am now interested in moving on to work on the mail client assessment daemon (perhaps "mvmcas"). ==== 20051222 ==== Sidetracked before moving onto new stuff. Bugs found and fixed, some significant ones including a problem in $str_debug, a problem in $cdb_get_string functions, a really old bug in "for" and "while" when no loops were executed, a bug in post-increment and post-decrement(!), a bug in preprocessor macro argument interpretation, and a bug when skipping false preprocessor conditionals. A number of new things added: ability to add to string pointers, ability to use C statements as expression terms (if enabled at configuration time), ability to use compound statements as expression terms, a whole slew of new built-in-functions, mostly $str_xxx but some others as well. ==== 20051214 ==== Some minor new $str_ and $msg_rip_ MFL functions, and a new set of $dns_ functions, and a function $ip_rev. Also a bugfix in setting envelope to/from strings, another edge case bugfix in token pool handling when FASTER_TOKBACK is enabled (which should be the default soon), and a really obscure bug also related to FASTER_TOKBACK. Also, a bug had crept into msgpart_hdrtoken(), preventing recognition of MIME structures. Quite a bit of attention to mvmtr, but nothing to write about yet. This is going out since I need some of the new stuff in a real use situation, and to have a checkpoint before some other new stuff. ==== 20051002 ==== A fix to two of the $cdb functions added last time, and some misc tinkering to existing code. ChangeLog has details. ==== 20050930 ==== Mainly a bugfix release- a bug in the mml library was failing to store the result of "?" matches as a matched string, e.g. available via mfl's $str_match() . So for example if you had a message with a subject of "I ate 3 snakes" and you did: sieve { header :matches "Subject" "* ate ? *" } submatch result #2 should contain "3" . Prior to this fix, result #2 would contain "snakes" and the substring matching "?" wasn't remembered. This release also adds the "db" built-in-function module to mfl, with the functions $cdb_open, $cdb_close, $cdb_get_string, and $cdbu_get_string . These can be used to query a CDB indexed file. ==== 20050824 ==== This release adds support for Sieve's "text:" multi-line text block, finally; it never seemed very important since multi-line quoted strings worked just fine. But it's now there for completeness. Also added is support for to-the-end-of-line comments beginning with '#'. MFL has always had the '//' equivalent, but with people wanting to use pure Sieve mode, I broke down and added this. The '#' style comments conflict with the use of '#' as a preprocessor introducer character. I've changed MFL to use '@' as a preprocessor character instead. If you need to retain the use of '#' for the preprocessor character, you can enable it via a configuration option. Also added is a configure switch to suppress warnings when an unknown preprocessor directive is encountered; this can help make '#'-style comments mix with '#'-style preprocessor directives. For new users, the recommendation is to use '@' for preprocessor and keep warnings enabled. The "dnsbl" Sieve capability name has been renamed to "vnd.mvmf.dnsbl"; this should always have been a vendor-specific name. My error. There's a configure switch to keep the "dnsbl" capability name as an alias. New users should probably not enable it. A "-n" command line flag has been added to mvmda, which will run a script in "test" mode, where mvmda will report the actions that it would have taken rather than actually taking the actions. ==== 20050821 ==== mvmda (actually MFL) now contains the initial implementation of the Sieve "vacation" statement. This implementation is still considered first-pass, and should be watched with care if used. It has some variance with the Sieve vacation draft: This implementation requires a ":from" tagged argument, as I don't want to try to guess the script owner's email address. The draft says that if a :handle is not given, one must be synthesized based on the other arguments. I haven't done this; rather, I assume a handle of "default" if one is omitted. The vacation history is tracked via a database file using the "ndbm" library. The configure script attempts to find either the ndbm library or the gdbm emulation of it. I'd add support for other dbm libraries if given enough information or access to them. The databases for mvmda are kept in a db file in a ".mvmda" directory in the user's homedir. Each database file is named "vacation-[handle]" (where "[handle]" is replaced by the actual :handle used). The location and name of the data directory (~/.mvmda) are likely to be made more flexible in the future (e.g. if you want to put them inside a Maildir, or other places for people who don't have homedirs). And in fact you can change the path to this data directory by using an MFL control function, perhaps in your .mvmda-rc startup file, e.g.: $control_string_set( "data_directory", "/path/to/.mvmda"); The sample file files/mvuser.mfl has been updated with an example of the vacation statement. Also see docs/draft-ietf-sieve-vacation-02.txt for a version of the draft spec.