New Features in Exim
--------------------

This file contains descriptions of new features that have been added to Exim,
but have not yet made it into the main manual (which is most conveniently
updated when there is a relatively large batch of changes). The doc/ChangeLog
file contains a listing of all changes, including bug fixes.


Version 4.21
------------

 1. There is a new log selector called smtp_incomplete_transaction. When a mail
    transaction is aborted by RSET, QUIT, loss of connection, or otherwise, the
    incident is logged, and the message sender plus any accepted recipients are
    included in the log line. This can provide evidence of dictionary attacks.

 2. If the option tls_remember_esmtp is set true, Exim violates the RFCs by
    remembering that it is in "esmtp" state after successfully negotiating a
    TLS session. This provides support for broken clients that fail to send a
    new EHLO after starting a TLS session.

 3. When an ACL is read dynamically from a file (or anywhere else), the lines
    are now processed in the same way as lines in the Exim configuration file.
    In particular, continuation lines are supported, which was not the case
    previously.

 4. The facility for constraining the result of dnslists lookups to certain IP
    addresses has been extended. The original facility uses a positive list,
    like this:

      deny  dnslists = a.b.c=127.0.0.2,127.0.0.3

    This example means "deny if the host is in the black list at the domain
    a.b.c and the IP address yielded by the list is 127.0.0.2 or 127.0.0.3". It
    is now possible to supply a negative list, like this:

      deny  dnslists = a.b.c!=127.0.0.2,127.0.0.3

    This example means "deny if the host is in the black list at the domain
    a.b.c and the IP address yielded by the list is not 127.0.0.2 and not
    127.0.0.3". In other words, the result of the test is inverted.

    Note that this kind of negation is not the same as negation in a domain,
    host or address list (which is why the syntax is different).

    If you are using just a single list, the new syntax does not gain you much.
    The previous example is precisely equivalent to

      deny  dnslists = a.b.c
           !dnslists = a.b.c=127.0.0.2,127.0.0.3

    However, if you are using multiple lists, the new syntax is clearer.
    Consider this example:

      deny  dnslists = sbl.spamhaus.org : \
                       list.dsbl.org : \
                       dnsbl.njabl.org!=127.0.0.3 : \
                       relays.ordb.org

    Using only positive lists, this would have to be

      deny  dnslists = sbl.spamhaus.org : \
                       list.dsbl.org
      deny  dnslists = dnsbl.njabl.org
           !dnslists = dnsbl.njabl.org=127.0.0.3
      deny  dnslists = relays.ordb.org

    which is less clear, and harder to maintain.

 5. The command line option -ti is equivalent to -t -i (this is for Sendmail
    compabitility).

 6. Sendmail's -Ooption=value command line option is now ignored.

 7. When execve() failed while trying to run a command in a pipe transport,
    Exim was returning EX_UNAVAILBLE (69) from the subprocess. However, this
    could be confused with a return value of 69 from the command itself. This
    has been changed to 127, the value the shell returns if it is asked to run
    a non-existent command. The wording for the related log line suggests a
    non-existent command as the problem.

 8. If received_header_text expands to an empty string, no Received: header
    line is added to the message.

 9. eximstats 1.82 has a new option -nt that allows certain transports to be
    omitted. This is useful for omitting spamassassin transport (for example)
    because otherwise each message gets counted twice.

10. If a DNS black list returns more than one A record, all the IP addresses
    are now included in the value of the $dnslist_value variable, comma-space
    separated.

11. The idea of the "warn" ACL verb is that it adds a header or writes to the
    log only when "message" or "log_message" are set. However, if one of the
    conditions was a negated address verification or a call to a nested ACL,
    the messages generated by the failing underlying test were being passed
    through. This no longer happens. The underlying message is available in
    $acl_verify_ message for both "message" and "log_message" expansions, so it
    can be passed through if needed.

12. This isn't a new feature: it's a note of a bug in the manual. In the
    description of $domain in section 11.8, it says "Whenever a domain list is
    being scanned, $domain contains the subject domain." This statement is NOT
    true when the domain list in a "sender_domains" condition in an ACL is
    being processed. It works this way so that, in a RCPT ACL, the
    sender_domains list can be dependent on the recipient domain (which is what
    is in $domain) if required. The sender's domain is in
    $sender_address_domain.

13. The way that the $h_ (and $header_) expansions work has been changed by the
    addition of RFC 2047 decoding. There are now three forms:

      $rh_xxx: and $rheader_xxx: give the original content of the header
        line(s), with no processing at all.

      $bh_xxx: and $bheader_xxx: remove leading and trailing white space, and
        then decode base64 or quoted-printable "words" within the header text,
        but do not do charset translation. If decoding of what looks
        superficially like a "word" fails, the raw string is returned. If
        decoding produces a binary zero character, it is replaced by a question
        mark - this is what Exim does for binary zeros that are actually
        received in header lines.

      $h_xxx: and $header_xxx: attempt to translate the $bh_ string to a
        standard character set. If this translation fails, the $bh_ string is
        returned. Translation is attempted only on operating systems that have
        the iconv() function. This is indicated by the compile-time macro
        HAVE_ICONV.

    In a filter file, the target charset can be specified by a command of the
    following form:

      headers charset "UTF-8"

    This command affects all references to $h_ (or $header_) in subsequently
    obeyed commands. In the absence of this command, the target charset in a
    filter is taken from the setting of the headers_charset option in the
    runtime configuration. The value of this option defaults to the value of
    HEADERS_CHARSET in Local/Makefile. The ultimate default is "ISO-8859-1".

    Some of the operating systems that supply iconv() do not support very many
    conversions. The GNU libiconv library (available from
    http://www.gnu.org/software/libiconv/) can be installed on such systems to
    remedy this deficiency, as well as on systems that do not supply iconv() at
    all. After installing libiconv, you should add HAVE_ICONV=yes to your
    Local/Makefile.

14. There is now support for Sieve filters (RFC 3028). Initial documentation
    is in doc/README.SIEVE. The RFC itself is also provided in the doc
    directory. Note that you cannot use a Sieve filter as a system filter,
    only as a per-recipient filter. The reason for this is that Sieve filters
    always specify what is to be done with a message (redirect, save, or
    discard). There is no mechanism for specifying "carry on and do the normal
    delivery".

15. The facility for testing for specific IP addresses in the yields of DNS
    black list lookups has been extended. The existing facility is to look for
    specific addresses by settings such as

      dnslists = a.b.c=127.0.0.1,127.0.0.2

    If the character '&' is used instead of '=', the comparison for each listed
    IP address is done by a bitwise 'and' instead of by an equality test. In
    other words, the listed addresses are used as bit masks. The comparison is
    true if all the bits in the mask are present in the address that is being
    tested. For example,

      dnslists = a.b.c&0.0.0.3

    matches if the address is x.x.x.3, x.x.x.7, x.x.x.15, etc. If you want to
    test whether one bit or another bit is present (as opposed to both being
    present), you must use multiple values. For example:

      dnslists = a.b.c&0.0.0.1,0.0.0.2

    matches if the final component of the address is an odd number or two times
    an odd number.

16. The smtp_printf() function is now available for (careful) use in
    local_scan() functions. Its arguments are like printf(), but it writes to
    the SMTP output stream. You should use this function only when there *is*
    an SMTP output stream, that is, when the incoming message has arrived via
    interactive SMTP. Two boolean global variables are available to enable you
    to test for this case:

      smtp_input           is TRUE for all SMTP input
      smtp_batched_input   is TRUE for BSMTP input

    Thus, the use of smtp_printf() should be restricted to cases where
    smtp_input is TRUE and smtp_batched_input is FALSE. If you want to test for
    an incoming message from another host, you can test the value of
    sender_host_address, which is non-NULL in this case.

    If an SMTP TLS connection is established, smtp_printf() uses the TLS output
    function, so it can be used for all forms of SMTP connection.

    The strings that are written by smtp_printf() from local_scan() must start
    with an appropriate response code: 550 if you are going to return
    LOCAL_SCAN_REJECT, 451 if you are going to return LOCAL_SCAN_TEMPREJECT,
    and 250 otherwise. Since you are writing the initial lines of a multi-
    line response, the code must be followed by a hyphen to indicate that the
    line is not the final response line. You must also ensure that the lines
    you write terminate with CRLF. For example:

      smtp_printf("550-this is some extra info\r\n");

    Note that you can also create multi-line responses by including newlines in
    the data returned via the return_text argument. The added value of using
    smtp_printf() is that, for instance, you could introduce delays between
    each output line.

    The smtp_printf() function does not return any error indication, because it
    does not automatically flush pending output, and therefore does not test
    the state of the stream. (In the main code of Exim, flushing and error
    detection is done when Exim is ready for the next SMTP input command.) If
    you want to flush the output and check for an error (for example, the
    dropping of a TCP/IP connection), you can call smtp_fflush(), which has no
    arguments. It flushes the output stream, and returns a non-zero value if
    there is an error.

17. The smtp transport has a new option called hosts_avoid_esmtp. This is for
    use with broken hosts that announce ESMTP facilities (for example,
    PIPELINING) and then fail to implement them properly. When a host matches
    hosts_avoid_esmtp, Exim sends HELO rather than EHLO at the start of the
    SMTP session. This means that it cannot use any of the ESMTP facilities
    such as AUTH, PIPELINING, SIZE, and STARTTLS.

18. Following a discussion on the list, the rules by which Exim recognises line
    endings on incoming messages have been changed. The -dropcr and drop_cr
    options are now no-ops, retained only for backwards compatibility. The
    following line terminators are recognized: LF CRLF CR. However, special
    processing applies to CR:

    (i)  The sequence CR . CR does *not* terminate an incoming SMTP message,
         nor a local message in the state where . is a terminator.

    (ii) If a bare CR is encountered in a header line, an extra space is added
         after the line terminator so as not to end the header. The reasoning
         behind this is that bare CRs in header lines are most likely either
         to be mistakes, or people trying to play silly games.

19. A new feature has been added to the processing of lists of hosts by the
    manualroute and queryprogram routers. An item in those lists can now take
    the form of a domain name followed by "/MX". This notation is an indirect
    way of specifying a list of hosts. For example, in a manualroute router you
    could have

      route_list = *  a.b.c/MX

    or

      route_list = *  x.y.z:p.q.r/MX:e.f.g

    If the hosts_randomize option is set, the order of the items in the list is
    randomized exactly as before. Then, when Exim is scanning the list to find
    an IP address for each host, items that end with "/MX" are replaced with
    the list of host names obtained by looking up MX records for the name. The
    order is determined by the preference values in the MX records, according
    to the usual rules. Because randomizing happens first, it does not affect
    the order that is defined by MX preferences.

    If the local host is present in the list obtained from MX records, but is
    not the most preferred host in that list, it and any equally or less
    preferred hosts are removed before the list is inserted.

    If the local host is the most preferred host in the MX list, what happens
    depends on where in the original list of hosts the "/MX" item appears. If
    it is not the first item (i.e. there are previous hosts), Exim just
    discards this item and any subsequent items in the list, just as it does
    for a non-MX item that turns out to be the local host.

    If the MX item is first in the list of hosts, and the local host is the
    most preferred host, what happens is controlled by the "self" option of the
    router.

    DNS failures are treated in the same way as DNS failures when looking up IP
    addresses: pass_on_timeout and host_find_failed are used when relevant.

20. There's a new generic transport option called rcpt_include_affixes. Its
    default value is FALSE. In this state, when an address that has had any
    affixes (prefixes or suffixes) removed from the local part is delivered by
    any form of SMTP or LMTP, the affixes are not included. For example, if a
    router that contains

      local_part_prefix = *-

    routes the address abc-xyz@some.domain to an SMTP transport, the envelope
    is delivered with

      RCPT TO:<xyz@some.domain>

    However, if rcpt_include_affixes is set TRUE, the whole local part is
    included in the RCPT command. This option applies to BSMTP deliveries by
    the appendfile and pipe transports as well as to the lmtp and smtp
    transports.

21. By default, the command line options -odi etc. override the setting of
    queue_only in the configuration file. This is now controlled by the
    queue_only_override option, which defaults true. If it is set false, the
    -od options cannot be used to override queue_only or queue_only_file.

22. When Exim is logging to syslog, it writes the log lines for its three
    separate logs at different syslog priorities so that they can in principle
    be separated on the logging hosts. Some installations do not require this
    separation, and in those cases, the duplication of certain log lines is a
    nuisance. If syslog_duplication is set false, only one copy of any
    particular log line is written to syslog. For lines that normally go to
    both the main log and the reject log, the reject log version (possibly
    containing message header lines) is written, at LOG_NOTICE priority.
    Lines that normally go to both the main and the panic log are written at
    the LOG_ALERT priority.

23. Formerly, the value of the AUTH= option on a MAIL command was trusted by
    an Exim server only if the client host had authenticated. An ACL can now be
    used as an alternative way of control this. The way that Exim handles an
    AUTH= option on a MAIL command is now as follows:

    . If the connection is not using extended SMTP (i.e. HELO was used), the
      use of AUTH= is a syntax error.

    . If the value of the AUTH= parameter is "<>" it is ignored.

    . If acl_smtp_mailauth is defined, this ACL is run. While it is running,
      the value of $authenticated_sender is set to the value obtained from the
      AUTH= parameter. If the ACL does not yield "accept", the value of
      $authenticated_sender is deleted.

    . If acl_smtp_mailauth is not defined, the value of the AUTH= parameter is
      ignored if the client has not authenticated.

    . If the AUTH= value was accepted by either of the two previous rules, and
      the client has authenticated, and the authenticator has a setting for the
      server_mail_auth_condition, the condition is checked. If it fails, the
      value of $authenticated_sender is deleted.

    . The ACL for MAIL, if defined, is run after AUTH= is accepted or ignored.
      It can make use of $authenticated_sender. The converse is not true: the
      value of $sender_address is not yet set up when the "mailauth" ACL is
      run.

    Whenever an AUTH= value is ignored, the incident is logged. Previously,
    logging happened only when server_mail_auth_condition testing failed.

    The acl_smtp_mailauth ACL may not return "drop" or "discard". If it defers,
    a temporary error code (451) is given for the MAIL command.

24. By default, callouts do not happen when testing with -bh. There is now a
    variant, -bhc, which does actually run the callout code, including
    consulting and updating the callout cache.

25. Exim now has support for Cyrus saslauthd authentication. This works in a
    similar way to Cyrus pwcheck authentication (which is now deprecated). To
    build Exim with saslauthd support, set CYRUS_SASLAUTHD_SOCKET in
    Local/Makefile. This must define the location of the socket that is used to
    communicate with the saslauthd daemon. For example:

      CYRUS_SASLAUTHD_SOCKET=/var/state/saslauthd/mux

    The daemon is called by a new expansion condition:

      ${if saslauthd{{user}{password}{service}{realm}}{yes-string}{no-string}}

    Up to four arguments can be supplied, but only two are required (service
    and realm are optional). That is why the arguments are enclosed in a set of
    braces. For details of the meaning of service and realm, and how to run the
    saslauthd daemon, consult the Cyrus documentation.

26. The way daemon_smtp_port, local_interfaces, and -oX interact has been
    extended, compatibily I hope. A new option called extra_local_interfaces
    has been added. Here is a full description of all of this, which will
    eventually be the basis of a new chapter in the manual.


              EXIM'S USE OF NETWORK INTERFACES

    A host that is connected to a TCP/IP network may have one or more physical
    hardware interfaces. Each of these interfaces may be configured as one or
    more "logical" interfaces, which are the entities that a program actually
    works with. Each of these logical interfaces is associated with an IP
    address. In addition, TCP/IP software supports "loopback" interfaces
    (127.0.0.1 in IPv4 and ::1 in IPv6), which do not use any physical
    hardware.

    Exim uses knowledge about the host's interfaces in three different ways:

    (1) When a listening daemon is started, Exim needs to know which interfaces
        and ports to listen on.

    (2) When Exim is routing an address, it needs to know which IP addresses
        are associated with local interfaces. This is required for the correct
        processing of MX lists by removing the local host and others with the
        same or higher priority values. Also, Exim needs to detect cases
        when an address is routed to an IP address that in fact belongs to the
        local host. Unless the "self" router option or the "allow_localhost"
        option of the smtp transport is set (as appropriate), this is treated
        as an error situation.

    (3) When Exim connects to a remote host, it may need to know which
        interface to use for the outgoing connection.

    Exim's default behaviour is likely to be appropriate is the vast majority
    of cases. If your host has only one interface, and you want all its IP
    addresses to be treated in the same way, and you are using only the
    standard SMTP port, you should not need to take any special action. The
    rest of this chapter does not apply to you.

    However, in more complicated situations, you may want to change the way
    Exim behaves, and for this reason there are a number of options that can be
    used to influence Exim's behaviour. The rest of this chapter describes how
    they operate.


    STARTING A LISTENING DAEMON

    When a listening daemon is started (by means of the -bd command line
    option), the interfaces and ports on which it listens are controlled by the
    following two options:

    . daemon_smtp_ports contains a list of default ports. (For backward
      compatibility, this option can also be specified in the singular.)

    . local_interfaces contains list of interface IP addresses on which to
      listen.

    The default list separator in both cases is a colon, but this can be
    changed as described in section 6.15. When IPv6 addresses are involved, it
    is usually best to change the separator to avoid having to double all the
    colons in IPv6 addresses. For example:

      local_interfaces = <; 127.0.0.1 ; \
                            192.168.23.65 ; \
                            ::1 ; \
                            3ffe:ffff:836f::fe86:a061

    A port number can be specified along with each IP address in
    local_interfaces. Two different formats are recognized:

     .   The port is added onto the address with a dot separator, for example,

           local_interfaces = <; 192.168.23.65.1234 ; \
                                 3ffe:ffff:836f::fe86:a061.1234

     .   The IP address is enclosed in square brackets, and the port is added
         with a colon separator, for example,

           local_interfaces = <; [192.168.23.65]:1234 ; \
                                 [3ffe:ffff:836f::fe86:a061]:1234

    When a port is not specified, the value of daemon_smtp_ports is used. The
    default setting contains just one port:

        daemon_smtp_ports = smtp

    In this list, ports can be identified either by name (defined in
    /etc/services) or by number. However, when ports are given with individual
    IP addresses in local_interfaces only numbers (not names) can be used.

    The addresses 0.0.0.0 and ::0 are treated specially. They are interpreted
    as `all IPv4 interfaces' and `all IPv6 interfaces', respectively. In each
    case, Exim tells the TCP/IP stack to `listen on all IPvx interfaces'
    instead of setting up separate listening sockets on each interface. The
    default value of local_interfaces is

      local_interfaces = 0.0.0.0

    when Exim is built without IPv6 support; otherwise it is:

      local_interfaces = <; ::0 ; 0.0.0.0

    When a message is received over TCP/IP, the interface and port that were
    used are set in $interface_address and $interface_port.

    The -oX command line option can be used to override the values of
    daemon_smtp_ports and/or local_interfaces for a particular daemon instance.
    Another way of doing this would be to use macros and the -D option.
    However, -oX can be used by any admin user, whereas modification of the
    runtime configuration by -D is allowed only when the caller is root or
    exim.

    The value of -oX is a list of items. The default colon separator can be
    changed in the usual way if required. If there are any items that do not
    contain dots or colons (that is, are not IP addresses), the value of
    daemon_smtp_ports is replaced by the list of those items. If there are any
    items that do contain dots or colons, the value of local_interfaces is
    replaced by those items. Thus, for example,

      -oX 1225

    overrides daemon_smtp_ports, but leaves local_interfaces unchanged, whereas

      -oX 192.168.34.5.1125

    overrides local_interfaces, leaving daemon_smtp_ports unchanged. (However,
    since local_interfaces now contains no items without ports, the value of
    daemon_smtp_ports is no longer relevant.)


    IPV6 ADDRESS SCOPES

    IPv6 addresses have `scopes', and a host with multiple hardware interfaces
    can, in principle, have the same link-local address on different
    interfaces. Thus, additional information is needed, over and above the IP
    address, to distinguish individual interfaces. A a convention of using a
    percent sign followed by something (often the interface name) has been
    adopted in some cases, leading to addresses like this:

      3ffe:2101:12:1:a00:20ff:fe86:a061%eth0

    To accommodate this usage, a percent sign followed by an arbitrary string
    is allowed at the end of an IPv6 address. By default, Exim calls
    "getaddrinfo()" to convert a textual IPv6 address for actual use. This
    function recognizes the percent convention in operating systems that
    support it, and it processes the address appropriately.

    Unfortunately, some older libraries have problems with "getaddrinfo()". If

      IPV6_USE_INET_PTON=yes

    is set in Local/Makefile (or an OS-dependent Makefile) when Exim is built,
    Exim uses "inet_pton()" to convert a textual IPv6 address for actual use,
    instead of "getaddrinfo()". (Before version 4.14, it always used this
    function.) Of course, this means that the additional functionality of
    "getaddrinfo()" - recognizing scoped addresses - is lost.


    EXAMPLES OF STARTING A LISTENING DAEMON

    The default case in an IPv6 environment is

      daemon_smtp_port = smtp
      local_interfaces = <; ::0 ; 0.0.0.0

    This specifies listening on the smtp port on all IPv6 and IPv4 interfaces.
    Either one or two sockets may be used, depending on the characteristics of
    the TCP/IP stack. (This is complicated and messy; for more information,
    read the comments in the daemon.c source file.)

    To specify listening on ports 25 and 25 on all interfaces, you could use

      daemon_smtp_ports = 25 : 26

    (leaving local_interfaces at the default setting) or, more explicitly:

      local_interfaces = <; ::0.25     ; ::0.26 \
                            0.0.0.0.25 ; 0.0.0.0.26

    To listen on the default port on all IPv4 interfaces, and on port 26 on the
    IPv4 loopback address only:

      local_interfaces = 0.0.0.0 : 127.0.0.1.26

    To specify listening on the default ports two specific interfaces only:

      local_interfaces = 192.168.34.67.25 : 192.168.34.67.26

    Note that such a setting excludes listening on the loopback interfaces.


    RECOGNISING LOCAL IP ADDRESSES

    The local_interfaces option is also used when Exim needs to determine
    whether or not an IP address refers to the local host. That is, the IP
    addresses of all the interfaces on which a daemon is listening are always
    treated as local.

    For this usage, port numbers in local_interfaces are ignored. If either of
    the items 0.0.0.0 or ::0 are encountered, Exim gets a complete list of
    available interfaces from the operating system, and extracts the relevant
    (that is, IPv4 or IPv6) addresses to use for checking.

    Some systems set up large numbers of virtual interfaces in order to provide
    many virtual web servers. In this situation, you may want to listen for
    email on only a few of the available interfaces, but nevertheless treat all
    interfaces as local. You can do this by setting extra_local_interfaces to a
    list of IP addresses, possibly including the `all' wildcard values. These
    addresses are recognized as local, but are not used for listening. Consider
    this example:

      local_interfaces = <; 127.0.0.1 ; ::1 ; \
                            192.168.53.235 ; \
                            3ffe:2101:12:1:a00:20ff:fe86:a061

      extra_local_interfaces = <; ::0 ; 0.0.0.0

    The daemon listens on the local interfaces and just one IPv4 and one IPv6
    address, but all available interface addresses are treated as local when
    Exim is routing.

    Another related option is hosts_treat_as_local. However, this uses host
    names rather than IP addresses. It operates when Exim is dealing with MX
    records.


    DELIVERING TO A REMOTE HOST

    Delivery to a remote host is handled by the smtp transport. By default, it
    allows the system's TCP/IP functions to choose which interface to use (if
    there is more than one) when connecting to a remote host. However, the
    "interface" option can be set to specify which interface is used. See the
    description of the smtp transport in chapter 29 for more details.

27. The new option process_log_path sets the name of the file to which Exim
    writes its "process log" when sent SIGUSR1. This is used by the exiwhat
    utility script. If this option is unset, the file called exim-process.info
    in Exim's spool directory is used. The ability to specify the name
    explicitly can be useful in environments where two different Exims are
    running, using different spool directories.


Version 4.20
------------

The documentation is up-to-date for the 4.20 release.

****
