offlineIMAP3/docs/rfcs/rfc4549.Sync_operations_for_disconnected_IMAP4_Clients.txt

Network Working Group                                   A. Melnikov, Ed.
Request for Comments: 4549                                    Isode Ltd.
Category: Informational                                        June 2006


       Synchronization Operations for Disconnected IMAP4 Clients

Status of This Memo

   This memo provides information for the Internet community.  It does
   not specify an Internet standard of any kind.  Distribution of this
   memo is unlimited.

Copyright Notice

   Copyright (C) The Internet Society (2006).

Abstract

   This document attempts to address some of the issues involved in
   building a disconnected IMAP4 client.  In particular, it deals with
   the issues of what might be called the "driver" portion of the
   synchronization tool: the portion of the code responsible for issuing
   the correct set of IMAP4 commands to synchronize the disconnected
   client in the way that is most likely to make the human who uses the
   disconnected client happy.

   This note describes different strategies that can be used by
   disconnected clients and shows how to use IMAP protocol in order to
   minimize the time of the synchronization process.

   This note also lists IMAP extensions that a server should implement
   in order to provide better synchronization facilities to disconnected
   clients.

















Melnikov                     Informational                      [Page 1]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


Table of Contents

   1. Introduction ....................................................3
      1.1. Conventions Used in This Document ..........................3
   2. Design Principles ...............................................3
   3. Overall Picture of Synchronization ..............................4
   4. Mailbox Synchronization Steps and Strategies ....................7
      4.1. Checking UID Validity ......................................7
      4.2. Synchronizing Local Changes with the Server ................8
           4.2.1. Uploading Messages to the Mailbox ...................8
           4.2.2. Optimizing "move" and "copy" Operations .............9
           4.2.3. Replaying Local Flag Changes .......................14
           4.2.4. Processing Mailbox Compression (EXPUNGE) Requests ..15
           4.2.5. Closing a Mailbox ..................................17
      4.3. Details of "Normal" Synchronization of a Single Mailbox ...18
           4.3.1. Discovering New Messages and Changes to Old
                  Messages ...........................................18
           4.3.2. Searching for "Interesting" Messages. ..............20
           4.3.3. Populating Cache with "Interesting" Messages. ......21
           4.3.4. User-Initiated Synchronization .....................22
      4.4. Special Case: Descriptor-Only Synchronization .............22
      4.5. Special Case: Fast New-Only Synchronization ...............23
      4.6. Special Case: Blind FETCH .................................23
   5. Implementation Considerations ..................................24
      5.1. Error Recovery during Playback ............................26
      5.2. Quality of Implementation Issues ..........................28
      5.3. Optimizations .............................................28
   6. IMAP Extensions That May Help ..................................30
      6.1. CONDSTORE Extension .......................................30
   7. Security Considerations ........................................33
   8. References .....................................................33
      8.1. Normative References ......................................33
      8.2. Informative References ....................................34
   9. Acknowledgements ...............................................34

















Melnikov                     Informational                      [Page 2]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


1.  Introduction

   Several recommendations presented in this document are generally
   applicable to all types of IMAP clients.  However, this document
   tries to concentrate on disconnected mail clients [IMAP-MODEL].  It
   also suggests some IMAP extensions* that should be implemented by
   IMAP servers in order to make the life of disconnected clients
   easier.  In particular, the [UIDPLUS] extension was specifically
   designed to streamline certain disconnected operations, like
   expunging, uploading, and copying messages (see Sections 4.2.1,
   4.2.2.1, and 4.2.4).

   Readers of this document are also strongly advised to read RFC 2683
   [RFC2683].

   * Note that the functionality provided by the base IMAP protocol
     [IMAP4] is sufficient to perform basic synchronization.

1.1.  Conventions Used in This Document

   In examples, "C:" and "S:" indicate lines sent by the client and
   server, respectively.  Long lines in examples are broken for
   editorial clarity.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [KEYWORDS].

   Let's call an IMAP command idempotent if the result of executing the
   command twice sequentially is the same as the result of executing the
   command just once.

2.  Design Principles

   All mailbox state or content information stored on the disconnected
   client should be viewed strictly as a cache of the state of the
   server.  The "master" state remains on the server, just as it would
   with an interactive IMAP4 client.  The one exception to this rule is
   that information about the state of the disconnected client's cache
   (the state includes flag changes while offline and during scheduled
   message uploads) remains on the disconnected client: that is, the
   IMAP4 server is not responsible for remembering the state of the
   disconnected IMAP4 client.

   We assume that a disconnected client is a client that, for whatever
   reason, wants to minimize the length of time that it is "on the
   phone" to the IMAP4 server.  Often this will be because the client is
   using a dialup connection, possibly with very low bandwidth, but



Melnikov                     Informational                      [Page 3]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   sometimes it might just be that the human is in a hurry to catch an
   airplane, or some other event beyond our control.  Whatever the
   reason, we assume that we must make efficient use of the network
   connection, both in the usual sense (not generating spurious traffic)
   and in the sense that we would prefer not to have the connection
   sitting idle while the client and/or the server is performing
   strictly local computation or I/O.  Another, perhaps simpler way of
   stating this is that we assume that network connections are
   "expensive".

   Practical experience with disconnected mail systems has shown that
   there is no single synchronization strategy that is appropriate for
   all cases.  Different humans have different preferences, and the same
   human's preference will vary depending both on external circumstance
   (how much of a hurry the human is in today) and on the value that the
   human places on the messages being transferred.  The point here is
   that there is no way that the synchronization program can guess
   exactly what the human wants to do, so the human will have to provide
   some guidance.

   Taken together, the preceding two principles lead to the conclusion
   that the synchronization program must make its decisions based on
   some kind of guidance provided by the human, by selecting the
   appropriate options in the user interface or through some sort of
   configuration file.  Almost certainly, it should not pause for I/O
   with the human in the middle of the synchronization process.  The
   human will almost certainly have several different configurations for
   the synchronization program, for different circumstances.

   Since a disconnected client has no way of knowing what changes might
   have occurred to the mailbox while it was disconnected, message
   numbers are not useful to a disconnected client.  All disconnected
   client operations should be performed using UIDs, so that the client
   can be sure that it and the server are talking about the same
   messages during the synchronization process.

3.  Overall Picture of Synchronization

   The basic strategy for synchronization is outlined below.  Note that
   the real strategy may vary from one application to another or may
   depend on a synchronization mode.

   a) Process any "actions" that were pending on the client that were
      not associated with any mailbox.  (In particular sending messages
      composed offline with SMTP.  This is not part of IMAP
      synchronization, but it is mentioned here for completeness.)





Melnikov                     Informational                      [Page 4]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   b) Fetch the current list of "interesting" mailboxes.  (The
      disconnected client should allow the user to skip this step
      completely.)

   c) "Client-to-server synchronization": for each IMAP "action" that
      was pending on the client, do the following:

      1) If the action implies opening a new mailbox (any operation that
         operates on messages), open the mailbox.  Check its UID
         validity value (see Section 4.1 for more details) returned in
         the UIDVALIDITY response code.  If the UIDVALIDITY value
         returned by the server differs, the client MUST empty the local
         cache of the mailbox and remove any pending "actions" that
         refer to UIDs in that mailbox (and consider them failed).  Note
         that this doesn't affect actions performed on client-generated
         fake UIDs (see Section 5).

      2) Perform the action.  If the action is to delete a mailbox
         (DELETE), make sure that the mailbox is closed first (see also
         Section 3.4.12 of [RFC2683]).

   d) "Server-to-client synchronization": for each mailbox that requires
      synchronization, do the following:

      1) Check the mailbox UIDVALIDITY (see Section 4.1 for more
         details) with SELECT/EXAMINE/STATUS.

         If UIDVALIDITY value returned by the server differs, the client
         MUST

         * empty the local cache of that mailbox;
         * remove any pending "actions" that refer to UIDs in that
           mailbox and consider them failed; and
         * skip step 2-II.

      2) Fetch the current "descriptors";

         I)  Discover new messages.

         II) Discover changes to old messages.

      3) Fetch the bodies of any "interesting" messages that the client
         doesn't already have.

   e) Close all open mailboxes not required for further operations (if
      staying online) or disconnect all open connections (if going
      offline).




Melnikov                     Informational                      [Page 5]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   Terms used:

   "Actions" are queued requests that were made by the human to the
   client's Mail User Agent (MUA) software while the client was
   disconnected.

   We define "descriptors" as a set of IMAP4 FETCH data items.
   Conceptually, a message's descriptor is that set of information that
   allows the synchronization program to decide what protocol actions
   are necessary to bring the local cache to the desired state for this
   message; since this decision is really up to the human, this
   information probably includes at least a few header fields intended
   for human consumption.  Exactly what will constitute a descriptor
   depends on the client implementation.  At a minimum, the descriptor
   contains the message's UID and FLAGS.  Other likely candidates are
   the RFC822.SIZE, RFC822.HEADER, BODYSTRUCTURE, or ENVELOPE data
   items.

   Comments:

   1) The list of actions should be ordered.  For example, if the human
      deletes message A1 in mailbox A, then expunges mailbox A, and then
      deletes message A2 in mailbox A, the human will expect that
      message A1 is gone and that message A2 is still present but is now
      deleted.

      By processing all the actions before proceeding with
      synchronization, we avoid having to compensate for the local MUA's
      changes to the server's state.  That is, once we have processed
      all the pending actions, the steps that the client must take to
      synchronize itself will be the same no matter where the changes to
      the server's state originated.

   2) Steps a and b can be performed in parallel.  Alternatively, step a
      can be performed after d.

   3) On step b, the set of "interesting" mailboxes pretty much has to
      be determined by the human.  What mailboxes belong to this set may
      vary between different IMAP4 sessions with the same server,
      client, and human.  An interesting mailbox can be a mailbox
      returned by LSUB command (see Section 6.3.9 of [IMAP4]).  The
      special mailbox "INBOX" SHOULD be in the default set of mailboxes
      that the client considers interesting.  However, providing the
      ability to ignore INBOX for a particular session or client may be
      valuable for some mail filtering strategies.






Melnikov                     Informational                      [Page 6]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   4) On step d-2-II, the client also finds out about changes to the
      flags of messages that the client already has in its local cache,
      and about messages in the local cache that no longer exist on the
      server (i.e., messages that have been expunged).

   5) "Interesting" messages are those messages that the synchronization
      program thinks the human wants to have cached locally, based on
      the configuration and the data retrieved in step b.

   6) A disconnected IMAP client is a special case of an IMAP client, so
      it MUST be able to handle any "unexpected" unsolicited responses,
      like EXISTS and EXPUNGE, at any time.  The disconnected client MAY
      ignore EXPUNGE response during "client-to-server" synchronization
      phase (step c).

   The rest of this discussion will focus primarily on the
   synchronization issues for a single mailbox.

4.  Mailbox Synchronization Steps and Strategies

4.1.  Checking UID Validity

   The "UID validity" of a mailbox is a number returned in an
   UIDVALIDITY response code in an OK untagged response at mailbox
   selection time.  The UID validity value changes between sessions when
   UIDs fail to persist between sessions.

   Whenever the client selects a mailbox, the client must compare the
   returned UID validity value with the value stored in the local cache.
   If the UID validity values differ, the UIDs in the client's cache are
   no longer valid.  The client MUST then empty the local cache of that
   mailbox and remove any pending "actions" that refer to UIDs in that
   mailbox.  The client MAY also issue a warning to the human.  The
   client MUST NOT cancel any scheduled uploads (i.e., APPENDs) for the
   mailbox.

   Note that UIDVALIDITY is not only returned on a mailbox selection.
   The COPYUID and APPENDUID response codes defined in the [UIDPLUS]
   extension (see also 4.2.2) and the UIDVALIDITY STATUS response data
   item also contain a UIDVALIDITY value for some other mailbox.  The
   client SHOULD behave as described in the previous paragraph (but it
   should act on the other mailbox's cache), no matter how it obtained
   the UIDVALIDITY value.








Melnikov                     Informational                      [Page 7]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


4.2.  Synchronizing Local Changes with the Server

4.2.1.  Uploading Messages to the Mailbox

   Two of the most common examples of operations resulting in message
   uploads are:

   1) Saving a draft message

   2) Copying a message between remote mailboxes on two different IMAP
      servers or a local mailbox and a remote mailbox.

   Message upload is performed with the APPEND command.  A message
   scheduled to be uploaded has no UID associated with it, as all UIDs
   are assigned by the server.  The APPEND command will effectively
   associate a UID with the uploaded message that can be stored in the
   local cache for future reference.  However, [IMAP4] doesn't describe
   a simple mechanism to discover the message UID by just performing the
   APPEND command.  In order to discover the UID, the client can do one
   of the following:

   1) Remove the uploaded message from cache.  Then, use the mechanism
      described in 4.3 to fetch the information about the uploaded
      message as if it had been uploaded by some other client.

   2) Try to fetch header information as described in 4.2.2 in order to
      find a message that corresponds to the uploaded message.  One
      strategy for doing this is described in 4.2.2.

   Case 1 describes a not particularly smart client.

      C: A003 APPEND Drafts (\Seen $MDNSent) {310}
      S: + Ready for literal data
      C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
      C: From: Fred Foobar <foobar@blt.example.COM>
      C: Subject: afternoon meeting
      C: To: mooch@owatagu.siam.edu
      C: Message-Id: <B27397-0100000@blt.example.COM>
      C: MIME-Version: 1.0
      C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
      C:
      C: Hello Joe, do you think we can meet at 3:30 tomorrow?
      C:
      S: A003 OK APPEND Completed

   Fortunately, there is a simpler way to discover the message UID in
   the presence of the [UIDPLUS] extension:




Melnikov                     Informational                      [Page 8]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


      C: A003 APPEND Drafts (\Seen $MDNSent) {310}
      S: + Ready for literal data
      C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
      C: From: Fred Foobar <foobar@blt.example.COM>
      C: Subject: afternoon meeting
      C: To: mooch@owatagu.siam.edu
      C: Message-Id: <B27397-0100000@blt.example.COM>
      C: MIME-Version: 1.0
      C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
      C:
      C: Hello Joe, do you think we can meet at 3:30 tomorrow?
      C:
      S: A003 OK [APPENDUID 1022843275 77712] APPEND completed

   The UID of the appended message is the second parameter of APPENDUID
   response code.

4.2.2.  Optimizing "move" and "copy" Operations

   Practical experience with IMAP and other mailbox access protocols
   that support multiple mailboxes suggests that moving a message from
   one mailbox to another is an extremely common operation.

4.2.2.1.  Moving a Message between Two Mailboxes on the Same Server

   In IMAP4, a "move" operation between two mailboxes on the same server
   is really a combination of a COPY operation and a STORE +FLAGS
   (\Deleted) operation.  This makes good protocol sense for IMAP, but
   it leaves a simple-minded disconnected client in the silly position
   of deleting and possibly expunging its cached copy of a message, then
   fetching an identical copy via the network.

   However, the presence of the UIDPLUS extension in the server can
   help:

      C: A001 UID COPY 567,414 "Interesting Messages"
      S: A001 OK [COPYUID 1022843275 414,567 5:6] Completed

   This tells the client that the message with UID 414 in the current
   mailbox was successfully copied to the mailbox "Interesting Messages"
   and was given the UID 5, and that the message with UID 567 was given
   the UID 6.

   In the absence of UIDPLUS extension support in the server, the
   following trick can be used.  By including the Message-ID: header and
   the INTERNALDATE data item as part of the descriptor, the client can
   check the descriptor of a "new" message against messages that are
   already in its cache and avoid fetching the extra copy.  Of course,



Melnikov                     Informational                      [Page 9]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   it's possible that the cost of checking to see if the message is
   already in the local cache may exceed the cost of just fetching it,
   so this technique should not be used blindly.  If the MUA implements
   a "move" command, it makes special provisions to use this technique
   when it knows that a copy/delete sequence is the result of a "move"
   command.

   Note that servers are not required (although they are strongly
   encouraged with "SHOULD language") to preserve INTERNALDATE when
   copying messages.

   Also note that since it's theoretically possible for this algorithm
   to find the wrong message (given sufficiently malignant Message-ID
   headers), implementers should provide a way to disable this
   optimization, both permanently and on a message-by-message basis.

   Example 1: Copying a message in the absence of UIDPLUS extension.

   At some point in time the client has fetched the source message and
   some information was cached:

      C: C021 UID FETCH <uids> (BODY.PEEK[] INTERNALDATE FLAGS)
      ...
      S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600"
          FLAGS (\Draft $MDNSent) BODY[] {1036}
      S: ...
      S: Message-Id: <20040903110856.22a127cd@chardonnay>
      S: ...
      S: ...message body...
      S: )
      ...
      S: C021 OK fetch completed

   Later on, the client decides to copy the message:

      C: C035 UID COPY 123 "Interesting Messages"
      S: C035 OK Completed

   As the server hasn't provided the COPYUID response code, the client
   tries the optimization described above:

      C: C036 SELECT "Interesting Messages"
      ...
      C: C037 UID SEARCH ON 31-May-2002 HEADER
          "Message-Id" "20040903110856.22a127cd@chardonnay"
      S: SEARCH 12368
      S: C037 OK completed




Melnikov                     Informational                     [Page 10]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   Note that if the server has returned multiple UIDs in the SEARCH
   response, the client MUST NOT use any of the returned UID.

4.2.2.2.  Moving a Message from a Remote Mailbox to a Local

   Moving a message from a remote mailbox to a local is done with FETCH
   (that includes FLAGS and INTERNALDATE) followed by UID STORE <uid>
   +FLAGS.SILENT (\Deleted):

      C: A003 UID FETCH 123 (BODY.PEEK[] INTERNALDATE FLAGS)
      S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600"
          FLAGS (\Seen $MDNSent) BODY[]
      S: ...message body...
      S: )
      S: A003 OK UID FETCH completed
      C: A004 UID STORE <uid> +FLAGS.SILENT (\Deleted)
      S: A004 STORE completed

   Note that there is no reason to fetch the message during
   synchronization if it's already in the client's cache.  Also, the
   client SHOULD preserve delivery date in the local cache.

4.2.2.3.  Moving a Message from a Local Mailbox to a Remote

   Moving a message from a local mailbox to a remote is done with
   APPEND:

   C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
       {310}
   S: + Ready for literal data
   C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
   C: From: Fred Foobar <foobar@blt.example.COM>
   C: Subject: afternoon meeting
   C: To: mooch@owatagu.siam.edu
   C: Message-Id: <B27397-0100000@blt.example.COM>
   C: MIME-Version: 1.0
   C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
   C:
   C: Hello Joe, do you think we can meet at 3:30 tomorrow?
   C:
   S: A003 OK [APPENDUID 1022843275 77712] completed

   The client SHOULD specify the delivery date from the local cache in
   the APPEND.

   If the [LITERAL+] extension is available, the client can save a
   round-trip*:




Melnikov                     Informational                     [Page 11]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
       {310+}
   C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
   C: From: Fred Foobar <foobar@blt.example.COM>
   C: Subject: afternoon meeting
   C: To: mooch@owatagu.siam.edu
   C: Message-Id: <B27397-0100000@blt.example.COM>
   C: MIME-Version: 1.0
   C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
   C:
   C: Hello Joe, do you think we can meet at 3:30 tomorrow?
   C:
   S: A003 OK [APPENDUID 1022843275 77712] completed

   * Note that there is a risk that the server will reject the message
     due to its size.  If this happens, the client will waste bandwidth
     transferring the whole message.  If the client wouldn't have used
     the LITERAL+, this could have been avoided:

   C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2004 05:26:59 -0600"
       {16777215}
   S: A003 NO Sorry, message is too big

4.2.2.4.  Moving a Message between Two Mailboxes on Different Servers

   Moving a message between two mailbox on two different servers is a
   combination of the operations described in 4.2.2.2 followed by the
   operations described in 4.2.2.3.

4.2.2.5.  Uploading Multiple Messages to a Remote Mailbox with
          MULTIAPPEND

   When there is a need to upload multiple messages to a remote mailbox
   (e.g., as per 4.2.2.3), the presence of certain IMAP extensions may
   significantly improve performance.  One of them is [MULTIAPPEND].

   For some mail stores, opening a mailbox for appending might be
   expensive.  [MULTIAPPEND] tells the server to open the mailbox once
   (instead of opening and closing it "n" times per "n" messages to be
   uploaded) and to keep it open while a group of messages is being
   uploaded to the server.

   Also, if the server supports both [MULTIAPPEND] and [LITERAL+]
   extensions, the entire upload is accomplished in a single
   command/response round-trip.






Melnikov                     Informational                     [Page 12]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   Note: Client implementers should be aware that [MULTIAPPEND] performs
   append of multiple messages atomically.  This means, for example, if
   there is not enough space to save "n"-th message (or the message has
   invalid structure and is rejected by the server) after successful
   upload of "n-1" messages, the whole upload operation fails, and no
   message will be saved in the mailbox.  Although this behavior might
   be desirable in certain situations, it might not be what you want.
   Otherwise, the client should use the regular APPEND command (Section
   4.2.2.3), possibly utilizing the [LITERAL+] extension.  See also
   Section 5.1 for discussions about error recovery.

   Note: MULTIAPPEND can be used together with the UIDPLUS extension in
   a way similar to what was described in Section 4.2.1.  [MULTIAPPEND]
   extends the syntax of the APPENDUID response code to allow for
   multiple message UIDs in the second parameter.

   Example 2:

   This example demonstrates the use of MULTIAPPEND together with
   UIDPLUS (synchronization points where the client waits for
   confirmations from the server are marked with "<--->"):

   C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
       {310}
   <--->
   S: + Ready for literal data
   C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
   C: From: Fred Foobar <foobar@blt.example.COM>
   C: Subject: afternoon meeting
   C: To: mooch@owatagu.siam.edu
   C: Message-Id: <B27397-0100000@blt.example.COM>
   C: MIME-Version: 1.0
   C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
   C:
   C: Hello Joe, do you think we can meet at 3:30 tomorrow?
   C:  (\Seen) " 1-Jun-2002 22:43:04 -0800" {286}
   <--->
   S: + Ready for literal data
   C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST)
   C: From: Joe Mooch <mooch@OWaTaGu.siam.EDU>
   C: Subject: Re: afternoon meeting
   C: To: foobar@blt.example.com
   C: Message-Id: <a0434793874930@OWaTaGu.siam.EDU>
   C: MIME-Version: 1.0
   C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
   C:
   C: 3:30 is fine with me.
   C:



Melnikov                     Informational                     [Page 13]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   S: A003 OK [APPENDUID 1022843275 77712,77713] completed

   The upload takes 3 round-trips.

   Example 3:

   In this example, Example 2 was modified for the case when the server
   supports MULTIAPPEND, LITERAL+, and UIDPLUS.  The upload takes only 1
   round-trip.

   C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
       {310+}
   C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
   C: From: Fred Foobar <foobar@blt.example.COM>
   C: Subject: afternoon meeting
   C: To: mooch@owatagu.siam.edu
   C: Message-Id: <B27397-0100000@blt.example.COM>
   C: MIME-Version: 1.0
   C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
   C:
   C: Hello Joe, do you think we can meet at 3:30 tomorrow?
   C:  (\Seen) " 1-Jun-2002 22:43:04 -0800" {286+}
   C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST)
   C: From: Joe Mooch <mooch@OWaTaGu.siam.EDU>
   C: Subject: Re: afternoon meeting
   C: To: foobar@blt.example.com
   C: Message-Id: <a0434793874930@OWaTaGu.siam.EDU>
   C: MIME-Version: 1.0
   C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
   C:
   C: 3:30 is fine with me.
   C:
   S: A003 OK [APPENDUID 1022843275 77712,77713] completed

4.2.3.  Replaying Local Flag Changes

   The disconnected client uses the STORE command to synchronize local
   flag state with the server.  The disconnected client SHOULD use
   +FLAGS.SILENT or -FLAGS.SILENT in order to set or unset flags
   modified by the user while offline.  The FLAGS form MUST NOT be used,
   as there is a risk that this will overwrite flags on the server that
   have been changed by some other client.

   Example 4:

   For the message with UID 15, the disconnected client stores the
   following flags \Seen and $Highest.  The flags were modified on the
   server by some other client: \Seen, \Answered, and $Highest.  While



Melnikov                     Informational                     [Page 14]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   offline, the user requested that the $Highest flags be removed and
   that the \Deleted flag be added.  The flag synchronization sequence
   for the message should look like:

      C: A001 UID STORE 15 +FLAGS.SILENT (\Deleted)
      S: A001 STORE completed
      C: A002 UID STORE 15 -FLAGS.SILENT ($Highest)
      S: A002 STORE completed

   If the disconnected client is able to store an additional binary
   state information (or a piece of information that can take a value
   from a predefined set of values) in the local cache of an IMAP
   mailbox or in a local mailbox (e.g., message priority), and if the
   server supports storing of arbitrary keywords, the client MUST use
   keywords to store this state on the server.

   Example 5:

   Imagine a speculative mail client that can mark a message as one of
   work-related ($Work), personal ($Personal), or spam ($Spam).  In
   order to mark a message as personal, the client issues:

      C: A001 UID STORE 15 +FLAGS.SILENT ($Personal)
      S: A001 STORE completed
      C: A002 UID STORE 15 -FLAGS.SILENT ($Work $Spam)
      S: A002 STORE completed

   In order to mark the message as not work, not personal and not spam,
   the client issues:

      C: A003 UID STORE 15 -FLAGS.SILENT ($Personal $Work $Spam)
      S: A003 STORE completed

4.2.4.  Processing Mailbox Compression (EXPUNGE) Requests

   A naive disconnected client implementation that supports compressing
   a mailbox while offline may decide to issue an EXPUNGE command to the
   server in order to expunge messages marked \Deleted.  The problem
   with this command during synchronization is that it permanently
   erases all messages with the \Deleted flag set, i.e., even those
   messages that were marked as \Deleted on the server while the user
   was offline.  Doing this might result in an unpleasant surprise for
   the user.

   Fortunately the [UIDPLUS] extension can help in this case as well.
   The extension introduces UID EXPUNGE command, that, unlike EXPUNGE,
   takes a UID set parameter, that lists UIDs of all messages that can
   be expunged.  When processing this command the server erases only



Melnikov                     Informational                     [Page 15]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   messages with \Deleted flag listed in the UID list.  Thus, messages
   not listed in the UID set will not be expunged even if they have the
   \Deleted flag set.

   Example 6:

   While the user was offline, 3 messages with UIDs 7, 27, and 65 were
   marked \Deleted when the user requested to compress the open mailbox.
   Another client marked a message \Deleted on the server (UID 34).
   During synchronization, the disconnected client issues:

      C: A001 UID EXPUNGE 7,27,65
      S: * ... EXPUNGE
      S: * ... EXPUNGE
      S: * ... EXPUNGE
      S: A001 UID EXPUNGE completed

   If another client issues UID SEARCH DELETED command (to find all
   messages with the \Deleted flag) before and after the UID EXPUNGE, it
   will get:

   Before:

      C: B001 UID SEARCH DELETED
      S: * SEARCH 65 34 27 7
      S: B001 UID SEARCH completed

   After:

      C: B002 UID SEARCH DELETED
      S: * SEARCH 34
      S: B002 UID SEARCH completed

   In the absence of the [UIDPLUS] extension, the following sequence of
   commands can be used as an approximation.  Note: It's possible for
   another client to mark additional messages as deleted while this
   sequence is being performed.  In this case, these additional messages
   will be expunged as well.

   1) Find all messages marked \Deleted on the server.

      C: A001 UID SEARCH DELETED
      S: * SEARCH 65 34 27 7
      S: A001 UID SEARCH completed

   2) Find all messages that must not be erased (for the previous
      example the list will consist of the message with UID 34).




Melnikov                     Informational                     [Page 16]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   3) Temporarily remove \Deleted flag on all messages found in step 2.

      C: A002 UID STORE 34 -FLAGS.SILENT (\Deleted)
      S: A002 UID STORE completed

   4) Expunge the mailbox.

      C: A003 EXPUNGE
      S: * 20 EXPUNGE
      S: * 7 EXPUNGE
      S: * 1 EXPUNGE
      S: A003 EXPUNGE completed

      Here, the message with UID 7 has message number 1, with UID 27 has
      message number 7, and with UID 65 has message number 20.

   5) Restore \Deleted flag on all messages found when performing step
      2.

      C: A004 UID STORE 34 +FLAGS.SILENT (\Deleted)
      S: A004 UID STORE completed

4.2.5.  Closing a Mailbox

   When the disconnected client has to close a mailbox, it should not
   use the CLOSE command, because CLOSE does a silent EXPUNGE.  (Section
   4.2.4 explains why EXPUNGE should not be used by a disconnected
   client.)  It is safe to use CLOSE only if the mailbox was opened with
   EXAMINE.

   If the mailbox was opened with SELECT, the client can use one of the
   following commands to implicitly close the mailbox and prevent the
   silent expunge:

   1) UNSELECT - This is a command described in [UNSELECT] that works as
      CLOSE, but doesn't cause the silent EXPUNGE.  This command is
      supported by the server if it reports UNSELECT in its CAPABILITY
      list.

   2) SELECT <another_mailbox> - SELECT causes implicit CLOSE without
      EXPUNGE.

   3) If the client intends to issue LOGOUT after closing the mailbox,
      it may just issue LOGOUT, because LOGOUT causes implicit CLOSE
      without EXPUNGE as well.

   4) SELECT <non_existing_mailbox> - If the client knows a mailbox that
      doesn't exist or can't be selected, it MAY SELECT it.



Melnikov                     Informational                     [Page 17]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   If the client opened the mailbox with SELECT and just wants to avoid
   implicit EXPUNGE without closing the mailbox, it may also use the
   following:

   5) EXAMINE <mailbox> - Reselect the same mailbox in read-only mode.

4.3.  Details of "Normal" Synchronization of a Single Mailbox

   The most common form of synchronization is where the human trusts the
   integrity of the client's copy of the state of a particular mailbox
   and simply wants to bring the client's cache up to date so that it
   accurately reflects the mailbox's current state on the server.

4.3.1.  Discovering New Messages and Changes to Old Messages

   Let <lastseenuid> represent the highest UID that the client knows
   about in this mailbox.  Since UIDs are allocated in strictly
   ascending order, this is simply the UID of the last message in the
   mailbox that the client knows about.  Let <lastseenuid+1> represent
   <lastseenuid>'s UID plus one.  Let <descriptors> represent a list
   consisting of all the FETCH data item items that the implementation
   considers part of the descriptor; at a minimum this is just the FLAGS
   data item, but it usually also includes BODYSTRUCTURE and
   RFC822.SIZE.  At this step, <descriptors> SHOULD NOT include RFC822.

   With no further information, the client can issue the following two
   commands:

      tag1 UID FETCH <lastseenuid+1>:* <descriptors>
      tag2 UID FETCH 1:<lastseenuid> FLAGS

   The first command will request some information about "new" messages
   (i.e., messages received by the server since the last
   synchronization).  It will also allow the client to build a message
   number to UID map (only for new messages).  The second command allows
   the client to

      1) update cached flags for old messages;

      2) find out which old messages got expunged; and

      3) build a mapping between message numbers and UIDs (for old
         messages).

   The order here is significant.  We want the server to start returning
   the list of new message descriptors as fast as it can, so that the
   client can start issuing more FETCH commands, so we start out by
   asking for the descriptors of all the messages we know the client



Melnikov                     Informational                     [Page 18]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   cannot possibly have cached yet.  The second command fetches the
   information we need to determine what changes may have occurred to
   messages that the client already has cached.  Note that the former
   command should only be issued if the UIDNEXT value cached by the
   client differs from the one returned by the server.  Once the client
   has issued these two commands, there's nothing more the client can do
   with this mailbox until the responses to the first command start
   arriving.  A clever synchronization program might use this time to
   fetch its local cache state from disk or to start the process of
   synchronizing another mailbox.

   The following is an example of the first FETCH:

   C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE
       RFC822.SIZE)

   Note 1: The first FETCH may result in the server's sending a huge
   volume of data.  A smart disconnected client should use message
   ranges (see also Section 3.2.1.2 of [RFC2683]), so that the user is
   able to execute a different operation between fetching information
   for a group of new messages.

   Example 7:

   Knowing the new UIDNEXT returned by the server on SELECT or EXAMINE
   (<uidnext>), the client can split the UID range
   <lastseenuid+1>:<uidnext> into groups, e.g., 100 messages.  After
   that, the client can issue:

      C: A011 UID fetch <lastseenuid+1>:<lastseenuid+100>
          (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)
      ...
      C: A012 UID fetch <lastseenuid+101>:<lastseenuid+200>
          (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)
      ...
      ...
      C: A0FF UID fetch <lastseenuid+901>:<uidnext>
          (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)

   Note that unless a SEARCH command is issued, it is impossible to
   determine how many messages will fall into a subrange, as UIDs are
   not necessarily contiguous.

   Note 2: The client SHOULD ignore any unsolicited EXPUNGE responses
   received during the first FETCH command.  EXPUNGE responses contain
   message numbers that are useless to a client that doesn't have the
   message-number-to-UID translation table.




Melnikov                     Informational                     [Page 19]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   The second FETCH command will result in zero or more untagged fetch
   responses.  Each response will have a corresponding UID FETCH data
   item.  All messages that didn't have a matching untagged FETCH
   response MUST be removed from the local cache.

   For example, if the <lastseenuid> had a value 15000 and the local
   cache contained 3 messages with the UIDs 12, 777, and 14999,
   respectively, then after receiving the following responses from the
   server, the client must remove the message with UID 14999 from its
   local cache.

      S: * 1 FETCH (UID 12 FLAGS (\Seen))
      S: * 2 FETCH (UID 777 FLAGS (\Answered \Deleted))

   Note 3: If the client is not interested in flag changes (i.e., the
   client only wants to know which old messages are still on the
   server), the second FETCH command can be substituted with:

      tag2 UID SEARCH UID 1:<lastseenuid>

   This command will generate less traffic.  However, an implementor
   should be aware that in order to build the mapping table from message
   numbers to UIDs, the output of the SEARCH command MUST be sorted
   first, because there is no requirement for a server to return UIDs in
   SEARCH response in any particular order.

4.3.2.  Searching for "Interesting" Messages.

   This step is performed entirely on the client (from the information
   received in the step described in 4.3.1), entirely on the server, or
   on some combination of both.  The decision on what is an
   "interesting" message is up to the client software and the human.
   One easy criterion that should probably be implemented in any client
   is whether the message is "too big" for automatic retrieval, where
   "too big" is a parameter defined in the client's configuration.

   Another commonly used criterion is the age of a message.  For
   example, the client may choose to download only messages received in
   the last week (in this case, <date> would be today's date minus 7
   days):

      tag3 UID SEARCH UID <uidset> SINCE <date>

   Keep in mind that a date search disregards time and time zone.  The
   client can avoid doing this search if it specified INTERNALDATE in
   <descriptors> on the step described in 4.3.1.  If the client did, it
   can perform the local search on its message cache.




Melnikov                     Informational                     [Page 20]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   At this step, the client also decides what kind of information about
   a particular message to fetch from the server.  In particular, even
   for a message that is considered "too big", the client MAY choose to
   fetch some part(s) of it.  For example, if the message is a
   multipart/mixed containing a text part and a MPEG attachment, there
   is no reason for the client not to fetch the text part.  The decision
   of which part should or should not be fetched can be based on the
   information received in the BODYSTRUCTURE FETCH response data item
   (i.e., if BODYSTRUCTURE was included in <descriptors> on the step
   described in 4.3.1).

4.3.3.  Populating Cache with "Interesting" Messages.

   Once the client has found out which messages are "interesting", it
   can start issuing appropriate FETCH commands for "interesting"
   messages or parts thereof.

   Note that fetching a message into the disconnected client's local
   cache does NOT imply that the human has (or even will) read the
   message.  Thus, the synchronization program for a disconnected client
   should always be careful to use the .PEEK variants of the FETCH data
   items that implicitly set the \Seen flag.

   Once the last descriptor has arrived and the last FETCH command has
   been issued, the client simply needs to process the incoming fetch
   items and use them to update the local message cache.

   In order to avoid deadlock problems, the client must give processing
   of received messages priority over issuing new FETCH commands during
   this synchronization process.  This may necessitate temporary local
   queuing of FETCH requests that cannot be issued without causing a
   deadlock.  In order to achieve the best use of the "expensive"
   network connection, the client will almost certainly need to pay
   careful attention to any flow-control information that it can obtain
   from the underlying transport connection (usually a TCP connection).

   Note: The requirement stated in the previous paragraph might result
   in an unpleasant user experience, if followed blindly.  For example,
   the user might be unwilling to wait for the client to finish
   synchronization before starting to process the user's requests.  A
   smart disconnected client should allow the user to perform requested
   operations in between IMAP commands that are part of the
   synchronization process.  See also Note 1 in Section 4.3.1.








Melnikov                     Informational                     [Page 21]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   Example 8:

   After fetching a message BODYSTRUCTURE, the client discovers a
   complex MIME message.  Then, it decides to fetch MIME headers of the
   nested MIME messages and some body parts.

   C: A011 UID fetch 11 (BODYSTRUCTURE)
   S: ...
   C: A012 UID fetch 11 (BODY[HEADER] BODY[1.MIME] BODY[1.1.MIME]
       BODY[1.2.MIME] BODY[2.MIME] BODY[3.MIME] BODY[4.MIME]
       BODY[5.MIME] BODY[6.MIME] BODY[7.MIME] BODY[8.MIME] BODY[9.MIME]
       BODY[10.MIME] BODY[11.MIME] BODY[12.MIME] BODY[13.MIME]
       BODY[14.MIME] BODY[15.MIME] BODY[16.MIME] BODY[17.MIME]
       BODY[18.MIME] BODY[19.MIME] BODY[20.MIME] BODY[21.MIME])
   S: ...
   C: A013 UID fetch 11 (BODY[1.1] BODY[1.2])
   S: ...
   C: A014 UID fetch 11 (BODY[3] BODY[4] BODY[5] BODY[6] BODY[7] BODY[8]
       BODY[9] BODY[10] BODY[11] BODY[13] BODY[14] BODY[15] BODY[16]
       BODY[21])
   S: ...

4.3.4.  User-Initiated Synchronization

   After the client has finished the main synchronization process as
   described in Sections 4.3.1-4.3.3, the user may optionally request
   additional synchronization steps while the client is still online.
   This is not any different from the process described in Sections
   4.3.2 and 4.3.3.

   Typical examples are:

    1) fetch all messages selected in UI.
    2) fetch all messages marked as \Flagged on the server.

4.4.  Special Case: Descriptor-Only Synchronization

   For some mailboxes, fetching the descriptors might be the entire
   synchronization step.  Practical experience with IMAP has shown that
   a certain class of mailboxes (e.g., "archival" mailboxes) are used
   primarily for long-term storage of important messages that the human
   wants to have instantly available on demand but does not want
   cluttering up the disconnected client's cache at any other time.
   Messages in this kind of mailbox would be fetched exclusively by
   explicit actions queued by the local MUA.  Thus, the only
   synchronization desirable on this kind of mailbox is fetching enough
   descriptor information for the user to be able to identify messages
   for subsequent download.



Melnikov                     Informational                     [Page 22]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   Special mailboxes that receive messages from a high volume, low
   priority mailing list might also be in this category, at least when
   the human is in a hurry.

4.5.  Special Case: Fast New-Only Synchronization

   In some cases, the human might be in such a hurry that he or she
   doesn't care about changes to old messages, just about new messages.
   In this case, the client can skip the UID FETCH command that obtains
   the flags and UIDs for old messages (1:<lastseenuid>).

4.6.  Special Case: Blind FETCH

   In some cases, the human may know (for whatever reason) that he or
   she always wants to fetch any new messages in a particular mailbox,
   unconditionally.  In this case, the client can just fetch the
   messages themselves, rather than just the descriptors, by using a
   command like:

      tag1 UID FETCH <lastseenuid+1>:* (FLAGS BODY.PEEK[])

   Note that this example ignores the fact that the messages can be
   arbitrary long.  The disconnected client MUST always check for
   message size before downloading, unless explicitly told otherwise.  A
   well-behaved client should instead use something like the following:

   1) Issue "tag1 UID FETCH <lastseenuid+1>:* (FLAGS RFC822.SIZE)".

   2) From the message sizes returned in step 1, construct UID set
      <required_messages>.

   3) Issue "tag2 UID FETCH <required_messages> (BODY.PEEK[])".

   or

   1) Issue "tag1 UID FETCH <lastseenuid+1>:* (FLAGS)".

   2) Construct UID set <old_uids> from the responses of step 1.

   3) Issue "tag2 SEARCH UID <old_uids> SMALLER <message_limit>".
      Construct UID set <required_messages> from the result of the
      SEARCH command.

   4) Issue "tag3 UID FETCH <required_messages> (BODY.PEEK[])".







Melnikov                     Informational                     [Page 23]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   or

   1) Issue "tag1 UID FETCH <lastseenuid+1>:* (FLAGS
      BODY.PEEK[]<0.<length>>)", where <length> should be replaced with
      the maximal message size the client is willing to download.

      Note: In response to such a command, the server will only return
      partial data if the message is longer than <length>.  It will
      return the full message data for any message whose size is smaller
      than or equal to <length>.  In the former case, the client will
      not be able to extract the full MIME structure of the message from
      the truncated data, so the client should include BODYSTRUCTURE in
      the UID FETCH command as well.

5.  Implementation Considerations

   Below are listed some common implementation pitfalls that should be
   considered when implementing a disconnected client.

   1) Implementing fake UIDs on the client.

      A message scheduled to be uploaded has no UID, as UIDs are
      selected by the server.  The client may implement fake UIDs
      internally in order to reference not-yet-uploaded messages in
      further operations.  (For example, a message could be scheduled to
      be uploaded, but subsequently marked as deleted or copied to
      another mailbox).  Here, the client MUST NOT under any
      circumstances send these fake UIDs to the server.  Also, client
      implementers should be reminded that according to [IMAP4] a UID is
      a 32-bit unsigned integer excluding 0.  So, both 4294967295 and
      2147483648 are valid UIDs, and 0 and -1 are both invalid.  Some
      disconnected mail clients have been known to send negative numbers
      (e.g., "-1") as message UIDs to servers during synchronization.

      Situation 1: The user starts composing a new message, edits it,
      saves it, continues to edit it, and saves it again.

      A disconnected client may record in its replay log (log of
      operations to be replayed on the server during synchronization)
      the sequence of operations as shown below.  For the purpose of
      this situation, we assume that all draft messages are stored in
      the mailbox called Drafts on an IMAP server.  We will also use the
      following conventions:  <old_uid> is the UID of the intermediate
      version of the draft when it was saved for the first time.  This
      is a fake UID generated on the client.  <new_uid> is the UID of
      the final version of the draft.  This is another fake UID
      generated on the client.




Melnikov                     Informational                     [Page 24]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


      1) APPEND Drafts (\Seen $MDNSent \Drafts) {<nnn>}
         ...first version of the message follows...

      2) APPEND Drafts (\Seen $MDNSent \Drafts) {<mmm>}
         ...final version of the message follows...

      3) STORE <old_uid> +FLAGS (\Deleted)

      Step 1 corresponds to the first attempt to save the draft message,
      step 2 corresponds to the second attempt to save the draft
      message, and step 3 deletes the first version of the draft message
      saved in step 1.

      A naive disconnected client may send the command in step 3 without
      replacing the fake client generated <old_uid> with the value
      returned by the server in step 1.  A server will probably reject
      this command, which will make the client believe that the
      synchronization sequence has failed.

   2) Section 5.1 discusses common implementation errors related to
      error recovery during playback.

   3) Don't assume that the disconnected client is the only client used
      by the user.

      Situation 2: Some clients may use the \Deleted flag as an
      indicator that the message should not appear in the user's view.
      Usage of the \Deleted flag for this purpose is not safe, as other
      clients (e.g., online clients) might EXPUNGE the mailbox at any
      time.

   4) Beware of data dependencies between synchronization operations.

      It might be very tempting for a client writer to perform some
      optimizations on the playback log.  Such optimizations might
      include removing redundant operations (for example, see
      optimization 2 in Section 5.3), or their reordering.

      It is not always safe to reorder or remove redundant operations
      during synchronization because some operations may have
      dependencies (as Situation 3 demonstrates).  So, if in doubt,
      don't do this.

      Situation 3: The user copies a message out of a mailbox and then
      deletes the mailbox.






Melnikov                     Informational                     [Page 25]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


         C: A001 SELECT Old-Mail
         S: ...
         C: A002 UID COPY 111 ToDo
         S: A002 OK [COPYUID 1022843345 111 94] Copy completed
         ...
         C: A015 CLOSE
         S: A015 OK Completed
         C: A016 DELETE Old-Mail
         S: A016 OK Mailbox deletion completed successfully

      If the client performs DELETE (tag A016) first and COPY (tag A002)
      second, then the COPY fails.  Also, the message that the user so
      carefully copied into another mailbox has been lost.

5.1.  Error Recovery during Playback

   Error recovery during synchronization is one of the trickiest parts
   to get right.  Below, we will discuss certain error conditions and
   suggest possible choices for handling them.

   1) Lost connection to the server.

      The client MUST remember the current position in the playback
      (replay) log and replay it starting from the interrupted operation
      (the last command issued by the client, but not acknowledged by
      the server) the next time it successfully connects to the same
      server.  If the connection was lost while executing a non-
      idempotent IMAP command (see the definition in Section 1), then
      when the client is reconnected, it MUST make sure that the
      interrupted command was indeed not executed.  If it wasn't
      executed, the client must restart playback from the interrupted
      command, otherwise from the following command.

      Upon reconnect, care must be taken in order to properly reapply
      logical operations that are represented by multiple IMAP commands,
      e.g., UID EXPUNGE emulation when UID EXPUNGE is not supported by
      the server (see Section 4.2.4).

      Once the client detects that the connection to the server was
      lost, it MUST stop replaying its log.  There are existing
      disconnected clients that, to the great annoyance of users, pop up
      an error dialog for each and every playback operation that fails.

   2) Copying/appending messages to a mailbox that doesn't exist.  (The
      server advertises this condition by sending the TRYCREATE response
      code in the tagged NO response to the APPEND or COPY command.)





Melnikov                     Informational                     [Page 26]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


      The user should be advised about the situation and be given one of
      the following choices:

      a) Try to recreate a mailbox.
      b) Copy/upload messages to another mailbox.
      c) Skip copy/upload.
      d) Abort replay.

   3) Copying messages from a mailbox that doesn't exist, or renaming or
      getting/changing ACLs [ACL] on a mailbox that doesn't exist:

      a) Skip operation.
      b) Abort replay.

   4) Deleting mailboxes or deleting/expunging messages that no longer
      exist.

      This is actually is not an error and should be ignored by the
      client.

   5) Performing operations on messages that no longer exist.

      a) Skip operation.
      b) Abort replay.

      In the case of changing flags on an expunged message, the client
      should silently ignore the error.

   Note 1: Several synchronization operations map to multiple IMAP
   commands (for example, "move" described in 4.2.2).  The client must
   guarantee atomicity of each such multistep operation.  For example,
   when performing a "move" between two mailboxes on the same server, if
   the server is unable to copy messages, the client MUST NOT attempt to
   set the \Deleted flag on the messages being copied, let alone expunge
   them.  However, the client MAY consider that move operation to have
   succeeded even if the server was unable to set the \Deleted flag on
   copied messages.

   Note 2: Many synchronization operations have data dependencies.  A
   failed operation must cause all dependent operations to fail as well.
   The client should check this and MUST NOT try to perform all
   dependent operations blindly (unless the user corrected the original
   problem).  For example, a message may be scheduled to be appended to
   a mailbox on the server and later on the appended message may be
   copied to another mailbox.  If the APPEND operation fails, the client
   must not attempt to COPY the failed message later on.  (See also
   Section 5, Situation 3).




Melnikov                     Informational                     [Page 27]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


5.2.  Quality of Implementation Issues

   Below, some quality of implementation issues are listed for
   disconnected clients.  They will help to write a disconnected client
   that works correctly, performs synchronization as quickly as possible
   (and thus can make the user happier as well as save her some money),
   and minimizes the server load:

   1) Don't lose information.

      No matter how smart your client is in other areas, if it loses
      information, users will get very upset.

   2) Don't do work unless explicitly asked.  Be flexible.  Ask all
      questions BEFORE starting synchronization, if possible.

   3) Minimize traffic.

      The client MUST NOT issue a command if the client already received
      the required information from the server.

      The client MUST make use of UIDPLUS extension if it is supported
      by the server.

      See also optimization 1 in Section 5.3.

   4) Minimize the number of round-trips.

      Round-trips kill performance, especially on links with high
      latency.  Sections 4.2.2.5 and 5.2 give some advice on how to
      minimize the number of round-trips.

      See also optimization 1 in Section 5.3.

5.3.  Optimizations

   Some useful optimizations are described in this section.  A
   disconnected client that supports the recommendations listed below
   will give the user a more pleasant experience.

   1) The initial OK or PREAUTH responses may contain the CAPABILITY
      response code as described in Section 7.1 of [IMAP4].  This
      response code gives the same information as returned by the
      CAPABILITY command*.  A disconnected client that pays attention to
      this response code can avoid sending CAPABILITY command and will
      save a round-trip.





Melnikov                     Informational                     [Page 28]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


      * Note: Some servers report in the CAPABILITY response code
        extensions that are only relevant in unauthenticated state or in
        all states.  Such servers usually send another CAPABILITY
        response code upon successful authentication using LOGIN or
        AUTHENTICATE command (that negotiates no security layer; see
        Section 6.2.2 of [IMAP4]).  The CAPABILITY response code sent
        upon successful LOGIN/AUTHENTICATE might be different from the
        CAPABILITY response code in the initial OK response, as
        extensions only relevant for unauthenticated state will not be
        advertised, and some additional extensions available only in
        authenticated and/or selected state will be.

   Example 9:

   S: * OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS
       AUTH=DIGEST-MD5 AUTH=SRP] imap.example.com ready
   C: 2 authenticate DIGEST-MD5
   S: 2 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN
       SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND]
       User authenticated (no layer)

   2) An advanced disconnected client may choose to optimize its replay
      log.  For example, there might be some operations that are
      redundant (the list is not complete):

      a) an EXPUNGE followed by another EXPUNGE or CLOSE;
      b) changing flags (other than the \Deleted flag) on a message that
         gets immediately expunged;
      c) opening and closing the same mailbox.

   When optimizing, be careful about data dependencies between commands.
   For example, if the client is wishing to optimize (see case b, above)

      tag1 UID STORE <uid1> +FLAGS (\Deleted)
      ...
      tag2 UID STORE <uid1> +FLAGS (\Flagged)
      ...
      tag3 UID COPY <uid1> "Backup"
      ...
      tag4 UID EXPUNGE <uid1>

   it can't remove the second UID STORE command because the message is
   being copied before it gets expunged.

   In general, it might be a good idea to keep mailboxes open during
   synchronization (see case c above), if possible.  This can be more
   easily achieved in conjunction with optimization 3 described below.




Melnikov                     Informational                     [Page 29]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   3) Perform some synchronization steps in parallel, if possible.

      Several synchronization steps don't depend on each other and thus
      can be performed in parallel.  Because the server machine is
      usually more powerful than the client machine and can perform some
      operations in parallel, this may speed up the total time of
      synchronization.

      In order to achieve such parallelization, the client will have to
      open more than one connection to the same server.  Client writers
      should not forget about non-trivial cost associated with
      establishing a TCP connection and performing an authentication.
      The disconnected client MUST NOT use one connection per mailbox.
      In most cases, it is sufficient to have two connections.  The
      disconnected client SHOULD avoid selecting the same mailbox in
      more than one connection; see Section 3.1.1 of [RFC2683] for more
      details.

      Any mailbox synchronization MUST start with checking the
      UIDVALIDITY as described in Section 4.1 of this document.  The
      client MAY use STATUS command to check UID Validity of a non-
      selected mailbox.  This is preferable to opening many connections
      to the same server to perform synchronization of multiple
      mailboxes simultaneously.  As described in Section 5.3.10 of
      [IMAP4], this SHOULD NOT be used on the selected mailbox.

6.  IMAP Extensions That May Help

   The following extensions can save traffic and/or the number of
   round-trips:

   1) The use of [UIDPLUS] is discussed in Sections 4.1, 4.2.1, 4.2.2.1
      and 4.2.4.

   2) The use of the MULTIAPPEND and LITERAL+ extensions for uploading
      messages is discussed in Section 4.2.2.5.

   3) Use the CONDSTORE extension (see Section 6.1) for quick flag
      resynchronization.

6.1.  CONDSTORE Extension

   An advanced disconnected mail client should use the [CONDSTORE]
   extension when it is supported by the server.  The client must cache
   the value from HIGHESTMODSEQ OK response code received on mailbox
   opening and update it whenever the server sends MODSEQ FETCH data
   items.




Melnikov                     Informational                     [Page 30]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


   If the client receives NOMODSEQ OK untagged response instead of
   HIGHESTMODSEQ, it MUST remove the last known HIGHESTMODSEQ value from
   its cache and follow the more general instructions in Section 3.

   When the client opens the mailbox for synchronization, it first
   compares UIDVALIDITY as described in step d-1 in Section 3.  If the
   cached UIDVALIDITY value matches the one returned by the server, the
   client MUST compare the cached value of HIGHESTMODSEQ with the one
   returned by the server.  If the cached HIGHESTMODSEQ value also
   matches the one returned by the server, then the client MUST NOT
   fetch flags for cached messages, as they hasn't changed.  If the
   value on the server is higher than the cached one, the client MAY use
   "SEARCH MODSEQ <cached-value>" to find all messages with flags
   changed since the last time the client was online and had the mailbox
   opened.  Alternatively, the client MAY use "FETCH 1:* (FLAGS)
   (CHANGEDSINCE <cached-value>)".  The latter operation combines
   searching for changed messages and fetching new information.

   In all cases, the client still needs to fetch information about new
   messages (if requested by the user) as well as discover which
   messages have been expunged.

   Step d ("Server-to-client synchronization") in Section 4 in the
   presence of the CONDSTORE extension is amended as follows:

   d) "Server-to-client synchronization" - For each mailbox that
      requires synchronization, do the following:

      1a) Check the mailbox UIDVALIDITY (see section 4.1 for more
          details) with SELECT/EXAMINE/STATUS.

          If the UIDVALIDITY value returned by the server differs, the
          client MUST

          * empty the local cache of that mailbox;
          * "forget" the cached HIGHESTMODSEQ value for the mailbox;
          * remove any pending "actions" that refer to UIDs in that
            mailbox (note that this doesn't affect actions performed on
            client-generated fake UIDs; see Section 5); and
          * skip steps 1b and 2-II;

      1b) Check the mailbox HIGHESTMODSEQ.  If the cached value is the
          same as the one returned by the server, skip fetching message
          flags on step 2-II, i.e., the client only has to find out
          which messages got expunged.






Melnikov                     Informational                     [Page 31]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


      2) Fetch the current "descriptors".

         I)  Discover new messages.

         II) Discover changes to old messages and flags for new messages
             using
             "FETCH 1:* (FLAGS) (CHANGEDSINCE <cached-value>)" or
             "SEARCH MODSEQ <cached-value>".

             Discover expunged messages; for example, using
             "UID SEARCH 1:<lastseenuid>".  (All messages not returned
             in this command are expunged.)

      3) Fetch the bodies of any "interesting" messages that the client
         doesn't already have.

         Example 10:

         The UIDVALIDITY value is the same, but the HIGHESTMODSEQ value
         has changed on the server while the client was offline.

      C: A142 SELECT INBOX
      S: * 172 EXISTS
      S: * 1 RECENT
      S: * OK [UNSEEN 12] Message 12 is first unseen
      S: * OK [UIDVALIDITY 3857529045] UIDs valid
      S: * OK [UIDNEXT 201] Predicted next UID
      S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
      S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited
      S: * OK [HIGHESTMODSEQ 20010715194045007]
      S: A142 OK [READ-WRITE] SELECT completed

   After that, either:

      C: A143 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 20010715194032001)
      S: * 2 FETCH (UID 6 MODSEQ (20010715205008000) FLAGS (\Deleted))
      S: * 5 FETCH (UID 9 MODSEQ (20010715195517000) FLAGS ($NoJunk
          $AutoJunk $MDNSent))
         ...
      S: A143 OK FETCH completed

   or:









Melnikov                     Informational                     [Page 32]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


      C: A143 UID SEARCH MODSEQ 20010715194032001 UID 1:20
      S: * SEARCH 6 9 11 12 18 19 20 23 (MODSEQ 20010917162500)
      S: A143 OK Search complete
      C: A144 UID SEARCH 1:20
      S: * SEARCH 6 9 ...
      S: A144 OK FETCH completed

7.  Security Considerations

   It is believed that this document does not raise any new security
   concerns that are not already present in the base [IMAP4] protocol,
   and these issues are discussed in [IMAP4].  Additional security
   considerations may be found in different extensions mentioned in this
   document; in particular, in [UIDPLUS], [LITERAL+], [CONDSTORE],
   [MULTIAPPEND], and [UNSELECT].

   Implementers are also reminded about the importance of thorough
   testing.

8.  References

8.1.  Normative References

   [KEYWORDS]    Bradner, S., "Key words for use in RFCs to Indicate
                 Requirement Levels", BCP 14, RFC 2119, March 1997.

   [IMAP4]       Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL -
                 VERSION 4rev1", RFC 3501, March 2003.

   [UIDPLUS]     Crispin, M., "Internet Message Access Protocol (IMAP) -
                 UIDPLUS extension", RFC 4315, December 2005.

   [LITERAL+]    Myers, J., "IMAP4 non-synchronizing literals", RFC
                 2088, January 1997.

   [CONDSTORE]   Melnikov, A. and S. Hole, "IMAP Extension for
                 Conditional STORE Operation or Quick Flag Changes
                 Resynchronization", RFC 4551, June 2006.

   [MULTIAPPEND] Crispin, M., "Internet Message Access Protocol (IMAP) -
                 MULTIAPPEND Extension", RFC 3502, March 2003.

   [UNSELECT]    Melnikov, A., "Internet Message Access Protocol (IMAP)
                 UNSELECT command", RFC 3691, February 2004.

   [RFC2683]     Leiba, B., "IMAP4 Implementation Recommendations", RFC
                 2683, September 1999.




Melnikov                     Informational                     [Page 33]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


8.2.  Informative References

   [ACL]         Melnikov, A., "IMAP4 Access Control List (ACL)
                 Extension", RFC 4314, December 2005.

   [IMAP-MODEL]  Crispin, M., "Distributed Electronic Mail Models in
                 IMAP4", RFC 1733, December 1994.

9.  Acknowledgements

   This document is based on version 01 of the text written by Rob
   Austein in November 1994.

   The editor appreciates comments posted by Mark Crispin to the IMAP
   mailing list and the comments/corrections/ideas received from Grant
   Baillie, Cyrus Daboo, John G. Myers, Chris Newman, and Timo Sirainen.

   The editor would also like to thank the developers of Netscape
   Messenger and Mozilla mail clients for providing examples of
   disconnected mail clients that served as a base for many
   recommendations in this document.

Editor's Address

   Alexey Melnikov
   Isode Limited
   5 Castle Business Village
   36 Station Road
   Hampton, Middlesex
   TW12 2BX
   United Kingdom

   Phone: +44 77 53759732
   EMail: alexey.melnikov@isode.com

















Melnikov                     Informational                     [Page 34]

RFC 4549        Synch Ops for Disconnected IMAP4 Clients       June 2006


Full Copyright Statement

   Copyright (C) The Internet Society (2006).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
   ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
   INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
   INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   ietf-ipr@ietf.org.

Acknowledgement

   Funding for the RFC Editor function is provided by the IETF
   Administrative Support Activity (IASA).







Melnikov                     Informational                     [Page 35]