IMAPsync/FAQ.d/FAQ.Use_cache.txt

$Id: FAQ.Use_cache.txt,v 1.11 2021/04/22 13:20:18 gilles Exp gilles $

This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Use_cache.txt


=======================================================================
               Imapsync  --usecache or --useuid options
=======================================================================

Questions answered in this FAQ are:

Q. What is --usecache good for?

Q. What is the technical principle of --usecache?

Q. On Windows, with --useuid and after a reboot or a new login, a new
   run generates duplicates. What can I do to avoid and fix that?

Q. On Windows, with --useuid or --usecache a problem occurs with long
   nested folder names. The error message is:
   "No such file or directory; The filename or extension is too long"

Q. Inode problem with --usecache or --useuid on Linux


Now the questions again and their answers.

=======================================================================
Q. What is --usecache good for?

R. The option --usecache is useful to speed up next syncs with --usecache.
   The first sync with --usecache between two imap accounts is as low 
   as the first sync without --usecache, the next syncs are faster.

=======================================================================
Q. What is the technical principle of --usecache?

R. Option --usecache creates a empty file per email message
   in order to keep the UIDs mapping between account1 and account2.
   
   The cache is maintened at each run. Messages UIDs enregistred 
   in the cache are not fetched again to be identified. It really
   speeds up the process of syncing.
   
   The cache directory is 
   
       <tmpdir>/imapsync_cache/
       
   where <tmpdir> is the temporary directory given by the --tmpdir option
   or the one by default, /tmp on Unix, variable on Windows.
   
=======================================================================
Q. On Windows, with --useuid and after a reboot or a new login, a new
   run generates duplicates. What can I do to avoid and fix that?

R. The cache is build in a temporary directory. Imapsync uses a system
call to get this temporary directory, it is not hardcoded and it
is system dependent, the call garanties a write access. Sometimes on 
Windows this directory name changes upon reboot or for any other reason. 
Anyway, the temporary directory used by imapsync can be set by the 
option --tmpdir and that's the way to avoid this issue. 
The new problem is that the temporary directory must be writable, which
is not a big deal to found.

Use for example either a) b) or c)

a) The current directory:

  imapsync ... --tmpdir .

b) The root directory:

  imapsync ... --tmpdir /

c) Any directory you want, like:

  imapsync ... --tmpdir "C:\TEMP"

Now, to fix the duplicates on host2, have a run with the option
--delete2duplicates

  imapsync ... --delete2duplicates


=======================================================================
Q. On Windows, with --useuid or --usecache a problem occurs with long
   nested folder names. The error message is:
   "No such file or directory; The filename or extension is too long"

R. This comes from a Windows limitation on pathnames.
No more than 260 characters are allowed for pathnames.
See more details on page
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx#maxpath
The workaround solution given at the previous link, 
ie using \\?\D:", does not work for imapsync. 
So this imapsync Windows bug is still pending and needs a fix using 
a different technique to cache, like a database file for example.

A solution to fix the issue is to use a Linux virtual host on a 
Windows box, with VirtualBox or VmWare etc. There is no bug this way,
pathnames can be several thousands characters long.
Better said that done but not so difficult nor expensive these days,
VirtualBox is free and VmWare Player is free for personal or test use.

If you have to stick on Windows, there are two good workarounds 
to reduce the cache directory name:

1) Use --tmpdir "D:\\temp" or simply --tmpdir "D:" and imapsync
will build and use the cache in the sub directory 
D:\imapsync_cache\

2) add two equivalent entries in the etc/hosts for host1 imap.truc.org 
and host2 imap.trac.org. 
If you map the ip of imap.truc.org just with the letter a 
and same thing for imap.trac.org then you gain characters

etc/hosts

192.168.12.1 a
192.168.55.3 b

Then use:

  imapsync --host1 a --host2 b ...

You can get the ip of a host with the ping command line
C:\> ping imap.truc.org

3) A third solution is to not use options --useuid nor --usecache

Fixing this long path problem directly in imapsync is in the TODO file
for a very long time.

=======================================================================
Q. Inode problem with --usecache or --useuid on Linux

R. You may run out of inodes using --usecache, especially with large
migration. Option --usecache creates a empty file per email message
in order to keep the UIDs mapping between account1 and account2.

So, if you plan to sync regularly 10 millions messages over a period 
of migration then the filesystem of --tmpdir needs 10 millions of free 
inodes. If it hasn't those free inodes then create a new special
filesystem devoted to the imapsync cache. 

# Create a file of 10 GB (10 millions*1024):

  dd if=/dev/zero of=/var/tmp/fscache bs=1M count=10000

# Create a filesystem where each file is only 1024 bytes per inode:

  mkfs.ext2 -F -i 1024  /var/tmp/fscache

# Mount this brand new filesystem

  mkdir -p /var/tmp/cachedir
  mount -o loop /var/tmp/fscache /var/tmp/cachedir
  df -i /var/tmp/cachedir # count inodes available

# Tell imapsync to use it

  imapsync ... --tmpdir /var/tmp/cachedir/

# When the migration is over and the cache becomes really useless

    umount /var/tmp/cachedir/
    rm /var/tmp/fscache


=======================================================================
=======================================================================