TUTORIAL_Unix.html 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487
  1. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
  2. <HTML>
  3. <HEAD>
  4. <META NAME="generator" CONTENT="http://txt2tags.org">
  5. <TITLE>Imapsync tutorial</TITLE>
  6. </HEAD><BODY BGCOLOR="white" TEXT="black">
  7. <CENTER>
  8. <H1>Imapsync tutorial</H1>
  9. <FONT SIZE="4"><I>Gilles LAMIRAL gilles@lamiral.info</I></FONT><BR>
  10. <FONT SIZE="4">$Id: TUTORIAL_Unix.t2t,v 1.27 2022/01/14 12:42:51 gilles Exp gilles $ </FONT>
  11. </CENTER>
  12. <P></P>
  13. <HR NOSHADE SIZE=1>
  14. <P></P>
  15. <UL>
  16. <LI><A HREF="#toc1">1. Good practices overview</A>
  17. <LI><A HREF="#toc2">2. Basic steps</A>
  18. <UL>
  19. <LI><A HREF="#toc3">2.1. Verifying imapsync works well</A>
  20. <LI><A HREF="#toc4">2.2. Working with your data</A>
  21. <LI><A HREF="#toc5">2.3. Prepare your credentials</A>
  22. <LI><A HREF="#toc6">2.4. Take a real user account as a source</A>
  23. <LI><A HREF="#toc7">2.5. Take a test user account as a destination</A>
  24. <LI><A HREF="#toc8">2.6. Edit your own script mysync</A>
  25. </UL>
  26. <LI><A HREF="#toc9">3. Background knowledge about mailboxes</A>
  27. <LI><A HREF="#toc10">4. Imapsync presentation</A>
  28. <LI><A HREF="#toc11">5. Conventions</A>
  29. <LI><A HREF="#toc12">6. Why start with a test account on destination host2?</A>
  30. <LI><A HREF="#toc13">7. Imapsync default behavior</A>
  31. <LI><A HREF="#toc14">8. To go further with imapsync</A>
  32. </UL>
  33. <P></P>
  34. <HR NOSHADE SIZE=1>
  35. <P></P>
  36. <A NAME="toc1"></A>
  37. <H1>1. Good practices overview</H1>
  38. <UL>
  39. <LI>Do the basic checks showing imapsync works by itself where you run it.
  40. <P></P>
  41. <LI>Next, applying imapsync to your data,
  42. continue with a <B>real user</B> account on the source imap server (host1)
  43. and a <B>test</B> account on the destination imap server (host2).
  44. <P></P>
  45. It's the best advice I can give to learn imapsync, be confident in it, and
  46. verify it will do what you expect it to do in your context, all of that
  47. without much pain. So try imapsync with a real account at the source, aka host1,
  48. and a test account at the destination, aka host2.
  49. <P></P>
  50. <LI>Next, once familiar and satisfied by the result on the host2 test account,
  51. change to a real user account on host2 or just stop considering it a test one.
  52. </UL>
  53. <A NAME="toc2"></A>
  54. <H1>2. Basic steps</H1>
  55. <A NAME="toc3"></A>
  56. <H2>2.1. Verifying imapsync works well</H2>
  57. <P>
  58. Open a terminal and go to the imapsync directory.
  59. The imapsync directory is the directory created by the extraction
  60. of the tarball (.tgz), its name is <CODE>imapsync-2.xxx</CODE> where <CODE>2.xxx</CODE>
  61. is imapsync release number (2.178 at the time of this proofread).
  62. </P>
  63. <PRE>
  64. cd imapsync-2.xxx/
  65. </PRE>
  66. <P>
  67. Verify imapsync runs on your system
  68. </P>
  69. <PRE>
  70. ./imapsync
  71. </PRE>
  72. <P>
  73. It should output the help message.
  74. </P>
  75. <P>
  76. If the previous command fails then there is an installation issue.
  77. Go back to the <A HREF="https://imapsync.lamiral.info/#install">https://imapsync.lamiral.info/#install</A> section, then
  78. read and apply the installation file corresponding to your
  79. system and drop me an email about your issue.
  80. </P>
  81. <P>
  82. Next, verify imapsync runs well with live tests. This check needs Internet
  83. access. It does a simple sync between two real dedicated
  84. imap mailboxes located at the host test.lamiral.info
  85. </P>
  86. <PRE>
  87. ./imapsync --testslive
  88. </PRE>
  89. <P>
  90. Now verify that the script examples/imapsync_example.sh runs fine:
  91. </P>
  92. <PRE>
  93. sh examples/imapsync_example.sh
  94. </PRE>
  95. <P>
  96. This script does the same thing as "<CODE>imapsync --testslive</CODE>" but it
  97. uses explicitly the 6 parameters so it will be a good start
  98. for your future own script.
  99. </P>
  100. <A NAME="toc4"></A>
  101. <H2>2.2. Working with your data</H2>
  102. <P>
  103. I consider you're still in the imapsync top directory.
  104. </P>
  105. <P>
  106. Make a copy of the script <CODE>examples/imapsync_example.sh</CODE>
  107. </P>
  108. <PRE>
  109. cp examples/imapsync_example.sh mysync
  110. </PRE>
  111. <P>
  112. Check that the copy works as the original
  113. </P>
  114. <PRE>
  115. sh mysync
  116. </PRE>
  117. <P>
  118. So far so good, now we're going to work with your data.
  119. </P>
  120. <A NAME="toc5"></A>
  121. <H2>2.3. Prepare your credentials</H2>
  122. <P>
  123. An IMAP account is accessed with 3 parameters,
  124. </P>
  125. <UL>
  126. <LI>the imap server <B>host</B>. It's a server name or an IP address
  127. <LI>the <B>user</B> name
  128. <LI>the <B>password</B>
  129. </UL>
  130. <P>
  131. Since imapsync job is to sync two imap accounts, we need 3 + 3 = 6 parameters:
  132. </P>
  133. <UL>
  134. <LI>Three parameters to read data from the source account: <B>host1</B>, <B>user1</B> and <B>password1</B>
  135. <LI>Three parameters to copy this data to the destination account: <B>host2</B>, <B>user2</B> and <B>password2</B>
  136. </UL>
  137. <A NAME="toc6"></A>
  138. <H2>2.4. Take a real user account as a source</H2>
  139. <P>
  140. Even during the learning time with imapsync, you can take a
  141. real user account as a source. There is also no problem if this account is
  142. currently used by a user. By default, this account will only be read, no change will
  143. be made by imapsync on it. It's safe to use a normal and live account as a source, even to learn imapsync.
  144. </P>
  145. <P>
  146. Assuming that the imap source server name host1 is <B>origin.example.com</B>,
  147. the user1 account name is <B>myuser1</B> and its password is <B>mysecret1</B>,
  148. we now have the first three parameters.
  149. </P>
  150. <UL>
  151. <LI>--host1 <B>origin.example.com</B>
  152. <LI>--user1 <B>myuser1</B>
  153. <LI>--password1 <B>mysecret1</B>
  154. </UL>
  155. <A NAME="toc7"></A>
  156. <H2>2.5. Take a test user account as a destination</H2>
  157. <P>
  158. Unlike the source side, the destination side will be modified by
  159. imapsync. Therefore, for learning, checking, and adjusting,
  160. it is not a good idea to use a real user imap account
  161. the first time you play with imapsync.
  162. </P>
  163. <P>
  164. If you really can't afford a test account on host2, it's ok,
  165. imapsync is not that bad but you may have some work to do to fix some unwanted behavior.
  166. Unwanted behavior is mostly folders names that you don't want to be the same on both sides.
  167. </P>
  168. <P>
  169. Assuming that the imap destination server name host2 is <B>destiny.example.com</B>,
  170. the user2 account name is <B>myuser2</B> and its password is <B>mysecret2</B>,
  171. we now have the next three parameters.
  172. </P>
  173. <UL>
  174. <LI>--host2 <B>destiny.example.com</B>
  175. <LI>--user2 <B>myuser2</B>
  176. <LI>--password2 <B>mysecret2</B>
  177. </UL>
  178. <A NAME="toc8"></A>
  179. <H2>2.6. Edit your own script mysync</H2>
  180. <P>
  181. Now edit the script <CODE>mysync</CODE> and replace the test values with yours.
  182. </P>
  183. <P>
  184. You're ready for a dry test on your accounts.
  185. </P>
  186. <PRE>
  187. sh mysync
  188. </PRE>
  189. <P>
  190. Since the <CODE>mysync</CODE> script is a copy of <CODE>examples/imapsync_example.sh</CODE>,
  191. your first run with your data should include three other options
  192. <CODE>--automap</CODE> <CODE>--justfolders</CODE> <CODE>--dry</CODE>.
  193. With <CODE>--dry</CODE> option, nothing will be done for real on host2
  194. but yet it will test whether the credentials are ok on both sides.
  195. You'll encounter two successful logins, or one login failure on host1 or host2, or two failures on host1 and host2.
  196. If the logins are ok, you will be able to observe that the folders mapping is ok.
  197. </P>
  198. <P>
  199. If login fails then double-check all three values that identify
  200. the account, which are the host, the login name, and the password.
  201. </P>
  202. <P>
  203. If the folders mapping proposed is not ok then you can fix it with
  204. the option <CODE>--f1f2</CODE>. The following example maps the source folder
  205. "Sent Messages" to the destination folder "Sent". The double-quotes
  206. are not part of the names of the folders but they should be used when special
  207. characters like blanks are in the names of the folders:
  208. </P>
  209. <PRE>
  210. ./imapsync ... --f1f2 "Sent Messages=Sent"
  211. </PRE>
  212. <P>
  213. As explained in the inline help or the README:
  214. </P>
  215. <PRE>
  216. --f1f2 str1=str2 : Force folder str1 to be synced to str2.
  217. </PRE>
  218. <P>
  219. You're ready for a real test on your accounts, restricted to
  220. folders. Remove <CODE>--dry</CODE> from the <CODE>mysync</CODE> script and rerun <CODE>mysync</CODE>:
  221. </P>
  222. <PRE>
  223. sh mysync
  224. </PRE>
  225. <A NAME="toc9"></A>
  226. <H1>3. Background knowledge about mailboxes</H1>
  227. <P>
  228. Three Internet protocols are used to access almost all email accounts:
  229. POP3, IMAP, HTTP.
  230. </P>
  231. <P>
  232. The oldest protocol still used to access mailboxes is POP3, the Post Office Protocol.
  233. POP3 gives access to only one main box called INBOX.
  234. </P>
  235. <P>
  236. With POP3, messages have no flags at all, no Seen/UnSeen, Forwarded, or Flagged labels.
  237. </P>
  238. <P>
  239. It's not systematic but messages are often removed from a POP3 server
  240. each time a software client looks into it,
  241. so messages only appear on the client host that fetched them,
  242. they are unavailable from any other system located elsewhere.
  243. </P>
  244. <P>
  245. The second protocol to deal with email messages is IMAP, Internet Message Access Protocol.
  246. IMAP gives access to a hierarchy of mailboxes also called folders. Other IMAP features are
  247. concurrent accesses, tagging with flags, search by many criteriums like date, subject, size, etc.
  248. </P>
  249. <P>
  250. The IMAP protocol presents most of the features POP lacks.
  251. Messages stay on the imap server so any client on the network can access them
  252. at any time from anywhere, the same messages with the same flags.
  253. </P>
  254. <P>
  255. The third protocol to access email messages is HTTP, HyperText Transfer Protocol.
  256. HTTP is the protocol to browse the web.
  257. </P>
  258. <P>
  259. Web browsers like Google Chrome, Mozilla Firefox, Internet Explorer and Safari
  260. are HTTP client software tools. You already know that so what's the point with HTTP mailboxes?
  261. HTTP mailboxes are called webmails.
  262. Webmails often offer the same features as imap servers
  263. because webmails underlying storage systems are often imap servers.
  264. </P>
  265. <P>
  266. Webmails systems like Gmail, Yahoo, Exchange, Zimbra, or Office365 are also accessible via imap.
  267. </P>
  268. <P>
  269. The conclusion of this protocol review is that mailboxes can be
  270. accessed using the IMAP protocol, most of the time.
  271. Here comes imapsync.
  272. </P>
  273. <P>
  274. In case the source mailbox is only accessible by the POP protocol,
  275. you can use the tool <CODE>pop2imap</CODE> to sync them.
  276. <CODE>pop2imap</CODE> is located at <A HREF="http://www.linux-france.org/prj/pop2imap/">http://www.linux-france.org/prj/pop2imap/</A>
  277. </P>
  278. <A NAME="toc10"></A>
  279. <H1>4. Imapsync presentation</H1>
  280. <P>
  281. Software imapsync is a command-line tool to
  282. copy, migrate, backup, or synchronize IMAP mailboxes.
  283. </P>
  284. <P>
  285. Command line means imapsync is not graphical, it is textual.
  286. Usually, with command-line tools, you have to type characters
  287. on your keyboard. But your fingers won't suffer much pain
  288. typing on the keyboard because script examples are given,
  289. nearly ready to run. Most of the time you only have to change
  290. the main values in those files and adapt them to your context.
  291. </P>
  292. <P>
  293. Don't be afraid, the mouse won't be forsaken.
  294. You can still use the mouse to launch an editor,
  295. select/copy/paste complete examples,
  296. and run the little script with a double-click.
  297. </P>
  298. <P>
  299. Imapsync runs on Linux, Windows, and OS X (Macintosh/Darwin world).
  300. Imapsync is written in the Perl language and, thanks to the
  301. Perl developers, Perl runs mostly everywhere. So does imapsync.
  302. </P>
  303. <P>
  304. While operating systems have a lot in common, they sometimes differ,
  305. especially within their syntax. I won't blame anyone, historically Windows
  306. came after Unix. The marvelous designers in these old times
  307. decided it would be very cool to not share the same syntax
  308. for doing the same things.
  309. </P>
  310. <P>
  311. To designate an end of the line, Unix uses the character \n
  312. Windows uses two characters \r\n and Mac use \r.
  313. Thanks to you guys, great thinking!
  314. Fifty years later, we still suffer from this...daily.
  315. </P>
  316. <P>
  317. To avoid some headaches with systems that no one masters
  318. I will give examples in both worlds, Unix and Windows.
  319. OS X users are in the Unix world nowadays so they must follow
  320. the Unix examples.
  321. </P>
  322. <A NAME="toc11"></A>
  323. <H1>5. Conventions</H1>
  324. <P>
  325. To simplify display or print,
  326. each imapsync command line is usually written in several lines
  327. but it could be written in one single line.
  328. </P>
  329. <P>
  330. If you prefer to use the whole command written in one single line then
  331. just remove the last visible character of each line ( \ or ^ ) and
  332. also the carriage return character.
  333. The last visible character \ or ^ means "command continues on next line";
  334. it's the backslash \ character on Unix and the caret ^ character on Windows.
  335. </P>
  336. <P>
  337. For example, on Unix, a command like the following:
  338. </P>
  339. <PRE>
  340. imapsync \
  341. --host1 test.lamiral.info \
  342. --user1 test1 \
  343. --password1 secret1 \
  344. ...
  345. </PRE>
  346. <P>
  347. is equivalent to:
  348. </P>
  349. <PRE>
  350. imapsync --host1 test.lamiral.info --user1 test1 --password1 secret1 ...
  351. </PRE>
  352. <P>
  353. and on Windows, a command like the following:
  354. </P>
  355. <PRE>
  356. imapsync.exe ^
  357. --host1 test.lamiral.info ^
  358. --user1 test1 ^
  359. --password1 secret1 ^
  360. ...
  361. </PRE>
  362. <P>
  363. is equivalent to:
  364. </P>
  365. <PRE>
  366. imapsync --host1 test.lamiral.info --user1 test1 --password1 secret1 ...
  367. </PRE>
  368. <A NAME="toc12"></A>
  369. <H1>6. Why start with a test account on destination host2?</H1>
  370. <P>
  371. A little explanation about this hint.
  372. Imapsync is safe with accounts on host1,
  373. it doesn't change anything on them, it just read them.
  374. The exception of this safe principle is when <CODE>--delete1</CODE> option is used,
  375. since <CODE>--delete1</CODE> removes on host1 each message successfully copied to host2,
  376. messages that couldn't be transferred stay on host1.
  377. </P>
  378. <P>
  379. It's not the same for destination accounts as imapsync writes on host2 accounts.
  380. Imapsync creates folders on them, adds messages, and sets flags on messages.
  381. It isn't safe on a real account. So don't use a real user account
  382. to test imapsync. Learn to use it and see what it does on a test account at host2.
  383. </P>
  384. <P>
  385. What can badly happen? The most common bad behavior is
  386. that folders mapping won't be what you expect because it is strictly
  387. reproduced from host1 to host2. The second bad behavior is getting
  388. duplicates on the second run and after; it's rare but it can happen
  389. when an imap server software changes the headers "<CODE>Message-Id</CODE>" or "<CODE>Received</CODE>".
  390. Solutions to avoid duplicates are often easy (There's a FAQ called <CODE>FAQ.Duplicates.txt</CODE> about that).
  391. It's also possible to remove the duplicates on host2 but it's better to avoid them on user accounts at first,
  392. users won't like that you mess up their mailboxes.
  393. </P>
  394. <A NAME="toc13"></A>
  395. <H1>7. Imapsync default behavior</H1>
  396. <P>
  397. By default, unless explicitly told to do something else:
  398. </P>
  399. <UL>
  400. <LI>Imapsync <B>goes ssl or tls</B> if possible
  401. <LI>Imapsync syncs <B>all folders</B> of host1
  402. <LI>Imapsync syncs <B>all messages</B> from host1, except duplicates.
  403. <LI>Imapsync doesn't sync already synced messages.
  404. <LI>Imapsync syncs <B>all flags</B>, at least all allowed by host2.
  405. <LI>Imapsync resyncs flags on messages already synced.
  406. </UL>
  407. <A NAME="toc14"></A>
  408. <H1>8. To go further with imapsync</H1>
  409. <P>
  410. Imapsync has many options but you can ignore most of them
  411. and still make great transfers.
  412. </P>
  413. <UL>
  414. <LI>Option names all begin with two minus characters <CODE>--</CODE>, like <CODE>--automap</CODE> or <CODE>--dry</CODE> etc. (using one minus, like <CODE>-dry</CODE>, is ok)
  415. <LI>Option names relative to the <B>source</B> account are ended with the number one <B><CODE>1</CODE></B>, like in <CODE>--host1</CODE>
  416. <LI>Option names relative to the <B>destination</B> account are ended with the number two <B><CODE>2</CODE></B>, like in <CODE>--host2</CODE>
  417. <LI>Some options need a <B>value</B> just after them, like <CODE>--host1</CODE> <B><CODE>source.example.com</CODE></B>, (the value is <CODE>source.example.com</CODE>)
  418. <LI>Some options are standalone, like <B><CODE>--automap</CODE></B>
  419. <LI>Any order is possible but when an option needs a value then the value must follow immediately its option name.
  420. </UL>
  421. <!-- html code generated by txt2tags 2.6 (http://txt2tags.org) -->
  422. <!-- cmdline: txt2tags -i doc/TUTORIAL_Unix.t2t -t html -\-toc -o doc/TUTORIAL_Unix.html -->
  423. </BODY></HTML>