IMAPsync/INSTALL.d/INSTALL.OnlineUI.txt

#!/bin/cat 
# $Id: INSTALL.OnlineUI.txt,v 1.56 2022/05/19 08:55:50 gilles Exp gilles $

This documentation is also located online at 
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.OnlineUI.txt

=======================================================================
               Installing imapsync online
=======================================================================


I'm now confident with /X since the /X service is up and running quite
well since January 2017. Anyway, if you run this service on your own,
online, you take responsibility for it.

=======================================================================
                    Hardware consideration

RAM used per imapsync process, mean value: 230 MB.
Average_bandwidth_rate: 345 KiB/s ~ 2.8 Mbps.
Load mean: 0.8 on a CPU 4 cores "Intel(R) i5-2320 3.00GHz K8-class"


=======================================================================
                    Installation

You have to be a little familiar with what a CGI script is and how to
activate a CGI script on the Apache HTTP server, or any other HTTP
server. I have received demands to run it on the Ngnix HTTP server but
I haven't played with it yet.  Linux is also a preferred platform (I
run /X service on Linux and FreeBSD).

I have tested this visual interface on Mac. It works.  For now, it
demands some skills few Mac users have.  Drop me a note in case you
want to do that.

I have tested this visual interface on Windows, it fails on Windows
because of some hardcoded Unix paths.  I'm working on it to be Windows
ok but it's not done yet (May 2020).

Some users have successfully installed a /X visual interface on
Windows using a Linux VM machine.

The web visual user interface frontend /X is compounded in four files:
a html5 file, a CSS file, a javascript file, and a logo image:

* https://imapsync.lamiral.info/X/imapsync_form_extra.html
* https://imapsync.lamiral.info/X/imapsync_form.css
* https://imapsync.lamiral.info/X/imapsync_form.js
* https://imapsync.lamiral.info/X/logo_imapsync_Xn.png


You can do a "view source" to see the HTML file as it is written, and
a "save" to get it locally.  The three other files can be saved the
same way or with a command named "wget". I strongly suggest using
wget, see below the ready-to-use command lines.

Those four files can be put anywhere on a web server, as long as they
stand in the same directory. If you want to put them in different
directories, just change the content of imapsync_form_extra.html to
reflect the change, ie, change the two lines referencing
imapsync_form.css and imapsync_form.js href="imapsync_form.css" (near
the beginning of imapsync_form_extra.html) src="imapsync_form.js"
(near the end of imapsync_form_extra.html) I let you change the image
logo as an exercise, it's safe if you fail.

The actual imap syncing work is done by imapsync acting as a CGI, the
visual interface is only there to give imapsync the parameters needed
for the sync.

Use at least Perl module CGI.pm release 4.08 (2014-10-18) to avoid the
bug "Undefined subroutine CGI::multi_param".  You can use the command
named cpanm to upgrade CGI.pm to its last version, it's the easiest
way.

Print the CGI.pm release with:

  perl -MCGI -e 'print "$CGI::VERSION\n"'

If it is under release 4.08 (2014-10-18) then upgrade it with

  cpanm CGI

It is a good thing to remove the old one if it was installed by a
distribution package, I let you this part as an exercise too. Ok, here
is a way:

  apt remove libcgi-pm-perl # for the Debian family
  dnf remove perl-CGI       # for the Centos family

To check and fix the Perl modules dependencies, run:

  cd
  wget -N https://imapsync.lamiral.info/prerequisites_imapsync
  sh prerequisites_imapsync

To make imapsync work as a CGI script, there are three conditions.

First, imapsync has to work by itself on the web host.  If imapsync
doesn't work by itself, as a command line, then it won't work as a CGI
script.

Second, imapsync has to work by itself on the web host using the Unix
user running the webserver.

Third, the file imapsync has to be considered as a cgi script.

Command lines to provide and verify those three conditions will be
provided further in this document, for the Debian family systems and
for the Centos family systems. You are strongly advised to follow this
commands if need and want help from me because I will first ask you to
run them before searching what you did wrong.

The imapsync_form_extra.html file in action calls the CGI location
/cgi-bin/imapsync
which has to be imapsync itself, the file script (not the directory).

The very latest and relatively stable imapsync is
https://imapsync.lamiral.info/imapsync
This file is the program file used verbatim for the service given at
https://imapsync.lamiral.info/X/

Copy the three files imapsync_form.* on a directory that is exported
by your HTTP server.

Copy the imapsync script on the cgi-bin/ directory
allowing CGIs and you'll have your imapsync visual interface
and service. The cgi-bin/ directory is usually outside the
hierarchy exported to anybody by the HTTP server.

The default Apache 2.4 timeout is 60 seconds, one minute, and 300
secondes for older Apache, 5 minutes.  See
https://httpd.apache.org/docs/2.4/mod/core.html#timeout

I use "Timeout 3600", 3600 seconds, an hour. I chose this huge timeout
value because imapsync can spend a long time without talking while
getting the headers of huge folders of 100k messages.  If you intend
to offer this service for huge mailboxes or for a long time, I
strongly recommand you to set this "Timeout 3600" in the Apache
configuration right now because you will sure end up with this timeout
issue in a few months. You can search for timeouts in the Apache error
log to see if you have timeout issues.

Now that I have explained the general context for any system, I'll
describe concrete examples on several systems, Debian/Ubuntu and
Centos. Feedbacks show that the Centos process is easier in case you
don't know very much any Linux distribution. But I have to add that if
you don't know very much the Linux distribution you use, then you
shouldn't install this imapsync service at all.


=============================================================================
A) Concrete example on a Debian server with Apache:

First, install Apache on your Debian system:

  apt install apache2

Imapsync script place on the server disk will be
/usr/lib/cgi-bin/imapsync

This classical /cgi-bin directory is usually already configured 
in the Apache configuration file
/etc/apache2/sites-available/default-ssl
or
/etc/apache2/sites-available/default

This configuration file contains the following section
somewhere, maybe in comments for now, ie, with 
some # characters at the beginning to make them ignored.
If you don't find the following section, keep reading,
the solution is below.

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
    AllowOverride None
    # Next line "no-gzip 1" is to avoid output buffering, 
    # clients can then see the log during the sync
    SetEnv no-gzip 1
    Options +ExecCGI -MultiViews
    
    # Choose either one or the other, depending on your Apache version
    # Lines beginning with # are ignored
    
    # For Apache 2.2
    #Order allow, deny
    #Allow from all
    
    # Apache 2.4
    Require all granted
</Directory>

In recent Debian distributions you can activate this cgi 
stuff with the following commands:

  a2enmod cgi
  a2enconf  serve-cgi-bin
  /etc/init.d/apache2 restart

If the cgi mode and the cgi-bin configuration are not activated then
you may encounter a 404 error when, later, you will run the command
wget -nv -S -O-  http://localhost/cgi-bin/imapsync?testslive=1

That's all for the Apache Debian family configuration side.

Now get, test, and install the latest imapsync:

  cd
  wget -N https://imapsync.lamiral.info/imapsync
  chmod +x imapsync

  # some basic tests
  ./imapsync 
  ./imapsync --testslive

  cp imapsync /usr/lib/cgi-bin/

Assuming that the Unix account running Apache is www-data, check that
it will work under Apache with this command:

  su -s /bin/sh -c 'SERVER_SOFTWARE=foo /usr/lib/cgi-bin/imapsync' www-data

You should end with something like:
Exiting with return value 0 (EX_OK: successful termination)

Test that imapsync is considered a cgi with: 

  wget -nv -S -O-  http://localhost/cgi-bin/imapsync?testslive=1

The last command should print something like:
Status: 200 OK to sync IMAP boxes. Load on bar is ...
...
Exiting with return value 0 (EX_OK: successful termination)

If you get a 404 or a 5xx here then review the cgi installation and
configuration part.


You can also verify that the webserver is not buffering its output with the 
command:

  wget -nv -S -O-  'http://localhost/cgi-bin/imapsync?testslive=1&simulong=10'

You should get the output as time goes on. If you don't get the output
as time goes on, ie you see no output then all output at once, it
means the webserver is buffering. Fix it with the "SetEnv no-gzip 1"
described above.


The UI front-end file place on the server disk in this example is
/var/www/html/X/imapsync_form_extra.html
but it can be placed anywhere on the disk, the important thing is that
it has to be served by the webserver.

  mkdir /var/www/html/X/
  cd /var/www/html/X/
  wget -N \
  https://imapsync.lamiral.info/X/imapsync_form_extra.html \
  https://imapsync.lamiral.info/X/imapsync_form.css \
  https://imapsync.lamiral.info/X/imapsync_form.js \
  https://imapsync.lamiral.info/X/logo_imapsync_Xn.png
  ln -s imapsync_form_extra.html index.html

The imapsync process working directory in cgi mode is
/var/tmp/imapsync_cgi/
it is not configurable unless changing it in imapsync directly, it is
hard-coded in imapsync.  In this directory will go the log files and
the pid files.

Check 
   http://yourhost/X/imapsync_form_extra.html
or the safer
  https://yourhost/X/imapsync_form_extra.html

Let's encrypt your site because crendentials should never travel in
clear form. Go to https://certbot.eff.org/instructions

See the Troubleshooting section to fix the systemd Apache 
PrivateTmp=true issue.

That's all for installing a /X service on a Debian family system.

=============================================================================
B) Here is a concrete example on a Centos 7 server with the Apache
   webserver httpd:

First, follow and apply the section "Centos 7 and latest imapsync" at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Centos.txt

Then:

  yum install httpd
  systemctl restart httpd

  cpanm CGI

  mkdir /var/www/html/X/
  cd /var/www/html/X/
  wget -N \
  https://imapsync.lamiral.info/X/imapsync_form_extra.html \
  https://imapsync.lamiral.info/X/imapsync_form.css \
  https://imapsync.lamiral.info/X/imapsync_form.js \
  https://imapsync.lamiral.info/X/logo_imapsync_Xn.png
  ln -s imapsync_form_extra.html index.html
  
  cd
  wget -N https://imapsync.lamiral.info/imapsync
  chmod +x imapsync
  # some basic tests
  ./imapsync 
  ./imapsync --testslive

  cp imapsync /var/www/cgi-bin/

Assuming that the Unix account running Apache is "apache",
which is the default Apache user on Centos system,
check that it will work under Apache with this command:

# a real synchronization but not in cgi context
  cd /tmp
  su -s /bin/sh -c '/var/www/cgi-bin/imapsync --testslive' apache
  
# in cgi context but just the imapsync command with no parameter
  cd 
  su -s /bin/sh -c 'SERVER_SOFTWARE=foo /var/www/cgi-bin/imapsync' apache
  
# a real synchronization in cgi context
  wget -nv -S -O-  http://localhost/cgi-bin/imapsync?testslive=1

The last command should print something like:
Status: 200 OK to sync IMAP boxes. Load on bar is ...
...

You can also verify that the webserver is not buffering its output
with the command:

  wget -nv -S -O-  'http://localhost/cgi-bin/imapsync?testslive=1&simulong=10'

You should get the output as time goes on. If you don't, no output
then all output at once, it means the webserver is buffering. Fix it
with the "SetEnv no-gzip 1" described above.

Now check 
   http://yourhost/X/imapsync_form_extra.html
or the safer
  https://yourhost/X/imapsync_form_extra.html

That's all for installing a /X service on Centos 7.


B bis) How about Centos 8?

Follow the procedure for Centos 7. While imapsync is ok on the command
line, you will encounter some permission denied in the CGI
context. Something like:

wget -nv -S -O-  http://localhost/cgi-bin/imapsync?testslive=1
...
Host1 failure: can not open imap connection on host1
[test1.lamiral.info] with user [test1]: Unable to connect to
test1.lamiral.info: Permission denied

The issue might come from SELinux. I haven't dig into SELinux enough
to give you the commands that will allow imapsync online and only it
while maintaining SELinux in enforcing mode.

Quick solution:

   getsebool    httpd_can_network_connect    # should show --> off
   setsebool -P httpd_can_network_connect=1
   getsebool    httpd_can_network_connect    # should show --> on
   wget -nv -S -O-  http://localhost/cgi-bin/imapsync?testslive=1  # no more "Permission denied"

The -P option installs the rule permanently, even after a reboot

To go back to the previous state:

   getsebool    httpd_can_network_connect    # should show --> on
   setsebool -P httpd_can_network_connect=0
   getsebool    httpd_can_network_connect    # should show --> off
   

Nota bene
=========

You may also want to avoid being placed by systemd in a directory like
(where xxx are crypto hash characters):

/var/tmp/systemd-private-xxx-httpd.service-xxx/tmp/

In that case, see the Troubleshooting section below.


=======================================================================
===================   Bandwidth statistics   ==========================
=======================================================================

If you want the bandwidth statistics like the ones at the bottom of
the page and following the image link, more detailed at
https://imapsync.lamiral.info/vnstat/vnstati.html

Those stats are generated by vnstat
https://humdi.net/vnstat/

Vnstat is already available as a package in most Linux distros.

The images are generated by the following commands, every minute:
vnstati -s  -o /var/www/html/vnstat/vnstat_s.png
vnstati -h  -o /var/www/html/vnstat/vnstat_h.png
vnstati -hg -o /var/www/html/vnstat/vnstat_hg.png
vnstati -hs -o /var/www/html/vnstat/vnstat_hs.png
vnstati -d  -o /var/www/html/vnstat/vnstat_d.png
vnstati -m  -o /var/www/html/vnstat/vnstat_m.png
vnstati -y  -o /var/www/html/vnstat/vnstat_y.png
vnstati -t  -o /var/www/html/vnstat/vnstat_t.png
vnstati -vs -o /var/www/html/vnstat/vnstat_vs.png
vnstati -5  -o /var/www/html/vnstat/vnstat_5.png




=======================================================================
======================   Troubleshooting   ============================
=======================================================================

The log says the temporary directory is 
/var/tmp/imapsync_cgi/
but this directory is not in the system. What a mystery!

It may be that the apache or httpd service is run by systemd with a
jailed temporary directory.

Solution:

  find /etc/systemd/ /usr/lib/systemd/ | xargs grep -s PrivateTmp

If systemd jails Apache, then you'll find a line like:
/etc/systemd/system/multi-user.target.wants/apache2.service:PrivateTmp=true
(Debian/Ubuntu)
or
/usr/lib/systemd/system/httpd.service:PrivateTmp=true
(Centos)

The goal is to override the line

PrivateTmp=true

found in /etc/systemd/system/multi-user.target.wants/apache2.service 
or
/usr/lib/systemd/system/httpd.service

with the line:

PrivateTmp=false

The right way to do it is by using the "systemctl edit ..." command
and then reload the systemd daemon and restart the apache2 service.
You can also edit directly the file override.conf if you know where
to do it. If you don't use the override.conf mechanism then your change will 
be canceled the next time the apache package is updated.

Debian:
  systemctl edit apache2

  cat /etc/systemd/system/apache2.service.d/override.conf
[Service]
PrivateTmp=false

  systemctl daemon-reload
  systemctl restart apache2
  systemctl status  apache2

Centos:
  systemctl edit httpd

  cat /etc/systemd/system/httpd.service.d/override.conf
[Service]
PrivateTmp=false

  systemctl daemon-reload
  systemctl restart httpd
  systemctl status  httpd

Then retry 

  wget -nv -S -O-  http://localhost/cgi-bin/imapsync?testslive=1

Look now if /var/tmp/imapsync_cgi/ is there.

=======================================================================
If you encounter this issue:

Failed to find a valid digest in the 'integrity' attribute for resource 
'https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js' 
with computed SHA-256 integrity 'kZMXypKF3if9/5v2tP9UHBvS/535tSyH7vjszruyCso='. 
The resource has been blocked.

It may be because of AdBlock.

Verification:

  wget https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js
  cat jquery.min.js | openssl dgst -sha384 -binary | openssl base64 -A

gives exactly what is in imapsync_form_extra.html

  more imapsync_form_extra.html
...
src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"
integrity="sha384-xBuQ/xzmlsLoJpyjoggmTEz8OWUFM0/RC5BsqQBDX2v5cMvDHcMakNTNrHIW2I5f"

So if your https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js
is not what it should be, your access looks compromised.

Thanks to Dominik Ulrich for this insight!

=======================================================================
=======================================================================

This part is for hackers only.

If you want to use the UI but make it more complicated things than
just run imapsync then use the following files:

imapsync_shell_wrapper      instead of imapsync itself 
imapsync_form_wrapper.js    instead of imapsync_form.js
imapsync_form_wrapper.html  instead of imapsync_form.html

How to get those files:

wget -N https://imapsync.lamiral.info/X/imapsync_shell_wrapper \
  https://imapsync.lamiral.info/X/imapsync_form_wrapper.js \
  https://imapsync.lamiral.info/X/imapsync_form_wrapper.html

Centos:
  chmod +x imapsync_shell_wrapper
  cp imapsync_shell_wrapper /var/www/cgi-bin/

Debian:
  chmod +x imapsync_shell_wrapper
  cp imapsync_shell_wrapper /usr/lib/cgi-bin/

Normally, you only have to change the script imapsync_shell_wrapper to
suit your needs.

Have in mind that the abort button will kill only one imapsync so it
is not a working button in case of successive imapsync runs.

=======================================================================
=======================================================================


====== mod_perl failure ======

This part is for mod_perl experts only.

The script imapsync doesn't work under Modperl::Registry nor under
ModPerl::PerlRun. So read on if you think you are better than me.

I tried the standard way, telling how any cgi Perl script can be run
under mod_perl perlrun, but it fails with imapsync.  Any hint welcome!

# This is a Debian example

# install mod-perl with

  apt-get install libapache2-mod-perl2

# edit the file /etc/apache2/mods-available/perl.conf
# with the following lines
  more /etc/apache2/mods-available/perl.conf

<IfModule mod_perl.c>
  PerlModule ModPerl::PerlRun
  Alias /perl-run/ /usr/lib/cgi-bin/
  <Location /perl-run>
      SetHandler perl-script
      PerlResponseHandler ModPerl::PerlRun
      PerlOptions +ParseHeaders
      Options +ExecCGI
  </Location>
</IfModule>

# Enable the Apache perl module

  a2enmod perl

# Verify perl.conf and perl.load are in directory mods-enabled/

  ls mods-enabled/perl.*

# Reload Apache

  apachectl graceful

# Verify imapsync works under perl-run

  curl http://localhost/perl-run/imapsync

=======================================================================
=======================================================================