Apr 01, 2018

Email Clients able to use serverside full text search

Since using a (local) Dovecot server with Solr as its search backend, I was interested in the ability of an Email client to make accessible and prominent use of a fulltext serverside IMAP SEARCHยน. Surveying the landscape, the "webmail" clients fared better. From the classical Linux desktop clients, not one does global search yet, but still gives back results based on folders. Dovecot offers a virtual mailbox that combines all relevant folders as workaround.

Methodology

Number of interactions needed to execute a server-side full-text search, preferably global on all directories, including the keystroke-shortcut and mouse-click to focus the input field, entering of searchterm and the submission. If the client does not default to search for first keystroke and shows results without submission in parallel, 2 interactions are neccessary at minimum, 3 if you use the mouse instead of a keyboard-shortcut to focus.

Overview

At the time of writing, 4 of the tested clients offered support for remote-side search. Also checkout the related column in Wikipedias E-mail client comparison.

client type version server-side full-text scope steps needed
Thunderbird app 52.7.0 yes yes folder only 7
Geary app 0.10.0 no - - -
Sylpheed app 3.5.0 no - - -
Trojita app 0.7 yes yes folder only 4
Mulberry app - - - - -
mutt app - - - - -
Rainloop web 1.11.3 yes yes folder only 5
Roundcube web 1.3.4 yes yes global 6
Mailpile web 1.0.0rc2 - - - -
Mailspring web 1.1.4 no - - -

Desktop Clients

Thunderbird

  • to use IMAP SEARCH, go to "search dialog" and explicity activate "on server"
  • 3 dimensions on the menu-navigation (2 clicks, 1 dropdown), or 1 keyboard shortcut (ctrl+shift+f). Then 1 click selection of "search on server", and 2 click in dropdown to select "Body" of message. 6+1 for the final enter step
  • is it then a real global search? No, the message count was correct only for searches in a specific folder. For the "recusive account server-side search" the numbers didn't add up
  • at the time of writing, the builtin indexer "GLODA" is for advanced local search in an internally populated sqlite database, that should meet most users needs
  • the quicksearch form uses the current folder and header fields only, no remote search
  • see bug 564168 - "RFE: Option to run all searches on the server when using an IMAP server"
  • can use an external send command

1 2 3

Geary

  • searches the body in what it has locally cached from the IMAP server
  • so no global fulltext search possible at the moment? theres a 0.12 version, I should check that out

1

Sylpheed

  • no server-side search feature
  • local fulltext search is possible. It will download the IMAP Folders content to search it. 5 clicks to get there
  • can use an external send command

1 2

Trojita (Qt)

  • does use IMAP SEARCH per folder. Does not show message count, but reading from the debug window output it matches the expected numbers
  • no keyboard-shortcut to the search-input field, you need to select the search of message body and the option will not be remembered across restarts. So depending if you want to select all fields or raw imap data to search in, it's 4-5 interactions needed
  • feels like a IMAP protocol explorer that can also send mails. This is a compliment
  • can use an external command for sending (msmtp/sendmail)
  • install: for Debian/Ubuntu do echo 'deb http://download.opensuse.org/repositories/home:/jkt-gentoo:/trojita/xUbuntu_16.04/ /' | sudo tee /etc/apt/sources.list.d/trojita.list, import the key-id and save in trustdb with gpg2 --armor --export A6A36662 | sudo apt-key add -, update your apt sources and apt install libmimetic0

2

mutt(-sidebar)

  • not tested yet
  • there is also notmuchmail and offlineimap

Mulberry

  • the last 32bit release is many years old, you have to recompile. Didn't muster the patience yet to go through with it on a Ubuntu
  • denoted as most IMAP compatible client

Browser Driven / "Webmail"

Rainloop

  • fulltext imap search, but not global. The search is specific to the selected folder, so a virtual unified folder is neccessary. Prepend searchterm with "text:" or via two clicks in the interface. The keywords are approchable, use "from:name text:needle" to filter by sender or recipients quickly
  • does not report number of found matches
  • global search wish in issues/73 and user feature suggestion page
  • misreporting email date? see issues/1673
  • some controversy about the author, when employed by afterlogic writing a similar product
  • setup: follow the guide and apt install php7.0-curl php7.0-xml, enable localhost domain with username rewrite in settings, if you have msmtp configured, start with php -d 'sendmail_path="/usr/bin/msmtp -C ~/.msmtprc -a <account> -t" -S localhost:8080'

1

Roundcube

  • 3 additional interactions to global fulltext search: focus field, enter searchterm, click dropdown left to entry form, click checkbox ("mail body" or "complete message"), click radio-button ("all folders"), press enter
  • this can't be set via settings as default
  • setup: unzip tarball, install neccessary php extensions (or clone git and use composer). Finally run bin/jsdeps in Roundcube < 1.4 and the lessc instructions from INSTALL if >= 1.4

1 1 1

Mailpile

  • by design not build for a "remote search"
  • notably easy install by the Readme instructions

running it made me actually understand what mailpile is about. It's not a syncing imap client for now, but a "personal computing email client" - what makes for implications contrary to a remote search, as the remote would need plaintext messages. Beyond the html view it offers a commandline driven search interface too.

"So Mailpile now internally supports multiple search indexes. At the moment it only searches one at a time, and some search indexes are not very good at searching yet... but the code is elegant and clean, works well and has interesting potential for the future. [...]. Maybe someday we'll have hybrid search, which searches both remote IMAP servers and the local index. Maybe someday we'll be able to pull in results from notmuch or some alternate index."

1

Mailspring

  • no IMAP SEARCH feature, it uses local indexes
  • if you select the "Inbox" folder, all messages get searched. If you select a subfolder, the search will target the subfolder
  • sleek interface in electron
  • offers tracking mechanisms in the pro/paid-for plans, a little spying help for the casual business user?
  • chatty on the network due to npm package(s) asking for the public ip every other action, checking if it's within a captive portal and some other stuff I didn't go to the bottom of
    15:27:31.446837 IP maschine.router.box.45409 > router.box.domain: 21863+ A? captive.apple.com. (35)
    15:27:31.446907 IP maschine.router.box.50954 > resolver1.opendns.com.domain: 21428+ A? myip.opendns.com. (34)
    15:27:31.448780 IP maschine.router.box.45409 > router.box.domain: 29741+ A? ipv4.icanhazip.com. (36)
    15:27:31.463898 IP maschine.router.box.42396 > defra3-vip-bx-008.aaplimg.com.http: Flags [S], seq 914141383, win 29200, options [mss 1460,sackOK,TS val 734571 ecr 0,nop,wscale 7], length 0
    
    $ grep -rn icanhazip mailspring-1.1.4-amd64/usr/share/mailspring/resources
    
    ./unpack/node_modules/public-ip/browser.js:9:   v4: 'https://ipv4.icanhazip.com/',
    

1

Gimmick: Enron Dovecot Setup

Screenshots: email listings have boring work subjects and I made sure to only open work mail, if you find yourself therein and want to be removed please contact mail address in page source header (ctrl+u)

Enron was a massive case of economic fraud. The mail archive of the firm got aquired by a researcher after the investigation and continues to live an interesting afterlife. A documentary was made on the topic - Enron: The Smartest Guys in the Room.

Fetch the dataset tarball at the CMU page here and work in a extracted to directory enron/.

$ cd /home/user/tinker/enron

# create enron users
find . -maxdepth 1 -type d | cut -c3-  | sort -n | sed 's/$/:{plain}enron/g' > /home/user/tinker/enron/enron-passwd

# create Maildir layout
for d in $(find . -type d); do mkdir -p $d/{cur,tmp,new}; mv $d/*. $d/cur/; done
rename 's/sent/Sent/g' */*

# add folders to IMAP subscriptions
for d in *; do find $d -maxdepth 1 -type d | grep -Ev '^tmp$|^cur$|^new$|^inbox$' | cut -d'/' -f2 > $d/subscriptions; done

create the corresponding Dovecot config

$ cat /etc/dovecot/conf.d/auth-passwdfile.conf.ext

# Authentication for passwd-file users. Included from 10-auth.conf.

# https://wiki2.dovecot.org/VirtualUsers#static_userdb
# https://wiki2.dovecot.org/UserDatabase/Static
# https://wiki2.dovecot.org/AuthDatabase/PasswdFile#Examples

passdb {
  driver = passwd-file
  args = /etc/dovecot/enron-passwd
}

userdb {
  driver = static
  args = uid=1000 gid=1000 home=/home/user/tinker/enron/virtual/%n mail=maildir:/home/user/tinker/enron/virtual/%n mail=maildir:~/:LAYOUT=fs:INBOX=~/inbox
}

include the auth type

$ grep -r passwdfile /etc/dovecot

./conf.d/10-auth.conf:!include auth-passwdfile.conf.ext

PS

There's still movement in the Email space, stay tuned for Ivelope which will bring a chat view and Email based delta.chat relying on the upcoming autocrypt standard.