Offlineimap, msmtp, nmh, and all the hours I've lost

I didn't do much the past few days. I didn't code much, I didn't go on tilde.team, I didn't post anything on this blog. I'd like to say I was doing something productive, that I was very busy with something out of my control. But the truth is that I've been spending almost all of that time setting up at least 5 different programs just to send and receive mail.

What can I say. I like messing with things. And I like using the terminal. And neomutt was really slow for me and also would fail to login any time I pressed backspace while typing in my password. I ended up looking for better mail clients and I stumbled upon mh, and then nmh, and 2 days later I'm here, sleepily writing this blog as I tell you all this.

For all the time wasted, I do like nmh and the many things it provides. It's not a mail client per say, it's more of an email manager. basically what it does is take some mail you have in a certain maildir. Nmh would look threw all the mail inside it, and put it in its own special mailbox. From there, you can use commands like show, scan, and rmm to manage them. It's also able to compose and reply to messages. What's interesting is that all of the things I've just mentioned are implemented using many different small commands. For example, there is a command called show, which, no points for guessing correctly, shows your current message. The comp command composes a new message, and so on.

Of course, nmh is useless on its own. It doesn't know how to get mail from a imap server for example. So you need some way to get that mail onto your computer. Back in the day, there were special daemons just to fill up your system mailbox. Back then mh would just read your mail from the system mailbox and call it a day. Of course, we don't do email that way, but threw some hacks and a bunch of configuring we can make it work.

## First, set up your password
Your account most likely has a password with which you can log in. You will of course need it for nmh as well, or rather a few helpers.

However, just keeping the password in plain text on your computer isn't safe, and its recommended that you store your password in a encrypted file.

While there are many ways to achieve it, the most common one, and the one I went with, is to use gpg to encrypt the file. Here's a command you can use.

	echo some_password >password
	gpg --symmetric --output password.gpg password

On some systems, the command is called `gpg2`, instead of `gpg`. If your shell screams at you about some kind of command not found business try using `gpg2`, and if that fails just install it. The package is usually called gnupg or something similar. It's a rather interesting tool if you're into encrypting or decryption things, especially mail. Take a look at its [website](https://gnupg.org/) to learn more.

## msmtp

As you might know, email isn't a simple protocol, it's more of an ancient eldritch bundle of protocols each doing something that some other protocol does in a different way. But of course, you aren't interested in all of them, you're most likely only going to use smtp (for sending email), and imap (for retrieving).

Imap will be covered in the next section, for now let's get smtp sorted out with help of a little interesting program called smtp. Msmtp is a sendmail client. It was mostly used by mutt but since its a standalone command line application any program could use its services. Nmh is no exception, but we'll get there.

First, get msmtp. It's most likely available in your distros / package manager's repository. Just search, try out a  few commands, and if all that fails, just compile it from source. Trust me, it's probably not that hard. The developer has probably made it as easy as `./configure && make && sudo make install`.

Once you got it installed in whichever way, you're ready to start configuring it. Point your text editor at ~/.msmtprc, and fill it with the following:

	account accountname
	port 587
	tls on
	tls_trust_file /etc/ssl/something.crt
	host smtp.server.com
	from your@email.address
	auth on
	user your_username
	passwordeval gpg --no-tty -q -d ~/password.gpg

	account default : accountname
Fill out the fields as you need. Most of these should be self-explanatory, so I'll explain just a few:

- The `account` directives make... well... accounts. The special account called `default` is used by default.
- `tls_trust_file` should point to your tls cert file. On my macos machine that's /etc/ssl/cert.pem. On linux it might be different, although from what I've seen it's usually inside the /etc/ssl directory.
- passwordeval is a shell command used for getting the password. I use gpg, but you can use pretty much any shell command here. Just make sure that if you're using gpg like me ~/password.gpg is where your encrypted password actually is. Also note that gpg will prompt you for the password you used for encrypting your email password. While that hasn't caused any problems with me, the msmtp manual suggests that the command shouldn't "mess with standard input". That's something to keep in mind.

Now, it's time to make sure it works. At your command prompt, write this:
	msmtp your@email.address
	This is a test
Once you've finished typing out the test message, press ctrl+d, for end of file. You should use your own email here, because you don't want people getting weird messages from you. If msmtp didn't say anything, it probably worked. If it errored out, it didn't work. I can't cover all possible error conditions here, but if you get some, check you wrote your username, host and passwordeval fields correctly. If it says that your `tls trust file` or something like that can't be found, then check your `tls_trust_file` field for any possible errors. You can remove that field fully if you want msmtp to try finding the certs on its own, but that didn't work with me, so don't count on it.

You might also be interested in setting up tor, or any other kind of socks proxy. You only need 2 fields for that:
	proxy_host 127.0.0.1
	proxy_port 9050
With everything set up, it's time to switch to imap.

## offlineimap

Okay, you can send email, but what about receiving it? Well, you do that via imap. And offlineimap is a program you can do to store your imap folders offline. I personally store it in a format called maildir, but mbox works just as well.

First, get offlineimap. Same as the previous time. Check package repository, if it isn't there, compile from source.

Now, to configure offlineimap. Most of it should be simple and familiar:

	[general]
	accounts = accountname
	pythonfile = ~/.offlineimap.py
	
	[Account accountname]
	localrepository = Local
	remoterepository = Remote
	
	[Repository Local]
	type = Maildir
	localfolders = ~/maildir
	
	[Repository Remote]
	type = IMAP
	remotehost = imap.some.server
	remoteuser = your@email.address
	ssl = yes
	sslcacertfile = /etc/ssl/some_file.pem
	folderfilter = lambda f: f == "INBOX"
	remotepasseval = getpass()

Alright, this is a little bit more complex, but not by much. Essentially, what we're saying is that there is an account called accountname, which has a local repository called local and a remote repository called remote. We defined the local repository as a repository of type maildir and say where it is. You can put it wherever you want, as long as offlineimap and nmh can read it. best option for that is your home directory. the `sslcacertfile` option is what `tls_trust_file` was to msmtp.

The 3 weirder options are `pythonfile`, `folderfilter`, and `remotepasseval`. The `folderfilter` option is by far the simplest. It's just a python lambda which says which folders should be synced. I personally only sync my inbox folder, but you can do other things as well. But wait, where the fuck did python lambdas come from? Well, offlineimap is written in python, so it has some sence that it would use python for things like this. It's a shame python's lambdas have to be so gimped, but I can rant about python some other time.

Pythonfile is just a file whose contents get evaluated. In it you can put things like functions and other definitions to be used for your configuration. In here we can for example define our getpass() function like this:

	import os
	import subprocess
	def getpass():
		password=os.path.join(os.getenv("HOME"), "password.gpg")
		process=subprocess.Popen(["gpg", "--no-tty", "--batch", "--yes", "--passphrase-fd", "0", password], stdout=subprocess.PIPE, stdin.subprocess.PIPE, stderr=subprocess.PIPE)
		process.stdin.write("password_to_decrypt_password")
		process.stdin.close()
		process.wait()
		return process.stdout.read()

Of course, you probably want to edit this a little, perhaps include error handling or some different way of submitting the password. This is just an example to get you started. And oh by the way, if you thought using python was bad, you have to use python2.

To change topic a little, if you want to send your imap requests threw tor, you can do that very easily by putting the following at the end of the remote repository definition:
	proxy = SOCKS5:127.0.0.1:9050

Of course, you should test out everything by running offlineimap from your command line. If everything worked, you should get your imap inbox downloaded into ~/maildir. If something failed, make sure you typed everything correctly, and use a search engine when in doubt.

## nmh

Of course, now its time to set up the star of your setup, nmh. It's actually not as hard to set up, especially once you've got offlineimap and msmtp set up.

But first, you need to install it. Same drill different program. Although, I'd suggest you install nmh with your package manager instead of from source, since ladder gave me some issues. But again, your mileage may vary.

Once you've installed nmh, run the install-mh command. It won't do much, it will just make a directory (by default ~/Mail) for storing your mail, and a file ~/.mh_profile for your settings. Now you only have to set up your .mh_profile file, and everything should just work. Here is a template you can use, just fill it out with your info:

	MH-Profile-Version: 1.0
	Path: Mail
	MailDrop: maildir/INBOX/
	send: -mts sendmail/pipe -sendmail /usr/local/bin/msmtp
	Local-Mailbox: Your Name <your@email.address>
	repl: -format

This file is very small compared to some of the others. Something that you'll have to note is that you can't have empty lines inside mh_profile, and there are no comments.

The `Path` is the place where nmh stores messages. The default is ~/Mail, and that's fine with me. Feel free to change it.

Offlineimap puts all your inbox messages in $your_maildir/inbox, so that's what the value of `MailDrop` is. Nmh also uses the $MAILDROP environment variable to get this information, so you can also do

	export MAILDROP=~/maildir/INBOX

in your .bashrc for example. I personally do both.

The value of Local-Mailbox is what nmh puts in your from field by default. It should take the form of `your name <email>`.

The `send` field has some options to make nmh use msmtp for sending mail. It basically works by telling nmh to first pipe your message into sendmail, and then telling it to use msmtp instead of the actual sendmail program.

Finally, the `repl` field just tells nmh, specifically its repl command, to always use -format. What that means is that is should always quote the message you're responding to. Rather useful.

Now, just use `inc` to incorporate your mail, and commands like show and next for reading them. Use `comp` to write a new message, and `repl` to reply to an existing one. All of these commands should work just fine. Although, there is a weird bug where, if you try to send a message, you get something like:

	Killed: 9

That means that something bad happened. Best way to know what exactly happened is by running send from the command line and looking at the errors you get.  It doesn't get killed that way.

## troubleshooting

As I said, there are some errors you can get. Some might be because of you misstyping some values, others because I made a mistake here, others because of limitations in nmh. I'll list some of them:

- If your `Local-Mailbox` field contains unicode characters, for example cyrillic letters, you might not be able to send messages because mhbuild chokes on things like that. There are 2 ways around that. One is to not use unicode characters. Another option is to first mime-encode it and put the result in `Local-Mailbox`. There are many different ways of doing that. and I can't cover them all. however, you're output should look like `=?utf-8?b?<random_letters_and_numbers>=?=`

## conclusion

Thanks for reading this. Over all, I had a lot of fun setting this all up, and I hope you'll find it as useful as I did. Of course, there are changes that can be made. For example, using a special password manager instead of gpg would help. Or maybe you can go a little bit higher level and use some clients based on nmh, like [mh-e](https://mh-e.sourceforge.io/). The choice is yours. There are many different resources for learning more about nmh, including its man-pages.

Now, I think I'll rest a little, before going back to my projects.

tags: mail, nmh, smtp, imap