Configuring fetchmail to retrieve your mail from the Exchange servers

Traditionally, mail for UNIX and Linux machines is delivered directly to the local mail spool on the system using SMTP (Simple Mail Transfer Protocol). The spool usually lives in a directory such as /var/spool/mail, usr/spool/mail, /var/mail, etc and typical UNIX/Linux mail software reads incoming mail from this spool. Direct SMTP mail feeds work well where systems have a permanent network connection and are powered up all the time, as is usually the case in academic institutions and commercial environments, as they will be able to receive incoming mail at any time.

But the growth in popularity of the Internet in the mid-1990's amongst home computer users - the vast majority of whom were using Windows PCs and Macintoshes which do not support incoming SMTP mail - to access the 'Net using dial-up modems inspired the development of alternative mail delivery protocols where instead of having mail pushed to them at any time of day or night, users could instead elect to store their mail on a designated mail server and pull it down at a time when it suited them. One such protocol is POP (Post Office Protocol), so named because it works in much the same way as a rural postal service; in many remote country districts, there are no postmen to bring your mail to your doorstep and you need to visit the local post office every so often to collect your mail. Another protocol is IMAP, a more powerful protocol that can be used for things other than mail. Home computer users can connect to the Internet via their modem and use a POP or IMAP-capable mail client to retrieve any mail waiting for them from their mailbox, usually on their ISP's POP or IMAP mail server.

As a result of the overwhelming popularity of POP and IMAP, very few ISPs bother to offer SMTP inbound mail facilities to their customers, even to those with broadband connections or those using UNIX-type systems at home. This caused a problem for these users as most UNIX mail software relies on mail being delivered to a local mail spool. To compensate, POP and IMAP facilities were added to many existing UNIX mail clients and to the mail component of web browsers like Netscape, Mozilla, Opera and new mail clients like Evolution and Thunderbird already include POP/IMAP facilities.

However, some of the functionality of mail clients like pine, elm and mutt is lost when using POP or IMAP; pine for example sets flags to indicate the status of a message - whether it is new, answered, awaiting deletion, etc - and it does this by inserting the flags into the message within the SMTP mail spool file. But when the mail resides on a remote POP server, although it can set the flags in memory during the session, it cannot save them on the POP server as it has no write access to the mail file on the server. So if the pine session is closed and then restarted at a later date, all the message flags are lost and the status of all messages reverts back to 'new'. For this reason and for those used to the continuous, seamless mail delivery of SMTP who found it irksome to use POP and IMAP mail clients - what was needed was a way of automating POP/IMAP mail retrieval to collect mail from a POP or IMAP server and and putting it into the local system's mail spool, in effect converting it back into an SMTP feed.

Enter fetchmail.

About fetchmail

fetchmail can connect to a mail server using a variety of mail protocols, retrieve mail for you and place it in your mail spool. It can be run as and when required in a 'single shot' fashon or it can operate continuously, polling the mail server at regular intervals and retrieving your mail. You can leave it running continuously and forget all about it and in effect, you are back in the good old days when you got your mail from a real UNIX-like mail server via SMTP.

Configuring fetchmail

fetchmail is configured using the .fetchmailrc file which stores the user's personal settings defining the address of the mail servers he/she wishes to use, usernames and passwords and a variety of other options. On some systems, a utility called fetchmailconf is installed which will create a .fetchmailrc configuration file for you. However, these notes will describe a typical simple set-up that can be easily implemented on any system and the example configuration file shown below will work in most instances.

set postmaster 'yourlocalusername'
set bouncemail
set no spambounce
set properties ""
set daemon 60
poll exchange.ic.ac.uk with proto POP3
       user 'yourICusername' there with password 'yourICpassword' is 'yourlocalusername' here

Important Update: Security changes for Imperial College mail servers on March 4th, 2008

From this date, the college mail servers will no longer support incoming mail retrieval using plain, unencrypted POP3 and IMAP4 mail protocols and they will be reconfigured to use only POP3 over SSL or IMAP4 over SSL. You will need to use the updated .fetchmailrc configuration file as shown below which enables the SSL functionality of fetchmail.

set postmaster 'yourlocalusername'
set bouncemail
set no spambounce
set properties ""
set daemon 60
poll exchange.imperial.ac.uk with proto POP3
       user 'yourICusername' there with password 'yourICpassword' is 'yourlocalusername' here ssl

Using your favourite text editor, create the file shown above in your home directory and replace:

• 'yourICusername' with your email account username

• 'yourICpassword' with the password you use for your email account

• 'yourlocalusername' with the username on the computer you are running fetchmail on (which might not be the same as your email username).

and save it as .fetchmailrc. Now since it contains your email server password, you must ensure that no-one else can read this file - set the permissions to -rw-------:

chmod 600 .fetchmailrc

Now start up fetchmail as a daemon that runs in the background every n seconds with:

fetchmail -dn

For example, if you want to check your mail every 15 minutes (900 seconds), then you would start fetchmail with:

fetchmail -d900

How often you check for incoming mail is up to you - anything under 60 seconds is probably excessive and you may get a rebuke from the ICT Exchange server manager as each fetchmail connection does use resources on the server.

The fetchmail configuration example shown assumes that you wish to delete your email from the Exchange server once it has been retrieved but this may not suit all users; if you want to leave the mail on the server, simply add the 'keep' keyword to the end of the line in your .fetchmailrc file that starts with the word 'poll' like this:

poll exchange.imperial.ac.uk with proto POP3 user 'yourICusername' there with password 'yourICpassword' is 'yourlocalusername' here ssl keep

Pine configuration for use with fetchmail

If you are using fetchmail, you simply use the standard pine configuration which looks for incoming mail in your system's mail spool. Just make sure that the inbox-path in the pine configuration is not set - pine will default to using the value of the system's inbox environment variable which will point to your mail spool; for most Linux systems, this will be /var/spool/mail but Debian and Debian-derived Linux systems and Solaris/Open Solaris use /var/mail while Digital/Compaq/HP Tru64 UNIX uses /usr/spool/mail. Here's a screenshot of a typical pine configuration with the inbox configuration highlighted:

Broadband ISP's offering inbound-SMTP

Althought distantly-related to fetchmail, I am sometimes asked which broadband providers actually offer inbound SMTP mail services and/or static IP addresses to their customers. There are four I know about:

Demon Internet

Newnet plc

BT Business Broadband

Pipex Business Broadband

and these will give you one or more static IP addresses as well as ensuring that port 25 (the port used for SMTP traffic) is unblocked in both directions. This allows you to set up your own mail server at home, say, which in turns allows you to have unlimited mail addresses for your own personal domains, etc. Newnet have a policy of not blocking any port - even on their domestic/consumer type broadband packages - and you are allowed to run web servers, gaming servers, etc over your home broadband connection. And having a static IP address that never changes - even if you are disconnected from the broadband network for a period of time - has many advantages including simplifying VPN, using certain services that control access by IP address, accessing system(s) in your home from outside, etc.