Exim servers removing sender message header

Once the ActionMailer base class was all patched up, I was able to add both the envelope sender and the message sender headers to outgoing messages. For test purposes I used the following simple test method:

def test(mail_to, mail_from)
subject “My simple rails mail”
recipients mail_to
from mail_from
sent_on Time.now
sender ” Accounts <accounts@mydomain.com>” #envelope and message header for SPF check
end

In addition, engine yard added an spf record to allow our messages to pass spf authentication checks:

“v=spf1 include72emailsrvr.com include72_spf.ey01.engineyard.com ~all”

Unfortunately, the messages I was sending myself as tests, once delivered, did not include the message sender header. Checking the production log confirmed that messages leaving our application included the message header.

After some troubleshooting, I discovered that the exim servers were stripping off the message sender header, as the message was being sent out This was because exim doesn’t allow non-local users to set the header (at least this seems to be a hard rule according to the documentation I’ve read). So as a workaround, we’re using another smtp provider that allows us to authenticate in, and then send mail. This is working great.

Ie. In the ActionMailer configuration:

ActionMailer::Base.smtp_settings = {
:domain => “yourdomain.com”,
:perform_deliveries => true,
:address => ‘smtp.yoursmtpserver.com’,
:port => 587,
:authentication => :login,
:user_name => “mailuser@yoursmtpserver.com” ,
:password => yourpassword
}

Where did my thawte generated .pvk go?

After going through the process of generating a microsoft authenticode signing certificate (spc) using Thawte, I was already to incorporate code signing into our build process. One problem. Where is my .pvk file?

The private key is generated during the certificate request process. Thawte never actually has access to your private key (this is obviously a good thing and by design). Unfortunately I originally went through the reqest process from my Vista dev. machine. Due to restricted security permissions,

Vista not only saves the private key into the registry (I knew this before hand and was ok with it), but it also meens that the private key can not be exported from this machine.

Thawte has a known issue article about this, but I’d failed to read it before hand.

The work around … re-issuing the certificate, but this time, placing the re-issue request from our build machine where the pvk is easily exportable.

Sending email on behalf of users – Part 2 – Patching ActionMailer

Our application was developed using Ruby on Rails, and leveraged RoR’s ActionMailer class to send emails. If you’re not using Rails, you can easily ignore the following.

Unfortunately, when using rails, perform_deliver_smtp of ActionMailer simply use mail.from as the envelope-from when setting the SMTP header. As mentioned earlier, we would like to take into consideration mail.sender if it exists. So I had to patch up the ActionMailer base class to specify the envelope-from as well as the Message sender header:

Once these changes were made, and I restarted our mongrel instance, I was able to set both the message and envelope sender.

Here’s the diff file of the changes to the base class that were needed:

Index: lib/action_mailer/base.rb

===================================================================

— lib/action_mailer/base.rb (revision 6287)

+++ lib/action_mailer/base.rb (working copy)

@@ -36,6 +36,7 @@

# * from – Who the email you are sending is from. Sets the From: header.

# * cc – Takes one or more email addresses. These addresses will receive a carbon copy of your email. Sets the Cc: header.

# * bcc – Takes one or more email address. These addresses will receive a blind carbon copy of your email. Sets the Bcc header.

+ # * sender – Takes just one email address. This address will be used to envelope-from of SMTP and appear in Sender: header.

# * sent_on – The date on which the message was sent. If not set, the header wil be set by the delivery agent.

# * content_type – Specify the content type of the message. Defaults to text/plain.

# * headers – Specify additional headers to be set for the message, e.g. headers 'X-Mail-Count' => 107370.

@@ -315,6 +316,9 @@

# header will be set by the delivery agent.

adv_attr_accessor :sent_on

+ # Specify the Sender address for the message

+ adv_attr_accessor :sender

+

# Specify the subject of the message.

adv_attr_accessor :subject

@@ -513,6 +517,7 @@

m.to, m.from = quote_any_address_if_necessary(charset, recipients, from)

m.bcc = quote_address_if_necessary(bcc, charset) unless bcc.nil?

m.cc = quote_address_if_necessary(cc, charset) unless cc.nil?

+ m.sender = sender unless sender.nil?

m.mime_version = mime_version unless mime_version.nil?

m.date = sent_on.to_time rescue sent_on if sent_on

@@ -548,16 +553,19 @@

def perform_delivery_smtp(mail)

destinations = mail.destinations

+ sender = mail.sender(nil) || mail.from

mail.ready_to_send

Net::SMTP.start(smtp_settings[:address], smtp_settings[:port], smtp_settings[:domain],

smtp_settings[:user_name], smtp_settings[:password], smtp_settings[:authentication]) do |smtp|

smtp.sendmail(mail.encoded, mail.from, destinations)

+ smtp.sendmail(mail.encoded, sender, destinations)

end

end

def perform_delivery_sendmail(mail)

IO.popen(“#{sendmail_settings[:location]} #{sendmail_settings[:arguments]}”,”w+”) do |sm|

+ arguments = sendmail_settings[:arguments].dup

+ arguments += ” -f#{mail.sender(nil)}” if mail.sender(nil)

+ IO.popen(“#{sendmail_settings[:location]} #{arguments}”,”w+”) do |sm|

sm.print(mail.encoded.gsub(/\r/, ”))

sm.flush

end

Sending email on behalf of users – Part 1

Before delving into the details of sending email on behalf of users, I thought I’d provide a bit of background information.

For simplicities sake, I’ll just scratch the surface when describing the headers of interest when sending messages out on behalf of users of your application. In my next posts, further description of these headers will be provided as needed.

Envelope sender – The envelope sender (return-to) address is, for the most part hidden from the user. This address receives bounces, and is typically, the address used for Sender Policy Framework (SPF) lookups, as well as domain key records.

Message sender – The message sender is also hidden from users more often than not. As I’ll discuss later, SenderID will generally use the Sender message header to authenticate the message. MUA’s such as hotmail and outlook (and I recon any other SenderID/Microsoft supported agent) expose the sender to the user. If you’ve ever received a message in outlook where you see something like “From John on behalf of Jane,” you’re seeing the effect of a Sender header. Ie. An application or mailing list has sent mail on behalf of Jane; so the message sender = John and the message from = Jane.

Message from – The message from is what is most often exposed to the user receiving the message. Surprisingly, although this header is what is exposed to an end user, it’s simple to impersonate. I quote from one of the users in the thread:

It is therefore, our goal to set the following headers in an effort to have our users receive email from users of our application as opposed to our application itself.

Envelope sender: invites@ourapplication.com

Message sender: invites@ourapplication.com

Message from: user@theiractualdomain.com

If we successfully set these headers, SPF checks will pass (assuming we create SPF records for ourapplication.com), SenderID checks will pass (again assuming we create SPF records), and users will receive invitations from actual users as opposed to our application.

As I found out, setting these headers, and having email land in the inbox of users was easier said than done. I’ll start to describe the steps needed to accomplish setting these headers in my following

Sending emails on behalf of users

Our application, much like many others, uses email as a means of communication between users. We recently made a design goal to send these emails from the users themselves as opposed to from our application. The decision itself seems to have been easier than the spam filters, spf records, domainkeys, and ActionMailer patch implementation details that ensued.

I’ll publish a list of articles that articulate the necessary technical steps to not only send email out on behalf of users (this is the easy part), but to get beyond spam filters, but for now I’m concentrating on motivations for sending out said emails as users.

The scenario that I’m considering is one familiar with users and developers of social networking sites. User John Doe invites Jane to use “My Application.” At this point, Jane will either receive an email from Accounts@MyApplication.com or else an email from John@hisdomain.com. I think it’s fairly obvious that a barrier of trust exists when users receive an invitation from some arbitrary domain owner …. such as “My Application,”regardless of how convincing the message is. Therefore I’m convinced that for virality, it’s ideal to invite users by using their email addresses, ie. John@hisdomain.com.

Before I begin to expand on implementation details in getting beyond spam filters, I must admit that at certain points, I began to question the nature of what we were doing. Part of me at times felt as though spam, and spoofing in particular, was exactly what we were trying to accomplish. I

In the end however, I am comforted and assured that what we are doing does not infringe on any privacy issues. Ultimately, users are intending on inviting others to collaborate with them. They intend to send a message to their friends or colleagues that they’d like to work with them; we enable these messages to be delivered in a natural way using the email addresses the users are working with.

What not to do when creating a software demo

I recently went through the pains of making a demo for an upcoming product. Perhaps the most obvious thing that I learned from this trial was that there is a reason people do this sort of thing professionally!

Needless to say, I took my lumps and in the end generated something that closely resembled a big steaming pile of poo for a demo. Fortunately, there will be a second crack … and we received some awesome feedback from Steven Blank, author of The Four Steps to Epiphany, a classic on bringing a product to market. Some lessons learned when making a demo:

4. Slow down
People watching the demo haven’t lived, breathed, and dreamed Attassa for the last year and a half. Chances are, it’ll take them a bit more time for things to sink in and make sense.

Screens need to be displayed for more than a fraction of a second, actions need to be explicitly pointed out, introduced, and explained.

3. Keep a running dashboard of benefits
One of the best suggestions that Steven had was to create a running list of benefits. As these benefits were being demonstrated, check them off of a list; making them painfully obvious. Clear Context does a nice job of the running dashboard of values.

2 . Create a demo scenario, not just a list of features to demonstrate
The most compelling demos are those that are able to show a real life usage that resonates with viewers. Without getting overly specific in any scenario, you can usually come up with one that is generic enough to convey value, while specific enough that the practicality of the software is beleviable.

1. The point of the demo is demonstrate value
It took me about 2 minutes to lose track of this and my engineering side to pop out as I proudly sliced together clips, transitioned audio, and in general, used every feature of the screen capturing software.