Month: March 2008

Exim servers removing sender message header

Once the ActionMailer base class was all patched up, I was able to add both the envelope sender and the message sender headers to outgoing messages. For test purposes I used the following simple test method:

def test(mail_to, mail_from)
subject “My simple rails mail”
recipients mail_to
from mail_from
sent_on Time.now
sender ” Accounts <accounts@mydomain.com>” #envelope and message header for SPF check
end

In addition, engine yard added an spf record to allow our messages to pass spf authentication checks:

“v=spf1 include72emailsrvr.com include72_spf.ey01.engineyard.com ~all”

Unfortunately, the messages I was sending myself as tests, once delivered, did not include the message sender header. Checking the production log confirmed that messages leaving our application included the message header.

After some troubleshooting, I discovered that the exim servers were stripping off the message sender header, as the message was being sent out This was because exim doesn’t allow non-local users to set the header (at least this seems to be a hard rule according to the documentation I’ve read). So as a workaround, we’re using another smtp provider that allows us to authenticate in, and then send mail. This is working great.

Ie. In the ActionMailer configuration:

ActionMailer::Base.smtp_settings = {
:domain => “yourdomain.com”,
:perform_deliveries => true,
:address => ‘smtp.yoursmtpserver.com’,
:port => 587,
:authentication => :login,
:user_name => “mailuser@yoursmtpserver.com” ,
:password => yourpassword
}

Where did my thawte generated .pvk go?

After going through the process of generating a microsoft authenticode signing certificate (spc) using Thawte, I was already to incorporate code signing into our build process. One problem. Where is my .pvk file?

The private key is generated during the certificate request process. Thawte never actually has access to your private key (this is obviously a good thing and by design). Unfortunately I originally went through the reqest process from my Vista dev. machine. Due to restricted security permissions,

Vista not only saves the private key into the registry (I knew this before hand and was ok with it), but it also meens that the private key can not be exported from this machine.

Thawte has a known issue article about this, but I’d failed to read it before hand.

The work around … re-issuing the certificate, but this time, placing the re-issue request from our build machine where the pvk is easily exportable.

Sending email on behalf of users – Part 2 – Patching ActionMailer

Our application was developed using Ruby on Rails, and leveraged RoR’s ActionMailer class to send emails. If you’re not using Rails, you can easily ignore the following.

Unfortunately, when using rails, perform_deliver_smtp of ActionMailer simply use mail.from as the envelope-from when setting the SMTP header. As mentioned earlier, we would like to take into consideration mail.sender if it exists. So I had to patch up the ActionMailer base class to specify the envelope-from as well as the Message sender header:

Once these changes were made, and I restarted our mongrel instance, I was able to set both the message and envelope sender.

Here’s the diff file of the changes to the base class that were needed:

Index: lib/action_mailer/base.rb

===================================================================

— lib/action_mailer/base.rb (revision 6287)

+++ lib/action_mailer/base.rb (working copy)

@@ -36,6 +36,7 @@

# * from – Who the email you are sending is from. Sets the From: header.

# * cc – Takes one or more email addresses. These addresses will receive a carbon copy of your email. Sets the Cc: header.

# * bcc – Takes one or more email address. These addresses will receive a blind carbon copy of your email. Sets the Bcc header.

+ # * sender – Takes just one email address. This address will be used to envelope-from of SMTP and appear in Sender: header.

# * sent_on – The date on which the message was sent. If not set, the header wil be set by the delivery agent.

# * content_type – Specify the content type of the message. Defaults to text/plain.

# * headers – Specify additional headers to be set for the message, e.g. headers 'X-Mail-Count' => 107370.

@@ -315,6 +316,9 @@

# header will be set by the delivery agent.

adv_attr_accessor :sent_on

+ # Specify the Sender address for the message

+ adv_attr_accessor :sender

+

# Specify the subject of the message.

adv_attr_accessor :subject

@@ -513,6 +517,7 @@

m.to, m.from = quote_any_address_if_necessary(charset, recipients, from)

m.bcc = quote_address_if_necessary(bcc, charset) unless bcc.nil?

m.cc = quote_address_if_necessary(cc, charset) unless cc.nil?

+ m.sender = sender unless sender.nil?

m.mime_version = mime_version unless mime_version.nil?

m.date = sent_on.to_time rescue sent_on if sent_on

@@ -548,16 +553,19 @@

def perform_delivery_smtp(mail)

destinations = mail.destinations

+ sender = mail.sender(nil) || mail.from

mail.ready_to_send

Net::SMTP.start(smtp_settings[:address], smtp_settings[:port], smtp_settings[:domain],

smtp_settings[:user_name], smtp_settings[:password], smtp_settings[:authentication]) do |smtp|

– smtp.sendmail(mail.encoded, mail.from, destinations)

+ smtp.sendmail(mail.encoded, sender, destinations)

end

end

def perform_delivery_sendmail(mail)

– IO.popen(“#{sendmail_settings[:location]} #{sendmail_settings[:arguments]}”,”w+”) do |sm|

+ arguments = sendmail_settings[:arguments].dup

+ arguments += ” -f#{mail.sender(nil)}” if mail.sender(nil)

+ IO.popen(“#{sendmail_settings[:location]} #{arguments}”,”w+”) do |sm|

sm.print(mail.encoded.gsub(/\r/, ”))

sm.flush

end

Sending email on behalf of users – Part 1

Before delving into the details of sending email on behalf of users, I thought I’d provide a bit of background information.

For simplicities sake, I’ll just scratch the surface when describing the headers of interest when sending messages out on behalf of users of your application. In my next posts, further description of these headers will be provided as needed.

Envelope sender – The envelope sender (return-to) address is, for the most part hidden from the user. This address receives bounces, and is typically, the address used for Sender Policy Framework (SPF) lookups, as well as domain key records.

Message sender – The message sender is also hidden from users more often than not. As I’ll discuss later, SenderID will generally use the Sender message header to authenticate the message. MUA’s such as hotmail and outlook (and I recon any other SenderID/Microsoft supported agent) expose the sender to the user. If you’ve ever received a message in outlook where you see something like “From John on behalf of Jane,” you’re seeing the effect of a Sender header. Ie. An application or mailing list has sent mail on behalf of Jane; so the message sender = John and the message from = Jane.

Message from – The message from is what is most often exposed to the user receiving the message. Surprisingly, although this header is what is exposed to an end user, it’s simple to impersonate. I quote from one of the users in the thread:

It is therefore, our goal to set the following headers in an effort to have our users receive email from users of our application as opposed to our application itself.

Envelope sender: invites@ourapplication.com

Message sender: invites@ourapplication.com

Message from: user@theiractualdomain.com

If we successfully set these headers, SPF checks will pass (assuming we create SPF records for ourapplication.com), SenderID checks will pass (again assuming we create SPF records), and users will receive invitations from actual users as opposed to our application.

As I found out, setting these headers, and having email land in the inbox of users was easier said than done. I’ll start to describe the steps needed to accomplish setting these headers in my following

Sending emails on behalf of users

Our application, much like many others, uses email as a means of communication between users. We recently made a design goal to send these emails from the users themselves as opposed to from our application. The decision itself seems to have been easier than the spam filters, spf records, domainkeys, and ActionMailer patch implementation details that ensued.

I’ll publish a list of articles that articulate the necessary technical steps to not only send email out on behalf of users (this is the easy part), but to get beyond spam filters, but for now I’m concentrating on motivations for sending out said emails as users.

The scenario that I’m considering is one familiar with users and developers of social networking sites. User John Doe invites Jane to use “My Application.” At this point, Jane will either receive an email from Accounts@MyApplication.com or else an email from John@hisdomain.com. I think it’s fairly obvious that a barrier of trust exists when users receive an invitation from some arbitrary domain owner …. such as “My Application,”regardless of how convincing the message is. Therefore I’m convinced that for virality, it’s ideal to invite users by using their email addresses, ie. John@hisdomain.com.

Before I begin to expand on implementation details in getting beyond spam filters, I must admit that at certain points, I began to question the nature of what we were doing. Part of me at times felt as though spam, and spoofing in particular, was exactly what we were trying to accomplish. I

In the end however, I am comforted and assured that what we are doing does not infringe on any privacy issues. Ultimately, users are intending on inviting others to collaborate with them. They intend to send a message to their friends or colleagues that they’d like to work with them; we enable these messages to be delivered in a natural way using the email addresses the users are working with.

What not to do when creating a software demo

I recently went through the pains of making a demo for an upcoming product. Perhaps the most obvious thing that I learned from this trial was that there is a reason people do this sort of thing professionally!

Needless to say, I took my lumps and in the end generated something that closely resembled a big steaming pile of poo for a demo. Fortunately, there will be a second crack … and we received some awesome feedback from Steven Blank, author of The Four Steps to Epiphany, a classic on bringing a product to market. Some lessons learned when making a demo:

4. Slow down
People watching the demo haven’t lived, breathed, and dreamed Attassa for the last year and a half. Chances are, it’ll take them a bit more time for things to sink in and make sense.

Screens need to be displayed for more than a fraction of a second, actions need to be explicitly pointed out, introduced, and explained.

3. Keep a running dashboard of benefits
One of the best suggestions that Steven had was to create a running list of benefits. As these benefits were being demonstrated, check them off of a list; making them painfully obvious. Clear Context does a nice job of the running dashboard of values.

2 . Create a demo scenario, not just a list of features to demonstrate
The most compelling demos are those that are able to show a real life usage that resonates with viewers. Without getting overly specific in any scenario, you can usually come up with one that is generic enough to convey value, while specific enough that the practicality of the software is beleviable.

1. The point of the demo is demonstrate value
It took me about 2 minutes to lose track of this and my engineering side to pop out as I proudly sliced together clips, transitioned audio, and in general, used every feature of the screen capturing software.

Why Edmonton is Deadmonton for Technology Startups


I love Edmonton. It’s a beautiful, friendly, family oriented city that I owe a lot to. I landed my first job here, gained countless amazing friends, and will always think positively about the city. But at the risk of sounding like a pre-madona, it’s a terrible place to be if you’re starting up a technology company.

I’ve bounced between Seattle and Edmonton the last 3 years in an attempt to balance a career, a software startup company, and a relationship; probably compromising all 3 fronts instead of fully investing. My assertion has always been that being in Seattle or San Francisco as a founder of a software startup is extremely advantageous

I’ll spare you the chest pounding rhetoric, and just give you the personal first hand challenges founding a company while spending time in Edmonton.

1. It’s expensive
Sure, web 2.0 startup costs are much cheaper than traditional models, but we experienced the expenses of Edmonton’s remoteness on a recent business trip. Since I was in Seattle at the time, I flew Seattle to San Francisco for $200 return. My co-founder flew Edmonton to San Francisco for $650 return. It also took him 6 more hours to get there.

2. Bootstrapping possibilities
When considering venture capital investment, our team considered continued funding through independent contract work. Finding contract work in Edmonton is tough. Government contracts do exist, but to be frank, it’s not always the most invigorating work. The following is a screenshot of the “Ruby” jobs listed in Seattle Area

… and Edmonton Area.

nuff said …

3. Finding Capital
We got lucky and raised money on a cold call early on in our efforts, so this claimed advantage is based more on assumptions than first hand experience. In our early planning, we always assumed we would have to hit the east coast, Seattle, or the Valley for Angel investment or Venture Capital; perhaps I’m wrong on my assumption.

My assumption is that Edmonton is filthy rich, but the money is tied up in Oil. Edmonton’s rich are Oil millionaires. They know the industry, understand it, believe in it, and trust their investments in it. I don’t blame them. Talking, and getting helpful investment from an ex Amazon, Web-Ex, Microsoft, Google, or Oracle millionare is much easier than a Talisman, or Husky Oil. Furthermore, investment is more than just $$, and the sweat equity gained from someone who understands the business is extremely valuable.

4. Talking to startup and technology peers Talking to peers in any professional domain is energizing. Physicians, lawyers, educators, laborers, are all motivated, educated, and energized talking shop with peers. Forget networking, schmoozing, or any other cliché, there hasn’t been a Seattle Tech Startup, Seattle lunch 2.0, or startup weekend that I’ve not been incredibly motivated by. Seattle offers events sponsored by Npost, Northwest Entreupreneurs, Startpad, Seattle Tech Startups, Biznik, Seattle Lunch 2.0, and many more. No such technology evens exist in Edmonton, and I can’t help but feel incredibly isolated in Edmonton.

5. Finding evangelists We’re very close to releasing Beta product. In addition to trying to find early adopters of our software via blogs, online marketing, etc, you can bet we’re going to go the grass roots route and talk to as many of our contacts as possible in an attempt to find a beach head customer. It’s a tough thing finding evangelists of new technology when they’re all bogged down with IT policy, software restrictions, as most corporate and governmental organizations are in Edmonton.

7. Finding co-workers Not sure what comes first the jobs or those qualified to perform them, but they go hand in hand. See above screenshots of Ruby jobs in Edmonton.

8. Standard of living Start hurling tomatoes Albertans …. But the *standard of living* offered by the immediate access to the Puget sound, lake Washington, Cascade Mountains, the metropolitan perks, is a convenience for someone with a demanding work schedule. Not having to drive more than 30 miles for these benefits provides a lot of the balance we all need in our lives. While Edmonton is beautiful, it’s also extremely isolated and unbelievably cold.

I’ll admit that my ability to hack away on a laptop, whether I’m in Gull Lake Saskatchewan, Seattle, San Francisco, Edmonton, or Timbuktoo, is about the same. If only starting up a tech company was as easy as typing away on a laptop.

Adding UI to Add Remove Programs Uninstall

As part of our uninstall flow, we want to give the user the option to manually save any of their user generated data before our uninstaller deletes it. Giving all users on the computer, not just the uninstalling user, this option, is a completely separate bag of worms that I’ll address in a later post.

Unfortunately, Windows launches uninstall in minimum UI mode when run from Add/Remove program files. This meens that as fancy and elegant as my underlying UI is on uninstall …. You’re not gonna see it when launched from Add remove programs. After going down a few false positive routes

– Changing the uninstallstring in the registry from /X to /I (windows ignores this)

The Rube solution was to

  • Hide the repair option from ARP

  • Enable the change option from ARP.
  • Modify my installer UI so that control elements displayed would be dependent on whether installing, repairing, or uninstalling

Installed

Wix bootstrapper setup.exe

As part of an effort to address several edgy installer and deployment requirements, I finally decided to wrap our installer .msi in a setup.exe. Previous deployments required our users to download a setup.exe which would download, and launch a second .msi. Being constrained by the wix toolset, and not having the luxury of install shield or another setup application made this a non-trivial task.

Our bootstrapper requirements were fairly straight forward and consistent with any other …

– Extract the internal msi

– Launch the external msi with maximum allowed privileges. If running as admin, install per / machine, else install per/user. This allows us to get around some of the Vista UAC nonsense in determining whether the installer is running as admin or not.

– Launch upgrade path REINSTALLMODE=vomus if the application was previously installed and running setup.exe.

There’s several bootstrapper stubs out there and available on the internet, some even containing code, but none of them seemed to do the job. Our Rube Goldberg solution ….

Use the wix 3.0 setupbld.exe tool along with the wix default default setup.exe stub.

Setupbld.exe is a nice tool that is part of wix 3.0 that beautifully wraps an msi with any setup.exe stub. The setup.exe stub that is bundled with wix 3.0 fulfills the above installer requirements that we have, so there was no need to even write our own stub. The only problem a person may have with this approach is not functional, but rather cosmetic. The generated setup.exe bootstrapper contains the resource information from the stub. Ie. You’re stuck with the generic icon as well as version information contained in the stub. One work around for this (that’s tough to automate in a build system) is to manually crack open the stub.exe in visual studio (simply file open) and change the resource information to that which you prefer.

So … our process ….

1. Autobuild generates the binaries

2. Autobuild builds the installer msi wrapping these binaries

3. Autobuild wraps the msi into a setup.exe

a. setupbld -out Setup.exe –mpsu setup.msi -setup setup.exe -title “Product Setup”

If we had wanted to inject our own branding into the build, we could update the setup.exe bundled with wix 3.0 with our own branding / version information.