Imagine you’re an overworked IT person managing an in-house email system and you keep getting requests to change it for various department and employee needs. Some departments have specific spam filtering needs while others want to autorespond for various purposes. You have a filter/autoresponder script, but each special case you add makes it more difficult to maintain. You’re looking to free-up time to play video games work on your other responsibilities and give the various users control to change it as they need. With just having one script across the whole company the result would be chaos! Various departments could make conflicting modifications and accidentally step on each other’s toes! You’d get the blame if they took the email server down! What if rather than giving shared control over all the email, instead, you could give everyone control over just their own email?

A primitive solution you may consider deploying is allow each user to upload a script and have the master autoresponder script call their script when their email address receives an email. This solution has so many holes it’s like swiss cheese; what programming language do you support? -Not everyone wants to write Perl like you. You don’t just chmod +x any uploaded file and run it, right? What if the script has dependencies, who is going to install those? What if a script gets stuck in an infinite loop or starves resources in other ways? This server does more than email, now all the company services are down. Running untrusted multi-tenant code is hard and it’s too risky to run without isolation. We want email “hackable” only by their owners, not email that’s easy to take over.

Thankfully, there’s an off shelf solution to safely integrate third party code with your system, the XTP platform. XTP is built by the folks at Dylibso on-top of the open source framework Extism.

Let’s take a closer look at how you could integrate XTP with your autoresponder script.

Table of Contents

Initial Code

To start, you may have a script that looks like this:

#!/usr/bin/perl
use strict; use warnings;

# parse the incoming email
my %email = parse_incoming_email(@ARGV);
my $sender = $email{sender};
my $receiver = $email{receiver};

# autoreply for various senders
if ($receiver eq 'marketing@companyname.com') {
    #....
    exit send_email($sender, $receiver, \%email);
} elsif ($receiver eq 'ceo@companyname.com' ) {
    #....
    exit send_email($sender, $receiver, \%email);
} elsif ($receiver eq 'engineering@companyname.com') {
    #....
    exit send_email($sender, $receiver, \%email);
} elsif (...) {
    #...
} else {
     exit send_email($receiver, $sender, \%email);
}

The Interface

To start, I recommend designing the plugin interface. The plugin interface can be broken down into two parts, exports and imports. Exports are the functions the plugin exposes to the program, in this case, the autoresponder script. Imports are functions you’d like to provide to plugins. As plugins are compiled to WebAssembly, they are sandboxed and are only able to call their own functions or the provided import functions.

For exports, we want each plugin to expose onEmail, an Extism function that takes an incoming email and outputs a result code. When the result code is non-zero we bounce the email. For imports, as the plugin needs to be able to send emails, you could expose send_email to the plugin. However, that would open our email system up for abuse as it could be used to send spam, even with spoofed senders to any email address. Instead, we’ll create safe wrappers of send_email with sender and receiver hard-coded, deliver and reply. deliver sends an email to the receiver and reply sends an email to the sender. reply will also have the To and From headers hard-coded to avoid getting a bad reputation for having spoofed or bogus headers with third-party email systems.

Once, you have an idea what your interface will be, it’s time to encode it into the XTP Schema:

version: v1-draft
exports:
  onEmail:
    description: |
      This function takes a IncomingEmail. What you do with it is up
      to you.
    codeSamples:
      - lang: c++
        source: |
          auto res = pdk::deliver(input);
          return res;
    input:
      contentType: application/json
      $ref: "#/components/schemas/IncomingEmail"
    output:
      contentType: application/json
      type: integer
      description: sendmail return code, set to non-zero to bounce
imports:
  deliver:
    description: |
      send it to the inbox, may only be called once
    input:
      contentType: application/json
      $ref: "#/components/schemas/IncomingEmail"
    output:
      contentType: application/json
      type: integer
      description: sendmail return code
  reply:
    description: |
      reply back, may only be called once
    input:
      contentType: application/json
      $ref: "#/components/schemas/ReplyEmail"
    output:
      contentType: application/json
      type: integer
      description: sendmail return code
components:
  schemas:
    IncomingEmail:
      description: the incoming email
      properties:
        sender:
          type: string
          description: from
        receiver:
          type: string
          description: to
        headers:
          type: object
          description: to, from, subject, etc
        body:
          type: string
          description: body of email
    ReplyEmail:
      description: a response email
      properties:
        subject:
          type: string
          description: email subject
        body:
          type: string
          description: body of email      

Any order will do, but first we encode the exports, then the imports, and then describe the types we used. The purpose of describing the interface in the XTP Schema is to make it easy for the XTP Bindgen to generate the functions and types for a large variety of languages for plugin development.

The XTP Platform

Extism handles running plugins and the XTP schema is free to use, but what about the third part of the problem, plugin management? Plugin authors need the ability to add and update their plugins in real time. Building out user authentication, plugin uploading, validation and hosting is no trivial task. The XTP platform does all of those things and more.

Sign-up for the free tier HERE. Create a team and app, and note down your "app id", it should be something like app_01j9w5k56wf9ev69z1h158axjc . Go to Guests and click "Add Yourself", which will create a guest user for your logged-in account. Finally, create an extension point, name the extension point "on email" , paste your schema created in the previous step, and click Submit.

Navigate to https://xtp.dylibso.com and click "tokens". We will generate a token in the next step.

Integrating XTP

In the same directory as autoresponder script, create a .env file with two variables, XTP_APPID and XTP_TOKEN, filling in the "app id" you noted down and generating and adding a token from the tokens page. When you’re done the file should look something like:

XTP_TOKEN=xtpXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XTP_APPID=app_01j9w5k56wf9ev69z1h158axjc

NOTE: It is critical to keep your token a secret!

Setup the autoresponder to load the .env file. As I was already launching the Perl autoresponder script with a bash script, I modified the bash script to load the .env file first.

DIR=$(realpath "$(dirname "${BASH_SOURCE[0]}")")
set -o allexport
source "$DIR/.env"
set +o allexport
exec perl "$DIR/autoresponder.pl" "$@"

Install the Perl XTP client: cpanm XTP. Note, this must be installed it a way that it is accessible to your autoresponder script.

With the configuration in place we can finally make the code modifications. Towards the top of autoresponder.pl load XTP, load our configuration, and define how we turn email addresses into guest keys:

use XTP;

# load XTP configuration
my $appid = $ENV{XTP_APPID};
my $token = $ENV{XTP_TOKEN};

sub email_to_guestKey {
    my ($address) = @_;
    unpack('H*', $address)
}

Add loading a plugin for the current user and calling it. Replace the end of the if ladder:

replace:

} else {
     exit send_email($receiver, $sender, \%email);
}

with:

} else {
    # try to get a plugin
    my $plugin = eval {
        # Initialize the host functions
        my $deliver = Extism::Function->new("deliver", [Extism_String], [Extism_String], sub {
            my ($input) = @_;
            my $email = decode_json($input);
            my $rc = send_email($sender, $receiver, $email);
            return JSON::PP::->new->utf8->allow_nonref->encode($rc);
        });
        my $reply = Extism::Function->new("reply", [Extism_String], [Extism_String], sub {
            my ($input) = @_;
            my $email = decode_json($input);
            $email->{headers} = {
                Subject => $email->{subject} // '',
                To => $sender,
                From => $receiver,
            };
            my $rc = send_email($sender, $receiver, $email);
            return JSON::PP::->new->utf8->allow_nonref->encode($rc);
        });
    
        # Initialize an XTP client
        my $client = XTP::Client->new({
            token => $token,
            appId => $appid,
            extism => {functions => [$deliver, $reply], wasi => 1}
        });
    
        # Finally actually try to get a plugin
        $client->getPlugin('on email', email_to_guestKey($receiver));
    };
    if ($@) {
        # no plugin, or initializing XTP failed, let the email pass through
        exit send_email($sender, $receiver, \%email);
    }
    
    # Call the guest's plugin
    my $rc = eval {
        my $res = $plugin->call('onEmail', \%email);
        JSON::PP::->new->utf8->allow_nonref->decode($res)
    };
    if ($@) {
        # Bounce, the plugin crashed
        exit 70; # EX_SOFTWARE
    }
    # return the plugin's exit code
    exit send_email($receiver, $sender, \%email);exit $rc;
}

That was a mouthful, but it can be broken down into four steps. First, we initialize our host functions (or imports from the plugins perspective), deliver and reply . Second, we create an XTP::Client. Third, we try to fetch a plugin for the receiver. Finally, if any of that failed (for example, the user doesn’t have a plugin), we just pass the email through to the receiver, otherwise, we try to call the plugin and exit with its result code or bounce on error.

Verify the modifications went smoothly by sending yourself an email and making sure it went through. If not, you probably got a failure delivery message with a stack trace pointing at your missing semicolon.

Creating a Plugin

To verify all the XTP parts are working, let’s create, push, and try out a plugin. Download the XTP CLI if you haven’t already: https://docs.xtp.dylibso.com/docs/cli/#installation and authenticate:

xtp auth login

Create a plugin with:

xtp plugin init

And select the app and extension point you created, select any programming language you’d like to use (though we will use C++) and follow the rest of the wizard.

To implement the Project XYZ handling featured in the video, open impl.cpp and replace the body of onEmail with:

  auto res = pdk::deliver(input);
  if (res && *res == 0) {
    std::string subject = input.headers.contains("Subject")
                              ? input.headers["Subject"].as<std::string>()
                              : "";
    if (subject.contains("Project XYZ") || input.body.contains("Project XYZ")) {
      pdk::ReplyEmail response;
      response.subject = "RE: " + subject;
      response.body =
          "Talking about Project XYZ is forbidden!!!\n\nPlease stop talking "
          "about Project XYZ!";
      res = pdk::reply(response);
    }
  }
  return res;

Once, you’re done writing code, from the plugin’s directory:

xtp plugin build
xtp plugin push

Once xtp plugin push finishes, send yourself an email that your plugin has special handling for and verify it performs as expected. For the Project XYZ example, putting Project XYZ in the subject will do and while the original email will go through, you’ll also get a reply back:

Inviting Guests

You can invite guests manually through an App’s guest page on the XTP website, but it likely would be better to have this process fully automated. Let’s add inviting guests to the autoresponder.

Go back to the if ladder and add a case for it@companyname.com:

elsif ($receiver eq 'it@companyname.com' && exists $email{headers}{Subject} &&
    $email{headers}{Subject} =~ m!^/hackablee?mail\s+signup!) {
    eval {
        my $client = XTP::Client->new({
            token => $token,
            appId => $appid,
        });
        my $invite = $client->inviteGuest({
            guestKey => email_to_guestKey($sender),
            deliveryMethod => 'link'
        });
        my %email = (
            headers => {
                Subject => 'Hackable Email Invite',
                To => $sender,
                From => $receiver,
            },
            body => $invite->{link}
        );
        exit send_email($sender, $receiver, \%email);
    };
    if ($@) {
        exit 70; # EX_SOFTWARE
    }
} 

If we receive an email directed at IT with the subject /hackableemail signup, we generate an invite link for them and reply back with it. The regex is actually a little more permissive to reduce support overhead with users mistyping it.

The easiest way to test it out is to remove yourself as guest from the app’s page on  https://xtp.dylibso.com and send an email with the subject /hackableemail signup , then repush your plugin using xtp plugin push and resend the test email (Project XYZ in our case).

Testing

One last thing you might want to do before encouraging your users to use your hackable email setup is to add some tests to the extension point. Without any additional work, when plugins are pushed, the XTP platform verifies plugins conform to the schema. You can block more malformed plugins from entering your system by adding tests that execute plugin code to verify it handles specific scenarios acceptably. It is still prudent to offer a secure host interface to plugins, but catching issues when plugins are pushed can provide a better developer experience than unexpected errors at runtime. Tests created the host that are ran by everyone are called simulations.

Simulations consist of a test module that contains tests and sometimes a mock host, that provides a simulated host environment to the plugin (the imports). If your schema has imports, you must provide a mock host in order to test it.

In the case of hackable email, we need to provide two functions, deliver and reply to create a mock host. The bodies of those functions are at simulation/host/impl.cpp. These functions are somewhat complicated as they use the email subject to configure their behavior to create more test scenarios.

We chose to use Go to write our tests:

// test.go
package main

import (
	"encoding/json"

	xtptest "github.com/dylibso/xtp-test-go"
)

func callOnEmail(input []byte) {
	output := xtptest.CallString("onEmail", input)
	xtptest.AssertNe("we got some output", "", output)
	var res int32
	err := json.Unmarshal([]byte(output), &res)
	xtptest.AssertEq("unmarshalling json", nil, err)
	if err != nil {
		xtptest.AssertEq("output", "", output)
	}
}

//go:export test
func test() int32 {

	xtptest.Group("Test with Deliver and Reply both succeeding", func() {
		input := []byte(`{
			"headers":{"Subject":"simulation00"},
			"body":"What body?",
			"sender":"a@example.com","receiver":"b@example.com"
		}`)
		callOnEmail(input)
	})
	xtptest.Group("Test with Deliver and Reply both failing", func() {
		input := []byte(`{
			"headers":{"Subject":"simulation11"},
			"body":"What body?",
			"sender":"a@example.com","receiver":"b@example.com"
		}`)
		callOnEmail(input)
	})
	xtptest.Group("Test with Deliver succeeding and Reply failing", func() {
		input := []byte(`{
			"headers":{"Subject":"simulation01"},
			"body":"What body?",
			"sender":"a@example.com","receiver":"b@example.com"
		}`)
		callOnEmail(input)
	})
	xtptest.Group("Test with Deliver failing and Reply succeeding", func() {
		input := []byte(`{
			"headers":{"Subject":"simulation10"},
			"body":"What body?",
			"sender":"a@example.com","receiver":"b@example.com"
		}`)
		callOnEmail(input)
	})
	xtptest.Group("Test without Subject header", func() {
		input := []byte(`{
			"headers":{},
			"body":"What body?",
			"sender":"a@example.com","receiver":"b@example.com"
		}`)
		callOnEmail(input)
	})
	return 0
}

func main() {}

For faster iteration, I recommend testing your simulation locally with the CLI:

xtp plugin test example-plugin/dist/plugin.wasm \
	--with simulation/test/test.wasm \
	--mock-host simulation/host/dist/plugin.wasm

When you’re satisfied with your simulation, upload it to the extension point on the XTP website as described at https://docs.xtp.dylibso.com/docs/host-usage/simulations . More details about testing plugins can be found at https://docs.xtp.dylibso.com/docs/concepts/testing-plugins

Wrap Up

The code for the autoresponder script, schema, test, and mock host is at https://github.com/dylibso/xtp-email-demo If you have any questions, don’t hesitate to reach out! We’re available on Slack, Discord, and email, links are on our support page: https://docs.xtp.dylibso.com/docs/support/

Gavin Hayes
Senior Software Engineer, Dylibso