How To Receive Twilio Messages In MongoDB Atlas Functions

Overview

There's plenty of documentation about how to send an SMS via Twilio, but very little about how to receive one. Receiving one in a MongoDB Atlas Function involves an extra complication. By the end of this post you'll know how to receive a text in Twilio, and have it successfully interact with your MongoDB Atlas Function. I'll also cover a couple nice-to-have's in Twilio that you'll probably want to set up anyway.

I'm hoping, that you'll discover, like I did, that this Serverless Function in the Cloud stuff is actually pretty easy.

Data Flow overview

an overview of the expected vs actual architecture. explained below.

The thing you need to know about interacting with Twilio is that it wants instructions from you in XML format. They've done a great job of not only making that XML simple, and documenting it, but also making libraries that make it trivial to generate without ever thinking about XML.

Twilio also makes it really easy to interact with external systems. You just plug the URL of a remote webhook into the appropriate, easy to find field, and you're good.

The problem is that MongoDB Atlas Functions only allow you to return JSON or EJSON. So, even though you can easily generate the XML you need in your Atlas Function, you can't return it. And, to complicate matters, what you tell the function to return, isn't actually what the function returns.

So, let's walk through setting this up. I'm going to skip over the bits that are well documented by Twilio and MongoDB, and focus on the bits that aren't.

Configuring Things in Twilio.

Twilio appears to have two forms of functions in the cloud: "TwiML Bins" and "Functions". You can find both if you click on "Explore Products" in the sidebar of your "Console". Once you click into it, be sure to click the three dots to the right of "TwiML Bins" in the sidebar and choose "Pin to Sidebar". You'll be glad you did.

Rejecting Voice Calls

I presume that most users don't use the voice functionality, so you the first thing you need to do is to refuse voice calls. Like everything else, you do this by giving Twilio instructions in its XML format (TwiML).

Create a new TwiML Bin and call it "Reject Calls".

Here's what it needs to contain.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Reject />
</Response>

Next, go to Phone Numbers > Manage > Active Numbers and click on your phone number. There are two sections "Voice and Fax" and "Messaging". Under "Voice and Fax" You'll see "A Call Comes in". Choose "Twiml Bin" from the first pull down, then your new "Reject Calls" from the second.

I didn't bother with anything else here. I didn't set a "Primary Handler Fails" method either because if they can't talk to their own system, I don't expect them to be able to talk to anyone else's.

You're now done with worrying about calls. Let's move on to the Atlas Function.

Your Atlas Function

First you'll need to create a cluster, and an App Services App. The MongoDB docs can guide you through that. Once you've done that, you go into the App Services App you created, click on the cluster you want to work with, and then click "Functions" in the left sidebar. Then click "Create New Function". Give it a useful name. Make sure the function is not private, and that authentication is set to "System".

npm Packages: You won't need the "twilio" npm package. It just makes life difficult here because of the JSON response. If you're like us and don't want the risk of storing someone's phone number you'll want to hash it. Click "Add Dependency" and add the "crypto" package dependency.

Now, let's make the function. This functions is where your real work is going to happen. In the example below I'm showing you only the core bits you'll probably want to copy. After that I'll give you an example of some input to test it with in their console.

// This function is the endpoint's request handler.
// NOTE: MAKE THIS async TO SAVE HEADACHES
exports = async function({ query, headers, body}, response) {
    const db = context.services.get("MY_LINKED_DATA_SOURCE").db("MY_DB_NAME");
    // we're going to hash the user's phone number because
    // we don't want the risk of storing that personal info

    const { createHash } = require('crypto');

    async function getOwnerObj(db, hashedPhoneNumber) {
      const collection = db.collection("NAME_OF_COLLECTION_WITH_PHONE_HASH");
      // below: phoneHash & ownerId are field names within our data
      // your names may be different
      const query = {"phoneHash": hashedPhoneNumber};
      const projection = {"ownerId" : 1};

      // Use findOne() to retrieve a single document from the collection
      return await collection.findOne(query, projection)
    }

    function cleanPhoneNumber(phoneNumber){
      // our app is US only so the phone numbers will start with
      // the US country code: +1
      // you may not want this, or may need a different regex
      return phoneNumber.replace(/^\+\d/, "");
    }

    function phoneHash(phoneNumber) {
      // it's a good idea to add in some "salt" here.
      return createHash('sha256').update(cleanPhoneNumber(phoneNumber)).digest('hex');
    }

    // --------------------------------------
    // BEGIN procedural code
    //
    // Raw request body
    // This is a binary object that can be accessed as a string using .text()
    const reqBody = body.text();

    // For a full list of parameters they send
    // check out https://www.twilio.com/docs/voice/twiml#request-parameters
    // We only care about who the message came from, and what the message was.
    const { From, Body } = JSON.parse(reqBody);

    // Set up the default data you'll return
    // I've decided to go with HTTP Status Codes, because they've already
    // got something to address most situations.
    var responseData = {
        "status": 200, // 200 if everything goes as expected
        "message": null // the message we'll return to the user
    };

    // hash the phone number, find a matching entry,
    // return it.
    // NOTE: we have to await the response, even though we already
    // awaited the response inside the functions we're calling.
    const ownerObj = await getOwnerObj(db, phoneHash(From));

    if (ownerObj != null) {
        // OK we have a user. Yay.

        // insert your message processing logic here.
        // if anything goes wrong, change the status code
        // in responseData
        // the message will be whatever message you want to return
        // to the user. Note that outbound texts are VERY rate limited
        // in twilio so, only respond with a message when you need to.

        responseData.message="it worked and i found a user";

    } else {
        responseData.status = 403;
        responseData.message = "No registered user with that phone number.";
    }
    return responseData;

};

Notes & Things to change

Note

The default exports function is not async. Be sure to change that.

It looks like you're returning something like this:

{
  "status": 200,
  "message": "my response SMS text"
}

BUT, what you're actually returning is more like this.

{
  "status": 200,
  "statusText": 'OK',
  "headers": {...},
  "config": {...},
  "request": {...},
  "data": {
    "status": 200,
    "message": "my response SMS text"
  }
}

Whatever JSON you chose to return will be stored as the value of the data key in the object that's returned.

Things to Change in the code above
  • MY_DB_NAME
  • NAME_OF_COLLECTION_WITH_PHONE_HASH
  • MY_LINKED_DATA_SOURCE (this defaults to "mongodb-atlas")

Change those to the relevant values. If you're unsure, there are a ton of videos on getting up and running with Atlas Functions "in 10 minutes". Just be aware that they used to be called "Realm Functions". Here's a video from MongoDB that should get you going.

Deploying Your Atlas Function

There's a tab for settings. There are two notable bits for now.

  1. Authentication should be "System" for now.
  2. Make it NOT be private.

You can change these later based on your needs, but for now, this setup will make testing WAY easier.

Deploying is trivially easy. Just save your function. That's it.

Unfortunately, we can't actually talk to it yet.

HTTPS Endpoint
  1. Click "HTTPS Endpoints" in the left sidebar.
  2. Click "Add an Endpoint"
  3. Give it a "route" In our case I made it /tell because our app is called "Tell Remmy" but it really doesn't matter. The general public is never going to see this URL.
  4. Turn on "Respond with Result" If you don't do this you won't get any data back.
  5. The Return Type should be JSON.
  6. Select your function by name.
  7. Don't muck with authentication. If you need to, muck with it once you've gotten things working.
  8. Copy the URL under "Operation Type". Save it somewhere for reference later.

Testing Your Atlas Function

There are two ways to test your function. Once you've got the HTTPS Endpoint set up you can easily test with Insomnia or Postman. "Deploying" your function is so fast, that I find it easier to test by actually hitting the endpoint, and just shoving useful debugging info into the returned JSON.

You can also test on their "console". I'll cover both.

The following assumes you've used the code above, and had a record in the DB with a matching phone number hash. You can also just comment out the code that interacts with the db and just return anything that isn't from that function if you want to just test fast and leave DB stuff for later.

Testing With Insomnia / Postman

Create a new POST request. Paste in the URL you saved when you made the HTTPS Endpoint.

Tell it to use a JSON body / payload. Here's my test one. It's got a couple extra keys in it that you'll be getting from Twilio, but not the full set.

{
"From": "+11234567890",
"To": "+15558675310",
"Body": "an example text message from the user",
"numMedia": 0
}

Run it. See what comes out.

Error messages are half-way decent but don't contain line numbers.

Testing with the Console

Under your function editor is a JavaScript console.

Here's some test input for it:

const textFunc = function(){
	return '{"From": "+11234567890", "Body": "an example text message from the user"}';
}

const request = {
	"query": null,
	"headers": {},
	"body": {"text": textFunc}
}
exports(request, {})

Note that body.text is a function. This is to simulate the real post body that you'll be receiving, and need to call .text on to get the JSON passed in from Twilio.

Stick that in the console, and click Run.

> result (JavaScript):
EJSON.parse('{"status":{"$numberLong":"200"},"message":"it worked and i found a user"}')

Note that if you end up with a problem in your code, console.log(...) is not going to be as helpful for debugging as you'd expect. It will only output anything if the function doesn't encounter any errors. Because of this, and because of the fact that the console is so very tiny with my large font size, I generally don't bother using it.

That's it.

Your Atlas Function is live, working, and callable from Twilio, or anywhere else.

Teaching Twilio To Talk To Atlas

Back in your Twilio console, click on "Explore Products" in the sidebar. Scroll down until you find "Functions and Assets". Click it, and then pin it to your sidebar for easy future access.

You'll need to create a service. This appears to be some sort of grouping of functions. I don't know, and I don't care. I just called it "FooProxy" where "Foo" is the name of the app in MongoDB Atlas that this function is acting as a proxy for.

Click into your new service, and click "Add +" > "Add Function"

Here's what your function should look like:

const axios = require('axios');

exports.handler = async (context, event, callback) => {
  // Create a new message response object
  const twiml = new Twilio.twiml.MessagingResponse();

  //twiml.message("bogus");
  //return callback(null, twiml);

  const instance = axios.create({
    baseURL: "THE DOMAIN NAME OF YOUR ATLAS FUNCTION'S URL",
    headers: { 'X-Custom-Header': 'Twilio' ,
               'Content-Type': 'application/json',
    },
  });

  try {
    // axios#post(url[, data[, config]])
    // const {status, message} = await instance.post(
    const mongoResponse = await instance.post(
      "EVERYTHING AFTER THE DOMAIN NAME OF YOUR ATLAS FUNCTION'S URL",
      {
        "From": event.From,
        "Body": event.Body
      }

    );

    if (mongoResponse.status == 200){
      // remember, or response is in the .data portion of the
      // big wrapper MongoDB Atlas put around it.
      const myStatus = mongoResponse.data.status;
      const message = mongoResponse.data.message;

      // maybe change something if myStatus != 200
      twiml.message(message);

    } else {
      // note: this is MONGO's status, not yours
      twiml.message(mongoResponse.statusText);
    }


    return callback(null, twiml);
  } catch (error) {
    // As always with async functions, you need to be sure to handle errors
    console.error(error);
    // Add a message to the response to let the user know that something went wrong
    twiml.message(`We received your message, but something went wrong 😭 ${error}`);
    return callback(error);
  }
};

Notes & things To Change

We'll need to insert the URL from your MongoDB HTTPS Endpoint into your Twilio Function. It probably looks like this:

https://us-east-1.aws.data.mongodb-api.com/app/cluster-name-abcd/endpoint/my_path

Replace "THE DOMAIN NAME OF YOUR ATLAS FUNCTION'S URL" with "https://us-east-1.aws.data.mongodb-api.com" , or whatever's at the start of yours.

Replace "EVERYTHING AFTER THE DOMAIN NAME OF YOUR ATLAS FUNCTION'S URL" with everything after that. In this example it'd be "/app/cluster-name/abcd/endpoint/my_path

In the instance.post method call, you may want to change the data you're passing on. For us From and Body was the only info we needed, so it's the only info we're passing on. You can always add additional fields later.

Deploying Your Twilio Function

  1. Save it.
  2. click "Deploy All"
  3. Go back to "Phone Numbers" > "Manage" > "Active Numbers"
  4. Click on your number.
  5. Scroll down to the "Messaging" section
  6. Under "A Message Comes In", choose "Function" and choose your function from the resulting pull-downs
  7. Click Save.
  8. Note that they've replaced your function with a webhook url. Don't worry about that. That's normal.

After the first time you just need to "Save" your function and click "Deploy All" to update it.

Testing Your Twilio Function

Go back to your function. "Functions And Assets" (in the sidebar) > "Services" > Click on your service.

Click "Enable Live Logs" below your function. Text your phone number. You should receive the contents of the "message" portion on your phone.

If you don't get a message, and don't see anything happen in the live logs it could be two things:

  1. Live logs simple isn't working. Sometimes it just doesn't. Reload the page.
  2. An error occurred.

    1. click "Monitor" in the Sidebar.
    2. click "Logs" > "Errors" > "Error Logs"
    3. Do not choose "Last Hour". It's broken. Probably a time-zone bug for anyone not in UTC.

If you see an error in the logs, click into it. The most important part is the "Response Body". Almost everything else was useless for me.

If your receive a "not delivered" message from your phone it's probably because you're still on the trial and it will ONLY let you interact with it from your phone number and no other ones.

Note that the macOS Messages app does NOT send from your phone number as far as Twilio is concerned. I haven't investigated what number it is.

Either way, your choice is to just use your phone, tell twilio about another phone number, or just start paying them. I just used my phone until I got everything working and then paid them so that others could use it.

Handling MongoDB Outages In Twilio

Sooner or later, MongoDB's servers will go down. That's what servers do. Twilio's will too, but there's not a lot you can do in Twilio if Twilio's servers are down.

Let's set up a TwiML Bin to handle what we hope will never happen, but probably will.

Create a new Twiml Bin. This one we'll call "Service Failure Fallback". Its contents should look like this.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Message>Oh No! Our Servers are not feeling well! Please try again in a few minutes.</Message>
</Response>

Then go manage your active phone number again, and under the call to your Twilio Function, that calls Mongo, associate your "Service Failure Fallback" TwiML Bin with the "Primary Handler Fails" operation.

Now, if MongoDB Atlas goes down your users will receive a text response with a message that you chose.

Summary

In the end it's pretty easy. It's just complicated by the JSON only responses of MongoDB Atlas Functions. If MongoDB eliminates that restriction, and stops wrapping responses with additional stuff, we can eliminate the Twilio Function entirely, and just use your Atlas Function's url as a webhook in Twilio. We'd also be able to use the twilio npm package in Mongo.

If you've got questions, feel free to ask me on Mastodon.