by Dan DeMeyere, Morgan Griggs, and Florian Lorrain
In the past year, thredUP's email marketing team has elevated their game to the next level. While they increased the frequency of their campaign cadence, they actually sent less emails per customer as a result of careful audience segmentation and campaign personalization. To segment and personalize in a way that would make our emails resonate more with our customers, our email marketing team had to analyze customer data via Looker (our business intelligence tool) and then upload lists for individual campaigns, which was a laborious and tedious process. This is something that we hoped our current email service provider (ESP) would have been capable of, but after more than a year of doing workarounds like this, our team arrived at the conclusion that we needed to find an ESP that empowered us instead of getting in our way.
Our CTO led discussions with a number of ESPs and ultimately decided on Oracle's Responsys solution. After the contract was signed, the integration project came down the pike to our web engineering team.
Our first step was getting to know our current ESP integration inside and out. Anyone who has completed an ESP migration before (this wasn't our first ESP migration rodeo at thredUP) knows that you can't just flip one ESP switch off and another ESP switch on. To warm up the new ESP's IP addresses, you have to integrate the new ESP into your existing system and ensure data is reconciled across all platforms. For example, if someone unsubscribes on your existing ESP, you need to update your web application's records as well as your new ESP's records to prevent sending someone an email in one system that already unsubscribed in a different system.
Once we had a feel for the existing endpoints, flows, and system dependencies, we had the necessary context for the next phase of the project, which was the research phase. Before doing a deep dive, we did a high level walkthrough to understand how we would be interacting with Responsys (ex: JSON? XML? SOAP?), how Responsys differed from our existing ESP and what terminology we should formalize to avoid confusion while discussing the Responsys API.
Responsys has a robust set of functionalities and along with it, they have a robust set of documentation. Since our team for the project was relatively small (3 engineers), we decided to divide and conquer. We divvied up the major components (profile extensions, list management, etc.) such that the team in aggregate had a thorough understanding of every major component. When someone felt they fully understood one of the components they were responsible for, they would do a mini presentation to the other two engineers. After 3 days (1 day for high-level, 2 days for deep dive), we were ready to start breaking down the project so we could get to work.
At this point, we did what most Rails engineers would do:
Every link was a dead end. Gems that didn't work anymore. Gems that were abandoned. We went multiple pages deep sifting through results and we tried every gem or library we could get our hands on. Nothing worked. With no viable gem options, we decided it was time for an architecture planning session.
Do we build our own Responsys gem? Do we create a self-contained layer in our app that talks directly to the Responsys API? Because our current ESP contract was expiring in less than a month, our project had a deadline. If we didn't complete the integration by the deadline, we would have to re-up our contract with our existing ESP and unfortunately they don't do month-to-month.
We couldn't afford to go down one road for a week or two and turn around once we discovered it was the wrong choice. We knew building the gem was the right way, but we didn't know if we could pull it off in time. We decided to have one person prototype the gem and one person prototype the non-abstracted internal approach. We time boxed the prototyping to one day and scheduled time to huddle the next morning.
The next morning we reviewed both prototypes and while the internal approach was further along, the gem was at a point where it was a viable option. We knew the gem would be more work and we knew we would have to be more thoughtful with decisions to ensure the gem was abstracted to the point where anyone could use it for their company, but we decided to bite the bullet and go for it.
We spent the next day or two deciding on how to organize the gem code, what naming conventions we should use (we decided to use the same terminology in the Responsys documentation) and what gem functionality would be required to complete the migration project. Until our project was complete, we wanted to focus our efforts on gem functionality that our web app required.
Once we identified the critical path for the project, we made sure the gem was always one to two steps ahead of the web app's integration. We huddled briefly every day to give updates and determine whether we needed to shift the team's resources around so that we could always be going full steam ahead on our web app's integration.
I'm happy to say we completed the project on time and that deciding to build the gem was the right decision: https://github.com/dandemeyere/responsys-api
We still plan to add more functionality to it as well as tighten up the documentation, but we've started what we think is a great base for ourselves and others to build upon. We've already had the fine folks on the TaskRabbit engineering team open pull requests to help improve the gem and we hope this post will be a catalyst for others to join the project as well.