The story behind salesforce-orm Gem, a successful hack we used to speed up Nestaway's Salesforce migration.

This post was migrated from my old personal blog. This was originally published on Jan 2018

What is Salesforce ORM

An ActiveRecord like ORM for Salesforce which can be used in ruby projects.

The story

In early 2017, Nestaway decided to move many of its processes to the Salesforce platform. As part of this, across the teams, many core components from the backend services have to be migrated to Salesforce. Few services became just a proxy to Salesforce, and for many services, the source of truth for data became either Salesforce or our backend systems, with the proper syncing mechanisms in place.

So in short, our backend systems have to play with a lot of Salesforce data.

Our devs had very little experience working Salesforce. So the initial learning curve for the team was very high.

As this migration process was huge, so the entire process was divided across teams base on service ownership. My team (Tenant experience) was the initial team who started working on this process. We had 2 main responsibilities:

• Completly reengineer and migrate service request market place (SRM) from Freshdesk to Salesforce service cloud.
• Migrate Nestaway users (both tenants & owners) to Salesforce and build a 2-way sync mechanism.

So we started with the SRM implementation. As the entire SRM system was being reengineered, we had decided to keep both Freshdesk and Service cloud to be active until we archive the data from Freshdesk. So we architected the system with the bridge pattern and built drivers for Freshdesk, service cloud, and Google cloud data store (for archival).

At Nestaway, most of the backend service was written in ruby and ROR.

So now on let's focus on the driver we built for the service cloud.

As I told earlier, our team had only very little experience working with Salesforce APIs. So as expected our initial code was full of boilerplates. We also did a 2nd time refactor to reduce the boilerplate. But still as and when the service started growing the boilerplate started increasing which made unit testing and code maintenance horrible.

For this implementation, we used a great Gem called restforce to work with Salesforce REST APIs.

That is when we actually came to know that, Salesforce supports a query language called SOQL. When we looked at it, we found that it is very similar to SQL syntax, and they support all CRUD operation similar to SQL. Also, we understood that restforce has direct API to execute SOQL queries.

That was a Eureka moment, we immediately started working on a basic SQL to SOQL converter. And tested it. With that positive impression on that same night, we built a basic working MVP that has a similar API like ActiveRecord. The next morning we demoed it to the team and everyone loved it. That gave us a green sign for building a more concrete Gem. But we were already halfway through the project and now we had to build this new Gem and migrate the existing code to use it (available time was less).

But with 3 to 4 days effort, we built the first usable version of the gem and we started integrating it into the running project. Surprisingly, that migration was super easy.

About the salesforce-orm

This Gem was providing an ActiveRecord like API to work with Salesforce objects. Any rails developer can easily learn this with their ActiveRecord experience.

What is the real hack behind it, which helped us build this entire stuff within 4 days?

In simple words, its the power of ruby, Metaprogramming.

With a proxy, we actually used ActiveRecord itself to do the trick. The trick was, how we generated the SQL query. With the help of nulldb we created a fake ActiveRecord instance, which made it generate SQL query.

Later this SQL query is converted to SOQL with a simple converter and forwarded it to Salesforce using restforce gem.

Listing the core modules of the Gem

• ObjectBase: Like ActiveRecord::Base, we have to create each Salesforce object (model) by inheriting this class. This class becomes, the primary proxy to bring in APIs similar to ActiveRecord.
• Base: This is the brain of this Gem. It takes care of building the query, making API requests to Salesforce, and all required data conversions.
• QueryBuilder: The real hack for generating SQL query.
• SqlToSoql: A simple SQL to SOQL converter. We know that it is not a solid converter. With the available time, we made it support most of the regular scenarios. And in most cases, we didn't have any issues with this converter.

Conclusion

The decision to build salesforce-orm came out really valuable for Nestaway because it helped us to speed up the future releases and it got widely adopted across Nestaway teams. Developers loved it because it was providing ActiveRecord like API. And they didn't have to worry about, what is happening underneath.

Overall this project was a huge success, and we were really happy to contribute the project to the community. \m/