We recently hosted a Nylas Developer Event on the topic of Developer Success Engineering. We transcribed the panel so that even if you weren't there, you wouldn't miss out on hearing everything you want to know about Developer Success Engineering from Slack, Segment.io, BugSnag and Nylas.

Panelists:

• David McCreath, Head of Developer Support at Slack
• Kelly Mason, Customer Engineering Team Lead here at Bugsnag
• Maggie Chu, Success Engineer at Segment
• Mike Pfister, Head of Solutions Engineering from Nylas

Moderator: Mary Thengvall, Developer Relations @ Nylas

Question 1: There are so many titles for the same role (developer success engineering, technical support, technical account management, solutions engineering). What do you call it at your company?

Mike: So I made my title up for myself -- I don’t know if other companies use it. To give some context on what we actually do: Nylas is currently 22 people, mostly engineers, building an API for email, calendar and contacts. Naming is hard in this general category of work [developer support engineering] - what’s more important is looking at the kind of customers and product you have to determine what your role will be. 

Maggie: At Segment.io my title is Success Engineer. I was also a success engineer at my previous company which was a really small company, it was like 30 people. Right now we’re a much bigger team at Segment.io (13 people just for success engineering), so it’s a little more siloed and more focused on support. Now, I deal with code every day whereas at my last company I looked up code maybe once per month. It’s pretty different - same title, but different company and different product.

Kelly: I will echo what Mike said in that I also made up my title as a customer engineer. When I started, it was the founders doing support. Since we’re an error monitoring tool and we’re getting down in the weeds with code, it just made sense. We do support and user education on the success side (when we have time). The title is less important than what you’re doing

David: My team is actually hierarchically under customer support,  but we also work with with the cross-functional platform team: DevRel, BD, developer support, product, engineering, and we all work together on the platform which is it’s own thing at Slack. My team came about because we were the agents handling the first 80 integrations we built which were super technical an buggy and required digging into logs. When we launched the platform, we ended . up being the ones that helped debug and QA the platform as it stood and we ended up just giving ourselves titles, so “developer support”.

Question 2: How do you work alongside engineering or other technical departments to make sure that you’re taking care of your customers?

David: Our support tickets are right now divided between customers who are using the integrations and developers who are building the integrations. Most of the time that we spend working cross functionally is with the platform engineering team. For a long time, we handled all of the manual QA that has now been passed off to a different team so we can focus more on the actual support of developers. We work with the product teams (a handful of product teams within platform) on making sure that they understand what the developers are looking for in a specific feature. Before a feature is released we build stuff to make sure it’s working properly. 

Kelly: We sit with the customer success team and are very in line with t heir goals, but we’re actually under the engineering umbrella. Part of it is any feedback we’re getting towards career development is shifting more towards engineering. One of the cool things we get to do is build our own admin tools, so we spend a lot more time in the code with the eng team and using our own product to help our customers.

Maggie: We build out our own Zendesk tools which we rely on all the time. We display customer information, how big their account is, who is their account manager. We rely on all of these things to ensure the customer gets proper support as they’re writing in. We use many other integrations like JIRA (communicating with our engineers and product teams).

Mike: Nylas as a whole is 22 people right now. The developer success team is just myself and Lou. We work very closely with all departments at Nylas (sales, product, engineering). We’re the front line of support. Since our business is an API and our customers are mostly developers so it’s very technical. The way we assist sales is we hop on a sales call and take on what might be called a sales engineering role, giving people technical advice on how they can integrate with our API, best practices, get them onboarded quickly. We help inform product decisions because we’re communicating with customers so frequently that we h ave great insight into what they want in terms of feature requests. With engineering, we’ll escalate any bugs that we’re seeing to them. We do daily standups with the engineering team that let us surface any customer issues on a daily basis.

Question 3: How do you protect your engineering team from getting bombared with customer support tickets? How do you balance that with their day to day engineering work?

Mike: It’s important that the developer success team protect engineering time. The engineers have their own long-term projects they’re working on, and it’s difficult for them to make progress when they’re getting interrupted with customer tickets on a daily basis context switching to fix bugs. To protect our engineering team, it’s about building trust. Every time we escalate something to them that we shouldn’t’ have escalated, it’s going to be hard the next time you want them to look at a ticket. It’s our job to prioritize and take the birds eye view from all the tickets we’re looking at and surface the most important ones. The things we look at to protect engineer’s time are:

• How many accounts are affected by this issue?
• Are there SLAs in place with the customer that we need to honor?
• Is this customer part of a deal we’re trying to close?

Maggie: For our company, we rely heavily on internal docs. So when we’re talking to our engineers, and they tell us “This is the reason why you’re seeing this error message”, we document that. We rely so heavily on this process and it’s not just our team looking at it, it’s teams from sales, customer success managers, and marketing. We all encounter the same questions across departments. We use Dropbox paper to maintain these docs.

Kelly: Give that we support pretty much every language and framework and I have only a few years of engineering experience, it’s quite often that I don’t now when I first start a problem if I can figure it out in an hour or if by the end of the day i’ll know anything more. Myself and my colleague (we’re also a two person team) run through tings in the morning and time box the ones that are configuration heavy or deep in the weeds of .net (something i just won’t know) and usually look at it for a solid 2 h ours then reassess whether to escalate or not. One person from our two engineering teams runs through everything we couldn’t resolve in the afternoon.

David: We have a similar process. Each of our engineering groups has what’s called a triage channel in Slack where support people (whether it’s us giving developer support or the customer team doing customer support) can drop in and say “I’ve got this problem and none of us can figure it out.” The engineers work in shifts on those in various departments. In platform, there’s two people and all they do is triage. But we try to protect them by understanding as much as we can about the APIs. 

Question 4: How do you get metrics around what features need to be added to the product?

David: For slack, the support is part of the product. That’s the pitch we’ve made since the product came out. If you’re paying for Slack, you’re paying for support. Over the last years, we’ve rolled out an enterprise product and scaled up some of that. Now, as a team we’re figuring out how we move from 1:1 email support to broader support for the enterprise. How do we look at a 1 to many instead of 1:1 solution?

Mike: At Nylas, we have a basic support SLA that all of our customers are on automatically for free. Being as mall startup, every team is very constrained in the resources we have. Obviously we would love to have 30 minute resolution times for every ticket that comes in but that’s just not possible. So we have other tiers of SLAs that people will pay for. Generally for newer customers, we’ll priotize their ticketes as well to help them get onboarded quickly.

Maggie: At Segment we have different pricing tiers, all the way from enterprise to a free developer plan. So if you’re on the the team plan or enterprise plan, it’s a faster response time and your issues get escalated faster. Enterprise plans have a dedicated support or customer success manager.

Kelly: We rely  a lot on our salesforce integration with our Zendesk. So if something is on an enterprise plan, it’ll get moved up int eh queue. But a lot of times it’s more informal in that I sit close to the sales team and the customer success team. I thought it was kind of goofy to sit with the customer success team at first, but it’s actually a huge benefit to be able to reach over to someone who has context and can help resolve something quickly.

Question 5: How do you get feature requests funneled to the right team/how do you speak on behalf of your customers?**

Kelly: We’re really interested in getting that customer feedback. It’s something we collect a lot because we value that feedback. Error monitoring is kind of new as an idea -  so grouping the way you might approach a problem of fixing the bugs in your application is so unique to every customer that we feel it’s best when we h ave a critical mass of feedback and can connect closely with our engineers who are building it. Once we see critical mass, we’ll start building something. 

Maggie: For any major feature request we have, we put them into JIRA reports. We have an integration with Zendesk where we can integrate Zendesk with JIRA reports. We also put the customer profile into the ticket so if they’re a big or small customer, we document everything so the product team can review it, do research, and decide which sprint to put the feedback into.

Question 6: How do you help your customers support themselves so you aren’t drowning in support tickets?**

Mike: Creating a lot of content. The most effective is a knowledge base article. i use a tool called Loom that helps you make a video and share the link with someone. We also host office hours on a weekly basis. There are 2-3 hours of weekly slots that are open, and our customers can call in and chat with a n engineer (which is basically me) every week. They love to talk about everything from their integration and how ti’s working well, to diving into a specific issue. If they don’t have an SLA, a lot of customers will use this t o get roe face time.

Maggie: Yeah, we have our public facing documents and we’re always trying to make them really beefy. Whenever a customer writes in and asks a question, they look at the docs. There’s a person on our team that focuses just on docs.

Question 7: What is the hardest part of developer success?

Maggie: A lot of our customers write in expecting us to know everything -  not just form your own product, but all the integration, all of your partners, questions that you can find on StackOverflow. They expect you to be an expert in everything technical, and obviously no one is, so we have to do a lot of our own research. The hardest part is to know when to push back and say “hey, this is something you might need to research on your own, it’s not in our scope.”

Mike: The hardest part for me was being the only support person for a long time. It’s a pretty emotional job when you’re trying to make customers happy and so not having a team member like Lou was really tough. I’m super excited because we just hired another DSE and I’m continuing to hire for more technical roles for our team, and so those days are over.

David: I think for us one of the trickiest things was the speed with which we grew as a platform suddenly became very difficult to keep up with. Even though we were the ones responsible for QA, things were slipping by us and getting released without QA happening and we had to put out fires.

Kelly: I would say ours is the number of ways we get a support request. We do get people writing customer support, talking to their success manager, but we also have a very large number of open source repositories so you can always great a GitHub issue there. And that’s great because once you’ve solved something you can reference it there and everyone can look at past GitHub issues, but it also is an interesting overlap between customer engineering and out platforms team that makes those libraries. Making sure the process holds up if they’re not overwhelmed with GitHub issues and knowing where to prioritize those within the rest of our customer support issues.

Question 8: What’s your favorite part about your job?

Kelly: I think it’s pretty cool that you very rarely know what the answer to someone's question is before I start it. The amount I've learned just form our customer’s questions and about technologies i would’ve never thought to look up on my own.

Maggie: We have a very international customer base, so it’s interesting to have customers form all over the world write in from different backgrounds and languages. They may ask the question very differently. 

David: For me, this role feels more like the old internet when you could just help somebody build their webpage and boom [they're just so excited]!

Mike: The kinds of people I work with are CTOs and tech leads, so my role is relationship based. It’s cool to establish that and work with someone through their entire integration. I like when I’m able to help those people and keep them happy and being able to automate processes in a way that finds elegant ways of not doing those small repeatable processes over and over again is really satisfying.

Question 9: What’s the one thing you wish someone had told you before you took on this role?

Kelly: I think the #1 thing would be everything there is to know about error handling. But given that that’s unlikely, maybe just the fact that understanding the question and figuring out what the user is getting at in their question and which part of their tech they think is going wrong. That’s the first step. Once . you can sift through that everything is easier to find.

Maggie: Knowing what the customer is actually asking. It could be so different than what you thought. I’d debug what i thought they were saying for 20 mins, and they’d come back and say “that’s not what I was asking” and I'd have to revisit the question. Always reconfirm what they’re truly asking for.

Mike: Back to my days on the graveyard shift (when I was doing support by myself), feeling the need that i Have to solve every issue as fast as possible. I wish i would’ve learned earlier that you can say no to customer request and be honest with people about expectations.

David: I think honesty is big part and not being afraid to tell the developer when “Yes there’s a problem with the API and it’s not going to be fixed today, but we’re working on it.” And then they at least know and can build around it.

Kelly: It’s kind of nice because most developers have been there before. They’re like “That’s a lot of work, I’m not going to build that”.

Question 10: Once you’ve dug into the code and know you can open a pull request for it, do you actually submit the PR or do you bring the problem to engineering?

Mike: At Nylas, yes as a developer success engineer you can get a customer support request through Zendesk, dig through Kibana, dive into logs using Honeycomb.io to figure out why things are broken, writing code, submitting to Phabricator, then actually deploying it. We have a command we’ve written at Nylas (CLI command) so you can take it from start to finish and get back to the customer and say it’s fixed. The types . of issues we fix in that way are smaller scoped. For big infra changes, we’d escalate to engineering.

David: I have commit privileges but I try not to use them. There’s enough engineers now that I’ll get it through to the PR then pass it off to the engineer to review and either commit it myself or let them do it.

Kelly: A lot of it is feeling out where that line is. Often we find that if something isn’t clear, we can make an example better. My co-worker updates a lot of our open source examples of how you might configure different parts of Bugsnag - that’s more common (random little QRs that are helpful but not specific to what our engineers are working on day to day).

Question 11: Do you have tips on stepping away from day to day work?

Maggie: For me it depend son how many people are on my ass about tickets. If i have people from many different teams saying i need to push something through right now, I get nervous about it. If it doesn’t need to be responded to asap, sometimes it can wait till tomorrow. You get the sense of urgency from the customer.

David: Letting people know you have not forgotten about them goes a long way.

Question 12: What key traits do you look for when interviewing for this role? Someone more technical or with a stronger support background?

Kelly: We look for people that are okay with being thrown in the deep-end of code. You must be comfortable starting with little information and learning as you go. Having technical background and being able to communicate well with customers is a unique trait and it’s hard to find people that can do both.

David: One of the things we screen for when hiring developer support or customer support is stamina. The people on the developer support team right now have all worked as engineers in the past and decided to move on to something else. The mindset you have to go from being heads down in your code all day to switching between other . people’s problems throughout the day is significant,  so we look for people who are genuinely interested in being helpful.

Mike: We’re hiring a developer success engineering right now. They’re 60-70% communication on email, phone, etc. The other 30-40% is diving into issues. We test for their communication skills and also their technical ability - we have a little coding challenge the run through. Prior coding experience is definitely something we look for. The partner engineer role is the flip side: 20-30% communication, more time spent diving into bugs.

Maggie: For us it’s pretty similar. Having a technical background is a good foundation, but we really stress great communication skills both verbally and in writing. 

Question 13: What kind of tooling do you invest in?

David: We have a pretty basic setup of Zendesk. It’s tied into Kibana and other instances. I think the code base of Slack is so unusual that we built most of our own tools around it and that’s stabilizing a little as we bring in more commercial stuff.

Kelly: We have support ticketing, and we use BugSnag to find people who have run into an error on our site. It’s very meta, but it’s really helpful when someone’s integration isn’t working. Normally that would be my least favorite thing to try to debug, but we send a custom event every time someone’s integration has failed, and I'll use Bugsnag to see if it was an incorrect password, or something bigger like the site being down. It’s funny using the tool they’re complaining about to fix the problem.

Mike: Internally we use a web app. Some of our accounts listing page - since we’re a fast growing startup, that page loads slower each day. As we hire more DSE’s, we’ll have more time to work on higher impact projects where we build out tools to support ourselves. A really awesome tool I like is called Honeycomb.io. It’s a great way to get structured log data and break down based on several fields. You can pinpoint why a particular thing is happening extremely quickly.

Question 14: How do you prevent an investigation on a ticket from becoming a rabbit hole?

David: I grab somebody else and get their eyes on it. If I realize i’m going down a rabbit hole, I’ve gone too far.

Maggie: I give myself maybe 30-40 minutes and if i haven’t figured it out, iIll ask somebody else. Sometimes I’ll ask the customer to repeat or rephrease the question or update them once I have more data,

Kelly: It’s a good point of getting more data. In that time, I’m hoping if I don’t know the answer I at least have the next question and can ask that to the user or ask the right team questions so that when I escalate it they’ve gotten a little further and I’ve saved them some time.

Mike: I live in rabbit holes on a daily basis. Working in email - that’s the value our business is providing. We abstract away all of the rabbit holes developers would have to deal with building out an email integration. I do the best that I can with the technical knowledge I have. I’ve spent a long time debugging Exchange ActiveSync issues, but at the same time I have to communicate with customers. I can’t go radio silence when I’m debugging something, so I try to document as much as possible what I find so that when I escalate the problem to engineering, they’re set up for success.