Before co-founding Sqreen, I spent years stressing applications both from the outside (as a security consultant) and from the inside (as part of Apple’s Red Team). As a result, I’m often asked by fellow CTOs what the best practices are in terms of SaaS security and penetration tests (pentests). So this article will shed light on pentest best practices and some mistakes you should avoid.

There are plenty of ways to proactively test an application’s security. One of the more common methods is performing penetration tests on your application. A pentest is a way to reveal security issues in your application and infrastructure from a neutral third party in a way that you can leverage to positively affect the whole company.

There are plenty of resources out there to help you prepare for a pentest. This article focuses on the actionable learnings a pentest should provide to you, beyond the pages of the report provided by the pentesters.

The 3 axes of a pentest

The learnings you can get out of a penetration test can be grouped into 3 dimensions.

• the technical learnings:
  • the state of your code security
  • the state of your infrastructure security
  • the quality of the monitoring and security alerting in place
• the organizational learnings:
  • how new services or infrastructure changes are handled (e.g. moving to the cloud)
  • whether processes and ownership are clearly identified
  • the effectiveness of the processes in place
• the human learnings:
  • the team’s reaction when security issues are reported
  • the way the company (hierarchy, stakeholders, etc.) handles security reports.

The technical report and the discussion with the auditors will mostly tell you about the technical axis. It is up to you, however, to carefully prepare your pentest in order to walk away with key insights and clear action items around the human and organizational dimensions.

A pentest also provides a way to assess how your security has evolved over time. Every penetration test should be followed by some days of validation tests – to ensure that the items you found and planned to fix have indeed been fixed. You should also provide previous test reports to the security auditors as a starting point to find initial vulnerabilities, iterate on the existing ones, and ensure they were fixed (if they’re present again, it means a regression occurred).

The technical learnings

A pentest will very likely give you new technical insight into your infrastructure and application. It’s important to thoroughly analyze how the technical side of your app responds to the pentesters, and how the tools you’ve set up to monitor and protect your app react. The report you get from the pentesters will give you a great overview, but you know your environment better than anyone, and there’s a lot to learn as well from how it responds during the pentest.

Prior to the pentest

Before the pentest begins, you and your dev team should review the backlog you have around security. Do you have a security tag in your bug tracker? How many tickets are tagged with security? How old are they?

No security tickets in the backlog most likely denotes a lack of attention on the security front. Otherwise, check how many have been solved in the past months. If the answer is none, it probably means you haven’t been prioritizing security enough!

Also, ensure your team is up to date on their technical security knowledge: does everyone knows what an XSS is? An injection? Same Origin Policy? CSRF? Content Security Policy? If your team has too many gaps, then you will need to start with updating your team’s knowledge prior to beginning any kind of fixes after the pentest concludes.

During the pentest

The quality and effectiveness of your monitoring and alerting tools is one factor to check when you’re doing a pentest. There are plenty of things a team monitors that may trigger because of a pentest. An exception manager, such as Sentry, will have unusual activity during most penetration tests that your dev team will need to interpret.

You should also check your performance monitoring tool (such as New Relic), which can highlight slow parts of the application during the pentest. The tests may slow down your application and put pressure on some specific parts of your infrastructure (e.g. a database or a microservice). This can lead to the detection of potential applicative DOS, and require specific optimization or rate limiting in order to prevent exhausting it. Most penetration tests do not look for DOS since they focus on testing the production: that’s something you need to look for yourself.

Your log monitoring solution will be a good source of information as well and can also tell you if your application is getting too slow and starts to answer with timeouts or errors. It’s worth checking your logs to see if you have issues in this space.

Eventually, your security tools will generate alerts regarding the attack. Make sure these alerts came from the testers (based on the IP address, the user names, etc.…) and take screenshots. The testers are gonna be pleasantly surprised you managed to catch them.

All of this information will most likely be unnoticed by a security researcher performing a test on your application, because they only have an external point of view on your system. It is up to you to be reactive when these kind of alerts occur and interpret them appropriately.

Refrain from fixing anything the testers may report to you until after the pentest fully wraps up. Doing so would prevent them from fully understanding and exploiting what they found, and also from finding more vulnerabilities in that vein.

After the pentest

The report that the pentesters deliver to you at the end of the test will cover any technical vulnerabilities that they’ve uncovered. It is up to you and your team to then prioritize and implement fixes. Many pentests come with a validation period, which is a time during which people can just ask their security contractor to ensure the reported weaknesses have been validated. Using a clear mapping between the pentest and your tickets will help you organize and prioritize this accordingly.

Take the time to speak in depth with the pentesters about what they’ve found in order to get the most you can out of the exercise. Pentesting companies often offer training, and organizing half a day of training on a particularly hard-to-understand vulnerability could be especially useful to level up your developers and get them interested in the subject.

Additionally, the report serves as a point-in-time understanding of the state of your infrastructure security and the state of your code security. It will also help you to take a step back and figure out where the gap areas are in your team. Should you train your team on a given family of vulnerabilities? Improve your processes to have a bigger security focus when writing specifications? Hire someone to challenge your team on their security point of view? Hire someone who could be a security champion? The pentest report can help you decide answers to questions like these.

tl;dr: actions for the technical axis

• Understand your security issue backlog and the state of security knowledge on your team
• Check that your monitoring solutions are working and are properly used by your developers
• Take the time to fully understand the report that the pentesters produce, and take concrete steps to improve your security knowledge on the team and your security within your applications

The organizational learnings

Pentests can give you insight into your organization as well. The way your security processes work (or don’t) in practice and the way your team functions as a unit are all key things to watch for and determine what needs improvement.

Prior to the pentest

This is the time to review your security processes. Ensure the responsibilities for your various processes are well-defined and clearly known. You should know who to alert when security vulnerabilities are found. It is not infrequent that penetration tests highlight existing vulnerabilities that are currently being exploited , and that may require a rapid response – you might need to shutdown a service while the vulnerability is being corrected. In this case make sure you know how to react and how to articulate and route this information within your company – this may involve the legal department as well as the communications department, particularly in the case of public companies.

During the pentest

How the team responds to security alerts is a key element of an efficient and effective security response. Is the team trained to do so? Was a process ready for the person on call to respond to this? These processes don’t get triggered very often, so is the team aware of how to find and implement this process? Have all the security alerts sent by your security tools been handed over to the appropriate team? For instance, a vulnerability lying in the code should go to the developer team to fix, and a training session could be conducted to ensure all engineers are now aware of this specific security threat. Also, any new developer joining the team needs to be given this training as well.

After the pentest

Some of the vulnerabilities discovered during the penetration test will come from a failure to enforce some processes – or from a lack of process. The initial fix is always technical, such as changing a default password or upgrading a library. But the root cause of the issue often lies in a missing or incomplete process (or an imperfect automatization).

This can be especially true when new technologies are at stake. The security specifics of a new database system, or a new cloud provider may remain fuzzy to your team, especially from a security point of view. In this case, training should be conducted so your team better understands these technical elements, and can ensure they are correctly configured. This also includes the need to improve the process of adding new technology to your infrastructure: engineers should be trained to reach a decent level of knowledge on the technology from a security point of view. A security review of any new family of elements can also be enforced.

tl;dr: actions for the organizational axis

• check you have procedures in place in case of a critical flaw discovery
• check that your processes have clear ownership defined
• test your processes to ensure that people know how to apply them

The human learnings

Pentests can tell you a lot about the individuals that you have on your team as well. Their reactions and responses can communicate a lot about their preparedness and mindset towards security.

Before the pentest: accepting the idea

For many technical people, whose job it is to build and operate systems, knowing that a penetration test is being run on what they built can be seen as an offense to their skills and intelligence. It can also be understood as a lack of trust from whoever ordered the tests.

Of course, this is a spontaneous reaction, driven by their ego, that may not last. The team will soon realize that a penetration test has two positive outcomes:

• First, it is an excellent opportunity to learn, since most security professionals will detail their reports in a pedagogical way. Learning is one of the most shared motivations amongst IT professionals.
• Second, it is an excellent opportunity to prioritize leftover tasks that generated security debt – which is a subset of technical debt.

The takeaway here is that the way you present the pentest to your team matters. Do not hide behind a compliance excuse as the reason for the tests, but rather clearly leverage these positive points to have the team welcome the security tests.

To ease the impact of the pentest on your team’s psyche, tell them in advance that a test is coming and promise that they will know the timeframe (e.g. 3-6 months out). Giving them a range rather than an exact date also creates an opportunity for them to get in the habit of thinking about security in their sprints, as well as improving or checking the alerting system for security issues. The team is often aware of security issues that have been long standing in the backlog, so communicating ahead gives them the chance to get around to fixing those issues.

During the pentest: monitoring & reacting to attacks

While the tests are ongoing, the team will be warned about the issues by your security alerting tools, if you have one in place (which you should). Critical issues will need immediate response (depending on your agreement with the security auditor) and will be handled by someone from the team.

Pay extra attention to the human side of these security issues. They will have stronger psychological impact to your team than usual bugs do, since they are proven to have security impact, and will be highlighted in the security auditor’s report that may flow all the way through the company’s hierarchy. This may reveal tensions and finger-pointing across team members. It is the right time to remind everyone about the benevolent motto: everyone is doing his or her best. If mistakes are made, they need to be understood and the team should act accordingly to ensure they won’t happen again. The important thing is correcting the bugs, not assigning blame, so this whole process should be conducted blamelessly.

After the pentest

Once the security audit is over, the team needs to debrief about the results. What was detected during the tests needs to be consigned in a shared document and enriched by everyone. Getting a broad team involved will make the results stronger.

Security successes (e.g. successfully detecting attacks, the parts of the system that the pentesters weren’t able to break, etc.) need to be highlighted and applauded, and failures (e.g. vulnerabilities detected and exploited, weaknesses discovered) have to be added to your bug tracking system, with a specific way to list them (with a tag such as “Q2 penetration test”). As with any other issues, these ones need to be prioritized. Most security consultants will rank their findings to help you prioritize what to fix, but they most often do not have the best understanding of your business. That means you need to go deeper with your team (and maybe the security consultants if an issue outcome is not perfectly clear) in order to properly rate and prioritize them.

Each issue found (e.g. MongoDB injection in the Company controller) needs to be generalized, multiple times. For example, “if we have injections in this controller, we may have some in other controllers”. “If we have injections, maybe we are not that good at sanitizing inputs, let’s check user inputs”. And so on. This will directly leverage your team’s problem solving skills.

Eventually, analyze your company’s reaction up and down the chain of command to the bad news that came up during the tests. Security is a sensitive topic that may put people in what they feel is a sensitive position, and have them react with more stress than usual. Good faith, honesty, and transparency are critical values for employees in order to have them proactively report security issues – rather than burying them.They have to be listened to and encouraged in order to bring these issues up. This means more than just stating values. People will read how everyone reacts during the pentest and internalize that.

A typical reaction when security vulnerabilities are found goes like this: “this is indeed an XSS, but it isn’t exploitable because only the user can do it to himself.” This reaction is understandable: if the vulnerability cannot be exploited, why would we fix it?

Let’s consider this vulnerability as a regular bug. What do you do with regular bugs? You fix the ones that are quick to fix, or can harm you, or can endanger your application. You also learn the lesson and make sure these kind of bugs do not occur again. Security bugs have to be considered the same way. If it occurred once, it may occur again, and if the part of the application where the vulnerability lies is later reused some place else, the bug will spread. In this case, its exploitability will have a different impact.

It is a regular bug, but one that has security consequences. That merits a different level of attention from your team.

tl;dr: actions for the human axis

• Take the time to talk with your team about the pentest before it happens and present it to them the right way
• Pay attention to the psychological side of your team during the pentest. Ensure that everyone knows that whatever is uncovered has nothing to do with blame
• Take the time to debrief fully with your team, and prioritize and log the security issues found in a fair way

Need some help?

If you’re interested in learning more about maximizing your pentest, we put together a Pentest Best Practices Checklist. Check it out!

Pentesting can give you an excellent point-in-time understanding of your security and show you the critical issues that you need to fix. Sqreen can help you solve these issues with real-time monitoring and protection, and actionable alerting. They work great together! Try it free today or sign up for a demo.

The post How to fully leverage your pentest appeared first on Sqreen Blog | Modern Application Security.