What to/not to expect from pentest

May 3, 2012


it has been a while since I posted something (nothing unusual), but I really wanted to touch on a sensitive/controversial topic. Firstly, the blog just represent my personal opinion and not that of my employer, so do not draw any conclusions!

So, to start the debate, I have a question:

Do you expect a black-box pentest to find *all* vulnerabilities?

I will comment on this with mainly with a black-box app pentest in mind but the same logic apply to other forms of pentest too (in my opinion that is)

My thoughts: Any pentest vendor you choose, they would try to find as many vulnerabilities as they possibly can. Most security consultancy companies (at-least all good ones) follow a set methodology, some sort of check-list, a number of in-house /commercial tools and other related material to ensure 2 most important things :

1. consistency
2. coverage

Pentesters are not robots, they are humans and different people have different expertise and different skills and of-course some are more skilled in one area than others. The methodology, in-house/commercial tools, check-lists etc help achieve some level of consistency between various pentesters.

Time factor: Black Box pentests are usually a function of time. That is you only get a few days to assess the security of a particular application. New vulnerabilities/technology makes our job more complicated/interesting. Most tests nowadays are scoped based around clients budget they have for security testing and hence the scope is limited to find as many vulnerabilities as one possibly can in that amount of time. The more time you will allow to find vulnerabilities, the more likely it is that new vulnerabilities will be found.

What is a black box pentest: The nature of black-box pentest is such that it can never guarantee that there are no more security flaws other than those reported in the pentest report. For a more thorough assessment, source code auditing is recommended. Black box pentest is done primarily to provide assurance that if a reputed pentest vendor cannot find anything major wrong with the application’s security than its less likely to suffer from high risk issue(s). I can easily code up applications with critical security flaws which can only be identified when the source code is available and these are nearly impossible to find in a black box pentest.

The length vs the breadth: Should you find a high risk vulnerability, how much time can you spend in exploiting it. Should you exploit it? Of-course, you should safely exploit it to demonstrate the true impact of the vulnerability. Often exploitation of one vulnerability leads to discovery of another vulnerability. E.g. exploiting a vulnerable file upload functionality can give you access to source code and could lead to discovery of a SQL Injection issue. But exactly how much time can you spend exploiting it. Can you afford to miss a Local file include vulnerability because you spent too much time exploiting some other vulnerability? Thus, when a critical issue has been identified I would always recommend that the retest don’t just focus on the 1 issue but at-least some more testing is done to ensure that the test has received a decent coverage.

The out-of-box thinking: Pentest by nature involves creative thinking. The more familiarized the pentesters are with the application, the results will be so much better. Most pentesters would start a pentest by getting familiarized with the application. If I perform a pentest of the same application which i tested six months ago, then I would have already had an understanding of the application’s behavior/security/logic/input validation etc and I can then spend some time to come up with some new innovative hacks. But unfortunately, the way it works, If i do find a clever hack in the recent test than sadly, I will be asked the question “Why the Hell did I miss in the first pentest”…….

Technology and Vulnerability move on: Again, even if you have not changed any line of code in your application it does not mean that the pentest will not find anything new this time. E.g. the world did not know of the padding oracle attack till 2 or so years ago; but because I know it now, I will identify and report it now. It does not mean I missed it then..

Thus there are so many factors which one should consider when requesting a pentest and set expectations accordingly. Also the pentester needs to consider several factors to ensure that they provide the best possible result in the allocated time. Hopefully, this also highlights the need for regular pentest.

Hope my rant dont upset too many people. Would love to hear what the other guys from the industry think about this….



  • Jim Bird says:

    From the customer-side of a pen test (I’m the guy who hires pen testers in to test my apps) everything that you say makes sense. Anyone who thinks that a manual pen test (especially a time-boxed test) will find all of the real vulnerabilities in their app is hopelessly naive. What am I looking for in a pen test? A better understanding of risks and threats that we can feedback into how we design and develop software.

    I understand that you can’t test everything and that how you test and how well you test will change over time as you learn more about the app and your tools and your craft. And I understand that the results of a pen test, like any test, depend on many factors like the approach that you take and the tools that you use or even how much sleep you’ve had lately, and on luck as well as skill and experience.

    If you can find real exploitable bugs in my app in a few days then we are doing something wrong and we need to understand what this is and what we need to fix – in the design and code and in the way that we design, build, deploy and test software. I want to know how hard each bug was to find and how exploitable it was or how exploitable you think it would be if you had more time.

    Sure, I want you to find as many bugs as you can so that we can fix them, but what I mostly want is information. I want help in understanding what you’ve found and where else I should look for other problems so that I know what I have to fix and so that I can fix the code and how we make it. I think that these are the expectations that you need to set – that your testing won’t be/can’t be exhaustive, that you are going to help identify areas of risk, and that it’s up to the development team to look deeper into what needs to be fixed: not just the code, but more careful design and programming, better tools and training, whatever.

  • I completely agree with your thoughts on what to expect from a black box pentest. I have been doing these pentests for almost 2 years now and I have dealt with all sorts of clients – From people who have unrealistic expectations to people who really understand what they are trying to gain from it.

    Also, one of the most frequent questions I get is “Why wasn’t this reported before when we haven’t changed anything in the code”. So, yeah I am guessing the industry is slowly getting used to these vulnerability assessments. Some are cool with it and some are not. It is just the way it is.

  • laz55 says:

    Why are you looking to penetrate an application (pen test)? Surely if you want to find what vulnerabilities exist a vulnerability assessment is the approach to take?

    Whilst people continue to brand every type of security testing a pen test there will always be confused people asking questions like this.

    To answer your first question ‘no’ I don’t expect a black box pen test to find all vulnerabilities, I expect it to determine whether it could achieve the goals it set out to achieve (such as transferring $1,000,000 from one account into another) and how that was achieved. Since real attackers don’t perform VA exercises I would be surprised to see a list of all vulnerabilities.

    However, if this was a black box web app security test I would expect a list of vulnerabilities. I would also hope that the testers prioritised their testing focusing primarily on areas of easiest win and communicated to me (in the report) the approach the have taken and the depth they have gone into in the timeframes so that we can make an assessment (with the testers) as to whether this was sufficient.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.