What to/not to expect from pentest

Hello,

it has been a while since I posted something (nothing unusual), but I really wanted to touch on a sensitive/controversial topic. Firstly, the blog just represent my personal opinion and not that of my employer, so do not draw any conclusions!

So, to start the debate, I have a question:

Do you expect a black-box pentest to find *all* vulnerabilities?
--
I will comment on this with mainly with a black-box app pentest in mind but the same logic apply to other forms of pentest too (in my opinion that is)
---
My thoughts: Any pentest vendor you choose, they would try to find as many vulnerabilities as they possibly can. Most security consultancy companies (at-least all good ones) follow a set methodology, some sort of check-list, a number of in-house /commercial tools and other related material to ensure 2 most important things :

1. consistency
2. coverage

Pentesters are not robots, they are humans and different people have different expertise and different skills and of-course some are more skilled in one area than others. The methodology, in-house/commercial tools, check-lists etc help achieve some level of consistency between various pentesters.

Time factor: Black Box pentests are usually a function of time. That is you only get a few days to assess the security of a particular application. New vulnerabilities/technology makes our job more complicated/interesting. Most tests nowadays are scoped based around clients budget they have for security testing and hence the scope is limited to find as many vulnerabilities as one possibly can in that amount of time. The more time you will allow to find vulnerabilities, the more likely it is that new vulnerabilities will be found.

What is a black box pentest: The nature of black-box pentest is such that it can never guarantee that there are no more security flaws other than those reported in the pentest report. For a more thorough assessment, source code auditing is recommended. Black box pentest is done primarily to provide assurance that if a reputed pentest vendor cannot find anything major wrong with the application's security than its less likely to suffer from high risk issue(s). I can easily code up applications with critical security flaws which can only be identified when the source code is available and these are nearly impossible to find in a black box pentest.

The length vs the breadth: Should you find a high risk vulnerability, how much time can you spend in exploiting it. Should you exploit it? Of-course, you should safely exploit it to demonstrate the true impact of the vulnerability. Often exploitation of one vulnerability leads to discovery of another vulnerability. E.g. exploiting a vulnerable file upload functionality can give you access to source code and could lead to discovery of a SQL Injection issue. But exactly how much time can you spend exploiting it. Can you afford to miss a Local file include vulnerability because you spent too much time exploiting some other vulnerability? Thus, when a critical issue has been identified I would always recommend that the retest don't just focus on the 1 issue but at-least some more testing is done to ensure that the test has received a decent coverage.

The out-of-box thinking: Pentest by nature involves creative thinking. The more familiarized the pentesters are with the application, the results will be so much better. Most pentesters would start a pentest by getting familiarized with the application. If I perform a pentest of the same application which i tested six months ago, then I would have already had an understanding of the application's behavior/security/logic/input validation etc and I can then spend some time to come up with some new innovative hacks. But unfortunately, the way it works, If i do find a clever hack in the recent test than sadly, I will be asked the question "Why the Hell did I miss in the first pentest".......

Technology and Vulnerability move on: Again, even if you have not changed any line of code in your application it does not mean that the pentest will not find anything new this time. E.g. the world did not know of the padding oracle attack till 2 or so years ago; but because I know it now, I will identify and report it now. It does not mean I missed it then..

Thus there are so many factors which one should consider when requesting a pentest and set expectations accordingly. Also the pentester needs to consider several factors to ensure that they provide the best possible result in the allocated time. Hopefully, this also highlights the need for regular pentest.

Hope my rant dont upset too many people. Would love to hear what the other guys from the industry think about this....