RamsThoughts

March 29, 2007 12:42 am

Exploratory Testing and Project Management

Filed under: Computers,Exploratory Testing,Software Testing,Tech — ramsblog @ 12:42 am

I have read some material about Exploratory testing, Rapid Application Testing etc on James Bach and CemKaner’s sites. From there I also got access to Pradeep’s blog. He has very interesting scenarios, user stories, hueristics, etc that are helpful for Exploratory testing. I read something about what kind of preparation is required for such an approach. I also talked to few architects in my local area and most people were positive about it. However, most organizations do believe in test cases, test plans etc.

Preparation is the Key! – Yes I do believe in this phrase. Prepare for Great and Worse situations and you will be better off in knowing and being capable of handling either way. So what kind of preparation made in traditional waterfall testing approach? Most organizations write Test Scenarios and Test cases during the Build phase when the application is being developed. And of course, the basic reference will be the Functional Specification document.

Based on documentation, Test plan, Test cases, it is kind of helpful from the Project Management perspective on a better planning. Does it mean testing will be complete in a planned time. May be or May be Not. It completely depends on the code quality.  So then, does it mean there is a good coverage in test cases and if you blindly execute the tests / test cases, you are done testing? No – way. There is never 100% testing complete in Software world. I once heard, even an airline industry has .001 defects in their products.

So, my point is, with all these pre-planned test plans, test cases, scenarios, Test Cases reviews, corrections, addition of more scenarios etc etc, it is hard to keep up the schedule and most time we will be able to keep up the schedule as well – but at least you have something to plan against and reference against. But as I read in Exploratory Testing, the Prep time is rather used for data preparation, scenario planning etc. One said, if you are trying Exploratory testing for the first time, set aside some amount of time in project deliverable for exploratory testing. Likewise, you would be able to base your test results / execution on test cases and still be able to go through the rapid testing.

Being directed ad-hoc in nature (or may be I am not knowledgeable enough make such comments),

  1. how would you plan for the execution time?
  2. What kind of commitments / goals would you set to the project management?
  3. How would you track the progress on your testing between a given time frame for testing? (let’s say you have 5 days on a medium sized application)
  4. when and how would you know you have tested enough? (remember you have some basis in traditional test case writing approach to track on – though with constraints)
  5. etc.

<updated tags>

{tags: project management, agile , exploratory testing}

About these ads

7 Comments »

  1. # how would you plan for the execution time?

    Why would you plan for the execution time? It has too many dimensions and variables attached to it which are not considered when planning.

    1. Tester A differs from Tester B and the plan for each ideally should be different. Tester B is forced to execute those tests in the same time Tester A did and ends up getting frustrated. A frustrated Tester B wouldn’t be finding many bugs… Now that is just one of many cases I have witnessed.

    2. The planning is usually done by the Lead or Manager after consulting testers on the estimated time to execute the tests. Poor testers, say a number that they think is good for them but later realize the product is more complex than they imagined, or some of the test cases were not good enough or so on … this makes testers to fear adding new tests to the test suite to reduce their own pressure of finishing things on task. Management never know that they are losing something important whenever testers meet the deadline by manipulation.

    There are so so many. Probably, I should consider writing a new post in my blog about this sometime :)

    # What kind of commitments / goals would you set to the project management?

    Goals to improve testers skills. A tester should be trained on situational awareness and skills listed in Rapid Software Testing ( http://www.satisfice.com/rst.pdf )

    Set a goal to not waste time in documentation that is WASTEFUL. For example: A test plan document, is prepared in a few CMM certified organizations for the sake of following process but they freeze it before the testing starts. If your plan freezes that means you are not open to changes. If you are not open to changes, you are not doing a good testing. Don’t blame the testers for not performing well after freezing your plan.

    # How would you track the progress on your testing between a given time frame for testing? (let’s say you have 5 days on a medium sized application)

    If I am the person who is supposed to track and this is a high priority work, I would first involve myself in testing with other testers, ensure that testers understand the mission towards which they are working, charter their mission, motivate them with some prizes for most number of bugs per day ( shell out from my pocket and not wait for the organization to approve this little stuff ), Go around the company – look for people who are free for the day, involve them in testing work with prize guarantee for every critical bug of course.

    Doesn’t that sound like something that has the real potential to find more bugs than spending time in meetings to track the progress of testing and sit in cafeteria with all testers and think of strategies of solving the problem.

    # when and how would you know you have tested enough? (remember you have some basis in traditional test case writing approach to track on – though with constraints)

    I know that when development manager comes to me ( Test Manager ) and say, “Can you take your testers out for a treat at my expense? Looks like, I might lose my developers if your testers keep pumping bugs like these”

    Again, that’s one such.. it depends on context and all above replies are with with assumptions that such a context exists or I have lived/heard in/about the contexts.

    People who would rule their testing business in future would do Exploratory Testing, in a skilled way and they would be smart enough of not revealing it outside so that their competitor knows the secret of their success.

    Comment by Pradeep Soundararajan — March 29, 2007 11:11 pm @ 11:11 pm | Reply

  2. Thank you for the detailed explanation, Pradeep.

    Giving right and prdictable status to the project management is where I think I have a problem going by exploratory testing. I have a team of 5 people working on a web UI application. It is a v1 application and we have a lot of ambiguities in the functional spec. We are testing the spec now at the same time writing the test cases off of the spec. We are following a SDLC / waterfall process for the current project and hence the group believes in test prep during the build phase and then moving on to the test execution phase. We are using this Build phase for functional specification document analysis, test case writing, test data preparation, UI automation, etc. We also have estimated the amount of time it could possibly take to execute each test case in the form of low/medium/high complexity. This kind of data helps in estimating the execution time.

    Any project, by definition, has end date attached to it. So, if we are delivering an application within 3months of time, and assuming end of 3months is a hard date that needs to be met for a set of pre-defined scoped application, Dev and test teams will need to work around the schedule and plan accordingly. This kind of data helps in better planning and execution. (Well, we always have constraints on how good the code will be).

    At this moment as we are reaching a situation where we don’t have enough time for test case prep due to spec ambiguities and requirements being fluid and change control in place, we are thinking of directed ad-hoc testing and of course the goal is to find more bugs and have them fixed (based on criticality in nature) to deliever a quality product. So, given this situation and we have 4 weeks of testing time with lot of things in scope – my challenge with exploratory is
    1. How much time should I/we set aside for exploratory testing? (we also have bunch of test cases written that need to be covered for execution)
    2. How would we report this to the project management on what percentage is complete on a daily basis?

    I like the idea of involving others in testing the application. That is something we have planned in the name of “Bug Bash” for each regression pass. We are also involving the customers on our Beta release to ensure we are meeting the right requirements and incorporate any missing features early and minimize churn when it actually goes to user acceptance testing phase.

    Now, regarding “2. The planning is usually done by the Lead or Manager after consulting testers on the estimated time to execute the tests. Poor testers, say a number that they think is good for them but later realize the product is more complex than they imagined, or some of the test cases were not good enough or so on … this makes testers to fear adding new tests to the test suite to reduce their own pressure of finishing things on task. Management never know that they are losing something important whenever testers meet the deadline by manipulation.
    ——-> I understand we need to let the brain flow free to be creative and find the right bugs. At the same time, you always need to set the destination or a goal. One may say “finding more bugs and testing until bug count becomes less is a Goal”; I agree, but unless you commit to the business customer on delivering a product within a timeframe, it doesn’t become a goal but just an objective. correct?

    Thanks again
    -Ram

    Comment by ramsblog — March 29, 2007 11:52 pm @ 11:52 pm | Reply

  3. >

    No one other than you can answer that question but I can help you answer that question.

    You need not have a focus exploratory session but mix very low cost exploratory tests with your scripted test. I think this is a pointer for you – http://testertested.blogspot.com/2007/03/mixing-scripted-and-exploratory-testing.html

    Also watch out Elizabeth Hendrickson’s cheat sheet and Rapid Software Testing heuristics to help your testers design low cost ( less time consuming) tests that don’t add to over head and yet are most of the times powerful than your scripted tests.

    Your testers are doing ET but they aren’t informing you or they aren’t aware of it.

    >

    # What percentage of what is complete?

    # If you say, “my team found 500 bugs today”, would they assume just 10% is left?

    You could tell a story and only that says information. Michael Bolton, once asked a test manager – how much do you love your wife?
    The manager reply was a story and a series of events and said finally said that there is lots more and he can’t express it completely. Michael then asked, why not you try the same with a number?
    That question from Michael Bolton left the manager thinking about the metrics and the need for a story.

    Your management would be happy with anything that helps towards achieving more customer satisfaction. If you could tell them a short story of how you have been doing on a daily basis, that should convince them. If they still ask for numbers, give it but warn them that numbers are misleading with examples of how numbers have mislead them so far.

    >

    More compared to what?
    Less compared to what?

    You must read this fairy tale or fable from Jerry Weinberg and also your management and customer read this : http://www.ayeconference.com/Articles/TestTrimmingFable.html

    Comment by Pradeep Soundararajan — March 30, 2007 12:18 am @ 12:18 am | Reply

  4. 1. How much time should I/we set aside for exploratory testing? (we also have bunch of test cases written that need to be covered for execution)

    No one other than you can answer that question but I can help you answer that question.

    You need not have a focus exploratory session but mix very low cost exploratory tests with your scripted test. I think this is a pointer for you – http://testertested.blogspot.com/2007/03/mixing-scripted-and-exploratory-testing.html

    Also watch out Elizabeth Hendrickson’s cheat sheet and Rapid Software Testing heuristics to help your testers design low cost ( less time consuming) tests that don’t add to over head and yet are most of the times powerful than your scripted tests.

    Your testers are doing ET but they aren’t informing you or they aren’t aware of it.

    2. How would we report this to the project management on what percentage is complete on a daily basis?

    # What percentage of what is complete?

    # If you say, “my team found 500 bugs today”, would they assume just 10% is left?

    You could tell a story and only that says information. Michael Bolton, once asked a test manager – how much do you love your wife?
    The manager reply was a story and a series of events and said finally said that there is lots more and he can’t express it completely. Michael then asked, why not you try the same with a number?
    That question from Michael Bolton left the manager thinking about the metrics and the need for a story.

    Your management would be happy with anything that helps towards achieving more customer satisfaction. If you could tell them a short story of how you have been doing on a daily basis, that should convince them. If they still ask for numbers, give it but warn them that numbers are misleading with examples of how numbers have mislead them so far.

    One may say “finding more bugs and testing until bug count becomes less is a Goal”; I agree, but unless you commit to the business customer on delivering a product within a timeframe, it doesn’t become a goal but just an objective. correct?

    More compared to what?
    Less compared to what?

    You must read this fairy tale or fable from Jerry Weinberg and also your management and customer read this : http://www.ayeconference.com/Articles/TestTrimmingFable.html

    Comment by Pradeep Soundararajan — March 30, 2007 12:20 am @ 12:20 am | Reply

  5. Thank you, Pradeep.

    Comment by ramsblog — April 1, 2007 12:17 am @ 12:17 am | Reply

  6. Thank you, Pradeep.

    May be I am skeptical – or a numbers minded. It appears from my personal style, I seem more analytical and study the numbers / facts. However, I understand the numbers might be misleading. I understand some reporting are always relative, like one of my questions above on an objective – but I still get down’to the numbers/percentage to know where my team is and where my project is towards meeting the commited date. So, if I have an end date of say, Mar 15th – for a piece of software to test and considering we have scripted test cases and also have heuristics to do exploratory testing, i still want to be able to provide my status (at least to myself) as to where I am with respect to meeting the date.

    From the fable’s example, it is agreed that the 3hrs of effort is what it takes to bake a cake and no shortcuts. However, 3hrs is an estimate that was given by the baker and he/she knows it takes taht time to complete. So, back to my ad-hoc testing – considering we have provided a similar estimate and what I referred in previous paragraph as Mar 15 – what is the level of confidence to say, I can be done by then. OR, does it mean, I won’t know about it until I am done. This is something I heard from someone working on SCRUM project model.

    Comment by Ram — April 30, 2007 12:27 am @ 12:27 am | Reply

  7. […] Tech — ramsblog @ 12:52 am Talking about Exploratory testing and the project management  [link] a few weeks ago, Pradeep from Bangalore, had shared a fable about TestTrimming or shortcuts. It […]

    Pingback by Exploratory Testing and documentation... « RamsThoughts — April 30, 2007 12:52 am @ 12:52 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 55 other followers

%d bloggers like this: