When we code a software, we choose an IDE which typically runs on an OS of a specific locale, having some Service Pack, co-existing with some software and so on. We compile the code and choose relevant targets, the compiler spits a binary and your job as a coder is done. In a complex world, there would be a build system which would get all the relevant files from relevant branches , from a source code control system , compile them and trigger the installer scripts. Installer script would create a package out of it. And its done.
When the package reaches the software tester, he has a big problem to solve. The first question which he has to answer that whether he should install the s/w on a fresh clean OS which has been installed just now, warm and inviting. OR he should rather install it on a dirty OS which has a plethora of software and would be more close to end-user. The answer is simple, do both.
The task doesn’t finish there, now comes the question about
* Which operating system, since it runs on all Windows. So should I install and make a clone of myself so that all of my clones can do simultaneous testing. To give you an example, MS Word runs on Windows and MAC. Within windows, it runs on Windows XP, Windows Vista and Windows 2000 and some other less popular flavors as well. Within Windows XP, it runs on all flavors of XP viz. SE, MCE, Professional. Win 2K has many more flavors and with Win Vista you need to do a MS certification to really say with surety on how many flavors it has. I am yet to talk about ‘Service Packs’.
You ask this question to a non-software-tester group head and he would say ‘Do it on All’.
* Assuming that your s/w either runs on browser then this would mean IE, FireFox, Safari and Netscape, if you ignore the others. With each browser there are multiple versions of each.
I can probably go on by adding variables like Locale, compatibility by installing other s/w, 32 bit – 64 bit versions, processors, cards, memory etc etc.
At the end of it, it would come down to Software Test Manager to decide on what all test-beds configurations one needs to test. So how as a software tester you should decide on the most optimized test-bed configuration matrix. Well, I do not have a silver bullet but I can share with you my plan on this. Essentially a bulleted list of items which might help you to do a better job in choosing a right test-bed config matrix as well as executing on that plan.
1. Find out what your users have. Ask Marketing or Product Management or your client to identify the most popular config. For example, if there is a web-photo site which helps you to share your photos with friends, then I would guess that most common test bed would be
OS – Windows XP Professional with latest service pack
Browser – Internet Explorer latest version if the latest has been there for at least 6 months, else use one but latest
Ratify this by high-n-mighty and then have this as a default test-bed for all the testers.
2. Once 1 is done then find out the next best test-bed config. Same formula as above. If your organization can support then have the secondary test-bed config available to all the testers all the time. At my place we usually have at least two machines so it works. We have a laptop for e-mails and other stuff so one can re-image the test systems all the time.
3. Dont go any beyond, just refuse. Its insane to continually test on more then 2 configs.
4. For all the other x configs, prepare a single machine of each config in your test lab and then have each of your software testers execute their cases on them, say couple of hours once a week. So at the end of first week, each tester would have done 2 hours worth of testing on 5 more configs, assuming that they are able to find a different config for 2 hours every day.
5. From 1-4, you probably would be able to cover all the popular configs but you still need to certify that your s/w at least does basic things on those really special and exotic configurations. Say running MS Word on a MS terminal server or something like that. For these, have someone who can prepare this config on 5 machines and then do a ‘bug hunt‘ for this config for two hours. Repeat this testing if you find many defects but else dont end up spending too much time on this.
6. Finally, go back and read 80-20 rule. Optimize your testing for 80 % of people and in most of the cases they can be covered by doing 1 and 2, so be realistic and enjoy.
I agree, but to an extent only. Most product companies need to print on the box – what all does this software support.
Taking your example of photo site might not be the best here but let me take example of a photo sharing software which resides on the desktop. A company who builds this product needs to put the list of supported OS, browsers, locales etc on the box. If Testing dept of that company hasn’t tested on say Vista (or IE8), this company can’t afford to put Vista or IE8 on the box as supported environment.
A customer who is using IE8 and Vista, when walks into the store to buy this software, would first read abt the supported software and would return the box back to the shelf not finding both of them there.
Not to mention, this whole scenario gets magnified to a different level when it comes to enterprise software. Million dollar deals could be lost just because your software didn’t support a particular version of browser or application server.
While 80-20 works for bugs, but then there are other criteria in the business world which bind testers to follow the matrix market defines…
Thanks Avinash for reading and sharing your thoughts on this.
Probably my post was not very clear but if I have to test a s/w as you mentioned and if Vista+IE8 doesn’t fall under 1 and 2 (default and 2nd default) then I would spend my testing time on this combination through 4 and 5.
As you could imagine, a mass market desktop software probably would end up running on every possible combination and in reality you wont have time and resources to test on every thing so you start optimizing.
I understand that in case of Enterprises , one would need to certify different configs because it may happen that a big client is using one of the configs and he wont change his h/w because of you. About 8 years back, I was working on an Enterprise Backup software and we had same issues. We did use to fall back on standards to save ourselves from certifying every thing. Essentially if a server supports HTTP then I wont bother too much on whether its a IIS or Apache. I would do most of my testing on most common and 2nd most common and then utilize 3 and 4 activities for certifying other configs.
Thanks again, probably your comment is the exact reason, I wrote this up. In my personal opinion, we still have to perfect this art and we are still learning. Before we reach there, we would end up doing much more then necessary to be safe.