When we code a software, we choose an IDE which typically runs on an OS of a specific locale, having some Service Pack, co-existing with some software and so on. We compile the code and choose relevant targets, the compiler spits a binary and your job as a coder is done. In a complex world, there would be a build system which would get all the relevant files from relevant branches , from a source code control system , compile them and trigger the installer scripts. Installer script would create a package out of it. And its done.
When the package reaches the software tester, he has a big problem to solve. The first question which he has to answer that whether he should install the s/w on a fresh clean OS which has been installed just now, warm and inviting. OR he should rather install it on a dirty OS which has a plethora of software and would be more close to end-user. The answer is simple, do both.
The task doesn’t finish there, now comes the question about
* Which operating system, since it runs on all Windows. So should I install and make a clone of myself so that all of my clones can do simultaneous testing. To give you an example, MS Word runs on Windows and MAC. Within windows, it runs on Windows XP, Windows Vista and Windows 2000 and some other less popular flavors as well. Within Windows XP, it runs on all flavors of XP viz. SE, MCE, Professional. Win 2K has many more flavors and with Win Vista you need to do a MS certification to really say with surety on how many flavors it has. I am yet to talk about ‘Service Packs’.
You ask this question to a non-software-tester group head and he would say ‘Do it on All’.
* Assuming that your s/w either runs on browser then this would mean IE, FireFox, Safari and Netscape, if you ignore the others. With each browser there are multiple versions of each.
I can probably go on by adding variables like Locale, compatibility by installing other s/w, 32 bit – 64 bit versions, processors, cards, memory etc etc.
At the end of it, it would come down to Software Test Manager to decide on what all test-beds configurations one needs to test. So how as a software tester you should decide on the most optimized test-bed configuration matrix. Well, I do not have a silver bullet but I can share with you my plan on this. Essentially a bulleted list of items which might help you to do a better job in choosing a right test-bed config matrix as well as executing on that plan.
1. Find out what your users have. Ask Marketing or Product Management or your client to identify the most popular config. For example, if there is a web-photo site which helps you to share your photos with friends, then I would guess that most common test bed would be
OS – Windows XP Professional with latest service pack
Browser – Internet Explorer latest version if the latest has been there for at least 6 months, else use one but latest
Ratify this by high-n-mighty and then have this as a default test-bed for all the testers.
2. Once 1 is done then find out the next best test-bed config. Same formula as above. If your organization can support then have the secondary test-bed config available to all the testers all the time. At my place we usually have at least two machines so it works. We have a laptop for e-mails and other stuff so one can re-image the test systems all the time.
3. Dont go any beyond, just refuse. Its insane to continually test on more then 2 configs.
4. For all the other x configs, prepare a single machine of each config in your test lab and then have each of your software testers execute their cases on them, say couple of hours once a week. So at the end of first week, each tester would have done 2 hours worth of testing on 5 more configs, assuming that they are able to find a different config for 2 hours every day.
5. From 1-4, you probably would be able to cover all the popular configs but you still need to certify that your s/w at least does basic things on those really special and exotic configurations. Say running MS Word on a MS terminal server or something like that. For these, have someone who can prepare this config on 5 machines and then do a ‘bug hunt‘ for this config for two hours. Repeat this testing if you find many defects but else dont end up spending too much time on this.
6. Finally, go back and read 80-20 rule. Optimize your testing for 80 % of people and in most of the cases they can be covered by doing 1 and 2, so be realistic and enjoy.