If you are software test engineer like me then almost every one in your office would have at least told you once that we must do automation and more if we are already doing it. Most of these people would be from non-testing function and probably they dont know enough and hence their feedback on doing-automation or doing-more-automation makes lot of sense, these guys are not biased and must be genuinely interested in seeing more automation.
To most of them, you would have replied with a sort of sheepish smile (like the one you just did), or with a mono-syllable answer, mostly Yes, or may be few lines, incase its an executive who is asking/suggesting.
But is doing automation or more automation really makes as much sense as everyone around, esp the non-testing ones, thinks so. Even though most of you would rate this question as rubbish, I thought that it may not and on that premise I started to think about general testing scenarios or software testing in general. And after some brooding over the subject in my large window office which opens to a big green field, I think that probably it does in an average case but it may not in all the other cases and who knows that your case is the one where it doesn’t make any sense at all to do any automation.
Lets try to think about some of the cases where it doesn’t.
1. If your job is to test something through qualitative analysis.
– For example, if you are searching whether a particular encoding of raw footage gives you a high quality video. There are some parameters which can be quantified say bps, total key frames etc and they help you in measuring over all quality but what about detail, colors and likewise.
– Similar would be the case of image encoding. When we switch formats and save a particular data in a format, do we lose detail and if yes then how much. When we save a file as JPG, we set a Q factor and its all subjective.
– If you are testing something like Red Eye Fix. For standard images, we can train the system to identify an image which has a Red Eye and a one which has this problem fixed but probably you would want this to be hand-tested rather then machine tested.
2. If investment to setup a test harness is much more then the returns.
– too short a test window to warrant any returns, better to hand test and move
– too complex to script
– testing s/w which are close to their end-of-life so you can’t re-use the scripts
– etc
3. Manual testing is easy and is not really a issue
– trivial work. Say testing a teeny-weeny functionality once in a while
4. To automate, you have to have different people then the ones who are currently testing
– automation testing is most effective when its best done by people who know testing. dont hire programmers and ask them to test. If you can’t get automation done by your testing people then better to hire folks who can do testing as well as automation, if you can’t then better to not automate
And I am sure there must be other 10 (or four if you are really alert) reasons to not automate. w At Adobe, in my group, we do automate and we really want to automate 🙂