Part IV: Supplementing Your Testing SE-307 Software Testing ContentsnAutomated Testing and Test ToolsnBug Bashes and Beta TestingPart IV: Supplementing Your Testing Automated Testing and Test ToolsContentsnThe Benefits of Automation and ToolsnTest ToolsnSoftware Test AutomationnRandom Testing: Monkeys and GorillasnRealities of Using Test Tools and AutomationThe Benefits of Automation and ToolsnBackgroundnRegression testing: This process of rerunning your tests is known as regression testing. nThe work of regression testing is enormousnSoftware test tools and automation can help solve this problem by providing a more efficient means to run your tests than by manual testing. The Benefits of Automation and ToolsnThe principal attributes of tools and automation arenSpeed. nEfficiency. nAccuracy and Precision. nResource Reduction. nSimulation and Emulation. nRelentlessness. nNOTEnSoftware test tools arent a substitute for software testers they just help software testers perform their jobs better.Test ToolsnAs a software tester youll be exposed to a wide range of testing tools. The types of tools that youll use are based on the type of software that youre testing and whether youre performing black-box or white-box tests.nImportant distinction between two types of tools non-invasive and invasive. nIf a tool is used only to monitor and examine the software without modifying it, its considered non-invasive. nIf, however, the tool modifies the program code or manipulates the operating environment in any way, its invasive. Test ToolsnThe major classes of testing tools and how theyre used. nViewers and Monitors nA viewer or monitor test tool allows you to see details of the softwares operation that you wouldnt normally be able to see. nCode coverage analyzers, communications analyzer, the code debuggers that come with most compilers. Test ToolsnDrivers nDrivers are tools used to control and operate the software being tested. nBatch file, replace the keyboard and mouse of the system being tested with an additional computer. nStubs nStubs, like drivers, were mentioned in Chapter 7 as white-box testing techniques. nEmulatorTest ToolsnStress and Load Tools nThe Stress program allows you to individually set the amounts of memory, disk space, files, and other resources available to the software running on the machine. nLoad tools are similar to stress tools in that they create situations for your software that might otherwise be difficult to create. nInterference Injectors and Noise GeneratorsnTheyre similar to stress and load tools but are more random in what they do. Test ToolsnAnalysis Tools nWord processing softwarenSpreadsheet softwarenDatabase softwarenFile comparison softwarenScreen capture and comparison softwarenDebuggernBinary-hex calculatornStopwatchnVCR or cameraTest Toolsn测试设计工具n用例生成，随机抽取n测试管理工具n测试运行跟踪，缺陷跟踪工具Software Test AutomationnMacro Recording and PlaybacknThe most basic type of test automation is as you run recording your keyboard and mouse actions your tests for the first time and then playing them back when you need to run them again. nMacro recorders and players are a type of driver tool. Drivers are tools used to control and operate the software being tested. nOn the Mac you can use QuicKeys; on Windows the shareware program Macro Magic is a good choice. nBe careful: nThe biggest problem is lack of verification. nA slight change in the GUI could throw off your captured steps. Software Test AutomationnProgrammed Macros nCreate programmed macros by programming simple instructions for the playback system to follow. nProgrammed macros such as this one have some real advantages over recorded macros. Although they still cant perform verification of the test results, they can pause their execution to prompt the tester (see Figure 15.8) with an expected result and a query for her to okay whether the test passed or failed. Software Test AutomationnProgrammed Macros (continue)nListing 15.1. A Simple Macro That Performs a Test on the Windows Calculatorn1: Calculator Test #2n2:n3: n4: 123-100=n5: n6: Software Test AutomationnProgrammed Macros (continue)nYoure still missing two important pieces, though, to perform complex testing.nProgrammed macros are limited to straight-line execution they can only loop and repeat. Variables and decision statements that youd find in a regular programming language arent available. nYou also dont have the ability to automatically check the results of your test. For these, you need to move to a comprehensive automated testing tool.Software Test AutomationnFully Programmable Automated Testing Tools nThe most important feature that comes with these automation tools is the ability to perform verification, actually checking that the software is doing whats expected. There are several ways to do this: nScreen captures. nControl values. nFile and other output. Software Test AutomationnFully Programmable Automated Testing Tools (continue)nTo get more information about several of the popular test automation products available, visit the following websites:nSoftware Development Technologies at www.sdtcorp.comnMercury at www.mercury.comnSegue Software at www.segue.comRandom Testing: Monkeys and GorillasnAnother type of automated testing, though, isnt designed to help run or automatically run test cases. Its goal is to simulate what your users might do. That type of automation tool is called a test monkey. nWhen your software is released to the public, it will have thousands or possibly millions of people using it. Despite your best efforts at designing test cases to find bugs, some bugs will slip by and be found by those users. Random Testing: Monkeys and GorillasnDumb MonkeysnIt just clicks or types randomly. n例：1：RANDOMIZE TIMERn 2: FOR i=1 TO 10000n 3: PLAY “CLICK”+STR$(INT(RND*640)+”,”+STR$(INT(RND*480)+”n 4:PLAY CHR$(RND*256)n 5:NEXT i Random Testing: Monkeys and GorillasnDumb MonkeysnIt doesnt seem to make sense that simple random clicking and typing could find a bug, but it does for a couple reasons:nGiven enough time and attempts, just like the monkeys writing Shakespeare, the random inputs will eventually stumble onto a magic sequence that the programmers and testers didnt think of. nA dumb monkey, with its continuous repetition and use, can expose bugs such as memory leaks that might not occur until many hours or days of normal use.Random Testing: Monkeys and GorillasnSemi-Smart Monkeys nDumb monkeys lack a few important features （如故障的重现）, though, that would make them even more effective. Adding these features （如日志或屏幕录像 ）raises your monkeys IQ a bit, making him semi-smart. nThe solution is to add logging to your monkey so that everything it does is recorded to a file. When the monkey finds a bug, you need only to look at the log file to see what it was doing before the failure. nIts also a good idea to program your monkey to operate only on the software youre testing. nAnother good feature to make your monkey semi-smart is crash recognition. Random Testing: Monkeys and GorillasnSmart Monkeys nSuch a monkey takes the effectiveness of random testing from his less-intelligent brothers and adds to that an awareness of his surroundings. He doesnt just pound on the keyboard randomly he pounds on it with a purpose. nA smart monkey can examine data as it goes, checking the results of its actions and looking for differences from what it expects. If you programmed in your test cases, the smart monkey could randomly execute them, look for bugs, and log the results. Realities of Using Test Tools and AutomationnTest automation isnt a panacea. When it is properly planned and executed it can make your testing immensely more efficient and find bugs that would have otherwise gone undiscovered. nHowever, countless test automation efforts have been abandoned and cost their projects dearly when they went astray. Realities of Using Test Tools and AutomationnYou should consider these important issues before you begin using the techniques described in this chapter:nThe software changes. nTheres no substitute for the human eye and intuition. nVerification is hard to do. nIts easy to rely on automation too much.nDont spend so much time working on tools and automation that you fail to test the software. nIf youre writing macros, developing a tool, or programming a monkey, youre doing development work. nSome tools are invasive and can cause the software being tested to improperly fail. nHighlights of this chapter includenWhy test tools and automation are necessarynExamples of simple test tools you can usenHow using tools migrates to test automationnHow to feed and care for monkeysnWhy test tools and automation arent a panaceaPart IV: Supplementing Your Testing Bug Bashes and Beta Testing Bug Bashes and Beta TestingnHaving Other People Test Your SoftwarenTest SharingnBeta TestingnOutsourcing Your TestingHaving Other People Test Your SoftwarenYoure likely under a tight schedule, you find as many bugs as possible in the time you have, but someone else can come in, test the same code, and find additional bugs. nThere are several reasons and solutions for it: nHaving another set of eyes look at the software helps break the pesticide paradox. nSimilarly, people dont just see differently from each other, they go about their testing differently, too. nHaving someone assist you in your testing helps eliminate boredom. nWatching how someone else approaches a problem is a great way to learn new testing techniques.Test Sharing nOne common approach is to simply swap test responsibilities with another tester for a few hours or a few days. Think of it as You run my tests and Ill run yours. nA fun way to share the testing tasks is to schedule a bug bash. A bug bash is a period of time (usually a couple hours) in which the entire test team stops their normally assigned testing tasks to participate in the bash. In a bug bash, a specific area of the software is selected and all the testers concentrate their testing on that one area or set of features. nA bug bash could determine if the bugs have been missed by normal testing or if its just well written code. nIts a good idea to get your product support team helping you test the product early in the design cycle to help identify and fix usability bugs. Beta TestingnBeta testing is the term used to describe the external testing process in which the software is sent out to a select group of potential customers who use it in a real-world environment. nBeta testing usually occurs toward the end of the product development cycle and ideally should just be a validation that the software is ready to release to real customers.Beta TestingnThere are several things to think about when planning for or relying on a beta test:nWho are the beta testers? nSimilarly, how will you know if the beta testers even use the software? nBeta tests can be a good way to find compatibility and configuration bugs. nUsability testing is another area that beta testing can contribute to if the participants are well chosen a good mix of experienced and inexperienced users. nBesides configuration, compatibility, and usability, beta tests are surprisingly poor ways to find bugs. nA beta test program can take up a lot of a testers time. Outsourcing Your Testing nConfiguration and compatibility testing are typically good choices for outsourcing. nLocalization testing is another example that often lends itself to outsourcing. nThe success, or failure, of the outsourcing job may well depend on you. Heres a list of things to consider and to discuss with your test manager or project manager to help make the job run more smoothly: nWhat exactly are the testing tasks that the testing company is to perform? Who will define them? Who will approve them?nWhat schedule will they follow? Who will set the schedule? What happens if the deadline is missed?Outsourcing Your TestingnWhat deliverables are you to provide to the testing company? The softwares specification, periodic software updates, and test cases are some examples.nWhat deliverables are they to provide to you? A list of the bugs they find would be the minimum.nHow will you communicate with them? Phone, email, Internet, central database, daily visit? Who are the points of contact at both ends?nHow will you know if the testing company is meeting your expectations? How will they know if theyre meeting your expectations?nHighlights of this chapter includenWhy its important to have other people testnHow you can get others looking at your softwarenWhat beta testing is and how testers are involvednHow to effectively outsource your test work