Open source testing tools target varied tasks

April 18, 2008 (4:00:00 PM)
By: Mayank Sharma

Testing is an important function of the software development process, no matter how big or small the development project. But not every company or developer has access to professional testing tools, which can run into hundreds and even thousands of dollars. The good news is that they don't need them, thanks to the tons of freely available open source software testing tools.

In simple terms there are two major approaches to testing software -- the manual way (a summer intern with a checklist) or through an automated program. With automated testing programs, you can spend a lot of money procuring these tools or distracting yourself from the task at hand by rolling out your customized automated testing software.

Instead, you could head over to sites like Open Source Testing (OST), QAForums, Open Testware, and others that catalog various testing tools and look for something that works for you.

"The largest category of open source testing tools," says Mark Aberdour, who manages OST, "is functional testing tools. This can cover a range of practices from capture-replay to data-driven tests, from Web application testing to Java application testing, and lots more in between." Aberdour, who before his current software development role spent 10 years on the other side of the fence in software testing and test management, says that the list of open source testing tools also includes many performance testing tools and test management/defect tracking tools, as well as a good number of security testing tools.

"If you include unit testing tools, then there are large numbers of tools for the more popular languages, particularly where test-driven development (TDD) is more popular," he says. There are several tools for testing Web and Java applications, but "as is the way in open source," Aberdour says, "if there's an itch, someone will scratch it, so there are tools available for all manner of obscure needs." OST lists testing tools for languages such as PHP, Perl, Ruby, Flash/ActionScript, JavaScript, Python, Tcl, XML, and so on. "The list is probably bigger than I've brought together on OST, and TDD practitioners should head over to testdriven.com, which has more focus on that area."

So how do they compare with the expensive proprietary tools? "In some cases, very well," says Aberdour. He points out WebLOAD and OpenSTA as examples that hold up well in the performance testing market -- no surprise there, since they were both originally commercial tools. Underlining his point, Aberdour says, "You have tools like Wireshark, which is huge in the security market, and Bugzilla and Mantis in the defect tracking sector. In functional testing there are a number of really great tools (Selenium, Abbot, Jameleon, jWebUnit, Marathon, Sahi, soapui, and Watin/Watir to name a few) with strong feature sets."

According to Aberdour, in addition to the tool itself, the advantage of the commercial tools is often their integration with an automated software quality (ASQ) suite, and of course an established company behind them, which will feature in a lot of people's selection criteria. Here again things are looking good for open source testing tools. Aberdour points out the RTH test case tool which integrates with Watir, HttpUnit, JUnit, MaxQ and the commercial WinRunner. Even bug trackers like Mantis and Bugzilla integrate with functional testing tools and test case tracking tools.

When it comes to reliability and accuracy, Aberdour says the classic open source arguments apply. The popular tools are tested and used by a high number of people, and many of the tools have evolved large and sustainable communities with many people feeding back on quality-related issues. "For many of these tools, innovation is high and release processes rapid. Yet it is not a free-for-all -- the code base may be open, but write access to the repository is closely guarded, and developers have to earn the right to commit."

According to Aberdour, the main issue for people using the tools is not whether their need is served, since in most cases you will find a tool to do the job, but whether the tool is mature enough to invest in. "Test automation is a major undertaking that has high costs in terms of upskilling your test team and creating test scripts, etc., and people need to know that the tool is a good investment, whether or not there is a license fee to pay. People will be asking if the tool is going to be around in five years' time, what levels of support are available, how good is the feature innovation, bug fixing, release schedules, and so on." Aberdour thinks there is a big gap in selection and evaluation support and paid technical support services, which is what he wants to focus on next with OST.

Of course there are strong and sustainable communities around the more popular tools that provide excellent support, but Aberdour says the market isn't at the point yet where there is a lot of commercial support available. Yet there are some examples of support companies evolving around open source products, and a lot of the product teams will hire themselves out. "In a commercial sense this market is still quite young, but it will mature, and when the product integration, community maturity, and commercial support are right, it has the potential to be highly disruptive. Gartner reckons the ASQ market is worth $2 billion a year, and while Hewlett-Packard and Mercury absolutely dominate, there are hundreds of smaller proprietary vendors at real risk from open source disruption. Things are certainly moving in the right direction."

Open source testing tools offer great performance and are a bargain compared to proprietary testing tools. The lack of a formal and consistent support structure might work against some tools being used to test mission-critical apps. But if you are an open source developer or with a software development company pondering over your testing budget, spend some time checking out the testing tools Web sites and forum boards. You might save yourself some serious bucks.