Distributed Deployment and Testing

January 28, 2008 by Isaac

I love Google videos. They have many great videos about software testing by some of the most renowned software testers in the industry. A couple weeks ago, while looking for new software testing videos, I stumbled on something called SmartFrog. SmartFrog was designed to be a distributed deployment system, but it can also be used for distributed testing.

I have been pondering lately on the topic of testing frameworks. There is a need at my work to create a new framework that can adapt to future needs. The current framework is about 6 years old and is brittle; it breaks very easily. The problem is that this framework is too brittle to just keep upgrading it. Rather than having a master plan for the framework it has just evolved into the beast that it is now. So, we are stuck. It has so much functionality now, but it is too brittle to keep patching.

My "ideal" framework will allow us to have a grid of machines that can be dynamically allocated to perform specific tasks. These machines would then be configured and have the appropriate software deployed. Test scripts would then run on each of the machines, according to their purpose, and the results would be centrally logged. Finally, anything done to the machine would be reversed and the machine returned to the pool of resources.

Using Open Source tools and a bunch of glue I imagine something like this would work:

• Torque to manage the resource pool.
• SmartFrog to deploy the software on the machines
• STAF/STAX to execute test scripts on each of the machines

This framework would also need some sort of job and network monitoring software. Each of these comes with their own tools, but it might be necessary to create a custom integrated monitoring application. There are also other issues that need to be addressed, such as file space and initial deployment. Should all the machines share a common file system, such as as PVFS, or should they have their own individual file systems? Another issue that I'd like to tackle is the initial installation of the machines. I like the approach that Rocks Clusters takes where installation of a new node is as simple as turning on the power. They distribute node bootstrap code via PXE and TFTP, which then installs a full version of Linux onto the machine. I have not fully investigated it yet, but I know there are "Rolls" for PVFS and Torque. I would probably have to create my own Roll for SmartFrog. The downside of using Cluster Rocks is that there is a single master node for the cluster and all the computational nodes are hidden behind that one node. I am not sure if this would work for our testing.

Anyway, there are some random thoughts on testing frameworks. Within the next couple months I want to leave the research and gathering info phase and start formal architecture. In the mean time, here is the Google video about SmartFrog.

Posted in Software Testing