Sometimes the most interesting bugs are discovered by pure accident.  I was picking up a few items at one of the United States major pharmacy chains and discovered a flaw in their software.  But first, let me back up and explain something called IIAS.

Inventory Information Approval System (IIAS) is used (or supposed to be used) by all the grocery stores and drug stores through the USA.  When you go to make a purchase of eligible goods using your Flexible Spending Account (FSA) credit card the IIAS is consulted and the register gets a "yes" or "no" on the item.  If the item is eligible for FSA spending and a FSA card is presented for use then only those items given a thumbs up from IIAS are supposed to be charged to the card.  The remaining items are supposed to be paid for by some other means.

At the end of my little shopping trip to the pharmacy I had some items that I knew were eligible for FSA and some that weren't.  After the cashier scanned all the items I swiped my FSA card.  Here's where the accident happened.  The cashier, obviously unaware of FSA cards or the IIAS, was confused by the message her register was giving her.  So told me the card hadn't worked and asked me to swipe it again.  At this point I had no reason to doubt her, so I swiped my card again.  The sale completed and after a brief moment of confusion I realized what had happened.  IIAS and the FSA card authorization system had failed.

What happened was the first time I swiped my card it had actually worked and only the FSA eligible items had been charged to the card.  The cashier didn't know what happened or why the balance remaining was not zero.  So, she thought the card had failed and asked me to try again.  The second time the system should have recognized the card as FSA and that there were no more items eligible items and should have rejected the card.  It didn't.  It allowed the transaction to continue.

This really isn't a life or death bug, but it really is a nuisance.  I had them reverse the charges on the card and then check my items out again.  This time I swiped a different card the second time and everything was happy.  The irritation would come later if I hadn't had them credit the card back.  I would have been contacted and asked to pay back the card for the non eligible items.

This seems like a test case that should have been run but was missed.  Worse yet, it might have been run and a bug filed but never fixed.  I can only speculate why this behavior was allowed, so allow me to rant a bit about a common problem encountered while bug hunting.

By a show of hands how many testers have ever received the "no one will ever do that" response to a bug filed?  Why on earth would anyone swipe a FSA card twice or when no FSA items are pending payment?

A common push-back I've seen from developers is "our users are smarter than that."  OK, so I work on high technology products and our users should be smarter than that.  When it comes to software I don't think it makes a difference how smart your users are supposed to be.  If there is a flaw in the program sooner or later someone will do something they shouldn't have done and the flaw will show its ugly head.  There are certainly going to be higher priorities in many cases, but a wise tester will know when to push to get these "dumb" bugs fixed.

A couple years ago I had the privilege of listening to Lee Copeland speak.  I quite enjoyed his talk and in my subsequent study of some of his other material, such as his book A Practitioner's Guide to Software Test Design I have come to really respect and enjoy his knowledge.

One of the greatest resources of all time for a tester is Google Videos. There are so many great videos by professional testers that it is amazing. Anyway, enough of an intro. Here's the video from Lee Copeland.

Buy the book. It's a great resource.

A year or so ago I bought a prepaid Net 10 phone.  During this year I have had numerous support phone calls, web site issues, and billing issues.  First, the Net 10 web site does not work with Firefox, which is my browser of choice.  Since I only use Windows when necessary this causes me problems when I need to log on and recharge my phone.  Most of the time I bought phone refill cards from the store, however, the two times that I tried recharging the phone online I had to fight with customer service to get my minutes.  This defeats the reason to allow people to recharge their phones online.  The 2nd time I tried recharging my phone via the web site it failed and I was told to call customer service.  After a while on the line I finally got the answer that I needed a new SIM card.  The card arrived a week or so later and I installed it.  I then called customer service back and they were able to activate the card, however, from there it was all down hill.  When they activated the phone they changed my phone number to a Texas phone number.  I don't live in Texas.  Where did that come from?  To top that off the phone didn't even work.  I could not place calls.  After two weeks of trying to get this issue resolved with them unable to change my phone number back, or get my phone working I asked for a refund of my money.  That's the last time they talked to me.  To this day I am out $30 on my refill.

OK, stepping off my soap box.... What does this have to do with QA?  Someone tested the telephone software.  I am sure QA was performed on the customer management software that Net 10 uses, and doubtless audits have been performed on the customer support team.  Someone has surely also tested the web site.

So, if these pieces were tested separately then what exactly went wrong?  The customer management software obviously didn't give the support reps the info or power they needed to be able to fix my phone and I was never able to get routed to anyone who had the needed power.  I don't think the problem was that the management software was defective, but it did not fill the role it needed to fill.

This is one of the fundamental problems in software QA; a piece of software can meet all the requirements and it can be user friendly, but can still fall short.  This is because it did not meet the need.  In this case the software prompted them with all the normal troubleshooting tips, but didn't give the service reps the power they needed to dig into the problem and fix it.  Having said that they should have been able to transfer me to someone who could help, but they didn't.

The problems with the web site were twofold: First, it did not work across browsers.  Second, it may be functional by itself but it did not integrate well with the phone crediting system.  Integration testing is another issue that comes up a lot.  Whenever you have one system that talks to another system there is bound to be a whole swarm of bugs waiting to be discovered.

Of course I need to add the disclaimer that this post represents my opinion and my experiences and that your experiences and opinions may vary.

I have been pondering lately on the topic of testing frameworks. There is a need at my work to create a new framework that can adapt to future needs. The current framework is about 6 years old and is brittle; it breaks very easily. The problem is that this framework is too brittle to just keep upgrading it. Rather than having a master plan for the framework it has just evolved into the beast that it is now. So, we are stuck. It has so much functionality now, but it is too brittle to keep patching.

My "ideal" framework will allow us to have a grid of machines that can be dynamically allocated to perform specific tasks. These machines would then be configured and have the appropriate software deployed. Test scripts would then run on each of the machines, according to their purpose, and the results would be centrally logged. Finally, anything done to the machine would be reversed and the machine returned to the pool of resources.

Using Open Source tools and a bunch of glue I imagine something like this would work:

• Torque to manage the resource pool.
• SmartFrog to deploy the software on the machines
• STAF/STAX to execute test scripts on each of the machines

This framework would also need some sort of job and network monitoring software. Each of these comes with their own tools, but it might be necessary to create a custom integrated monitoring application. There are also other issues that need to be addressed, such as file space and initial deployment. Should all the machines share a common file system, such as as PVFS, or should they have their own individual file systems? Another issue that I'd like to tackle is the initial installation of the machines. I like the approach that Rocks Clusters takes where installation of a new node is as simple as turning on the power. They distribute node bootstrap code via PXE and TFTP, which then installs a full version of Linux onto the machine. I have not fully investigated it yet, but I know there are "Rolls" for PVFS and Torque. I would probably have to create my own Roll for SmartFrog. The downside of using Cluster Rocks is that there is a single master node for the cluster and all the computational nodes are hidden behind that one node. I am not sure if this would work for our testing.

Anyway, there are some random thoughts on testing frameworks. Within the next couple months I want to leave the research and gathering info phase and start formal architecture. In the mean time, here is the Google video about SmartFrog.

I used to work for a company that at the beginning of a project they would spend months on documentation. A full 1/3 of the project schedule was spent in this phase. All the latest in tools and UML diagrams and use cases was employed. The project was thoroughly planned out before any coding commenced. Sounds great, right? Nope. Once all of the documentation and modeling were completed they were never referenced again. The project proceeded along by verbal instructions. The systems architects would verbally tell the developers what they wanted, which usually differed from what was documented. Little wonder that the various pieces of the system didn't integrate well and the project ran *way* over schedule and budget (we're talking millions here). Of course in the end the test team caught a lot of flack for not doing their job and not catching all the bugs earlier. Gee, we only had a verbally defined moving target of a system to work with. Oh, and a developer to tester ratio of 10:1 didn't help either.

In any case, this company was CMMI and ISO certified. They had the processes and documentation to prove it. The problem, as is pointed out in the above article, is that the organization was sick and didn't realize it. The company kept trying to impose process to help alleviate these problems, but inertia kept things from moving in the right direction. Traditionally, the senior project staff always just told the junior staff what to do. By spending 1/3 of the project doing documentation they stuck to the letter of the law, but then immediately discarded all that work in favor of tradition. What happened is that the more senior management tried to implement processes the tighter the lowly developers and testers were squeezed and actually threatened (not a joke).

Contrast that to my present employer. They are neither ISO nor CMMI certified, but the corporate culture is very amiable to change and cooperation. Instead of a dictatorship of senior engineers there is a mutual respect for each other. There is still a clear management chain, but junior and senior engineers all work side by side as opposed to chain of command. The end result is that the project is actually on schedule and instead of enmity between developers and testers there is cooperation.

So, a "fool with a tool is still a fool." Or, maybe I should say "a manager with a process is still a manager."

In a nutshell, they create several modules that each accomplish one task. At the end of the task they return the system to a known state. Since each module returns to the same known state you can construct complex tests by stringing these modules together. Quite an interesting concept.

• http://www.beijinginsight.com/ftpspace/softwaretesting/mbt/1999-icse.pdf
• http://www.beijinginsight.com/ftpspace/softwaretesting/mbt/PSTT_2004_blackburn.pdf

There are a bunch of general papers on software testing here.

A call for chapters in an upcoming book is here.

• http://www.qaforums.com/
• http://www.stickyminds.com/
• http://www.sqatester.com/
• http://www.satisfice.com/
• http://www.testingeducation.com
• http://www.testingreflections.com
• http://blogs.msdn.com/micahel
• http://www.developsense.com/blog
• http://www.kohl.ca
• http://www.testing.com
• http://www.homeoftester.com/
• http://www.agilemodeling.com/
• http://www.testinggeek.com/
• http://brains4softwaretesting.blogspot.com/