Month: September 2007

Distributed automated quality assurance

Distributed continuous Quality Assurance

I’m gonna toss this google tech talk into the *wow, that sounds amazing, but how the he#$# would you implement something like that in real software systems without developing a testing harness of equal or greater complexity than the product under test … but there are practical lessons to be learned from theory” bucket.

Stumbling around QA videos on google tech talks tonight I happened to be interested enough in Adam Porter’s Google talk regarding distributed continuous integration testing to devote an hour of my life to this cause. The basic premise (obviously I’m skimming over vasts amount of detail given a 1hr speech is now condensed into 4 points) goes something along the lines of

a) Derive a list of legal configurations and variables for the system under test

b) Develop a centralized “Quality assurance space” server I’ll coin it the QAS which will hold the conditions under tests, results, and heuristics for deploying these to test clients who call back home.

c) Deploy a number of said rogue testing clients to thousands of test machines (vm, clusters, and others) that will call back home to the centralized machine

d) Machine learning and pattern recognition within the QAS

Aside from the fact that there’s obviously constraints involved in deploying any significant number of rogue clients even with the advent of VMware, I completely agree with Adam that modern systems are bigger, more complex, and more geographically distributed than in the past, and traditional Qa functional and automated testing results in controlled environments are being reduced to nill. However, 99.9 % of quality assurance groups (let’s just say everyone outside of Microsoft and Google), lack the resources (both processional and machine) for such an elaborate system. However, there were a few pieces of data that I pulled out from the speech that I can easily apply to Collabomatic automated tests.

a) In Adam’s test results, traversing 2 way option paths yielded nearly as many defects as did an exhaustive test of a complex system. In other words a high wide test of scenarios covering at least 2 combos of all of valid input as opposed to an exhaustive search was just about as good but required about 1/10000th the execution and design time. Because of computational power of my 3 Collabomatic test machines (I’m guessing that no one would allow me to deploy rogue software yet), 2 combos is about as much as I could expect. A system such as this could be administered from the centralized fitnesse server I have deployed.

b) A good set of configuration settings that easily map back to automated tests allows for a focus on variables that really do matter. This allows for tests to become more intelligent over time in stressing areas and configurations with a higher defect yield. In the practical sense, this allows a software engineer in test to derive automated tests in those areas, flipping those configurations around, and leave areas

c) Adam himself admits that quite possibly the most valuable data they gained from these tests was the exercise of deriving all of the legal configurations. This is very much the same 80/20 school of thought where QA involvement at the design level finds bugs when they are cheap to fix (ie. Before the code is even written). Anyway, we’ve all heard that song and dance but it was interesting to hear that even after a beast of an automation tool was designed one of the most effective testing results was gained through a glorified design review.

Anyway, I’d highly recommend viewing the video for anyone interested in emerging QA technologies. I’m excited about such a system as described, have gained some ideas for that I can apply to Collabomatic automated tests.

Email sucks for collaboration

Collaboration via email sucks.

Collaboration using email is inherently broken. While Collabomatic … the company that I have co-founded base our business primarily on this hypothesis … this was truly validated recently in a real life scenario.
—- Insert Plug Her —-
Our basic premise with Collabomatic is that email for ad-hoc collaboration is broken … and we aim to fix this. Knowledge workers experience significant pain around using email for team projects, not the least of which includes fractured documents and revision history; splintered conversations with accidental omissions of people and lack of context around individual emails; and pain with fluid team membership. Ultimately managing files and participants within these collaborations becomes unruly and using email as a solution ultimately breaks down, leading to lost information. Existing solutions don’t completely solve the pains as they force users to change the way they work … either by forcing them to a 3rd party web site like google groups … or engaging a menacing IT team to install a product like sharepoint.

Collabomatic solves the pain by seamlessly integrating into the Windows shell and into Outlook to provide team functions that do not force the users to change the way that they work by integrating into normal Outlook use and Windows use. Users can continue to work as they always have but get all the benefits of collaboration solutions.

—– End Plug Here —–
I experienced the pains of email collaboration first hand recently when working on contract. One of the outputs from my consulting agreement was a test plan. This plan (a 25 page word doc) needed to be sent off for review by the stakeholders in the project. These reviewers in turn sent the document out to external reviewers for comment. 6 comment (all sent back to me via email) and 2 days working through these comments I finally had a second revision to get reviewed. This copy was sent to the project manager for sign off. Much to my surprise, I was being asked similar questions that I had been asked before. What the #$($#. Ok, long story short, I’m sure you can see where this is going, but the fie revisions had become so unruly that the document sent out for review was actually the same document as sent out originally. While I’ll spare you all the nitty gritty details of how Collabomatic could have saved the day …. It sure does feel good when the pain that you envision you are solving is reinforced through personal experience.

QA Process in Plan driven environments

Background
It had only been two days since I left Saflink … a small-sized software company, working within an enthusiastic, agile project group in Seattle Washington … to a large, bureaucratic, development team working on a government contract in Edmonton Alberta – and the differences were already starting to wear on me.

Before I continue, I feel the need to state my intended purpose for writing this blog. While I do admit a certain amount of spite and malcontent (and I’m sure this will become obvious), I do not intend to speak ill of plan-driven environments and their effectiveness – I’d be foolish to think that these methodologies are not needed in certain scenarios such as large, governmental, highly formalized projects. However, I do believe there are many lessons to be learned and practices to be adopted from the agile startup world that could benefit these top heavy projects. Having a background … and what I like to refer to as a startup mind … I am constantly seeking ways to infiltrate the seemingly impenetrable wall of process definition, best practices, and documentation with injections of agile practices. The intended purpose of this original blog is to define these two environments. Further blog contributions will discuss ways in which I (and hopefully others) have effectively instilled agile methodologies within a process-driven environment.

The agile culture
Instead of going to great lengths describing the previous agile, startup environment I worked in, I will attempt to summarize briefly. Our small project team was proud, passionate, and had ownership and belief in the product that we were building. We were small, adaptable and not bound by procedures and best practices. Each team member was not defined by their role (as a matter of fact, we avoided using words like project managers, developers, testers) however we were expected and trusted to do whatever work was necessary to succeed. This included seeking common or unnoticed tasks and completing them. Side effects of such a culture were long hours, hard work, innovation, new technologies, team lunches, playstation sessions, over the shoulder code reviews and troubleshooting, ad-hoc collaboration, outside the box thinking, and a lot of progress towards our end goal.

The plan-driven culture
In contrast, the environment which I have been immersed in recently is a plan-driven culture. People feel comfortable and empowered only when there are clear policies and procedures (and a document) defining their roles. Their work is done when their current task can be ticked off on a spreadsheet. As a matter of fact, you are discouraged from starting up a new task until given direct instructions (in the form of a new task in a gant chart). Developers (who insist on being referred to as “analysts”) are instructed (only by project managers) on exactly what to build and exactly how long they have to do it. I often reflect on a comment made by a project manager – “Our goal is to define the software to such a degree, that we can hand off the specification document and developers can simply *puke* out code”. While this would be about enough to cause any believer in agile practices to jump off the 10th floor, I have tried (unsuccessfully however) to believe that the nature of some projects (such as ours) require such definition and precision. After all, developers … er I mean analysts here … don’t seem to mind this mentality.

Boardrooms outnumber test labs 5 to … well … there are no test labs. This would disgust Vincent Maraia who wrote in “The Build Master” that the success of a software organization can be directly measured by the ratio of computer labs to board rooms.

Finally, the sense of ownership and pride is non-existent. On my first day, my question on how to acquire remote access to the domain was answered by a look of confusion. People simply don’t have an affiliation and a sense of ownership over what is being built.

Removing barriers and applying agile methodologies
Again, I am a strong believer in agile approaches and feel this is where I’m most effective. However, I believe there are some projects that call for more of a structured approach. The challenge is finding the balance between the two … I believe the lessons I have learned from the agile startup environment can be applied to this top heavy process. It is our challenge (as startup minds in slow-down cultures) to cut through the red tape, and fly under the radar in our attempts to speed up (while not undermining) the processes currently in place … and overall, maintain our sanity.

Quality Assurance in a software startup

Friday morning 7:10AM: I stare down the sprinting lane at the University indoor track. I am anxious, not about my final sprint in a series of 10, but of an analogy that has just entered my mind. I am one of 3 founders trying to startup our company collabomatic, produce an alpha, get users, and raise some capital. I’m at the track this morning training for the sport of ultimate Frisbee, an activity that I try to cram into a day filled with 9 hours of contract work, 1.5 hours of commuting, 5 hours of work towards our alpha product, and 5 hours of sleep.

I am troubled as I’ve just been reminded by my co-trainer that a sprinter gains maximal velocity after just a few strides and spends the rest of the race slowing down. I find this incredibly analogous to a startup company. A successful startup is the one that manages to slow down the least. My quandary in this analogy is my role as QA lead in past software projects, and the heavy regulations that my teams have often injected into the development process. While the two other founders and I have fully adopted the startup/agile approach avoiding titles like the plague, I can’t help but worry that my past experience and strengths in QA will prove to be more of a douse on our sprint than fuel.

From here my mind wanders back to childhood days training for hockey; a game that I was heavily involved in. Boy, did I feel fast skating around the arena on Sunday afternoons – free of constricting shin pads, vision limiting visors, and bulky hockey pants that had hindered me throughout a week of practices. Are shin pads, shoulder pads, and hockey sticks comparable to test plans, code reviews, and regression test passes?

At the beep of my Timex, I am off sprinting … thoughts of founding a company and my role within left temporarily behind. The relief is short lived however, and as my heart rate returns to normal, so do my previous thoughts. As I think through my predicament, some compromises start to arise. While one of the keys to a successful start up may be getting to the finish line fastest, there really is more to a sprint than maximal velocity.

– As everyone knows, the closest path between two points is a straight line. Aha!! … part of my role as a founder has been to ensure that we’re not spending energy delivering features the customer may not want.
– The speed of Paul Kariya would be in vain if it was applied in a sprint to the wrong finish line. Yet another win for my role … I have been providing the customer voice and looking to ensure that their needs are being met and that we’re moving towards that voice.
– Once you’re at the finish line, we ought to be somewhat prepared for what awaits us. What good is the puck in the slot without an Easton Synergy to blast it home! Win! The automated tests that I have been developing are hitting the mark ensuring the product works as designed – and these tests haven’t been slowing development down much at all.
– While speed is essential, a certain level of precautionary equipment like a helmet is necessary along the way. Surely an unprotected website holding credit card information needs SSL even though it may require the investment of a day or 2 of sweat equity.

Alas my role has some definition … I’ll insist on for the helmet, the hockey stick and provide the hollering and encouragement as I help fuel and steer our team towards that finish line …Go Go Go ……

Ah relief … I am convinced that I have a role again … I am providing value … and I have just finished my final 100 yard sprint of the day … and now I’m going home to start the real sprint. Posted by David Q. at 11:06 AM 0 comments