Confessions Of A Beta Tester

Blogger Lagrange Audio gives us tips on testing      05/12/13

4) The testing methodology and chronology - In the IT community it is widely recognised that testing is a professionally accredited activity in its own right. Professional testers take their responsibilities very seriously and support their work with professional software tools and processes that can leverage every aspect of the product. Testing has many distinct chronological phases such as unit testing, system integration testing, user acceptance testing, stress and performance testing, beta testing, working with early release pilot groups and so on. It's important to understand where you fit in the overall scheme of things. This is particularly important depending on the development methodology approach the developer has taken and recognising that in reality both development and testing phases overlap by a significant degree. Certainly in IT circles it is recognised that the majority of development actually occurs after the first release goes out.

Leading from that it's also important to understand where the product is at when you get your first look at it. Is it an early alpha or a late beta? Determining the products maturity sets a starting point for you. In terms of methodology it's worth noting the definition, which is "a set of guidelines and principles that a practitioner of a particular discipline operates by...". For late beta's I would suggest from my own experience that your own workflow is sufficient to adopt as a methodology as hopefully most of the deep issues have been resolved. For earlier releases I would adopt a lower level set of tasks before swinging your particular workflow into play. This can be a discipline issue for some people because it's tempting to be swayed by the products 'wow' factor. Unfortunately there is a whole bunch of things that have to be checked first before a single sound is created. These might include the install procedure for example, is everything being put where it's supposed to be. Other 'boring' areas to highlight are audio interfaces and drivers, for a software instrument can it see your driver(s) and switch through all of its capabilities, the same applies for Midi interfaces and so on. Patch management and UI is another area and on it goes and of course every test you perform should be repeated on every platform you have available that the vendor intends to deliver it on, obviously there are some practical constraints here. I have two Windows machines for example that can be dual booted into a number of OS's and configurations, which is necessary while we are still in that 32bit to 64bit transition phase and will be for some time yet.

This next bit is especially important if for no other reason that your approach here sets you apart from your peers. In my world any reported issue to a vendor will not be accepted without a reasonable test case for the vendor to understand and replicate the problem so make sure you adopt a reasonably documented approach considering the key items below:

a) What is the platform configuration the product was running on including any host containers, typically DAW's?

b) What were the steps leading up to the issue? When clicking through the interface for example do not rush this bit because if the issue is not immediately repeatable then you will have a frustrating time attempting to recall your lead-in steps.

c) Record (screen shhots preferred) every error message displayed and its source (if known) i.e the product, the OS, the drivers noting specfic libraries, file paths etc

d) Was the error recoverable or non-recoverable? Non-recoverable essentially means that at the very least the software in question has to be forced shutdown and likely this includes related software (DAWs etc) and OS's (re-boots).

e) Is the error repeatable or random? Actually there is no such thing as a random event in software development, only a much reduced probability that you will replicate a combination of factors that you are yet to understand.

Put simply the last thing the developer wants to hear is "it just crashed...", that helps no-one. The above provides the developer with the essential test case, which allows them to configure an environment where they can replicate the issue. This is particularly important given the likely distance between the parties involved. I live in Australia and all my beta-testing to date has been products developed in Europe and the US. Regression testing is also another discipline that's worth noting as well. The history of software development is resplendent with the scenario where an existing function has been broken as the result of fixing an unrelated issue. That means existing functions have to be re-tested, a painful but necessary exercise unfortunately. A lot of people miss this one!!

One final thing to mention is that it is completely unrealistic to expect perfect software or for that matter every issue you raise will be addressed. Software companies walk the commercial tightrope to determine the release date. It is a delicate balancing act between the desire to make the best product possible and the need to generate revenue. Ultimately it's the companies call, some pick their moment well, others less so.

5) Understanding software development - or more importantly recognising the challenges that software developers face. Part of the philosophical debate is whether software development today is easier than in the past. My IT vendors argue that it is easier on the basis that frameworks make it so and I have to resist the temptation of reaching across the meeting room table and smacking them around the face with a large wet fish when they tell me these things. The truth is that software development is as hard as it ever was for no other reason that the poor developer has to bolt their product on top of a layer cake of hardware and software infrastructure with all of its attendant libraries, interfaces and API's. Further the developer then has to rely on the fact that all of these things built by people they will never meet all are working well with each other in the first instance before they have a hope of getting their product to work. And finally hoping beyond hope that nothing then changes at the very least at a frequency that the developer then spends all of their time porting and upgrading rather than doing what they would rather be doing which is developing new products and getting them to market. As a beta tester a healthy understanding of these issues is paramount. Hope is a wonderful thing.

6) Confidentiality - I have to confess this is a bit of an annoyance to me. In my day job confidentially is essential, if in doubt it is always implied. What I have noticed however in the music tech space and it's various online forums is a consistent number of people professing to be beta-testers for company X or currently testing product Y. Unless these people have approval to advertise their activities I would suggest this is a wholly unprofessional approach to take and I wouldn't encourage it. As I highlighted above as a beta tester you have an effective stake in the companies objectives so for safety's sake you must assume that everything is 'Commercial-In-Confidence'. There are exceptions to this obviously but really only in the area where there is a clearly stated relationship between the company and the people involved, stated by the company. There is also the temptation to shout about the product on release day and your involvement with it. It's probably OK for example to say that a product has been released but I would suggest you don't put your name to it unless you have cleared that with the vendor first. A lot of this is common sense after all and can be guided by the dynamic of the relationship you have with the people concerned.

7) Your place in the world - this relates to your stature in the great endeavour. Software vendors will most likely have immediate go-to people for first looks. Bigger companies for example may have in-house people who they approach first, they could be their own certified trainers for example who need that heads-up to start preparing their own training materials. Vendors often have, informally or otherwise, an artists roster who serve not only as professional musicians providing valuable insight but also may lead into promotional activities. It's very likely, especially starting out, that you do not fall into any other these groups, you will be on a lower rung. So it's important to recognise that you are still a valuable participant. albeit a silent one.

8) The pay-off - Put simply, it's certainly not a career that pays but that's not the point, the pay-off comes from getting a look at very interesting products before other people do and in some small way stamping a little bit of yourself on their development, which is very satisfying on its own. Please note however that you should not under any circumstances expect anymore than this. Never expect or ask for freebies. It generally doesn't work that way.

Jason Durbin (aka Lagrange Audio) has been a synth and music tech enthusiast for 30 years since getting his hands on his first synth in 1983 at the tender age of 16. He hasn't earned a single Aussie dollar from music but the journey has been nothing short of incredible and he has met and interacted with some amazing people along the way. Jason is a true enthusiast doing it for nothing more than the pure love of it.




More From: LAGRANGE AUDIO
Even more news...

1 Comments...  Post a comment    original story
passbrains    Said...

Good thoughts on Beta testing

08-Dec-13 11:35 PM


Post a comment 
 


More Videos

Sonic LAB: M-Audio Trigger Finger Pro 

Beats, control, sounds. Value?


Video: New AIRA Firmware And SH-101 Plug-out 

Many fixes and tweaks to the range, official release of SH-101 plug out


SonicTouch: EP 33 - Max Out USB - WIn CME Xkeys 

We hookup 7 USB devices to a single iPad and play


Sonic LAB Review: Happy Nerding FM AID and Super Sawtor Modules 

We take a look at two Eurorack modules