When to write the test specification

[socaroka]

I'm currently engaged in a hot debate regarding when is the most appropriate time in a software development project to draft the proposed test specification. I say it should be 'drafted' alongside the requirements specification - ie. at the same time as the technical spec., functional spec., etc. etc.
Working on a new user interface for a car accessories firm.

[techgnome]

It starts during the design phase. As soon as the Use Cases are built, the test specs can be started. They should be developed concurrently and independently with the design. In addition, they should be written implementation agnostic. Meaning the steps laid out shouldn't care if the entry is a text box or a date time picker. The test spec should read "Enter such and such date into the XYZ Date field."

I also believe that while the design and test specs should be created independantly, the two should work together to make sure that changes by one is reflected by the other (what happens if the designer decided to lock down the XYZ Date Field and simply populate it with the current date? The above test fails.)

Once those are done and handed off to the developer (I've found having the test cases helps to proactively test the development, it helps to see what will work and what won't and why) then the detailed test scripts/documents can be created, and should be kept from the developer - that's testing's Ace in the hole to determine if itwill pass or not.

-tg

[alex_read]

It starts during the design phase.

Completely agree - the draft should be started as early as possible as the project is being planned and before any code is written. Finalisation of the test specification (sometimes a master test plan) has to happen after the project plan is accepted (as then you'll know the accepted dates and budgeted cost set aside for test team members & possibly the different testing types requested at this stage). Again the finalisation should happen as early as possible, likely before any code has started.

As soon as the Use Cases are built, the test specs can be started. They should be developed concurrently and independently with the design.

Agree again but sadly in many projects so far I've not seen use cases (due to different companies developing & testing or due to budgeting to create these docs). They aren't always available in my experience. :c(

In addition, they should be written implementation agnostic. Meaning the steps laid out shouldn't care if the entry is a text box or a date time picker. The test spec should read "Enter such and such date into the XYZ Date field."

Now we go a little off topic - this is more the test case design area which comes after the specification is produced. When I first started testing, I'd put everything specifically ("click the red button with the text 'don't click me' in the top right of the page" etc.). Looking back now, I'd agree with your comments on being a bit more granular! :cP

I also believe that while the design and test specs should be created independantly, the two should work together to make sure that changes by one is reflected by the other (what happens if the designer decided to lock down the XYZ Date Field and simply populate it with the current date? The above test fails.)

Again, I'd completely agree to the first part here. The 2nd part, well seems more related to the test cases/script creation phase again to me. A few comments I'd add here, would be:

• The 3 should work together - the customer/end acceptance team ought to be involved too.
• The developer team's functional specification document (if produced) should list this & should always be get acceptance/sign off before the code/design starts incase that's not clear to anyone else reading this thread.
• Hmm... in your given example I'd say the testing team's main priorities should be (i) the field & screen matches the end needed requirements and (ii) that the submission works as expected. But yes largely the teams should communicate as much as possible and the test cases should be written in a way that matches both the requirements and the design being released.

[techgnome]

Wow... this is a call back to the waybac machine here...

The 3 should work together - the customer/end acceptance team ought to be involved too.

It depends ... what is this test case that's being developed for? Or rather WHO is the test case for?
Where I work, there are basically four testing rounds that everything is supposed to go through:
The first is the unit testing - this can be automated (sadly it usually isn't and is often the first thing developers throw out the window)
The second is what we call RATS - Requirement Acceptance Testing - they are test cases added by QA based on the requirements, and to be executed by the developer before promoting the work to QA.
The third round is the usual QA testing... this is where form meets function... sure, the user can click the button and the report appears, so it's functionally correct, but the button is transparent color with white text, making it white on white and unable to see... so a defect gets filed.
The fourth test is the UAT - User Acceptance Testing - this is where it finally makes it into the hands of the users that will be using the system.

Each round of testing has it's own needs and document requirements. The first - unit testing - we play fast and loose with that and half the time I'm happy if my devs simply check in something that compiles. We use continuous integration, so as soon as something is checked into TFS, the build runs and deployed to the staging environment. We get an email every time it happens and whether is succeeded or failed.

The second one - RATS - this is where QA takes the requirements document and creates a RAT task indicating 1) the requirement and 2) how to test that requirement. Because our requirements are written to be testable-standalone items, RATs are actually quite easy. They are NOT a test case though... but multiple RATs could roll up into a full test case. An example of a RAT: The requirement is that the Commit date on a Revenue transaction must allow the user to enter any future date and be in the ISO yyyy-mm-dd format. From that, we get three RATS: 1) User must not be able to select a date in the past. 2) User must be able to select any date in the future (and yeah, we'll check out to 3030) 3) The format of the displayed date is to be in yyyy-mm-dd format. So now when the developer looks at the task in our system, they will see multiple items... the User Story - which is the highlevel big picture requirement... then derrived from that are all the individual requirements... on each requirement are the RATs... when they are done with the development of that requirement, they run the RATs and sign off on each one. When they sign off on all RATs for a requirement, that item can then be promoted to QA. Part of the intent of the RATs is to catch the obvious stuff so we don't waste QA's time. It is a checkpoint for the developer, and it's something that can't be ignored.

Then there's QA ... QA has their own test cases that are developed based on the requirements and the user stories. These document the features of the development and the steps QA will undertake to test them. These test case are for QA and QA only ... the user/client has no say in what that testing does or is. (this is why I said it depends on the audience in my initial comment).

The only time the client/user has any say in the test spec is for their own UAT ... because that's THEIR test that THEY will perform. It is how they will test the functionality and determine if it meets their needs or not. And the client does this. They typically do not get input from us on how to conduct this. It's the same reason I don't (normally) let clients in on the technical design aspect of the development... I don't know their job any better than they know mine. And it would be foolish for either of us to presume we know better.

In our case, while we have Use Cases built into our requrements document (which I'm not sure I 100% agree with) ... they simply outline the general expected/proposed functionality and should be used as guideline for creating the QA Test Cases and the client's UAT test scripts ... but when it comes to those test cases and scripts, each one has it's own audience and involved people.

Just my two cents.

-tg

[alex_read]

I appreciate both the correction & getting to hear/learn different styles, so thanks for that one! I especially love the term RATS too just as a side note - one of the best things about being in IT is the names people give projects, apps etc.

Yep that maybe ought to be "could" not "should" above there then. I've also worked in a software house which produced a spec, went off & coded then handed over just the end product with little contact between too. In my own experience (& with a lot of hindsight), that was pretty disasterous & led to tons of bugs. I've read several books & articles since that time (most notably the head first software development book which I thought had some great ideas) & really wish we'd have done things differently, involving the cutomer tons more. Maybe that's not applicable to all places though & I hadn't considered every project type or development team there.

Places where the "could" part will fit in better for anyone else reading this post... (i) agile projects (ii) onsite projects where you're writing code or new features ad hoc & in a time and materials basis or (iii) you're working in a multivendor project largely or partly onsite with different vendors & teams.