Open Credo

June 15, 2016 | Software Consultancy

You Are Ignoring Non-­functional Testing

It’s as simple as that – and as a consultant, it’s a problem I see all the time. Testing is always focused on functional testing. Non-functional testing, by comparison, is treated like a second class citizen. This means that functional requirements get refined, and non-functional requirements are ignored until the very end.


Matt Long

Matt Long

Senior Consultant

You Are Ignoring Non-­functional Testing

Products have floundered because of it, and lessons never get learned.

Let’s talk about what I mean by functional and non-functional testing.

Functional testing

Functional testing is what most software teams think of as testing.  For example, if I want to test a search engine, I’d think about things like:

  • Can I search for something?
  • Can I browse through many results?
  • Can I see my results as I type?
  • Can I search using my voice?

Notice that all these issues begin with “can I….” – there’s a reason for this. Functional testing only cares about whether these things are possible. It’s a checklist. Simple yes or no answers. It does not take into account how well they’re done.

That’s because anything other than bare functionality is covered by non-functional testing.

Non-functional testing

Non-functional testing is everything else. It’s just as essential to software quality – and this is the problem with ignoring it. We’re talking about performance, security, usability, accessibility. Questions like:

  • How fast does this page load?
  • Is this web page laid out intuitively?
  • What happens if 100,000 people all go on our website at once?
  • Are we likely to leak our customer’s data?

These are interesting questions with complicated answers. Answers that come with reports and complex analysis. Compare this against the straightforward passes and fails of functional tests.  

Because they’re difficult to process, it can be tempting to push them to the back of your mind. As a result, I don’t see these types of questions raised often enough.

Some of the questions cover disaster scenarios. Events that can bring a product to its knees. Some of them are things your customers will notice, but won’t tell you about. They’re the subtle things that make a good product great. In a competitive marketplace, you will stand out by caring about these things. Just think about these products, and how they stand against their competitors:

  • Google searchFast, minimal, pleasant to use, great search results
  • iPhone/iOS: Fast, attractive, intuitive
  • Mozilla Firefox/Chrome: Fast, secure, extendable
  • Uber: Convenient, simple, scalable

None of those competitive advantages would ever be fully covered by functional testing. They’re a result of great product design, enforced with both functional and non-functional testing.

Software scales incredibly well – and tends to be winner takes all. A couple of fantastic products will dominate each market. In a world like this, non-functional testing isn’t just a nice to have. It’s essential for successful software.

Why don’t more projects do it?

I’ve been a developer-in-test on many projects, and it’s often the same story. It could be performance testing, it could be penetration testing:

  • “We’ll fix it later”
  • “That’s not a priority right now”
  • “We’re under a lot of pressure to release”

These are not arguments, these are excuses. You won’t fix it later (nobody ever does). It is always a priority. You may be under pressure. But it doesn’t matter. You only get one chance to make a first impression on your customers, so make it matter. A rushed product will do more damage to your brand than any marketing can fix. Spending a bit more time and money upfront could save you an enormous amount in the long term. It’s vastly preferable to a rebrand or re-release, which may not even work.

The lack of uptake is often due to risk. Risk is a difficult, abstract concept. You don’t know for certain whether you’ll have a database breach. Maybe your application will go viral. It’s difficult to judge. Instead of grappling with these complex issues, many would prefer to bury their head in the sand.

The thing is, some non-functional testing is insuring against risk, which appears to be of dubious value. But the risk is almost certain if your product becomes successful. You will scale up. You will become a target for hackers. You will be compared against the best on the market. Are you really certain you want to bet against your product being successful?

It gets even worse, though. Most people just aren’t equipped to prioritize technically difficult stories. Imagine you are a project manager given the following stories:

  1. “We need to introduce this new dashboard widget which shows recent activity”
  2. “We need to change the application to use this more secure protocol”

One is a concrete new feature that everybody can understand. You can easily work out how important it is. The other is intellectual and abstract. It may have minimal effect, it may lead to a security disaster.

Let’s face it: not everyone is a security expert. Even if, as a manager, you were to ask your developers and you understood the story, you may have to justify it to your managers. Then there’s your shareholders, your customers, everyone else. It’s a hard sell. Maybe it’s just easier to put it off, and not worry about it.

Part of the problem is our management and communication structure through agile. We’ve become adept at communicating the importance of functional requirements. Agile isn’t being used effectively for non-functionals. It’s time we changed that.

How to change things

When non-functional testing is adopted, it’s usually considered a compliance issue. You rubberstamp the system to cover yourself if it goes wrong.

This approach fails because you only learn about your problems right before release. If we’ve learned anything from agile, it’s that testing is about failing fast. Then you can fix it fast. If you wait until the end, your problems are going to be insurmountable.

Don’t put non-functional testing as part of your release process. Have it as part of your development process. Get some basic requirements right at the start of the project. Create some small tests, and run them regularly. Just like you should with functional tests. Build on them as your product grows. Regular information about the full quality of your product will inform your developers. Security and performance are often tied to core architectural decisions. The sooner you know, the easier it will be to fix. You may even need to redesign large parts of the system – an impossible choice if faced with an impending release.

You should get to a point where the entire product team can check if the product is becoming less secure. They can point to trends in speed and scalability. It’s no longer an abstract problem – it’s a concrete one with numbers and graphs.

Stop being reactive – be active

Non-functional tests are too important to neglect. Don’t be tempted to leave it until the end. Without them, you’re flying blind. You won’t know how fast or secure or usable your system is until it’s far too late. And it might be functional, but that’s not the whole picture. You need to stop ignoring non-functional testing. Truly great software needs it.


This blog is written exclusively by the OpenCredo team. We do not accept external contributions.



Twitter LinkedIn Facebook Email