'Approved' product is a gift horse into whose mouth you need to look with great care. You pays for what you gets...
Certified products are a common find in the security market. The basic idea is quite simple: if you are going to use a commodity, there should be some guarantee that it does what it is supposed to. As the majority of us don't have the time, expertise or budget to fully break apart all the products that we trust with our data, this testing is left to third parties.
There is a range of approval schemes, with a sliding scale of price versus level of effort. A basic evaluation might cost the vendor £25K, with a more heavy-duty option easily ten times that.
Pretty much all these schemes have one thing in common: the product vendor provides the evaluator with a statement of what the product is supposed to do, and it is tested against that statement, and only that. So, for example, you could get a ballpoint pen evaluated if all you claimed was its basic writing functionality.
As a customer, the first thing you are supposed to do is check that the product's claims fit with your expected use of it. Sadly, many people ignore this and just look for the ‘Approved' sticker.
This is a common problem. I recently had a need for some disk-wiping software, and picked a CESG-approved product (CESG is the UK's ‘national technical authority' in infosec). I noticed that part of the usage conditions were in a separate document.
To cut a long story short, it seems I was the first person ever to ask for this document, as neither the vendor nor CESG had it to hand (and in the end I was left without the document, as it's subject to a non-disclosure agreement). This is a product that is a market leader, so I'm unlikely to be its first customer.
It is no surprise then to find out that recently a range of secure USB sticks approved to FIPS 140-2, a US standard for cryptographic products, was found lacking in the security department. The crypto was fine, AES was perfectly implemented, but the key-handling interface between the crypto and the user was fundamentally broken (see http://bit.ly/bDYgbJ).
The result was that FIPS-approved products from Kingston, Verbatim and Sandisk were all easily broken with a custom ‘unlock' program. A review of the FIPS documentation for these products (http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140val-all.htm) shows that they use the same crypto chip, hence the common mode failure. But the interface between the chip and the user isn't part of the FIPS accreditation, so the code with the bug in escaped scrutiny. This is not unusual: vendors tend to opt for the simplest solution compliant with market requirements.
More worrying is the likelihood that, as all three products had identical vulnerabilities, all three vendors just took the original code without checking its security and badged it up.
This has been brushed off by the vendors as not a major issue, as it would require ‘skilled attackers'. This is daft: the only reason you pay the price premium for a FIPS-approved USB stick is precisely that: you are worried about skilled attackers.
So does this mean that accreditation schemes are useless? No, far from it. Such schemes are useful, but only if you carefully review exactly what it is that has been tested.
Non-FIPS USB sticks may still have dumb key-handling code, but they may also have broken AES implementations, random number generator bugs and so forth. A better option would be a formal evaluation that includes the whole key-management process, but that is unlikely to happen for economic reasons.
I'm reminded of one of my good friends, who has spent most of his working life being shot at. He chastised me for using the term ‘bullet-proof'. There's no such thing, he pointed out, there's only ‘bullet-resistant'. But that doesn't mean he walks around without his body armour.
Evaluation schemes reduce the risk of exploitation of security systems. But they are not perfect, and will never guarantee your safety. It may do what it says on the tin, but you need to make sure you have the right tin for the job.