From xkcd this morning:
In the security industry, we call this "rubber-hose cryptanalysis" -- a euphemism coined by Marcus Ranum for getting the target to give you access to the encrypted data via coercion. The name implies physical torture, but psychological coercion (threatening physical harm, litigation, etc) is pretty much the same thing. I personally consider this a variety of Social Engineering since it relies on manipulating your mark rather than using technology to directly attack the assets.
When attacking many modern cryptosystems, a technological attack is often going to take a lot longer than simply bludgeoning it out of someone. Some technological attacks that don't directly involve breaking the crypto through brute force can sometimes get decrypted data :
- Dumping the contents of the target's RAM and Swap file (can contain the crypto key, unencrypted data, evidence of encrypted data or metadata about the encrypted files)
- Operating system history "recently used" (can store history data and reveal structure of the encrypted volume)
Simply put, deniable encryption most often refers to encrypted data which can resolve to both the genuine data and decoy data, depending on the key used to decrypt the data. The decoy data should appear to be "secret" in nature, and it's best if the decoy data appears to be the same kind of data that you're really trying to protect. If you encrypted a bunch of proprietary documentation, your attacker may know what they're looking for. You'd be best to make the decoy data look like proprietary documentation while remaining innocuous, perhaps loaded with misleading facts. Using a video of Rick Astley's Never Gonna Give You Up might tip the attacker off that they need to keep looking (and torturing you) for the real goods.
While not the only tool available, one commodity piece of free, cross-platform software that handles this task quite nicely is TrueCrypt. It handles full-disk encryption, deniable encryption (which is called a "hidden volume" in TrueCrypt), and can even boot an operating system from a hidden volume. That's right, TrueCrypt can boot an entirely different operating environment based on which pre-boot passphrase you enter.
I'll save the merits and woes of full-disk encryption for another day.