Cryptographic Exception - Bad Data (DESCryptoServiceProvider)

[Tyler]

I have read that a similar problem to the one I describe below has been
discussed on the microsoft.public.dotnet.security newsgroup, but I was
unable to see a solution to the problem with which I am struggling. I am
hoping that someone reading here will be able to assist me.

As far as I can tell, I'm doing exactly what the article at
http://www.dotnetthis.com/Articles/Crypto.htm does, but I keep getting this
exception. If someone could assist or point me to a good, relevant article,
that would be great.

Here's a snippet of code that demonstrates my problem:

-------------------------------------------------------------------------
// Input - actually obtained from output of previous encryption
byte [] atvyData = new byte [8] {0xE4, 0x83, 0x16, 0xD4, 0x89, 0x47, 0xAE,
0x68};

// Encryption key and initialization vector to use
byte [] atvyKey = new byte [8] {0xFA, 0xFA, 0xFA, 0xFA, 0xFA, 0xFA, 0xFA,
0xFA};
byte [] atvyIV = new byte [8] {0, 0, 0, 0, 0, 0, 0, 0};

// Decrypt
MemoryStream tvOutput = new MemoryStream();
DESCryptoServiceProvider tvDESAlg = new DESCryptoServiceProvider();
CryptoStream tvCryptStream = new CryptoStream(tvOutput,
tvDESAlg.CreateDecryptor(atvyKey, atvyIV), CryptoStreamMode.Write);
tvCryptStream.Write(atvyData, 0, 8);
tvCryptStream.FlushFinalBlock();

byte [] atvyResult = tvOutput.ToArray();
-------------------------------------------------------------------------

Every time I execute the 'tvCryptStream.FlushFinalBlock()' I get the 'Bad
Data' exception - why? I have tried numerous tweaks and variations to the
code above and have been unable to get rid of the exception.

Any help would be greatly appreciated!

Thanks, Tyler

 

[Jon Skeet [C# MVP]]

I have read that a similar problem to the one I describe below has been
discussed on the microsoft.public.dotnet.security newsgroup, but I was
unable to see a solution to the problem with which I am struggling. I am
hoping that someone reading here will be able to assist me.

As far as I can tell, I'm doing exactly what the article at
http://www.dotnetthis.com/Articles/Crypto.htm does, but I keep getting this
exception. If someone could assist or point me to a good, relevant article,
that would be great.

Here's a snippet of code that demonstrates my problem:

<snip>

That shows decrypting, but you need to also show how you encrypted the
data to start with. If you could produce a single short but complete
program which demonstrates that, so that we could reproduce it, that
would be great.

See http://www.pobox.com/~skeet/csharp/complete.html for what I mean.

 

[Tyler]

In this post is what I hope is a single, short, but complete program that
demonstrates my problem.

I am curious - why does it matters how I generate the data that is to be
decrypted? I had assumed that the only relevant information was in my
previous posting where I provided the data I was attempting to decrypt and
code the generated the exception. As I understand the DES algorithm, it
should not matter how one creates the 8 bytes of data to be decrypted - the
algorithm simply performs an operation on those 8 bytes of data and returns
the 8 byte result. Do you think that the DES crypto service provider works
differently from that?

Thanks for your assistance,

Tyler


-------------------------------------
using System;
using System.IO;
using System.Text;
using System.Security.Cryptography;

namespace Decrypt
{
/// <summary>
/// Summary description for Class1.
/// </summary>
class Class1
{
/// <summary>
/// The main entry point for the application.
/// </summary>
[STAThread]
static void Main(string[] args)
{
// Input
ASCIIEncoding tvEncoder = new ASCIIEncoding();
byte [] atvyInput = tvEncoder.GetBytes("THIS IS MY TEST DATA");

// Pad to ensure multiple of 8 bytes
byte [] atvyData = new byte [atvyInput.Length + (8 - (atvyInput.Length %
8))];
Array.Copy(atvyInput, 0, atvyData, 0, atvyInput.Length);

// Encryption key and initialization vector to use
byte [] atvyKey =
new byte [8] {0xFA, 0xFA, 0xFA, 0xFA, 0xFA, 0xFA, 0xFA, 0xFA};
byte [] atvyIV =
new byte [8] {0, 0, 0, 0, 0, 0, 0, 0};

// Encrypt
MemoryStream tvOutput = new MemoryStream();
DESCryptoServiceProvider tvDESAlg = new DESCryptoServiceProvider();
CryptoStream tvCryptStream =
new CryptoStream(tvOutput,
tvDESAlg.CreateEncryptor(atvyKey, atvyIV),
CryptoStreamMode.Write);
tvCryptStream.Write(atvyData, 0, atvyData.Length);
tvCryptStream.Flush();

byte [] atvyResult = tvOutput.ToArray();

// Decrypt
atvyKey = new byte [8] {0xCD, 0xCD, 0xCD, 0xCD, 0xCD, 0xCD, 0xCD, 0xCD};
tvOutput = new MemoryStream();
tvCryptStream =
new CryptoStream(tvOutput,
tvDESAlg.CreateDecryptor(atvyKey, atvyIV),
CryptoStreamMode.Write);
tvCryptStream.Write(atvyResult, atvyResult.Length - 8, 8);

// *** Exception here ***
// - if I just invoke Flush, no data is retrieved from the ToArray()
method
tvCryptStream.FlushFinalBlock();

atvyResult = tvOutput.ToArray();
}
}
}

 

[Jon Skeet [C# MVP]]

In this post is what I hope is a single, short, but complete program that
demonstrates my problem.

I am curious - why does it matters how I generate the data that is to be
decrypted? I had assumed that the only relevant information was in my
previous posting where I provided the data I was attempting to decrypt and
code the generated the exception. As I understand the DES algorithm, it
should not matter how one creates the 8 bytes of data to be decrypted - the
algorithm simply performs an operation on those 8 bytes of data and returns
the 8 byte result. Do you think that the DES crypto service provider works
differently from that?

There are a few things wrong in your code:

1) You don't need to pad the data - and the data will come back
unpadded too. I don't know exactly how that works with the way DES
itself is normally implemented - I suspect the padding and unpadding
happens inside the implementation - but you don't need to do all the
things you did. Just write the contents of atvyInput instead.

2) You're calling Flush on the encryption, but not FlushFinalBlock.

3) (Most importantly) You're creating a different key to decrypt with -
so it's hardly surprising that it failed!

Here's the modified code which does the decryption and shows the
result:

using System;
using System.IO;
using System.Text;
using System.Security.Cryptography;

namespace Decrypt
{
/// <summary>
/// Summary description for Class1.
/// </summary>
class Class1
{
/// <summary>
/// The main entry point for the application.
/// </summary>
static void Main(string[] args)
{
// Input
ASCIIEncoding tvEncoder = new ASCIIEncoding();
byte [] atvyInput = tvEncoder.GetBytes
("THIS IS MY TEST DATA");

// Encryption key and initialization vector to use
byte [] atvyKey =
{0xFA, 0xFA, 0xFA, 0xFA, 0xFA, 0xFA, 0xFA, 0xFA};
byte [] atvyIV = {0, 0, 0, 0, 0, 0, 0, 0};

// Encrypt
MemoryStream tvOutput = new MemoryStream();
DESCryptoServiceProvider tvDESAlg = new
DESCryptoServiceProvider();
CryptoStream tvCryptStream =
new CryptoStream(tvOutput,
tvDESAlg.CreateEncryptor(atvyKey, atvyIV),
CryptoStreamMode.Write);
tvCryptStream.Write(atvyInput, 0, atvyInput.Length);
tvCryptStream.FlushFinalBlock();

byte [] atvyResult = tvOutput.ToArray();

tvOutput = new MemoryStream();
tvCryptStream = new CryptoStream(tvOutput,
tvDESAlg.CreateDecryptor(atvyKey, atvyIV),
CryptoStreamMode.Write);
tvCryptStream.Write(atvyResult, 0, atvyResult.Length);
tvCryptStream.FlushFinalBlock();

atvyResult = tvOutput.ToArray();
string x = Encoding.ASCII.GetString(atvyResult);
Console.WriteLine ("'{0}'", x);

}
}
}

 

[Tyler]

Unfortunately, the changes you recommend do not address the problem I am
trying to work through. The changes you made accomplish the task of
encrypting and then decrypting a stream of data - assuming that I want to be
able to obtain the original result when decrypting.

However, that's not what I'm trying to do. To give you the whole picture,
what I'm trying to do is to mimic a hardware hash algorithm. I want to copy
the hardware algorithm in software so that I can output the intermediate
steps. The hash algorithm processes the stream in 8 byte pieces as follows
(assuming the stream provided has a length that is a multiple of 8 bytes):
1. Take the 1st 8 bytes as buffer
2. Encrypt the buffer (with an 8 byte DES key) and store the result in the
buffer
3. If all bytes have been processed, goto 6
4. XOR the buffer with the next 8 bytes and store the result in the buffer
5. Goto 2
6. Decrypt the buffer (with a different 8 byte DES key - this is
intentional) and store the result in the buffer
7. Encrypt the buffer (with the original DES key) and store the result in
the buffer
8. Buffer now contains the desired result

As you can see, I'm not interested in being able to decrypt the 8 bytes to
get my original 8 byte input. I simply want to decrypt 8 bytes and use the
result of that decryption. I didn't think it was really relevant to the
problem, but I hope it clarifies my intentions and explains why the code you
provided does not address my problem.

To address your 2nd point, I did not call FlushFinalBlock because the final
block of data is not relevant to the DES encryption result in which I am
interested. I take the following sample data from FIPS 113
(http://www.itl.nist.gov/fipspubs/fip113.htm) - it covers steps 1-5 I
identified above:
- key = 0123456789ABCDEF
- IV = 0000000000000000
- DATA = 3736353433323120 4E6F772069732074 68652074696D6520 666F7220
The FIPS result is: 21FB193693A16C28 6C463F0CB7167A6F 956EE891E889D91E
F1D30F6849312CA4.
The .NET result (when I change the key and input data in the sample I
provided previously) is: 21FB193693A16C28 6C463F0CB7167A6F 956EE891E889D91E
F1D30F6849312CA4 112633EB7B8D80AA.

Notice how the .NET result is 8 bytes longer - everything matches, except
for the final 8 bytes (112633EB7B8D80AA). I'm not sure why those bytes are
there, but they're not part of the DES result - do you know what they are
for? As far as I can tell, a DES encryption of a buffer that has a length
that is a multiple of 8 bytes should have an output that matches the size of
the input - the output should not be 8 bytes longer.

Obviously, the crypto stream is doing something that I do not want it to do
and I assume this is also the reason why I am having trouble decrypting
using the crypto stream. To do what I want to do, should I not use the
crypto streams and try to use the ICryptoTransform interface directly on the
data I want to transform?

Thanks, Tyler

Tyler <[email protected]> wrote:

1) You don't need to pad the data - and the data will come back
unpadded too. I don't know exactly how that works with the way DES
itself is normally implemented - I suspect the padding and unpadding
happens inside the implementation - but you don't need to do all the
things you did. Just write the contents of atvyInput instead.

While I also thought that should be true, when I do not pad the data, the
output I get (when using FlushFinalBlock) is: 21FB193693A16C28
6C463F0CB7167A6F 956EE891E889D91E 7045EECDCDE48FF2. Notice how the last 8
bytes do not match those of the FIPS 113 and the output generated by .NET
when I pad the data (less the last 8 bytes which I questioned above)? It
must be doing something different than just padding the data with 0's to an
8 byte boundary.

2) You're calling Flush on the encryption, but not FlushFinalBlock.

Covered above.

3) (Most importantly) You're creating a different key to decrypt with -
so it's hardly surprising that it failed!

Covered above.


Thanks, Tyler

 

[Jon Skeet [C# MVP]]

However, that's not what I'm trying to do. To give you the whole picture,
what I'm trying to do is to mimic a hardware hash algorithm. I want to copy
the hardware algorithm in software so that I can output the intermediate
steps. The hash algorithm processes the stream in 8 byte pieces as follows
(assuming the stream provided has a length that is a multiple of 8 bytes):
1. Take the 1st 8 bytes as buffer
2. Encrypt the buffer (with an 8 byte DES key) and store the result in the
buffer
3. If all bytes have been processed, goto 6
4. XOR the buffer with the next 8 bytes and store the result in the buffer
5. Goto 2
6. Decrypt the buffer (with a different 8 byte DES key - this is
intentional) and store the result in the buffer
7. Encrypt the buffer (with the original DES key) and store the result in
the buffer
8. Buffer now contains the desired result

Right. I don't know the details of DES, but the "bad data" exception
you're getting *suggests* to me that you can't just decrypt a whole
buffer with the wrong key.

To address your 2nd point, I did not call FlushFinalBlock because the final
block of data is not relevant to the DES encryption result in which I am
interested. I take the following sample data from FIPS 113
(http://www.itl.nist.gov/fipspubs/fip113.htm) - it covers steps 1-5 I
identified above:

<snip>

Right. I don't think you want a CryptoStream at all in that case - I
think you just want an ICryptoTransform which lets you transform a
block at a time. Here's a sample program which follows the above and
comes out with the "right" answer:

using System;
using System.Text;
using System.IO;
using System.Security.Cryptography;

public class Test
{
static void Main()
{
// Encryption key and initialization vector to use
byte [] key = {0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef};
byte [] iv = new byte [8];

// Pad data if necessary
byte[] inputData = Encoding.ASCII.GetBytes
("7654321 Now is the time for ");
if ((inputData.Length % 8) != 0)
{
byte[] tmp = new byte[((inputData.Length+7)/8)*8];
Array.Copy (inputData, 0, tmp, 0, inputData.Length);
inputData = tmp;
}

// Start with empty buffer
byte[] buffer = new byte[8];
for(int i=0; i < inputData.Length; i+=8)
{
// XOR result with previous contents of buffer
for (int j=0; j < 8; j++)
{
buffer[j] ^= inputData[i+j];
}
// Now encrypt the buffer
buffer = Encrypt (buffer, key, iv);
Console.WriteLine (BitConverter.ToString(buffer));
}
}

static byte[] Encrypt (byte[] buffer, byte[] key, byte[] iv)
{
DESCryptoServiceProvider provider = new
DESCryptoServiceProvider();

ICryptoTransform transform = provider.CreateEncryptor(key, iv);

byte[] ret = new byte[8];
transform.TransformBlock (buffer, 0, 8, ret, 0);

return ret;
}
}

Does that help?

 

[Tyler]

Right. I don't think you want a CryptoStream at all in that case - I
think you just want an ICryptoTransform which lets you transform a
block at a time. Here's a sample program which follows the above and
comes out with the "right" answer:

<snip>

Yes! I think that is exactly what I wanted - it was what I was going to try
next - thanks. I'd really like to know what the CryptoStream is doing with
those extra bytes that it generates (and seems to require to decrypt), but I
can make my stuff work using the ICryptoTransform.

Thanks again for your assistance,

Tyler

 

[Tyler]

Unfortunately, I may have been a little too quick to claim success. While
the sample you provided does the encryption, it doesn't do the decryption -
and that's where I have hit a brick wall ... always the decryption.

Below I have included an updated sample based on your sample that
demonstrates the problem. Can you see anything that I'm doing that is
obviously wrong? I was really hoping this ICryptoTransform was the answer.

The algorithm is:
1. Take the 1st 8 bytes as buffer
2. Encrypt the buffer (with an 8 byte DES key) and store the result in the
buffer
3. If all bytes have been processed, goto 6
4. XOR the buffer with the next 8 bytes and store the result in the buffer
5. Goto 2
6. Decrypt the buffer (with a different 8 byte DES key - this is
intentional) and store the result in the buffer
7. Encrypt the buffer (with the original DES key) and store the result in
the buffer
8. Buffer now contains the desired result

The sample is:
static void Main(string[] args)
{
// Input
ASCIIEncoding tvEncoder = new ASCIIEncoding();
byte [] atvyInput = tvEncoder.GetBytes("7654321 Now is the time for ");

// Pad to ensure multiple of 8 bytes
byte [] atvyData = new byte [atvyInput.Length + (8 - (atvyInput.Length %
8))];
Array.Copy(atvyInput, 0, atvyData, 0, atvyInput.Length);

// Encryption key and initialization vector to use
byte [] atvyKey = new byte [8] {0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD,
0xEF};
byte [] atvyIV = new byte [8] {0, 0, 0, 0, 0, 0, 0, 0};

// Encrypt (steps 1-5)
DESCryptoServiceProvider tvDESAlg = new DESCryptoServiceProvider();
ICryptoTransform tvEncryptor = tvDESAlg.CreateEncryptor(atvyKey, atvyIV);

byte [] atvyResult = new byte [atvyData.Length];
tvEncryptor.TransformBlock(atvyData, 0, atvyData.Length, atvyResult, 0);

// Value of atvyResult is now:
// 21FB193693A16C28 6C463F0CB7167A6F 956EE891E889D91E F1D30F6849312CA4
// as expected.

// Decrypt the last 8 bytes F1D30F6849312CA4 (step 6)
byte [] atvyKey2 = new byte [8] {0xCD, 0xCD, 0xCD, 0xCD, 0xCD, 0xCD,
0xCD, 0xCD};
byte [] atvyDecrResult = new byte [8];
ICryptoTransform tvDecryptor = tvDESAlg.CreateDecryptor(atvyKey2,
atvyIV);
tvDecryptor.TransformBlock(atvyResult, atvyResult.Length - 8, 8,
atvyDecrResult, 0);

// Value of atvyDecrResult is now: 00000000 00000000
// and should be: 1355250A 3EED48BD

// Encrypt the result 0000000000000000 (step 7)
atvyResult = new byte [8];
tvEncryptor = tvDESAlg.CreateEncryptor(atvyKey, atvyIV);
tvEncryptor.TransformBlock(atvyDecrResult, 0, 8, atvyResult, 0);

// Value of atvyResult is now: 18EC386C 7BC01E5E
// and should be: A502759B E569D70F (obviously because the decrypt didn't
work)
return;
}

Thanks, Tyler

 

[Jon Skeet [C# MVP]]

Unfortunately, I may have been a little too quick to claim success. While
the sample you provided does the encryption, it doesn't do the decryption -
and that's where I have hit a brick wall ... always the decryption.

Below I have included an updated sample based on your sample that
demonstrates the problem. Can you see anything that I'm doing that is
obviously wrong? I was really hoping this ICryptoTransform was the answer.

Comments inline:

The sample is:
static void Main(string[] args)
{
// Input
ASCIIEncoding tvEncoder = new ASCIIEncoding();

There's no need to create a new encoding - just use Encoding.ASCII.

byte [] atvyInput = tvEncoder.GetBytes("7654321 Now is the time for ");

// Pad to ensure multiple of 8 bytes
byte [] atvyData = new byte [atvyInput.Length + (8 - (atvyInput.Length %
8))];

That's fine *this* time, but not if the input is already exactly a
multiple of 8 bytes, as you're then padding unnecessarily.

Array.Copy(atvyInput, 0, atvyData, 0, atvyInput.Length);

// Encryption key and initialization vector to use
byte [] atvyKey = new byte [8] {0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD,
0xEF};
byte [] atvyIV = new byte [8] {0, 0, 0, 0, 0, 0, 0, 0};

You don't need the new byte[8] here, although I suppose it makes sure
you've got 8 bytes

// Encrypt (steps 1-5)
DESCryptoServiceProvider tvDESAlg = new DESCryptoServiceProvider();
ICryptoTransform tvEncryptor = tvDESAlg.CreateEncryptor(atvyKey, atvyIV);

byte [] atvyResult = new byte [atvyData.Length];
tvEncryptor.TransformBlock(atvyData, 0, atvyData.Length, atvyResult, 0);
// Value of atvyResult is now:
// 21FB193693A16C28 6C463F0CB7167A6F 956EE891E889D91E F1D30F6849312CA4
// as expected.

If you say so - I wasn't aware that a block of more than 8 bytes would
happen to have the effect you wanted, but if you say it does, I have no
reason to doubt you.

// Decrypt the last 8 bytes F1D30F6849312CA4 (step 6)
byte [] atvyKey2 = new byte [8] {0xCD, 0xCD, 0xCD, 0xCD, 0xCD, 0xCD,
0xCD, 0xCD};
byte [] atvyDecrResult = new byte [8];
ICryptoTransform tvDecryptor = tvDESAlg.CreateDecryptor(atvyKey2,
atvyIV);
tvDecryptor.TransformBlock(atvyResult, atvyResult.Length - 8, 8,
atvyDecrResult, 0);

// Value of atvyDecrResult is now: 00000000 00000000
// and should be: 1355250A 3EED48BD

Is there anything in DES which specifies that you should definitely be
able to "decrypt" a block with a different key? It still sounds a very
dodgy thing to do to me.

However, there's something odd going on, as decrypting the block even
with the *original* key gives all zeroes.

I've no idea why, I'm afraid

 

[Jon Skeet [C# MVP]]

// Value of atvyDecrResult is now: 00000000 00000000
// and should be: 1355250A 3EED48BD

Got it!

For some reason, you need to call TransformBlock *twice* to get the
data. The data used in the second call is irrelevant (so long as
there's enough to flush the first block.) I believe basically the call
to TransformBlock is likely to effectively transform the previous
block, or something like that.

Note that the first call to TransformBlock returns 0 (no bytes written)
but the second call returns 8 (8 bytes written). That's how it works
properly in a stream - and could explain why there's the extra block
when using streams to encrypt and decrypt.

 

[Tyler]

Thanks - I appreciate your assistance. I included some comments below.

// Encrypt (steps 1-5)
DESCryptoServiceProvider tvDESAlg = new DESCryptoServiceProvider();
ICryptoTransform tvEncryptor = tvDESAlg.CreateEncryptor(atvyKey, atvyIV);

byte [] atvyResult = new byte [atvyData.Length];
tvEncryptor.TransformBlock(atvyData, 0, atvyData.Length, atvyResult, 0);
// Value of atvyResult is now:
// 21FB193693A16C28 6C463F0CB7167A6F 956EE891E889D91E F1D30F6849312CA4
// as expected.

If you say so - I wasn't aware that a block of more than 8 bytes would
happen to have the effect you wanted, but if you say it does, I have no
reason to doubt you.

It does because the default cipher mode of the DES provider is CBC (which
does the XOR/encrypt stuff for each 8 byte block). If thet cipher mode was
set to something like ECB, I couldn't do that.

// Decrypt the last 8 bytes F1D30F6849312CA4 (step 6)
byte [] atvyKey2 = new byte [8] {0xCD, 0xCD, 0xCD, 0xCD, 0xCD, 0xCD,
0xCD, 0xCD};
byte [] atvyDecrResult = new byte [8];
ICryptoTransform tvDecryptor = tvDESAlg.CreateDecryptor(atvyKey2,
atvyIV);
tvDecryptor.TransformBlock(atvyResult, atvyResult.Length - 8, 8,
atvyDecrResult, 0);

// Value of atvyDecrResult is now: 00000000 00000000
// and should be: 1355250A 3EED48BD

Is there anything in DES which specifies that you should definitely be
able to "decrypt" a block with a different key? It still sounds a very
dodgy thing to do to me.

However, there's something odd going on, as decrypting the block even
with the *original* key gives all zeroes.

Yes, it is quite possible - DES is simply an operation that is perform on an
input block. I used the D3DES implementation written by Richard Outerbridge
(the source is publicly available and used in a variety of applications) to
get my expected values and that's why I knew my .NET code wasn't working
quite right.

I've no idea why, I'm afraid

I appreciate the help that you've provided. Apparently neither of us
understand what the DES crypto service provider is doing with the decryptor.
I guess I'll have to do try doing some interop to the D3DES instead of using
the .NET classes.

Thanks, Tyler

 

[Tyler]

Awesome! Thank you so much - I never would have found that. Very bizarre,
but it works!

Tyler