CryptoStream // Memory Problems

[Caroline]

Hello,
I need to encrypt about 5 million records in a flat file. For a specific
reason, each record (32 bytes) must be encrypted by itself, (not all records
can be encrypted at once). Also I must use the Rinjndael encryption.

So I am running into memory problems, because I am creating a new
CryptoStream every time. I couldn't figure out how to encrypt using one
single CryptoStream, please help!

Could anyone help? This code is really simple, but it's hogging a lot of
memory. Please give me some advice.
using System.Data.SqlClient;

using System.Security.Cryptography;

using System.IO;


int i = 0;
try
{

SqlConnection conn;
SqlCommand cmd;
SqlDataReader r;

ICryptoTransform ict;
StreamWriter swData;
FileStream fs, fsData;

System.Security.Cryptography.RijndaelManaged cryptObj = new
RijndaelManaged();
byte[] KEY_128 = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6};
byte[] IV_128 = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6};

MemoryStream ms = null;
CryptoStream cs = null;
StreamWriter sw = null;

fs = new FileStream("..\\..\\isbncrypt.txt", FileMode.Create,
FileAccess.ReadWrite, FileShare.ReadWrite);

ict = cryptObj.CreateEncryptor(KEY_128, IV_128);

string s = "";
int blockSizeEnc = 32;
byte[] buf = new byte[blockSizeEnc];

conn = new SqlConnection("server=homeserver;Integrated
Security=SSPI;database=NorthWind;");
conn.Open();

cmd = new SqlCommand("selectrecords_ssp", conn );
cmd.CommandType = CommandType.StoredProcedure;

r = cmd.ExecuteReader();

while ((r.Read()) && i < 5000000) // 5 million records
{
i++;
ms = new MemoryStream();
cs = new CryptoStream(ms, ict, CryptoStreamMode.Write);
sw = new StreamWriter(cs);

s = r.GetString(1).Substring(0,32);

sw.Write(s);
sw.Flush();
cs.FlushFinalBlock();

ms.Position = 0;
buf = new byte[blockSizeEnc];
ms.Read(buf, 0, blockSizeEnc);
fs.Write(buf, 0, blockSizeEnc);

if(conn.State != ConnectionState.Open)
conn.Open();

}
fs.Write(buf, 0, blockSizeEnc);


ms = null;
fs.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message + "*** " + i);
}

 

[Jon Skeet [C# MVP]]

I need to encrypt about 5 million records in a flat file. For a specific
reason, each record (32 bytes) must be encrypted by itself, (not all records
can be encrypted at once). Also I must use the Rinjndael encryption.

So I am running into memory problems, because I am creating a new
CryptoStream every time. I couldn't figure out how to encrypt using one
single CryptoStream, please help!

If every item needs to be encrypted individually,

Could anyone help? This code is really simple, but it's hogging a lot of
memory. Please give me some advice.

Firstly, you should use "using" statements to dispose of your streams
when you're done with them. That may well be part of the problem.

Your MemoryStream handling could be more efficient though. If you
create a single byte array of the right size, and pass the same one
into the constructor each time, making the stream a fixed length so
that exceptions will get thrown if you unexpectedly try to use too much
space, you can write the contents of the buffer directly to the
FileStream, using the Length property of the MemoryStream to find out
how much to write.

 

[Caroline]

Jon,
You are awesome!
I followed your advice close the streams and set a fixed size to the buffer,
bam! that was it!

Thanks a lot!

 

[Helge Jensen]

byte[] IV_128 = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6};

Usually the IV is used to provide nondeterministic cryptography, where
the same payload is encrypted differently each time. That property may
or may not be valuable to you, but if you need it you should choose
random IV's and store those with the encrypted data.

The usual story of importance of nondeterministic encryption is:

A general sends the message encrypt("Attack at dawn") to his troops. The
enemy intercepts the message, but cannot decrypt it -- so they don't
know when the attack will come and the general is victorious.

Two weeks later the general again sends encrypt("Attack at dawn") to his
troops in an attack on another town, the enemy intercepts the message
and cannot decrypt it -- but the sneaky enemy recognises the message
from two weeks ago and is ready to defend when the attack comes,
incurring terrible losses.