M
Mark Watts
I have some questions how about solutions/projects should be structured
for unit testing, what others consider as best practices and your
general ideas and inputs. I am trying to get a structure in place for
a product currently in development and welcome input from others with
experience in this area.
BTW: I am aware of the upcomming feature in Whidbey of Friendship
Assemblies that will hopefully help the situation, however, I have to
come up with a workable solution now for VS.NET 2003 and v1.1 of the
framework.
Here are the options that I am aware of.
1. Place all tests for an assembly in a second assembly that depends
on the production build of the assembly being tested. In reviewing
publicly available code bases this seems to be the most common.
* Problem: You are forced to do black-box testing (only test those
classes/methods that are public). This either limits your tests to the
defined public interface of the assembly (those classes and methods
that are public) or in order to provide unit tests for all classes,
make everything public. Both are undesirable. Black box testing means
it is much easier to miss test cases and making everything public is
not really a good programming practice, esp. when you are producing
libraries and/or components that are designed to be used by other
programmers.
2. Two projects, one for production, one for test where the test
project in VS.NET includes the files with the unit tests.
* Problem: This is a pain to maintain as new production files must be
added to both projects.
* Problem: Compile errors are reported twice, once in the production
files and again in the same production files in the test projects.
* Problem: If you have a file open in either the production or test
project, trying to open the same file in the other project results in a
VS.NET error that the file is already open.
* Problem: If your product is dynamic in nature where assemblies are
loaded at runtime using reflection then you have situations where types
can exist in two assemblies (production and test) and both be loaded
resulting in much hair pulling trying to figure out why things don't
work.
* Problem: In the test version of your assembly if you start defining
security attributes on the assembly, NUnit AND csUnit will either fail
to load the assembly or fail to find any tests. May be a defect in
NUnit/csUnit but it is repeatable in both frameworks. This means you
can't do testing involving security while using this configuration.
This limitation is a real issue when developing Rich Clients.
3. Have a single project that includes both production and test code
and ships the unit tests in production.
* Problem: Security issues mentioned in item 2 above.
* Problem: You end up shipping unit tests. Do NOT like this.
* Problem: You are forced to also ship your unit test framework. This
may be illegal with commercial unit testing frameworks and potentially
with NUnit although I am not sure. Because NUnit is licensed under the
GPL not LGPL or GPL with an exception for linking with your code, I
believe that this option would cause the viral nature of the GPL to
extend to my code which in a corp. setting will not work.
* NOTE: This might be a workable solution IF VS.NET would allow
Directories and Assembly Dependencies to be conditional. i.e. Release
target would not include the TEST directory or depend on
nunit.framework.dll but the Debug target would. However, this is not
the case and this does not address the problem with loading assemblies
with security restrictions in NUnit and csUnit.
* Another NOTE: This option will still work if production builds are
done with a command line build tool like NAnt. It's use of filesets
will allow the exclusion of the Test files from the build resulting in
a GPL and Test Framework free production build.
4. Use conditional compilation and place your unit tests for a class
in the same file as the class or perhaps as additional methods of the
class.
* Problem: Security issues mentioned in item 2 above.
* Problem: The use of conditional defines tends to make the code messy
and more difficult to read/maintain.
* Problem: Dependencies start getting hairy. Often testing a
unit/class will require other classes or mock objects and establishing
dependencies on those other classes/mock objects makes your assembly
depenencies and your 'using' or 'imports' clauses complex.
* Problem: Because VS.NET does not allow conditional dependencies for
an assembly, one missed IFDEF around something establishes a production
build dependency on your unit testing framework.
TIA
-Mark
for unit testing, what others consider as best practices and your
general ideas and inputs. I am trying to get a structure in place for
a product currently in development and welcome input from others with
experience in this area.
BTW: I am aware of the upcomming feature in Whidbey of Friendship
Assemblies that will hopefully help the situation, however, I have to
come up with a workable solution now for VS.NET 2003 and v1.1 of the
framework.
Here are the options that I am aware of.
1. Place all tests for an assembly in a second assembly that depends
on the production build of the assembly being tested. In reviewing
publicly available code bases this seems to be the most common.
* Problem: You are forced to do black-box testing (only test those
classes/methods that are public). This either limits your tests to the
defined public interface of the assembly (those classes and methods
that are public) or in order to provide unit tests for all classes,
make everything public. Both are undesirable. Black box testing means
it is much easier to miss test cases and making everything public is
not really a good programming practice, esp. when you are producing
libraries and/or components that are designed to be used by other
programmers.
2. Two projects, one for production, one for test where the test
project in VS.NET includes the files with the unit tests.
* Problem: This is a pain to maintain as new production files must be
added to both projects.
* Problem: Compile errors are reported twice, once in the production
files and again in the same production files in the test projects.
* Problem: If you have a file open in either the production or test
project, trying to open the same file in the other project results in a
VS.NET error that the file is already open.
* Problem: If your product is dynamic in nature where assemblies are
loaded at runtime using reflection then you have situations where types
can exist in two assemblies (production and test) and both be loaded
resulting in much hair pulling trying to figure out why things don't
work.
* Problem: In the test version of your assembly if you start defining
security attributes on the assembly, NUnit AND csUnit will either fail
to load the assembly or fail to find any tests. May be a defect in
NUnit/csUnit but it is repeatable in both frameworks. This means you
can't do testing involving security while using this configuration.
This limitation is a real issue when developing Rich Clients.
3. Have a single project that includes both production and test code
and ships the unit tests in production.
* Problem: Security issues mentioned in item 2 above.
* Problem: You end up shipping unit tests. Do NOT like this.
* Problem: You are forced to also ship your unit test framework. This
may be illegal with commercial unit testing frameworks and potentially
with NUnit although I am not sure. Because NUnit is licensed under the
GPL not LGPL or GPL with an exception for linking with your code, I
believe that this option would cause the viral nature of the GPL to
extend to my code which in a corp. setting will not work.
* NOTE: This might be a workable solution IF VS.NET would allow
Directories and Assembly Dependencies to be conditional. i.e. Release
target would not include the TEST directory or depend on
nunit.framework.dll but the Debug target would. However, this is not
the case and this does not address the problem with loading assemblies
with security restrictions in NUnit and csUnit.
* Another NOTE: This option will still work if production builds are
done with a command line build tool like NAnt. It's use of filesets
will allow the exclusion of the Test files from the build resulting in
a GPL and Test Framework free production build.
4. Use conditional compilation and place your unit tests for a class
in the same file as the class or perhaps as additional methods of the
class.
* Problem: Security issues mentioned in item 2 above.
* Problem: The use of conditional defines tends to make the code messy
and more difficult to read/maintain.
* Problem: Dependencies start getting hairy. Often testing a
unit/class will require other classes or mock objects and establishing
dependencies on those other classes/mock objects makes your assembly
depenencies and your 'using' or 'imports' clauses complex.
* Problem: Because VS.NET does not allow conditional dependencies for
an assembly, one missed IFDEF around something establishes a production
build dependency on your unit testing framework.
TIA
-Mark