Class Library Projects and References

I dont think I'm doing something right here or maybe I am misunderstanding
the concept. Hope someone can help.

I have a solution with 5 projects. Each one builds into its own class
library. They are not dependent on each other in terms of specific build
order; but related items in my application that I am aggregating for the
sake of organization. However, each library does reference the others in
code. Compilation works fine.

However, if I want to add a library by itself to another, different project,
I have to add all the other projects that are added as references. I assumed
that once added as references originally, when the dll was thrown into the
bin folder, upon release build, this would be "taken care of behind the
scenes", so you would only have to add one file at a time. Instead I'm
adding references to all the different libraries I used in the original.
Make sense?

So why do I have to do this? Is there a way to stop this from occurring?
Hopefully someone knows what I am trying to say (not sure I do after reading
it through lol).

Thanks