deleting msgs etc. that still stays in hard drive (Microsoft)

Once I delete emails or files etc. that goes into the
recycle bin and then delete, I want to know why it still
sits in my hard drive and is never really deleted. What
is the purpose then if all is still buried and perhaps
can be accessed by some hacker? Search and Recovery at a
cost can correct any problems but Microsoft should have
that in their system when you buy it and everything
deleted should be deleted permanently. Any answers to my
questions would be greatly appreciated.
Tks.
G