That is a quotation symbol, and used to indicate that the sender did not
write the indicated words. Normally (unless the author deliberately inserts
them, as you did). If you are going to dispose of them, are you going to use
some other means of indicating that the text is a quoted reply? If so, a
simple "search and replace" operation should be sufficient (as suggested by
Diane Poremsky).