Font sizes in chart objects are affected by the size of the object relative
to the original picture size (as seen in Right click>Format Object>Size).
Only if the chart size is 100% relative to original picture size will fonts
be the correct size, otherwise they will be scaled accordingly.
The problem begins when you resize a chart object just by selecting it and
dragging the resize handles.
The solution is to reset the chart to 100%, then double click on it and
resize it in chart edit mode. That way you can make it the size you want,
including fonts, and PowerPoint will still treat it as 100% of its original
size.