It's not really a privacy concern any more than the information you enter into your own phone. The issue here is that there isn't a setting to control the behavior or to manually wipe the data. I suspect/hope we might see something like that in the future. Something else Apple might do is eliminate any association of the data with dates.

The data on mine definitely shows very rough areas I've been "around" but it's very far from exhaustive. It also displays a start date earlier than my iPhone purchase. The points on the map are a lot more localized than the areas I've actually been to with the phone.