I've been calling it Mumbai the last week or so but had previously called it Bombay mostly on the grounds that I think it's silly to keep changing the names of well-established places.

That pretty much nails it. Or, looking at it another way, we don't call other cities by their local names.

The argument is a little different, DCL. When the city was built it was called “Saint Petersburg” and it remained that way for a couple of hundred years. The Bolsheviks changed it to Leningrad. After the USSR collapsed the Russians changed it back.

The city in India had never been called “Mumbai” except in the imaginations of a few nationalists. Bombay comes from the Portuguese for a “good port”.

You've actually come across the best solution, which is to use "St. Petersburg" and "Bombay" so that Sympathy For The Devil still makes sense.

The city in India had never been called “Mumbai” except in the imaginations of a few nationalists. Bombay comes from the Portuguese for a “good port”.

I see no particular reason for English speakers to try to call foreign places by the names the people who live there use when we've got accepted English names for them. Otherwise we'd be saying “Sankt Pehtehrsboorg” (to use the example above), “Paree”, and “Roma”.