I don't mind them. I live next to fields in the countryside next to the sea and I often see Foxgloves and Buttercups flowering in the borders. Someone once said weeds are wildflowers in the wrong places.
They attract bees and butterflies and lady birds and they can't be that bad can they? I suppose I just don't want them to take over completely.
In September I plan on lifting and dividing one of the herbaceous borders which is planted right next to a field. Who but me would be arrogant enough and would try to grow perennials next to nettles, long grass and other pernicious weeds?
It's like the definition of an hedge: A statement of man's arrogance towards nature.
Can you live with wildflowers invading your lawns and borders?