You don't have to worry about accidentally weeding them out because you can never get rid of them that way. They will always leave little bulblets behind to haunt you unless this is what you really want to grow. They will crowd out and cover over other bulbs (in my case lilies). This is what jmorth means when they say they are aggressive. It won't happen overnight but the day will come...
I started out with one bulb and now they are massed along the back of the house, pretty much keeping me from using that real estate for other plants that I would like to grow.
I say, if you really like them keep them in a container and DEADHEAD as soon as bloom is finished.