Hi Karalk and
to the site here!
I don't think the sweet peas will be long gone when the lilies emerge. If you don't mind the peas growing up the lily stems I would say go for it. These two might make an interesting duo though I have never tried it. When the sweet peas finish blooming and brown out you would probably want to pull off the dead foliage.
If this doesn't sound appealing to you then you need a plan B
.