Unfortunately, the plastic/gravel thing is a cover-up that is supposed to grow nothing. It's supposed to block light and water from getting to the soil underneath to discourage whatever was growing there from coming back and make it difficult for deep-rooted vegetation to re-establish from seeds that blow in. A low (not no) maintenance, sterile sort of a thing--easily pull anything that sprouts in the gravel or spray with poison.
Depending on the plastic and how deep it is--it may not be very degraded.
Typically, what happens is, stuff accumulates and decomposes between the gravel and becomes soil, which is very favorable for seed sprouting; weed seeds sprout, and then left to do their thing, they do that too.
You can do many great things with that space for sure, but if you add some soil and rocks on top in places, you can still only grow shallow rooted stuff on top of the plastic. Maybe that's all you need?
I don't know what all you are planning to grow or how deep the plastic is. How deep is it?
Plants that would prefer to have a deeper and cooler root run would not be very happy or drought tolerant. (And, it would drive me crazy to be limited that way in choice, and running into plastic all the time. I ripped out all of the gravel covered plastic full of weeds as the first order of business when I moved into where I live now.)
Also, do consider that whatever soil you add on top will tend to get relocated by water over time to the bottom--wherever that is--down through that large gravel, to the plastic, down the slope and then out between the rocks in the wall?
I have quite a few places that I have to keep adding soil back to.
