I believe you are right that folks would dump trash through the holes, particularly cigarette butts. I only see the one board that has a knot in it. I would definitely replace that board. If you have the money to do the garden side, I would assume you would have the money to redo the street side. I would replace what is there with tongue and groove/ lap paneling. Then you could stain it whatever color you prefer. I would hesitate to do both sides unless you plan to insulate (don't know about outdoor insulation, you'd have to ask a contractor.)
I'm curious as to what is holding up your glass roof.
