SoCal, mulberry trees may create dense shade. By the way, I love mulberries, since childhood. The only reason I didn't plant a mulberry tree in my garden was dense shade that it may create.
You can try, for example, if there are at least two to three hours of sunlight below the tree, it might even work. You can always move the peonies.
