A friend of mine brought a budding Plumaria inside last fall and let it bloom by a sunny window, then let it go dormant as usual. Gotta enjoy those first blooms! I would not worry about setting the plant back, especially with a four-year-old one. After all, we are growing them for the chance that they will Bloom.