I got this little Yachiyo Tsubaki graft from Alex and planted it some three weeks ago. When I planted it, I noticed that the scion is connected with the rootstock only on one side. Now I am thinking that I should dig it up and firmly tape everything together. What are your thoughts? Please advise.

I think I would, you want the graft to be in contact and as secure as possible.

You can tighten it with some rubber band, the one with the stretch that could not restrict the root growth. I started to use cuts from kitchen rubber gloves this year once I ran out of rubber bands. The good thing is you can cut it wide or narrow as needed.