As long as everyone knows what you're talking about there shouldn't be a problem.
I think it's more of a problem for birders because there's an official governing body and there's a competitive aspect to it. Most gardeners don't keep life lists or anything.