Do you think I was wrong?

No. He is your dog. However why not drop him off there sometimes for a visit. Kind of like you would take a child to play at someone elses house. That way everyone would get to be with the dog.