Gulliver is resentful of going back, and when he does, he realizes he does not want to be around his own wife and child because they remind him of the Yahoos.
Kind of sad that he left his pregnant wife to travel and now does not want to return to her and even finds her annoying. He also is not super fond of being around his own child, just wants to go back to the talking horses that are apparently better than his family.