Was it wrong to rehome this cat, especially if he was miserable with us?

You totally did the right thing. You're happier, the cat's happier, no one's getting hurt. It would have been a bad choice to keep him. You didn't go against what you believe in. Just because you're not the one taking care of him, you do know who is. At least you didn't throw him out on the streets like some people do.