He's said before he struggles to put weight on, he's tried "bulking" and it just doesn't work for him. Him and Zoe both have very slim builds naturally, so it is partly their genetics meaning he has a naturally slim build. That's just the way he is.He might not be able to, my boyfriend isn’t far from Joes size and is physically unable to put weight on (unless he ate a horrendous unsustainable amount of calories a day anyway). Some men are just like that. It can be really hurtful for them when people point out how small they are as if it’s somehow not manly enough.
Like you say, it's pretty hurtful to be so negative about, and body image is a big issue for men, mental health and so called "manliness", can lead to major issues with eating disorders and body dysmorphia, and men ending up in the gym or roiding up to get a so called "manly" body.