OP - you don't need to grow up. Leaving a job is daunting. I've handed my notice in at 3 jobs now (I'm 25) and it's felt daunting and nervewracking every time.
I think by the way you've described how he's treated you, then you are better off out of there. And if that means you feel uncomfortable returning to there to enjoy food with friends and family then so be it, there will be plenty other lovely options to choose from to visit for food. You have to consider your happiness. If you're unhappy and being treated poorly in a job, which it sounds like you are, it will impact you on so many levels. Your happiness is always most important. The new job you have mentioned might just be a blessing in disguise, it sounds like you're better off out of this current role.
If I were you I would follow the link I provided earlier, there's tons of really helpful info on the gov website about employment rights etc. If you feel that your rights have been breached then gather as much advice as you can about that, before going to your employer to hand your notice in - you'll want to make sure you're as clued up as can be!
Good luck.