He doesn't really annoy me to be honest, but he just upsets me when he swears or says I am being delirious .I think you need to delete him, tbh. There sounds nothing fun about the chats you're having with him and you both seem to be irking each other. Inadvertently in your case. But he's being a diberate bellend by the sounds of it. Not the best prelude to meeting
You are right though, an I constantly feel like I'm walking on eggshells and that he's going to take everything I say the wrong way. We already met on Saturday and had a nice time I thought.