You shouldn't have to juggle your relationship with him and your relationships with your mother and daughter - a good partner wouldn't try and make you. At the end of the day, your daughter was there before him and she'll be there after him too.
The fact that you say he doesn't care if you're upset is so concerning too, what kind of partner can't support someone when they're upset?
Ultimately its your choice and its so easy to sit at a keyboard and say you need to end it, but for the sake of your own mental health and your relationship with your daughter, its something you need to give some serious thought.