I think you only need consent if you’re moving abroad or if you’re taking someone else’s child abroad. (When my mum took my step sister abroad, without my step dad, he wrote a letter didn’t get asked though)
Im pretty sure yours and your children’s passports will be linked. Like they would have seen you have travel together before.
I am 19 so my information might be a little out of date, but I think as long as your kids father doesn’t reported them as missing when you go on holiday it seems like it would be fine. It’s more as long as you have his permission it’s fine rather than specifically written permission