In my experience it's largely people who would have been considered part of the les community who are attempting to change the parameters though. AMAB non binary people and trans women have rarely been the ones causing problems.Anyone seeking to get you to redefine yourself to suit their agenda is a red flag in my opinion and I’m not surprised the non-binary identity has made you wary. As someone who doesn’t accept the premise of gender ideology, it concerns me that people who would formerly not have been considered part of the les community are using this ideology to gain access to les spaces and change the parameters of what it means to be les /gay/ bi.
Edited to add: this is based on real life anecdotes though, online things seem a bit different