The bullying I received when my teacher tried that because they couldn't find things for me to do.
Plus kids "love" being taught by the geek in the room.
I will say the flip side of that coin you're tossing around is that "gifted/talented" often turns into "highly motivated/comes from a good family." I understand people getting upset about this type of segregation as well.
I am actually very sympathetic to the argument that schools can't make that much of a difference given the challenges in many childrens' home lives. But at some point, they have to explain why they need lots of my tax money but they won't educate my kid beyond some minimum threshold (admittedly, I am in California, where there is no money for G/T but where my local elementary school has a very well-funded anti-bullying curriculum, so I may be extra cynical). Or they need to admit that their job is only to get kids to that minimum standard, and accept that a lot of parents will opt out.