The Queen’s Men aren’t named necessarily for their dedication to Queen Selyse. Instead it refers to their conversion to the R’hllor faith. And while many might be concerned that their true loyalties lie with the Red Priestess for the time being they are just as willing to head to battle in service of Stannis Baratheon. In time their true nature might come to light.