Varys also sounds like a Valyrian-styled name. Or Targaryen, though they may be one in the same. There's only one other Valyrian noble house in Westeros and we don't see much of them.
I googled it and apparently people are speculating he could be descended from a Blackfyre from across the Narrow Sea, which lines up with his background from what we know.