The Mishna, Kiddushin 66b, states: "Whenever kiddushin would not take effect between she and he, but she would have kiddushin with another man, the offspring is a mamzer. What case is this? This is one who came upon one of the arayos in the Torah." That the offspring is a mamzer requires further elaboration: does the fatehr's lineage still exist (and if the father was a kohen, the offspring will be a kohen mamzer) or does the mamzerus uproot the mishpachas av?
Koveitz He'aros 44:2 says: "In Shu"t HaRashba...without the Rashba we could have said another thing regarding that which a mamzer does not have kehuna - the psul removes him from from the kedushas kahal, it certainly removes him from the status of kohen....and we see this in Gittin 59 and Tosefta Yevamos 8:1 a female Levi who was held captive.... It is simple from this that mamzeirus removes the mishpachas av.
This is also implied in Mishna Brurah 135:30: "Even if they know their father is a Levi, we must suspect that their father married a mamzeres or nesina and disqualified his children from the kedushas Leviyah, and as a mamzer he reads from the Torah (i.e., gets an aliya) like any other Yisrael.
Chidushei Hagram Halevi (p. 54) s.v. Yevamos 87 says: the din that the offspring follows the psul of either of them is only a status of psul. The offspring of a regular Yisrael and an Egyptian convert is in the khal Yosrael and not in khal geirim, because his family is his father's mishpachas av. Only, he also has a status of psul of Mitzri...
This is also the shita of Rav Chaim, Isurei Biah 15:9.
See Tosfos Kiddushin 68b s.v. vladah kimosah: He explains the question of the gemara that how do we know the vlad of a non-Jewish woman follows her - "what is the source that the offspring is like her, that it will not be called his offspring for anything, not even his son but [a son who is] a mamzer...." According to Koveitz He'aros, if mamzeirus removes mishpachas av, what is Tosfos saying? We see Tosfos holds like Rav Moshe and Rav Chaim that the offspring is a mamzer and has mishpachas av. But the son of a non-Jewish woman will not be considered the Jewish father's son at all.
--Heard from Rav Chaim Ilson