Will libertarianism lead to some sort of national socialist racial state?

No, quite the opposite. Libertarians favor the smallest possible government. Under a true Libertarian regime, people would be left alone regardless of race, creed, color, gender, national origin or even ual orientation. The root word of Libertarian is "Liberty." This is the opposite of Nazism or any other form of authoritarian/totalitarian oppression.