white population
At the time the university was established, America was not really a diverse nation and included mostly white people who hailed from England and other European nations. The majority of the population, especially citizens, was white and black slaves were not even considered citizens. Even if there was immigration, the immigrants mainly came here to work and were not thought of as the population that is receiving education. While it is unfortunate that equality didn't exist back then, it is also necessary to understand that the circumstances were different; the founders tried to appeal mainly to the population that would actually attend college there, the white Americans.
While it is unfortunate that equality among races didn't exist back then, it would be wrong to expect such of America in 1818. The people then thought of blacks as lesser than them and the progressive concept of treating all races and genders equally didn't exist back then. Faulting them for such mentality would be an example of "ex post facto" where we would be blaming them of wrongdoings that weren't considered wrongdoings back then. For instance, before the 1860s-1870s, cousin marriage was thought to be normal and accepted by the society. Now everyone knows about the dangers of reproducing among closely related family members, making it a rare instance in modern America. What is right and what is wrong changes with time and we should have an open mind when analyzing such situations in order to get an accurate picture of history.