This blue-eyed hag was hither brought with childAnd here was left by the sailors. Thou, my slave,As thou report'st thyself, wast then her servant;And, for thou wast a spirit too delicateTo act her earthy and abhorr'd commands,
The "monstrous" characters in this are presumably darker of skin, owing to the origin story of Caliban's parentage as well as the general idea of an "Island People." Historically, this lines up with colonization of Europe, even the colonization of the United States, with worry of running into "Indians." All of this really "others" all of the indigenous people to this mystical island, and portrays them as savages.