This is a great question and I ask it frequently with many different answers.
I've not seen specific numbers, but I suspect that the majority of Hypothes.is users are annotating in small private groups/classes using their learning management system (LMS) integrations through their university. As a result, using it and hoping for a big social experience is going to be discouraging for most.
Of course this doesn't mean that no one is out there. After all, here you are following my RSS feed of annotations and asking these questions!
I'd say that 95+% or more of my annotations are ultimately for my own learning and ends. If others stumble upon them and find them interesting, then great! But I'm not really here for them.
As more people have begun using Hypothes.is over the past few years I have slowly but surely run into people hiding in the margins of texts and quietly interacted with them and begun to know some of them. Often they're also on Twitter or have their own websites too which only adds to the social glue. It has been one of the slowest social media experiences I've ever had (even in comparison to old school blogging where discovery is much higher in general use). There has been a small uptick (anecdotally) in Hypothes.is use by some in the note taking application space (Obsidian, Roam Research, Logseq, etc.), so I've seen some of them from time to time.
I can only think of one time in the last five or so years in which I happened to be "in a text" and a total stranger was coincidentally reading and annotating at the same time. There have been a few times I've specifically been in a shared text with a small group annotating simultaneously. Other than this it's all been asynchronous experiences.
There are a few people working at some of the social side of Hypothes.is if you're searching for it, though even their Hypothes.is presences may seem as sparse as your own at present @tonz.
Some examples:
@peterhagen Has built an alternate interface for the main Hypothes.is feed that adds some additional discovery dimensions you might find interesting. It highlights some frequent annotators and provide a more visual feed of what's happening on the public Hypothes.is timeline as well as data from HackerNews.
@flancian maintains anagora.org, which is like a planet of wikis and related applications, where he keeps a list of annotations on Hypothes.is by members of the collective at https://anagora.org/latest
@tomcritchlow has experimented with using Hypothes.is as a "traditional" comments section on his personal website.
@remikalir has a nice little tool https://crowdlaaers.org/ for looking at documents with lots of annotations.
Right now, I'm also in an Obsidian-based book club run by Dan Allosso in which some of us are actively annotating the two books using Hypothes.is and dovetailing some of this with activity in a shared Obsidian vault. see: https://boffosocko.com/2022/03/24/55803196/. While there is a small private group for our annotations a few of us are still annotating the books in public. Perhaps if I had a group of people who were heavily interested in keeping a group going on a regular basis, I might find the value in it, but until then public is better and I'm more likely to come across and see more of what's happening out there.
I've got a collection of odd Hypothes.is related quirks, off label use cases, and experiments: https://boffosocko.com/tag/hypothes.is/ including a list of those I frequently follow: https://boffosocko.com/about/following/#Hypothesis%20Feeds
Like good annotations and notes, you've got to put some work into finding the social portion what's happening in this fun little space. My best recommendation to find your "tribe" is to do some targeted tag searches in their search box to see who's annotating things in which you're interested.