Mapping a Community Online
Recently, I inquired on the ~S~ Seeker’s messageboard about how one would go about trying to find every site on a certain subject or which is part of a certain community on the internet. In the course of the discussion which followed, another poster asked me if I would mind writing an essay about the tricks I’ve used in trying to map a community online. This essay is the result.
The Target
As you may already be aware, if you’ve read anything published on the main domain of Dreamhart.org, I’m a member of the otherkin community online. I’ve been involved with it for the past 11 years. Recently, I’ve begun a project to try to create what I believe is both the most current and the most comprehensive listing of active otherkin websites, forums, chatrooms, and other resources online.
If you really want to know about otherkin, I recommend checking out the rest of Dreamhart.org and the various otherkin resources I’ve collected there. As a very brief definition, though, otherkin consider themselves in some way (most often spiritually) nonhuman. I realize that sounds pretty bizarre to those outside our subculture, possibly even insane. We are, of course, but I hope that won’t stop you from reading on and seeing what I learned in the course of mapping out our unique digital community. Besides, if you already found your way to this site you’re probably pretty weird too. ;-)
A Starting Point
Every search has a beginning. I started my search somewhere around the beginning of June this year. At that time I had a list of six websites, two of which were already defunct. These had been the main websites I linked to from an earlier version of my website. If you’re setting out to map a community, perhaps you’ll have a similar set of links to begin with. Or maybe you’ll need to build one.
If you’re not already familiar with the community, the best way to begin your search is to look for the community’s largest hubs. Otherkin.net, for instance, would be one of the larger hubs of the otherkin community while Searchlores.org would probably count as the largest hub for the serious web searching community. You might find these hubs by a quick and dirty search on google, but a more reliable method is checking directories like Yahoo and DMOZ. Wikipedia can also be a good source if it has an article on your subject: take a look at the citations and external links.
Combing
Once you’ve got links to a handful of community hubs, you’re ready to really begin. These hub sites should contain links (or other information) pointing to other websites where members of the community gather. Focus on those for now, not links to personal websites. You can always come back and comb the personal sites later, but to cover as much ground as quickly as possible it’s a good idea to focus on making a list of community resources at first: forums, mailing lists, directories, and the like. Anything that attracts lots of users from your target community.
For my search, Otherkin.net has a section listing otherkin mailing lists. Many of them are dead, or simply silent, but some are active and I made note of those. More importantly, though, if I wasn’t already familiar with the community it would have pointed me towards Yahoo Groups as a place where many otherkin have created mailing lists. Naturally, I would then move on to searching that site directly.
Similarly, your target community may have certain providers that host a large number of mailing lists, forums, or other community resources. Once you find them, be sure to search those sites yourself rather than relying on community directories to list them all. And once you find the resources there, be sure to check them for links to still more resources… YahooGroups have a links area, LiveJournal communities often include links on their profile pages, forums often have a board for suggested links. Letting others do the searching for you dramatically speeds the process of mapping a community. Work smart, not hard.
Once you’ve gone through the big community hubs you can even comb the links pages of personal sites to find areas of the community that may have fallen through the gaps. Also, don’t forget to look around for other projects trying to map the same community. You may find other searchers have already done a lot of work for you, as I did when I came across The Were Directory and the Otherkin Timeline. And if your community is of a more controversial nature, don’t forget to comb the sites of the people trolling it: often they have a fairly extensive list of links.
Tools
It’s a good idea to do most of your combing by hand, but with larger sites that takes too much time or you may just want to double-check your work. If so, I’d recommend getting a bot to help you out. The two I used were Sphider and the W3S Araña Web Personal by Mhyst. Set them to a reasonable link depth, and you can’t go wrong. Note: the largest setting is not a reasonable link depth. Five deep may even be pushing it.
W3S outputs all links exactly the way they were originally formatted on the pages they appeared in, so you may also want to make use of a bookmarklet to change the link text to their full urls when analyzing its output.
Blast from the Past
If the community you’re targeting is an older one, chances are that some of its major resources have gone defunct over time. They may still be of use to you… if you can find the URL of the dead site, check it out through Archive.org. It may contain links that more current sites no longer index, or which are no longer conveniently listed together. For the otherkin community, such sites included Pathwayz.org, the Otherkin Resource Center, and the original TirNanOc.
Digging Deeper
Combing sites will only get you so far when mapping a community. Communities are cliquish, and while some main resources may be linked to, most of the connections in a community are just friends linking to friends. Relatively few people will link to those they dislike. From a searching perspective, this creates many small pockets of interlinked sites that may or may not link back into the main community. Some may even start to splinter off into their own communities, with their own specialized jargon distinct from that of the main group.
To find these sites, you need to go back to the search engines. Try some vertical search engines to dig through twitter, blogs, webrings and forum posts. If that doesn’t work, try Rollyo and create a custom search roll (or several) for free website or forum providers to scan through. Finally, head back to the main search engines and try some substitutions: instead of searching for the normal name of your community, try any synonyms you’ve discovered in the course of searching. Or try substituting a part for the whole… for instance, “dragonkin” or “angelkin” instead of “otherkin”. Or try phrases that might express the concept. Basically, all the things you would normally do at the start of a search. We’re doing them last because it’s faster and more efficient to let others do your searching for you, and then use these methods to fill in the gaps and pick up any strays.
Also, since you should by now have a large group of sites that fit your search criteria, take a good look at them. Are there any characteristics that stand out that you didn’t know at the beginning of your search? Any common phrases you haven’t already searched for? Any authors or essayists you haven’t searched for? What about checking for sites that link to ones you’ve already found? Use these pearls as seeds to grow your search, expanding each specific example into a treasure trove of new results.
One last thing you might try? Ask. Go on community forums, chatrooms, etc and just ask. Maybe you won’t get any responses, maybe you will, but you’ll never know unless you ask. And who really knows the community better than the people who are part of it?