In an age of LLMs, is it time to reconsider human-edited web directories?
In an age of LLMs, is it time to reconsider human-edited web directories?
In an age of LLMs, is it time to reconsider human-edited web directories?
Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.
These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/Fashion/etc.
Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.
Typically, these directories also had a limited web search that would crawl through the pages of websites listed in the directory.
Lycos, Excite, and of course Yahoo all offered web directories of this sort.
(EDIT: I initially also mentioned AltaVista. It did offer a web directory by the late '90s, but this was something it tacked on much later.)
By the late '90s, the standard narrative goes, the web got too big to index websites manually.
Google promised the world its algorithms would weed out the spam automatically.
And for a time, it worked.
But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.
And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.
My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?
Do we really want to search every single website on the web?
Or just those that aren't filled with LLM-generated SEO spam?
Or just those that don't feature 200 tracking scripts, and passive-aggressive privacy warnings, and paywalls, and popovers, and newsletters, and increasingly obnoxious banner ads, and dark patterns to prevent you cancelling your "free trial" subscription?
At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of trustworthy, quality websites?
And is it time to begin considering what a modern version of those early web directories might look like?
@degoogle #tech #google #web #internet #LLM #LLMs #enshittification #technology #search #SearchEngines #SEO #SEM
Lycos, Excite, AltaVista, and of course Yahoo all were originally web directories of this sort.
Both Wikipedia and my own memory disagree with you about Lycos and AltaVista. I'm pretty sure they both started as search engines. Maybe they briefly dabbled in being "portals".
@bsammon And this Archive.org capture of Lycos.com from 1998 contradicts your memory: https://web.archive.org/web/19980109165410/http://lycos.com/
See those links under "WEB GUIDES: Pick a guide, then explore the Web!"?
See the links below that say Autos/Business/Money/Careers/News/Computers/People/Education /Shopping/Entertainment /Space/Sci-Fi/Fashion /Sports/Games/Government/Travel/Health/Kids
That's exactly what I'm referring to.
Here's the page where you submitted your website to Lycos: https://web.archive.org/web/19980131124504/http://lycos.com/addasite.html
As far as the early search engines went, some were more sophisticated than others, and they improved over time. Some simply crawled the webpages on the sites in the directory, others
But yes, Lycos definitely was definitely an example of the type of web directory I described.
1998 isn't "originally" when Lycos started in 1994. That 1998 snapshot would be their "portal" era, I'd imagine.
And the page where you submitted your website to Lycos -- that's no different than what Google used to have. It just submitted your website to the spider. There's no indication in that snapshot that suggests that it would get your site added to a curated web-directory.
Those late 90's web-portal sites were a pale imitation of the web indices that Yahoo, and later DMoz/ODP were at their peak. I imagine that the Lycos portal, for example, was only managed/edited by a small handful of Lycos employees, and they were moving as fast as they could in the direction of charging websites for being listed in their portal/directory. The portal fad may have died out before they got many companies to pony up for listings.
I think in the Lycos and AltaVista cases, they were both search engines originally (mid 90s) and than jumped on the "portal" bandwagon in the late 90s with half-assed efforts that don't deserve to be held up as examples of something we might want to recreate.
Yahoo and DMoz/ODP are the only two instances I am aware of that had a significant (like, numbered in the thousands) number of websites listed, and a good level of depth.
Main problems are:
-
Link rot
-
Sneakily inserted sponsored links
@Moonrise2473 @ajsadauskas
3. Infinitely growing list of categories.
4. Mis-categorisationi remember learning HTML (4.0) and reading that you should put info in a <meta> tag about the categories your page fits in, and that would help search engines. Did it also help web directories?
-
@ajsadauskas @degoogle I guess the problem though is how you make sure they are actually maintained by a human acting in good faith. The way community Facebook groups meant to be for this kinda thing get spammed by likely fake businesses doesn’t give me hope
@joannaholman @degoogle Good point.
If it were run as a private company, I think the solution might be just to pay actual humans as employees.
If it's a community-run project, the challenge would be to come up with a robust moderation system...
@ajsadauskas @joannaholman @degoogle maybe a mix of wikipedia and search engine would be nice. WikiSearch?
I suppose any measures at all would cut out a massive number of spam pages already.
@ajsadauskas @degoogle What we need to do is re-visit the GnuPG philosophy of building rings of trust. If one emerges with enough people proven to provide quality aggregators/summarizers then we can start to depend on that, or those.
@ajsadauskas @degoogle Webrings! Bring back Webrings!
Reddit and Lemmy are supposed to be what you want: link aggregators.
We're supposed to link to sites and pages and people vote on how good they are in the context of the sub community topic.
Of course, then Ron Paul happened, and now it's just memes and Yank politics so... maybe deploy Lemmy and turn off comments.
I think you are mostly right, except Lemmy and reddit are not organized.
Yeah, it's the lack of organisation that is the issue and if we are thinking about web directories, there is the missing element of deliberate creation.
I'd argue that link aggregators like Lemmy (from which I'm posting o/) are the new world version of that. Link aggregators are human-edited web directories; humans post links and other humans vote whether those links are relevant to the "category" (community) they're in. The main difference is that it's an open communal effort with implicit trust rather than closed groups of permitted editors.
The problem is bots
I'm sadden to say that one of my jobs in 2014 was to build bots for a company. And the first thing they did was use it to spam social media with links, and bots to reply to the original bot to appear more human and give it "authority" and "social proof". That practice boosted sales dramatically.
Some of the bot libraries that are openly available, along with AI, makes the things I did look like child play.
All instances of malicious bots I saw around here were downvoted into oblivion.
I used them and contributed to links as well - it was quite a rush to see a contribution accepted because it felt like you were adding to the great summary of the Internet. At least until the size of the Internet made it impossible to create a user-submitted, centrally-approved index of the Net. And so that all went away.
What seemed like a better approach was social bookmarking, like del.icio.us, where everyone added, tagged and shared bookmarks. The tagging basically crowd-sourced the categorisation and meant you could browse, search and follow links by tags or by the users. It created a folksonomy (thanks for the reminder Wikipedia) and, crucially, provided context to Web content (I think we're still talking about the Semantic Web to some degree but perhaps AI is doing this better). Then after a long series of takeovers, it all went away. The spirit lives on in Pinterest and Flipboard to some degree but as this was all about links it was getting at the raw bones of the Internet.
I've been using Postmarks a single user social bookmarking tool but it isn't really the same as del.icio.us because part of what made it work was the easy discoverablity and sharing of other people's links. So what we need is, as I named my implementation of Postmarks, Relicious - pretty much del.icio.us but done Fediverse style so you sign up to instances with other people (possibly run on shared interests or region, so you could have a body modification instance or a German one, for example) and get bookmarking. If it works and people find it useful a FOSS Fediverse implementation would be very difficult to make go away.
Oh indeed there are services out there that do something similar to Delicious, but I put a lot into that site only for it all to disappear due to the whims of some corporate overlord and I am not doing that again. What I am looking for is an easy Fediverse solution so my data is never lost again. Postmarks is definitely getting there but as a single-user service it isn't quite what I am looking for.
@Emperor
This this this! Some kind of service that would sit alongside a fedi instance and serve as a community directory.
@ajsadauskasIndeed. Places like Lemmy and Reddit might be called "link aggregators" but they are, ultimately, jumped up web forums (and that's no slight, I'm a web forum guy through and through) and are nothing like the social bookmarking sites, like Delicious, which had greater breadth and depth (just look at your own bookmarks, you'd only share a fraction on here but you put a larger percentage into social bookmarking) but, crucially, essentially crowd-sourced the organisation and categorisation of those links.
Some kind of service that would sit alongside a fedi instance
I have been pondering the idea of "Fediverse plug-ins" that would do that, extending the core functionality of the service.
So in the case of, what we'll call, Fedilicious users of the service could either punt over links they post to Mastodon or Lemmy to a social bookmarking plug-in where it is stored and categorised (or you could run a not to do this automatically) but they could also add links that might not be worth a new post or storing away for future reference, etc. You would then have a curated, easily-accessible repository of links that reflect the interests of that instance.
It needn't itself be federated but if you did, you could have some "everything" sites (fedilicious.world?) which would accepted all links from other Fedilicious instances it is federated with (which would tend to be set to broadcast mode, so categorised links go out, they don't receive all the links, although users could be allowed to add links to it from elsewhere).
@Emperor @ajsadauskas I've been thinking about this myself lately - but I had wondered how a curated directory might scale, I hadn't considered federated social bookmarking and honestly that sounds like a brilliant solution. I'd love to see something like that happen, maybe even contribute
As the links show, Relicious/Fedilicious has been on my mind a while and I have been mourning the loss of Delicious for a long time. However, the above got me jotting down some notes.
It should be doable. I haven't had a root through PostMark's code but it might be they have done the bulk of the work already and it just needs a multiuser interface bolting on top of it.
@Wren @Emperor @ajsadauskas Back in the day people's web sites had a links page and if their site was good it was always worth looking at what they listed as worthy links. I still have one but it's out of habit rather than being useful. Might rethink now tho.
@Emperor @ajsadauskas that's Lemmy?
Although Lemmy is called a link aggregator it is really just a kind of web forum and nothing like a social bookmarking service.
I had this exact thought earlier today. Either curated directories, or a ground-up, vetted search engine that only pulls from pre-screened sources.
This is how it's gonna go. we'll get human-curated search results, before someone "innovates" by mildly automating the process until someone "innovates" again by using AI to automate it further. Time is a circle
@ajsadauskas @degoogle i love this idea, i'm going to start my own web directory.
Do it!
Then federate it.
@ajsadauskas @degoogle Since I run a small directory this is a fascinating conversation to me.
There is a place for small human edited directories along with search engines like Wiby and Searchmysite which have human review before websites are entered. Also of note: Marginalia search.
I don't see a need for huge directories like the old Yahoo, Looksmart and ODP directories. But directories that serve a niche ignored by Google are useful.
@bradenslen @ajsadauskas @degoogle looksmart! There's a blast from the past.
As a very early internet user (suburbia.org.au- look it up, and who ran it) and a database guy, what I learnt very early is that any search engine needed users who knew how to write highly selective queries to get highly specific results.
Google - despite everything - can still be used as a useful tool - if you are a skilled user.
I am still surprised that you are not taught how to perform critical internet searching in primary school. It is as important as the three Rs
But directories that serve a niche ignored by Google are useful.
This is a good point - as search is increasingly enshittified too (from top down, with corporate interests, and bottom up, from SEO manipulation and dodgy sites) it makes sense for topics or communities often drowned out by the noise.
I also see you are using webrings - another blast from the past that has it's uses.
@ajsadauskas @degoogle I actually contributed to one! I was a writer at LookSmart for four years; we manually created categories and added websites to then, with short descriptive reviews. Though an algorithm listed more sites below our selections, we could force the top result, eg we'd make sure the most relevant website was the first result of a search on that topic. Old-skool now, but had better results in some ways.
And is it time to begin considering what a modern version of those early web directories might look like?
Something like fmhy.net?
The tale of the internet has been curation, and I would describe it a little differently.
First we had hand made lists of website (Yahoo directory, or we had a list of websites literally written in pen in a notebook saying "yahoo.com" and "disney.com").
Then it was bot-assisted search engines like Google.
Then there was so much content we didn't even know where to start with Google, so we had web rings, then forums, then social media to recommend where to go. Then substack style email newsletters from your chosen taste makers are a half-step further curated from there.
If that is all getting spammed out of existence, I think the next step is an AI filter, you tell the AI what you like and it sifts through the garbage for you.
The reasons we moved past each step are still there, we can't go back, but we can fight fire with fire.
@ajsadauskas Back when, UW Madison hosted an outfit called The Internet Scout Project that was in the curation business for web resources. The decaying state of search (alternatively the growth of web resources intended to serve interests other than their visitors') has me thinking it would be good to work with public libraries to convene and host this sort of thing.
Librarianship is the right sort of ethos for it, and libraries are infrastructure for human-mediated discoverability.
@ajsadauskas @degoogle definitely something o be thinking about. More and more I’m using my followed hashtags, mastodon lists, and links to resources other people provide rather than just finding useful things in search results. But the big gap is still when I want to find quality info on a new topic. Cannot trust any of the damn results searching for how and how often to clean my kid’s new aquarium, for example. So much LLM and SEO crap info.
Don't clean his new aquarium for like months.
If you can see this response (this half baked Mastodon integration annoys the shit out of me), I'd be happy to talk you through it.
@TheOctonaut What do you mean by aquarium?
@ajsadauskas @degoogle I used to be one of those human editors. I was the editor of Scotland.org from about 1994 to about 1997, back in the days when it was exactly one of those hierarchical web directories – with the intention of indexing every website based in Scotland.
@ajsadauskas @degoogle having said that, the patents on Google's PageRank algorithm have now all expired, and a distributed, co-op operated search engine would now be possible. Yes, there would be trust issues, and you'd need to build fairly sophisticated filters to identify and exclude crap sites, but it might nevertheless be interesting and useful.
@ajsadauskas @degoogle DMOZ was once an important part of the internet, but it too suffered from abuse and manipulation for traffic.
For many DMOZ was the entry point to the web. Whatever you were looking for, you started there.
Google changed that, first for the better, then for the worse.
Yes to all. For a while I've been de facto using a miniscule subset of the web. My gateway to other, relevant websites are via human-to-human recommendations, primarily in a place like this.
And just now, as seen at the bottom of a blog post:
"Post a Comment
Unfortunately because of spam with embedded links (which then flag up warnings about the whole site on some browsers), I have to personally moderate all comments. As a result, your comment may not appear for some time. In addition, I cannot publish comments with links to websites because it takes too much time to check whether these sites are legitimate."
It looks like there's a couple projects to continue the directory DMOZ. I hope they're sharing work with each other!
Got any links?
Yeah. Sorry, I was hesitant to post links at first before I vetted them.
It looks like "Curlie" is the official continuation of the DMOZ project:
The other ones I was seeing, it turns out, are static mirrors of 2017 DMOZ.
And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.
True, but these things can also be used by us, to curate/maintain a high quality link collection. However, I'm not sure 'pages' will be read by humans in 5 years, so I have a feeling we wont need such a collection anymore. Well, not for humans but probably for our individual LLM's.
Just to add to your list of steps and consequences: I also think academic studies about information retrieval, indexing and crawling became less popular. Aspirant students hearing the message: those studies / workfields will become obsolete once AI does all that.
I remember a time when you could be a paper magazine every other week with curated lists of link on various topics. There were ads, but just paper ads :)
@ajsadauskas @degoogle
I've already seen new webrings forming.Or maybe that was old webrings updating?
Yeah, I was just looking at a webring and thinking "these still have a use". They could definitely help with discoverablity on a broad front. I help Admin feddit.uk and had pondered reaching out to other British Fediverse services to make a Britiverse. However, how to hold it all together and navigate between them was proving tricky or clunky until I was looking at the webring and thought "FedRing". Now that could work.
@ajsadauskas @degoogle Curation is elation.
@ajsadauskas @degoogle a bit of history of Yahoo here, started as a web directory https://www.wired.com/1996/05/indexweb/
@ajsadauskas @degoogle So, classic mid-90s Yahoo. Or LookSmart, which was initially curated by Reader's Digest.
@SDWolf @ajsadauskas @degoogle fork the web
What's to say we won't have AI-curated lists and directories? That way we don't have to deal with link rot and the like. I think the issue is the algorithms used for search. We need better ones, better AI, not more frivolous human labor.
@ajsadauskas @degoogle I mean we could still use all modern tools. I'm hosting a searxng manually and there is currently an ever growing block list for AI generated websites that I regularly import to keep up to date. You could also make it as allow list thing to have all websites blocked and allow websites gradually.
@ajsadauskas @degoogle I started that because it bothered me that you couldn't just report a website to duckduckgo that obviously was a stackoverflow crawler. This problem persists since reddit and stackoverflow are a thing themselves. why are there no measurements from search engine to get a hold of it.
I never understood that.
@ajsadauskas @degoogle hopefully they don't look like Dmoz, because i still have unpleasant flashbacks of that dark time 😋
@ajsadauskas @degoogle i encountered https://aboutideasnow.com/ recently
it's not exactly what you describe and remember, but it's... at least real.
i was reminded by this post of my days from like 1998-2001ish on About.com (RIP)
@ajsadauskas @degoogle it sounds a bit like Kagi‘s Small Web initiative and search. have you seen it? https://blog.kagi.com/small-web
I don’t know if this is the intent of their small web effort, but the first impression I got just now when I clicked through and saw this was, “gross”.
Scummy looking/feeling marketing hustle is a huge turn off. (edit: I went back to the Kagi small website again and it seems to be sort of like stumbleupon, as it served up a different, way less gross site, and has the next site button to browse through whatever they’re featuring)
@ajsadauskas @degoogle ah the good ol' days. I was a curator on yahoo's directory for a few years, before it ended.
@ajsadauskas @degoogle Curlie https://curlie.org/ is the continuation of the ODP
@ajsadauskas @degoogle I've always wanted to try or contribute to one of these!
@ajsadauskas Lemmy instances without comments?
It would be sad to go back to walled gardens like AOL, particularly since they were corporate-owned. But a sort of Kite Mark, certifying a site is free of LLMs, would be useful. Then users could choose for themselves.
@ajsadauskas @degoogle
Fagan Finder
https://www.faganfinder.com/
All the Internet
https://www.alltheinternet.com/
https://start.me/p/wM7y15/search-engine
https://www.searchall.net/
Awesome lists about all kinds of interesting topics
https://github.com/sindresorhus/awesome@ajsadauskas I think Github's awesome lists are kind of like this. They're human-maintained catalogues of worthwhile websites on a specific topic.
@ajsadauskas sounds like you want https://curlie.org/ - which seems to be up to date and interesting.
@ajsadauskas @degoogle
New online family game is coming next month ! Only first 1000 will get to play it for free for 1 month !Check out https://www.meeteli.com