While Google will remove PageRank scores from public view in the coming weeks, the way those scores dramatically reshaped the web will remain.
The Age of PageRank is Over
When Sergey Brin and Larry Page came up with the concept of PageRank in their seminal paper The Anatomy of a Large-Scale Hypertextual Web Search Engine (Sergey Brin and Lawrence Page, Stanford University, 1998) they profoundly...
Is Google about to destroy the web?
Google says adding more AI to its search engine will rejuvenate the internet. Others predict an apocalypse for websites. One thing is clear: this era of online history is closing.
The Web Can Thrive Without Google’s Search Monopoly
Viable browsers and meaningful contributions to web standards can be sustained with more modest revenue streams, writes Alissa Cooper.
Solutions for Link Rot on the Modern Web
Web links are fundamental to the web, enabling navigation between pages and citations in research articles. However, web links suffer from "link rot", a phenomenon in which links are likely to become inaccessible over time. This can occur if a link’s site disappears, making it impossible to resolve the hostname or establish a server connection, or the linked page has been deleted, resulting in an HTTP 404 "Not Found" error. Today, a common solution to tackle link rot is to rely on web archives, which capture snapshots of web pages for future reference. However, the modern web has evolved significantly since web archives were first introduced, leading to several limitations in their effectiveness. First, the scale of the web has grown tremendously, making it infeasible to crawl every page whenever it changes. As a result, many broken links either have no archived copies, or the archived copies have stale content. Second, modern web pages rely heavily on increasingly complex and diverse JavaScript. This shift has made it more challenging for preserving fidelity in archived copies, significantly increasing both the computational cost of operating browser-based crawlers and engineering effort required to maintain accurate replay systems. This thesis presents a set of solutions to cope with link rot on the modern web. My work aims to mitigate the various limitations of web archives. First, for broken links without any archived copy, or if the archived copy includes stale content or unavailable functionalities, I built Fable. Fable revives the dead link with the new URL to the same page whenever available. Compared to prior approaches, Fable revives 4.6K broken links—a 50% increase—with much higher accuracy. Second, for pages that require dynamic crawling, I show how web archives can achieve a better tradeoff between efficiency and fidelity. By carefully choosing 8.9% of pages to crawl dynamically and strategically reusing resources from those crawls, an archive can serve 99% of the remaining statically crawled pages without any fidelity loss. Third, for fidelity violations in archived copies that are caused by incorrect edits to crawled scripts, I built FidEx. FidEx reliably detects when an archived page differs from its original version and pinpoints the root cause. After fixing the most common errors pinpointed by FidEx, I reduced the fraction of pages for which FidEx reports a violation of fidelity from 15% to 9%.
Solutions for Link Rot on the Modern Web
Web links are fundamental to the web, enabling navigation between pages and citations in research articles. However, web links suffer from "link rot", a phenomenon in which links are likely to become inaccessible over time. This can occur if a link’s site disappears, making it impossible to resolve the hostname or establish a server connection, or the linked page has been deleted, resulting in an HTTP 404 "Not Found" error. Today, a common solution to tackle link rot is to rely on web archives, which capture snapshots of web pages for future reference. However, the modern web has evolved significantly since web archives were first introduced, leading to several limitations in their effectiveness. First, the scale of the web has grown tremendously, making it infeasible to crawl every page whenever it changes. As a result, many broken links either have no archived copies, or the archived copies have stale content. Second, modern web pages rely heavily on increasingly complex and diverse JavaScript. This shift has made it more challenging for preserving fidelity in archived copies, significantly increasing both the computational cost of operating browser-based crawlers and engineering effort required to maintain accurate replay systems. This thesis presents a set of solutions to cope with link rot on the modern web. My work aims to mitigate the various limitations of web archives. First, for broken links without any archived copy, or if the archived copy includes stale content or unavailable functionalities, I built Fable. Fable revives the dead link with the new URL to the same page whenever available. Compared to prior approaches, Fable revives 4.6K broken links—a 50% increase—with much higher accuracy. Second, for pages that require dynamic crawling, I show how web archives can achieve a better tradeoff between efficiency and fidelity. By carefully choosing 8.9% of pages to crawl dynamically and strategically reusing resources from those crawls, an archive can serve 99% of the remaining statically crawled pages without any fidelity loss. Third, for fidelity violations in archived copies that are caused by incorrect edits to crawled scripts, I built FidEx. FidEx reliably detects when an archived page differs from its original version and pinpoints the root cause. After fixing the most common errors pinpointed by FidEx, I reduced the fraction of pages for which FidEx reports a violation of fidelity from 15% to 9%.
It's time to rebuild the web
The web was never supposed to be a few walled gardens of concentrated content owned by a few major publishers; it was supposed to be a cacophony of different sites and voices.
Bridging-Based Ranking
There is significant concern about the engagement-based ranking systems used by TikTok, Facebook, YouTube, etc. to recommend content. Bridging-based ranking systems can address one of the most dangerous aspects of such algorithmic recommendations—the push toward polarization and divisiveness that is tearing nations apart—and do so without reducing anonymity or increasing censorship. This report explores what bridging-based ranking is, how it helps (overcoming downsides of chronological feeds and middleware), addresses common objections, and provides early examples of its use and benefits in the wild. The report concludes by providing next steps for platforms, governments, funders, and researchers in order to accelerate the deployment of bridging.
Personalized PageRank Estimation and Search: A Bidirectional Approach
We present new algorithms for Personalized PageRank estimation and Personalized PageRank search. First, for the problem of estimating Personalized PageRank (PPR) from a source distribution to a...
Personalized PageRank Estimation and Search: A Bidirectional Approach
We present new algorithms for Personalized PageRank estimation and Personalized PageRank search. First, for the problem of estimating Personalized PageRank (PPR) from a source distribution to a...
The people who ruined the internet
SEO experts got very rich filling the web full of garbage. But are they to blame or is Google?
Why Google’s New AI-Saturated Search Page Will Be A Disaster
Google didn’t invent full-text search of the Internet – that honor belongs to early pioneers such as WebCrawler, Lycos and AltaVista. But for the last 25 years or so, Google has…
What the ephemerality of the Web means for your hyperlinks
Hyperlinks are a powerful tool for journalists and their readers. Diving deep into the context of an article is just a click away. But hyperlinks are a double-edged sword; for all of the internet’s boundlessness, what’s found on the Web can also be modified, moved, or entirely vanished. The fragility of the Web poses an […]
How Google helped destroy adoption of RSS feeds
Learn how one of the biggest tech companies played a major role in the reason many people stopped using RSS feeds
Imagine things being so bad that your best solution is to cut yourself off from Google Search traffic. I don't think that people realise just how much of a bad actor Google has been, for year. No other company has damaged the web nearly as much. 🧵 adweek.com/media/publishers-opt-out-goog…
Publishers Are Preparing to Opt Out of Google Search
www.adweek.comImagine things being so bad that your best solution is to cut yourself off from Google Search traffic. I don't think that people realise just how much of a bad actor Google has been, for year. No other company has damaged the web nearly as much. 🧵 adweek.com/media/publishers-opt-out-goog…
Publishers Are Preparing to Opt Out of Google Search
www.adweek.com