Welp, Pocket shuts down tomorrow despite our pleas for it to stay. While migrating1 all of my saved articles, I noticed that I’ve got almost 900 saved articles spanning nearly 7 years. That’s a goldmine of stuff-I-like data! Some quick analysis using xsv2: 𝄢 unzip pocket.zip && xsv headers part_000000.csv 1 title 2 url 3 time_added 4 tags 5 status 𝄢 xsv sample 1 part_000000.csv | xsv flatten title The Uncertain Future of American Libraries url https://mek.fyi/posts/the-uncertain-future-of-american-libraries time_added 1678243022 tags status unread 𝄢 xsv count part_000000.csv 878 𝄢 xsv select time_added part_000000.csv | xsv stats | xsv select min,max | xsv flatten | while read header value; do echo -n "$header " date -I -d "@$value" done min 2018-07-12 max 2025-06-26 Experiment Recalling Simon Willison’s recent geoguessing challenge for o3, I considered, “What might o3 be able to tell me about myself, simply based on a list of URLs I’ve chosen to save?” I opened a temporary chat with o3 (doesn’t use ChatGPT’s memory feature) and prompted:
Newspaper3k: Article scraping & curation — newspaper 0.0.2 documentation
Inspired by requests for its simplicity and powered by lxml for its speed:
Housing Data
Note: We make occasional changes to CSV download paths. Monthly data is updated on the 16th of each month and most weekly data is updated every Tuesday. To explore the data, please visit the market explorer dashboard.
Apache Calcite | Proceedings of the 2018 International Conference on Management of Data
As part of the Digital Library's transition to Open Access, new features for researchers are available in the Premium Edition. Click here to learn more.
File Over App: A Philosophy for Digital Longevity
My take on the 'file over app' philosophy and why it’s essential for keeping my data resilient and built to last.
Solutions for Link Rot on the Modern Web
Web links are fundamental to the web, enabling navigation between pages and citations in research articles. However, web links suffer from "link rot", a phenomenon in which links are likely to become inaccessible over time. This can occur if a link’s site disappears, making it impossible to resolve the hostname or establish a server connection, or the linked page has been deleted, resulting in an HTTP 404 "Not Found" error. Today, a common solution to tackle link rot is to rely on web archives, which capture snapshots of web pages for future reference. However, the modern web has evolved significantly since web archives were first introduced, leading to several limitations in their effectiveness. First, the scale of the web has grown tremendously, making it infeasible to crawl every page whenever it changes. As a result, many broken links either have no archived copies, or the archived copies have stale content. Second, modern web pages rely heavily on increasingly complex and diverse JavaScript. This shift has made it more challenging for preserving fidelity in archived copies, significantly increasing both the computational cost of operating browser-based crawlers and engineering effort required to maintain accurate replay systems. This thesis presents a set of solutions to cope with link rot on the modern web. My work aims to mitigate the various limitations of web archives. First, for broken links without any archived copy, or if the archived copy includes stale content or unavailable functionalities, I built Fable. Fable revives the dead link with the new URL to the same page whenever available. Compared to prior approaches, Fable revives 4.6K broken links—a 50% increase—with much higher accuracy. Second, for pages that require dynamic crawling, I show how web archives can achieve a better tradeoff between efficiency and fidelity. By carefully choosing 8.9% of pages to crawl dynamically and strategically reusing resources from those crawls, an archive can serve 99% of the remaining statically crawled pages without any fidelity loss. Third, for fidelity violations in archived copies that are caused by incorrect edits to crawled scripts, I built FidEx. FidEx reliably detects when an archived page differs from its original version and pinpoints the root cause. After fixing the most common errors pinpointed by FidEx, I reduced the fraction of pages for which FidEx reports a violation of fidelity from 15% to 9%.
Solutions for Link Rot on the Modern Web
Web links are fundamental to the web, enabling navigation between pages and citations in research articles. However, web links suffer from "link rot", a phenomenon in which links are likely to become inaccessible over time. This can occur if a link’s site disappears, making it impossible to resolve the hostname or establish a server connection, or the linked page has been deleted, resulting in an HTTP 404 "Not Found" error. Today, a common solution to tackle link rot is to rely on web archives, which capture snapshots of web pages for future reference. However, the modern web has evolved significantly since web archives were first introduced, leading to several limitations in their effectiveness. First, the scale of the web has grown tremendously, making it infeasible to crawl every page whenever it changes. As a result, many broken links either have no archived copies, or the archived copies have stale content. Second, modern web pages rely heavily on increasingly complex and diverse JavaScript. This shift has made it more challenging for preserving fidelity in archived copies, significantly increasing both the computational cost of operating browser-based crawlers and engineering effort required to maintain accurate replay systems. This thesis presents a set of solutions to cope with link rot on the modern web. My work aims to mitigate the various limitations of web archives. First, for broken links without any archived copy, or if the archived copy includes stale content or unavailable functionalities, I built Fable. Fable revives the dead link with the new URL to the same page whenever available. Compared to prior approaches, Fable revives 4.6K broken links—a 50% increase—with much higher accuracy. Second, for pages that require dynamic crawling, I show how web archives can achieve a better tradeoff between efficiency and fidelity. By carefully choosing 8.9% of pages to crawl dynamically and strategically reusing resources from those crawls, an archive can serve 99% of the remaining statically crawled pages without any fidelity loss. Third, for fidelity violations in archived copies that are caused by incorrect edits to crawled scripts, I built FidEx. FidEx reliably detects when an archived page differs from its original version and pinpoints the root cause. After fixing the most common errors pinpointed by FidEx, I reduced the fraction of pages for which FidEx reports a violation of fidelity from 15% to 9%.
homepages.news — News Homepages documentation
An open-source archive that gathers, saves, shares and analyzes news homepages
Crossref: The sustainable source of community-owned scholarly metadata
This paper describes the scholarly metadata collected and made available by Crossref, as well as its importance in the scholarly research ecosystem. Containing over 106 million records and expanding at an average rate of 11% a year, Crossref’s metadata has become one of the major sources of scholarly data for publishers, authors, librarians, funders, and researchers. The metadata set consists of 13 content types, including not only traditional types, such as journals and conference papers, but also data sets, reports, preprints, peer reviews, and grants. The metadata is not limited to basic publication metadata, but can also include abstracts and links to full text, funding and license information, citation links, and the information about corrections, updates, retractions, etc. This scale and breadth make Crossref a valuable source for research in scientometrics, including measuring the growth and impact of science and understanding new trends in scholarly communications. The metadata is available through a number of APIs, including REST API and OAI-PMH. In this paper, we describe the kind of metadata that Crossref provides and how it is collected and curated. We also look at Crossref’s role in the research ecosystem and trends in metadata curation over the years, including the evolution of its citation data provision. We summarize the research used in Crossref’s metadata and describe plans that will improve metadata quality and retrieval in the future.
franssjz/cpr-vcodex
CPR-VCODEX : Improving the reading experience and consistency for Xteink X4. One-click install or edit stats via the link below: ⬇⬇
Readeck: Home
Save interesting articles, long read, pictures, videos. Read or revisit them later.
The State of Papers, Retractions, and Preprints: Evidence from the CrossRef Database (2004-2024)
A 20-year analysis of CrossRef metadata demonstrates that global scholarly output -- encompassing publications, retractions, and preprints -- exhibits strikingly inertial growth, well-described by exponential, quadratic, and logistic models with nearly indistinguishable goodness-of-fit. Retraction dynamics, in particular, remain stable and minimally affected by the COVID-19 shock, which contributed less than 1% to total notices. Since 2004, publications doubled every 9.8 years, retractions every 11.4 years, and preprints at the fastest rate, every 5.6 years. The findings underscore a system primed for ongoing stress at unchanged structural bottlenecks. Although model forecasts diverge beyond 2024, the evidence suggests that the future trajectory of scholarly communication will be determined by persistent systemic inertia rather than episodic disruptions -- unless intentionally redirected by policy or AI-driven reform.
WikiKV: Schema-Evolving Path-Indexed Storage for Hierarchical Knowledge Navigation
LLM-curated hierarchical knowledge bases, namely a tree-structured wiki whose nodes summarize an underlying corpus, have become a dominant substrate for retrieval-augmented applications, yet their storage layer is still treated as an implementation detail. This workload is hierarchical, query-intensive, and continuously evolving, and no existing storage model natively captures all three properties at once. We present WikiKV, a path-indexed key-value storage model purpose-built for this workload, comprising three components: (i) a data-driven schema that bootstraps the hierarchy via Intent-Anchored Schema Induction and refines it through Continuous Evolution Operators; (ii) a consistency protocol for the path-indexed storage model that precludes partial-read observations under concurrent offline rewrites without read-path locking; and (iii) a budgeted navigation operator whose search-accelerated routing reduces the expected number of LLM-assisted descent steps from d to O(1) while preserving anytime semantics with progressively refined answers. We evaluate WikiKV through real-world deployment for the WeChat Official Account AI Assistant and benchmark it against diverse baselines on the AuthTrace dataset, where it achieves balanced low per-operator latency across four query operators against relational, graph, and FS backends, and reaches 63.2% end-to-end answer correctness, exceeding multiple RAG baselines, with the gap widening on low- and high-fan-in multi-document questions. Ablation study further confirms the effectiveness of WikiKV's components.
The thing that I find missing from most new blogging software, which includes even not-so-new ones like Ghost, is decent Archives navigation. I want to see per-year, per-month, per-tag, per-category nav. I came up in Wordpress and it is the only thing WP still does better than everyone by default.
Excited about this article fetcher. That could be a way for @semble.so to efficiently store full copies of its saved articles. You guys should talk :)
GitHub - stormlightlabs/lectito: web page/article fetcher inspired by readability
github.comHow long does a web page live 🤔 >5yrs ago I asked the internet, and made some new friends: x.com/dietrich/status/1135778377766… After a multi-year research collab with @archive.org, ODU and Filecoin Fdn... (largest study like this ever done) ... we have results! tldr: It's bad. fil.org/blog/the-web-isn-t-forever-ne…
