Diggernaut - Turn website content into datasets
Web scraping is just became easy. Extract any website content and turn it into datasets. No programming skills required.
I Don’t Need No Stinking API: Web Scraping For Fun and Profit | Hacker News
I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an update on my phone when it dawned on me that the mobile.twitter site is, for the most part, simple html stuff. Once I realized that, I was able to programatically log into my account with no problems, and the rest of twitter was unlocked for me. I could scrape and post to (almost) my heart content.
How to Scrape Web Content for RAG with Readability.js
firecrawl/firecrawl
The context API to search, scrape, and interact with the web at scale. 🔥
Scraping Framework for Golang
Scraping framework for extracting the data you need from websites, used for a wide range of applications, like data mining, data processing or archiving
Advanced web scraping with Mechanize
The personal website of Chris Mytton: programming, life, nature and, lately, maps and the landscape.
Interact — Scrape a page, then take actions on it with AI prompts or code | Firecrawl
Scrape any page, then click, type, fill forms, and extract data with natural language or Playwright code. Live view, persistent profiles, session chaining.
Newspaper3k: Article scraping & curation — newspaper 0.0.2 documentation
Inspired by requests for its simplicity and powered by lxml for its speed:
Dweb: Social Feeds with Secure Scuttlebutt – Mozilla Hacks - the Web developer blog
Scuttlebutt is a free and open source social network with unique offline-first and peer-to-peer properties. Mainstream closed platforms have become a more popular way of creating and consuming content than ...
Common Crawl - Open Repository of Web Crawl Data
We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.
All The Places
A growing set of web scrapers designed to output consistent geodata about as many places of business in the world as possible.
zio.sh/didweb.watch
🦋🤖 Bluesky bot to track did:web usage — https://bsky.app/profile/didweb.watch
Aggressive AI scrapers are making it kinda suck to run wikis
Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
Keynote: The Death of the Browser - Rachel-Lee Nabors, AgentQL
alan-turing-institute/ReadabiliPy
A simple HTML content extractor in Python. Can be run as a wrapper for Mozilla's Readability.js package or in pure-python mode.