Web scraping is just became easy. Extract any website content and turn it into datasets. No programming skills required.
ParseHub | Free web scraping - The most powerful web scraper
Scraping Framework for Golang
Scraping framework for extracting the data you need from websites, used for a wide range of applications, like data mining, data processing or archiving
WDC - RDFa, Microdata, and Microformat Data Sets
More and more websites have started to embed structured data describing products, people, organizations, places, and events into their HTML pages using markup standards such as Microdata, JSON-LD, RDFa, and Microformats. The Web Data Commons project extracts this data from several billion web pages. So far the project provides 12 different data set releases extracted from the Common Crawls 2010 to 2023. The project provides the extracted data for download and publishes statistics about the deployment of the different formats.
firecrawl/firecrawl
The context API to search, scrape, and interact with the web at scale. 🔥
How to Scrape Web Content for RAG with Readability.js
Common Crawl - Open Repository of Web Crawl Data
We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.
Advanced web scraping with Mechanize
The personal website of Chris Mytton: programming, life, nature and, lately, maps and the landscape.
Interact — Scrape a page, then take actions on it with AI prompts or code | Firecrawl
Scrape any page, then click, type, fill forms, and extract data with natural language or Playwright code. Live view, persistent profiles, session chaining.
A Web of Data
Thoughts on making a "Web of Data" instead of a "Web of Pages", and how that might let us take a step away from the dominance of large, complicated browsers.
All The Places
A growing set of web scrapers designed to output consistent geodata about as many places of business in the world as possible.
Aggressive AI scrapers are making it kinda suck to run wikis
Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
Launching the Wolfram Data Repository: Data Publishing that Really Works
Launching the Wolfram Data Repository: a library of datasets for immediate computation, visualization & analysis; a platform for publishing your own. Machine-learning training sets, book texts, networks, 3D models, geospatial data...
I Don’t Need No Stinking API: Web Scraping For Fun and Profit | Hacker News
I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an update on my phone when it dawned on me that the mobile.twitter site is, for the most part, simple html stuff. Once I realized that, I was able to programatically log into my account with no problems, and the rest of twitter was unlocked for me. I could scrape and post to (almost) my heart content.
SCRAPING AND CLEANING YOUR DATA WITH GOOGLE SHEETS: A CLOSER LOOK
GOOGLE GUARANTEED - Ever found yourself copy and pasting data, wishing there was an easier way? Take a foray into the world of data scraping using Google Sheets with Jeremy Gottlieb!
Websets | People Search, AI SDR & B2B Lead Generation
Use Websets for people search, AI SDR discovery, lead generation, and B2B marketing. Explore sales automation tools and AI research tools in one place.