Global InformationScrapy information
 | |
| Developer(s) | Zyte (formerly Scrapinghub) |
|---|---|
| Initial release | 26 June 2008 |
| Stable release | 2.11.1[1] 
/ 14 February 2024 |
| Repository |
|
| Written in | Python |
| Operating system | Windows, macOS, Linux |
| Type | Web crawler |
| License | BSD License |
| Website | scrapy |
Scrapy (/ˈskreɪpaɪ/[2] SKRAY-peye) is a free and open-source web-crawling framework written in Python. Originally designed for web scraping, it can also be used to extract data using APIs or as a general-purpose web crawler.[3] It is currently maintained by Zyte (formerly Scrapinghub), a web-scraping development and services company.
Scrapy project architecture is built around "spiders", which are self-contained crawlers that are given a set of instructions. Following the spirit of other don't repeat yourself frameworks, such as Django,[4] it makes it easier to build and scale large crawling projects by allowing developers to reuse their code.
Some well-known companies and products using Scrapy are: Lyst,[5][6] Parse.ly,[7] Sayone Technologies,[8] Sciences Po Medialab,[9] Data.gov.uk’s World Government Data site.[10]
- ^ "Release 2.11.1". 14 February 2024. Retrieved 20 February 2024.
- ^ Commit 975f150
- ^ Scrapy at a glance.
- ^ "Frequently Asked Questions". Frequently Asked Questions, Scrapy 2.8.0 documentation. Retrieved 28 July 2015.
- ^ Bell, Eddie; Heusser, Jonathan. "Scalable Scraping Using Machine Learning". Archived from the original on 4 June 2016. Retrieved 28 July 2015.
- ^ Scrapy | Companies using Scrapy
- ^ Montalenti, Andrew (October 27, 2012). "Web Crawling & Metadata Extraction in Python". Web Crawling & Metadata Extraction in Python - Speaker Deck. Retrieved May 11, 2015.
- ^ "Scrapy Companies". Scrapy | Companies using Scrapy.
- ^ Hyphe v0.0.0: the first release of our new webcrawler is out!
- ^ Ben Firshman [@bfirsh] (November 4, 2010). "World Govt Data site uses Django, Solr, Haystack, Scrapy and other exciting buzzwords http://bit.ly/5jU3La #opendata #datastore" (Tweet) – via Twitter.