Skip to content
@commoncrawl

Common Crawl Foundation

Common Crawl provides an archive of webpages going back to 2007.

Pinned Loading

  1. cc-pyspark cc-pyspark Public

    Process Common Crawl data with Python and Spark

    Python 460 97

  2. cc-crawl-statistics cc-crawl-statistics Public

    Statistics of Common Crawl monthly archives mined from URL index files

    Python 231 18

  3. cc-index-table cc-index-table Public

    Index Common Crawl archives in tabular format

    Java 132 17

  4. cc-warc-examples cc-warc-examples Public

    Forked from Smerity/cc-warc-examples

    CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

    Java 38 17

  5. cc-citations cc-citations Public

    Scientific articles using or citing Common Crawl data

    Jupyter Notebook 31 4

  6. cc-notebooks cc-notebooks Public

    Various Jupyter notebooks about Common Crawl data

    Jupyter Notebook 69 11

Repositories

Showing 10 of 91 repositories
  • crawler-commons Public Forked from crawler-commons/crawler-commons

    A set of reusable Java components that implement functionality common to any web crawler

    commoncrawl/crawler-commons's past year of commit activity
    Java 2 Apache-2.0 95 0 2 Updated Oct 2, 2026
  • cc-webgraph Public

    Tools to construct and process Common Crawl webgraphs

    commoncrawl/cc-webgraph's past year of commit activity
    Java 114 Apache-2.0 7 7 (1 issue needs help) 7 Updated Oct 1, 2026
  • cc-index-table Public

    Index Common Crawl archives in tabular format

    commoncrawl/cc-index-table's past year of commit activity
    Java 132 Apache-2.0 17 5 8 Updated Oct 1, 2026
  • cc-warc-examples Public Forked from Smerity/cc-warc-examples

    CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

    commoncrawl/cc-warc-examples's past year of commit activity
    Java 38 MIT 45 0 0 Updated Oct 1, 2026
  • ia-hadoop-tools Public Forked from Aloisius/ia-hadoop-tools

    Web archiving tools on Hadoop

    commoncrawl/ia-hadoop-tools's past year of commit activity
    Java 4 29 2 1 Updated Oct 1, 2026
  • ia-web-commons Public Forked from Aloisius/ia-web-commons

    Web archiving utility library

    commoncrawl/ia-web-commons's past year of commit activity
    Java 12 Apache-2.0 81 4 1 Updated Oct 1, 2026
  • cc-crawl-statistics Public

    Statistics of Common Crawl monthly archives mined from URL index files

    commoncrawl/cc-crawl-statistics's past year of commit activity
    Python 231 Apache-2.0 18 2 7 Updated Oct 1, 2026
  • news-crawl Public

    News crawling with StormCrawler - stores content as WARC

    commoncrawl/news-crawl's past year of commit activity
    Java 379 Apache-2.0 42 12 2 Updated Sep 30, 2026
  • commonlid-eval Public

    Benchmark for language identification by the CommonLID project

    commoncrawl/commonlid-eval's past year of commit activity
    Python 3 Apache-2.0 2 4 1 Updated Sep 23, 2026
  • cc-webgraph-statistics Public

    Statistics of Common Crawl monthly Web Graphs

    commoncrawl/cc-webgraph-statistics's past year of commit activity
    JavaScript 8 Apache-2.0 1 0 0 Updated Sep 22, 2026
View all repositories