Search Results - RepositoryStats

news-please fhamborg

412

2.0k

apache-2.0

52

news-please - an integrated web crawler and information extractor for news that just works

Created 2016-12-18

736 commits to master branch, last one 3 days ago

cc-pyspark commoncrawl

85

393

mit

22

Process Common Crawl data with Python and Spark

wet spark pyspark sparksql wat-files warc-files commoncrawl common-crawl

Created 2017-04-12

104 commits to main branch, last one 2 months ago

news-crawl commoncrawl

34

309

apache-2.0

32

News crawling with StormCrawler - stores content as WARC

news warc crawler commoncrawl web-crawler apache-storm common-crawl storm-crawler

Created 2016-07-18

159 commits to master branch, last one 6 months ago

comcrawl michaelharms

38

220

mit

6

A python utility for downloading Common Crawl data

data python scraping commoncrawl common-crawl deep-learning training-dataset

Created 2020-01-09

122 commits to master branch, last one 3 years ago

cdx_toolkit cocrawler

29

155

apache-2.0

12

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

cdx warc python cdx-api commoncrawl web-archives web-archiving

Created 2018-03-03

243 commits to main branch, last one 2 years ago

ungoliant oscar-project

14

154

apache-2.0

2

:spider: The pipeline for the OSCAR corpus

nlp oscar crawler fasttext commoncrawl common-crawl corpus-linguistics language-classification

Created 2021-02-15

419 commits to main branch, last one 7 months ago

fundus flairNLP

64

142

mit

6

A very simple news crawler with a funny name

nlp rss corpus python cc-news crawler scraper sitemap web-corpus commoncrawl news-crawler web-scraping news-scraping text-extraction

Created 2022-10-28

1,986 commits to master branch, last one a day ago

gogetcrawl karust

16

137

mit

5

Extract web archive data using Wayback Machine and Common Crawl

golang crawler webarchive commoncrawl concurrency wayback-machine

Created 2019-06-14

28 commits to master branch, last one about a year ago

cc-crawl-statistics commoncrawl

9

124

apache-2.0

17

Statistics of Common Crawl monthly archives mined from URL index files

statistics commoncrawl common-crawl

Created 2016-07-14

209 commits to master branch, last one 23 hours ago

c4-dataset-script shjwudp

12

111

mit

5

Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.

nlp spark python dataset commoncrawl massivetext

Created 2022-05-27

26 commits to master branch, last one about a year ago

cc-index-table commoncrawl

9

99

apache-2.0

13

Index Common Crawl archives in tabular format

sql spark aws-athena commoncrawl apache-parquet columnar-storage

Created 2017-11-09

89 commits to main branch, last one 9 months ago

cc-webgraph commoncrawl

4

74

apache-2.0

11

Tools to construct and process webgraphs from Common Crawl data

pagerank webgraph commoncrawl common-crawl webgraph-framework centrality-measures

Created 2017-11-12

97 commits to main branch, last one 23 hours ago

CommonCrawlDocumentDownload centic9

20

59

bsd-2-clause

13

A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

java warc cdx-files mime-types commoncrawl

Created 2015-04-22

286 commits to master branch, last one 12 days ago

KeywordAnalysis CI-Research

13

57

unknown

6

Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends

wordcount commoncrawl cluster-analysis keyword-extraction

Created 2017-03-27

139 commits to master branch, last one 5 months ago