Search Results - RepositoryStats

77

2.2k

apache-2.0

25

Up to 10x faster strings for C, C++, Python, Rust, and Swift, leveraging NEON, AVX2, AVX-512, and SWAR to accelerate search, sort, edit distances, alignment scores, etc 🦖

csv html json simd laion ndjson parser string dataset substring common-crawl beautifulsoup string-search string-parsing string-matching sorting-algorithms pattern-recognition string-manipulation information-retrieval

Created 2020-08-14

836 commits to main branch, last one 10 hours ago

cc-pyspark commoncrawl

86

405

mit

23

Process Common Crawl data with Python and Spark

wet spark pyspark sparksql wat-files warc-files commoncrawl common-crawl

Created 2017-04-12

104 commits to main branch, last one 7 months ago

news-crawl commoncrawl

35

321

apache-2.0

33

News crawling with StormCrawler - stores content as WARC

news warc crawler commoncrawl web-crawler apache-storm common-crawl storm-crawler

Created 2016-07-18

159 commits to master branch, last one 10 months ago

comcrawl michaelharms

41

225

mit

6

A python utility for downloading Common Crawl data

data python scraping commoncrawl common-crawl deep-learning training-dataset

This repository has been archived (exclude archived)

Created 2020-01-09

122 commits to master branch, last one 4 years ago

ungoliant oscar-project

14

162

apache-2.0

2

:spider: The pipeline for the OSCAR corpus

nlp oscar crawler fasttext commoncrawl common-crawl corpus-linguistics language-classification

Created 2021-02-15

419 commits to main branch, last one 12 months ago

cc-crawl-statistics commoncrawl

11

151

apache-2.0

18

Statistics of Common Crawl monthly archives mined from URL index files

statistics commoncrawl common-crawl

Created 2016-07-14

222 commits to master branch, last one 8 days ago

troll-a crissyfield

11

134

apache-2.0

6

Drill into WARC web archives

warc security common-crawl security-tools internet-archive command-line-tool

Created 2023-12-07

78 commits to main branch, last one 21 days ago

goclassy oscar-project

6

86

apache-2.0

9

An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.

nlp fasttext common-crawl corpus-linguistics language-classification

This repository has been archived (exclude archived)

Created 2019-03-01

17 commits to master branch, last one 3 years ago

cc-webgraph commoncrawl

4

79

apache-2.0

12

Tools to construct and process webgraphs from Common Crawl data

pagerank webgraph commoncrawl common-crawl webgraph-framework centrality-measures

Created 2017-11-12

103 commits to main branch, last one 16 days ago

cc-notebooks commoncrawl

9

46

apache-2.0

18

Various Jupyter notebooks about Common Crawl data

aws-athena commoncrawl common-crawl webarchiving jupyter-notebook webgraph-framework

Created 2019-07-19

23 commits to main branch, last one 2 years ago