Search Results - RepositoryStats

63

1.9k

apache-2.0

19

Up to 10x faster strings for C, C++, Python, Rust, and Swift, leveraging SWAR and SIMD on Arm Neon and x86 AVX2 & AVX-512-capable chips to accelerate search, sort, edit distances, alignment scores, et...

csv html json simd laion ndjson parser string dataset substring common-crawl beautifulsoup string-search string-parsing string-matching sorting-algorithms pattern-recognition string-manipulation information-retrieval

Created 2020-08-14

716 commits to main branch, last one 2 months ago

cc-pyspark commoncrawl

85

393

mit

22

Process Common Crawl data with Python and Spark

wet spark pyspark sparksql wat-files warc-files commoncrawl common-crawl

Created 2017-04-12

104 commits to main branch, last one 2 months ago

news-crawl commoncrawl

34

309

apache-2.0

32

News crawling with StormCrawler - stores content as WARC

news warc crawler commoncrawl web-crawler apache-storm common-crawl storm-crawler

Created 2016-07-18

159 commits to master branch, last one 6 months ago

comcrawl michaelharms

38

220

mit

6

A python utility for downloading Common Crawl data

data python scraping commoncrawl common-crawl deep-learning training-dataset

Created 2020-01-09

122 commits to master branch, last one 3 years ago

ungoliant oscar-project

14

154

apache-2.0

2

:spider: The pipeline for the OSCAR corpus

nlp oscar crawler fasttext commoncrawl common-crawl corpus-linguistics language-classification

Created 2021-02-15

419 commits to main branch, last one 7 months ago

troll-a crissyfield

12

131

apache-2.0

5

Drill into WARC web archives

warc security common-crawl security-tools internet-archive command-line-tool

Created 2023-12-07

69 commits to main branch, last one 5 months ago

cc-crawl-statistics commoncrawl

9

124

apache-2.0

17

Statistics of Common Crawl monthly archives mined from URL index files

statistics commoncrawl common-crawl

Created 2016-07-14

209 commits to master branch, last one 22 hours ago

goclassy oscar-project

6

85

apache-2.0

9

An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.

nlp fasttext common-crawl corpus-linguistics language-classification

This repository has been archived (exclude archived)

Created 2019-03-01

17 commits to master branch, last one 3 years ago

cc-webgraph commoncrawl

4

74

apache-2.0

11

Tools to construct and process webgraphs from Common Crawl data

pagerank webgraph commoncrawl common-crawl webgraph-framework centrality-measures

Created 2017-11-12

97 commits to main branch, last one 22 hours ago