Purpose

The goal of this project is to build a distributed, decentralized crawler to scrape 1 billion web pages using a couple hundred dollars of commodity AWS hardware.

Name		Name	Last commit message	Last commit date
.terraform/plugins/darwin_amd64		.terraform/plugins/darwin_amd64
env		env
src		src
statistics		statistics
tests		tests
.eslintrc.json		.eslintrc.json
Dockerfile		Dockerfile
Pipfile		Pipfile
Pipfile.lock		Pipfile.lock
README.md		README.md
crawler.key		crawler.key
example.js		example.js
manageInstances.py		manageInstances.py
notes.md		notes.md
package.json		package.json
prepare-alexa-top-million.js		prepare-alexa-top-million.js
seed.json		seed.json
test.js		test.js
yarn.lock		yarn.lock

Navigation Menu

Repository navigation

About

Repository files navigation

Purpose

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Purpose

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages