Sources of big data

Big data communities and platforms

1. http://archive.ics.uci.edu/ml/index.php
2. Kaggle
3. Statista
4. Data World
5. Data Hub
6. AWS open data
7. Google public data

Crime

1. https://www.fbi.gov/services/cjis/ucr
2. https://www.pcr.uu.se/research/UCDP/
3. https://www.drugabuse.gov/drug-topics/trends-statistics

Internet

1. https://wiki.dbpedia.org/
2. https://trends.google.com/trends/explore
3. Reddit datasets

Government

1. https://www.ukdataservice.ac.uk/
2. https://data.gov.uk/
3. https://data.london.gov.uk/
4. https://www.data.gov/
5. https://opengovernmentdata.org/data/
6. https://www.cia.gov/library/publications/the-world-factbook/
7. https://data.gov.au/
8. https://opendata.cityofnewyork.us/
9. https://open.canada.ca/en/open-data

House prices

1. UK house prices

Mathematics

• I got 717 problems... Project Euler problems

Health

1. https://healthdata.gov/
2. https://digital.nhs.uk/data-and-information/data-collections-and-data-sets
3. https://www.who.int/data/collections
4. https://www.who.int/gho/maternal_health/reproductive_health/en/
5. http://portals.broadinstitute.org/cgi-bin/cancer/datasets.cgi
6. https://www.cdc.gov/datastatistics/
7. https://www.fda.gov/drugs/drug-approvals-and-databases/drugsfda-data-files
8. https://github.com/publichealthengland/coronavirus-dashboard
9. 1000 Genomes project
10. USDA food composition

Business

1. https://www.glassdoor.com/research/type/data-sets/
2. https://opencorporates.com/

Transport

1. https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page

Climate

1. Africa climate
2. https://openaq.org/
3. Historic weather
4. NOAA tides and currents data
5. Global temperatures
6. Nasa Earth data
7. Reeep

Nature

1. NYC squirrel census - see also squirrel attacks

Travel

1. https://www.ustravel.org/research

Current affairs

1. Five Thirty Eight

Entertainment

1. BFI industry data insights
2. Spotify developer API

Sport and gambling

1. Historical Football Results and Betting Odds Data

Social media

1. Facebook API
2. Instagram API
3. YouTube API

Finance

1. Google Finance
2. Cryptocompare API
3. XE API (free trial)
4. https://datahub.io/collections/stock-market-data
5. https://data.imf.org/?sk=388dfa60-1d26-4ade-b505-a05a558d9a42
6. https://atlas.cid.harvard.edu/
7. World Bank Open Data
8. Financial Times markets data

To sort

1. https://www.gapminder.org/data/
2. http://storage.googleapis.com/books/ngrams/books/datasetsv2.html
3. https://github.com/rfordatascience/tidytuesday

Tools

• https://oxylabs.io
• https://scrapinghub.com
• https://import.io
• https://webscraper.io

References

• https://www.kdnuggets.com/2017/12/big-data-free-sources.html
• https://www.springboard.com/blog/free-public-data-sets-data-science-project/
• https://www.dataquest.io/blog/free-datasets-for-projects/
• https://www.forbes.com/sites/bernardmarr/2016/02/12/big-data-35-brilliant-and-free-data-sources-for-2016/#4166564fb54d
• https://www.tableau.com/en-gb/learn/articles/free-public-data-sets
• https://piktochart.com/blog/100-data-sets/
• https://github.com/awesomedata/awesome-public-datasets
• https://mode.com/analytics-dispatch/interesting-data-sets/
• https://www.dataquest.io/blog/free-datasets-for-projects/
• https://github.com/BuzzFeedNews/nics-firearm-background-checks
• https://www.springboard.com/blog/free-public-data-sets-data-science-project/
• https://www.kdnuggets.com/2017/12/big-data-free-sources.html
• https://github.com/awesomedata/awesome-public-datasets
• https://github.com/sindresorhus/awesome