• View
• Edit
• History
• Print

Data matrix

see also AImatrix

Principle

Collect the key repositories of easy to interesting, accurate, large and easy exploit datasets.

PS : individual specialized dataset should not be listed here, only repositories.

Encompassing structure

• voiD (from "Vocabulary of Interlinked Datasets") from W3C

Matrix

Name	Specificity	Institution	Size	Last update
CKAN	Handling datasets as packages and having its own "client" datapkg*	Open Knowledge Foundation (irc)	860 registered packages	live
The Map of Data by Sindice	?	DERI	?	?
Linked Data Sets as RDF Dumps	from the ESW Wik	W3C	?	?
Public Datasets	Dedicated to EC2 usage	Amazon AWS	?	?
Free Redistributable Rich Data Sets	not always easy to use because of scarce data or "old" formats	InfoChimps.org	?	?
data sets	"for people with large data sets"	theinfo.org	?	?
datasets from ManyEyes	?	IBM	?	?
Data.gov	restricted to the USA, no mention of DARPA data (as of June 2009)	US government	?	?
data.gov.uk	restricted to the UK	?	?	?
Twine	?	Radar Networks	?	?
Freebase	?	Metaweb	?	?
Worldometers	?	?	?	?
Community Open Data Tables	prepared for Yahoo's YQL	Yahoo/community-driven	?	?
Datasets from Programming For Peace	Datasets specialized in political conflicts	Multiple research groups	<10	?
NYTimes Linked Open Data	SKOS File + API	New York Times	?	live
Concept Web	relying on wiki structure	WikiProfessional	1 Million	?
LinkedData.org	provides a (non-working...) RSS feed	administered by Tom Heath	?	?
ScraperWiki	wiki with scapers to configure	?	?	?

||Name||Specificity||Institution||Size||Last update|| * Systems installed on Seedea.

Visualizers/Explorers

Most (if not all?) are linked to one specific set of well formatted datasets, be sure to check if the data your want are there of easy to convert first.

• parallax for FreeBase
• Vispedia : Interactive Visual Exploration of Wikipedia Data via Search-Based Integration
• ManyEyes by IBM
  • used by NYT Visualization Lab
• ReSIST RKB Explorer, provided by the ReSIST Project
  • its data sources
• n² Platform API reference for the web services exposed from the Talis Platform at api.talis.com
• Potentially Google with Adding search power to public data, Official Google Blog, April 2009
• political/economical data explorers
  • OECD eXplorer
  • Subsidyscope.com by The Pew Charitable Trusts
  • Socrata Making Data Social
  • Datamob Public data put to good use
    • highlights the connection between public data sources and the interfaces people are building for them.
• Tableau Public explore, create, share
• Google Public Data Explorer with less than 20 datasets as of early 2010

References

• Tim Berners-Lee speaks on Linked Data
  • This slide set was presented at the TED 2009 conference, "The Great Unveiling" in Long Beach, CA. USA, 4, Feb 2009
• New AWS Public Data Sets - Anthrokids, Twilio/Wigle.net, Sparse Matrices, USA Spending, Tiger, Amazon Web Services Blog, April 2009
• The DataPortability Project
• The dangers of Webcrawled datasets by Graeme Bell, First Monday Volume 15 Number 2 February 2010

To do

• Consider tablefy or Google Squared (q=dataset) to organize the comparison
• use W3C checklink to periodically insure that links are valid
  • point to archive.org mirror for the dead links
    • see also Google cache
• consider web services (nearly functional approach, code is data, the opposite is not necessarily true though)
  • cf Services#Sources
• explore http://www.biotorrents.net/ first discovered through freenode #hplusroadmap then http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0010071
• The World Bank Data including over 2000 indicators
• handle specific datasets properly
  • Barcodepedia.com community based online barcode database.
• Get the Data Open Data Q&A Forum
• Public SPARQL Endpoints availability
  • from CKAN list
  • with its SPARQL endpoint
• http://www.quora.com/Data/Where-can-I-get-large-datasets-open-to-the-public
• pskomoroch's http://delicious.com/pskomoroch/dataset and http://delicious.com/pskomoroch/redistributable+dataset