Exploratory Analysis of a Terabyte Scale Web Corpus |
|---|
| Vasilis Kolias, Ioannis Anagnostopoulos, Eleftherios Kayafas |
- Abstract:
- In this paper we present a preliminary analysis over the largest publicly accessible web dataset: the Common Crawl Corpus. We measure nine web characteristics from two levels of granularity using MapReduce and we comment on the initial observations over a fraction of it. To the best of our knowledge two of the characteristics, the language distribution and the HTML version of pages have not been analyzed in previous work, while the specific dataset has been only analyzed on page level.
- Keywords:
- web characterization, web measurement, big data, mapreduce
- Download:
- IMEKO-TC4-2014-291.pdf
- DOI:
- -
- Event details
- IMEKO TC:
- TC4
- Event name:
- TC4 Symposium 2014
- Title:
20th IMEKO TC4 Symposium on Measurements of Electrical Quantities (together with 18th TC4 International Workshop on ADC and DCA Modeling and Testing, IWADC)
"Research on Electrical and Electronic Measurement for the Economic Upturn"- Place:
- Benevento, ITALY
- Time:
- 15 September 2014 - 17 September 2014