MediawikiToJsonCLI converts a Wikipedia Dump in Json.
MediawikiToJsonCLI wikipedia-dump.xml.bz -output wikipedia-dump.json[.gz] -lang [en|it]
produces in wikipedia-dump.json the JSON version of the dump. Each line of the file contains an article
of dump encoded in JSON. Each JSON line can be deserialized in an Article object, which represents an
enriched version of the wikitext page. The Article object contains:
- the title (e.g., Leonardo Da Vinci);
- the wikititle (used in Wikipedia as key, e.g., Leonardo_Da_Vinci);
- the namespace and the integer namespace in the dump;
- the timestamp of the article;
- the type, if it is a standard article, a redirection, a category and so on;
- if it is not in English the title of the corrispondent English Article;
- a list of tables that appear in the article ;
- a list of lists that that appear in the article ;
- a list of internal links that appear in the article;
- if the article is a redirect, the pointed article;
- a list of section titles in the article;
- the text of the article, divided in paragraphs;
- the categories and the templates of the articles;
- the list of attributes found in the templates;
- a list of terms highlighted in the article;
- if present the infobox.
Once you have created (or downloaded) the JSON dump (say
wikipedia.json
), you can iterate over the articles of the collection
easily using this snippet:
RecordReader<Article> reader = new RecordReader<Article>(
"wikipedia.json",new JsonRecordParser<Article>(Article.class)
).filter(TypeFilter.STD_FILTER);
for (Article a : reader) {
// do what you want with your articles
}
You can also add some filters in order to iterate on only certain articles (in the example
we used only the standard type filter, which excludes meta pages e.g., Portal: or User: pages.