it.cnr.isti.hpc.dexter.lucene

Class LuceneHelper

java.lang.Object

it.cnr.isti.hpc.dexter.lucene.LuceneHelper

public class LuceneHelper
extends Object

LuceneHelper provides utilities for indexing, retrieving, and ranking Wikipedia articles.

Author:

Diego Ceccarelli Created on Aug 27, 2013

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.lucene.document.FieldType	STORE_TERM_VECTORS
static org.apache.lucene.document.FieldType	STORE_TERM_VECTORS_NOT_STORED

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	LuceneHelper(File wikiIdtToLuceneIdSerialization, File indexPath) Opens or creates a lucene index in the given directory

Method Summary

Methods

Modifier and Type	Method and Description
void	addDocument(it.cnr.isti.hpc.wikipedia.article.Article a) Indexes a Wikipedia Article
protected void	addDocument(int id, String content) Adds a Wikipedia Article (added just for testing)
void	clearIndex() Clears the index
void	closeWriter()
void	commit()
protected void	dumpWikiIdToLuceneId() Dumps the map containing the conversion from the Wikipedia ids to the Lucene Ids.
it.cnr.isti.hpc.wikipedia.article.Article	getArticle(int id) Retrieves an article from the index
it.cnr.isti.hpc.wikipedia.article.Article	getArticleSummary(int id) Retrieves only the article summary and the title from the index
double	getCosineSimilarity(int x, int y) Returns the cosine similarity between two documents
double	getCosineSimilarity(int x, int y, String field) Returns the cosine similarity between two documents
static LuceneHelper	getDexterLuceneHelper() Returns an instance of the Dexter's Lucene index.
int	getFreq(String query)
int	getFreq(String query, String field)
protected int	getLuceneId(int wikiId)
float	getSimilarity(org.apache.lucene.search.Query query, int wikiId) Returns the TFIDF-similarity between a given string and an article
int	getWikiId(int luceneId)
static boolean	hasDexterLuceneIndex()
void	loadWikiIdToLuceneId() Loads the map containing the conversion from the Wikipedia ids to the Lucene Ids.
int	numDocs()
protected void	parseWikiIdToLuceneId() Loads the map containing the conversion from the Wikipedia ids to the Lucene Ids.
List<Integer>	query(String query)
List<Integer>	query(String query, String field)
void	rankBySimilarity(SpotMatch spot, EntityMatchList eml, String context) Sorts a list of entities by their similarity (full text) with the string context.
void	rankBySimilarity(SpotMatch spot, EntityMatchList eml, String context, String field) Sorts a list of entities by their similarity with the string context.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

STORE_TERM_VECTORS

public static final org.apache.lucene.document.FieldType STORE_TERM_VECTORS

STORE_TERM_VECTORS_NOT_STORED

public static final org.apache.lucene.document.FieldType STORE_TERM_VECTORS_NOT_STORED

Constructor Detail

LuceneHelper

protected LuceneHelper(File wikiIdtToLuceneIdSerialization,
            File indexPath)

Opens or creates a lucene index in the given directory

Parameters:

wikiIdtToLuceneIdSerialization - - the file containing the serialized mapping between wiki-id and Lucene documents ids

indexPath - - the path of the directory with the Lucene's index

Method Detail

hasDexterLuceneIndex

public static boolean hasDexterLuceneIndex()

Returns:

true if the dexter lucene index exists, false otherwise

getDexterLuceneHelper

public static LuceneHelper getDexterLuceneHelper()

Returns an instance of the Dexter's Lucene index.

Returns:

an instance of the Dexter's Lucene index

parseWikiIdToLuceneId

protected void parseWikiIdToLuceneId()

Loads the map containing the conversion from the Wikipedia ids to the Lucene Ids.

dumpWikiIdToLuceneId

protected void dumpWikiIdToLuceneId()

Dumps the map containing the conversion from the Wikipedia ids to the Lucene Ids.

loadWikiIdToLuceneId

public void loadWikiIdToLuceneId()

Loads the map containing the conversion from the Wikipedia ids to the Lucene Ids.

getLuceneId

protected int getLuceneId(int wikiId)

Returns:

the Lucene id of an article, given its wikiId

getSimilarity

public float getSimilarity(org.apache.lucene.search.Query query,
                  int wikiId)

Returns the TFIDF-similarity between a given string and an article

Parameters:

query - - the query containing the query to compare with the article

wikiId - - the id of the article to compare with the query

Returns:

the TFIDF-similarity between the query and wikiId

getCosineSimilarity

public double getCosineSimilarity(int x,
                         int y)

Returns the cosine similarity between two documents

Parameters:

x - - the WikiId of the first document

y - - the WikiId of the first document

Returns:

a double between 0 (not similar) and 1 (same content), representing the similarity between the 2 documents

getCosineSimilarity

public double getCosineSimilarity(int x,
                         int y,
                         String field)

Returns the cosine similarity between two documents

Parameters:

x - - the WikiId of the first document

y - - the WikiId of the first document

field - - the field on which to compute the similarity

Returns:

a double between 0 (not similar) and 1 (same content), representing the similarity between the 2 documents

addDocument

public void addDocument(it.cnr.isti.hpc.wikipedia.article.Article a)

Indexes a Wikipedia Article

Parameters:

a - the article to index

addDocument

protected void addDocument(int id,
               String content)

Adds a Wikipedia Article (added just for testing)

Parameters:

id - - the id of the Wikipedia Article

content - - the text of the Wikipedia Article

clearIndex

public void clearIndex()

Clears the index

commit

public void commit()

getFreq

public int getFreq(String query,
          String field)

Parameters:

query - - a query

field - - the field where to search the query

Returns:

number of documents containing the text in query in the given fields

getFreq

public int getFreq(String query)

Parameters:

query - - a query

field - - the field on which to perform the query

Returns:

number of documents containing the text in query in the given fields

numDocs

public int numDocs()

Returns:

the number of documents indexed

closeWriter

public void closeWriter()

query

public List<Integer> query(String query,
                  String field)

Returns:

the top Lucene Document ids matching the query

query

public List<Integer> query(String query)

Returns:

the top Lucene Document ids matching the query

getArticle

public it.cnr.isti.hpc.wikipedia.article.Article getArticle(int id)

Retrieves an article from the index

Parameters:

id - - the Wikipedia Id of the Article

Returns:

the document from the index

getArticleSummary

public it.cnr.isti.hpc.wikipedia.article.Article getArticleSummary(int id)

Retrieves only the article summary and the title from the index

Parameters:

id - - the Wikipedia Id of the Article

Returns:

the document from the index

getWikiId

public int getWikiId(int luceneId)

rankBySimilarity

public void rankBySimilarity(SpotMatch spot,
                    EntityMatchList eml,
                    String context,
                    String field)

Sorts a list of entities by their similarity with the string context.

Parameters:

spot - - the spot for which the entities are sorted

eml - - the entity list to sort

context - - the context text, entities are sorted based on their similarity with the context.

field - - sort the entity based on the similarity between their text in this field and the context.

rankBySimilarity

public void rankBySimilarity(SpotMatch spot,
                    EntityMatchList eml,
                    String context)

Sorts a list of entities by their similarity (full text) with the string context.

Parameters:

spot - - the spot for which the entities are sorted

eml - - the entity list to sort

context - - the context text, entities are sorted based on their similarity with the context.