FAQ
Hi list,


I'm working on an app in which longish text chunks (could be up to a few MB in size, and stored either in flat text files or in fields of database records - TBD) need to be searched for the presence of a combination of string constants, where the string constants can be combined with boolean operators (in the user input given in a web form, for the search). The DB is MongoDB, accessed from Python using the PyMongo driver / client.


The text chunks are associated with structured fields of database records, of users (say, suppliers), so the end result wanted is that the app user should be able to specify a search using boolean operators and the app should search across all the text chunks belonging to all users (users to text chunks is a one-to-many relationship, whether the chunks are stored as long varchar in the DB or as flat files, with a field of a database record containing the filename of the text file containing the chunk). Then the app should return both the user names and the corresponding text chunks, for chunks that matched the boolean expression.


Example return value:


user2
    filename1.txt
    filename4.txt
user5
    filename3.txt
user7
    filename6.txt
(where the filenames.txt could instead be the chunks of text, if the chunks are stored in the DB instead of the file system).


The boolean expressions can be like these:


apple and orange
apple or pear
apple and (pear or mango)
(apple and pear) or mango
not orange
apple or not (pear and orange)
etc.


What are some good approaches to doing this using Python?


I'm first looking at pure Python code approaches, though have considered using search tools like Solr (via PySolr), ElasticSearch, etc. Right now the app is in the prototype stage so I first want to establish the feasibility and a simple approach to doing the search, though later, if the app needs to scale, may investigate the tools like PySolr or ES.


I think I may need to use one of the parsing libraries for Python such as PLY and others mentioned here:


https://wiki.python.org/moin/LanguageParsing


Let's say a parse tree is built up by whichever approach is used. A related question would be how to use this parse tree to search in the text chunks.


I realize that it is a somewhat complex questiom, so am not looking for complete solutions, but for suggestions on how to go about it.


Thanks for any suggestions.
--
Vasudev Ram
www.dancingbison.com
jugad2.blogspot.com

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
grouppython-list @
categoriespython
postedOct 25, '13 at 10:33p
activeOct 25, '13 at 10:33p
posts1
users1
websitepython.org

1 user in discussion

Vasudev Ram: 1 post

People

Translate

site design / logo © 2022 Grokbase