Check out the Hacking page for more context on the code.
Late Breaking News/Known Problems
- Integration with twisted logging is poor.
- We are in the middle of moving from using the manually executed 'run-raindrop.py' script to the couchdb managed 'couch-raindrop.py' script which is controlled by the front end. If both processes are running you can expect one to see regular conflict errors.
Install couchdb, version 0.10 or later. 0.9 will not work. See:
- http://couchdb.apache.org/downloads.html for most platforms.
- http://janl.github.com/couchdbx/ for a MacOSX installer
- http://wiki.apache.org/couchdb/Windows_binary_installer for Windows.
When building couchdb from source, it can be run without installing via a 'dev' install:
cd couchdb; ./bootstrap && ./configure && make && make dev ./utils/run
- Install Python 2.5 or 2.6 (note that Python 3 is not yet supported).
- Install Mercurial (http://mercurial.selenic.com/)
- Checkout raindrop. For the 0.1 code that is shown on https://labs.mozilla.com/raindrop, use the 0.1 tag:
% hg clone -r 0.1 http://hg.mozilla.org/labs/raindrop
The trunk code is very much in flux, but if you prefer to use it, leave off the tag:
% hg clone http://hg.mozilla.org/labs/raindrop
- Execute the 'check-raindrop.py' script in the raindrop/server/python directory.
This should check all dependencies and let you know what you are missing, then check your couchdb for certain configuration requirements.
If you are brave, try executing 'check-raindrop.py --configure' (prefixing that command with 'sudo ' on some platforms). This will attempt to automatically install all dependencies necessary and configure couchdb.
If you are less brave, run it without the --configure option, then manually resolve any dependency or configuration issues it reports, then repeat until no problems are found.
A summary of the dependencies are: setuptools, twisted, PyOpenSSL, Skype4Py, python-twitter, simplejson, feedparser and paisley.
Notes for mac users:
- ERROR: module 'twisted>=8.2' has version conflict: (Twisted 2.4.0 (/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python), Requirement.parse('twisted>=8.2')) happens when trying to install twisted to the system python. Install Python 2.6 and make sure you're using it.
- doing easy_install PyOpenSSL after May 1st, may install some badly. It may be a PyOppenSSL 0.9 issue. 0.8 is known to work. Try the latest code, but if that does not work, switch to 0.8.
Twisted 0.9 (which is what you get if using easy_install or check-raindrop.py --configure) seems to cause problems with raindrop: .
After check-raindrop.py reports everything is in order, the next step is to set up the accounts to use with raindrop, then have raindrop fetch the messages
Raindrop uses a file called .raindrop to hold your account configurations. This file lives in your home directory.
Important: the user account that runs CouchDB will need to have read and write access to the .raindrop and .raindrop.temp file in your home directory for it to run.
If you installed CouchDB with the standard defaults and you just want to configure one Gmail and one Twitter account, you do not need to manually set up the .raindrop file, and you can skip this section. You will be prompted in the web UI to set up the Gmail and Twitter account via OAuth. However, the user account that runs CouchDB needs to have read and write permissions for the .raindrop and a .raindrop.temp file to work.
Manual .raindrop configuration
If you just want to These steps should become less necessary as time moves forward (it will be possible in the future to do this work though a web interface), but for now, you need to set up the accounts you want to use in raindrop via a .raindrop file:
configure raindrop by editing ~/.raindrop
if your 'local' couchdb isn't on localhost, add a section along the lines of:
[couch-local] host=hostname port=port
Add one new section per account; note the "account-" prefix is important when defining accounts:
- Add gmail accounts along the lines of
[account-gmail-username] proto=imap kind=gmail email@example.com password=topsecret ssl=True
- Add twitter accounts along the lines of
[account-twitter-username] proto=twitter kind=twitter username=username password=topsecret
- Add imap accounts along the lines of
[account-imap-username] proto=imap kind=imap host=imap.example.com port=993 username=username password=topsecret ssl=True
- Add skype accounts along the lines of
[account-skype-username] proto=skype kind=skype username=username password=topsecret
- Add RSS feed along the lines of
[account-rss-feed] proto=rss uri=http://example.org/rss.xml
Accounts name must be unique. A 'rss' account already exists, which fetch messages from getsatisfaction.com
If you did not manually configure the .raindrop file, then you can run this command in the raindrop/server/python directory to set up the web UI:
When that is done, go to this URL:
You will be prompted to set up a Gmail and Twitter account via OAuth. Finish the account setup flow. Once you are done you will have to do another command to start syncing messages:
Once you are done with the web account setup or manually configuring ~/.raindrop, then go to the command line, to the raindrop/server/python directory and run the following command:
% ./run-raindrop.py sync-messages --max-age=5days
It will take a few minutes for run-raindrop to complete syncing messages. The above command gets the last 5 days of messages for each account configured above. Once the above command finishes, then open your favorite Firefox or WebKit-based browser and go to:
You should see some of your messages displayed.
There are many issues with the UI and how messages are processed, but hopefully you now have a feel for how raindrop works.
The run-raindrop.py script is capable of fetching messages, re-running extensions, etc. It is expected that over time, this functionality will be rolled into the couchdb controlled couch-raindrop.py script. As a result, some functionality is duplicated in these scripts.
In particular, there is no protection against 'couch-raindrop.py' and run-raindrop.py attempting to perform the same thing at the same time, which will cause many errors to be logged. Before using the advanced features of run-raindrop, you should ensure couch-raindrop.py is not running by executing a command similar to:
% curl http://127.0.0.1:5984/raindrop/_raindrop/exit
When run-raindrop.py is executed, it always checks to see if the accounts and other documents required by raindrop (eg, the front-end applications) are up to date, then performs the requested operation, then terminates. If no operation is requested a web-browser pointing at raindrop will be opened. This means you can edit any of the files in the 'schema', 'client' or 'couch_doc' directories, then see those changes reflected in the front-end when run-raindrop is next started. We don't detect the changes while we are running though, only at startup.
Get messages from all accounts:
% ./run-raindrop.py sync-messages --max-age=2days --folder=inbox
Both --max-age and --folder are optional - see --help for details. Reload the front-end applications to see the messages as they arrive.
To delete all the intermediate messages in the DB, execute:
% ./run-raindrop.py unprocess
The next time you execute a 'process-backlog' command, all your messages will be reprocessed.
If there is an error during a 'process' operation (ie, an exception in the converter), we will write an 'error document' and continue processing the work queue. We don't make a very loud noise at the end when this happens - you need to notice it in the log output as it is running. At any time you can execute:
% ./run-raindrop.py retry-errors
To re-process those messages - but this will generally result in exactly the same error unless the converter has been fixed to deal with the condition causing the error. Alternatively, execute 'unprocess' to force reprocessing of all messages, not only those which previously caused an error.
Test everything using the test suite