SummerOfCode/2012/ISPDB

Introduction

This page was created to show the progress of my GSoC 2012 project which aims to get ISPDB into production. It will be available a list of pending features and bugs, weekly status updates, and other stuff related to the project.

Feel free to contribute and give me your opinions. I'll be available on irc.mozilla.org #maildev xsergio

Activities before coding period begins

• To make things easier for contributors, ispdb repo is now in GitHub: https://github.com/mozilla/ispdb
• The project has been updated to run on Django 1.4
• A list of bugs and pending features is being constructed

Mid-term evaluation

What has been done so far:

• Updated the project to run on Django 1.4
• Add BrowserID support
• Add an edit domain support #526467
• Add delete domain support
• Add issues report support with a merge feature.
• Improve comments support
• Small fixes, cleanups

What I plan to do until the end of the program:

• #543614: Automated Checks
• #591353: Allow to pin down some important configs of big ISPs
• #588136: Make ispdb cache-friendly
• #588155: Allow all values from config file XML to be put in the ISPDB database
• Internationalization and localization
• Improve UI (using Bootstrap and/or JQueryUI)

Weekly status updates

In this section I'll be writting updates status of the project. What I've been working on, next steps, etc.

27/04 - 04/05

During this week I've been working on the following items:

• Study the components used in the project, such as django, django-nose, django-openid-auth.
• Collect bugs and pending features
• Django 1.4 integration
• Discuss next steps

Much of my time was spent on Django 1.4 integration. Since the project is running with an old Django version (1.0 or 1.1), there were a few changes on Django API, project layout and project files.

You can find the pull request with these changes in https://github.com/mozilla/ispdb/pull/2

For the next week, I'll work on BrowserID integration, and edit domains feature (bug #526467). Also, if we find any error in Django 1.4 integration code, I'll work on them.

04/05 - 11/05

Since the beginning of this week, Django 1.4 integration code was merged into Mozilla ISPDB repo, so developers can work on the last stable version of Django.

Also, ISPDB is now using BrowserID for authentication. There is a django-browserid module available, which is full integrated to django.contrib.auth system. The code will be soon in upstream repo as well.

For the next week, I'll be working on edit domains feature.

11/05 - 18/05

First news: BrowserID integration code has been merged into Mozilla's ISPDB repo. Now you can clone Mozilla's ISPDB repo and test it: https://github.com/mozilla/ispdb

During this week I've been working on bug #526467 which aims to give support for editing configs. This bug required a lot of changes, including the concept of domain models. Althought it is not finished, some code was already pushed to my github. You can check it out on my edit_domain branch: https://github.com/sergiocharpineljr/ispdb/tree/edit_domain

18/05 - 25/05

Now, most of the work of bug #526467 is done. You can check out the code on https://github.com/sergiocharpineljr/ispdb/tree/edit_domain . Before I submit my pull request, I'll work on a feature that allows users to send changes requests for any domain they don't have permission to edit, which I expect to finish until the end of the next week.

Besides that, for the next week I'll be fixing the existing tests and adding now ones to check the edit domain code. Also I'll be working on the votes feature. We plan to extend it to permit users to vote on existing configurations as well.

25-05 - 01/06

During this week I've been working to allow users and admins delete their configurations. Also I've made some changes in admin site and in the models (UnclaimedDomain was removed and we are using DomainRequest instead). These changes and some fixes are available on my edit_domain branch. You can checkout out them here.

Next week I'll be working on delete configurations feature to remove the confirmation popup and add a UNDO button instead, which will be available for X hours. After that period the configuration can only be recovered through admin interface. Besides that I'll work on a problem report feature, which will allow users to report problems and optionally send an updated configuration. After that I'll take some time to review the code and make the pull request.

01/06 - 08/06

This week I'll be working on the same features that I'd worked last week, which are: delete domains and issue report. Most of the work is done but there is already some pending tasks. Until Friday, I hope I have a reviewed pull request with both features on it.

Next task I'll be working on is the comments. The project supports a very basic comment model, which is implemented by Django module. The idea is to extend it with comments edition, deletion, etc.

08/06 - 15/06

Finally, I've finish my edit_domain branch, merged to main master branch and opened the pull request. I worked on the following items:

• add support for domain deletion in config form
• add support for configuration edition
• add support for issues report (with merge option) for approved configurations
• add more tests

A few small bugs were also fixed.

For the next week, I'll work to improve comments support which are available through Django comments framework.

15/06 - 22/06

After a few fixes, my pull request was reviewed and merged into Mozilla's repo. Now I'm finishing some pending tasks on comments feature to make another pull request. You can find the code on my comments branch.

After I finish the comments branch, I'll start to work on Automated Checks.

22/06 - 06/07

During these two weeks I've been working on Automated Checks feature. Currently it's implemented the following checks:

• Domain Checks
  • Compare nameservers of the first domain against the others.
  • Check if domains has at least one MX server
• Config Checks
  • Check if there is a better connection method available for incoming (IMAP and POP3) and outgoing servers (SMTP).
  • Check if supplied connection method is supported
  • Check if supplied authentication method is supported

You can check out the code on my automated_checks branch.

06/07 - 13/07

Now automated checks feature is almost done. I've added checks for suggesting better authentication methods (if available) and for testing if the configured auth method is supported. Those informations are obtained from ehlo responses. These checks are now being displayed on the details page instead of open a new tab/window.

Now I'll work on checking tld of MX servers to compare with tld of configured incoming/outgoing servers. And I'll have to create tests for all of this code. I'll use pymox to mock dnspython, smptlib, poplib and imaplib.

After finishing automated checks feature, I'll start to work on Bug 588155 - Allow all values from config file XML to be put in the ISPDB database. I hope to finish Bug 526468 - Can't enter several documentation URLs until the end of this week.

13/07 - 20/07

Finally, I'm done with automated checks feature. You can check out the code here.

Now I'm working on Bug 526468 - Can't enter several documentation URLs and Bug 518319 - Allow documentation pages in more languages. It's getting a little tricky because we have nested forms with ManyToMany fields.

20/07 - 27/07

I haven't finished bugs Bug 526468 - Can't enter several documentation URLs and Bug 518319 - Allow documentation pages in more languages yet. The view code was getting too complex, so I decided to transfer this complexity to the form and baseformset classes.

Next week I'll finish these bugs, and work to get pull request 7 approved.

27/07 - 03/08

After simplifying the views I could finish 526468: Can't enter several documentation URLs. Also I have progressed on Bug 518319 - Allow documentation pages in more languages, but we don't have support for all (or most) of the languages yet because I'm trying to find a better solution to add forms dynamically so we won't need to build manually a repeated hidden form.

After that, I will work on 526469: No field for enableURL.

03/08 - 10/08

During this week I've been working on Bug 518319 - Allow documentation pages in more languages and Bug 526469 - No field for enableURL. Like I said last week, we had to find a better solution to add forms dynamically. I'll explain better the problem.

We are using hidden textarea elements which contains our "generic" forms. These generic forms have on their attributes JQuery Validation tags, in our case "{1}-{0}", which are replaced with form prefix and indexes (it's the way Django deals with formsets). These generic forms were rendered manually (using html tags) instead of letting django render it for us. So, if we wanted to add a new language choice, we would need to change the model and the template. The template was getting huge too.

To fix this I've tried to use django-dynamic-formset. But it doesn't support nested forms easily. So I've decided to use the django formset's attribute "empty_form". By default, it will create a form with a "__prefix__" string, which should be replaced by the form index. But our code is expecting a "{1}-{0}" string. So I found that creating a base BaseModelFormSet class and override _get_empty_form method to change the prefix would be our best option.

Now we are almost in the end of the program and I still have a few things to work on (hopefully most of them are small tasks). The list includes:

• Finish automated_checks branch tests: Blake asked me to test if the errors and warnings are being created correctly.
• Bug 588136: Make ispdb cache-friendly
• Bug 591353: Allow to pin down some important configs of big ISPs
• Documentation for developers (DEVELOPERS.md): Write a documentation file to help new developers.
• Clean up code and make the project pep8 compatible

Besides these tasks I hope I'll have time to work on Internationalization and localization. The plan is to localize all of the translatable strings so translators can contribute with the project.

10/08 - 20/08

This is the last week of the project. Thanks Mozilla and specially bwinton for everything! I hope I'll contribute much more to the project.

Now, to the tasks. While reviewing bugs Bug 518319 - Allow documentation pages in more languages, Bug 526469 - No field for enableURL code, we've decided to create a common class for DocURL and EnableURL (and DocURLDesc and EnableURLInst) since they are similar. So CommonConfigURL and CommonURLDesc abstract model classes were created.

For forms classes, we have created a LanguageDescModelForm class (for DocURLDesc and EnableURLInst). We should do the same for DocURL and EnableURL. The problem is: these classes are nested formsets (e.g. a config has a DocURL formset and each DocURL form has a DocURLDesc formset) and is not that simple to generalize it. But it should be done in the future.

On Bug 591353: Allow to pin down some important configs of big ISPs, we've created a small locked boolean attr that can be changed using the admin interface. So, locked configs can't be changed.

Also a small_fixes branch was created with a few small fixes like: remove unused attributes and methods, move form classes to forms.py, remove tabs, change port and socket_type order because js code changes the port when the user selects a socket type, submit it! link is not being updated, add end slash to URLS re in urls.py, etc. They are on pull request 11.

On bug 588136, I've proposed these cache settings:

• add a default max-age of 600s.
• for export_xml/id (exports a config XML): set no-cache to Cache-Control (it will cache but will always check with the server if the cache file is updated, and we can provide the last-update value using config.last_update_datetime)
• list/xml/ (export list with config IDs and its last_update_datetime) - set no-cache to Cache-Control and use last-update as the most recent config.last_update_datetime

They are already implemented on my bug_588136_cache branch.

I didn't have time to work on Internationalization and Localization and to improve UI. Nested formsets took much more time than I had previewed. But I'll continue contributing to the project, and I hope these tasks will get done soon.