Personal tools
You are here: Home Software Center do2gether Documentation Architecture Draft Reference Architecture
Document Actions

Draft Reference Architecture

Note: This is the print view with all the Reference Manual pages on one page. The paginated version is available here, if you prefer that.

Hows of a possible high-quality and scalable deployment.

1. Introduction (by Lupi)

Draft Architecture Introduction
Scope of this document
This web application will follow a software as a service model. It’s very important that our software scales well horizontally - adding new servers to handle more load should be a breeze.
This document details the scalability and other non-functional requirements and proposes a reference architecture that support those goals.
Software platform
The reference platform is:
    Plone 2.5[1]
    Zope 2.9
The latest minor versions of these packages should be preferred.



[1] Plone 3.0, which runs on Zope 2.10, has not been released yet, but the expected date is May 10th, 2007.
I recommend to talk with Plone release manager regarding the possibility of moving the development to Plone 3.0, he can advise us on the stability of the new release and can estimate better than anyone else if the expected date will be met.

2. Scalability Requirements (by Lupi)

Scalability Requirements by Roberto Lupi.
Usage Scenario
Partecs plans to offer its basic product as a service. Organizations will be free to sign up for the basic service and will pay for additional services or to host the software on their own servers.
    we expect a high number of signups to try out our software.
    Some of these organizations may find our software unsuitable to their needs and abandon their accounts.
    Among those that continue to use our software, we expect users to divide into these classes:
    Active, high value users that use Partecs’ software as a key part of their work, they will probably sign up for additional services. These are high load, high return users and we must give them the best possible service.
    Low load, low value users that find Partecs’ software suitable for them, but do not consider it central to their work. Most probably these organizations won’t invest much in value-added services on our platform, but it’s important to keep them happy for marketing purposes. It is important to give them a good service.
    High load, low value users that find Partecs’ software useful but do not currently invest in value added services. These organizations have the potential to drain resources from other more profitable users, we must avoid it and try to steer them to become higher value users. It is important to minimize their load on the servers, while still giving them a passable service.
 
Requirements
Creation of new organizations should be fully automated
We cannot support a high number of signups unless the marginal cost to add a new organization is zero. It should be a fully automated process, initiated by the user:
1.        The user signs up thru a web form on our site.
2.        The software chooses a suitable server to host it.
3.        The server creates the required environment to host the user’s organization.
Inactive accounts should not affect server performance
If many organization sign up for our service but decide not to use their account and abandon it, our software’s performance should not be affected.
An unused account must:
    Have no impact on CPU load.
    Have no impact on memory load.
Storage space is cheap, so it is fine to use it to store the unused account data for some time. An automated process should however be created to track account usage, so unused account can be removed after some time.[1]
Migration of organizations between servers should be easy
We want to partition users by their class.
    High load, High value users will have their own servers.
    Low load, High value users will be separated from other users so they always get a good service and are kept happy.
    Low value users will be put together. A mixture of high load and low load users will be put on each server in order to balance the load among our server farm.
We need an easy procedure to migrate an entire organization from a server or instance to another one. This procedure will be used to move users between our servers as they change their value for us and as we add new server to support the load over time.
Ideally, we want to use Zope built-in import/export tools to handle the migration:
1.        export the whole organization from the Zope management interface (ZMI) to a file.
2.        move the file to the new Zope instance, potentially on another server.
3.        import the new organization thru the ZMI.
4.        update any system-wide configuration file[2] to point to the new location.
The system should scale horizontally with servers
We will be adding more servers over time. It is important that the performance of our server farm scales linearly as we add more boxes.


[1] It can be easily done by tracking the last login times or by grouping the web statistics by organization.
[2] e.g. front-end web server configuration or load balancers configuration.

3. Reference Architecture (by Lupi)

This chapter describes a reference architecture that fulfills the requirements set forth in the previous part of this document. It is not the most performing architecture, instead horizontal scalability and ease of management and deployment have been preferred.
Front-end web server
An Apache web server or another similar system will be used to server static content, it will forward requests to the back-end servers.
The load-balancing component will not be present until more than one back-end server is used.
Configuration files or a database will map the user organizations[1] to the right back-end server. Initially, Apache rewrite rules and proxy capabilities will suffice; eventually, an Apache module that looks up organizations-to-host mapping in a database can be created.

Reference Architecture

Back-end server
We will have as many back-end servers as needed. When one’s capacity is saturated, a new one will be deployed and new organizations will be added on the new server. It is a good idea to create a “template” installation image that can be deployed on a new server by just copying it over the hard disk.
Due to the way Python works, the Zope application server doesn’t scale so well when run across multiple CPU cores. For this reason, each back-end server will host one Zope application server for each CPU core.
Each organization will have its own instance of Partecs software, they will be created as objects inside one of the Zope application servers. These instances should be self contained: users, content, groups and settings should all be stored inside these objects and their children.[2] In that way, it’s possible to use Zope’s built-in import/export mechanism to move organizations between Zope application servers or even back-end machines.
Scaling guidelines
It is impossible to forecast perfectly what will be the main bottleneck of this architecture until the software is developed. However, there are two possible scenarios.
The number of concurrent requests that a Zope server can fulfill is limited. If this becomes the bottleneck, then organizations should be shuffled around between Zope application servers - even on the same back-end machine - so that organizations with lot of concurrent users will get their own app servers. This will prevent very active organizations to starve less active organizations.
On the other hand, if CPU, memory or disk bandwidth become the bottleneck, then the organizations should be shuffled across back-end servers. Each machine should host a mixture of high-load and low-load organizations, in order to make sure that each system is used effectively.
In both cases, if an organization becomes too big to handle on a single server, it will be moved to a dedicated ZEO cluster. ZEO is Zope solution to connect multiple zope instances to a single database.


[1] The user’s organization will be extract from the URLs or host names.
[2] Alternatively, for high value customers that want more than one instance, the security and authentication data (users, groups and settings) can be stored externally in an LDAP directory.

4. Rich Text Editor Choice (by Lupi)

Evaluation of a WYSIWYG editor suitable for the platform.
Rich Text Editor choice: FCK vs. MCE
I analyzed both FCK and MCE, diving into their APIs and plugin architectures. I think that there is no significant advantage in using one over the other in your particular projects, both are valid pieces of software and quite easily extendable with a good documentation. In either case, I don’t see any particular stumbling block or barrier that you’d face in the future. Both are valid foundations for D2G.
I think the choice of which editor to use (among these two) should be left to the software developer. Pre-existing working experience on either one is what is going to make a difference.
Both FCK and MCE seems to be backed by valid companies and developers. I suggest that you ask them for a quote for the D2G development.
If they don’t want to bid for the whole project but instead suggest that they develop only the editor part, I advise not to choose that option; instead, forward their offer to the companies that bid for the whole project and let them deal with subcontractors. You should strive to have a single, accountable development party.

5. AJAX Frameworks (by Lupi)

Discussion over a preferred Ajax framework for the platform.
AJAX frameworks: KSS and Dojo, Yahoo UI Library
I evaluated KSS in details, I think it’s quite an interesting piece of software but it’s not mature enough yet. The cornerstone feature is that it turns Javascript coding (hard) into CSS-style coding (easy) in the areas covered by KSS, mostly AJAX interactions. It’s going to be integrated in Plone 3, so it will most probably be readily available in the software platform for -at least- Do2Gether. On the other hand, I find the documentation a bit lacking and KSS at the moment is too limited to cover all your needs. I expect that KSS will improve sharply this spring, boosted from the PloneSprint in March and the soon to be released Plone 3.
Dojo on the other hand is much more feature rich and complete. It’s mature and has very broad industry support, the Dojo foundation has impressive sponsors. I think that it is the best choice for what you need in both Do2gether and Draft2gether, except for the text editor in the latter one.
In addition, I suggest a look at the Yahoo UI Library (http://developer.yahoo.com/yui/). It’s not a monolithic piece of software, in fact it’s very modular and you can get just pieces from it. There is a single piece that you should consider using, no matter what AJAX framework you choose, that piece is their portable, liquid layout CSS stylesheets. Another interesting piece is the Container framework, which can be used to develop the kind of dialog-style UI that you want for the settings/preferences window in D2G.
I suggest that you choose Dojo as your reference AJAX framework, complementing it with YahooUI CSS stylesheets and, perhaps, Containers. Your existing code that use Mochikit will be most probably replaced during the development, so its existance is a non-issue.

6. Other Media: FAX, SMS... (by Lupi)

Products & Services enabling interaction over other media channels.
Other media: fax, SMS
I can provide references to a few services that I used or evaluated in the past.
    Sending and receiving faxes. I use a service called www.popfax.com and I am quite satisfied by it. The cost for sending faxes is 0,05 euro per page in Europe. Receiving faxes is free. It works worldwide. The only issue I had with this service is when sending faxes to VoIP numbers in Italy, which was to be expected since we don’t have yet reached VoIP interoperability in Italy.
    Sending SMS. Various providers allow you to send short messages (SMS) to mobile phones for a fee. I have evaluated Energit (http://sms.energit.it/) and Aruba. The former has better customer support but higher prices, the latter has a big cost advantage.
 
All of these services can be accessed thru web services and can be easily integrated in your application.
I have no direct experience with sending regular mail, so I cannot recommend any particular service.

7. Miscellaneous Notes on Architecture (by Lupi)

Here is a general sketch of the architecture that I expect you will be using for Do2gether and future Draft2gether (if it will be developed in Plone).

Reference Architecture 01

Apache will act as a gateway and proxy, serve static content (javascript frameworks, CSS stylesheets, user interface images), translate addresses like org1.d2g.com into requests to your Zope instances (using mod_rewrite).
There will be one production Zope instance per CPU in your servers.
These Zope instances may be distributed to one or more physical hosts, you will add boxes as your users grow.
Each Zope instance will accommodate one or more Plone sites (Do2gether, Draft2gether). Each Plone site will serve a single organization. You should create a master/template installation for each of your products. Zope allow you to easily clone that template directly from the Web, making it very easy to add a new organization.
When a Plone site becomes too big and you need to scale up, you will move it to a separate Zope instance on a new box.
I hope that these notes will be useful.
Yours Truly,
Roberto Lupi

8. Product Recommendations (by Alec)

Recommended Plone Products after a brief in-person meeting in Rome, navigating on current do2gether.net code (v0.8).

For Member Profiles it will be necessary to transition away from your current custom solution in favor of one which integrates with PAS. I suggest looking at:

remember

I suspect that the new portlet architecture will be essential to the development of your product, both for your right column and your "dashboard" type views. It would be very wise to have all developers study this architecture intensely. It is included with Plone 3.0 in the packages:

plone.portlets, plone.app.portlets

For calendar related items I recommend a look at:

Plone4Artists Calendar

Which uses Zope 3 techniques to turn any folder (or smart folder) containing events into a calendar with two way sync from desktop applications like iCal. This should also allow a blog to be viewed in a calendar like fashion when suitable adapters for the posts are provided.

For audio and video blog support, I suggest extending existing blog products (including your own via adaptation. Some Zope 3 products which provide relevant adaptations are:

Plone4Artists Audio, p4a.videoembed

The former of which allows 'File' objects to acquire special behavior and UI when marked as containing audio, and the latter of which allows users to embed videos from e.g. YouTube without allowing them to enter potentially dangerous unfiltered HTML (<embed>, <object>, <script>, etc.).

For blogroll, planet, and other RSS aggregation needs it would be worthwhile to look at:

feedfeeder and the rss portlet included in Plone 3.0 in plone.app.portlets

Testimonials should be relatively easy to implement, especially on top of a product like remember. It seems you already have a sketch of an implementation in your MemberProfile product, though that code is not likely to be directly useful in Plone 2.5+, it could reasonably be applied to a remember based member object.

The email, sms, bulletin, ... notification system should be easily implementable using Zope 3 event channels along with some means of storing/retrieving the user preference information.

For a global ratings system look at

'contentratings' or perhaps 'lovely.ratings'

which allows for customizable multivector ratings (though as far as I know it only works in Zope 3).

Globally available voting for and flagging of content should be easily implementable using single adaptation along with annotations, 'contentratings' provides a basic code example of how this sort of thing is accomplished.

9. Implementation of the Generic Listing View Components (by Alec)

Suggested implementation for the Generic List Views and useful reading.

My suggestion is that these be implemented as a series of ViewletManagers, where each viewlet is a representation of a list element. There would be a few simple interfaces involved. First the interfaces describing a list items and actions available on these list items:

class IActionItems(Interface):
"""A description of an action available on an object"""
# This may need to be generalized a bit to support actions which
# are more complex than a simple link
title=TextLine(u"Title")
description=TextLine(u"Description")
url=URI(u"Action URL")
category=TextLine(u"Action Category")

class IListItem(Interface):
"""An interface to describe an object which is part of a listing"""
title = TexLine(u"Title")
description = Text(u"Description")
actions = List(u"Actions",
description=u"List of Actions Available on the Item",
value_type=Object(u"Action Item",
interface=IActionItem))
You would then register adapters from each of your potential list items to IListItem. These adapters would likely be incredibly trivial. Then you would register the relevant viewlet managers and viewlets. You would have a named viewlet manager for each type of potential list item ('comment_listing', 'amendment_listing', ...) these implementations would all use the same basic implementation, changing only the content type they choose to list and the name they are registered under.

Anyone who ends up making your architectural decisions or doing development should read and understand the very thorough README.txt included in zope.viewlet, which describes a similar pattern for building an extensible folder contents listing using viewlets.

The base implementation of the viewlet manager would get the objects of the desired type (possibly batched), and lookup a viewlet corresponding to each object (a viewlet adapts a context, request, view and the viewlet manager itself). Internally the viewlet would make the adaptation to IListType to get the desired data, and would render some relevant html for that piece. More specific viewlets would be registered for each object type (comment, amendment, etc.) as needed, but 90% of the implementation would remain the same, and these more specific renderings could be added incrementally.


Useful Reading


The main text that developers and architects should read (aside from standard Zope and Plone stuff) is 'Web Component Development with Zope 3' by Philipp von Weitershausen.

Philipp's site also has a lot of useful info, particularly regarding using this stuff in Zope 2. Additionally, Martin Aspeli will be publishing a book on development with Plone 3.0 in the near future, and that will certainly be worth a
look.

Alec.


Powered by Plone CMS, the Open Source Content Management System

This site conforms to the following standards: