Monday, April 24, 2006

Record everything

A couple of months ago I attended a conference at the Universidad Rey Juan Carlos in Madrid in which one of the speakers highlighted a principle that Linux Torvalds keeps related the Linux kernel development: all technical discussions must be made in the mailing list. He refuses to discuss anything by any other method be it in person, by IM, etc. In other words:


Linus defends that the Linux kernel developers should Record Everything.


Since then I've been watching how several open source projects work to determine if this is a common practice, which tools do they use to do the recording and how it may relate with the community around the project.

During this time I've happened to become a user of the Apache Jackrabbit project and I now think that it is a great example of the benefits of the Record Everything practice. Here is how they do it:


  • Everything is discussed through the developers' mailing list: this is usual in many open source projects, although some also use personal mails and those are not recorded.

  • Track every feature implemented: Whenever some bug solution or new feature is going to be coded it is added to its tracking software (JIRA). When it is not clear if some idea or issue is really a nice feature request or bug report it is first discussed through the mailing list. Only some consensus has been achieved it is added to JIRA. Then the discussion keeps going through JIRA's discussion system. One of the very nice things about this is that it is very easy to reference pending functionalities or even those in which people are working. It is also very easy to know when some functionality was implemented because everything is tracked through JIRA

  • All changes to the tracking software are mailed to the mailing list: this solves the possible problem of having to be aware of changes of JIRA issues manually. Everything is centralized through the mailing list but at the same time it is given an appropriate structure.



My personal feeling as a user of this projects is that thanks to these practices I feel more involved with it. I've been able to notify a couple of bugs and even sent a patch with some of my teammates with much more confidence than in most projects I've worked with. The reason is that I know what is going on within the project. I'm not a developer (yet) but no information is hidden from me. Jackrabbit was recently graduated from the Apache Incubation project with a great appraisal from its mentor. He particularly mentioned the existence of a very healthful community around the project even though it is very new and many developers are paid by a single company. I believe that following the Record Everything best practice has been very important to achieve this process success.

For this reason I've added it to the Collaborative Development patterns catalog. Your feedback and contributions to it are welcome.

Friday, March 24, 2006

Ward Cunningham on collaborative development

Ward Cunningham known as the inventor of CRC cards and wiki and one of the fathers of extreme programming and design patterns talks in eweek about his (very positive) view of collaborative development and community forming in Open Source projects. In his own words:

I'm betting on open source being a big trend, And it's not just because of cost, but because of end-user innovation. No end user wants to be a programmer; they just want to get their jobs done, [..] I think of software being a work very much like a wiki being a work where people see an area that's weak and they make it stronger.

Ward Cunningham recently left Microsoft to join the Eclipse Project to work on building its community. I've always seen him as one of the greatest innovators on software development methodologies and it's very nice to hear from himself such a big commitment to collaborative development. In fact he believes that even Microsoft should go towards this style of development:
They have to inch toward this community style development, otherwise it would be irresponsible to their stock holders

Thursday, February 09, 2006

Sources of best practices

Which sources could be used to find best practices for free software development? There are several good places to start from:


  • Successful projects such as Apache, GNOME, KDE, Linux, Debian, Ubuntu, Fedora, Gentoo, BSD, Eclipse, JBoss, Cherokee, Liferay, etc. Some of these projects have clearly visible practices which are even sometimes written down. How are the practices of different projects related? What are their consequences? It should be considered that some of the projects listed above are similar but others are completely different so while they probably share some good process practices other only make sense within each project's own context.
    Apache is one of my favorites because it has grown up from one project to 29 top level projects and lots of subprojects and has gained the respect of both the free software and the closed software world. But most important about apache is that they have been able to establish a strict set of rules to guarantee that all its projects are truly free and that are developed using a collaborative model (no one controls the project by his/her own). This rules have been based on known good practices so the questions that come up are: to what degree they are respected? How do they relate to project success?

  • Quantitative measurement of libre software projects: A line of investigation which has already provided nice results by analyzing several kinds of data produced by the projects such as the source code created, CVS logs or mailing lists. These studies can, for example, discover what practices are being used by a project and then correlate them to the grade of success of the project to determine if they are good practices in that context. The studies can also provide valuable information about context of the project which, as was explained in a previous post, is necessary in order to be able reuse a best practice.
    A group of the Universidad Rey Juan Carlos at Madrid, the GSyC/Libresoft group has published several very interesting papers about their results with these type of studies.

  • Articles which provide information about the experience of individual people with free software projects. Starting with The Cathedral and the Bazaar and continuing with lots of replies either of appraisal or proposing a different model.

  • Inspiration of software development processes usually associated to the development of closed source products. Mostly from agile methodologies that promote self organization and the books and articles about organizational patterns. SCRUM is a very valuable source of information. It is a software development methodology (with a very deep theoretical background) which establishes a very low ceremony process

  • Results of several surveys performed to open source developers which show how they are organized and what are their motivations. Some interesting questions which may be answered by them are: how is evolving the volunteer to paid worker ratio of free soft developers over the year? What motivates each of them?


My next posts will be exploring each of these sources of information.

Looking for best practices in free software development

Some free software projects are very successful while others fail completely. What's the difference between them? We should be able to find what the successful projects do and document them as the best practices for running free software projects. Project leaders could use this information to know get an idea of what they should do (depending on the project characteristics) to improve their chances of success.

Several issues appear in this quest to find such best practices. The first and most important one is Are best practices always best practices? probably not.

For this reason the practices shown as good for some projects should be described within its context. By providing the context it will be possible for anybody to determine if his/her own context is close enough or not to consider the practice useful. Patterns are a formal way to document best practices and provide enough information to help determine in which cases they are applicable and how to best apply them.

For this reason the Collaborative Development Wiki has decided to use patterns to document our findings. Join it to give your opinions or provide patterns you have found.

Saturday, February 04, 2006

Alan Cox reviews the development process of the Linux Kernel

In an interview performed by News.com Alan Cox has given some interesting answers about the process used to develop the linux kernel and how it has changed since its origins. Here are the most interesting points I've found:

  • Years ago most developers contributed on their free time while now the big majority of developers are hired by some company, then there is a minority of students and finally some casual contributors. One of the consequences of this change is that now patches are mostly sent Monday to Friday rather than weekends. I had never thought about this, although now it seems obvious: it is possible to study when changes are made to a project to determine if it is being driven by people working for the project within companies or by free-time volunteers.
  • Everybody can contribute to the project but there is a lot of control and review. The interviewer compares this to Debian which has a previous process to become a contributor. I wonder what are the characteristics in Debian which have made them choose that model. Is it the right decision? Alan Cox states that by not having a formal process they allow for casual contributions. Doesn't Debian need those? Do they have another way to allow them?
  • It has been created a kernel newbies project in which new contributors learn how things should be done. This may indicate awareness from the core developers that there is a steep learning curve for beginners. Another fact that makes this project useful is that quality standards are very important for the kernel and patches sent which do not meet them will not be accepted so it is important that newcomers learn those standards.
  • It has also been created a kernel janitors project where people work on small things, such as clean-ups and reviewing code. This seems to me as an interesting idea to make sure that some boring but simple tasks get done. By listing and highlighting them through this project you get two benefits: core developers acquire a commitment to have them done but at the same time by delimiting some small, not too difficult, tasks it is easier for newcomers to help with them.
  • A final important fact that may be used to evaluate the process used is that Alan sees the kernel as a mature product in terms of functionality so the focus is currently in improving its performance and quality.

Tuesday, January 31, 2006

Introducing a collaborative model for developing software

Free software development has shown how a model for developing software centered on the collaboration of its parts can produce large software products with high quality levels.

Eric Raymond first described the difference of this model, which he called the Bazaar, compared to the model which is usually found on closed source projects, which he called the Cathedral. But the fact is that not all software projects work the same way and not even all of them are successful.

This blog is part of an effort to analyze and describe the processes that are used to develop free/open source software and create a body of knowledge for it. Here I'll be posting my thoughts and findings about the subject, while all the conclusions and important discoveries will be documented in the form of patterns in the Collaborative Development wiki.