Distributed Bug Tracking

I was positively surprised to see Hubert Figuière write about decentralising bug tracking yesterday. This is something I've also been discussing with friends over the course of the past year. Unfortunately I hadn't taken the time to organise my thoughts, but this gave me a good opportunity to do just that.

In my career I've yet to come across a good system for bug tracking. The options are usually (with some notable exceptions) behemoths of the last century. There has been little or no real innovation on that front for 15 years, when Mozilla first released Bugzilla in 1998.

Bug tracking software is pretty much still fairly simple web applications with some page to look up issues, a form to file new issues, a search functionality (that is usually pretty bad, since I never seem to find what I'm looking for), and some way of commenting. The more complex alternatives bake in resource tracking and time management. But more fundamentally, everyone are hosted environments with traditional server-client setups backed by your standard SQL database.

Yet other pieces of our technology stack have received significantly more attention. The Linux kernel, being one of the bigger free software projects of its day, started reconsidering the server-client relationship for sharing changesets amongst developers early on, and switched to BitKeeper, one of the pioneering distributed revision control softwares. The basic idea is simply that instead of requiring users to talk to a central server, you let changesets be synchronised amongst individual users. This is done by the users pulling in eachother's changes. Admittedly you still need canonical servers to ease the initial checkout, but it was a novel concept that really just makes that server another node in the grid.

git's branching model (CC BY 3)

Distributed version control was a novelty back then, and soon led to a myriad of different implementations. Amongst the more popular ones we find git, hg, and bzr. Combined, these power the vast majority of new software projects hitting the open source world, so they must be doing something right. Perhaps users are seeing the benefits of working offline, or perhaps the new generation of RCSes are simply better engineered than the more traditional cvs and svn.

In fact people are finding them so generally useful that we are beginning to see other use cases. Centralised hosting providers such as Github are offering you to host websites through the gh-pages branch of your repo. And I find it really interesting how our repositories are starting to incorporate all of these things that traditionally used to live on hosted servers elsewhere; code, web content, wiki; and becoming these self-hosted and distributed entities, which in essence makes you less reliant on a single remote to get work done.

Now for a little thought experiment: What if we were to take this idea and apply it to bug tracking? An in-repo branch may hold bug issues backed by a program to manipulate issues and add comments. Not totally dissimilar to the way Maildir works. mu is a great example of a program to interact with that format. This needn't mean that we couldn't have centralised web interfaces on hosted servers anymore; in fact they would be much easier to clone and host. With a load balancer and individual syncing in the backend they should also be able to more easily cope with heavy load, which is an absolute requirement today.

But more excitingly one could also envision local browsing of issues, either through spinning up a local web server (much like godoc -http :6060) or through a more traditional gitk or gitg UI. In fact Mercurial already does this through hg serve, that spins up a local HTTP server for browsing files, changesets, and more. For a distributed bug tracking system, one could foresee using this as a possible interface to search, add comments, and triage bugs in. Because it might sit in the same repository as your code, you'd get a much tighter integration with your code than you'd normally get in traditional BTSes.

This also also has several practical aspects to it: I'm used to working on really large software projects such as browsers. I worked on Presto for five years before I moved on to WebKit/Blink, and more recently Gecko. My team at Opera was partly responsible for hosting the in-house JIRA instance we used for tracking bugs. The several hundreds of thousands of bugs filed in that exceeded Atlassian's recommendation by two or three times, and so required the machines hosting it to be a fairly beefy monsters. For example, I believe we maxed out our database server at 256 GB of memory shortly before I left to join Mozilla, where we similarily just recently turned our 1 millionth bug.

Is the time ripe for rethinking bug tracking? I think so. A successful solution needs to be simple to understand from a technical perspective, e.g. no complicated backend for storing the data. It needs to implicitly deal with conflicts that would probably be quite hard to achieve. It needs to have a great toolchain to make it easy to adopt and integrate with. And it needs to rely on proven technical solutions instead of reinventing the wheel at every opportunity.

Thanks to Berge Schwebs Bjørlo for valuable feedback.