Test Results

First of all, WOW, thanks for all the responses I got. As I will explain later on, the received feedback is extremely useful. To the people who sent me an e-mail, I'll respond personally monday or tuesday, this weekend is extremely busy (I'm selling fish all weekend!).

Okay, a few people didn't get anything from the syslog, but did get a 'General OS Error' on doing a 'cat /dev/bogus/usb_test'. The general OS error is a good sign (when your computer doesn't crash), this means that the USB stack is started. If you have ProcessController, you'll probably find a thread called 'usb_hub_explore'in the kernel team. I forgot that syslog isn't enabled by default. For now, I have enough test results to continue my work, but I'll be posting instructions on how to enable the syslog for the next test. People without syslog, even though you might not be able to help me directly, you've still been a great help by letting me know you are still following my work.

For the other reports, there were roughly three kinds. I haven't had te time to analyse all completely, but I know that there were people who experienced computer crashes. That's a pity. Because I've been testing my stack on a single computer, I haven't been able to follow all available code paths, and since other hardware may work differently, you probably followed untested code paths leading to crashes. Those are kinks to be worked out later this stage.

The second group of people were the people who sent me a useful syslog which stated something that made me extremely happy. For those of you who haven't studied the USB and UHCI spec, here's a very rough idea of how the UHCI controller works. All transfers over the USB hardware goes based on a schedule on the host controller. The part where I was stuck at, was when I gave the host controller the task of starting schedule execution, that it would stop. Reading through the hardware specs didn't give me a conclusion on why it would have stopped, so I was expecting that it could be my type. To be conclusive: there were some people who had the schedule running succesfully. This means that I shouldn't check general behaviour, but I should check FreeBSD's stack and the linux stack on specific exceptions for my hardware. Thank you.

The third group of people were those who had very deviant behaviour. Because my USB stack is very verbose, it will be nice to analyse those logs as well, to learn more about the behaviour and why it works. I'll be studying your logs as well.

It's time to go to sleep now, I have to get up early tomorrow, I'll answer your mails and questions monday and tuesday.

Thanks.

The next step

For the last few weeks I've been stuck at a certain point in development. For some reason, the UHCI host controller stops as soon as I start it (but the strange thing is that it does get started). I'm fooling around a little, but so far I haven't had anything that works. To rule out that it would be my controller, I'd like to use you all to test it for me. I'm working on a script that installs my usb driver and that can help you with providing test results to me.

Speak to you all later.

Update

Okay, here's the procedure. First of all, you need an UHCI usb busmanager. These cards are based on Intel's hardware specification. There is a more than moderate chance that you have this type. If you have an USB 2.0 controller (the EHCI spec), you should still check if you have UHCI (EHCI has an older host controller for backward compatibility). You can check it by going to the Devices preflet, and checking the info on your USB entry (or entries). If one of these says "base: c Subtype: 3 Interface: 0", you're ready to participate in this public experiment.

1. Go to your /boot/beos/system/add-ons/kernel/busses/usb and your /boot/beos/system/add-ons/kernel/bus_managers and move out uhci in the first and usb in the second one. Keep it if you need usb support and put it back in when you finish this exercise (no need to remove the files you are about to install).
   
2. Reboot. This means that if you use an USB mouse or keyboard that it probably won't work anymore. So don't try this if you don't have a spare PS/2 one.
   
3. Download this archive [http://home.planet.nl/~ksctrre/usbtest.zip] and unzip.
4. Open a terminal, go the the location of the archive and perform an ./installusb.
   
5. Here comes the actual testing part. Open two terminals. In the first, do a tail -f /var/log/syslog. In the second one, do cat /dev/bogus/usb_test. At this point, the first terminal should go berserk with many lines. If you have repeatedly two lines, then you're done (CTRL-C it), and copy and mail all the output above to me.
   
6. Restart to unload my usb driver. If you want your normal usb stack, place the files you have moved out (in the first step) back in.
   

Thanks in advance for your help.

Still Alive

For those of you wondering: Im still alive. At the moment I´m on holiday in the Czech Republic, until the 20th, then I´ll post a technical status update (on a number of fronts). I´ll also will be responding to mail from that date on (more frequently).

Thanks for not forgetting me.

Garden post

It's just lovely weather, and thanks to wifi, I'm transmitting this message live from my garden. There's a clear blue sky, a nice warm sun and a comfy chair, and it just makes me want to live near the beach (which I'm seriously considering, by the way).

I'll just report my current status of the 'tinderbox'-clone. I've got the rails app up and running reasonably fast, and the communication mechanisme between the build machines and the server works (though it's not error-proof yet). I'm currently implementing a view. At one hand, I think something like mozilla's tinderbox is good (the table view), but the rest of the interface is a bit cryptic and messy (so I'll be experimenting with that). As soon as I've got something of an interface, I'll put the thing online to show the early progress.

I have yet to find a place in Haiku's SVN for it (or any other SVN for that matter). It will be pretty nice though, and it might even have a beneficial effect on developer productivity and code quality.

Tinderbox replacement

Okay, so I thought I could get away with criticizing the Haiku project without actually doing something? Wrong, at this moment, the first pieces of a 'tinderbox' replacement are being assembled on my computer. At the moment I'm going to test the communication between the build machines and the server, so expect some results soon. I hope to prove that this is something that can be done in one day (so I hope results are coming out tonight!)

Quality Assurance: Getting Started

In this document I'd like to set out a few thoughts regarding Quality Assurance. As a somewhat active participant in Open Source projects, I have received a reasonable amount of observations regarding quality assurance and it's practises in Open Source software. I must admit, thought, that I don't have any professional experience in quality assurance, or software development in general. However, that doesn't belittle the fact that the Haiku project is at a certain point in its life where a good road to QA should be chosen.

Note: This is a 'nicer' font than the message sent to the haiku mailinglist. It would probably be best to keep the discussion over there.

In order to work with the observations and ideas, we should be first set out the definition of quality assurance. My view is that quality assurance is the ongoing process by which the quality of an end product is constantly monitored and controlled. Software quality assurance is no different, perhaps with the only problem being that software in some ways will always be 'broken' (or buggy) - the trick is to have it working for as many people as possible.

This simple definition reveals the ultimate problem which Open Source software is having: QA is a process - meaning it needs constant attention and care. But the open software projects are generally catered by developers. The consensus seems to be that whoever contributes, gets something to say about the project. The fact that developers provide the most substantial part of code, leaves them to say things about what happens next. The ultimate problem is twofold. First of all, most developers code what they need/require (if it works for them, it's okay). The other side of the coin is that there comes a point where the developer:user ratio becames greatly disturbed, so that developers will be overwhelmed with bug reports to an extend they can only ignore them. Furthermore, a developer might not have all the patience and care needed to provide a descent amount of QA.

Mozilla

When you carefully look at most Open Source projects, you see that they do recognise the need for QA. Most obvious leader in OSS QA is the mozilla project. This is probably most due to the fact that it stems from Netscape - a company with a dubious QA record, but nevertheless a company. Companies have commercial interests to protect, so they usually employ some kind of QA mechanism. The policies of Netscape seem to have been flowing into the Mozilla project.

The implementation of the Mozilla project is two-fold. First of all, they have a system called 'tinderboxes'. A tinderbox is merely a server that's continually building and running unit tests on the latest code from CVS. The boxes report their status back to a central server which neatly publishes all data on a web page. The nice thing is that everybody is responsible for their own changes. If you commit something that breaks on other hardware, it's your job to make sure the repository get fixed. So this is actually an ongoing process to ensure a constantly working tree (= a more constant quality).

The second main tool is their custom developed bug-tracker 'Bugzilla'. Bugzilla is the number one choice for bug tracking. The interesting thing is that it has a stable code base and it is well maintained. The good things about it are the power it has, and the workflow it imposes. Naturally, there are some issues that lower the fun a little. First of all, it's a usuability nightmare for the users. Second of all, over the years it has evolved (with the Mozilla project) to a new level where it functions more like a project tracker tool, instead of merely a bug tracker.

I might need to illustrate that point. Over the years (ever since Netscape was taken over by AOL and has taken its hands from the Mozilla project) the use of Bugzilla has changed. Instead of a bug tracker for users to report their bugs in, it has moved over to be more like a large container of all patches that need to be tested. This means that the emphasis on the users has started to disappear. This is reflected in the fact that they don't usually assign QA contacts to bugs anymore. Also a lot of features in Bugzilla have turned it into helping developers and complicating the user interface even more. That said, bugzilla still offers great features for bug tracking, it just needs a tweaked UI (which can be done quite easily).

It's not the tools; it's the use

Even though the Mozilla projects seem to have tools to properly perform QA projects, they still lack something. First of all, you will find many bugs in their database that seem to be forgotten, or even never been looked at. Second of all, the project still believes that it will find most issues by issueing alpha and beta builds. I would argue that issueing builds won't have the influence on quality we expect, and I intend to prove that claim.

But first on the first issue. One of the reasons to use a bugtracker is to lower the entrypoint for bug reports, and for developers to keep track of the reported bugs (instead of using e-mail to do that). However, you won't prevent the problem mentioned earlier: at a certain moment in time, the user:developer ratio changes to such an extend the developer will still have too much bug reports, and the sad thing is that nothing happens with it anyway. This leads to the point that employing a bug tracker to improve communication with the users is a myth.

The second issue, posting builds in order to get feedback to improve the product, is also overhyped. The common idea is that 'more eyes' will improve the general quality. While in fact this may be theoretically true (or not), it's at least the case that when someone is a developer, betas and alphas are usually employed by users to check out the 'cool new stuff'. Users are generally not developers, and most probably won't report bugs. The ones that do, will often publish low-quality (useless) bug reports that in no way can be reproduced. Add this up to the previous point, and you get no improvement in Quality Assurance whatsoever.

When you add all this up, you'll understand that Quality Assurance might not be a job for developers. Thus an alternative way of keeping QA up to par in Open Source needs to be devised.

Solving the problems


In order to try to work around the common pitfalls of Open Source Quality Asurance, I will present some ideas/concepts that may be able to counter the problem.

Just as a reminder: QA is a continual process that checks the code of the project. The first thing that implies, is that at every point in time, the code in the subversion repository should be checked, and regular builds should be provided so that people will be able to keep an eye on what's working and what's not. Second of all, regular unit testing should be designed in such a way that they would be able to check the most basic failures. In effect, this probably means a system like the tinderboxes. This would also allow to have continual build checks on multiple systems (haiku, R5, R5-bone, zeta) and multiple compilers (gcc 3).

The second part of the solution is a bit different. It requires a specific QA team whose task it is to provide all the work developers might not have the time for. In an ideal world, every part of the codebase should have a QA manager that does three things. First of all, he/she is a regular user of the code, and preferably continually stress tests the part. He/she keeps an open communication with the developer(s) of that part, which implies a healthy relationship. Second of all, the QA manager will be the one that is involved with the whole process of bug reporting and resolution. He checks every bug that enters his realm, tries to reproduce it, and works with the reporter to make it easily reproducible for the developer(s). After a fix is made, the QA manager tries to verify the fix, together with the user, which then cleanly resolves the issue. This approach has several advantages: the developers are released of a lot of work regarding bug reports, the users have a larger chance for a response and a fewer number of bug reports will enter the realm of obscurity. The final advantage of a QA team would be that it would create a lot of openings for non-coders to participate. Many times one hears remarks on the mailing lists from 'lurkers' who just don't know what to do. This approach generates a lot of valuable 'jobs'.

A third thing is needed to improve OSS QA. This is what I would call the 'openly closed beta build'. This would mean that a select amount of people would be invited to a special forum or mailing list, would get a special build of the software to test, and more importantly, would get the opportunity to have a good influence on the project. By creating small test groups of enthousiasts with many different configurations, you create a perfect environment to do intense testing. The developers would participate directly with the users, but because there are a few of these users, that communication is possible. The selection would also find enthousiast testers, who will probably find a lot more bugs that you would with regular betas. Even if that's not the case, the general quality of the beta bug reports would be better than in another fashion. When you finally do release a beta build in the open (which would attract a larger audience), you will have ironed out the most obvious bugs, and most certainly also a lot more obscure ones.

Getting to a conclusion

All these solutions may seem a bit strange to OSS developers. It requires dropping the whole idea of 'whom codes, controlls the project'. This attitude will have to change to get this idea working. With a young open source project (like the Haiku project) this mentality change is probably a reacheble one. Without many rigid structures commonly found in older OSS projects, and an open area for discussion, we might reach a world where Quality Assurance is an integral part of the equation. Furthermore, even the older OSS projects seem to be walking into a different direction (albeit a bit slower). The KDE Quality team seems to be a vast new force to improve the general Quality of the code. GNOME has the bug-squat. These teams both try to get to a world which I'm promoting as well. And that's the next challenge of getting Open Source Software to the desktop: promoting a higher quality standard.

Stalled again

Hey guys,

It's been a while, so it might be nice to give a little update. For the last two weeks, I've been working on preparing for my exams (I think I made 2 out of 3, so I'm reasonably satisfied).

About USB. Here in Amsterdam I've got an OHCI controller, but for some reason I cannot get it to work. I've got the registers properly setup (I think), but when I try to read some data from a register, I only get bogus information. So that's not quite working yet.
Finally, I'm also considering moving all my development into the haiku SVN. The world isn't ready for distributed development, and the point of using it when I'm the only one using it might be somewhat beyond the goals of distributed development.

Niels