jmmv's weblog

Blog migrated to Blogger

2005-10-22T15:49:09Z

As I outlined a week ago, I was considering the migration of this blog from Livejournal to Blogger... and I finally did it.

The new address is http://julipedia.blogspot.com/ and the blog is now named The Julipedia. Please see the welcome message for more information. Remember to update your subscriptions!

This blog will no longer be updated. Its contents will be kept intact for a long time.

Thanks to Livejournal for their service during the life of this blog.

GNOME 2.12.1 hits pkgsrc

2005-10-16T14:33:44Z

As usual, the latest stable version of the GNOME Platform and Desktop, 2.12.1, has been integrated into pkgsrc. It has been a tough job due to all the affected packages and comes a bit late, compared to all the previous updates, but I hope you'll enjoy it.

Please see the official announcement for more information.

Blogger

2005-10-15T11:37:05Z

Two years ago or so, I registered a weblog at Blogger, which only lasted a week (don't bother looking for it; it was deleted). At that time, I didn't like that site much, specially because it lacked a very important feature: integrated support for comments. Yes, you could use external utilities/sites to host comments, but that was complex.

However, a few minutes ago, I discovered that they now have comments support; yay! I spent some time surfing their site and found it very intuitive. Also, I registered a new weblog (whose name I won't unveil yet, but a friend of mine will know which it is ;-) just to see which features they support now.

And all I can say is that it seems better than Livejournal... Here are some reasons:

Blogger applies the weblog style to all pages: posts and comments. Here at Livejournal, only the front page gets the look you design, but all comment ones have a standard, non-customizable and ugly look (I don't know about paid accounts).
As regards the style, Blogger lets you customize it completely, editing the HTML code directly. Livejournal does not let you do this unless you have a paid account.
Blogger addresses are shorter and easier to remember: http://<blogname>.blogspot.com/ vs. http://www.livejournal.com/users/<user>.
The blogger interface seems to be more intuitive than Livejournal's one.
Blogger lets you post images on your blog (without external links).
Blogger lets you post by e-mail.
Neither Livejournal nor Blogger have support to track blog statistics. However, Blogger lets you add statistics trackers from other sites, which is a plus.
My beloved Drivel supports both systems, so easier migration from this POV.

So... I'm seriously considering migrating this blog there, because the reasons mentioned above. However, there are two things against this idea: people will have to update their address (though I don't think there are many readers out there ;) and, worst of all, I don't know if it's possible to easily migrate all the existing posts to the new site (and this is a must).

Anyone has comments on this? Any counter-arguments or suggestions? :-)

Edit 18:08: The statistics item was added.

Articles: Lightweight web serving with thttpd

2005-10-14T16:20:40Z

Several months ago I started writing an article about setting up and using the lightweight web server thttpd, focusing on the NetBSD operating system. I finally decided to finish it a month ago and submitted it for publication. You can now read it on-line at ONLamp.

Hope you find it useful!

C++: Constructors and global data

2005-10-12T11:44:08Z

As a general rule of thumb, accessing global data from within class constructors is a dangerous operation and ought to be avoided. The C++ specification does not describe the order in which global variables are initialized, so there are high chances that your constructor accesses uninitialized data, producing unexpected resultsn (e.g., crashes in some circumstances).

It is important to remark this. This behavior is undefined, so it can change across compilers and/or architectures. If your code relies on it, it's broken: something that apparently works on one machine may not work on another one.

Let's see an example. First of all, consider an Integer class that wraps an integer (like Java's one) and a Foo class with a not-yet-specified constructor:

class Integer {
    int m_value;

public:
    Integer(int v) { m_value = v; }

    int get_value(void) { return m_value; }
};

class Foo {
public:
    Foo(void);
};

Up to this point, everything is correct. Now, given that both classes are usable, we declare two global variables, one of each type:

static Integer Global_Integer(5);
static Foo Global_Foo;

As these are objects, the C++ runtime will call their constructors when initializing them, but we can't predict the order in which this will happen.

Now we define Foo's constructor body, which accesses and prints Global_Integer's value:

Foo::Foo(void)
{
    std::cout << Global_Integer.get_value() << std::endl;
}

With this done, if you add an empty main function and run the program, you should see an integer; it will be probably be 5 (at least with GNU G++ 4.0.1 on a Linux/powerpc box), but it may not. Whichever value you get, try reversing the variable declaration lines, defining Global_Foo first and see if the results change (I get 0 in this box).

Clear? OK, this example is extremely simple, but imagine if this same structure was split among multiple files and classes in a large project... You couldn't easily predict what happens under the hood.

Monotone: Using mini-branches to apply patches

2005-10-08T21:42:30Z

The Monotone VCS provides the concept of mini-branches. A mini-branch is a lightweight branch created inside a formal branch whenever a commit causes "conflicts" with the actual contents of the repository. For example, if your working copy is not up to date and you commit something, you will create a new head within the branch (that is, a mini-branch), that you will later need to (possibly manually) merge with the other head to remove the divergence.

Mini-branches can be used to easily apply externally-provided patches to your software project. Consider the following "collapsed" revision subgraph:

      tag: foo-2.1        tag: foo-3.0
          |                   |
A -> B -> C -> D -> E -> F -> G -> H -> I -> J

As you can see, some development happened in revisions A, B and C, at which point the program was considered stable and the 2.1 release was made. Some time later, and after lots of changes, revision G was tagged as 3.0 and that release was made.

One of this project's users notices a bug in the 2.1 version, tracks it down and fixes it. For whatever reason, he cannot update to 3.0 to see if his changes work with the latest version, so he decides to submit his fix as a patch against 2.1 to the mainstream developers.

So, how do they handle the patch? It will doubtfully apply cleanly to their current code-base, which is far past 3.0. Of course, they can inspect it, adjust it and apply it directly to revision J, but all this process won't be tracked anywhere. Users and developers could later be confused when looking at the original patch and the patch that was really applied — "why were those changes done?".

Here is where mini-branches come to help. The developers ask Monotone to check out a clean copy of C (the same that the user had), ensuring that the patch will apply cleanly. At that point, they apply the fix and commit it to the tree, thus "storing" the original patch file in it. As a result, the revision tree could look like:

      tag: foo-2.1        tag: foo-3.0
          |                   |
A -> B -> C -> D -> E -> F -> G -> H -> I -> J
           \
            \
             K

As you can see, the repository now has two heads (J and K) in the same branch (which can be inspected using monotone heads). J is a lot farther than K in terms of development, but that doesn't matter to the VCS system. Note that, at this point, the revision K carries the code in 2.1 plus the changes submitted by the user verbatim; they still haven't been modified to apply to J, and J is not affected at all by that commit.

Once this is done, and after inspecting why the patch does not apply, the developer decides to merge the heads (monotone merge), thus creating a new revision L that holds all J's code plus the fix added in K:

      tag: foo-2.1        tag: foo-3.0
          |                   |
A -> B -> C -> D -> E -> F -> G -> H -> I -> J -> L
           \                                  /
            \                                /
             K -----------------------------'

Voila! There is now a single head, L, which holds all your code plus the fix sent by the user. Furthermore, the repository has kept track of all the patching process, storing the original and the modified versions of the changes.

Note that this has assumed that revisions are marked as tags rather than as formal branches. Of course, a similar process could be followed if each version was on its own branch (as done with any other VCS).

Games: Half-Life 2

2005-10-06T15:21:44Z

Half-Life 2's Game Of The Year edition was published past Friday. I finally bought it (have been waiting for this since I threw away my illegal copy several months ago); this edition is cheaper than the original game and comes with some goodies.

And just a few minutes ago, I completed it :-) (had started with my old saved game, which was almost at the end). All I can say is that the game is really stunning. If you like FPSs, this is The One.

There is a lot of variety in the game, as opposed to Doom 3, which I find quite repetitive. Each chapter has its own style and suits the story well. There are also multiple weapons and, as you know, the physics are really well done.

Now, it's time to play it again at a higher difficulty level. I think I'll wait until I renew my video card, though. And also, time to play Half-Life: Source, no matter I've already finished the first game three times :-)

C++: Templates and the ABI

2005-10-03T11:24:53Z

In the previous post, we saw why inlined code is dangerous in a public API. Unfortunately, I carelessly put templates in the same bag in an attempt to generalize the idea too much, giving an incorrect impression of how they work. A reader (fellow_traveler) spotted the mistake and here is this new post to clarify what's up with templates and an ABI... if one can relate the two concepts at all.

C++ makes intensive use (or abuse?) of templates to achieve genericity; one can quickly notice this in the STL, where each container is parametrized based on one or more types. Other libraries, such as Boost, go further and use templates in a lot of situations where one couldn't have ever imagined so.

In order to understand what goes on with templates, let's remind how they work. A template defines a piece of code (be it a class or a function) that is parametrized by a type given by the developer. The template does not exist as binary code, because, simply put, that is impossible: it lacks type information to be compiled.

Here is a trivial function that returns the sum of two objects whose type is defined by Type; we will use it to illustrate some examples below. Also, and to focus on the ABI, let's assume that this code is part of a public library; therefore, it must be placed in a header file (e.g., foo.hpp) to be useful to other users (if it were in a foo.cpp file, it'd simply be private and not usable outside that file).

template <class Type>
Type
add(const Type& p1, const Type& p2)
{
    return p1 + p2;
}

When the template is used in someone else's code — in other words, it is instantiated — the compiler grabs the template's source code, fills in the parametrized gaps with the type given by the developer and creates the final object code. For example, given:

int foo = add<int>(2, 3);
float bar = add<float>(2.4, 3.5);

The compiler gets the verbatim add function's code from the header file, replaces Type with int, generates the object code for the resulting function and stores it alongside the user's binary. The same happens with the float instance. Notice how the binary code is not in the library where the template came from, and also notice that the user's binary has gained two new functions, one for each instantiation.

So what happens? Templates cannot take advantage of (binary) shared libraries. Whenever the code in a template changes, the template's user is forced to rebuild his code (if he wants to get the new changes, of course). Imagine that there was a security bug (or any other serious bug) in the template's code: you'd need to make sure to rebuild all its uses to fix the issue, something well-known by users of static binaries.

Of course, the library developer could explicitly instantiate some common types in his library's binary so that the user needn't duplicate the code. This could work in some cases, but as he cannot predict what types will the developer use, this is not a complete solution.

Other developers create templates in a two-layered design. The public template is a very thin wrapper over a private class that achieves genericity by using void * types. This way, the public template is unlikely to change, and the developers can safely change their internal code without requiring external rebuilds. I think I saw this in the STL itself, or maybe in QT, cannot remember.

Summarizing: as my reader said, it makes no sense to talk about templates and ABIs, because a template never has an ABI. It is only an API that, once compiled in third-party code, becomes part of it. I'm now wondering how Java 1.5's templates work or if they suffer from these issues too...

C++: Inlined code and the ABI

2005-10-02T13:54:49Z

There are many development libraries that provide inline functions (or macros) as part of their public API. This is often done for efficiency reasons, although some times it's done because developers don't know the consequences of doing such things (this last thing is just a guess, but it can perfectly happen). Providing such inlined functions breaks the whole idea of encapsulation and shared libraries. Let's see why.

Consider the following simple class:

 /* In foo.h. */

class foo {
    int m_value;

public:
    int get_value(void) { return m_value; }
    void set_value(int v);
    /* ... */
};

/* In foo.c; this is _not_ inlined. */
void
foo::set_value(int v)
{
    m_value = v;
}

Now imagine that this class belongs to a shared library, say libbar.so.1.0. Given this, our Joe user does this in his code, which is perfectly legal:

foo a;
a.set_value(5);
/* ... do whatever with 'a' ... */
int b = a.get_value();

When this code is compiled, the compiler replaces the call to foo::get_value() with the method's code, avoiding a function call, a return and all the stack set up; all the action takes place in the user's code, not in the library. Typically, getting a value from a structure means reading a concrete position of memory within it, described by its offset from the beginning. OTOH, the call to foo::set_value() is correctly made into a regular function call inside the shared library's text.

Some time later, the libbar developers decide to change the internal representation of the foo class for whatever reason. According to the encapsulation principle used in object oriented designs, they should be able to, after all. Let's suppose they add a new integer before the m_value field, called m_id. Unwillingly, the developers have just changed the ABI of their library and, if they don't take care to update the library's major number, seriuos problems will arise. But, why?

Our Joe user again sees a new release of libbar, say 1.1, so he rebuilds and updates it in his machine, replacing libbar.so.1.0 with libbar.so.1.1; these two libraries typically share the same soname, libbar.so.1, because they are compatible in theory. According to how shared libraries work, he oughtn't rebuild his application.

The set_value() call will continue to work correctly because the application will call the new function in the updated shared library. However, the execution of get_value() will be broken; oops! Remember the sample code shown above? It was compiled as an offset within the class, which is now different! This getter will return an incorrect value, no matter what he does. He'll be forced to rebuild his application to adjust to the new ABI.

Conclusion: be very careful when defining inlined methods and macros. If you need to fix a mistake or modify the internal representation of your code in the future, you will be unable to. Personally, I avoid inlined code in all public interfaces, despite this introduces a small performance degradation; however, they are perfectly fine for internal code.

It's a pity that careless C++ developers make so intensive use of such inlined code. BTW, note that although this has focused on C++, the same is true for, e.g., C99, which provides an inline keyword.

Edit (Oct 3rd): Based on this reply, I've removed some (really minor) references to templated code from the article; they certainly didn't belong here.

New utility: verifypc

2005-10-01T13:59:48Z

After five months or so of not touching the code, I've finally cleaned up my verifypc utility and imported it into pkgsrc. Its purpose is to sanity check the dependencies of a given package based on its calls to the pkg-config program.

For more information see the announcement in the tech-pkg@ mailing list or this past post in which I detailed the idea.

Lose vs. loose

2005-09-30T20:55:08Z

Some days ago, I misspelled lose as loose in one of my posts. One of the readers pointed me out the error and suggested to look at this comic strip from Queen of Wands (never read it before). I think I'll never make the same mistake again ;-)

Oh, and BTW, yesterday's strip is kind of funny too; you ought to read it.

Cleaning up a Debian box

2005-09-26T19:27:14Z

I have to confess that, for a long time, I was a big Debian fan; this changed when I switched to the BSDs, but I still think that it is (one of) the best GNU/Linux distribution. This, and because its excellent hardware support, is why it's the main system on my iBook G3.

As a user, it's very common to install lots of packages just to try them, followed by a quick deinstallation when you realize they don't suit your needs. Unfortunately, following common procedures often leaves useless garbage in the file system.

The first annoying thing are installed packages that provide libraries but which aren't used by any other package. These are often called orphan packages and can be listed by using the deborphan utility (which must be installed first). Assuming you have it, do the following to remove all orphaned libraries:

# dpkg --purge $(deborphan)

Note that this in turn may obsolete other libraries, so rerun this command until deborphan prints nothing. Also, please bear in mind that this is dangerous if you have installed software from other means — not from Debian packages — as those libraries may still be used.

Then we have obsolete configuration files. Whenever you remove a package, all its files are removed except the configuration ones, in case you did some modification you want to preserve during an eventual reinstallation. This may be OK in some scenarios but, personally, when I delete something I want it really gone. Here comes dpkg's --purge option to help, which removes all leftovers of a previously installed package.

But how to purge all configuration files from packages that were once installed? Just do the following:

dpkg --purge $(dpkg --list | grep ^rc | awk '{ print $2; }')

This constructs a list of packages which are removed but still have their configuration files installed (denoted by the ^rc pattern) and passes this list to the package management utility to purge all obsolete files.

NFS exports lists rototill

2005-09-23T12:35:34Z

After two weeks of work, the NFS exports lists rototill that I briefly outlined in this past post is finished and committed into NetBSD's source tree. Believe it or not, the whole set of changes was triggered by a XXX mark in mountd(8)'s code (in other words, fixing code marked as such is not always trivial).

In the past, when a file system wanted to support NFS, it had to include two fields in a fixed position of its mount arguments structure due to the broken way in which mountd(8) handled the mount(2) calls. Furthermore, each file system had to deal internally with the NFS exports list, duplicating code among all NFS-aware file systems; this feature was clearly generic, so it had to be placed in an upper generalization level. At last, you had to use the mount(2) system call in a wired way to change the exports.

This was wrong because it broke extensibility and code cleanliness: you couldn't add a new NFS-aware file system without modifying and rebuilding mountd(8). There was also a lot of code duplication and the interface was just ugly from a design point of view. It looked like a quick band-aid over old code to add support for more than one NFS-aware file system.

The changes I just committed have centralized all NFS exports lists handling in a single place and it has been removed from the kernel's core. This stuff is now only compiled in when the NFSSERVER option is enabled, reducing the final kernel size by around 5KB.

The next logical step is to rework mountd(8) to update the whole exports list for a given mount point atomically; this wasn't possible at all before. The new kernel interface, introduced as a new command to the nfssvc(2), has been designed to support this functionality, although it's not used yet due to the big changes required in the userland utility. I won't be working on this anytime soon, as I have other stuff to do before it (hence the entry I added to src/TODO).

If you want to know a bit more, the commit message contains a detailed list of all the changes.

PS: I have to thank Google's Summer of Code 2005 again. I couldn't have found this issue nor had enough knowledge to fix it if I hadn't been working on tmpfs during the summer.

Quality of digital signals

2005-09-22T20:58:14Z

I have seen many times people thinking that a signal has better quality if it's in digital form rather than in analog form, yet this is false. When you convert an analog signal to a digital format, you always lose quality because you are converting a continuous signal to a discrete one. Let's see a little example:

Suppose we want to digitize a 20KHz analog audio signal so that we can store it on disk. Following the Nyquist theorem, we must take a minimum of 40K samples per second; this guarantees that we can recreate the original signal in its exact form (i.e., maximum quality).

The lost of quality arises at this point: how can we represent each sample without losing precision (assuming, of course, that the sample is accurate)? Samples are measured in a continuous scale, but in order to store them we must round their value up or down to a discrete value we can represent (think about representing a real number, which leads to the same problem). We'd need infinite precision to store each sample in its exact form, which means infinite disk space. Not possible.

Note that the example above assumes a PCM modulation technique where each sample is codified and stored in a fixed amount of bits. As you can imagine, any other technique will suffer from this problem.

This post comes after today's SPD class, an optional course I'm taking this semester at university. It's all about the low level details of public data networks. Amazing stuff, IMHO ;-)

Linker's link sets

2005-09-18T12:20:00Z

I don't know about other linkers, but GNU ld provides a very useful feature: link sets. A link set is a list of symbols constructed during link time which can then be inspected in regular code. This is very interesting in situations when you want to initialize several subsystems from a centralized place but don't know which of these will be available; that is, you don't know which ones will be in the final binary.

To illustrate this feature, let's see how NetBSD manages file system initialization. In NetBSD, you can configure which file systems you want in your kernel binary and which not. Each file system has a vfs_init hook that must be called during system initialization but... how to do this in a clean and extensible way? For a moment, imagine we didn't have link sets. In this case, NetBSD would need a function similar to the following:

void
init_file_systems(void)
{
#ifdef UFS
    ufs_init();
#endif
#ifdef MSDOSFS
    msdosfs_init();
#endif
...
#ifdef TMPFS
    tmpfs_init();
#endif
}

Isn't this ugly? Very much, although one might support the visual ugliness. However, the real problem is that, in order to add a new file system, you need to edit other parts of the system; this might seem acceptable in an open source project, but what if the initialization code was closed source? You couldn't add your file system. This clearly goes against extensibility.

Using link sets, this problem goes away: each independent module is able to add itself to a global list of symbols. Continuing our file systems example, take a look to the vfsinit function in kern/vfs_init.c. The first thing you see there is the declaration (__link_set_decl) of a new link set, called vfsops, that will hold pointers to struct vfsops objects. Then, the code uses the __link_set_foreach construction to iterate over the list, correctly initializing all the file systems. The following is an extract of this, with many details removed:

void
vfsinit(void)
{
    __link_set_decl(vfsops, struct vfsops);
    struct vfsops * const *vfsp;  /* Iterator */

    __link_set_foreach(vfsp, vfsops) {
        (*vfsp)->vfs_init();
    }
}

At last, how does a module add itself to the link set? Using the __link_set_add_data operation, which takes the name of the link set as its first argument and the object to be added as the second one. E.g.:

struct vfsops barops = {
    ...
    ufs_init,
    ...
};
__link_set_add_data(vfsops, barops);

As you can see, using link sets the kernel does not need to know at all which file systems are compiled into it, yet the initialization code will be executed.

Interface to change NFS exports

2005-09-11T11:48:28Z

While adding NFS support to tmpfs, I found how NetBSD currently manages NFS exports from userland. The interface is, IMHO, scary. As NetBSD aims for clean code and design, it must be fixed.

See my mail to the tech-kern@ mailing list for more details on the issue and a preliminary patch.

tmpfs: Project merged into NetBSD

2005-09-10T21:13:54Z

After listening to many queries from developers asking when tmpfs could be integrated into NetBSD, I finally imported the code into the CVS repository. I'm really happy about this :-) Development will be simplified from now on and it will be a lot easier for interested parties to test the code. Please read the announcement for more information.

I'd like to comment now some of the improvements I've been doing during the past days, which mostly addressed optimization. I started by removing the storage of . and .. directory entries from directories, generating them on the fly when requested. This was done for three reasons. First, to remove redundancy: as . points to the directory itself, the entry can always be generated; similarly, .. can be generated from the pointer to the parent directory stored in the nodes. Secondly, to simplify the code: there were multiple assertions to ensure that these entries were correct and there was code to update them on file-system changes; this code was hard to understand and, as you can see, avoidable. Lastly, to reduce memory consumption: this removed around 1KB of storage from each directory (given to a change I've done recently, this gain is lower, because entries are now a lot smaller, but still this saves 40 bytes from each directory).

Then I fixed a long-standing and very important issue: the assignment of unique node numbers and generation numbers to nodes. Among other reasons, this was done to be NFS-friendly. See my previous post on NFS file handles for more details.

At last, I removed fixed-size string buffers from nodes and directory entries, replacing them with dynamically allocated memory from a string pool. Strings are now stored more efficiently and waste far less space. Note that, before this change, each directory entry occupied around 255 bytes and each node took more than 1KB. As an example, the simple creation of 10K empty files previously required around 20MB of memory, whereas it now needs 1.5MB. This should be reduced even more, but, as you can see, this is a lot better than before.

And of course, several bug fixes and miscellaneous improvements were done all around.

NFS file handles

2005-09-10T09:31:26Z

NFS uses a structure called a file handle to uniquely identify exported files. When a client requests to access a file, the server constructs a file handle that identifies it; from this point on, this identifier will be used in all communications between the server and the client to access that specific file. (Look at the fs_vptofh and fs_fhtovp hooks in NetBSD's VFS layer to see how this mapping works.)

In order to identify a file univocally, you need three things: the file-system identifier (i.e., something unique to each mount point), the node number (e.g., the inode number in FFS terminology) and a generation number. It is clear why the first two are needed, but the generation number may be confusing. In order to explain this last concept, let's see an example:

Suppose you have a file using the 1234 inode of the root file-system; let's call this foo. Now, you delete foo and create a new file, called bar, which ends up reusing the 1234 inode. These two files are clearly different, but they could look the same from the point of view of a file handle if you used the file-system number and the inode number exclusively. Therefore, we need a third component, the generation number, that makes these two files different. This component is incremented each time an inode is reused for a different file.

So far, so good. But now we get to the ugly part: given a file-system number, a node number and a generation number, you can generate a file handle that refers to any file within the exported file-system. If these values were available to everybody, a malicious client could easily generate file handles to access files he is not allowed to touch. This is why the generation number is hidden to user space: if you apply stat(2) to a file, you'll always get a zero value in the st_gen field (some systems may return the correct number if you are root). Similarly, the getfh(2) system call is restricted to the super-user.

But can't a user guess the generation number? Yes, he can. This is why they are often generated based on a random seed, which makes the guessing difficult. See fsirand(8) for more information.

All this explains why there is no such thing as exporting a subtree in the NFS world. When you export a directory from a file-system, you are implicitly exporting the whole mount point because malicious file handles could be used to access files outside the subtree.

Thanks go to William Studenmund for explaining me this during tmpfs development.

FFS, LFS and MFS

2005-09-09T09:05:21Z

The designers of the Fast File System (FFS) organized it in two different layers aiming for flexibility and code reuse; this happened in 4.4BSD (or maybe earlier, I don't know for sure). The upper layer communicates with the VFS and vnode code while the lower layer accesses the underlying devices and only receives requests from the upper layer. It goes like this:

VFS/vnode calls <-> FFS upper layer <-> FFS lower layer <-> backing device

The upper layer handles all logical details of a Unix file-system, abstracting, among many other things, inode representation and management, tree manipulations and verification of credentials. It also lays out data in different blocks and passes them to the lower layer.

The lower layer receives the blocks generated by the upper layer and lays them out on the physical device (e.g., a disk). It is also responsible of reading the blocks requested by the upper layer and transferring them to it.

Thanks to this design, three different lower layers were implemented, providing three file-systems:

Fast File System (FFS): It assumes that the average file size is small and that some files (those in the same directory) will be accessed all at once. Given these assumptions, it divides the disk surface in different cylinder groups and lays out blocks that may be accessed together in the same group. This reduces expensive seeks.
Log-structured File System (LFS): It assumes that the system bottleneck are writes to the disk, so it packs all writes in a concrete area of the disk, treating the overall space as an append-only log (no seeks required).
Memory File System (MFS): Reserves a contiguous region of memory and lays out blocks in it. This is why MFS is considered inefficient, because it is using the concepts of an on-disk file-system over memory.

The different layers are clearly visible in the source code. The src/sys/ufs/ufs directory contains the upper layer's code while src/sys/ufs/ffs, src/sys/ufs/lfs and src/sys/ufs/mfs contain the three respective lower layers.

Note that the above is an over-simplification of the Real Thing (TM). If you want detailed information, read Chapter 6 of The Design and Implementation of the 4.4 BSD Operating System.

SoC: The end

2005-09-01T14:58:23Z

So... the deadline for Google's Summer of Code 2005 program arrived some time between yesterday and today (don't know exactly due to timezones). The final results from my side: a functional memory file-system (not efficient yet) for NetBSD named tmpfs, as well as the beginnings of a book/article on file-system development under NetBSD.

As regards the file-system itself, its code can be found in the CVS repository. Despite it has some bugs and misfeatures, it is functional from the user's point of view. The file-system's code is around 4000 lines (plus 500 from the mount utility) and the regression test suite around 2000 (half of which are license texts). I know these numbers are low, but man... this is the hardest code I've ever written (mostly due to lack of documentation and having to reverse engineer existing stuff). I've also written a document describing tmpfs' internals (as said in the initial proposal), which is available in the form of a manual page in the repository and is around 700 lines long.

With respect to the documentation, it is not yet available online. I'm still discussing with my mentor (Bill Studenmund) how it shall be published and I don't like it in its current form to make it public (still has many mistakes and is heavily disorganized... and is already 2500 lines long). It looks like we both agree that it should be in the form of a (online) book, leaving room for future additions (e.g., UVM, device drivers, etc.). However, I will also try to write a little article explaining all the steps I followed to write tmpfs. Writing an independent article will be better because it'll let me focus on a single task and it'll be possible to mix information from several subsystems without making it look messy. But there is still a long way to go to have these done.

I would like to make it clear that despite the program has ended, I will continue working on all this stuff until it is ready to be integrated into NetBSD's source tree; all this work musn't have been in vain! Oh, and having been part of this program has been great. It has forced me to work on something I've always wanted to do (kernel development) but never found time to learn about it... and I've done paid development for the best free operating system! ;-) I have to say that I've learned a damn lot in these two months.

And today, just as a little prize for the program's completion, I've bought myself a copy of Doom 3 (because I decided to go legal a while ago, yay!) and this month's Dr. Dobb's Journal edition. This is the first time I buy this magazine. Why? They will be publishing a set of articles describing SoC projects, so I want to know about their style — which, as far as I've seen, is very technical, something that is nice. Furthermore, this edition has some articles that I found interesting when I first saw them in the web site; i.e., C++, STL, and custom pool allocators and C++ exceptions and the Linux kernel.

A prompt that does not get in the way

2005-08-31T12:19:01Z

Yesterday, I saw a screenshot of a developer's box. His prompt was composed of two characters only: a colon and a semicolon (:;). This looks wired, doesn't it? Yes, it does, but there is a rationale behind it: using this prompt, you can copy and paste complete lines from a terminal into another one and they will just work. Why? Because those characters are ignored by the shell (the colon returns success and the semicolon starts another command).

Oh yes, this prompt does not provide useful information, but that's a matter of preference. E.g., I used a simple $ prompt for a long while, so :; is not any worse than it.

SoC: Project announced

2005-08-26T09:23:17Z

Despite I don't like doing premature announcements of my projects, I've been kind of forced to do it for tmpfs. The reason is that SoC's deadline is really close now and people should have a chance to test it. Not to mention that the code won't suffer any serious improvements in the subsequent days, so delaying the announcement is not worth it either.

You can read the announcement in my mail to the tech-kern@ mailing list, which also includes a step-by-step guide to test tmpfs.

pkgsrc's strengths

2005-08-24T22:07:49Z

Jeremy C. Reed has collected an excellent list of pkgsrc's strengths. I encourage you to read his post to tech-pkg@ in case you still had doubts about why to use pkgsrc ;-)

His mail contains also some bad things about this packaging system, although the list is very short. Followups to this email list some more items on the negative side.

Local sockets' permissions

2005-08-24T12:58:53Z

A few days ago, I was trying gamin under NetBSD which unfortunately didn't work at all. The first problem I encountered was that it complained about the excessive permissions given to newly created local sockets (those stored in the file-system, also known as "Unix sockets" historically). After analyzing the issue, I saw that those files were given 777 permissions, regardless of the user's umask. Strangely, the code was explicitly checking for this mode after creation, so I was probably missing something.

I wrote a little test program that creates a local socket and ran it under Linux, FreeBSD and OpenBSD. All of them correctly respected the user's umask (setting the permissions to 755). So, what was going on with NetBSD?

After asking in the tech-kern@ mailing list, I was told the following: the traditional Unix behavior when creating local sockets was to give them 777 permissions to mimic real sockets (i.e., everybody can connect to them). Therefore, the portable way to create them in a secure way is to first make a directory with safe permissions (say, 700) and then create the socket inside it.

However, I'm thinking about changing NetBSD to honor the user's mask in this case too (it's a trivial fix). It does not hurt in any way and it may improve portability of some "non-portable" programs. (Though, this will hide portability bugs in some programs, which makes me dubious about the change...)

By the way, gamin was doing the right thing (creating sockets inside secure directories) so I don't know why it wanted to ensure that the sockets were "safe".

SoC: Status report 6

2005-08-23T16:17:14Z

This past week has not been excessively productive because I spent some time dealing with long overdue pkgsrc tasks (mainly updating GNOME to 2.10.2, the latest stable version) and was away from computer more time than usual. Anyway, I have done a bunch of things, although they are not as visible as the work from other weeks (this is, in part, why I felt less productive).

I started by adding support for local sockets, which was easy enough to do but caused panics (specially when switching /tmp to tmpfs and starting an X session).

Then I tried to switch memory pools to use anonymous memory, so that all file meta-data was pageable. I did several clumsy tests and, while it mostly worked, the system crashed after intensive usage of the file-system (such as decompressing pkgsrc over it). I had to give this up because I had some other stuff to do (see below) and this was consuming much time.

The next thing I did was to rework the way pools were handled by making them file-system specific and by implementing a tmpfs-specific extension of them. There are several benefits from this change, but the most important one is that tmpfs is now able to control memory usage in a fine-grained manner.

Later on I added NFS support by defining the VFS operations I was missing. This was harder than I thought, mostly because the lack of documentation, but after some time I got something that basically works. There are still some some serious issues, so I'd not call tmpfs as NFS-ready yet. I'm afraid that these problems are not specific to NFS; i.e., NFS is only exposing some real bugs in the code. We'll see.

Related to the previous, I was disappointed to see that I had to modify mountd's code to be able to export tmpfs file-systems. I have some ideas to improve this, thus making mountd file-system independent, but I'll delay them until SoC's deadline has passed.

Furthermore, I cleaned up some parts of the code: split the different vnode operations vectors into different source files (tmpfs_vnops.c, tmpfs_fifoops.c and tmpfs_specops.c) for better readability and to get shorter files, split the main tmpfs.h file into several headers, cleaned up inclusions and centralized some code.

At last, I wrote the document describing tmpfs internals. This is in the form of a manual page in the 9th section, and can be found in the CVS. If you want to visualize it, you can run the following command: groff -Tascii -mdoc tmpfs.9 | less after fetching the file. It may lack some details, but I feel it is quite complete, though it assumes prior knowledge of how file-systems work within NetBSD.

What I'm going to target this week (after prior agreement with my mentors) is the other so-promised document I've been talking about: a complete document that describes how to write file-systems for NetBSD. I started it yesterday, and it is proving to be very difficult to explain things in a clean and well-structured way (mostly because I want to avoid big code examples without real explanations).