ASST0: Introduction to OS/161

After completing ASST0 you should:

Another assignment objective is to get you working efficiently in a Linux development environment using standard command-line development tools. Unlike many of the things you will find online ASST0 is not a tutorial. You will go slower, but you will learn a lot more from figuring out how to: "Run the first OS/161 thread test and then shut down your kernel" than how to:

Following these kinds of written instructions will also better prepare you to interact with other OS/161 hackers in person and in the online forum.

Another objective of this course is to help you learn to use common development tools, particularly Git. However, like most programming tools the best way to learn Git is through the extensive online documentation and multiple good online tutorials (1, 2, etc.).

As a result, we have removed Git coverage from this introduction. Any documentation we could provide is unlikely to be complete and up-to-date, and this will also provide you with good practice in online learning. Feel free to ask your course staff or other students for help, either in person or using the forum. Chances are many of them use Git on a daily basis.

Like anything else, learning Git takes practice, more practice, and patience. But here’s one tip: read the output generated by Git commands. Git produces remarkably useful output, particularly when an error occurs or when it didn’t accomplished what you wanted. Frequently the output tells you exactly what to do to fix the problem—​down to the commands that you should type.

ASST0 focuses on getting your environment set up and familiarizing you with the source code you will be modifying. There is a small amount of coding involved, but not too much. Here are guidelines for how to work with your partner—​if you have one—​and with other students:

We distribute the ops-class.org OS/161 source code using Git. Starting with a clone of our repository makes it easy for us to distribute updates, which can be merged easily into your repository.

First, choose a directory to work in:

Let’s say you’ve chosen a directory called src, which should either not exist or (in the case of the Vagrant VM) be empty. Clone our ops-class.org Git repository. into that directory. Note that we refer to this directory as your source directory. In contrast, your root directory is where your built kernel and other binaries are installed and where you run sys161.

The next step is to configure the OS/161 sources by running the configure command located at the base of your source tree. You need to do this (very short) step only when you completely remove your source tree for some reason. The only configuration step is to set up where various binaries—​including system executable and your kernel—​will be created when you run make in later steps. Run configure --help to find out more including available command line options.

Note that by default OS/161 installs things to the root directory $HOME/os161/root, which is a fine plan to put things if you are working on a shared machine. For our dedicated VM we use $HOME/root to shorten the directory paths a bit, but this requires that you use the --ostree argument to configure. If you forget this argument either now or when you need to rerun configure later, you will install things into $HOME/os161/root. This has caused confusion for some students previously, so please be careful.

The kernel sources for OS/161 are in the kern subdirectory, which has its own configuration script. Change into kern/conf and look around. You should notice a configuration script, a base configuration file (conf.kern), and four configurations that include conf.kern.

You should take a look at conf.kern and one of the configurations to get a sense of the format. But for now, the only thing we’re concerned about is ensuring that we enable dumbvm for this assignment. You’re going to write a full-fledged virtual memory (VM) system in ASST3, but for the first few assignments dumbvm provides enough of a "dumb" VM to allow you to proceed. Configure a kernel now with dumbvm enabled.

The OS/161 kernel configuration process sets various options that control how your kernel gets built, so unlike the configuration step above you will probably need to modify these files at some point during later assignments. In particular, conf.kern determines what source files get included in your kernel build, so if you add sources to the kernel you’ll need to add them to conf.kern as well.

Once you’re successfully configured your OS/161 kernel you should have a directory to compile in, as well as a reminder about a build step that you might forget. Once you change into that directory you are ready to build a kernel!

One important note before you start. You are probably used to using GNU make to build software on UNIX-like systems. However, the OS/161 sources use BSD make, which has a different Makefile syntax 3. To avoid confusion, BSD make has been installed on your system as bmake. So while you might normally run make clean to reset your build and remove all of the build targets, when working with OS/161 you would run bmake clean.

There are three steps:

Run these three commands now and check that they complete successfully. Then change into your root directory and look around. You should see a fresh kernel. If you don’t, review the steps above until your kernel builds successfully.

Now that you have a kernel, the next step is to run it. But how? Given that your kernel doesn’t yet have any useful features, it would be impossible to run it on real hardware, or even in a fully-featured VM like VirtualBox.

Instead, OS/161 kernels are built to be run by a special-purpose system simulator called sys161, or System/161. Compared to other VMs or full-system simulators, sys161 is simpler and faster but retains enough realism to enable OS/161 kernel development. Apart from floating point support and certain issues relating to RAM cache management, it provides an accurate emulation of a MIPS processor 4.

Unlike OS/161, we do not expect you to modify sys161. However, you do need to configure the simulated machine that sys161 provides by choosing the number of simulated CPU cores, the amount of simulated memory, and the number of simulated disk drives. Here is a sys161.conf file that you can use to get started. You should read and understand the structure of this configuration file so that you can modify it as needed in later assignments.

Now that you have a kernel and a sys161 configuration file you should be ready to go. Fire up your kernel and see what happens. Poke around a bit at the menu. Run a test or two. And then shut down.

What just happened? You ran one computer program (sys161) that loaded your kernel (from the kernel) file. Your kernel is itself a program expressed as a series of MIPS r3000 instructions, which were interpreted by sys161 and simulated as if they had executed on real hardware. Of course, this includes the ability read from and write to a console device, allowing you to interact with your running kernel.

Examine the output produced by your kernel as it boots and shuts down. You should be able to answer the following questions:

Before going on try the following exercises:

As you saw above, building an OS/161 kernel from scratch involves five steps:

The first step only needs to be done when you download a new OS/161 source tree. The second step only needs to be done when you start a new assignment or add files to your kernel build by editing kern/conf/conf.kern. Rebuilding the dependencies in step three is also not usually necessary unless you have reconfigured your kernel.

So that leaves the last two (bmake ; bmake install) as your normal kernel development workflow. Note that bmake is usually smart enough to detect what you changed and not recompile things unnecessarily, but if you think that it hasn’t accomplished that correctly you can always run a bmake clean to force it to start over.

However, if things aren’t building properly you may want to rerun the kernel configuration and dependencies steps just to be sure. Step 1 is almost never necessary to repeat unless you’ve completely started over and removed your entire previous OS/161 source tree.

If you change into your root directory you should see only a few files, including your compiled kernel, a symbolic link pointing to that kernel, and the sys161.conf file that configures sys161. That’s fine for now, and all your need until ASST2.

But your OS/161 kernel would not be very interesting or useful if it couldn’t run user programs. So let’s build those now. Head over to the base of your source directory and run bmake followed by bmake install. This generates a lot of output, but when it’s done return to your root directory. You should see a directory structure including bin and testbin directories containing cross-compiled user binaries that your OS/161 kernel will eventually be able to run.

Note that building the user space tools is not part of the kernel development cycle. Unless you modify or add tests in the userland subdirectory of your source directory—​which you are encouraged to do—​you should not need to rebuild or reinstall these binaries. And until you begin ASST2 your kernel can’t run user binaries anyway, so this part of the build process is completely useless. We only point this out because compiling and installing the user space tools takes enough time to be annoying, so don’t let it slow you down unnecessarily. Focus your development loop on your kernel.

Your OS/161 source directory contains the following files:

The source directory contains the following subdirectories:

If you have previously configured and built in this directory there are also some additional files and directories that have been created, such as defs.mk and build/.

In the userland/ source subdirectory, you will find:

You don’t need to understand the files in userland/bin/, userland/sbin/, and userland/testbin/ now, but you certainly will later on. Eventually, you will want to modify these or write your own utilities and these are good models. Similarly, you need not read and understand everything in userland/lib and userland/include but you should know enough about what’s there to be able to get around the source tree easily. The rest of our code walk-through is going to focus on kern/.

Now let’s navigate to the kern/ source subdirectory. Once again, there is a Makefile. This Makefile installs header files but does not build anything. In addition, we have more subdirectories for each component of the kernel as well as some utility directories and configuration files.

Use your new-found knowledge of the OS/161 source code to answer the questions that follow. You may also find standard UNIX utilities like find and grep useful when searching through your OS/161 source code.

While we don’t specifically support any particular editing or code browsing software, we suggest that you use an editor that is designed for working with source code. Command line tools like vim work well and, when combined with ctags and tmux, produce a very powerful command-line development environment. Graphical editors like Eclipse or Visual Studio are also fine but more of a hassle to set up. Decide what works best for you.

When you read and begin to modify source code, you will also want to absorb it’s stylistic conventions. Like any other hacker, David Holland has his preferences about indentation, line width, function signatures, where to put braces, and tabs versus spaces. Adopting these will make it much easier to work on OS/161 and ensure that your changes fit in. This is also a critical skill to learn as you begin to contribute to other shared code bases. At minimum, you and your partner should agree on style so that you don’t drive each other crazy or spend hours reformatting each others' code.

As described previously, we are not going to go into the use of Git in detail. But we will point out that using Git is not optional for completing the ops-class.org assignments. We distribute our base sources using Git and will use Git to push updates to you. Our testing tool, test161, submits assignments for testing using Git.

One additional requirement is that you have a private Git repository so that you do not inadvertently share your solutions with others. The test161 submission system will refuse to grade your assignments if it detects that your Git repository is public. Getting your hands on a private Git repository is not hard. If you are a student, GitHub will allow you to set up a limited number of private repositories for instructional use. GitLab.com provides private Git repositories for free, as do other sites such as BitBucket. And your course staff may also set up private Git repositories for you to use.

If you are completing the ops-class.org assignments alone, you may wonder whether you need Git. The short answer is yes. Every programmer, including you, should get in the habit of setting up version control every time you start a project. It’s the first thing that you should do. Always. Every time. No exceptions. Why? There are a lot of reasons. Google them.

GDB—​or the GNU debugger—​is another extremely well-documented tool which we will let you learn on your own. Unlike Git, GDB is not required to complete the ops-class.org assignments. But that’s like saying that shoes aren’t required to climb Mt. Everest. You can make it without GDB, through good old printf debugging and pure deductive reasoning. But it will be very, very painful. You will be much happier if you learn to use GDB.

The only complication to using GDB to debug your OS/161 kernel is that the machine simulator sys161 gets in the way. As a result, the way that you hook up the debugger to your running kernel is a bit different than you might be used to if you have used GDB previously. For example, if you try this in your root directory:

you will end up debugging sys161, not your kernel, which is not what you want. And if you run

nothing will happen at all because you haven’t started the sys161 simulator required to run your kernel.

Instead, you need to start the simulator and the debugger separately. However, it is critical that they run in the same directory. A terminal multiplexer like tmux comes in handy here. Here’s what to do:

Unfortunately, you are not quite done. You may have noticed that the kernel is still waiting for a debugger connection. To establish that connection, type the following at the GDB prompt:

At this point GDB should confirm that it is connected to the sys161 simulator and you can proceed. Note that the kernel is stopped at this point as if you have set up a breakpoint, so you need to instruct it to continue.

Happily, new versions of sys161 will wait explicitly at shutdown for a debugger connection if something goes wrong. Try booting your kernel and running the panic command to observe this behavior. This gives you a chance to connect a debugger and poke around in cases where your kernel panics and you weren’t expecting it. That said, we suggest that you always run your kernel with the debugger attached from boot.

If you get tired of typing these commands, there are ways to set up a GDB alias for the target command and have it be run when GDB starts. For the even more adventurous, you can set up a tmux script that will automatically create two windows, boot your kernel in one and start the debugger in the other. Programming FTW!

Finally, note that because GDB is debugging your kernel through the sys161 system simulator, not all GDB features are supported. Watchpoints, for example, are known not to work. In addition, when stepping through code you should keep in mind that your kernel is multithreaded and that other threads may have run in-between each step.