I'm writing a kernel (only 5-10% complete a.t.m), and I have been writing some modules for it as well (for filesystem, window management, network I/O, etc) and have been working on system calls and callable functions. I am now wondering which is the most secure design to go for. The three designs I am considering are (in preferrential order):

1. Modular Monolithic
2. Hybrid
3. Microkernel

I did not choose solid monolithic becaise changing one module means changing the entire kernel and recompiling the whole kernel image. The exokernel is also phased out for me as I cannot make such a design secure enough for my purpose (exokernels allow direct hardware access!).

Can someone explain why a microkernel would be more secure?

Still waiting...

Microkernels are said to be more secure because the different functions are seperated - If a malicious program finds a fault in one piece of code, it cannot exploit that fault to cause trouble elsewhere. In a microkernel the different parts could be at different addresses, depend on different APIs, etc. In a monolithic kernel, all the code has access rights to all the other code, so one bug could be used by a hacker in any number of ways.

I hope that helps.

What about a hybrid kernel? How come so many OSes are using hybrid kernels?

So that phases out "Modular Monolithic" as an option

According to FreeBSD Unleashed, Linux and Windows 98 are monolithic kernels in which the code is more optimized for performance and to minimize context switching. This makes for code that's cleaner and easier for developers to maintain, but the administrator is required to recompile the kernel every time support for a new device is added. (It goes on to mention there are differences in that statement today)

A microkernel is fairly small and modular and new devices can be more frequently added but the differences with monolithic are more a philosophical one than anything having to do with devices.

The biggest and most fundamental difference is that a microkernel has a mechanism for passing certain non-core system calls to a user level process rather than handling it internally. This keeps only essential functions in the kernel and makes it more robust, manageable and understandable. It also provides a measure of inherent security.

Although the ideal of total modularity is all well and good, a kernel in which every kind of device and option is modular and loadable during runtime is as theoretical as perpetual motion.

Parts of the above were edited by me for brevity.