Kernel Subsystems

The choice of boundary crossing methods depends largely on the part of the kernel into which you are adding code. In particular, the boundary crossing method preferred for the I/O Kit is different from that preferred for BSD, which is different from that preferred for Mach.

If you are writing a device driver or other related code, you are probably dealing with the I/O Kit. In that case, you should instead read appropriate sections in IOKit Fundamentals, Accessing Hardware From Applications, and IOKit Device Driver Design Guidelines.

If you are writing code that resides in the BSD subsystem (for example, a file system), you should generally use BSD APIs such as syscall or sysctl unless you require high bandwidth or exceptionally low latency.

If you are writing code that resides anywhere else, you will probably have to use Mach messaging.

Bandwidth and Latency

The guidelines in the previous section apply to most communication between applications and kernel code. The methods mentioned, however, are somewhat lacking where high bandwidth or low latency are concerns.

If you require high bandwidth, but latency is not an issue, you should probably consider doing memory-mapped communication. For large messages this is handled somewhat transparently by Mach RPC, making it a reasonable choice. For BSD portions of the kernel, however, you must explicitly pass pointers and use copyin and copyout to move large quantities of data. This is discussed in more detail in Memory Mapping and Block Copying.

If you require low latency but bandwidth is not an issue, sysctl and syscall are not good choices. Mach RPC, however, may be an acceptable solution. Another possibility is to actually wire a page of memory (see Memory Mapping and Block Copying for details), start an asynchronous Mach RPC simpleroutine (to process the data), and use either locks or high/low water marks (buffer fullness) to determine when to read and write data. This can work for high-bandwidth communication as well.

If you require both high bandwidth and low latency, you should also look at the user client/device interface model used in the I/O Kit, since that model has similar requirements.

Using Well-Defined Ports

Before you can use Mach IPC for task communication, the sending task must be able to obtain send rights on the receiving task’s task port. Historically, there are several ways of doing this, not all of which are supported by OS X. For example, in OS X, unlike most other Mach derivatives, there is no service server or name server. Instead, the bootstrap task and mach_init subsume this functionality.

When a task is created, it is given send rights to a bootstrap port for sending messages to the bootstrap task. Normally a task would use this port to send a message that gives the bootstrap task send rights on another port so that the bootstrap task can then return data to the calling task. Various routines exist in bootstrap.h that abstract this process. Indeed, most users of Mach IPC or Mach messaging actually use Mach remote procedure calls (RPC), which are implemented on top of Mach IPC.

Since direct use of IPC is rarely desirable (because it is not easy to do correctly), and because the underlying IPC implementation has historically changed on a regular basis, the details are not covered here. You can find more information on using Mach IPC directly in the Mach 3 Server Writer’s Guide from Silicomp (formerly the Open Group, formerly the Open Software Foundation Research Institute), which can be obtained from the developer section of Apple’s website. While much of the information contained in that book is not fully up-to-date with respect to OS X, it should still be a relatively good resource on using Mach IPC.

Remote Procedure Calls (RPC)

Mach RPC is the most common use for Mach IPC. It is frequently used for user-kernel communication, but can also be used for task to task or even computer-to-computer communication. Programmers frequently use Mach RPC for setting certain kernel parameters such as a given thread’s scheduling policy.

RPC is convenient because it is relatively transparent to the programmer. Instead of writing long, complex functions that handle ports directly, you have only to write the function to be called and a small RPC definition to describe how to export the function as an RPC interface. After that, any application with appropriate permissions can call those functions as if they were local functions, and the compiler will convert them to RPC calls.

In the directory osfmk/mach (relative to your checkout of the xnu module from CVS), there are a number of files ending in .defs; these files contain the RPC definitions. When the kernel (or a kernel module) is compiled, the Mach Interface Generator (MIG) uses these definitions to create IPC code to support the functions exported via RPC. Normally, if you want to add a new remote procedure call, you should do so by adding a definition to one of these existing files. (See Building and Debugging Kernels for more information on obtaining kernel sources.)

What follows is an example of the definition for a routine, one of the more common uses of RPC.

routine thread_policy_get(

                    thread      : thread_act_t;

                    flavor      : thread_policy_flavor_t;

            out     policy_info : thread_policy_t, CountInOut;

            inout   get_default : boolean_t);

Notice the C-like syntax of the definition. Each parameter in the routine roughly maps onto a parameter in the C function. The C prototype for this function follows.

kern_return_t thread_policy_get(

    thread_act_t            act,

    thread_policy_flavor_t  flavor,

    thread_policy_t         policy_info,

    mach_msg_type_number_t  *count,

    boolean_t               get_default);

The first two parameters are integers, and are passed as call-by-value. The third is a struct containing integers. It is an outgoing parameter, which means that the values stored in that variable will not be received by the function, but will be overwritten on return.

From there it becomes more interesting. The fourth parameter in the C prototype is a representation of the size of the third. In the definition file, this is represented by an added option, CountInOut.

The MIG option CountInOut specifies that there is to be an inout parameter called count. An inout parameter is one in which the original value can be read by the function being called, and its value is replaced on return from that function. Unlike a separate inout parameter, however, the value initially passed through this parameter is not directly set by the calling function. Instead, it is tied to the policy_info parameter so that the number of integers in policy_info is transparently passed in through this parameter.

In the function itself, the function checks the count parameter to verify that the buffer size is at least the size of the data to be returned to prevent exceeding array bounds. The function changes the value stored in count to be the desired size and returns an error if the buffer is not large enough (unless the buffer pointer is null, in which case it returns success). Otherwise, it dereferences the various fields of the policy_info parameter and in so doing, stores appropriate values into it, then returns.

In addition to the routine, Mach RPC also has a simpleroutine. A simpleroutine is a routine that is, by definition, asynchronous. It can have no out or inout parameters and no return value. The caller does not wait for the function to return. One possible use for this might be to tell an I/O device to send data as soon as it is ready. In that use, the simpleroutine might simply wait for data, then send a message to the calling task to indicate the availability of data.

Another important feature of MIG is that of the subsystem. In MIG, a subsystem is a group of routines and simpleroutines that are related in some way. For example, the semaphore subsystem contains related routines that operate on semaphores. There are also subsystems for various timers, parts of the virtual memory (VM) system, and dozens of others in various places throughout the kernel.

Most of the time, if you need to use RPC, you will be doing it within an existing subsystem. The details of creating a new subsystem are beyond the scope of this document. Developers needing to add a new Mach subsystem should consult the Mach 3 Server Writer’s Guide from The Open Group (TOG), which can be obtained from various locations on the internet.

Another feature of MIG is the type. A type in MIG is exactly the same thing as it is in programming languages. However, the construction of aggregate types differs somewhat.

type clock_flavor_t     = int;

type clock_attr_t       = array[*:1] of int;

type mach_timespec_t    = struct[2] of int;

Data of type array is passed as the user-space address of what is assumed to be a contiguous array of the base type, while a struct is passed by copying all of the individual values of an array of the base type. Otherwise, these are treated similarly. A “struct” is not like a C struct, as elements of a MIG struct must all be of the same base type.

The declaration syntax is similar to Pascal, where *:1 and 2 represent sizes for the array or structure, respectively. The *:1 construct indicates a variable-sized array, where the size can be up to 1, inclusive, but no larger.

Calling RPC From User Applications

RPC, as mentioned previously, is virtually transparent to the client. The procedure call looks like any other C function call, and no additional library linkage is needed. You need only to bring the appropriate headers in with a #include directive. The compiler automatically recognizes the call as a remote procedure call and handles the underlying MIG aspects for you.

General Information on Adding a sysctl

When adding a sysctl, you must do all of the following first:

• add the following includes:

  #include <mach/mach_types.h>

  #include <sys/systm.h>

  #include <sys/types.h>

  #include <sys/sysctl.h>

• add -no-cpp-precomp to your compiler options in Project Builder (or to CFLAGS in your makefile if building by hand).

Adding a sysctl Procedure Call

Adding a system control (sysctl) was once a daunting task requiring changes to dozens of files. With the current implementation, a system control can be added simply by writing the appropriate handler functions and then registering the handler with the system at runtime. The old-style sysctl, which used fixed numbers for each control, is deprecated.

The preferred way of adding a sysctl looks something like the following:

SYSCTL_PROC(_hw, OID_AUTO, l2cr, CTLTYPE_INT|CTLFLAG_RW,

    &L2CR, 0, &sysctl_l2cr, "I", "L2 Cache Register");

The _PROC part indicates that you are registering a procedure to provide the value (as opposed to simply reading from a static address in kernel memory). _hw is the top level category (in this case, hardware), and OID_AUTO indicates that you should be assigned the next available control ID in that category (as opposed to the old-style, fixed ID controls). l2cr is the name of your control, which will be used by applications to look up the number of your control using sysctlbyname.

CTLTYPE_INT indicates that the value being changed is an integer. Other legal values are CTLTYPE_NODE, CTLTYPE_STRING, CTLTYPE_QUAD, and CTLTYPE_OPAQUE (also known as CTLTYPE_STRUCT). CTLTYPE_NODE is the only one that isn’t somewhat obvious. It refers to a node in the sysctl hierarchy that isn’t directly usable, but instead is a parent to other entries. Two examples of nodes are hw and kern.

CTLFLAG_RW indicates that the value can be read and written. Other legal values are CTLFLAG_RD, CTLFLAG_WR, CTLFLAG_ANYBODY, and CTLFLAG_SECURE. CTLFLAG_ANYBODY means that the value should be modifiable by anybody. (The default is for variables to be changeable only by root.) CTLFLAG_SECURE means that the variable can be changed only when running at securelevel <= 0 (effectively, in single-user mode).

L2CR is the location where the sysctl will store its data. Since the address is set at compile time, however, this must be a global variable or a static local variable. In this case, L2CR is a global of type unsigned int.

The number 0 is a second argument that is passed to your function. This can be used, for example, to identify which sysctl was used to call your handler function if the same handler function is used for more than one control. In the case of strings, this is used to store the maximum allowable length for incoming values.

sysctl_l2cr is the handler function for this sysctl. The prototype for these functions is of the form

static int sysctl_l2cr SYSCTL_HANDLER_ARGS;

If the sysctl is writable, the function may either use sysctl_handle_int to obtain the value passed in from user space and store it in the default location or use the SYSCTL_IN macro to store it into an alternate buffer. This function must also use the SYSCTL_OUT macro to return a value to user space.

“I” indicates that the argument should refer to a variable of type integer (or a constant, pointer, or other piece of data of equivalent width), as opposed to “L” for a long, “A” for a string, “N” for a node (a sysctl that is the parent of a sysctl category or subcategory), or “S” for a struct. “L2 Cache Register” is a human-readable description of your sysctl.

In order for a control to be accessible from an application, it must be registered. To do this, you do the following:

sysctl_register_oid(&sysctl__hw_l2cr);

You should generally do this in an init routine for a loadable module. If your code is not part of a loadable module, you should add your sysctl to the list of built-in OIDs in the file kern/sysctl_init.c.

If you study the SYSCTL_PROC constructor macro, you will notice that sysctl__hw_l2cr is the name of a variable created by that macro. This means that the SYSCTL_PROC line must be before sysctl_register_oid in the file, and must be in the same (or broader) scope. This name is in the form of sysctl_ followed by the name of it’s parent node, followed by another underscore ( _ ) followed by the name of your sysctl.

A similar function, sysctl_unregister_oid exists to remove a sysctl from the registry. If you are writing a loadable module, you should be certain to do this when your module is unloaded.

In addition to registering your handler function, you also have to write the function. The following is a typical example

static int myhandler SYSCTL_HANDLER_ARGS

{

    int error, retval;

    error = sysctl_handle_int(oidp, oidp->oid_arg1, oidp->oid_arg2,  req);

    if (!error && req->newptr) {

        /* We have a new value stored in the standard location.*/

        /* Do with it as you see fit here. */

        printf("sysctl_test: stored %d\n", SCTEST);

    } else if (req->newptr) {

        /* Something was wrong with the write request */

        /* Do something here if you feel like it.... */

    } else {

        /* Read request. Always return 763, just for grins. */

        printf("sysctl_test: read %d\n", SCTEST);

        retval=763;

        error=SYSCTL_OUT(req, &retval, sizeof retval);

    }

    /* In any case, return success or return the reason for failure  */

    return error;

}

This demonstrates the use of SYSCTL_OUT to send an arbitrary value out to user space from the sysctl handler. The “phantom” req argument is part of the function prototype when the SYSCTL_HANDLER_ARGS macro is expanded, as is the oidp variable used elsewhere. The remaining arguments are a pointer (type indifferent) and the length of data to copy (in bytes).

This code sample also introduces a new function, sysctl_handle_int, which takes the arguments passed to the sysctl, and writes the integer into the usual storage area (L2CR in the earlier example, SCTEST in this one). If you want to see the new value without storing it (to do a sanity check, for example), you should instead use the SYSCTL_IN macro, whose arguments are the same as SYSCTL_OUT.

Registering a New Top Level sysctl

In addition to adding new sysctl options, you can also add a new category or subcategory. The macro SYSCTL_DECL can be used to declare a node that can have children. This requires modifying one additional file to create the child list. For example, if your main C file does this:

SYSCTL_DECL(_net_newcat);

SYSCTL_NODE(_net, OID_AUTO, newcat, CTLFLAG_RW, handler, "new  category");

then this is basically the same thing as declaring extern sysctl_oid_list sysctl__net_newcat_children in your program. In order for the kernel to compile, or the module to link, you must then add this line:

struct sysctl_oid_list sysctl__net_newcat_children;

If you are not writing a module, this should go in the file kern/kern_newsysctl.c. Otherwise, it should go in one of the files of your module. Once you have created this variable, you can use _net_newcat as the parent when creating a new control. As with any sysctl, the node (sysctl__net_newcat) must be registered with sysctl_register_oid and can be unregistered with sysctl_unregister_oid.

Adding a Simple sysctl

If your sysctl only needs to read a value out of a variable, then you do not need to write a function to provide access to that variable. Instead, you can use one of the following macros:

• SYSCTL_INT(parent, nbr, name, access, ptr, val, descr)

• SYSCTL_LONG(parent, nbr, name, access, ptr, descr)

• SYSCTL_STRING(parent, nbr, name, access, arg, len, descr)

• SYSCTL_OPAQUE(parent, nbr, name, access, ptr, len, descr)

• SYSCTL_STRUCT(parent, nbr, name, access, arg, type, descr)

The first four parameters for each macro are the same as for SYSCTL_PROC (described in the previous section) as is the last parameter. The len parameter (where applicable) gives a length of the string or opaque object in bytes.

The arg parameters are pointers just like the ptr parameters. However, the parameters named ptr are explicitly described as pointers because you must explicitly use the “address of” (&) operator unless you are already working with a pointer. Parameters called arg either operate on base types that are implicitly pointers or add the & operator in the appropriate place during macro expansion. In both cases, the argument should refer to the integer, character, or other object that the sysctl will use to store the current value.

The type parameter is the name of the type minus the “struct”. For example, if you have an object of type struct scsipi, then you would use scsipi as that argument. The SYSCTL_STRUCT macro is functionally equivalent to SYSCTL_OPAQUE, except that it hides the use of sizeof.

Finally, the val parameter for SYSCTL_INT is a default value. If the value passed in ptr is NULL, this value is returned when the sysctl is used. You can use this, for example, when adding a sysctl that is specific to certain hardware or certain compile options. One possible example of this might be a special value for feature.version that means “not present.” If that feature became available (for example, if a module were loaded by some user action), it could then update that pointer. If that module were subsequently unloaded, it could set the pointer back to NULL.

Calling a sysctl From User Space

Unlike RPC, sysctl requires explicit intervention on the part of the programmer. To complicate things further, there are two different ways of calling sysctl functions, and neither one works for every control. The old-style sysctl call can only invoke a control if it is listed in a static OID table in the kernel. The new-style sysctlbyname call will work for any user-added sysctl, but not for those listed in the static table. Occasionally, you will even find a control that is registered in both ways, and thus available to both calls. In order to understand the distinction, you must first consider the functions used.

sysctlbyname(char *name, void *oldp, size_t *oldlenp,

        void *newp, u_int newlen)

int get_debug_bufsize()

{

char *name="debug.bpf_bufsize";

int bufsize, retval;

size_t len;

len=4;

retval=sysctlbyname(name, &bufsize, &len, NULL, 0);

/* Check retval here */

return bufsize;

}

sysctl(int *name, u_int namelen, void *oldp, size_t *oldlenp,

        void *newp, u_int newlen)

int get_bus_speed()

{

int mib[2], busspeed, retval;

unsigned int miblen;

size_t len;

mib[0]=CTL_HW;

mib[1]=HW_BUS_FREQ;

miblen=2;

len=4;

retval=sysctl(mib, miblen, &busspeed, &len, NULL, 0);

/* Check retval here */

return busspeed;

}