Read Me About CFLocalServer

===========================

1.2

CFLocalServer is a sample that shows how to use UNIX domain sockets to communicate between client and server programs running on the same machine.  This sample was written specifically to show how a daemon can manage communication with multiple clients.  It supports RPC [1] communications (where the client makes a request to server and blocks waiting for a reply) and asynchronous notification (where the server sends messages to all clients).  It wraps each UNIX domain socket in a CFSocket to demonstrate the integration of UNIX domain sockets with a run-loop based program.

CFLocalServer should work on all versions of Mac OS X, but it has been primarily tested on Mac OS X 10.4.x.

[1] Remote Procedure Call

About UNIX Domain Sockets

-------------------------

UNIX domain sockets are somewhat like TCP/IP sockets, except that the communication is always local to the computer.  You access UNIX domain sockets using the same BSD sockets API that you'd use for TCP/IP sockets.  The primary difference is the address format.  For TCP/IP sockets, the address structure (that which you pass to bind, connect, and so on) is (struct sockaddr_in), which contains a IP address and port number.  For UNIX domain sockets, the address structure is (struct sockaddr_un), which contains a path.

When the server binds to a UNIX domain socket, the system creates a file system object that represents the socket.  For example, PPP's UNIX domain socket is /var/run/pppconfd.  When you look at this with "ls -l", you'll see the first character of the listing is "s", indicating that this object is a socket.

$ ls -l /var/run/pppconfd 

srwxrwxrwx  1 root  daemon  0  9 May 12:57 /var/run/pppconfd

Once the server is running, the client can connect to it by simply passing this path to the connect call.  Once the connection is in place, communication proceeds as it would for TCP/IP sockets.

To learn more about UNIX domain sockets, consult any standard UNIX reference.  I particularly recommend "UNIX Network Programming" by Stevens.

Why UNIX Domain Sockets?

------------------------

Mac OS X has numerous inter-process communications (IPC) APIs.  When implementing a daemon, the UNIX domain sockets API offers a number of advantages over other IPC mechanisms.

o Compared to TCP/IP, it guarantees that only processes running on your local machine can connect to your server.  You can also get this guarantee with TCP/IP, but it requires extra work.

o Compared to Apple events, it works on all versions of Mac OS X (there are problems using Apple events from a daemon prior to Mac OS X 10.2).  Also, it's a connection-oriented API, so the server automatically learns about the death of a client and it's easy for the server to asynchronously notify the client.

o Compared to Mach messages (and any high-level wrappers, like CFMessagePort), it eliminates any Mach bootstrap port namespace concerns.  Also, when you use Mach messages, you have to do extra work for the server to be notified when a client dies unexpectedly.

o It can be easily integrated into any server architecture, including those based on threads, a runloop (using CFSocket), a select loop, or kqueues.

o It is a well known POSIX API with numerous sources of good documentation.

o Source code based on UNIX domain sockets code is portable to other POSIX platforms.

Packing List

------------

The sample contains the following items:

o Read Me About CFLocalServer.txt -- This document.

o CFLocalServer.xcodeproj -- An Xcode 2.1 project that builds the sample.

o CFLocalServer.xcode -- An Xcode 1.5 project that builds the sample.

o Server.c -- Source code for the server.

o Client.c -- Source code for the client.

o Protocol.h -- Definitions for the client/server communication protocol.

o Common.h -- Interface to the common utilities.

o Common.c -- Implementation of the common utilities.

o build -- A directory containing pre-built versions of the client and server.

Using the Sample

----------------

To try out CFLocalServer, first open a Terminal window, change into the "build" directory, and then run the server.

server$ ./CFLocalServer

CFLocalServer: Starting up (pid: 2764).

The server will continue running until you tell it to quit by either typing ^C or  sending it a "quit" command from one of the clients.

To start a client, open a new Terminal window, change into the build directory, and then run the client as shown below.

client$ ./CFLocalClient nop

       nop

The client arguments represent commands to send to the server.  In this case the "nop" command does nothing; it simply tests the RPC path between the client and the server.  If you look at the server window you'll see that the server has printed something like:

0x500da0: NOP

ClientGotSpace: Client 0x500da0 lifted write-side flow control.

The first line is a record of the "nop" command.  The second is a debugging message which isn't particularly relevant to this example.

The client supports a number of other commands.  You can see a list by running the client with no arguments.

client$ ./CFLocalClient 

usage: CFLocalClient command...

       commands: nop

                 whisper <message>

                 shout   <message>

                 listen  (must be last)

                 quit

The "quit" command tells the server to quit.  The "whisper" command tells the server to print <message>.  The "shout" and "listen" commands are used to demonstrate asynchronous server-to-client notification.  To see this in action, open a third terminal window and run the "listen" command in that.

listener$ ./CFLocalClient listen

    listen Press ^C to quit.

This starts the client and tells it to listen for messages from the server.  These messages are asynchronous; the server generates the message internally and sends it to all listening clients.

To send an asynchronous notification message, execute the "shout" command on the first client.

$ ./CFLocalClient shout 'Hello Cruel World!'

     shout "Hello Cruel World!"

On the server you'll see the following, indicating that the server got the "shout" command:

0x501060: Shout   "Hello Cruel World!"

On the listening client you'll see the following:

listener$ ./CFLocalClient listen

    listen Press ^C to quit.

           heard "Hello Cruel World!"

You can run multiple listeners, and each will see this message.

If you want to examine the state of the server, you can send it a SIGINFO signal.

client$ kill -INFO 2764

The server will then dump its current state, as shown in the example below.

CFLocalServer State

-------------------

Clients:

  Client 0x500da0:

    fSockFD            = 8

    fSockCF            = 0x500f80

    fRunLoopSource     = 0x500e90

    fBufferedData      = 0x500df0 (count: 0)

    fPendingSends      = 0x500e20 (count: 0)

    fPendingSendOffset = 0

    fListening         = true

Descriptors:

   0 file    '/dev/ttyp2'

   1 file    '/dev/ttyp2'

   2 file    '/dev/ttyp2'

   3 socket  SOCK_STREAM ? -> ?

   4 socket  SOCK_STREAM ? -> ?

   5 socket  SOCK_DGRAM  ? -> ?

   6 socket  SOCK_DGRAM  ? -> ?

   7 socket  SOCK_STREAM /var/tmp/com.apple.dts.CFLocalServer/Socket -> ?

   8 socket  SOCK_STREAM /var/tmp/com.apple.dts.CFLocalServer/Socket -> ?

Building the Sample

-------------------

The sample was built on Mac OS X 10.4.x using Xcode 2.1.  To build the project, open the project and choose Build from the Build menu.  There is nothing inherently 10.4-specific about this sample, and it should work all the way back to Mac OS X 10.1.

How it Works

------------

The client and server communicate by exchanging messages, known as packets.  All packets start with a header, with four fields:

o fMagic is always kPacketMagic, which allows me to quickly detect any communications problems

o fType is the packet type; for example, a value of kPacketTypeNOP represents a "nop" command

o fID is a sequence number designed to allow the client to match up requests with replies; kPacketIDNone is used for non-RPC messages; any other value is used for RPC requests; for RPC requests, the server always sends the reply with an fID that matches the fID of the request; currently a client connection can only be used for RPC or for asynchronous messages (not both), so this matching of request and reply is not really needed

o fSize is the overall size of the packet, including the header

Data after the fSize field is packet-type specific.

The client is made up of two logically separate components: the connection abstraction and the command line tool.  The command line tool is simply the UI of the client; it parses arguments and prints results.  It doesn't know anything about UNIX domain sockets.  It does all of its work via the connection abstraction.

The connection abstraction is a set of routines that are called by the command line tool to connect to the server (ConnectionOpen), send RPC requests (ConnectionRPC), and so on.  The connection abstraction is basically a big wrapper around a UNIX domain socket.  ConnectionOpen opens a connection to the server by calling socket <x-man-page://2/socket>, to create the socket, and connect <x-man-page://2/connect>, to connect it.  ConnectionRPC uses write <x-man-page://2/write> to send the request and read <x-man-page://2/read> to receive the response.

The most complex part of the client relates to listening.  This performs two actions.  Firstly, it sends a "listen" command to the server, to tell the server that this client has switched away from RPC mode and is now listening for asynchronous messages.  Secondly, it calls the connection abstraction to schedule this socket with the runloop (ConnectionRegisterListener) and then runs the runloop.  When data arrives, the connection abstraction (ConnectionGotData) is called by CFSocket, and it in turn calls the tool's packet handler (GotPacket).

The server is also divided into two logical components.  The first is the server framework, which is responsible for creating a listening socket, scheduling that socket with the runloop, and running the runloop to accept new connections.  When it gets a new connection (ListeningSocketAcceptCallback), it calls the second layer, client state management, to create (ClientCreate) a new client record (of type ClientState).  This client record tracks the state of the client.  The client state management layer maintains a global set of client records (gClients), one for each connected client.  Once a client has been created, it can do a bunch of different things.

o If the client sends data to the server, CFSocket calls the client state management's socket callback (ClientEvent) with the kCFSocketDataCallBack event.  This calls ClientGotData, which attempts to receive a packet from the client.  If the new data doesn't represent a full packet, the client state management layer buffers the data (in fBufferedData) and waits for more data to arrive.  If the new data does represent a whole packet (or completes a previously buffered packet), the client state management layer processes that packet.

o If the client sends the server any form of a bogus packet, the server destroys the client record (ClientDestroy).

o If the server needs to send data to the client, it places the data in the fPendingSends buffer and then calls the transmission routine (ClientSendPending).  That either sends all of the data, or it fails with an error.  If the error indicates write-side flow control (that is, the client has stopped receiving data and the socket buffer is full, causing write to return EAGAIN), the client state management layer tells CFSocket that it wants to be notified when the flow control is lifted, and returns.  If it gets any other error, it just destroys the client.

o If a flow controlled client starts receiving data again, CFSocket calls the client state management's socket callback (ClientEvent) with the kCFSocketWriteCallBack event.  This calls ClientGotSpace, which tries to send any pending data to the client.

o If the amount of data waiting to be sent to the client exceeds a certain threshold (100 messages, kClientMaximumPendingSends), the server assumes that the client has gone permanently deaf and destroys the client.

o If a client goes away in an orderly fashion, it sends the server a "goodbye" packet.  The server responds by destroying the client.

o If a client dies unexpectedly, CFSocket notifies the client state management layer by telling it that no data is available from the client.  ClientGotData detects this and destroys the client.

Caveats

-------

This is a sample of client/server communications using UNIX domain sockets wrapped in CFSockets.  Client/server communication is a critical part of writing a daemon, but it is not the only part.  To write a fully fledged daemon, you must organise to launch the daemon (typically this is done using launchd <x-man-page://8/launchd> on 10.4 and later, or from a startup item on earlier systems) and you must implement proper logging (using ASL <x-man-page://3/asl> on 10.4 and later, or syslog <x-man-page://3/syslog> on earlier systems).

The sample does not support communication between 32- and 64-bit processes.  This is primarily because CFSocket is not supported for 64-bit programs.  However, even if that was addressed, you'd have to modify the sample to ensure that the structures used in the communication were 64-bit invariant.  See the comments in "Protocol.h" for more details.

As it's currently written, the sample is not endian neutral; the server and all its clients must use the same endian flavour.  Thus, for example, on a Developer Transition System, you can't run a native server and connect to it from a PowerPC program being executed by Rosetta.  It would be easy to modify the sample to support this (you just have to decide on a standard endianess, and byte swap the multibyte fields of the structures defined in "Protocol.h"), but I decided against doing this because a) mixed-endian code on the /same machine/ is pretty unusual, and b) it obscures the primary goal of the sample.

In the current architecture, each connection between the client and the server is used either for RPC or for asynchronous notification, but never for both.  If you want a client that does both, you need to open two connections.  This design simplifies the RPC code (it doesn't have to worry about asynchronous notification messages being mixed in with an RPC reply), but it is somewhat wasteful of resources.

The current implementation is not tuned for performance.  For example, the server's packet reception loop will often make unnecessary copies of the packet data.  If you adapt this sample to a performance critical application, you will have to profile the code and optimise its algorithms.

Credits and Version History

---------------------------

If you find any problems with this sample, mail <dts@apple.com> and I'll try to fix them up.

1.0 (May 2005) was the first shipping version.

1.1 (Jun 2005) corrected a bug where the packet's fID field wasn't being initialised in the RPC case.

1.2 (Jul 2005) updated to include an Xcode 2.1 project to produce universal binaries; there were no code changes required for it to run correctly on the Developer Transition Systems.

Share and Enjoy.

Apple Developer Technical Support

Networking, Communications, Hardware

20 Jul 2005

$Log: Read\040Me\040About\040CFLocalServer.txt,v $

Revision 1.4  2005/07/20 14:24:57  eskimo1

Updated to discuss the issues associated with building a universal binary.

Revision 1.3  2005/06/30 20:27:15  eskimo1

Updated for version 1.1.

Revision 1.2  2005/05/18 13:36:32  eskimo1

Fixed various documentation/comment changes.

Revision 1.1  2005/05/17 12:19:27  eskimo1

First checked in.