Jesse Caulfield

To determine an endpoint’s operating system one must first understand how to measure differences in a network protocol stack. Start by studying the TCP state machine, understanding how TCP connections are made, where implementations may vary, and creating a series of tests to expose those variances.

To initiate and manage TCP connections endpoints (hosts) exchange packets with certain flags, or control bits, set. There are eight flags available; to identify a host operating system we are interested with the following three:

1. ACK – Indicates that the acknowledgment field is significant
2. RST – Resets the connection
3. SYN – Synchronizes sequence numbers

Transmission Control Protocol is defined in RFC 793 and uses a 3-way handshake protocol to initialize new connections between hosts.

The packet sequence is:

1. Caller sends SYN
2. Recipient responds with SYN, ACK
3. Caller sends ACK

If the connection cannot be made the recipient will typically respond with a RST packet to abort the connection instead of the SYN + ACK to establish it. A successful 3-way handshake is illustrated in the host packet exchange shown below. The process of establishing a connection can be traced on the TCP state diagram below right as (CLOSED) > (SYN_SENT) > (ESTABLISHED) while a failed connection would be (CLOSED > (SYN_SENT) > (CLOSED).

To detect a TCP connection and determine between caller and recipient one must capture only two packets with the SYN (Step 1) and SYN+ACK (Step 2) or RST bits set. To do this note the structure of a TCP packet header and build a packet filter expression.

A TCP header typically holds 20 octets of data. The first line of the packet header contains 32 bits or 4 octets numbered 0 – 3, the second line octets 4 – 7, etc. The TCP control bits are contained in octet 13. In a TCP datagram with the SYN bit set octet 13 resembles the TCP Flags detail illustrated above. By filtering on TCP flags one may capture all packets necessary to profile a host operating system.

References

• RFC 793 – Transmission Control Protocol

Gnome-Session in Ubuntu 7.10 has a very frustrating default configuration which breaks VNCServer by defaulting to an incorrect keyboard map. (i.e. scrambling what you type.)

Here is how to fix it:

In a working terminal (probably not via the scrambled vnc connection) open gconf-editor as your user (not as root) then navigate to:

desktop > gnome > peripherals > keyboard > kbd

Replace the empty (or incorrect) layout value of [] or [us] with any other two letters. I used “aa”.

Close gconf-editor and restart your vncserver. The problem should now be fixed.

The standard Ubuntu & Debian VNCServer package does not run out-of-the-box due to a missing font dependency. Inspecting the log file, you see the error message:

Fatal server error:
could not open default font ‘fixed’

Here is how to fix it.

Update the vncserver configuration file with the following font configuration lines

$ sudo vi /etc/vnc.conf

# Added for Ubuntu Compatibility
$fontPath .= "/usr/share/fonts/X11/misc/,";
$fontPath .= "/usr/share/fonts/X11/75dpi/:unscaled,";
$fontPath .= "/usr/share/fonts/X11/100dpi/:unscaled,";
$fontPath .= "/usr/share/fonts/X11/Type1/,";
$fontPath .= "/usr/share/fonts/X11/75dpi/,";
$fontPath .= "/usr/share/fonts/X11/100dpi/,";
$fontPath .= "/usr/share/fonts/X11/encodings/";

Then restart the vncserver.

$ vncserver

Lastly, Ubuntu has some VNC default color-depth and desktop size strangeness. Fix this with command options.

$ vncserver :1 -geometry 1024x768 -depth 24 -alwaysshared