Jesse Caulfield

How to run Tomcat on Port 80 with user account privileges.

For our development server we don’t have any static content, and wanted to run Tomcat on port 80 for simplicity.

On Linux, authbind simplifies this process. Here are the steps required:

Step 1: Install authbind

Step 2: Make port 80 available to authbind (you need to be root)
touch /etc/authbind/byport/80
chmod 500 /etc/authbind/byport/80
chown glassfish /etc/authbind/byport/80

Step 3: Make IPv4 the default (authbind does not currently support IPv6). To do so, create the file TOMCAT/bin/setenv.sh with the following content:
CATALINA_OPTS="-Djava.net.preferIPv4Stack=true"

Step 4: Change startup.sh
exec authbind --deep "$PRGDIR"/"$EXECUTABLE" start "$@"
# OLD: exec "$PRGDIR"/"$EXECUTABLE" start "$@"

The spectrum monitoring project is starting to collecting a LOT of data. That’s the good part, but it raises the obvious and oft-encountered issue of how to store all this data so it can be rapidly retrieved.

The structure of our data is a long byte array (spectrum measurements) with associated meta-data (sample parameters, geo-location, etc). Each spectrum sample is about 200,000 8-bit data points, and the average database record is almost 256 kByte. At 8 samples per second, this adds up fase. Because each sample represents a distinct measurement location and time, change-detection compression is not possible. However, much of the data is redundant (i.e. long stretches of vacant spectrum), and so each sample lends itself well to atomic compression.

Samples are persisted into a standard SQL database (MySQL or Derby depending upon platform).

To reduce storage requirements I applied GZIP compression to the raw data, while storing meta-data in discrete fields for index, query and retrieval.

Within the Java entity object, I inserted a GZIP shim that compresses and de-compresses the raw BYTE data whenever data is written to or read from the database.

Here’s an outline of the compress/de-compress technique, stripped of the JPA encapsulation:

public class testGzip {

public static void main(String[] args) throws IOException {
// Original Data
byte[] dataBytes = new byte[]{1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 5, 6, 67, 7, 8, 9};
// A grow-able, memory-resident byte array
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
// A gzip processor to write into the memory array
GZIPOutputStream gzipos = new GZIPOutputStream(byteArrayOutputStream);
// Write the data to the gzip processor
gzipos.write(dataBytes);
// Close the ‘file’ – this adds the gzip checksum, etc.
gzipos.close();
// Debug – show what we’re compressing
System.out.println(“data bytes len ” + dataBytes.length);
int len = dataBytes.length;
System.out.print(” data: “);
for (byte b : dataBytes) {
System.out.print(b + ” “);
}
System.out.println(“”);
// Get the compressed data – this can be persisted
byte[] gzBytes = byteArrayOutputStream.toByteArray();
// Debug – show that the compressed data is different (and presumably shorter)
System.out.println(“compressed bytes len ” + gzBytes.length);
System.out.print(” data compressed: “);
for (byte b : gzBytes) {
System.out.print(b + ” “);
}
System.out.println(“”);
// A structure to read memory-resident byte arrays
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(gzBytes);
// A gzip expander
GZIPInputStream gzipis = new GZIPInputStream(byteArrayInputStream);
// A place to hold the uncompressed data — the length is carried over from the original data
byte[] unzip = new byte[len];
// Read the compressed data into the uncompressed byte array
gzipis.read(unzip, 0, len);
// Debug – show that the recovered data exactly matches the input data
System.out.println(“un-compressed bytes len ” + unzip.length);
System.out.print(” data un-compressed: “);
for (byte b : unzip) {
System.out.print(b + ” “);
}
System.out.println(“”);
System.out.println(“Input data matches Output data: ” + Arrays.equals(dataBytes, unzip));
System.out.println(“Compression ratio: ” + String.format(“%.2f”, (double) gzBytes.length / (double) dataBytes.length));
}
}

We’re trying to create a series of cross-referencing database tables in MySQL using foreign keys. However, despite following the documentation the linking did not seem to work.

I found that MySQL by default uses MyISAM for the DB engine, which does not support FOREIGN KEYs.

The solution is to force MySQL to use the INNODB engine.

For example: This will not work.

CREATE TABLE parent (id INT NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE child (id INT, parent_id INT,
INDEX par_ind (parent_id),
FOREIGN KEY (parent_id) REFERENCES parent(id)
ON DELETE CASCADE
) ENGINE=MyISAM;

This is how is should be written:

CREATE TABLE parent (id INT NOT NULL,
PRIMARY KEY (id)
) ENGINE=INNODB;
CREATE TABLE child (id INT, parent_id INT,
INDEX par_ind (parent_id),
FOREIGN KEY (parent_id) REFERENCES parent(id)
ON DELETE CASCADE
) ENGINE=INNODB;

A frustrating detail.

This week I sat down with Shep Kendall, an old friend and colleague, at a local Starbucks to discuss web applications. Shep accomplished in 20 minutes what 6 months of casual inquiry could not: explain clearly how things are done today, what is the model-view-controller architecture, how one would employ it, and why one would bother.

As background, fresh out of college I was the “resident HTML expert” and developed the first CGI codebase for Proxima (later Proxicom) that we used in our non-static web sites. I’m dating myself to say this was pre-Netscape IPO, but the point is that dynamic web services then relied exclusively on a client-server model. Dynamically generated HTML pages were created on the server and dynamically linked to other pages or CGI scripts.

Each browser page query was stateless and cookies hadn’t yet been invented. Stateful, multi-page transaction required values (like shopping carts) to be encoded and propagated within the URL. There was also no Javascript yet and the browser function was limited just to rendering HTML. Folks still used the <flash> tag and animated GIFs were still considered cool.

This basic client-server (CS) architecture I worked is still widely in use today in the guise of pre-processed HTML techniques like PHP, JSP, etc. where the HTML pages and CGI-BIN programming logic have been combined.

Client Server Model, or Sweet Nostalgia:
How we built web applications back in the day

Over the past decade however a whole new corpus of technology has been developed. While the end result is the same (rendered HTML) there are few other similarities to how dynamic web services are implemented today.

The most important development is in today’s browsers, which are infinitely more flexible platforms capable of “client-side” processing and logic through the use of Javascript, Flash, Active-X and other virtual-machine plug-ins. This has also enabled the server side to evolve.

The client / server evolution also enables a more optimised web application delivery achitecture. The addition of intelligence in the browser basically means that an HTML “page” is no longer a static, stateless object, but rather the canvas (or framework) within which information may be rendered according to user inputs (clicks, drags, etc.) The innovation called Web 2.0 is essentially an intelligent browser that can dynamically update portions of the rendered HTML on-the-fly.

The most scaleable architecture I’m aware of for such dynamic web services is Model-View-Controller, or MVC. MVC, like the client / server model illustrated above, also isolates business logic (programs) from the user interface (browser) while adding a two-day communications concept directly between the browser page and the server.

In layman’s terms, this means the browser fetches a page while the page fetches its own content.

MVC Model: Enables dynamic, user-driven content

In the MVC architecture configuration shown the web server is Apache. A separate application server called Tomcat is also installed. Tomcat may also run on it’s own without Apache. In the scenario illustrated above Tomcat registers its services (servlets) with the Apache web server and maps them to URLs. The net effect of the above configuration is that URL requests are directed either to HTML pages via Apache or to servlets via Apache + Tomcat transparently to the user.

The important addition to the earlier CS model is the concept of servlets, which are essentially funtion-specific web servers. In practice, servlets are paired up with Javascript/Flash/Active-X functions on the web page, respond to specific inquiries and send data formated for the requesting function. For example, a function to create a table display will be paired with a servlet that sends data in tabular format.

Servlet example: executing a database inquiry and formatting the data into a JSON-encoded object

It’s helpful to “follow the token” and the transaction flow for a page-load and subsequent page data request are illustrated below.

Page and page data transaction flow

A brief word about libaries and frameworks

There is a tremendous amount of configuration required to get all these pieces working together properly. Software frameworks have been developed essentially to automate the process of registering servlets and their capabilities, then exposing their data to client browser functions. Framework examples include Sprint, JBOSS, etc. More on frameworks at a later date.

Links >

• Model–view–controller
• PHP
• Shep Kendall
• JBOSS
• Spring Framework

In this tutorial I will describe how to install JPCap from source code on Linux (Ubunty 7.10 or Debian 4.0).

JPCap is an open-source Java library released under the GNU LGPL and designed to enable the capturing and sending network packets in Java. JPCap is developed by Keita Fujii and the University of California and Irvine.

The JPCap home page is  http://netresearch.ics.uci.edu/kfujii/jpcap/doc/

JPCap is a Java Native Library Implementation (JNI) of the popular libpcap library and should therefor work on any OS which supports libpcap. libpcap is a user-level packet capture library that provides a common, system-independent API for low-level network monitoring. libpcap is also open-source software developed and maintained by TCPDump.org.

We will use JPCap and libpcap on the Linux operating system but the both libraries work on Microsoft Windows (WinPcap), Linux, FreeBSD, and Mac OS X.

JPCap supports the following types of network data:
Layer 2:    Ethernet Datagrams
Layer 3:    IPv4 & Ipv6     ARP/RARP
TCP, UDP and ICMPv4.

JPCap recognizes the packet types enumerated above but can capture any type of network traffic as a raw packet (i.e., as an instance of the Packet class) which contains the Packet’s whole data. This feature allows Java applications to analyze any packet type.

Step 1:    Preparing the System

Following are instructions to install JPCap on a fresh installation of Ubuntu 7.10 of Debian 4.0.

Install the necessary development software packages to create a usable development environment:

Install the GNU compiler and basic libraries
% sudo apt-get install build-essential

Install the linux pcap library
% sudo apt-get install libpcap0.8

Install the Java SDK from Sun
% sudo apt-get install sun-java6-sdk

Download the latest JPCap source and extract it into your working directory

http://netresearch.ics.uci.edu/kfujii/jpcap/

You can also find version 0.7 here: JPCap 0.7

Step 2:    Building & Installing the JPCap JNI library

In a terminal window, navigate to the [jpcap]/src/c directory. For example:
~/jpcap-0.7/src/c

[IMPORTANT] Edit the JAVA_DIR Makefile entry to point to your version of jni.h
You can find it quickly with the following command:
$ find /usr -name jni.h
/usr/lib/jvm/java-6-sun-1.6.0.03/include/jni.h

The Makefile should then read
JAVA_DIR = /usr/lib/jvm/java-6-sun-1.6.0.03

In the terminal window under the [jpcap]/src/c directory type make to create the shared library
$ make
This will create the file ‘libjpcap.so’.

Copy ‘libjpcap.so’ your system’s Java JNI library directory
‘[Java-dir]/jre/lib/<arch>’ where <arch> is either ‘i386′ or ‘sparc’.
For my installation, this was /usr/lib/jvm/java-6-sun/jre/lib/i386
Another option is to copy the library to the directory where your application is located.

Step 3:    Building and Installing the JPCap JAR file

In a terminal window, nagivate to the [jpcap]/src/java directory. For example:
~/jpcap-0.7/src/java
There should be one subdirectory named ‘jpcap’

Compile all of the .java files in the ‘jpcap’ and ‘jpcap/packet’ directories.
$ find . -name “*.java” -exec javac {} \;

You may see the following error for IPPacket.java – ignore it.
“Note: ./jpcap/packet/IPPacket.java uses unchecked or unsafe operations.”

Create the JAR file
jar -cf jpcap.jar jpcap

Confirm the contents of your JAR file with the following command:
jar -tvf jpcap.jar

Copy the new jpcap.jar file to your Java extentions directory
$ cp jpcap.jar ‘[Java-dir]/jre/lib/ext/
For my installation, this was /usr/lib/jvm/java-6-sun/jre/lib/ext
Another option is to copy the JAR file to your application’s directory and add it to your CLASSPATH.

You are now ready to begin using the JPCAP Java Native Interface Library.