Soap-Stone: How Fast Is Your Network Today? White Paper Results Submit Your Results Support Products Download Soap-Stone

Related Resources:

1 Soap-Stone: How Fast Is Your Network Today?

1.1 Introduction

Soap-Stone is the network benchmark application which can answer these questions for you by putting your network under load and conducting automatic benchmark and recording activities.

1.2 Getting Started

For the rest of this paper the actual JAR file name will be referred to as soap-stone.jar, when in practice the filename will have release information built into it (for example soap-stone-full-0.952.jar). Please make appropriate substitutions when running the application on your local machine.

The diagram in Fig.1 shows a localhost loopback benchmark with Soap-Stone running as a client and server on the same physical machine.

Fig.1: Using Soap-Stone to conduct a simple one-way localhost benchmark

What does this do? It runs a local loopback test from one program to another, on the same machine. Here's what the output for the client and server looks like on a 3 GHz Pentium-IV Hyper-Threaded system:

Fig.2: Client & Server output for simple one-way loopback benchmark.

The interesting results are shown in the Client window: the Server window will sit there quietly unless there is a failure. The result from this test is interesting: for 1500 byte size packets, sent 10000 at a time, the average throughput is 408 Mbit/sec! Sounds impressive, but what does this actually mean?

More than you might think. True it doesn't measure a real network between two machines. But when you look at a networking stack the physical network is only one part of the overall overhead in communicating information between agents (Fig.3).

Fig.3: The Network Stack

What this first test represents is the best-case one-way throughput achievable on a given CPU architecture, running a given Java Virtual Machine, with no physical network overhead. Is this useful? Wait until we examine some results for communication procotols other than the raw sockets in java.net.Socket before casting judgement!

What conclusions can we draw from these results? Interestingly, this setup is capable of sustaining network traffic at rates faster than 100Mbit/sec, so a 100baseT physical network would not be fast enough to support even two similar machines communicating using this raw sockets protocol.

So we have our first conclusion: the latest desktop and server machines running Java should be provisioned with Gigabit Ethernet if they are bandwidth intensive.

1.3 Multiple Protocols

Fig.4: Soap-Stone Built-in Help

The -help option passed into SoapStone shows the normal two usage modes of the soap-stone.jar, namely:

• java -jar soap-stone.jar server: Launch a server with default options
• java -jar soap-stone.jar client: Launch a client with default options

It is important to launch the server before the client, since the client will immediately start to establish communications with the server when it is started.

1.4 How Fast Is Your Protocol Today?

SoapStone also supports the concept of scenarios: a scenario is a set of tests to be performed by the client against a running server. (The client automatically instructs the server to switch protocols during testing, so scenarios are not applicable to the server, which unlike the client never terminates once it is stated). Let's take a look at the first scenario in action.

1.4.1 Scenario 1: All Protocols

Arguments	Effects
-protocol raw	Raw sockets protocol (the fastest?)
-protocol rmi	The Java RMI protocol (Remote Method Invocation)
-protocol http	HTTP protocol, typically used with web-servers, as implemented using Jetty
-protocol soap	An implementation of the SOAP protocol, running on top of Jetty

This scenario assumes that you have a fully populated plugins subdirectory below the directory where you invoke soap-stone.jar. If you are running the soap-stone-full.jar this will be created and populated for you automatically. By "fully populated" we mean that the directory should contain the files raw-plugin.jar, rmi-plugin.jar, http-plugin.jar, and soap-plugin.jar. However, any missing protocols will result in benchmark numbers of zero Mbit/sec in the results.

After starting a server, run this scenario as follows: $ java -jar soapstone-full.jar -scenario 1. The resulting output is shown in Fig.6:

Fig.6: Scenario 1: Protocols

How does the selection of protocol affect the network performance? Dramatically, as the chart in Fig.7 shows. More on this in one moment.

Fig.7: Protocol Bandwidths

1.4.2 Logging

You may be wondering how this chart was created? The driving SoapStone program writes out a comma-separated variable file suitable for direct importation into a spreadsheet. The file is called soap-stone.csv, and is overwritten for each scenario.

The file is written in such a way as to facilitate importing and charting in a spreadsheet program such as OpenOffice SpreadSheet. Consider the following scenario:

java -jar soap-stone.jar client -protocol [raw,http] -echo [false,true] -size [10,100,1000]

This results in a total of 12 test cases (2*2*3). What does the soap-stone.csv file look like? The easiest way to explain it is to show it as it appears in a spreadsheet.

Fig.8a: CSV Logging Tables

For each parameter in the scenario, other than the right-most, a new table is created. So in this case there are four tables (2*2) by default. Each table is preceded by the combination of parameters that are in force for the numbers in the table (e.g. protocol=raw, echo=true for the first table). The right most parameter forms the first column of each table. The rows of the table are the average, minimum and maximum bandwidths measured during the test, in Mbit/sec.

This tabular format is reasonably convenient for graphing the results in a spreadheet.

Clearly, however, the number of tables will explode exponentially as the number of parameters is increased. So SoapStone supports some powerful mechanisms for logging only the data that you want, and controlling the table formats to simplify subsequent spreadsheet analysis.

The first mechanism allows control over the number of tables. By specifying parameter names as values for the -tables option, the other parameters are forced down into the column names for the data.

java -jar soap-stone.jar client -protocol [raw,http] -echo [false,true] -size [10,100,1000] -tables echo

Fig.8b: Reducing Number of Tables

This presents the same information as in Fig.8a, but as two tables instead of four. The tables are named (prefixed with) the parameter(s) specified in the -tables argument.

This approach can be driven to its logical extreme: a single table with all the results in it, by using the -tables argument with no table names:

java -jar soap-stone.jar client -protocol [raw,http] -echo [false,true] -size [10,100,1000] -tables

Fig.8c: Logging to One Table

In all probability, this is too much data for use in a spreadsheet. SoapStone supports another method of reducing the amount of data, the -columns option.

java -jar soap-stone.jar client -protocol [raw,http] -echo [false,true] -size [10,100,1000] -tables -columns avg

Fig.8d: Further Reducing the Log

1.4.3 Performance Analysis

What can we conclude about communication performance from this scenario? Clearly the Raw Sockets communications is by far the fastest way to communicate between two programs on the same machine (at least for 1500 byte packets, and in a unidirectional fashion). In fact, Raw Sockets are more than ten-times faster than the nearest rival, which is RMI, and more than one-hundred times faster than the SOAP protocol! To make this more evident, let's modify Fig.7 to use a logarithmic Y axis for the bandwidth:

Fig.9: Protocol Bandwidths (Log Scale)

At this point you are probably thinking "no big deal". Everybody knows SOAP is an inefficient transport mechanism, the SOAP envelope overhead is substantial with a payload of only 1500 bytes, so that probably explains the poor performance. Unfortunately, things aren't that simple as we will see next.

1.4.4 Payload Size

What payload size is reasonable for communications? For XML based service-oriented architectures (SOA'a) payload sizes in the 1Mbyte range are not unreasonable. Why? Simply because the SOA architecture tends to be based on stateless agents, and the entire state of a transaction thererfore has to be contained in a document. As an XML document flows through the SOA system, it will grow as it accumulates intermediate results as part of its being processed.

These sizes are in distinct contrast to those experienced when using remote-procedure call system architerctures, where many small messages will be exchanged between stateful agents. In such cases payload sizes will be at or below 100 bytes.

So how is performance impacted by payload size? As the payload size increases towards infinity, one would expect all of the protocols to converge to similar throughputs, since the various overheads would be neglegible with respect to the increasingly large payload sizes. But in practice this is not the case, at least with the four protocol plugins we are testing in this experiment as the following two figures show (the second chart uses a logarithmic Y-axis scale, the first chart uses a linear Y-axis scale).

The scenario being tested here is as follows:

scenario 2: client -protocol [raw,rmi,http,soap] -size [100,1k,10k,100k,1000k] -tables -columns avg

The client and server programs are run on a single 3.0GHz Pentium-V/HyperThreading system, so no physical network is being measured. HyperThreading works well in this situation, with each virtual CPU running at only around 50% utilization. The results are therefore useful in predicting what speed of network would be required to fully service this class of machine in processing data of various sizes.

Note that this scenario does not echo data back from the server to the client, so this represents something of a best-case for throughput, since no extra burden is imposed by the return of data to the client.

Fig.9b: Protocol Bandwidths versus payload size

Fig.9c: Protocol Bandwidths versus payload size (log scale)

1.4.5 Observations

The raw protocol out-performs all other protocols at all payload sizes, sometimes by one or two orders of magnitude.

As payload size approaches 1,000,000 bytes, the raw and http protocols converge towards respectable throughputs in the 350Mbit/sec range. The rmi performance starts to lag for payloads at and above 100,000 bytes.

Small payloads (100 bytes) have very poor throughput, the best case being the raw protocol which manages to achieve around 30 Mbit/sec.

The soap protocol performance is simply dismal. It is an order of magnitude worse than the rmi and http protocols. The performance relative to http is disturbing: since SOAP uses HTTP as its transport one would expect that their performance would converge as the payload size increased. We speculate that the SOAP protocol is introducing a lot of extra overhead in the serialization and deserialization of data which is simply unacceptable for large payload sizes.

The throughput of all of the protocols starts to fall off for payload sizes above 100k.

1.4.6 Conclusions?

1.4.7 Meanwhile, back on the Server

As an aside, note that the server window does show something interesting in this case: as the client switches from protocol to protocol, the server is commanded to follow it, and it starts other servers (HTTP, RMI, SOAP) on demand to support the client (Fig.8):

Fig.10: Scenario 1: Servers

1.5 How Fast Is Your Phone-Net Today?

However, sometimes it can establish a connection where it really should not. Here's a real example. The physical network is a 10 MBps NetGear Phone-Net running over a typical home phone line setup. The home phone-line network was installed and worked fine for a couple of years, but something happened to the front room connection which made network traffic appear to be much slower than it should have been. How much slower? Soap-Stone provided the answer: for 1500 byte packets over a RAW protocol the connection from the front room of this house was one-hundred times slower than from the living room! Some crawling about under the house appears to be in order. Note that one cause of this kind of problem is inadvertantly plugging in your Phone-Net behind the line-filters intended for analog equipment. This was, unfortunately, not the case here.

Fig.11: Phone-Net Problems

1.6 Help!

The design of SoapStone is hierarchical: the top-level SoapStone driver program invokes the Client and Server programs based on the client or server argument passed to the top-level program. The Client and Server programs depend on protocol plugins, which themselves have their own applicable options. Options for server-side plugins are specified at the client, and passed to the server through the control commands issued at runtime.

The client and server programs each generate their own help. The plugins generate help relevant to their own operation:

Fig.11: Client and Server Options

1.7 Where Next?