Show Table Of Contents
Soap-Stone: How Fast Is Your Network Today?

White Paper
Results
Submit Your Results
Support
Products
Download Soap-Stone

Related Resources:
download soap-stone-full-0.952.jar soap-stone home-page at sourceforge.net simontuffs.com home-page licensing information contact us SourceForge.net home-page
Copyright P. Simon Tuffs, 2004. All Rights Reserved.Version 0.95

1 Soap-Stone: How Fast Is Your Network Today?

by P. Simon Tuffs, Software Architect and Consultant (www.simontuffs.com).


1.1 Introduction

So you just installed a new Gigabit Ethernet network. Or you just installed a WiFi access point for your intranet. It seems fast, but just how fast can you really pump data through your shiny new network? More specifically, how fast can your applications talk various protocols (Sockets, HTTP, RMI, SOAP) one-way and round-trip from inside a Java Virtual Machine?

Soap-Stone is the network benchmark application which can answer these questions for you by putting your network under load and conducting automatic benchmark and recording activities.

1.2 Getting Started

Soap-Stone is delivered as a single Java JAR archive which can act as both client and server in a point-to-point network test. You can download the latest Soap-Stone JAR file here: Download Soap-Stone (Note: Soap-Stone is packaged using One-Jar, another simontuffs.com technology)

For the rest of this paper the actual JAR file name will be referred to as soap-stone.jar, when in practice the filename will have release information built into it (for example soap-stone-full-0.952.jar). Please make appropriate substitutions when running the application on your local machine.

The diagram in Fig.1 shows a localhost loopback benchmark with Soap-Stone running as a client and server on the same physical machine.


Fig.1: Soap-Stone Benchmarking

Fig.1: Using Soap-Stone to conduct a simple one-way localhost benchmark

What does this do? It runs a local loopback test from one program to another, on the same machine. Here's what the output for the client and server looks like on a 3 GHz Pentium-IV Hyper-Threaded system:


Fig.2: Client/Server Output

Fig.2: Client & Server output for simple one-way loopback benchmark.

The interesting results are shown in the Client window: the Server window will sit there quietly unless there is a failure. The result from this test is interesting: for 1500 byte size packets, sent 10000 at a time, the average throughput is 408 Mbit/sec! Sounds impressive, but what does this actually mean?

More than you might think. True it doesn't measure a real network between two machines. But when you look at a networking stack the physical network is only one part of the overall overhead in communicating information between agents (Fig.3).


Fig.3: The Network Stack

Fig.3: The Network Stack

What this first test represents is the best-case one-way throughput achievable on a given CPU architecture, running a given Java Virtual Machine, with no physical network overhead. Is this useful? Wait until we examine some results for communication procotols other than the raw sockets in java.net.Socket before casting judgement!

What conclusions can we draw from these results? Interestingly, this setup is capable of sustaining network traffic at rates faster than 100Mbit/sec, so a 100baseT physical network would not be fast enough to support even two similar machines communicating using this raw sockets protocol.

So we have our first conclusion: the latest desktop and server machines running Java should be provisioned with Gigabit Ethernet if they are bandwidth intensive.

1.3 Multiple Protocols

In Fig.3 you will notice that the Client and Server applications can exchange date using various protocols. What exactly does this mean? The default protocol for the Client is to use raw socket based communications with the server, and to send data in one direction only. But in fact Soap-Stone supports an extensible set of protocols, as well as various options for packet size, and other aspects of the communication. A typical configuration of SoapStone is shown in Fig.4:


Fig.4: Help!

Fig.4: Soap-Stone Built-in Help

The -help option passed into SoapStone shows the normal two usage modes of the soap-stone.jar, namely:

It is important to launch the server before the client, since the client will immediately start to establish communications with the server when it is started.

1.4 How Fast Is Your Protocol Today?

SoapStone also supports the concept of scenarios: a scenario is a set of tests to be performed by the client against a running server. (The client automatically instructs the server to switch protocols during testing, so scenarios are not applicable to the server, which unlike the client never terminates once it is stated). Let's take a look at the first scenario in action.

1.4.1 Scenario 1: All Protocols

Scenario processing is handled by the SoapStone driver program: all that the Client program sees is a sequence of invocations with different arguments. From the built-in help we see that scenario 1 reads: client -protocol [raw,http,rmi,soap] This simply invokes the Client program four times, with the following arguments:

ArgumentsEffects
-protocol raw Raw sockets protocol (the fastest?)
-protocol rmi The Java RMI protocol (Remote Method Invocation)
-protocol http HTTP protocol, typically used with web-servers, as implemented using Jetty
-protocol soap An implementation of the SOAP protocol, running on top of Jetty

This scenario assumes that you have a fully populated plugins subdirectory below the directory where you invoke soap-stone.jar. If you are running the soap-stone-full.jar this will be created and populated for you automatically. By "fully populated" we mean that the directory should contain the files raw-plugin.jar, rmi-plugin.jar, http-plugin.jar, and soap-plugin.jar. However, any missing protocols will result in benchmark numbers of zero Mbit/sec in the results.

After starting a server, run this scenario as follows: $ java -jar soapstone-full.jar -scenario 1. The resulting output is shown in Fig.6:


Fig.6: Help!

Fig.6: Scenario 1: Protocols

How does the selection of protocol affect the network performance? Dramatically, as the chart in Fig.7 shows. More on this in one moment.


Fig.7: Protocol Bandwidths

Fig.7: Protocol Bandwidths

1.4.2 Logging

You may be wondering how this chart was created? The driving SoapStone program writes out a comma-separated variable file suitable for direct importation into a spreadsheet. The file is called soap-stone.csv, and is overwritten for each scenario.

The file is written in such a way as to facilitate importing and charting in a spreadsheet program such as OpenOffice SpreadSheet. Consider the following scenario:

This results in a total of 12 test cases (2*2*3). What does the soap-stone.csv file look like? The easiest way to explain it is to show it as it appears in a spreadsheet.


Fig.8a: CSV Logging Tables

Fig.8a: CSV Logging Tables

For each parameter in the scenario, other than the right-most, a new table is created. So in this case there are four tables (2*2) by default. Each table is preceded by the combination of parameters that are in force for the numbers in the table (e.g. protocol=raw, echo=true for the first table). The right most parameter forms the first column of each table. The rows of the table are the average, minimum and maximum bandwidths measured during the test, in Mbit/sec.

This tabular format is reasonably convenient for graphing the results in a spreadheet.

Clearly, however, the number of tables will explode exponentially as the number of parameters is increased. So SoapStone supports some powerful mechanisms for logging only the data that you want, and controlling the table formats to simplify subsequent spreadsheet analysis.

The first mechanism allows control over the number of tables. By specifying parameter names as values for the -tables option, the other parameters are forced down into the column names for the data.


Fig.8b: Reducing Number of Tables

Fig.8b: Reducing Number of Tables

This presents the same information as in Fig.8a, but as two tables instead of four. The tables are named (prefixed with) the parameter(s) specified in the -tables argument.

This approach can be driven to its logical extreme: a single table with all the results in it, by using the -tables argument with no table names:


Fig.8c: Logging to One Table

Fig.8c: Logging to One Table

In all probability, this is too much data for use in a spreadsheet. SoapStone supports another method of reducing the amount of data, the -columns option.


Fig.8d: Further Reducing the Log

Fig.8d: Further Reducing the Log

1.4.3 Performance Analysis

What can we conclude about communication performance from this scenario? Clearly the Raw Sockets communications is by far the fastest way to communicate between two programs on the same machine (at least for 1500 byte packets, and in a unidirectional fashion). In fact, Raw Sockets are more than ten-times faster than the nearest rival, which is RMI, and more than one-hundred times faster than the SOAP protocol! To make this more evident, let's modify Fig.7 to use a logarithmic Y axis for the bandwidth:


Fig.9: Protocol Bandwidths (Log Scale)

Fig.9: Protocol Bandwidths (Log Scale)

At this point you are probably thinking "no big deal". Everybody knows SOAP is an inefficient transport mechanism, the SOAP envelope overhead is substantial with a payload of only 1500 bytes, so that probably explains the poor performance. Unfortunately, things aren't that simple as we will see next.

1.4.4 Payload Size

What payload size is reasonable for communications? For XML based service-oriented architectures (SOA'a) payload sizes in the 1Mbyte range are not unreasonable. Why? Simply because the SOA architecture tends to be based on stateless agents, and the entire state of a transaction thererfore has to be contained in a document. As an XML document flows through the SOA system, it will grow as it accumulates intermediate results as part of its being processed.

These sizes are in distinct contrast to those experienced when using remote-procedure call system architerctures, where many small messages will be exchanged between stateful agents. In such cases payload sizes will be at or below 100 bytes.

So how is performance impacted by payload size? As the payload size increases towards infinity, one would expect all of the protocols to converge to similar throughputs, since the various overheads would be neglegible with respect to the increasingly large payload sizes. But in practice this is not the case, at least with the four protocol plugins we are testing in this experiment as the following two figures show (the second chart uses a logarithmic Y-axis scale, the first chart uses a linear Y-axis scale).

The scenario being tested here is as follows:

The client and server programs are run on a single 3.0GHz Pentium-V/HyperThreading system, so no physical network is being measured. HyperThreading works well in this situation, with each virtual CPU running at only around 50% utilization. The results are therefore useful in predicting what speed of network would be required to fully service this class of machine in processing data of various sizes.

Note that this scenario does not echo data back from the server to the client, so this represents something of a best-case for throughput, since no extra burden is imposed by the return of data to the client.


Fig.9b: Protocol Bandwidths versus payload size
Fig.9b: Protocol Bandwidths versus payload size

Fig.9b: Protocol Bandwidths versus payload size


Fig.9b: Protocol Bandwidths versus payload size (log scale)

Fig.9c: Protocol Bandwidths versus payload size (log scale)

1.4.5 Observations

As might be expected, increasing payload size also increase throughput. But the increase is not monotonic: in each protocol there is a point beyond which increasing payload size causes a drop in throughput. For example, the raw protocol appears to be most efficient for payloads around 10,000 (10k) bytes in size, whereas the rmi protocol is most efficient for payloads around 100,000 bytes.

The raw protocol out-performs all other protocols at all payload sizes, sometimes by one or two orders of magnitude.

As payload size approaches 1,000,000 bytes, the raw and http protocols converge towards respectable throughputs in the 350Mbit/sec range. The rmi performance starts to lag for payloads at and above 100,000 bytes.

Small payloads (100 bytes) have very poor throughput, the best case being the raw protocol which manages to achieve around 30 Mbit/sec.

The soap protocol performance is simply dismal. It is an order of magnitude worse than the rmi and http protocols. The performance relative to http is disturbing: since SOAP uses HTTP as its transport one would expect that their performance would converge as the payload size increased. We speculate that the SOAP protocol is introducing a lot of extra overhead in the serialization and deserialization of data which is simply unacceptable for large payload sizes.

The throughput of all of the protocols starts to fall off for payload sizes above 100k.

1.4.6 Conclusions?

What can be concluded from this initial test? When provisioning a system for communicating data, the choice of communication protocol has a huge impact on the CPU resources that will have to be provisioned in order to achieve a given throughput. The SOAP protocol (at least in the implementation tested here) is quite inefficient for both smaller and larger payload sizes, and will require the most computational resources. RMI and HTTP are good choices for payloads above 10k, but for smaller payloads raw socket communications would be the best choice.

1.4.7 Meanwhile, back on the Server

As an aside, note that the server window does show something interesting in this case: as the client switches from protocol to protocol, the server is commanded to follow it, and it starts other servers (HTTP, RMI, SOAP) on demand to support the client (Fig.8):


Fig.10: Servers

Fig.10: Scenario 1: Servers

1.5 How Fast Is Your Phone-Net Today?

The spread-spectrum technology which underlies home-networking protocols such as Wi-Fi and Phone-Net makes the best use of available spectrum, spreading the signal around so as to be able to share the same physical medium with other communicating devices. It's extremely flexible, and can workaround large amounts of interference.

However, sometimes it can establish a connection where it really should not. Here's a real example. The physical network is a 10 MBps NetGear Phone-Net running over a typical home phone line setup. The home phone-line network was installed and worked fine for a couple of years, but something happened to the front room connection which made network traffic appear to be much slower than it should have been. How much slower? Soap-Stone provided the answer: for 1500 byte packets over a RAW protocol the connection from the front room of this house was one-hundred times slower than from the living room! Some crawling about under the house appears to be in order. Note that one cause of this kind of problem is inadvertantly plugging in your Phone-Net behind the line-filters intended for analog equipment. This was, unfortunately, not the case here.


Fig.11: Phone-Net Problems

Fig.11: Phone-Net Problems

1.6 Help!

The design of SoapStone is hierarchical: the top-level SoapStone driver program invokes the Client and Server programs based on the client or server argument passed to the top-level program. The Client and Server programs depend on protocol plugins, which themselves have their own applicable options. Options for server-side plugins are specified at the client, and passed to the server through the control commands issued at runtime.

The client and server programs each generate their own help. The plugins generate help relevant to their own operation:


Fig.11: Options

Fig.11: Client and Server Options

1.7 Where Next?

The next article in this series dives into the extended test modes that are supported by SoapStone. It also covers the plugin architecture which allows new protocols and dimensions to be easily added to the SoapStone framework so that you can customize it to test aspects of your own personal network topology.


If you like SoapStone then you might want to check out some of the other Open-Source projects developed by simontuffs.com:
soap-stone at sourceforge.net simontuffs.com