iPerf Network Performance Testing

iPerf is an open source network testing and troubleshooting tool. The purpose of iPerf is to measure end-to-end network throughput and line quality within data centers and internet connection. iPerf is also used to baseline cloud performance and troubleshoot connectivity issues. That include physical hosts, VMs, containers, and Kubernetes. iPerf does not provide reliable latency testing or application metrics.

The software is based on a client/server model where TCP or UDP traffic streams are generated between client and server sockets. This is referred to as memory-to-memory socket-level communication for network testing. iPerf3 supports disk read/write tests as well that identify server hardware as the performance bottleneck instead of the network. It is also a popular tool for network engineers and DevOps to troubleshoot network and application problems.

Use Case Examples

  • LAN/WAN throughput testing
  • Voice and video line quality
  • Troubleshooting performance
  • VM cloud throughput
  • Network stress testing
  • Wireless throughput
  • DevOps socket buffer tuning
  • Firewall open port testing

iPerf2 vs iPerf3 vs Speedtest

There are similarities and differences between iPerf2 and iPerf3 that affect network testing. It is important to understand the differences and limitations with each version to generate accurate reports. iPerf3 is Linux-based and not recommended or supported with Windows for various reasons. It is the most recent version and included with most Linux distro packages.

iPerf3 is not compatible with iPerf2 whether running on Linux or Windows. Each testing platform supports similar and different features that affect test results and reports generated. iPerf2 is an older testing platform that is supported on Windows and Linux.

There is anecdotal evidence that Windows-based testing can be problematic. Any test machine should be updated with the most current version of iPerf and Windows for best results. You can do testing between Windows and Linux machines as well provided they are running the same iPerf version (iPerf2 or iPerf3). Some features only available with iPerf3 include omitting TCP slow start from results, disk read/write tests, SCTP protocol, verbose reporting, and bandwidth options. There is also a control channel that sends test parameters to the server.

Speedtest is a popular online tool for measuring throughput that has both advantages and disadvantages. It is used primarily for testing the upload and download speed of your internet connection from a web browser. Speedtest also reports idle and loaded latency based on the server location that you select.

Speedtest does not test throughput between host endpoints within your network or administrative domain. For example, you cannot use Speedtest to measure throughput between your data center and a cloud server or another data center server. This is an important distinction between Speedtest and iPerf that reports on actual performance metrics for your network.

Speedtest is an effective tool for estimating bandwidth delay product (bdp) on an internet link where most latency occurs. Speedtest reports average speed in both directions, along with idle latency and loaded latency. Speedtest can only estimate performance to a data center in Los Angeles for example based on a local Speedtest server. Your actual internet connection would terminate at an ISP in Los Angeles where throughput and latency could vary. There are other factors that also affect results since it is not end-to-end testing to your data center.

It should be noted that many performance testing tools such as Ookla Speedtest report higher throughput than iPerf since they include headers. iPerf does not include protocol headers so throughput report is actual data payload. For example, iPerf would report 960 Mbps at best when testing Gigabit Ethernet (1000 Mbps) theoretical speed. There is approximately 4% deducted for IP and TCP headers that reduces actual goodput data transfer.

Throughput vs Latency

TCP-based applications comprise most internet and data center traffic. TCP window size and bandwidth delay product (bdp) will have the most effect on maximum data throughput. This is important to understand when analyzing iPerf or Wireshark reports with TCP attributes such as MSS, window size, scaling factor, and throughput. Application developers and DevOps can use iPerf and Speedtest for fine-tuning memory buffers from Windows and Linux default settings.

TCP throughput is most affected by network latency since it is used to calculate bandwidth delay product (bdp). Network engineers must minimize latency and packet loss particularly across internet links where network performance is most affected. TCP window size can expand to equal bandwidth delay product (bdp) when packet loss is minimal. The maximum TCP window size is only 64 KB unless window scaling is enabled and working correctly on client and server. UDP is much less affected by network latency since it is connectionless with less protocol overhead than TCP. It sends datagrams as fast as possible based on average latency and any rate limiting applied.

You could intuitively conclude that UDP throughput is higher than TCP however there is packet loss with UDP that could cause similar or even lower throughput. Fine-tuning UDP for larger packet size could increase throughput while balancing acceptable packet loss. DevOps engineers should also verify window scaling is working correctly and apply fine-tuning of memory buffers. This is most applicable to a long fat network (LFN) since it has high bandwidth and latency.

BDP (bps) = bandwidth (bps) x RTT latency (sec)

TCP window size (bytes) = BDP / 8 (bits/byte)

Throughput (data transfer) = TCP window size (bytes) / RTT latency (sec)

Data Throughput Calculation

You have an internet connection to Los Angeles that Speedtest reports as 150 Mbps download speed and 60 ms idle latency. What is the average data throughput per second you could expect when doing a file transfer?

BDP = bandwidth (bps) x RTT latency (sec)
= 150,000,000 bps x 0.060 sec
= 9,000,000 (9 Mbps)

TCP window size (bytes) = BDP / 8 (bits/byte)
= 9,000,000 bps / 8 bits/byte
= 1,125,000 bytes (1.125 MB) -> window scaling required (> 64 KB)

Throughput (data transfer) = TCP window size / RTT latency
= 1,125,000 bytes / .060 sec
= 18,750,000 (18.7 MB/sec)

Speedtest provides bandwidth and idle RTT latency to their nearest test server for an estimate only. You could also use MTR or ping for RTT latency and iPerf report for bandwidth to calculate bandwidth delay product (bdp).

iPerf reports throughput (data transfer) in MBytes (MB) and bandwidth (bit rate) in bps. This is based on report interval (-i) that you select with the default being 1 second. Throughput and bandwidth are interchangeable since bandwidth (bps) = data throughput / 8 bits. iPerf throughput reported represents how many bytes were sent per time interval. This is preferred since videos, files, and web pages are specified in bytes instead of bits. Throughput would typically double if the report interval is 2 seconds.

What Causes Latency?

Network latency is the most important performance metric that affects both throughput and application performance. There are five sources of latency that include propagation delay, transmission delay, queuing delay, processing delay, and protocol delay. Any reported latency represents the sum of all sources with a single RTT value. The affect of propagation delay is by far the biggest contributor. This is based on the distance between source and destination and varies with network media type (fiber, copper, wireless) and number of hops.

This also explains the popularity of CDN services that provide local access points where content is cached locally. The other significant contributor to latency is transmission delay that represents interface speed (bandwidth). Any bandwidth upgrade will reduce latency but also explains why it is not a quick fix solution since there are multiple causes of latency. Protocol delay is inherent to TCP with flow control and retransmits. DNS and SSL/TLS handshakes are a common source of protocol delay as well with web-based applications.

Most performance tools will report network latency as round-trip time (RTT). Measure latency from both directions since is not necessarily symmetrical and reverse path could be higher. This could be the result of a different reverse path with more hops and lower bandwidth links or queuing delays. Jitter is the amount of variation in latency also reported with iPerf and MTR. It is most relevant to voice and video delay-sensitive applications.

IP Fragmentation (MTU/MSS/PMTUD)

The topic of data throughput should also include MTU packet size and MSS data payload size. iPerf allows you to configure MSS size as a testing attribute and verify how that affects throughput. The global default for MTU packet size is 1500 bytes including headers and data payload. There is maximum 1460 bytes available for MSS data payload after deducting IPv4 header (20 bytes) and TCP header (20 bytes). The recommendation for network engineers and application developers is to use MTU 1500 bytes. The exception is VPN tunnels where network engineers implement MTU 1400 to account for additional headers. Most data applications will write the maximum MSS 1460 bytes payload required to maximize data throughput without causing fragmentation.

Fragmentation occurs when packets from a client arrive at a server or router with MTU larger than the inbound interface MTU size. This will cause the interface to fragment packet into multiple packets of the configured MTU size. It should be noted that fragmentation is typically enabled since packets with larger MTU would be discarded if fragmentation is disabled. MTU mismatch is not recommended since it causes processing delay and increases overall latency. PMTUD protocol is usually enabled by default on hosts to determine the lowest MTU on a network path and set accordingly. You can verify MTU setting on hosts and network devices with the following commands.

# tracepath -n (Linux)

c:\netsh interface ipv4 show destinationcache (Windows)

# mturoute -t (Windows/Linux)

How to Install iPerf

There are various methods available to install iPerf on your client and server machine. Most Linux distros now include iPerf3 however it is often not updated to the most current version. The following Linux commands will update Linux package then install iPerf2 or iPerf3. The links also provide binaries for multiple different Linux platforms.

iPerf2 for Windows is a single file download that is copied to C:\Windows\System32\ directory. Do not install iPerf3 with Windows since it is not supported.

iPerf2 / iPerf3 (Linux)
sudo apt-get update
sudo apt-get -y install iperf3
sudo apt-get -y install iperf

https://software.es.net/iperf/obtaining.html
https://downloads.es.net/pub/iperf/

iPerf2 (Windows)
https://sourceforge.net/projects/iperf2/
select download

Run Windows CMD as administrator
copy iperf-2.2.1-win64.exe from downloads to C:\Windows\System32 (or any other directory)

How iPerf Works

iPerf3 client creates a control channel on TCP port 5201 to start and stop tests. iPerf3 server listens on the same TCP or UDP port for test data based on selected protocol. This means your firewall must allow the same TCP and UDP port number for testing with iPerf3.

iPerf2 uses only a single TCP channel for all communication whether TCP or UDP tests. iPerf3 test parameters from the client are sent across the control channel to server along with test results from server. iPerf3 server listens on TCP port 5201 for a client connection by default and iPerf2 server listens on TCP port 5001. The client is randomly assigned an ephemeral (dynamic) port greater than 1023. You can assign a non-default port to server instead of 5201 or 5001 with -p option. The client command must include the new destination port as well since client will send packets to default port if not specified.

Tests will sometimes end with data that is still in-flight between client and server. This effect can be significant for tests less than 5 seconds, however it is often negligible for longer tests. iPerf commands on the TCP control channel can terminate testing before all of the test data has been processed. The result is a mismatch between the data (bytes) sent from client and what was received at server (data transfer) or vice versa. There is also server buffer overflow that could account for more data sent than received as well. iPerf also assigns an identifier [4] [5] that represents a unidirectional session from client to server or vice versa.

default mode
[4] sender (client) – upload speed (bit rate) from client to server
[4] receiver (server) – download speed (bit rate) on server

reverse mode
[5] sender (server) – upload speed (bit rate) from server to client
[5] receiver (client) – download speed (bit rate) on client

Tips and Tricks

  • Run full bandwidth tests off-hours or maximum 10 seconds during the day to minimize bandwidth and CPU usage.
  • Stop server mode service after testing to prevent unauthorized client connections.
  • Identify any host-based and/or network firewalls to request open ports TCP/UDP 5201 (iPerf3), TCP 5001 (iPerf2), or select a common port (TCP 443).
  • Reverse mode supports bidirectional testing across NAT and firewalls.
  • Client network interface speed determines the maximum bit rate available for testing.
  • Update to the most current version of iPerf for best results.
  • RDP can initiate tests remotely from different locations to test servers or laptops.
  • iPerf reports actual data payload throughput and not theoretical capacity

Bidirectional vs Full-Duplex

It is important to understand the difference between bidirectional and full-duplex network communication. Most applications (HTTP, FTP, DNS, etc.) are bidirectional. Ethernet and serial interfaces support multiple full-duplex bidirectional application sessions across a single physical link. The noted exception is your wireless network that is bidirectional half-duplex communication.

Each application session creates a single socket on client and server. This is referred to as bidirectional same socket communication and represents TCP protocol. The exception is an application such as FTP that has separate sockets for control channel and data channel. Most bandwidth usage however is across the data channel. Full-duplex traffic is sent in both directions simultaneously over the same socket. This is really only a Layer 2 concept that does not apply to upper layer protocols.

UDP creates unidirectional sessions since it is connectionless and well-suited to voice and video traffic. UDP sends traffic in a single direction and supports bidirectional traffic using a separate socket for some different function. Each side of a voice call for example would have separate UDP sockets. There is also a UDP socket that is designated for control signaling. This is the logical equivalent of starting a new TCP application session from another host.

There are multiple bidirectional testing options available with iPerf2 and iPer3. They are labeled bidirectional, dualtest, tradeoff, reverse mode, and full-duplex. iPerf2 and iPerf3 support reverse mode (-R) testing that measures throughput from server to client. The default setting for iPerf is testing throughput from client to server. The advantage of reverse mode is same socket communication initiated from client that enables NAT and firewall traversal. The firewall for example would not permit a new socket from an existing connection in the reverse direction. There is bidirectional mode (–bidir) with iPerf3 that also supports NAT and firewall traversal since client initiates socket.

iPerf2 bidirectional full-duplex test is based on concurrent read/write operations at client and server. This measures throughput that simulates Ethernet physical layer traffic . There is dualtest and tradeoff mode with iPerf2 that is a variant of unidirectional or bidirectional half-duplex mode. They create separate sockets while sending traffic in both directions. NAT and firewall traversal is not supported with dualtest and tradeoff mode since client and server initiate separate sockets independently.

Symmetrical vs Asymmetrical

Throughput test results are directly affected by the ISP assigned bit rate for upload and download speed. Symmetrical bandwidth refers to a network link with the same upload and download speed such as Ethernet, serial, and fiber. This occurs whether the link transits an ISP or within a data center. Asymmetrical bandwidth refers to a network link that is assigned different upload and download bit rates. This is typical of broadband services such as DSL, LTE, and cable where download speed is often faster than upload speed. This aligns with web-based applications that make requests and download large amount of data such as videos, large web pages, and files.

iPerf reports results as upload speed from client and download speed to server. This is a unidirectional flow from client to server unless you select reverse mode or bidirectional testing. Throughput for a symmetrical link should be identical in both directions unless rate limiting or throttling exists between client and server. The same testing applies to asymmetrical links where iPerf will report upload and download speed. The reports refer to throughput as data transfer (number of bytes) and bit rate (bps) as bandwidth.

  • average bit rate (bps)
  • maximum throughput (bytes)
  • packet loss (%)
  • jitter (ms)
  • retries (retransmits)
  • read/write buffer size
  • socket buffer size
  • congestion window (cwnd)
  • window scaling
  • MSS size for path forwarding
  • CPU usage with verbose report

The results for iPerf report average bit rate (bandwidth) and maximum data throughput in bytes. The amount of throughput increases with test duration. There is also the option of testing with larger TCP window size or UDP packet size to measure data throughput.

TCP window scaling is enabled by default with Windows and Linux. The purpose of window scaling is to enable TCP window size (RWND) that is larger than default maximum 64 KB. This is provided there is low packet loss on the link and bandwidth delay product is more than 64 KB. You will also notice the TCP congestion window (CWND) is larger than 64 KB as well. TCP congestion window and receive window is based on network conditions so that when latency and/or packet loss is detected, they are reduced along with throughput. accordingly. The lower of congestion window and receive window is implemented at any point in time.

There are also retries (retransmits) reported with iPerf that refer to number of packet drops per time interval. For example, you could have a duplex mismatch on your ISP link causing packet loss on an Ethernet interface. TCP protocol would request retransmission and that would report as retransmit. The recommended maximum number of retries should be less than 1% of data throughput. The following calculation will provide some context when 100 retries are reported for 10 MB data throughput. The calculation suggests this is worth investigating further since it is marginal performance with more than 1% retries. Another cause of retries is ISP rate-limiting for example when internet traffic bursts exceed CIR.

100 retransmits x 1500 bytes (packet) / 10,000,000 bytes = 1.5%

iPerf Reference Guide: Network Engineers / DevOps

iPerf reference guide includes commands, test examples, and troubleshooting notes.

  • TCP Throughput
  • TCP Reverse Mode Throughput
  • TCP Parallel Streams Throughput
  • Line Quality (voice and video)
  • Link Saturation Throughput
  • Network Stress Testing
  • Disk Read/Write Throughput
  • Non-Default Server Listening Port
  • Jumbo Frames MTU
  • What-if Simulation Testing

Example: TCP Throughput

The purpose of this test is to measure TCP throughput and retries (retransmits) from client to server with MSS 1460 bytes. TCP throughput measures average bit rate and maximum data transfer for a single session by default. Throughput testing is limited by the client network interface. You cannot test 10 Gigabit Ethernet links with a client that only has a Gigabit interface.

TCP control channel does PMTUD to verify lowest MTU on forwarding path. TCP channel detects the default MTU 1500 bytes on forwarding path and assigns MSS 1448 bytes. There are 12 bytes deducted from MSS 1460 since TCP timestamps option is enabled. UDP derives the same default packet size value from TCP PMTUD/MTU discovery without fragmentation. The data channel assigns MSS 1460 with -M 1460 parameter setting for this test. You could also omit -M parameter and iPerf would automatically detect path MTU and include with report for troubleshooting purposes.

The other settings are to allow TCP slow start (-O 2) to finish before collecting results and enable verbose mode (-V) for a detailed report. The advantage of TCP slow start option is more accurate results for data throughput by removing initial protocol overhead. TCP slow start command omits the first two seconds of testing.

TCP will use all bandwidth available by default with a report interval (-i) of 1 second. Consider a longer test duration (-t) of 5 minutes (300 seconds ) instead of 10 seconds (default) when troubleshooting. Start the server first to listen for client connections on default port TCP 5201. The server mode service is stopped with Ctrl + C key to prevent unauthorized testing.

Start iPerf3 server mode

# iperf3 -s

Start iPerf3 client mode

# iperf3 -c 192.168.122.3 -M 1460 -O 2 -V

Network Performance Troubleshooting Masterclass