Wlanpi a la Mode

Have you had a chance to check out the wlanpi project yet? Wlanpi is a tiny odroid linux box that packs a punch with several free test tools pre-installed on its default image. Go to wlanpi.com for more info including where to source parts and software. Fortunately, there has been a lot written on how to use the existing wlanpi tools and how to turn it into a remote packet sniffer/wlan sensor. Here I’ll show you how to add another cool tool to your wlanpi kit for throughput testing and graphical analysis.

This tool is named Flent – FLExible Network Tester developed by Toke Høiland-Jørgensen. Flent is a set of python scripts that wrap and run netperf, netserver and other tools which allow you to take your throughput testing to another level…with graphs…lots and lots of data-driven, presentation-ready sharp looking graphs.  The graphing capability is IMHO what sets this tool apart, specifically because it allows you the ability to easily compare data sets from previous tests and see differences between test runs.

NOTE: You may want to experiment on a freshly etched wlanpi image on a separate memory card so you do not fubar anything you have already setup on your existing install. I am using two usb-to-wifi adapters and have unmasked the wpa_supplicant.service so I can use wlan1 as a traffic source/sink with the following command:
wlanpi@wlanpi:~$ sudo systemctl unmask wpa_supplicant.service

Once unmasked, you can use the nmtui to configure wlan1.
wlanpi@wlanpi:~$ sudo nmtui

Here is the method of installation I recommend based on my testing on the 1.5.0 wlanpi image running on my wlanpi with an internet connection!

Flent – install from github.
wlanpi@wlanpi:~$ git clone https://github.com/tohojo/flent.git
wlanpi@wlanpi:~$ cd flent
wlanpi@wlanpi:~/flent$ sudo python setup.py install

Netperf/Netserver – install from github and be sure to use “–enable-demo=yes” and compile netperf to enable netperf to emit “interim results” at semi-regular intervals…as stated in the user guide for this feature.
wlanpi@wlanpi:~$ git clone https://github.com/HewlettPackard/netperf.git
wlanpi@wlanpi:~$ cd netperf
wlanpi@wlanpi:~/netperf$ sudo ./autogen.sh
wlanpi@wlanpi:~/netperf$ sudo ./configure --enable-demo=yes
wlanpi@wlanpi:~/netperf$ sudo make
wlanpi@wlanpi:~/netperf$ sudo make install

Fping – version 3.15 is already on the wlanpi image, but Flent would like a higher version. Install from github.
wlanpi@wlanpi:~$ git clone https://github.com/schweikert/fping.git
wlanpi@wlanpi:~$ cd fping
wlanpi@wlanpi:~/fping$ sudo ./autogen.sh
wlanpi@wlanpi:~/fping$ sudo ./configure
wlanpi@wlanpi:~/fping$ sudo make
wlanpi@wlanpi:~/fping$ sudo make install

You can install flent and netperf/netserver using apt on the wlanpi, but you will be using older versions than what is available from github.

Flent works similarly to iperf in that there is a server and a client required to run tests. The server runs on one system and the client runs on another. It is up to you to decide how you want to run tests, but there are command-line options to swap upload and download direction for example if you need it. I recommend running the flent tests on the wlanpi which means a netserver process will need to run on a separate system such as a laptop that is also connected to the same network under test. For a more detailed explanation…head over to Flent intro page: https://flent.org/intro.html

For your laptop or other system that will be acting as a netserver, you will need to install the flent-gui as well as flent, netserver and fping so you can run tests and then scp results files from your wlanpi and view graphical results. You will have to install additional packages to get the flent-gui to display correctly. The flent-gui does have the capability to run tests directly from its interface, but I am not covering that feature in this post. Here is what I installed on a Ubuntu 18.04 laptop and then I was able to install flent, netserver and fping and then run flent-gui.

sudo apt install python-setuptools-git autotools-dev automake python-qt4 python-matplotlib

After the installs are completed, you can now run tests. Start netserver on your laptop or other server system by simply typing “netserver” and then ssh to the wired interface on the wlanpi and run one or more of the flent tests on the wireless interface.

To see what tests are available, type “flent –list-tests” (it takes a couple of minutes for this command to complete…not sure why). I have not tried all of the available tests and you may need other software installed to run some of the other tests, but the main ones I have used are tcp_download, tcp_upload, rrul and udp_flood. I have experimented a bit with the multi stream tcp and the different rrul tests, but you can try the ones that look interesting to you and see what happens!

wlanpi@wlanpi:~$ flent –list-tests
Available tests:
bursts : Latency measurements under intermittent UDP bursts
bursts_11e : 802.11e Latency measurements under intermittent UDP bursts
cisco_5tcpup : RTT Fair Realtime Response Under Load
cisco_5tcpup_2udpflood : Cisco 5TCP up + 2 6Mbit UDP
cubic_bbr : Cubic VS BBR smackdown
cubic_cdg : Cubic VS CDG smackdown
cubic_dctcp : Cubic VS DCTCP smackdown
cubic_ledbat : Cubic VS Ledbat smackdown
cubic_ledbat_1 : Cubic vs LEDBAT upload streams w/ping
cubic_reno : Cubic VS Reno smackdown
cubic_westwood : Cubic VS Westwood
dslreports_8dn : 8 down – dslreports dsl test equivalent
http : HTTP latency test
http-1down : HTTP get latency with competing TCP download stream
http-1up : HTTP get latency with competing TCP upload stream
http-rrul : HTTP get latency with competing RRUL test
iterated_bidirectional : Iterated TCP bidirectional transfers example
ledbat_cubic_1 : Cubic vs LEDBAT upload streams w/ping
ping : Ping test (ICMP and UDP)
qdisc-stats : Capture qdisc stats
reno_cubic_westwood_cdg : Realtime Response Under Load
(with different congestion control algs)
reno_cubic_westwood_ledbat : Realtime Response Under Load
(with different congestion control algs)
reno_cubic_westwood_lp : Realtime Response Under Load
(with different congestion control algs)
rrul : Realtime Response Under Load
rrul46 : Realtime Response Under Load – Mixed IPv4/6
rrul46compete : Realtime Response Under Load – Mixed v4/v6 compete
rrul_100_up : 100 up vs 1 down – exclusively Best Effort
rrul_50_down : 50 down vs 1 up – exclusively Best Effort
rrul_50_up : 50 up vs 1 down – exclusively Best Effort
rrul_be : Realtime Response Under Load – exclusively Best Effort
rrul_be_iperf : Realtime Response Under Load – exclusively Best Effort (Iperf TCP)
rrul_be_nflows : Realtime Response Under Load – Best Effort, configurable no of flows
rrul_cs8 : Realtime Response Under Load CS8, one flow per CS/precedence level
rrul_icmp : Realtime Response Under Load – Best Effort, only ICMP ping
rrul_noclassification : Realtime Response Under Load – no classification on data flows
rrul_prio : Realtime Response Under Load – Test Prio Queue
rrul_torrent : Torrent-like competition
rrul_up : Realtime Response Under Load – upload only
rtt_fair : RTT Fair Realtime Response Under Load
rtt_fair4be : RTT Fair Realtime Response Under Load
rtt_fair6be : RTT Fair Realtime Response Under Load
rtt_fair_up : RTT Fair upstream only
rtt_fair_var : RTT Fair – variable number of hosts
rtt_fair_var_down : RTT Fair – variable number of hosts (download only)
rtt_fair_var_mixed : RTT Fair – variable number of hosts (mixed up and down)
rtt_fair_var_up : RTT Fair – variable number of hosts (upload only)
sctp_vs_tcp : SCTP vs TCP
tcp_12down : TCP download – 12 streams w/ping
tcp_12up : TCP upload – 12 streams w/ping
tcp_1down : Single TCP download stream w/ping
tcp_1up : Single TCP upload stream w/ping
tcp_1up_noping : Single TCP upload stream
tcp_2down : TCP download – 2 streams w/ping
tcp_2up : TCP upload – 2 streams w/ping
tcp_2up_delay : Two TCP upload streams; 2nd stream started delayed
tcp_2up_square : Two TCP upload streams; 2nd stream started delayed
tcp_2up_square_westwood : Two TCP upload streams; 2nd stream started delayed
tcp_4down : TCP download – 4 streams w/ping
tcp_4up : TCP upload – 4 streams w/ping
tcp_4up_squarewave : Four TCP upload streams; 2nd streams started delayed, cubic vs BBR
tcp_6down : TCP download – 6 streams w/ping
tcp_6up : TCP upload – 6 streams w/ping
tcp_8down : TCP download – 8 streams w/ping
tcp_8up : TCP upload – 8 streams w/ping
tcp_bidirectional : Bidirectional TCP streams w/ping
tcp_download : TCP download stream w/ping
tcp_ndown : TCP download – N streams w/ping
tcp_nup : TCP upload – N streams w/ping
tcp_upload : TCP upload stream w/ping
tcp_upload_1000 : 1000 up – exclusively Best Effort
tcp_upload_prio : TCP upload stream w/ToS prio bits
udp_flood : UDP flood w/ping
udp_flood_var_up : UDP flood w/ping – variable number of hosts
udp_flood_var_up_stagger : UDP flood w/ping – variable number of hosts, staggered start
voip : VoIP one-way stream test
voip-1up : VoIP one-way stream test with competing TCP stream
voip-rrul : VoIP one-way stream test with competing RRUL test

A word about RRUL – the Realtime Response Under Load test attempts to saturate the traffic link with four TCP upload streams and four TCP download streams each of which are marked to correspond to different QoS traffic classes. The point of this test is to identify potential bufferbloat in the network under test. Bufferbloat is the effect of having too many network buffers causing unwanted and excessive delay in a network. The advantage of running the RRUL test on a wireless network is that it can show how a highly contentious RF environment can behave as badly or worse than a bufferbloated wired network. By running RRUL and its variants, you can measure how the existing or modified queuing mechanisms implemented by vendors of your networking equipment stand up to a traffic load as well as collect the data to visually inspect what happens when someone complains that “the wifi is slow.” ™
For additional info on this topic, see the following:
The Blind Men and the Elephant
Make Wifi Fast Project
Battle Mesh write-up

To view the results, scp the *.flent.gz results files to your flent-gui system, then run the flent-gui to view the data. Add additional data sets by going to the Data menu. As you can see there are plenty of graphs to choose from.

Flent_GUI

Below is an example comparison of a TCP upload test on a wifi network with 25 other transmitting stations and again without other stations using the network. You can see the difference in latency results between the tests when many stations vs one station are contending for airtime.

You can use IP addresses for the -H and –local-bind options, or you can edit your /etc/hosts file on the wlanpi to use human readable names. The -x collects metadata about your test, –socket-stats collects TCP stats but only for the upload test and -t allows you to give your test run a title. There are many other options of course to control how flent behaves. Type “flent –help” to see them or visit flent.org

wlanpi@wlanpi:~$ sudo flent -x -H laptop --local-bind wlan1 tcp_upload --socket-stats -t tcp-up-25stas-tx
wlanpi@wlanpi:~$ sudo flent -x -H laptop --local-bind wlan1 tcp_upload -t tcp-up-no-stas

The above two commands each ran a 70 second TCP upload test which resulted in two *.flent.gz files which I scp’d to the laptop and then opened with flent-gui to obtain the following graphs of the two data series. The box plot shows better upload results and the Ping CDF plot shows better ping times when comparing no stations to many transmitting stations.

tcp-up-25stas-tx_vs_tcp-up-no-stas_box-plot-totalstcp-up-25stas-tx_vs_tcp-up-no-stas_ping-cdf

Next up is a rrul test where a comparison of the same test network once with 25 transmitting stations and again without other stations using the network.

wlanpi@wlanpi:~$ sudo flent -x -H laptop --local-bind wlan1 rrul --socket-stats -t rrul-25stas-tx
wlanpi@wlanpi:~$ sudo flent -x -H laptop --local-bind wlan1 rrul --socket-stats -t rrul-no-stas

The box plot shows better TCP download results and the ICMP CDF plot shows better ping times when comparing no stations to many transmitting stations.

rrul-25stas-tx_vs_rrul-no-stas_box-plot-totalsrrul-25stas-tx_vs_rrul-no-stas_icmp-cdf

You can also build flent batch files to run different tests or the same test over and over again instead of running the tests one at a time. The batch capability is also quite useful because you can kick off other scripts or initiate a wireshark capture when a particular test runs.

The batch file below named flent-batch.txt names three tests and repeats each of the three tests five times.

wlanpi@wlanpi:~$ cat flent-batch.txt
[Batch::tcp-down]
test_name = tcp_download
hosts = laptop
title = wlanpi-tcp-down rep:${repetition}
filename_extra = wlanpi-tcp-down-${repetition}
repetitions = 5
[Batch::tcp-up]
test_name = tcp_upload
hosts = laptop
title = wlanpi-tcp-up rep:${repetition}
filename_extra = wlanpi-tcp-up-${repetition}
repetitions = 5
[Batch::rrul]
test_name = rrul
hosts = laptop
title = wlanpi-rrul rep:${repetition}
filename_extra = wlanpi-rrul-${repetition}
repetitions = 5

Then to run the entire batch file, you use the following command:

wlanpi@wlanpi:~$ sudo flent -x --local-bind wlan1 -b tcp-down -b tcp-up -b rrul -B flent-batch.txt

This lengthy post only scratches the surface of what Flent can do, but I wanted to demonstrate that this tool can be very useful for collecting data on how your networks are behaving. The fact that it runs on linux and the wlanpi is a sweet bonus! With the many test options, batch operations and graphing capabilities of Flent, you can turn your wlanpi into a significant little test system for your wifi networks. A big thanks to @jolla @wlanpi and others for their work on this project. Keep Testing!

Additional Articles on using WlanPi

Here is the main user guide found at wlanpi.com
https://docs.google.com/document/d/13W31uoOfefEJIlWu1m7gt5pFws8YSt_xcbRZt2hidQA/edit

Here are a handful of links to articles written about using the wlanpi:

Fun with iperf

For a quick throughput tester, iperf can come in very handy. Here are some basics with setting up iperf and running tests. Note that iperf refers to version iperf2.

Install on Linux or Windows – Just download the binary you need from iperf.fr and go!

   Keys to remember: iperf serveR Receives traffic, iperf clienT Transmits traffic. There are options and methods to change this behavior, but in general this is the default. How do I know this? Well, the answer is experience! and sniffing traffic!

   After installation, the next step is to start an iperf server on the system you want to receive traffic…how do you decide? Well, it depends on the type of environment you are testing. If you’re testing a wired setup on a local LAN it doesn’t much matter. If you’re testing a wifi network you may want an upload or download test…the bottom line is for most cases, the iperf server will be receiving the traffic flow from the client.

Start an iperf server on any interface:
iperf -s
iperf3 -s

Start an iperf server on a specific interface:
iperf -s -B
iperf3 -s -B

⇒   After starting your iperf server, move along to the next system and start the client.

Start an iperf client with TCP traffic to run for 30s and report every 1s:
iperf -c -t30 -i1

Start an iperf client with UDP traffic to run for 30s, report every 1s and try to achieve 1Gbps throughput:
iperf -c -t30 -i1 -u -b1G

Start an iperf client with TCP traffic for 30s reporting every 1s and use a specific interface for the connection.
iperf -c -t30 -i1 -B

Run the client in reverse mode (-R) where the client receives and the server transmits. Although the man page for iperf2 does not show this option, it worked for me on both iperf2 and iperf3.
iperf -c 192.168.3.2 -B 192.168.2.14 -t30 -i1 -R

Run the client in dualtest mode which is a simultaneous bidirectional test where the client and the server transmit and receive at the same time! This option requires the use of -L for a listen port to receive bidirectional tests on and importantly only works with iperf2 and not with iperf3.
iperf -c 192.168.3.2 -B 192.168.2.14 -t30 -i1 -d -L 5002

Run the client in tradeoff mode which is an individual bidirectional test. Tradeoff (-r) first sets the client to transmit for 30s in this example, then sets the server to transmit and run for 30s. This option also requires the use of -L for a listen port to receive bidirectional tests on and importantly only works with iperf2 and not with iperf3.
iperf -c 192.168.3.2 -B 192.168.2.14 -t30 -i1 -r -L 5002

Run the client with multiple parallel streams for multiple connections to the server. Although this option is available for iperf2, I was only able to reliably get it to work with iperf3.
iperf3 -c 192.168.3.2 -B 192.168.2.14 -t30 -i1 -P 5

Iperf has many other options that you can experiment with to find the right mix for your testing. One of the cool things about iperf is that there are a few different Android and IOS apps that also allow you to use your mobile device as an iperf endpoint. Here is an example of  the HE.net app described at https://networktools.he.net/

An iperf3 client to connects to another iperf3 server:

Because iperf can be run on many different platforms with many different options, it is a great tool to learn how to use for your throughput testing toolbox. Enjoy!

802.11r Fast BSS Transition

This is not an in-depth post on 802.11r. Instead, it is just about one piece of the 802.11r puzzle that will hopefully remain in my brain long enough for certification and beyond!

For a detailed review of 802.11r, I suggest Devin Akin’s excellent white paper:
802.11_RSN_FT.pdf

And, Rasika Nayanajith’s excellent blog posts:
802.11r FT over the DS and 802.11r FT over the air

I was having trouble understanding the difference between two of the wpa_cli commands ft_ds and roam. The wpa_cli is the WPA command line interface for interacting with wpa_supplicant (as a station in client mode).

The wpa_cli command help gave me a clue, but wasn’t giving me the full story:
ft_ds = request over-the-DS FT with
roam = roam to the specified BSS

So I setup a Cisco testbed with a WLC2504 and two 1602 APs. Then setup a LANforge-WiFIRE test system with a client to roam between the two APs and with a separate monitor interface for packet captures.

With the wpa_cli ft_ds command I obtained the following capture:
wlan0-ath9k-2ft_ds-2roam.pcapng-002

With the wpa_cli roam command I obtained the following capture:
wlan0-ath9k-2ft_ds-2roam.pcapng-005

Staring at the packet captures and re-reading the following helped me see the difference:
Over-the-DS
• STA communication to the target AP via current AP using FT Action frames.
Over-the-Air
• STA communicates directly with the target AP using FT Authentication frames.

The command wpa_cli ft_ds dc:a5:f4:ff:4f:af initiates a FT Request (Action Frame) from the station 00:0e:8e:43:3a:71 to the target AP dc:a5:f4:ff:4f:af via the current AP dc:a5:f4:f3:ce:9f that the station is connected to.
wlan0-ath9k-2ft_ds-2roam.pcapng-003-marked.png

The command wpa_cli roam dc:a5:f4:ff:4f:af initiates a FT Authentication from the station 00:0e:8e:43:3a:71 over the air directly to the target AP dc:a5:f4:ff:4f:af.
wlan0-ath9k-2ft_ds-2roam.pcapng-006-marked.png

Both commands initiate an 802.11r Fast BSS Transition, but the distinction between the two is important to understanding how clients will transition. Using this command is a helpful way to test 802.11r wireless network setups and verifying that when it comes time for real clients to roam, you will have some confidence that it will work as expected.

Comparing USB-to-Wifi Adapters

After purchasing a handful of the Eye P.A. supported USB-to-Wifi adapters to see how well they could capture wifi traffic when compared to a dedicated PCI-E wifi NIC, I found that some of the adapters were better than others but none as good as the dedicated NIC.

Test Setup: Using a LANforge CT523b wifi traffic generator and an Aruba 315 AP setup on channel 100, I setup a single wifi client for a 10 second over-the-air UDP download at 500Mbps with a 1472B payload size. A second wifi NIC in monitor mode was used on the LANforge system to perform a 15 second wifi capture to compare against each USB adapter capture using the Eye P.A. capture tool.

The first 3 rounds were control tests to see how well the dedicated NIC would capture wifi traffic:

Round 1
    wiphy2 RX Pkts 217788
    wlan2 RX Pkts 424631
    moni5a
        captured 109306
        dropped 106672
Round 2
    wiphy2 RX Pkts 214776
    wlan2 RX Pkts 424618
    moni5a
        captured 109218
        dropped 106119
Round 3
    wiphy2 RX Pkts 215087
    wlan2 RX Pkts 424568
    moni5a
        captured 108803
        dropped 107875

The next 5 rounds compared different USB adapters to the dedicated NIC wifi capture:

Round 4 – adding capture device Edimax 7833
Eye P.A. captured 8730 pkts with 4240 data frames
    wiphy2 RX Pkts 213697
    wlan2 RX Pkts 424569
    moni5a
        captured 114925
        dropped 102400
Round 5 – capture device Linksys 6300
Eye P.A. captured 4970 pkts with 9 data frames
    wiphy2 RX Pkts 213462
    wlan2 RX Pkts 416002
    moni5a
        captured 109170
        dropped 106777
Round 6 – capture device Asus AC68
Eye P.A. captured 7240 pkts with 3276 data frames
    wiphy2 RX Pkts 138598
    wlan2 RX Pkts 241384
    moni5a
        captured 91625
        dropped 65146
Round 7 – capture device Asus AC68
Eye P.A. captured 7957 pkts with 4269 data frames
    wiphy2 RX Pkts 213985
    wlan2 RX Pkts 424611
    moni5a
        captured 108988
        dropped 109630
Round 8 – capture device Comfast Realtek 8812
Eye P.A. captured 6560 pkts with no data frames captured
    wiphy2 RX Pkts 226968
    wlan2 RX Pkts 424522
    moni5a
        captured 111036
        dropped 58637

Next steps are to try this test again under Linux or modify some of the other test setup to see if there is any benefit. Also I’d like to try the test against other APs and wifi modes, but the overall result appears to be that the USB-to-wifi adapters are not a good as a dedicated wifi NIC…but you probably already knew that!

Hidden SSIDs

How hidden are hidden SSIDs? Hidden SSIDs are only “hidden” because the AP doesn’t put the SSID (wlan_mgt.ssid) in the Beacon (wlan.fc.type_subtype==0x0008) and that’s it.

hidden-ssid-beacon

Beacon

However, if any device that already has the SSID configured, sends an
Association Request (wlan.fc.type_subtype==0x0000) or a
Probe Request (wlan.fc.type_subtype==0x0004), then the SSID is not hidden anymore.

hidden-ssid-assoc-req

Association Request

hidden-ssid-probe-req

Probe Request

As demonstrated here, if you’re running a wifi capture and happen to sniff these two requests or you happen to get the Probe Response (wlan.fc.type_subtype==0x0005), then you get the SSID!!!

hidden-ssid-probe-resp

Probe Response

It’s kinda like telling your toddler “Don’t tell Mommy we got her a surprise birthday cake!” If anyone asks how her day was, it only takes about 5 seconds for her to blurt out “We got cake!”

The takeaway here is…don’t bother hiding your SSID…there are better ways to secure your wlan network as well as surprise your loved ones with cake!!!

Measuring Beacon Interval

Here is my attempt to take a wifi capture file and use the IO Graph feature in Wireshark to measure the beacon interval as well as see how beacons behave with wifi traffic.

The wifi beacon from the AP is configurable and the default is usually around 100ms. But, this is more accurately described in the CWNP article:
https://www.cwnp.com/cwnp-wifi-blog/80211-beacon-intervals/

In the Wireshark capture, I can filter on beacons by using wlan.fc.type_subtype==0x0008 and see the time delta from the previous displayed frame as 0.1025 seconds then 0.1023 seconds then 0.1024 seconds and so on which makes sense…but a graph would be much better to visualize what is happening.

Starting with the how-to I found here,
https://www.fir3net.com/Networking/How-Tos/how-to-graph-tcp-delta-times-in-wireshark.html
I started building a graph for beacon intervals. Going to Wireshark – Statistics – IO Graphs brings up the IO Graphs window. The default graph is for All Packets, but you modify this with a display filter and then add Y-axis filters if needed. After adding the entries for time deltas for Min/Max/Avg beacons, I also added a display filter for QoS Data to show when traffic was being transmitted.
Wireshark · IO Graphs · wifi-capture_045
The default time scale is too big to see what is going on, so next I set the Interval to 100ms.

Wireshark · IO Graphs · wifi-capture_040
Here the time scale to too compact, so next I zoom in.

Wireshark · IO Graphs · wifi-capture_041
Now you can see that the Max Beacon Interval before traffic starts and after traffic stops is measurable at 100000 microseconds or 100ms. There is a big gap at 10.24 seconds where the beacon was delayed for some reason, but I haven’t figured that out yet. When traffic starts, the beacon intervals are scattered among the traffic carrying frames because now there is more contention for air time.

Wireshark · IO Graphs · wifi-capture_043
Zooming in further shows how closely yet separate the beacons are from the data traffic.

Wireshark · IO Graphs · wifi-capture_044

Now with the original goal accomplished, measuring the beacon interval, this post also reveals that beacon intervals can change as air time becomes limited. Another cool trick is that you can also measure the beacon interval with this tshark command:

tshark -r wifi-capture.pcap -q -Y wlan.fc.type_subtype==0x0008 \
-z io,stat,0.01,”AVG(frame.time_delta_displayed)frame.time_delta_displayed” \
|grep -v “0.000000”

Partial output of this is:

====================================================================
| IO Statistics                                                    |
|                                                                  |
| Duration: 29.39 secs                                             |
| Interval:  0.01 secs                                             |
|                                                                  |
| Col 1: AVG(frame.time_delta_displayed)frame.time_delta_displayed |
|------------------------------------------------------------------|
|                |1         |                                      |
| Interval       |    AVG   |                                      |
|---------------------------|                                      |
|  0.10 <>  0.11 | 0.102500 |                                      |
|  0.20 <>  0.21 | 0.102273 |                                      |
|  0.30 <>  0.31 | 0.102406 |                                      |
|  0.40 <>  0.41 | 0.102391 |                                      |
|  0.51 <>  0.52 | 0.102361 |                                      |
|  0.61 <>  0.62 | 0.102445 |                                      |
|  0.71 <>  0.72 | 0.102400 |                                      |
|  0.81 <>  0.82 | 0.102398 |                                      |
|  0.92 <>  0.93 | 0.102402 |                                      |
|  1.02 <>  1.03 | 0.102402 |                                      |
|  1.12 <>  1.13 | 0.102409 |                                      |
|  1.22 <>  1.23 | 0.102393 |                                      |
|  1.33 <>  1.34 | 0.102381 |                                      |

Thanks for taking a look at this post!