MAP65 Statistics during 2012 ARRL International EME Competition

During the second weekend of the 144 MHz portion of the 2012 ARRL International EME Competition I wanted to gather some data regarding the performance of MAP65 and various configurations of WSJT.  To do this, I ran MAP65, with input from Leif Asbrink's WSE Hardware and Linrad, in parallel with an instance of WSJT fed by my FT1000MP with either H or V pol input, and 4 slave instances of Linrad, each receiving both H and V pol input from the same instance of Linrad that fed MAP65, and with each of the 4 slave instances of Linrad locked respectively to H, V, +45, or -45 polarization angle.  Details of how I did this are described at LinradMAP65.htm

MAP65 ran throughout the contest, and was my primary receiver.  To keep the huge amount of data being sent from the main instance of Linrad to MAP65 and the 4 slave instances of Linrad [all of which run on the same computer], the Linrad/MAP65 computer was isolated from the local area network.  A simple homebrew program sent, via USB, information to the logging computer about the MAP65 QSO frequency, and this was used to control the frequency of the FT1000MP for both transmit and receive.  As is noted on the webpage referenced above, the master and slave instances of Linrad automatically follow the MAP65 QSO frequency, as a result of some additions I made to the Linrad software base, that are also described on the web page referenced above.

The data I analyzed was obtained from the following sources:

1.  The file map65_rx.log contains all MAP65 decodes

2.  The file ALL.TXT contains all WSJT9 decodes.  There were 5 instances of this file used for this analysis, one for the instance of WSJT9 receiving the FT1000MP data and 4 for the 4 Linrad slave instances described above. 

Each line of each file contained information describing a single decoded signal, or "decode".  The data structures for the MAP65 and WSJT programs are not identical, and I adjusted the data so that the data analyzed from the two data sources would be compatible.  The data structure looks like this:

The first two data records shown in this illustration are WSJT data, and the third record is from MAP65.  You can see that the MAP65 decode has frequency data which the WSJT data does not, but that otherwise the records are similar.

The data was processed and analyzed by entering it into an SQLite database, and then using SQLite Expert Professional 3.4.47.2269 to analyze the data and then display and record the results.

The data was first processed by removing all lines containing decodes where W3SZ was the transmitting station.  This was necessary because the 5 WSJT instances decoded the W3SZ transmit signal during each transmit cycle, and these decodes needed to be removed from the dataset prior to analysis.  Removing these records from the dataset left 10046 records remaining for analysis.

Further data processing prior to analysis was necessary because I have several powerful, relatively close stations that were also operating during the ARRL contest.  When they transmitted, it would produce large numbers of false decodes.  I removed the obvious false decodes before beginning my analysis.  There may be remaining false decodes of which I am unaware.  Removing the false decodes reduced the total number of records [i.e. decodes] from 10046 to 8805. 

Of these 8805 decodes, 5895 were standard messages, and the rest were shorthand messages.  These 5895 standard messages were sent by 262 unique callsigns ("unique calls").  A listing of how these were divided among MAP65 and the 5 instances of WSJT is given below. 

NOTE THAT THE OPERATION OF THE STATION WAS SUCH THAT MAP65 WAS USED TO DISCOVER THE FREQUENCIES WHERE STATIONS WERE OPERATING, AND THE WSJT9 RECEIVERS WERE THEN PLACED ON THE DESIRED QSO FREQUENCY TO RECEIVE SIGNALS FROM THE STATION INITIALLY DISCOVERED BY MAP65.  MANY FEWER DECODES BY THE WSJT RECEIVERS WOULD HAVE RESULTED HAD MAP65 NOT BEEN USED TO FIND STATIONS AND THEN AUTOMATICALLY, OR IN SOME CASES SEMI-AUTOMATICALLY, PUT THE WSJT RECEIVERS ONTO ITS QSO FREQUENCY.

Note that the designation "WSJT" in my tables and discussion actually includes 5 instances of WSJT, one for the FT1000MP and one for each of the 4 slave instances of Linrad. 

Note that the designation "Linrad" as I use it in the tables and when discussing the data includes 4 instances of WSJT, one for each of the 4 slave instances of Linrad.  It is a subset of the "WSJT" designation noted immediately above.

The primary question that I wanted to address was, "How do the MAP65 decoder and the WSJT decoder compare when they are operating on the same signals, on the MAP65 QSO frequency?"  To answer this question I looked at the performance of the programs' decoders when operating on the signals of stations that W3SZ worked during the second weekend of the contest.  I chose this dataset to explore this question because in each W3SZ QSO all of the receivers were on the same [QSO] frequency, and the decoders could be fairly compared.  The table below shows how MAP65 and WJST performed during the contacts that W3SZ made during the second weekend:

You can see that about 10% of the time only MAP65, and not the WSJT receivers, decoded the standard message containing both stations' callsigns and the grid square.  Also, about 10% of the time only WSJT, and not the MAP65, decoded the standard message with both stations' callsigns and the grid square.  So based on this criterion, the performance of both decoders is similar.  Due to statistical variation, in a small percentage of cases, one or the other of the decoders decodes a signal that the other one does not, but the numbers do not suggest that one progrem's decoder is superior to the other's.  The four slave instances of Linrad did not add any decodes to those accomplished by MAP65 and the FT1000MP instance of WSJT.

From the data contained in the table above, as noted, we see that both MAP65 and WSJT do very well.  As previously noted, most of the QSOs completed by W3SZ, i.e. the QSOs that resulted in the data used to create the table above, had the initial sequences decoded by MAP65 (and not by WSJT).  This table shows that only 3 QSOs of the 44 QSOs completed by W3SZ the second weekend did not have the initial decode by MAP65.  In contrast, for 28% of the WSJT-received QSOs, WSJT never decoded the initial sequence, although WSJT did decode the exchange containing both stations' callsigns and grid square.  That is due to the way in which MAP65 and the WSJT receivers were used at W3SZ during the contest.  MAP65 found the stations of interest, and then all receivers were brought to the MAP65 QSO frequency.

There are some results found on this table that appear to be counterintuitive when this table is compared to the tables below.  This is due to differences in the details of the record definitions used for the data analysis that was performed to create this table as compared with the definitions used for subsequent tables. 

Note that if the first message of a QSO series was a CQ decoded by only MAP65, or was initiated by W3SZ after W3SZ saw a CQ appearing only on MAP65, but then the next message from the "other" station was only decoded by one of the WSJT instances, that decode would be recorded in the above analysis as a "Callsign calling W3SZ decoded ONLY by a WSJT receiver", because the message used to "count" these contacts, one containing "W3SZ" as the first component and the other station's call as the second component, and a grid square, would have been copied only by a WSJT station.  But in fact, the contact would never have taken place if MAP65 had not decoded the initial message, which the WSJT instances did not decode.  So this analysis tends to "favor" the WSJT instances in terms of suggesting that WSJT alone was responsible for the completion of some QSOs, when in fact MAP65 was also essential for the completion of those QSOs. 

Nevertheless, the above data answers the question regarding relative performance of the MAP65 and WSJT decoders, and suggests that their performance is similar.  Other interesting data was available from the MAP65 and WSJT9 log files, and it is presented below.

Another way of looking at the data is to look at all stations that called W3SZ during the second weekend.  There were a total of 201 decodes of stations calling W3SZ.  These 201 decodes represented 96 unique callsigns calling W3SZ.  The breakdown of these decoded messages is shown below:

Receiver	Decodes of Stations Calling W3SZ	Unique Callsigns Calling W3SZ	Callsigns Decoded ONLY by This Receiver	Callsigns Decoded ONLY by This Receiver as % of Total Unique Callsigns by All Receivers
MAP65	76	65	29	30.2%
WSJT	125	67	31	32.3%
FT-1000MP	49	43	7	7.3%
Linrad	76	55	22	22.9%
H	30	28	15	15.6%
V	20	20	5	5.2%
+45	15	12	1	1.0%
-45	11	10	1	1.0%

Here we see that there were 76 decodes by MAP65 of stations calling W3SZ, and 125 decodes by WSJT.  The higher number of decodes by the WSJT receivers is likely a result of the fact that there were 5 WSJT receivers running simultaneously, all tuned to MAP65's QSO frequency. The number of decodes for any ONE of the WSJT receivers is less than the number of MAP65 decodes.  We also see that 30% of the unique callsigns calling W3SZ were decoded ONLY by MAP65 and 32% were decoded ONLY by the WSJT receivers.  Again, these numbers are comparable, suggesting that the efficacy of the decoders in MAP65 and WSJT9 is similar.

A counterintutive result contained in the table above is that there are 31 callsigns decoded only by WSJT, which includes decodes from the FT1000MP and the slave Linrad receivers.  However, there are 7 such decodes by the FT1000MP, and 22 by the Linrad slave receivers.  How did the number 31 appear, given that 7 + 22 = 29?  Well, W0XG was decoded by both the FT1000 and by the -45 Linrad slave receiver.  And YU7XG was decoded by both the FT1000 and the H pol Linrad slave receiver.  So these two callsigns added to the count of WSJT-only decodes, but not to any of its subsets, because the callsigns were duplicated in two different receivers contained WITHIN the WSJT dataset.   There are also other examples of "counterintuitive" results present in the results, which also have simple explanations; this one is included here just to allay any fears it might have caused concerning the robustness of the data.

The table below compares, for the entire set of  8805 decodes remaining after initial processing of the data, the performance of MAP65 with the 5 instances of WSJT regarding the total number of decodes performed by each receiver, the number of standard messages decoded by each receiver, the number of unique callsigns decoded by each receiver, and the number of unique callsigns recorded ONLY by a particular receiver and not by any other receivers:

Note that the majority of decodes (87.1%) were by MAP65, and that 133 of the callsigns decoded (50.8%) were decoded ONLY by MAP65 and not by any of the other receivers.  This large percentage of both total decodes and of decodes unique to MAP65 is of course primarily due to the fact that MAP65 was decoding all signals contained in an 80 kHz span throughout the contest, whereas the WSJT receivers were each limited to about a 2 kHz span.

In contrast to the 50.8% UNIQUE decode rate with MAP65, only 3 callsigns (1.1%) were ONLY decoded by the FT1000MP/WSJT9 combination, and just 42 callsigns, or 16% of the total, were decoded ONLY by the Linrad slaves.  These differences point up the great utility of MAP65 in FINDING stations of interest.  However, because of the difference between MAP65 and the WSJT receivers in analyzed bandwidth, and because of the fact that the WSJT receivers only decoded on the MAP65 QSO frequency, these number do NOT address the relative efficacy of the MAP65 and WSJT decoders.  That analysis was done as described in the first table, where a fair comparison of the decoders was possible, on the MAP65 QSO frequency looking at the standard message from stations that worked W3SZ containing both W3SZ's and that station's callsigns and that station's gridsquare.  The second table, looking at the data from all stations calling W3SZ, provided confirmatory data.

 Note that some COUNTERINTUITIVE results pop up here, as for example only three callsigns picked up ONLY by the FT-1000MP in this dataset, vs the 7 callsigns ONLY picked up by the FT-1000MP in the second dataset/table. This difference occurs because the other four callsigns were picked up by other receivers when they were calling CQ or calling other stations, and those decodes were therefore appropriately counted during the data analysis included in the table immediately above (thus reducing the count of "ONLY decoded by FT-1000MP" records from what would have been 7 to 3), but they of course would NOT be counted in the data analysis contained in the second table of decodes of stations calling W3SZ, and thus the count of "ONLY decodes" for theFT-1000MP in that "Calling W3SZ" table is increased to 7.

The results overall, as as summarized in these 3 tables, do not suggest a deficiency of MAP65's decoder, which in fact appears to be similar to WSJT9's decoder, but rather emphasize the great utility of MAP65 in finding stations to work, and then completing the QSOs. 

There IS an incremental value to adding additional WSJT decoders to MAP65, but I believe that the results are consistent with no more than a statistical variation among decodes achieved by the different decoders.  They do not imply a superiority of the WSJT decoder over MAP65 or vice versa.

One can speculate on the potential benefits brought by using MAP65 and multiple instances of WSJT.  The number of calls decoded during the second weekend but not worked during the contest is is 171. So the number of stations worked would have increased from 125 to 125 + 171 = 296 if W3SZ had worked all stations decoded during the second weekend.

Another way of looking at this is that the number of stations decoded that were calling W3SZ but that were not worked during contest is 49.  So stations worked would have increased from 125 to 125 + 49 = 174 if W3SZ had worked all stations decoded calling him during the second weekend.  This is a more conservative estimate.

If you wish, you can work out the marginal utility of various combinations of MAP65 and WSJT receivers by using the data contained in the tables above.

The data obtained can also be used not for the purposes of evaluating MAP65 or WSJT, but to explore contest conditions as heard at W3SZ for each hour of the contest in terms of total station activity, number of new stations heard, etc., as shown on the following tables.  On these tables UTC is defined as hours since UTC 0000 Saturday, the start of the contest for the second weekend.  It is thus the time in UTC for the first night, and the UTC time plus 24 for the second night:

FIRST PASS:

UTC	# Decodes During Hour	# Unique Calls Decoded During Hour	# Unique Calls Decoded During Hour and Not in Log	# Unique Calls Not Previously Decoded and Not in Log	Total # Calls Worked by End of Hour	# Calls Worked During Hour
00	17	5	3	3	81	0	0040 moon is at 10 degrees
01	17	7	4	3	81	0	0130 moon is at 20 degrees
02	27	10	3	2	81	0	0130 Texas window opens
03	72	27	10	9	83	2	0300 California window opens
04	374	54	26	19	87	4
05	372	53	25	11	92	5
06	328	36	15	8	94	2
07	298	57	33	22	97	3
08	259	38	18	11	101	4
09	232	31	18	11	104	3	0930 Eu window (GM) closes
10	126	18	11	7	106	2	0950 JA window opens
11	153	21	12	8	107	1
12	68	14	5	4	107	0	1220 moon is at 20 degrees
13	100	9	1	1	107	0	1320 moon is at 10 degrees
14	63	7	2	1	107	0	1410 moonset
15	2	1	1	0	107	0
16	0	0	0	0	107	0

SECOND PASS:

UTC	# Decodes During Hour	# Unique Calls Decoded During Hour	# Unique Calls Decoded During Hour and Not in Log	# Unique Calls Not Previously Decoded and Not in Log	Total # Calls Worked by End of Hour	# Calls Worked During Hour
24	6	3	1	1	107	0	2440 is moonrise
25	181	27	11	8	107	0	2530 moon is at 10 degrees
26	311	51	28	13	110	3	2630 moon is at 20 degrees
27	351	59	28	14	112	2	2630 Texas window opens
28	383	54	19	7	112	0	2750 California window opens
29	340	50	17	7	113	1
30	398	63	27	13	117	4
31	502	60	26	11	120	3
32	380	51	23	9	122	2
33	129	23	9	2	122	0
34	58	12	7	4	122	0	3440 Eu window (GM) closes
35	96	8	5	1	124	2	3550 JA window opens
36	107	16	7	3	124	0
37	105	15	5	2	125	0	3700 moon is at 20 degrees
38	30	9	3	1	125	1	3800 moon is at 10 degrees
39	0	0	0	0	125	0	3900 moonset

You can see that the contest is busiest in the mid portion of each pass, and that while there are hundreds of decodes per hour, the number of unique calls not previously decoded and not already in the log is quite small for any hour during the second weekend, but that by continuing to operate, the number of stations in the log is gradually increased.

I did not have the data to do a detailed analysis of the first weekend, but I did construct similar tables using the MAP65-only data which I had from that weekend.  There were a total of 8299 decodes of which 5952 were standard decodes, representing 181 unique calls.  86 stations calling W3SZ were decoded (of which 81 were worked the first weekend).  It is interesting that there were more total unique calls for the second weekend (262) than during the first weekend (181), and there were also generally more total and new calls heard per hour during the second weekend.  Perhaps this is a result of the competition provided by other contests in Europe during the first weekend.

FIRST PASS

UTC	# Decodes During Hour	# Unique Calls Decoded During Hour	# Unique Calls Decoded During Hour and Not in Log	# Unique Calls Not Previously Decoded and Not in Log	# Calls Worked During Hour	Total # Calls Worked by End of Hour
00	19	0	0	0	0	0	moonrise is at 0050
01	157	20	20	20	2	2	0150 moon is at 10 degrees
02	319	24	23	12	6	8	0240 moon is at 20 degrees
03	304	38	33	14	6	14	0250 Texas window opens
04	372	43	32	12	3	17	0420 California window opens
05	517	44	30	8	5	22
06	434	40	26	6	4	26
07	478	39	31	13	4	30
08	413	39	30	9	6	36
09	360	35	22	6	4	40
10	117	14	10	1	3	43	1100 JA window opens
11	100	10	6	2	2	45	1130 Eu window (GM) closes
12	97	11	5	1	2	47
13	128	16	8	5	2	49	1340 moon is at 20 degrees
14	109	15	6	4	1	50	1440 moon is at 10 degrees

SECOND PASS

UTC	# Decodes During Hour	# Unique Calls Decoded During Hour	# Unique Calls Decoded During Hour and Not in Log	# Unique Calls Not Previously Decoded and Not in Log	# Calls Worked During Hour	Total # Calls Worked by End of Hour	Moonrise is at 2550
25	14	3	1	1	0	50
26	19	4	10	7	0	50
27	153	19	11	4	5	55	2640 moon is at 10 degrees
28	233	22	13	1	3	58	2740 moon is at 20 degrees
29	327	25	15	3	3	61	2740 Texas window opens
30	469	43	29	9	2	63	2900 California window opens
31	627	61	31	9	3	66
32	653	67	27	6	3	69
33	588	67	27	8	3	72
34	372	54	14	5	4	76
35	252	32	4	1	2	78	3600 JA window opens
36	132	14	5	3	0	78	3600 Eu window (GM) closes
37	140	12	8	3	1	79
38	110	14	7	3	1	80
39	155	15	5	1	1	81

Back to Top

Copyright 1997-2012 COPYRIGHT Roger Rehr W3SZ. All Rights Reserved

Brought to you by the folks at W3SZ