Building a Amateur radio DMR contacts list – Attempt #1

Building a Amateur radio DMR contacts list – Attempt #1

posted in: Ham radio | 6

UPDATE.

The DMR Last heard system is currently unavailable, so I will take my pages offline until It comes online again

 

 

 

[Original Article starts here]

One cool feature of Amateur radio DMR, is that the other station’s details can often be shown on the transceiver’s display.

The trick is to have the other station’s details installed as a Contact in the transceiver’s Codeplug database.

 

 

 

Update.

With firmware version 3.0.6 (and newer), a new feature was added to separately upload , up to 10920 “DMR ID”s. This feature is not integrated into the main CPS, and uses a separate program called ActiveClient and requires the transceiver to be turned on and connected to USB while several buttons are held down.

In some respects this feature superseded the information in this post, However….
There is one drawback with this feature. Radioddity have only allocated 8 characters for the callsign, rather than the 16 characters available for the “Digital Contact” in the CPS, and there is only enough characters for the callsign and not enough to include the name e.g. a Digital contact can hold “VK3KYY Roger….”, where as the DMR ID system can only hold “VK3KYY R”.
So IMHO, its not as cool, as stations have to give their callsign.

 

Anyway. To continue my original post, assuming you want to make the most of the 1024 Digital contacts which can store callsign and name..

 

This sounds great in theory, however the problem is that the entire DMR-MARC database, has over 83,000 entries, at the time of writing this post, and most transceivers can only store a much smaller number of Contacts.

My Radioddity GD-77 (aka Tytera MD-760) can only store 1024 Contacts using the current firmware (v3.0.6), and 15 of those “Contacts” are taken up with TalkGroups, so I only have space for just over 1000 Contacts

 

Filtering the DMR-MARC database to only include Australian “VK” callsigns doesn’t help because there are around 1300 listed (at the time of writing), and I expect the number will rise as more people buy these cheap handheld transceivers.

The question is which 1000 of the 1300 to include and one possible solution, which I’m currently using is to use the DMR-MARC “Last Heard” page http://ham-digital.org/dmr-lh.php with a filter for all callsigns starting with VK

 

e.g.

 

http://ham-digital.org/dmr-lh.php?callsign=VK*

 

By default page only shows the last 30 entries, however the n=xx parameter allows the maximum number of entries to be changed to much bigger numbers e.g 1000 or more.

Several Caveats.

There is a big problem caused by the paginated form in which the data is available. On several occasions the data has changed between page 1 and page 2 etc, so that a callsign (ID) has been lost. The method described below only works correctly when there is no traffic for the callsigns in question e.g. VK* while all necessary pages are being retrieved.

 

Even though the DMR-MARC database has over 1300 ID’s listed for Australian stations, the “Last heard” page only seems to be currently listing around 708 entries. Its possible that some people have registered ID’s and never used them, or used them prior to 2014, which seems to be the oldest date in the Last Heard system, but I currently think its the best option to get a list of the most active ID’s

 

 

Currently this page is only available as HTML, and also it is paginated with the maximum number of entries per page of 100, so initially,ย  I manually cut and paste the contents of each page, into Excel, and removed unwanted columns, however as new callsigns are being added to the DMR-MARC database all the time, I would need to do this about once a week, and I have better ways to use my time ๐Ÿ˜‰

 

As a solution to this problem, I decided to write a PHP script to “scrape” the “last heard” page and output the result as a CSV file.

I thought this would be relatively straight forward, as PHP has a DOM object which can parse HTML pages into nicely accessible objects, but I was wrong… The “Last heard” page will not parse because the HTML authoring is invalid in multiple places, and the only way to extract the data was to treat the whole page as a string, and search for specific markers.

This is highly likely to break if the page HTML is modified, but I think even using the DOM object would only be marginally better.

 

I’ve put the code in the scripts folder, and an example usage would be

 

http://rogerclark.net/scripts/dmr-contacts.php?callsign=VK*&limit=1000ย  (All VK callsigns limited to 1000 entries)

http://rogerclark.net/scripts/dmr-contacts.php?callsign=F*&limit=1000 (all F callsigns, limited to 1000 entries)

 

The format is an index number in the first column, Callsign and name in the second column and ID in the last column. This format is not to suit any specific transceiver, so you need to append any additional columns that your transceiver software needs, and you’ll probably need to append your Talk Groups and re-number the file, to suit.

But its a lot easier than cutting and pasting from the HTML.

 

Update.

As I have a GD-77, I’ve updated the script to include a “radio” parameter e.g.

 

http://rogerclark.net/scripts/dmr-contacts.php?callsign=VK*&limit=1000&radio=gd-77

 

Currently the only “radio” option which does anything is “gd-77” (or “GD-77”), and this format is only compatible with the CPS 2.0.5

(Note. For older CPS versions, the last column has to be renamed from “Call Receive Tone” to “Rx Tone”)

 

This is my current code, which works, but is not exactly elegant ๐Ÿ˜‰ and is available for on Github a https://gist.github.com/rogerclarkmelbourne/1612a2455bf68d45a1fb6f03093e1210

 

 

Update 18th Jan 2018

 

I have been experimenting with using the DMR IR system in conjunction with the Digital Contacts, and unfortunately the firmware in the GD-77 does not do the logical thing.

 

If I load the most active 700 VK callsigns (including names) into the Digital Contacts, and then download the full list of VK DMR ID’s , I’ve found that the name is not being displayed.

Presumably this is because the firmware in the GD-77 is searching the DMR ID dataset first ๐Ÿ™

Ideally the GD-77 firmware would search the Digital Contacts first and would therefore display the callsign and name.

Currently my workaround is to download the entire database from DMR MARC, filter for VK* then remove ID’s from this list which are in my main Digitial Contacts.

But this is is a laborious process, and needs to be automated.

 

Update 19th Jan 2018

 

I’ve found another problem with the DMR-LastHeard page

Looking at the page, without applying any callsign filters, I noticed one line which didn’t contain the callsign or name (see line 58 below)

 

57 2018-01-19 00:52:22 H 3106320 AF6BY Jeff 310668 KI6KQU San Diego 31066 TG31066 BM 2 GVF
58 2018-01-19 00:51:59 H 1107738 310613 W6TCP Mt. Allison 91 TG91 BM 1 GVF
59 2018-01-19 00:51:56 H 2142260 EA2SB LUKEN 214215 ED2ZAF Arnotegi – Bilbao 214 TG214 BM 1 GVF

I initially thought that perhaps someone just entered a random ID into a DMR radio and connected via a repeater to the network

However, when I checked the main DMR-MARC database search, it found

 

DMR ID Callsign Name City State/Prov Country Remarks
1107738 KC7VHI Dennis P Low Sunnyvale California United States DMR

 

This is a valid ID, so I think the LastHeard page must not directly lookup the name etc from the main DMR database, but must take snapshots, and if the page is run where a new user is using the network, its possible that their details will not display

 

As I intend making a new script, I need to take this into consideration, and not have this bug in my own code, and the script should not use the callsign and name on the LastHeard page, but should lookup the ID in a copy of the DMR-MARC database

I could download a new database each time an unknown ID is encountered, but I think this would probably create an unnecessary load on the DMR-MARC database server, as my script would probably end up needing a new database every time it ran e.g. every 5 minutes.

So its probably best to only download the latest DMR-MARC database on a longer schedule e.g. perhaps once every 6 hours, and simply not include any users newer than 6 hours in the Contacts list it creates

6 Responses

  1. Vic
    |

    What would the script be to search criteia fro VA3* and VE3* callsigns …………….I can use you example do them individually !!
    Great idea to use only ACTIVE USERS ………….

  2. Roger Clark
    |

    At the moment, the only way to do it would be to use the script twice, once for VA3* and once for VE3*, because the “DMR-LastHeard” page only accepts one callsign filter and it doesn’t seem to allow filters like “VA3* OR VE3*” or “VA3* || VE3*” or regular expressions

    I did try contacting the web admin for the DMR-LastHeard, who is DL5DI, to ask about additional features, via the email address on that page, but I’ve not had a reply.
    I may try contacting him directly, as this seems to be his own website http://dl5di.prgm.org/

  3. Vic
    |

    Thanks for your rply and interest……… Interesting reading regarding your approach ……..tnx have a great week-end !

  4. Roger Clark
    |

    No worries

    I have started to investigate whether I could import the entire DMR-MARC database (from the CSV data dump) and run a script every 5 mins to keep track on the most active stations across the whole database.
    Then just display data that has been previously scraped.

    However, it looks like this approach is impractical, as its taking PHP on my local machine a huge amount of time to import the CSV (10 minutes), and it would be even slower on my shared web server.

    So I’m going to revisit my current script, and work out why occasionally it seems to have some stations mission

  5. Anonymous
    |

    If you remove the duplicates there is enough room

  6. Roger Clark
    |

    There isnt enough room in the Digital Contacts for all the VK callsigns registered with DMR-MARC

    There are over 1000 and the max number of Digital Contacts is 1000

Leave a Reply