The DMR Last heard system is currently unavailable, so I will take my pages offline until It comes online again
[Original Article starts here]
One cool feature of Amateur radio DMR, is that the other station’s details can often be shown on the transceiver’s display.
The trick is to have the other station’s details installed as a Contact in the transceiver’s Codeplug database.
With firmware version 3.0.6 (and newer), a new feature was added to separately upload , up to 10920 “DMR ID”s. This feature is not integrated into the main CPS, and uses a separate program called ActiveClient and requires the transceiver to be turned on and connected to USB while several buttons are held down.
In some respects this feature superseded the information in this post, However….
There is one drawback with this feature. Radioddity have only allocated 8 characters for the callsign, rather than the 16 characters available for the “Digital Contact” in the CPS, and there is only enough characters for the callsign and not enough to include the name e.g. a Digital contact can hold “VK3KYY Roger….”, where as the DMR ID system can only hold “VK3KYY R”.
So IMHO, its not as cool, as stations have to give their callsign.
Anyway. To continue my original post, assuming you want to make the most of the 1024 Digital contacts which can store callsign and name..
This sounds great in theory, however the problem is that the entire DMR-MARC database, has over 83,000 entries, at the time of writing this post, and most transceivers can only store a much smaller number of Contacts.
My Radioddity GD-77 (aka Tytera MD-760) can only store 1024 Contacts using the current firmware (v3.0.6), and 15 of those “Contacts” are taken up with TalkGroups, so I only have space for just over 1000 Contacts
Filtering the DMR-MARC database to only include Australian “VK” callsigns doesn’t help because there are around 1300 listed (at the time of writing), and I expect the number will rise as more people buy these cheap handheld transceivers.
The question is which 1000 of the 1300 to include and one possible solution, which I’m currently using is to use the DMR-MARC “Last Heard” page http://ham-digital.org/dmr-lh.php with a filter for all callsigns starting with VK
By default page only shows the last 30 entries, however the n=xx parameter allows the maximum number of entries to be changed to much bigger numbers e.g 1000 or more.
There is a big problem caused by the paginated form in which the data is available. On several occasions the data has changed between page 1 and page 2 etc, so that a callsign (ID) has been lost. The method described below only works correctly when there is no traffic for the callsigns in question e.g. VK* while all necessary pages are being retrieved.
Even though the DMR-MARC database has over 1300 ID’s listed for Australian stations, the “Last heard” page only seems to be currently listing around 708 entries. Its possible that some people have registered ID’s and never used them, or used them prior to 2014, which seems to be the oldest date in the Last Heard system, but I currently think its the best option to get a list of the most active ID’s
Currently this page is only available as HTML, and also it is paginated with the maximum number of entries per page of 100, so initially, I manually cut and paste the contents of each page, into Excel, and removed unwanted columns, however as new callsigns are being added to the DMR-MARC database all the time, I would need to do this about once a week, and I have better ways to use my time 😉
As a solution to this problem, I decided to write a PHP script to “scrape” the “last heard” page and output the result as a CSV file.
I thought this would be relatively straight forward, as PHP has a DOM object which can parse HTML pages into nicely accessible objects, but I was wrong… The “Last heard” page will not parse because the HTML authoring is invalid in multiple places, and the only way to extract the data was to treat the whole page as a string, and search for specific markers.
This is highly likely to break if the page HTML is modified, but I think even using the DOM object would only be marginally better.
I’ve put the code in the scripts folder, and an example usage would be
http://rogerclark.net/scripts/dmr-contacts.php?callsign=VK*&limit=1000 (All VK callsigns limited to 1000 entries)
http://rogerclark.net/scripts/dmr-contacts.php?callsign=F*&limit=1000 (all F callsigns, limited to 1000 entries)
The format is an index number in the first column, Callsign and name in the second column and ID in the last column. This format is not to suit any specific transceiver, so you need to append any additional columns that your transceiver software needs, and you’ll probably need to append your Talk Groups and re-number the file, to suit.
But its a lot easier than cutting and pasting from the HTML.
As I have a GD-77, I’ve updated the script to include a “radio” parameter e.g.
Currently the only “radio” option which does anything is “gd-77” (or “GD-77”), and this format is only compatible with the CPS 2.0.5
(Note. For older CPS versions, the last column has to be renamed from “Call Receive Tone” to “Rx Tone”)
This is my current code, which works, but is not exactly elegant 😉 and is available for on Github a https://gist.github.com/rogerclarkmelbourne/1612a2455bf68d45a1fb6f03093e1210
Update 18th Jan 2018
I have been experimenting with using the DMR IR system in conjunction with the Digital Contacts, and unfortunately the firmware in the GD-77 does not do the logical thing.
If I load the most active 700 VK callsigns (including names) into the Digital Contacts, and then download the full list of VK DMR ID’s , I’ve found that the name is not being displayed.
Presumably this is because the firmware in the GD-77 is searching the DMR ID dataset first 🙁
Ideally the GD-77 firmware would search the Digital Contacts first and would therefore display the callsign and name.
Currently my workaround is to download the entire database from DMR MARC, filter for VK* then remove ID’s from this list which are in my main Digitial Contacts.
But this is is a laborious process, and needs to be automated.
Update 19th Jan 2018
I’ve found another problem with the DMR-LastHeard page
Looking at the page, without applying any callsign filters, I noticed one line which didn’t contain the callsign or name (see line 58 below)
|57||2018-01-19 00:52:22||3106320||AF6BY||Jeff||310668||KI6KQU||San Diego||31066||TG31066||BM||2||GVF|
|58||2018-01-19 00:51:59||1107738||310613||W6TCP||Mt. Allison||91||TG91||BM||1||GVF|
|59||2018-01-19 00:51:56||2142260||EA2SB||LUKEN||214215||ED2ZAF||Arnotegi – Bilbao||214||TG214||BM||1||GVF|
I initially thought that perhaps someone just entered a random ID into a DMR radio and connected via a repeater to the network
However, when I checked the main DMR-MARC database search, it found
|1107738||KC7VHI||Dennis P Low||Sunnyvale||California||United States||DMR|
This is a valid ID, so I think the LastHeard page must not directly lookup the name etc from the main DMR database, but must take snapshots, and if the page is run where a new user is using the network, its possible that their details will not display
As I intend making a new script, I need to take this into consideration, and not have this bug in my own code, and the script should not use the callsign and name on the LastHeard page, but should lookup the ID in a copy of the DMR-MARC database
I could download a new database each time an unknown ID is encountered, but I think this would probably create an unnecessary load on the DMR-MARC database server, as my script would probably end up needing a new database every time it ran e.g. every 5 minutes.
So its probably best to only download the latest DMR-MARC database on a longer schedule e.g. perhaps once every 6 hours, and simply not include any users newer than 6 hours in the Contacts list it creates