Hello from new member!

[VK3KYY]

I'm unsure if we are looking at the same data, so I downloaded the user.csv file from the same URL taht the OpenGD77 CPS uses and inspected it. The whole file is utf-8 encoded, I checked it with the iconv utility. When you say "it's a mess", what exactly do you mean? That rows can have different character encodings? Or something else? Could you give an example of a "messed up" entry in that file?

73 de SA0ASM

Daniel was referring to the internal string / char mapping used by the firmware.

The display font is 8 bit (bitmap), i.e only 255 chars, so the language strings used in the firmware are not a standard 'code page'
Its effecively a custom code page, but because there are only 255 characters, its not possible to support even all accented characters, hence why its not possible to even produce versions for eastern European character sets.
And, the Japanese version has to be a completely different firmware build, with completely differnt bitmap fonts.

This has not changed in several years. The published source code for the GD77 and UV380 has all the bitmap fonts including coments about which characters are supported.

If you look you'll see there is almost no space for additional characters in the font.


For the DMR ID database; it is compressed to a form of 6 bit ascii, becasue originally the DMR ID database did not contain any non (A-Z,a-z,0-9,\.) characters, and the compression greatly increased the number of ID's that would fit into the small memory of a lot of radios.

Its only in the last year that the UV380 (and clones) have been supported, which have more flash memory to store the DMR ID database.

However, simply switching back to uncompressed data for these has other problems, because the data transfer is quite slow and not 100% reliable.

So changing to uncompressed on the UV380 / DM1701 etc, will increase the upload time noticably and also make the uploads less reliable.

Also, I can't remember if the code to handle uncompressed DMR ID's is still in the firmware codebase, because there is a problem of ROM size on all radios, especially the older radios like the GD77/RD5R/DM1801 etc , with almost no ROM space remaining.

So there is no room for unused code, and we generally remove it

The UV380 etc have much more ROM however, its not that simple, because a large portion of the ROM map is taken up by the binary blob of the official firmware, which has to be merged duing DFU, by the CPS, because of the legal / IP problems with the AMBE codec.

So even though the UV380 etc have 2 x the ROM size of the GD77, it does not mean that there is 2 x the available ROM space for the OpenGD77 firmware, becuase the binary blob of the official UV380 firmware , that we need to merge is probably larger

Possibly in one of the old versions of the CPS we may have had a checkbox to enable / disable compression, but we definitely don't have that now.


Anyway. I suggest that you start by taking a look at the firmware source code.

[SA0ASM]

Thank you for the comprehensive reply!

I still think we can improve by introducing some pre-processing on the id database contents before compression, but I'll run some tests first before suggesting any specific changes.

73 de SA0ASM

[VK3KYY]

Thank you for the comprehensive reply!

I still think we can improve by introducing some pre-processing on the id database contents before compression, but I'll run some tests first before suggesting any specific changes.

73 de SA0ASM

Converting the unicode accent characters to non-accented versions, is initially the best approach, assuming there is a reliable method to do this.

AFIK. We don't have a mapping table from unicode to the custom character table, and I don't know how difficult it would be to do this.
Even if that was implemented in the CPS, the functionality would need to be optional, because there are a large number of users who don't care about accented characters, but do care about the number of IDs that will fit in their radios, or the time that it takes to upload the database into the radio.

Also. Many users do not have an understanding of the complexities of unicode etc, and simply having a checkbox in the CPS , even if it says , "Support Accented Characters" , would potentially cause problems, because this would need to disable compression, and also not all accented characters can be supported becuase the font does not support all possible accented characters or even Japanese characters.

This also opens up a can of worms with regards to Japanese names in the DMR ID database, becuase from memory I think there are some Hams who enter Japanese characters into RadioID.net when they register their name and other data.

But the Japanese firmware build is different from the non-Japanese build and we don't have a for the CPS to detect whether the firmware is Japanese.

Also, we don't have a method to detect what firmware version is in the radio, and ultimately the firmware version is the git revision, and it would be a require a lot of work to like git revisions to specific CPS related functionality

etc
etc
etc

[SA0ASM]

Actually, the stack overflow article that the code for DMRDataItem.RemoveDiacritics() was taken from (https://stackoverflow.com/questions/249 ... ing-in-net) does supply several alternative ways of "ascii-fying" roman text. I'd suggest using the dictionary lookup method (suggested by user "CIRCLE"), as it would cover more than "stripping diacritics", such as providing support for greek characters.

Anyway, I'll leave this one for another day. Thanks for the introduction both of you!

73 de SA0ASM

[VK3KYY]

If you test each method with real data, please let us know which method gives the best results, please test with multiple languages not just the Nordic languages

[SA0ASM]

Re: Writing the CPS again from scratch

This would take a huge amount of time, which I personally don't have.

There is nothing to stop you writing your own CPS from scratch if you have time.

Hi!

I took some time during the holidays to start work on this project. The initial version is far from feature complete, though I've implemented enough to edit channels, zones, contacts and groups as well as some initial code for reading and writing to a radio. I'd very much appreciate any feedback on this, especially around the radio-specific serial comms details that I may have misinterpreted. There's a long way to go before it's actually on parity with the existing code base for most users, CSV import and flashing firmware being the obvious ones (though I was hoping you might want to contribute some of your existing code).

The project is on github at https://github.com/SA0ASM/OpenGD77CPS-new if you want to check it out before I post a request for beta testers in the forum...

73 de SA0ASM

[VK3KYY]

Re: Writing the CPS again from scratch

This would take a huge amount of time, which I personally don't have.

There is nothing to stop you writing your own CPS from scratch if you have time.

Hi!

I took some time during the holidays to start work on this project. The initial version is far from feature complete, though I've implemented enough to edit channels, zones, contacts and groups as well as some initial code for reading and writing to a radio. I'd very much appreciate any feedback on this, especially around the radio-specific serial comms details that I may have misinterpreted. There's a long way to go before it's actually on parity with the existing code base for most users, CSV import and flashing firmware being the obvious ones (though I was hoping you might want to contribute some of your existing code).

The project is on github at https://github.com/SA0ASM/OpenGD77CPS-new if you want to check it out before I post a request for beta testers in the forum...

73 de SA0ASM

Is this a Windows application?

I presumed you'd be writing something which is cross platform so that Linux users could run it as a native application, and potentially mobile devices.

I do not personally have time to work on any new projects, so I'm unable to help with this.

The CSV code used in the CPS was supplied by Colin G4EML and is written in Visual Basic.
If you want his code, you would need to contact him, because I'm not at liberty to give away other people's code.

There are Python versions of the flashing code, which you could use.
The Windows firmware flashing code probably has licensing issues preventing it being released as OpenSource as its derived from multiple examples, some with non GPL licenses. e.g. I don't think ST's code license allows their source code to be rereleased as OpenSource / GPL

Edit.

BTW.
You do not seem to have specified a source code license for your code e.g with a license.txt
Your souce code files also do not include any copyright or license information.
Normal practice is to specify the code author and code copyright in every appropriate file, i.e any file that does not only contain automatically generated /boilerplate code.

[SA0ASM]

Thanks for the feedback! Would any particular license suit the OpenGD77 project better? I would probably choose some BSD or MIT license variant, but I'm open to suggestions.

73 de SA0ASM