Contact lookup is consuming large amounts of CPU time
Contact lookup is consuming large amounts of CPU time
Guys
I was investigating accesses to the 1Mb flash memory, and I came across something which I think needs to be changed.
When the radio receives a DMR signal, it searches all the way through the entire contacts section of the codeplug, which is 1024 records, loading each contact one by one.
This takes around 75mS and happens twice every time a signal is received.
The problem with accessing Flash is that its also processor intensive, because all the hardware SPI busses on the MCU are connected to the C6000, and the entirety of the transfer needs to be handled by the MCU including generating the clock and data signals.
I appreciate that people want to be able to see the name etc of stations in their contacts, but I think we need to think of a smarter way to do this.
Firstly, we should be caching the look-ups and we are already doing this to some extent with the Last Heard system, but we're not storing the name or DMR ID data as part of the Last Heard data and we should be doing this.
Also, and more controversially, I think that the Contacts DB should be checked after the DMR ID data not before.
This is because the DMR ID lookups are a binary tree search and only need to access the flash memory around 15 times to search 32,000+ ID's, hence its very quick.
The official firmware checks the DMR ID data first, which is a problem because the official firmware only supports 8 characters per ID, but the OpenGD77 supports 16 character and can be made to support more, hence both the callsign and name can be stored.
So I don't see a reason to lookup the Contact first, as it will not contain more information than the DMR ID database.
Also, if a DMR ID can't be found in either the DMR ID or Contacts DB, I think the caching system needs to handle this, so the next time the same station is heard, the system does not go through the whole process again.
I was investigating accesses to the 1Mb flash memory, and I came across something which I think needs to be changed.
When the radio receives a DMR signal, it searches all the way through the entire contacts section of the codeplug, which is 1024 records, loading each contact one by one.
This takes around 75mS and happens twice every time a signal is received.
The problem with accessing Flash is that its also processor intensive, because all the hardware SPI busses on the MCU are connected to the C6000, and the entirety of the transfer needs to be handled by the MCU including generating the clock and data signals.
I appreciate that people want to be able to see the name etc of stations in their contacts, but I think we need to think of a smarter way to do this.
Firstly, we should be caching the look-ups and we are already doing this to some extent with the Last Heard system, but we're not storing the name or DMR ID data as part of the Last Heard data and we should be doing this.
Also, and more controversially, I think that the Contacts DB should be checked after the DMR ID data not before.
This is because the DMR ID lookups are a binary tree search and only need to access the flash memory around 15 times to search 32,000+ ID's, hence its very quick.
The official firmware checks the DMR ID data first, which is a problem because the official firmware only supports 8 characters per ID, but the OpenGD77 supports 16 character and can be made to support more, hence both the callsign and name can be stored.
So I don't see a reason to lookup the Contact first, as it will not contain more information than the DMR ID database.
Also, if a DMR ID can't be found in either the DMR ID or Contacts DB, I think the caching system needs to handle this, so the next time the same station is heard, the system does not go through the whole process again.
Re: Contact lookup is consuming large amounts of CPU time
I do not load a contact DB as I must use a pi-star hotspot and I watch the dashboard on my tablet or screen. I do assume it does not spend time looking for nothing...
Would it make sense to add a switch in options to allow or not allow a search in the DB?
In my case, I really do not care if any information is displayed regarding the remote station.
Would it make sense to add a switch in options to allow or not allow a search in the DB?
In my case, I really do not care if any information is displayed regarding the remote station.
Last edited by KB8AOB on Sat Jan 25, 2020 3:10 pm, edited 1 time in total.
Re: Contact lookup is consuming large amounts of CPU time
Hi Roger,
Okay. Well, I agree about the contact source priority, as memory is faster.
Also, the first thing I would like to change is the way lookup is made, as it read the whole contact record while looking up.
I think reading only the 4 first bytes about the ID, then the full contact if it matches would save save time.
Cheers.
---
Daniel
Okay. Well, I agree about the contact source priority, as memory is faster.
Also, the first thing I would like to change is the way lookup is made, as it read the whole contact record while looking up.
I think reading only the 4 first bytes about the ID, then the full contact if it matches would save save time.
Cheers.
---
Daniel
Re: Contact lookup is consuming large amounts of CPU time
Hi Roger,
About dmrLookup(), caching the matching contact record in a rolling contact buffer, of lets say 3 or 4 entries, would also save time, as it could be used on the next call(s).
Cheers.
---
Daniel
About dmrLookup(), caching the matching contact record in a rolling contact buffer, of lets say 3 or 4 entries, would also save time, as it could be used on the next call(s).
Cheers.
---
Daniel
Re: Contact lookup is consuming large amounts of CPU time
Hi Daniel
I did implement a single record cache, and we could implement a bigger cache, but the Last Heard is already a big cache, so I think it makes more sense to use the Last Heard DB first.
I also noticed on the Last Heard screen it is doing a DMR ID lookup for every item in the list (that has data in it). This is also a waste of CPU power.
Re: Searching the contacts
I don't think there is enough space in RAM to cache all 1024 contacts.
The contact record is 28 bytes, (including an 'int' we compute and add to the data).
So we'd need 28672 bytes of RAM.
I think probably only 24 bytes of the 28 are actually used, but it will still take a lot of RAM to cache everything.
Re: DMR ID lookup reading the whole record and not just the ID.
Yes. It would be slightly better if we only read the ID number, except for the last lookup, when we will need perform another read to access the data, so this will take extra time
Overall, I still think there is a problem even if we cache, because the first lookup of a Contact is still going to take a very long time and a lot of CPU processing power
I did implement a single record cache, and we could implement a bigger cache, but the Last Heard is already a big cache, so I think it makes more sense to use the Last Heard DB first.
I also noticed on the Last Heard screen it is doing a DMR ID lookup for every item in the list (that has data in it). This is also a waste of CPU power.
Re: Searching the contacts
I don't think there is enough space in RAM to cache all 1024 contacts.
The contact record is 28 bytes, (including an 'int' we compute and add to the data).
So we'd need 28672 bytes of RAM.
I think probably only 24 bytes of the 28 are actually used, but it will still take a lot of RAM to cache everything.
Re: DMR ID lookup reading the whole record and not just the ID.
Yes. It would be slightly better if we only read the ID number, except for the last lookup, when we will need perform another read to access the data, so this will take extra time
Overall, I still think there is a problem even if we cache, because the first lookup of a Contact is still going to take a very long time and a lot of CPU processing power
Re: Contact lookup is consuming large amounts of CPU time
Hi Roger,
I agree.
Cheers.
---
Daniel
Sure last heard prevail.
Okay, same apply, instead of crawling through the whole 1024 entries, max, each time.
I never meant to cache the whole thing, but just the last 3 or 4 last successful lookup.VK3KYY wrote: ↑Sat Jan 25, 2020 5:15 amRe: Searching the contacts
I don't think there is enough space in RAM to cache all 1024 contacts.
The contact record is 28 bytes, (including an 'int' we compute and add to the data).
So we'd need 28672 bytes of RAM.
I think probably only 24 bytes of the 28 are actually used, but it will still take a lot of RAM to cache everything.
Yes, the last full read will takes extra time, but the other x * (contact text lenght) are saved, even if the current code jumps in the middle of the list, and tries to find the way to go. Winner-winner, chicken dinner
I agree.
Cheers.
---
Daniel
Re: Contact lookup is consuming large amounts of CPU time
I just checked and we can't cache all the contacts as there isn't enough RAM.
Well.. there may be enough RAM but the linker does not seem to allocate much of the SRAM_LOWER_BOTTOM section
e.g. here I tried to allocate the RAM, and the linker tries to put everything in the SRAM_UPPER
I think there is probably something wrong with the linker setting that Alex setup, because I don't see the SRAM_LOWER_TOP in this listing either.
Well.. there may be enough RAM but the linker does not seem to allocate much of the SRAM_LOWER_BOTTOM section
e.g. here I tried to allocate the RAM, and the linker tries to put everything in the SRAM_UPPER
Code: Select all
Memory region Used Size Region Size %age Used
PROGRAM_FLASH: 476 KB 494 KB 96.36%
SRAM_UPPER: 71428 B 64 KB 108.99%
SRAM_LOWER_BOTTOM: 604 B 27488 B 2.20%
SRAM_LOWER_AMBE: 0 GB 8744 B 0.00%
Re: Contact lookup is consuming large amounts of CPU time
One thought.
Assuming people don't still use loads of contacts instead of the DMR ID lookup....
If the CPS compacted the contacts to the start of the contacts memory, then the first empty contact is the end of the data, and the lookup would be much quicker.
But I investigated this before and its not that simple to compact the contacts data to remove fragmentation.
Assuming people don't still use loads of contacts instead of the DMR ID lookup....
If the CPS compacted the contacts to the start of the contacts memory, then the first empty contact is the end of the data, and the lookup would be much quicker.
But I investigated this before and its not that simple to compact the contacts data to remove fragmentation.
Re: Contact lookup is consuming large amounts of CPU time
I was also checking the code.
Since everything goes through lasthheard first, I think lastheard should embed contact text (using contactID / dmrID lookups, as it's done afterward).
This way, menuUtilityRenderQSOData() will stop to lookup, everywhere on every call, to get the text from various locations, and only use lastheard TA only.
Cheers.
EDIT: and currently, lasheard is used as a fallback, if lookups fail.
Since everything goes through lasthheard first, I think lastheard should embed contact text (using contactID / dmrID lookups, as it's done afterward).
This way, menuUtilityRenderQSOData() will stop to lookup, everywhere on every call, to get the text from various locations, and only use lastheard TA only.
Cheers.
EDIT: and currently, lasheard is used as a fallback, if lookups fail.
Re: Contact lookup is consuming large amounts of CPU time
I agree the Last Heard list should contain the contact name, and not just the TA data.
Actually perhaps TA data and the contact name can probably share the same buffer.
Actually perhaps TA data and the contact name can probably share the same buffer.