Contact lookup is consuming large amounts of CPU time

Discussions related to the firmware code development
VK3KYY
Posts: 7486
Joined: Sat Nov 16, 2019 3:25 am
Location: Melbourne, Australia

Contact lookup is consuming large amounts of CPU time

Post by VK3KYY » Sat Jan 25, 2020 2:38 am

Guys

I was investigating accesses to the 1Mb flash memory, and I came across something which I think needs to be changed.

When the radio receives a DMR signal, it searches all the way through the entire contacts section of the codeplug, which is 1024 records, loading each contact one by one.

This takes around 75mS and happens twice every time a signal is received.
flash_contacts_lookup.png
flash_contacts_lookup.png (54.84 KiB) Viewed 5131 times
The problem with accessing Flash is that its also processor intensive, because all the hardware SPI busses on the MCU are connected to the C6000, and the entirety of the transfer needs to be handled by the MCU including generating the clock and data signals.

I appreciate that people want to be able to see the name etc of stations in their contacts, but I think we need to think of a smarter way to do this.

Firstly, we should be caching the look-ups and we are already doing this to some extent with the Last Heard system, but we're not storing the name or DMR ID data as part of the Last Heard data and we should be doing this.

Also, and more controversially, I think that the Contacts DB should be checked after the DMR ID data not before.
This is because the DMR ID lookups are a binary tree search and only need to access the flash memory around 15 times to search 32,000+ ID's, hence its very quick.

The official firmware checks the DMR ID data first, which is a problem because the official firmware only supports 8 characters per ID, but the OpenGD77 supports 16 character and can be made to support more, hence both the callsign and name can be stored.

So I don't see a reason to lookup the Contact first, as it will not contain more information than the DMR ID database.

Also, if a DMR ID can't be found in either the DMR ID or Contacts DB, I think the caching system needs to handle this, so the next time the same station is heard, the system does not go through the whole process again.

KB8AOB
Posts: 29
Joined: Sun Dec 29, 2019 7:44 pm
Location: West Virginia

Re: Contact lookup is consuming large amounts of CPU time

Post by KB8AOB » Sat Jan 25, 2020 4:16 am

I do not load a contact DB as I must use a pi-star hotspot and I watch the dashboard on my tablet or screen. I do assume it does not spend time looking for nothing...

Would it make sense to add a switch in options to allow or not allow a search in the DB?

In my case, I really do not care if any information is displayed regarding the remote station.
Last edited by KB8AOB on Sat Jan 25, 2020 3:10 pm, edited 1 time in total.

User avatar
F1RMB
Posts: 2518
Joined: Sat Nov 16, 2019 5:42 am
Location: Grenoble, France

Re: Contact lookup is consuming large amounts of CPU time

Post by F1RMB » Sat Jan 25, 2020 4:34 am

Hi Roger,

Okay. Well, I agree about the contact source priority, as memory is faster.

Also, the first thing I would like to change is the way lookup is made, as it read the whole contact record while looking up.
I think reading only the 4 first bytes about the ID, then the full contact if it matches would save save time.

Cheers.
---
Daniel

User avatar
F1RMB
Posts: 2518
Joined: Sat Nov 16, 2019 5:42 am
Location: Grenoble, France

Re: Contact lookup is consuming large amounts of CPU time

Post by F1RMB » Sat Jan 25, 2020 4:46 am

Hi Roger,

About dmrLookup(), caching the matching contact record in a rolling contact buffer, of lets say 3 or 4 entries, would also save time, as it could be used on the next call(s).

Cheers.
---
Daniel

VK3KYY
Posts: 7486
Joined: Sat Nov 16, 2019 3:25 am
Location: Melbourne, Australia

Re: Contact lookup is consuming large amounts of CPU time

Post by VK3KYY » Sat Jan 25, 2020 5:15 am

Hi Daniel

I did implement a single record cache, and we could implement a bigger cache, but the Last Heard is already a big cache, so I think it makes more sense to use the Last Heard DB first.

I also noticed on the Last Heard screen it is doing a DMR ID lookup for every item in the list (that has data in it). This is also a waste of CPU power.



Re: Searching the contacts
I don't think there is enough space in RAM to cache all 1024 contacts.

The contact record is 28 bytes, (including an 'int' we compute and add to the data).
So we'd need 28672 bytes of RAM.

I think probably only 24 bytes of the 28 are actually used, but it will still take a lot of RAM to cache everything.


Re: DMR ID lookup reading the whole record and not just the ID.

Yes. It would be slightly better if we only read the ID number, except for the last lookup, when we will need perform another read to access the data, so this will take extra time



Overall, I still think there is a problem even if we cache, because the first lookup of a Contact is still going to take a very long time and a lot of CPU processing power

User avatar
F1RMB
Posts: 2518
Joined: Sat Nov 16, 2019 5:42 am
Location: Grenoble, France

Re: Contact lookup is consuming large amounts of CPU time

Post by F1RMB » Sat Jan 25, 2020 5:35 am

Hi Roger,
VK3KYY wrote:
Sat Jan 25, 2020 5:15 am
Hi Daniel

I did implement a single record cache, and we could implement a bigger cache, but the Last Heard is already a big cache, so I think it makes more sense to use the Last Heard DB first.
Sure last heard prevail.
VK3KYY wrote:
Sat Jan 25, 2020 5:15 am
I also noticed on the Last Heard screen it is doing a DMR ID lookup for every item in the list (that has data in it). This is also a waste of CPU power.
Okay, same apply, instead of crawling through the whole 1024 entries, max, each time.
VK3KYY wrote:
Sat Jan 25, 2020 5:15 am
Re: Searching the contacts
I don't think there is enough space in RAM to cache all 1024 contacts.

The contact record is 28 bytes, (including an 'int' we compute and add to the data).
So we'd need 28672 bytes of RAM.

I think probably only 24 bytes of the 28 are actually used, but it will still take a lot of RAM to cache everything.
I never meant to cache the whole thing, but just the last 3 or 4 last successful lookup.

VK3KYY wrote:
Sat Jan 25, 2020 5:15 am
Re: DMR ID lookup reading the whole record and not just the ID.

Yes. It would be slightly better if we only read the ID number, except for the last lookup, when we will need perform another read to access the data, so this will take extra time
Yes, the last full read will takes extra time, but the other x * (contact text lenght) are saved, even if the current code jumps in the middle of the list, and tries to find the way to go. Winner-winner, chicken dinner ;-)

VK3KYY wrote:
Sat Jan 25, 2020 5:15 am
Overall, I still think there is a problem even if we cache, because the first lookup of a Contact is still going to take a very long time and a lot of CPU processing power

I agree.

Cheers.
---
Daniel

VK3KYY
Posts: 7486
Joined: Sat Nov 16, 2019 3:25 am
Location: Melbourne, Australia

Re: Contact lookup is consuming large amounts of CPU time

Post by VK3KYY » Sat Jan 25, 2020 5:45 am

I just checked and we can't cache all the contacts as there isn't enough RAM. :-(

Well.. there may be enough RAM but the linker does not seem to allocate much of the SRAM_LOWER_BOTTOM section

e.g. here I tried to allocate the RAM, and the linker tries to put everything in the SRAM_UPPER

Code: Select all

Memory region         Used Size  Region Size  %age Used
   PROGRAM_FLASH:        476 KB       494 KB     96.36%
      SRAM_UPPER:       71428 B        64 KB    108.99%
SRAM_LOWER_BOTTOM:         604 B      27488 B      2.20%
 SRAM_LOWER_AMBE:          0 GB       8744 B      0.00%
I think there is probably something wrong with the linker setting that Alex setup, because I don't see the SRAM_LOWER_TOP in this listing either.

VK3KYY
Posts: 7486
Joined: Sat Nov 16, 2019 3:25 am
Location: Melbourne, Australia

Re: Contact lookup is consuming large amounts of CPU time

Post by VK3KYY » Sat Jan 25, 2020 5:48 am

One thought.

Assuming people don't still use loads of contacts instead of the DMR ID lookup....

If the CPS compacted the contacts to the start of the contacts memory, then the first empty contact is the end of the data, and the lookup would be much quicker.

But I investigated this before and its not that simple to compact the contacts data to remove fragmentation.

User avatar
F1RMB
Posts: 2518
Joined: Sat Nov 16, 2019 5:42 am
Location: Grenoble, France

Re: Contact lookup is consuming large amounts of CPU time

Post by F1RMB » Sat Jan 25, 2020 5:56 am

I was also checking the code.
Since everything goes through lasthheard first, I think lastheard should embed contact text (using contactID / dmrID lookups, as it's done afterward).
This way, menuUtilityRenderQSOData() will stop to lookup, everywhere on every call, to get the text from various locations, and only use lastheard TA only.

Cheers.

EDIT: and currently, lasheard is used as a fallback, if lookups fail.

VK3KYY
Posts: 7486
Joined: Sat Nov 16, 2019 3:25 am
Location: Melbourne, Australia

Re: Contact lookup is consuming large amounts of CPU time

Post by VK3KYY » Sat Jan 25, 2020 8:29 am

I agree the Last Heard list should contain the contact name, and not just the TA data.

Actually perhaps TA data and the contact name can probably share the same buffer.

Post Reply