script: Unaccounted size in ram/rom reports

[elliot-wdtl]

Describe the bug
It seems like there is a variable amount of missing data in the ram/rom report output.
For example:
{ "symbols": { "name": "root", "size": 101962, "identifier": ":", "children": [ ... ] "total_size": 421321 }
If I sum the size of all of the top level children of "root" I don't end up with 421321 and in this case I actually ended up with 326307.
Each of the top level children of "root" seem to have a size which is the sum of their children's sizes which makes sense.
This doesn't seem to apply to "root" itself though and I don't know what that "root" size value represents.
It's not the difference between the sum of the children and the total although it often seems like it is close to that.
In this particular case I end up with 95014 bytes that I don't know what they are being used for in flash (this is one of the most extreme cases I've seen).

To Reproduce
Steps to reproduce the behavior:

1. Build a project.
2. Run the rom/ram report on it.
3. Add the size of all of the children of the "root" node.
4. Compare it to the total size or ram/rom used.

Expected behavior
Ideally the json generated by the rom/ram report script would account for all resource use.

Impact
It just makes it a little difficult to figure out what is taking up resources.

Environment (please complete the following information):

• OS: Windows
• Toolchain: Python 3.9.0
• Commit SHA or Version used: fork from v2.6

[dcpleung]

Do you have a particular sample that I can use to figure out what is going on? The root node is actually using the ELF file's sections to calculate its size instead of the sum of its children. In your case, it is possible that the script is double counting some sections.

[dcpleung]

We talked offline and found out that the issue has been fixed in main branch.

[elliot-wdtl]

I wouldn't say it's fixed in main branch. It is different but the sizes of things still don't make sense to me. Here is an example of the output:

{
    "symbols": {
        "name": "Root",
        "size": 65403,
        "identifier": "root",
        "children": [
            {
                "name": "(no paths)",
                "size": 5235,
                "identifier": ":",
                "children": []
            },
            {
                "name": "ZEPHYR_BASE",
                "size": 35797,
                "identifier": "path/to/zephyr",
                "children": []
            },
            {
                "name": "/",
                "size": 0,
                "identifier": "/",
                "children": [
                    {
                        "name": "C:\\",
                        "size": 20275,
                        "identifier": "C:\\",
                        "children": []
                    }
                ]
            },
            {
                "name": "(hidden)",
                "size": 24371,
                "identifier": "(hidden)"
            }
        ]
    },
    "total_size": 65403
}

Now we get different first level children in root "(no paths)", "ZEPHYR_BASE", "(hidden)" and "/" (which seems to mostly be all sources other than zephyr.
Now if we add the sizes of the first 3 we get the size of the parent node ("Root").
This means "/" which contains all the sizes of my symbols must be overlapping with one of the other 3 or something.

[dcpleung]

Hm... possibly due to how paths are handled under Python in Windows. Is it possible to build your code under Linux as comparison?

[elliot-wdtl]

Looks like you are right. When I run it under linux the top level "/" node actually gets a size and the "(hidden)" one is around 20k smaller. The 4 top level nodes actually add up to the total too.
What exactly is "(hidden)" because it still accounts for 4k out of my 64k total?

[dcpleung]

"(hidden)" are symbols taking up memory but there no associated debug information and/or symbols. Most of the time these are strings used directly in function calls. For example, printk("Hello World!"); where "Hello World" has no symbol associated but still taking up space.

[elliot-wdtl]

Thanks @dcpleung. Your example would account for some hidden things in a rom report right? Do you have an example of what would end up in "(hidden)" in the ram report?

[dcpleung]

Hm... can't think of one at the moment. Eventually it depends on the toolchain whether it wants to emit certain symbols. Since those symbols are not emitted, there is no direct way of figuring out where they reside in memory either. So if you really want to get to the bottom of it, you will need to figure out the memory regions not covered by all other symbols, and wait for those to get accessed in debugger.

[elliot-wdtl]

Great thanks for the info. I couldn't think of anything either but at least I can account for a large portion of "(hidden)" in my rom reports.