[perf] Improve analyze file filter for header

[gzzi]

In this PR:

• Compute dependencies of each source file of the compilation database only once for all header provided
• Parse the compilation database in parallel

Close #4949

[bruntib]

Hi @gzzi,

Thank you for the path, this is a really nice catch and a great improvement!

However, I have some concerns related to the solution design. It doesn't seem intuitive for a library to provide a "dependencies" as a parameter because it is supposed to be the result of the function, more or less. I mean, only an extra filtering happens, based on the file filter.

Couldn't get_dependent_sources() be redesigned in a way that header_path becomes header_paths: Optional[List[str]]. This way, this function could be called only once from get_affected_file_paths() instead of calling it many times in a loop.

What do you think?

[gzzi]

Hello @bruntib,

Agree, I tried to keep my change on analyze.py low. Now I pushed a version where the loop iteration is done on the tu_collector lib. Let me know what you think about it.

Note that now the order of file_paths in get_affected_file_paths is not the same as before. All the source affected by header are placed at the end.

[bruntib]

Hi @gzzi,

Thank you for the fixes. I have just two small comments:

1. Can header_file variable be renamed to header_files?
2. The default value of jobs in get_dependent_sources() function parameter could be 1 instead of None. This way there is no need to check if jobs is None.

Also, one more question. I understand the goal of the development, and it is really useful in case of big projects. However, if I #include "dummy.hpp" in the beginning of single_entry.cpp and run the command CodeChecker analyze --file dummy.hpp ... then it still doesn't analyze anything, because --file works as if one single file was provided in a skipfile with --skip. Since single_entry.cpp doesn't match dummy.hpp, it won't be analyzed.
This is an independent question from this PR, because this was the situation earlier, too. I'm just wondering, what is the usage of --file flag in your project in this regard?

Thank you again this development. Please, consider applying these two minor fixes, otherwise we can merge it.

[gzzi]

Hello @bruntib,

Requested changes have been applied.

if I #include "dummy.hpp" in the beginning of single_entry.cpp and run the command CodeChecker analyze --file dummy.hpp ... then it still doesn't analyze anything

The path are absolute on the compile_commands.json. You need to provide --file $(pwd)/dummy.hpp. Here is a script showing that:

#!/bin/bash

function generate_db {
    echo "Generating database..."

    echo "[" > compile_commands.json

    for _ in {1..1000}; do
        echo "{\"directory\": \"$(pwd)\", \"command\": \"g++ dummy.cpp -c -o dummy.o\", \"file\": \"dummy.cpp\", \"output\": \"dummy.o\"}," >> compile_commands.json
    done
    echo "{\"directory\": \"$(pwd)\", \"command\": \"g++ single_entry.cpp -c -o dummy.o\", \"file\": \"single_entry.cpp\", \"output\": \"dummy.o\"}" >> compile_commands.json
    echo "]" >> compile_commands.json
}

generate_db
touch dummy.cpp
echo "#include \"dummy.hpp\"" > single_entry.cpp
touch dummy.hpp
touch dummy_2.hpp

echo "Relative path ------------------------------------------"
time CodeChecker analyze compile_commands.json -o ./reports --file dummy.hpp

echo "Absolute path ------------------------------------------"
time CodeChecker analyze compile_commands.json -o ./reports --file $(pwd)/dummy.hpp

I'm just wondering, what is the usage of --file flag in your project in this regard?

On our CI, when an analysis of a pull request is made, we check only files that was modified compared to main branch fork point. This save us quite a lot of time. The full analysis are made only nightly.

[bruntib]

Hi @gzzi,

I see now. Thank you for the explanation and thank you for the patch!