Wrong commit id from FileHistoryCache lead to wrong dynamic file content view

[mmmarq]

Describe the bug
I have a public git project indexed and checked out at master branch. This branch point to the commit e54084b5d1542187e5b8560f9f29e372680f4052. But when I browse through the project and open a file named core/build_id.mk I got the following URL:

source/xref/build-ori/core/build_id.mk?r=390c2376

The revision id does not match to project HEAD. That is ok because this revision comes from core/build_id.mk history, right? Yes, but if I check this file history I got a diff top commit id, which is 63e74fdff18d4f7f5a859969ef984e37bf4a8fa7. Checking the project history I can see that file revision get by opengrok (390c2376) is far above the true file top commit (63e74fdf). Checking opengrop file history page, I see changes that are not really related to the file.

It would be a minor issue if my code was not indexed with "--economical" flag. With this flag, opengrok generate a dynamic page to show file content, and since it use the "wrong" revision (390c2376), file content is also wrong:

workspace file last line:
BUILD_ID=AOSP.MASTER

Last line from opengrok webapp (source/xref/build/core/build_id.mk?a=true&r=390c2376)
BUILD_ID=TQ1A.230205.002

Is there any way to check how opengrok get this revision?

To Reproduce
Clone public repository:
git clone https://android.googlesource.com/platform/build

Checkout master branch:
git checkout master

Index command:
/opengrok/opengrok-tools/bin/opengrok-indexer --java /tools/jre/bin/java --java_opts=-server --java_opts=-Xmx24g --java_opts=-Dorg.opengrok.indexer.history.git=/bin/git --jar /opengrok/lib/opengrok.jar -- --ctags /tools/ctags/ctags --dataRoot /opengrok/data -i d:.repo -i d:.git -H --projects --depth 50 --source /opengrok/src_root --search --writeConfig /opengrok/configuration.xml --progress --economical

Expected behavior
When dynamic page is generated, file content should be the same as the file in git workspace (disk)

Additional context

Opengrok 1.7.41
JDK openjdk12.0.2-x64
Tomcat apache-tomcat-10.0.12
OS Ubuntu 18.04.6 LTS
Git version 2.39.0

[vladak]

I can see the same thing, however it is not clear to me what is going on in Git. Before I summarize what was found so far, let me explain how history cache creation works in OpenGrok: to avoid getting history for each file individually (which would take a very long time for the overall indexing), a trick is used for source code management systems that support getting history for directory. The indexer retrieves the history for the top-level directory of given repository, along with all the files that have changed in some way in each commit/revision. For Git, that would be something like git log --stat <top_level_directory>, except these days the Git support is implemented in Java using JGit, but it should not matter. Next, a hash map is constructed that maps a commit ID to list of files changed/added/deleted in given commit. Then the hash map is inverted so that it a new hash map is created that maps file names to change IDs in which the file was changed/added/deleted. This hash map is then traversed and used to store history cache for individual files. This means that the commits in history cache for given file do not necessarily have to correspond to the history retrieved for the file individually, e.g. to the equivalent of git log --stat <top_level_directory>/some/path/file.txt.

Now, 63e74fdff18d4f7f5a859969ef984e37bf4a8fa7 is a merge commit, so that is not listed in history cache unless merge commits are explicitly enabled for indexer. It's parent is 1849ad5c5f367b11ba066b05ce662140e44d5c40 which is regular commit, changing the core/build_id.mk file. This commit is visible in the history cache generated for the file. It's the 808th changeset when getting the commits from the history cache file:

$ gzip -dc data.android/historycache/android-build/core/build_id.mk.gz | grep -A 1 '\"revision\"' | grep '<string>' | cat -n | grep 1849ad5
   808        <string>1849ad5c</string>

There are 5501 commits in the history cache for that file. Plain git log for the file shows much less than that:

$ git log --pretty=oneline core/build_id.mk | wc -l
594

however when run with --follow, the count it almost matches (might be interesting to see that 1 differing commit):

$ git log --follow --pretty=oneline core/build_id.mk | wc -l
5502

So, it looks as if the history cache was generated with renamed file handling enabled, which it was not.

Looking at the differences between commit IDs reported by git log --follow and what is actually stored in the history cache for that file, I can see a divergence after changeset b2897025. The history cache entries continue with f75e2547 while the git log --follow output with ab5e1624f0. This sequence then reunites at changeset d8822209.

There is another divergence at the beginning (or end, depending on how one is looking at the sequence). There is 1 extra commit - 05806d7af6 (diff -y output), sort of explaining the 5501/5502 total commit count difference:

a7e72e17                                                      | a7e72e17df
f122bec4                                                      | f122bec404
6bce2058                                                      | 6bce205856
88b60799                                                      | 88b607994a
475fa12a                                                      | 05806d7af6
dcc08f07                                                      | 475fa12ade
b6c1cf6d                                                      | dcc08f073b
                                                              > b6c1cf6de7

In that changeset, the core/build_id.mk was deleted. The abbreviation used by the indexer does not match the abbreviation used by the git log command, perhaps it does not matter in this case.

I'm afraid this brings more questions than answers. In particular, why does Git consider the file renamed and why is there the longer commit divergence. The ultimate question would be if the indexer could do something differently.