Fix binary handling #1999

bill-rich · 2023-10-25T19:00:25Z

Binary diffs were being dropped because the content length was zero.

rgmz · 2023-10-25T19:10:07Z

Wouldn't the diffs still be devoid of content?

$ git log -p --full-history --all
commit cb6448321d3fb5b6db253c0a7d68770826bd7e09 (HEAD -> master)
Author: John <[email protected]>
Date:   Wed Oct 25 15:06:29 2023 -0400

    2

diff --git a/demo.zip b/demo.zip
index 6a3d370..47ddff8 100644
Binary files a/demo.zip and b/demo.zip differ

commit 6df4bff0dd1f19015823c070494f39f1ada86820
Author: John <[email protected]>
Date:   Wed Oct 25 15:05:05 2023 -0400

    test

diff --git a/demo.zip b/demo.zip
new file mode 100644
index 0000000..6a3d370
Binary files /dev/null and b/demo.zip differ

Does TruffleHog handle binary diffs differently, or use --text somewhere that I'm not seeing?

$ git log -p --full-history --all --text
commit cb6448321d3fb5b6db253c0a7d68770826bd7e09 (HEAD -> master)
Author: John <[email protected]>
Date:   Wed Oct 25 15:06:29 2023 -0400

    2

diff --git a/demo.zip b/demo.zip
index 6a3d370..47ddff8 100644
--- a/demo.zip
+++ b/demo.zip
@@ -1,33 +1,27 @@
-PK^C^D^T^@^H^H^H^@̈YW^@^@^@^@^@^@^@^@^@^@^@^@^E^@5^@demo/UT^M^@^G^PK9e^OK9e^PK9e
-^@ ^@^@^@^@^@^A^@^X^@2<DA>_<95>e^G<DA>^A<DD>^V_<95>e^G<DA>^A2<DA>_<95>e^G<DA>^A^C^@PK^G^H^@^@^@^@^B^@^@^@^@^@^@^@PK^C^D^T^@^H^H^H^@ˈYW^@^@^@^@^@^@^@^@^@^@^@^@^T^@5^@demo/settings.gradleUT^M^@^G^OK9e^OK9e^OK9e
-^@ ^@^@^@^@^@^A^@^X^@<DD>^V_<95>e^G<DA>^A<DD>^V_<95>e^G<DA>^A<DD>^V_<95>e^G<DA>^A+<CA><CF>/    (<CA><CF>JM.<D1><CB>K<CC>MU<B0>UPOI<CD><CD>W<E7>^B^@PK^G^H<C8>^S<D2>:^\^@^@^@^Z^@^@^@PK^C^D^T^@^H^H^H^@ˈYW^@^@^@^@^@^@^@^@^@^@^@^@^L^@5^@demo/gradlewUT^M^@^G^OK9e^OK9e^OK9e
-^@ ^@^@^@^@^@^A^@^X^@<EE>=_<95>e^G<DA>^A<EE>=_<95>e^G<DA>^A<EE>=_<95>e^G<DA>^A<BD>Y<EB>v<D3>H^R<FE><EF><A7>h<94><80><E3>Ld<87>03?^R<U+008E>I^B<98>!<97>^S^G^B^GX<A6>-<B5><ED>&<B2><A4>QKq| <F3>>{<F6>^Q<E6>^_O<B6>_U<B7>.^N^Y<D8>=g<CE>z.<B1><A5><EA><EA><AA>ꪯ.<BD>r<A7>7<D2>q<CF>L[<AD><95>֊<D8>K<D2>E<A6>'<D3>\|<F9><97><D8>ڼ<FF><93><BF><B5><B9>u_<E4>S%^R<ױ<8C><84>,<F2>i<92><99>./x<A1>^C^U<8A>"^NUƄ<FD>T^F<F8><E3><DE>l<88>W*3:<89><C5>VwS<AC>^Q<81><E7>^y<9D>^]pX$<85><98>Ʌ<88><93>\^TF<81><85>6b<AC>#%<D4>U<A0><D2>\<E8>X^D<C9>,<8D><B4><8C>^C%<E6>:<9F><F2>6<8E>  <C4>^Po^\<8B>d<94>KPKЧ<F8>5n<D2>        <99><B3><C0><FC><99><E6>yj<B6>{<BD><F9>|ޕ,n7<C9>&<BD>Ȓ<9A>ދ<C1><DE><C1><D1><F0><C0><87>ȼ<E8>e^\)cD<A6>~/t^FeG^K!SH^T
<C8>^Q<E4><8C><E4>^\<D6>^Qr<92>)<BC><CB>^S<92>x<9E><E9>\Ǔ^Ma<92>q><97><99>^B<97>P<9B><ӣ"_2W)^_<B4>n^R<C0>`2^V^^?(^FCO<<EE>^O^G<C3>^M<F0>8^_<9C>=;~y&<CE><FB><A7><A7><FD><A3><B3><C1><C1>P^\<9F><8A><BD><E3><A3><FD><C1><D9><E0><F8>^H<BF><9E>
<88><FE><D1>^?C(^X^Kۨ<AB>4#<F9>!<A4>&C<AA><90><AC>6TjI<80>qb^E2<A9>
+PK^C^D^T^@^H^H^H^@ɘYW^@^@^@^@^@^@^@^@^@^@^@^@^E^@5^@demo/UT^M^@^G*g9e*g9e*g9e
+^@ ^@^@^@^@^@^A^@^X^@<84><93><D4>Uv^G<DA>^A <A9><D3>Uv^G<DA>^A<84><93><D4>Uv^G<DA>^A^C^@PK^G^H^@^@^@^@^B^@^@^@^@^@^@^@PK^C^D^T^@^H^H^H^@ɘYW^@^@^@^@^@^@^@^@^@^@^@^@^L^@5^@demo/gradlewUT^M^@^G*g9e*g9e*g9e
+^@ ^@^@^@^@^@^A^@^X^@A<F7><D3>Uv^G<DA>^A0<D0><D3>Uv^G<DA>^AA<F7><D3>Uv^G<DA>^A<BD>Y<EB>v<D3>H^R<FE><EF><A7>h<94><80><E3>Ld<87>03?^R<U+008E>I^B<98>!<97>^S^G^B^GX<A6>-<B5><ED>&<B2><A4>QKq| <F3>>{<F6>^Q<E6>^_O<B6>_U<B7>.^N^Y<D8>=g<CE>z.<B1><A5><EA><EA><AA>ꪯ.<BD>r<A7>7<D2>q<CF>L[<AD><95>֊<D8>K<D2>E<A6>'<D3>\|<F9><97><D8>ڼ<FF><93><BF><B5><B9>u_<E4>S%^R<ױ<8C><84>,<F2>i<92><99>./x<A1>^C^U<8A>"^NUƄ<FD>T^F<F8><E3><DE>l<88>W*3:<89>
...

bill-rich · 2023-10-25T19:53:46Z

Wouldn't the diffs still be devoid of content?

$ git log -p --full-history --all
commit cb6448321d3fb5b6db253c0a7d68770826bd7e09 (HEAD -> master)
Author: John <[email protected]>
Date:   Wed Oct 25 15:06:29 2023 -0400

    2

diff --git a/demo.zip b/demo.zip
index 6a3d370..47ddff8 100644
Binary files a/demo.zip and b/demo.zip differ

commit 6df4bff0dd1f19015823c070494f39f1ada86820
Author: John <[email protected]>
Date:   Wed Oct 25 15:05:05 2023 -0400

    test

diff --git a/demo.zip b/demo.zip
new file mode 100644
index 0000000..6a3d370
Binary files /dev/null and b/demo.zip differ

Does TruffleHog handle binary diffs differently, or use --text somewhere that I'm not seeing?

$ git log -p --full-history --all --text
commit cb6448321d3fb5b6db253c0a7d68770826bd7e09 (HEAD -> master)
Author: John <[email protected]>
Date:   Wed Oct 25 15:06:29 2023 -0400

    2

diff --git a/demo.zip b/demo.zip
index 6a3d370..47ddff8 100644
--- a/demo.zip
+++ b/demo.zip
@@ -1,33 +1,27 @@
-PK^C^D^T^@^H^H^H^@̈YW^@^@^@^@^@^@^@^@^@^@^@^@^E^@5^@demo/UT^M^@^G^PK9e^OK9e^PK9e
-^@ ^@^@^@^@^@^A^@^X^@2<DA>_<95>e^G<DA>^A<DD>^V_<95>e^G<DA>^A2<DA>_<95>e^G<DA>^A^C^@PK^G^H^@^@^@^@^B^@^@^@^@^@^@^@PK^C^D^T^@^H^H^H^@ˈYW^@^@^@^@^@^@^@^@^@^@^@^@^T^@5^@demo/settings.gradleUT^M^@^G^OK9e^OK9e^OK9e
-^@ ^@^@^@^@^@^A^@^X^@<DD>^V_<95>e^G<DA>^A<DD>^V_<95>e^G<DA>^A<DD>^V_<95>e^G<DA>^A+<CA><CF>/    (<CA><CF>JM.<D1><CB>K<CC>MU<B0>UPOI<CD><CD>W<E7>^B^@PK^G^H<C8>^S<D2>:^\^@^@^@^Z^@^@^@PK^C^D^T^@^H^H^H^@ˈYW^@^@^@^@^@^@^@^@^@^@^@^@^L^@5^@demo/gradlewUT^M^@^G^OK9e^OK9e^OK9e
-^@ ^@^@^@^@^@^A^@^X^@<EE>=_<95>e^G<DA>^A<EE>=_<95>e^G<DA>^A<EE>=_<95>e^G<DA>^A<BD>Y<EB>v<D3>H^R<FE><EF><A7>h<94><80><E3>Ld<87>03?^R<U+008E>I^B<98>!<97>^S^G^B^GX<A6>-<B5><ED>&<B2><A4>QKq| <F3>>{<F6>^Q<E6>^_O<B6>_U<B7>.^N^Y<D8>=g<CE>z.<B1><A5><EA><EA><AA>ꪯ.<BD>r<A7>7<D2>q<CF>L[<AD><95>֊<D8>K<D2>E<A6>'<D3>\|<F9><97><D8>ڼ<FF><93><BF><B5><B9>u_<E4>S%^R<ױ<8C><84>,<F2>i<92><99>./x<A1>^C^U<8A>"^NUƄ<FD>T^F<F8><E3><DE>l<88>W*3:<89><C5>VwS<AC>^Q<81><E7>^y<9D>^]pX$<85><98>Ʌ<88><93>\^TF<81><85>6b<AC>#%<D4>U<A0><D2>\<E8>X^D<C9>,<8D><B4><8C>^C%<E6>:<9F><F2>6<8E>  <C4>^Po^\<8B>d<94>KPKЧ<F8>5n<D2>        <99><B3><C0><FC><99><E6>yj<B6>{<BD><F9>|ޕ,n7<C9>&<BD>Ȓ<9A>ދ<C1><DE><C1><D1><F0><C0><87>ȼ<E8>e^\)cD<A6>~/t^FeG^K!SH^T
<C8>^Q<E4><8C><E4>^\<D6>^Qr<92>)<BC><CB>^S<92>x<9E><E9>\Ǔ^Ma<92>q><97><99>^B<97>P<9B><ӣ"_2W)^_<B4>n^R<C0>`2^V^^?(^FCO<<EE>^O^G<C3>^M<F0>8^_<9C>=;~y&<CE><FB><A7><A7><FD><A3><B3><C1><C1>P^\<9F><8A><BD><E3><A3><FD><C1><D9><E0><F8>^H<BF><9E>
<88><FE><D1>^?C(^X^Kۨ<AB>4#<F9>!<A4>&C<AA><90><AC>6TjI<80>qb^E2<A9>
+PK^C^D^T^@^H^H^H^@ɘYW^@^@^@^@^@^@^@^@^@^@^@^@^E^@5^@demo/UT^M^@^G*g9e*g9e*g9e
+^@ ^@^@^@^@^@^A^@^X^@<84><93><D4>Uv^G<DA>^A <A9><D3>Uv^G<DA>^A<84><93><D4>Uv^G<DA>^A^C^@PK^G^H^@^@^@^@^B^@^@^@^@^@^@^@PK^C^D^T^@^H^H^H^@ɘYW^@^@^@^@^@^@^@^@^@^@^@^@^L^@5^@demo/gradlewUT^M^@^G*g9e*g9e*g9e
+^@ ^@^@^@^@^@^A^@^X^@A<F7><D3>Uv^G<DA>^A0<D0><D3>Uv^G<DA>^AA<F7><D3>Uv^G<DA>^A<BD>Y<EB>v<D3>H^R<FE><EF><A7>h<94><80><E3>Ld<87>03?^R<U+008E>I^B<98>!<97>^S^G^B^GX<A6>-<B5><ED>&<B2><A4>QKq| <F3>>{<F6>^Q<E6>^_O<B6>_U<B7>.^N^Y<D8>=g<CE>z.<B1><A5><EA><EA><AA>ꪯ.<BD>r<A7>7<D2>q<CF>L[<AD><95>֊<D8>K<D2>E<A6>'<D3>\|<F9><97><D8>ڼ<FF><93><BF><B5><B9>u_<E4>S%^R<ױ<8C><84>,<F2>i<92><99>./x<A1>^C^U<8A>"^NUƄ<FD>T^F<F8><E3><DE>l<88>W*3:<89>
...

They would still be empty, but the git source uses IsBinary and PathB to pull the file from the commit using go-git. Here is where it gets the file and scans it.

ahrav · 2023-10-26T03:16:37Z

Should we be concerned about the Perf test regression, or is this an aberration?

bill-rich · 2023-10-26T15:10:59Z

Should we be concerned about the Perf test regression, or is this an aberration?

The difference in time is just from actually scanning the binary files. I tested by skipping handleBinary (so its just the gitparse changes in effect) and it had similar performance.

Fix binary handling

df929ee

bill-rich requested a review from a team as a code owner October 25, 2023 19:00

ahrav approved these changes Oct 26, 2023

View reviewed changes

bill-rich merged commit 00a00ef into main Oct 26, 2023
8 of 9 checks passed

bill-rich deleted the fix_git_binary_handling branch October 26, 2023 17:07

Phoenix591 pushed a commit to Phoenix591/trufflehog that referenced this pull request Oct 27, 2023

Fix binary handling (trufflesecurity#1999)

7e0be12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix binary handling #1999

Fix binary handling #1999

bill-rich commented Oct 25, 2023

rgmz commented Oct 25, 2023 •

edited

Loading

bill-rich commented Oct 25, 2023

ahrav commented Oct 26, 2023

bill-rich commented Oct 26, 2023

Fix binary handling #1999

Fix binary handling #1999

Conversation

bill-rich commented Oct 25, 2023

rgmz commented Oct 25, 2023 • edited Loading

bill-rich commented Oct 25, 2023

ahrav commented Oct 26, 2023

bill-rich commented Oct 26, 2023

rgmz commented Oct 25, 2023 •

edited

Loading