-
Notifications
You must be signed in to change notification settings - Fork 3
/
howtodebug
140 lines (112 loc) · 6.67 KB
/
howtodebug
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
This is a short guide on how to find, debug and fix reproducibility issues in openSUSE OBS package build results. Some of it will also apply to other subjects.
0. Fetch this git repo, follow the setup instructions in the README.md file.
1. find (detect) issues
1.1. run 'rbk'
1.2. for noarch packages run 'noarchcheck'
1.3. run 'nachbau' once the above is all good
all of them will store their outcome in dot files in the current dir for later use.
A result of '0' usually indicates 'success' or 'reproducible' in UNIX fashion.
2. debug (track down) issues
This section assumes you found a package that produced diffs in rbk
2.1. run autoclassify
this script will need a while and report a bitmask of what non-deterministic influences had to be disabled to produce reproducible results.
If it is "1 1 1 1 1 1 1 1 1 1 1 1 99" it remained unreproducible even with all tweaks applied.
Debugging the remaining diff can still be easier and once that is fixed, autoclassify can be rerun.
If there are multiple bits left, you can debug one of them at a time by running 'rbkt' with fewer bits set.
See rbkt source for the meaning of bits.
2.2. run autoprovenance $unreproduciblefile
this script will tell you what command produced the file along with a call trace. Requires 'stracebuild' to be run before once.
2.3 other ways to find provenance
2.3.1 review diffs in .rb.buildroot.diff - e.g. .c .h .a configure Makefile and track down their provenance
2.3.2 check build log (might need to use make V=1 VERB=1)
2.3.3 grep source for strings next to diff (e.g. grep "This file was autogenerated at")
2.3.4 grep source for known troublemakers (`date, $(date, %Y, year, strftime, ctime, asctime, stat (mtime,inode,dev) readdir, os.listdir, os.walk, glob.glob, find, random, fprofile-use, -march=native, -mtune=native)
2.3.5 use difflog to see if something interesting happens related to indeterministic files
2.3.6 review relevant files (configure{,.ac}, Makefile{,.in,.am}, CMakeLists.txt, setup.py)
2.3.7 manually add strace -f
3. fix issues
Once you know exactly which code line introduced non-determinism, fixing it can be pretty straightforward.
https://salsa.debian.org/reproducible-builds/reproducible-website/tree/master/_blog/posts numbers 100+ has 2 years worth of links to a large collection of Bernhard's patches along with short summaries like "date" or "readdir" or "ASLR" that map to one of the entries in
https://github.com/bmwiedemann/theunreproduciblepackage
The workflow for fixing issues usually involves:
source $path_to_reproducibleopensuse/helperfuncs
qs # shorthand for quilt setup + cd + apply old patches
quilt new reproducible.patch
quilt edit $file
quilt diff # optional
quilt refresh
cd ..
/usr/lib/build/spec_add_patch reproducible.patch
rbk # retest
For patches that can not be upstreamed:
bcocd
cp -a ../../*.{patch,spec} .
osc vc -m "Add reproducible.patch to make package build reproducible"
osc addremove
osc status # optional
osc checkin --noservice
osc submitrequest
3.1 drop unreproducible elements
sometimes this is the simplest to get upstream because it can reduce the overall complexity of the build system
3.2 normalize unreproducible elements
sort lists (that usually varied from filesystem readdir order or ASLR)
replace build date with fixed date (e.g. SOURCE_DATE_EPOCH or ChangeLog file mtime)
3.3 fix bugs
sometimes unreproducible output is the result of actual bugs and then actual upstream bugfixes rather than downstream distribution workarounds are recommended.
https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/race has several examples
https://github.com/projectM-visualizer/projectm/pull/176
4. Do upstream patches, if possible
sometimes non-determinism will be introduced by .spec files or downstream-only patches - then a osc submitrequest is enough.
4.1 locate the upstream git repo, bug tracker or website
4.1.1 grep -e ^Url -e ^Source *.spec
4.1.2 google $package git OR contribute OR bugtracker
4.2 check if the project had activity in the last 12 months
if not, it may have moved or been orphaned. You can still submit a downstream patch.
4.3 make the patch
git clone $URL
patch -p1 < $path_to/reproducible.patch # or edit $file
git checkout -b $topic
# possibly check other contribution guidelines about "Signed-off-By" lines or where/how to submit fixes
git commit -a
# use good commit messages, that can include phrases like
Do $SOMEFIX to make builds reproducible.
See https://reproducible-builds.org/ for why this is good.
Without this patch, build results varied in `$file` with this diff:
```diff
$DIFF_EXAMPLE
```
4.4 submit the patch
4.4.1 For projects hosted on github this can use the `hub` package:
hub fork
git push $yourfork
hub pull-request
4.4.2 For projects doing reviews on gerrit, use 'git-review' package
git review -s
git commit --amend # to add a ChangeID
git review
4.4.3 For projects using phabricator use the 'arcanist' package
arc diff # FIXME this is incomplete
4.4.4 For projects using mercurial, use hg or https://github.com/felipec/git-remote-hg
4.4.5 For projects using mailing lists for patches:
if only members can post, subscribe to ML
git send-email --to "$ML_ADDR" HEAD^
check ML archive 10-30 minutes later
unsubscribe from ML once patch discussion is over
4.4.6 If projects do not specify how to contribute, but you have a git repo, try
git log $changedfile
git send-email --to "$mainauthor" HEAD^
4.5 Tell the world about your fix
using https://salsa.debian.org/reproducible-builds/reproducible-website monthly draft blog post entries to link to your fix and become famous - one patch at a time.
4.6 Track fixes for yourself
Often it takes maintainers some days or weeks to review and merge your fix, but sometimes fixes fall through cracks and then it helps to try and remind them.
Also when doing your next fix, you can re-use patches and submit message from previous submissions.
4.6.1 re-check after a day if there is work left for you to do like signing a CLA, adding ChangeLog/NEWS entries, addressing outstanding concerns (either with comments or code changes)
4.6.2 after a month, ping in your submission, especially if git history shows other patches going in
Can include polite helpful phrases like:
Is there anything I can do to help get this simple patch merged?
4.6.3 try alternative routes of communication (not all at the same time): IRC, email, ...
4.6.4 after 6 months, if project appears orphaned:
4.6.4.1 look for active forks in github -> Insights -> Network and ask there
4.6.4.2 propose the package for inclusion in https://github.com/distropatches/
4.6.4.3 adopt it: fork it and collect patches from others and contact previous authors and downstreams
4.6.4.4 submit the patch downstream in a distribution