Going through the unverified contracts during migration

[kuzdogan]

When moving from the old infra to GCP, we had to re-verify all contracts on the GCP instance. This is because the old infra only had files (after the incident, we had the backups) and to have the DB data, we need to compile them all and verify.

In the end we verified +5M contracts, this is the final sync table, ordered by not_synced, totals on the last row:

Table

chain_id	synced	not_synced	total_sync_status
42	0	45231	45231
1	684901	35830	720731
5	151834	942	152776
69	10963	684	11647
42161	505612	240	505852
11155111	279989	124	280113
8453	688097	115	688212
5700	69	106	175
84531	49732	98	49830
56	10665	96	10761
4	124145	95	124240
80001	207251	71	207322
137	89574	56	89630
28	364	50	414
3	76503	47	76550
10243	16	47	63
10	1923456	36	1923492
1149	0	32	32
42220	3727	24	3751
42261	215	21	236
43113	16160	21	16181
100	8244	19	8263
82	818	19	837
122	467	17	484
44787	6304	17	6321
83	930	15	945
23294	142	15	157
71402	89	15	104
1101	139	14	153
71401	262	13	275
77	1140	12	1152
0	0	11	11
11155420	12004	11	12015
9001	283	10	293
7700	862	9	871
8217	19	9	28
202401	0	9	9
10200	257	8	265
534352	117	8	125
42262	77	7	84
43114	15937	7	15944
2222	1582	7	1589
7672	137	6	143
13337	321	6	327
106	115	6	121
369	130902	5	130907
97	12012	5	12017
1115	712	5	717
41	987	5	992
420	5235	4	5239
888	66	4	70
288	416	4	420
17000	11339	4	11343
5000	139	3	142
51	41	3	44
40	1444	3	1447
1313161555	166	2	168
9996	126	2	128
23295	189	2	191
314	74	2	76
62621	33	2	35
666666666	58	2	60
1116	3698	2	3700
7701	172	2	174
9977	0	2	2
1313161554	1101	2	1103
999	55	1	56
252	32	1	33
295	7	1	8
11297108099	547	1	548
1285	96	1	97
7001	247	1	248
11297108109	196	1	197
9000	159	1	160
57000	6	1	7
1287	245	1	246
2037	33	1	34
421613	20583	1	20584
421611	4212	1	4213
30	26	1	27
53935	55	1	56
73799	122	1	123
1433	0	1	1
2522	6	0	6
4000	3	0	3
8	12	0	12
37714555429	36	0	36
534	11	0	11
11235	3	0	3
19	27	0	27
42170	59	0	59
33101	38	0	38
1284	130	0	130
200901	10	0	10
32769	49	0	49
44	3	0	3
335	56	0	56
6321	1	0	1
19011	2	0	2
255	10	0	10
1291	1	0	1
710420	46	0	46
7000	48	0	48
13381	2	0	2
6119	36	0	36
25	138	0	138
648	25	0	25
246	1	0	1
2358	17	0	17
22776	67	0	67
61	43	0	43
78431	3	0	3
39797	5	0	5
999999999	143	0	143
4337	199	0	199
1339	2	0	2
11111	73	0	73
2000	29	0	29
57	65	0	65
59141	3	0	3
5003	1	0	1
222000222	426	0	426
49797	22	0	22
1127469	5	0	5
12898	3	0	3
1001	71	0	71
7777777	122	0	122
35441	9	0	9
570	25	0	25
62320	480	0	480
841	1	0	1
534351	159	0	159
34443	102	0	102
35443	10	0	10
420666	28	0	28
641230	12	0	12
4157	10	0	10
16180	1	0	1
99	15	0	15
167006	7	0	7
660279	3	0	3
333000333	57	0	57
314159	22	0	22
1030	28	0	28
10850	5	0	5
78430	4	0	4
919	87	0	87
80002	1355	0	1355
10849	5	0	5
100010	3	0	3
103090	1	0	1
356256156	6	0	6
62831	1	0	1
7668	70	0	70
111000	2	0	2
432204	23	0	23
212	18	0	18
192837465	15	0	15
2021	1	0	1
84532	57937	0	57937
690	11	0	11
59144	7	0	7
200810	37	0	37
2221	22	0	22
54211	1	0	1
167005	14	0	14
336	13	0	13
4200	4	0	4
10242	25	0	25
17069	10	0	10
421614	1756	0	1756
250	142	0	142
	5133646	84318	5217964

Bear in mind there are chains like Ethereum Goerli (5) that are deprecated ie. not supported. Ex-Kovan, now Lukso (42) will not be added as this is an unresolved issue about the chainId collusion.

We need to look at why these contracts don't verify. Some give "There is not contract at 0x...` which means the contract might have been destroyed. Some give "bytecodes don't match" which is weird because how did they get verified in the first place? Some give compiler errors which is also unexpected.

We should categorize these error and go through them. I believe we'll gain a lot of insights and potentially find some bugs with this. The difficult thing is there are a lot of contracts and we have to recognize some patterns, group the contracts and execute and report systematically.

[kuzdogan]

Sharing findings here:

I categorized the failed verifications under the following errors. I am going to share the number of errors for the Mainnet contracts only because with a total of 32k contracts, it's the majority of unverified contracts.

Metadata file not found

Error message: "Metadata file not found. Did you include \"metadata.json\"?"
Contracts in Mainnet: 17

This case is quite unexpected as they indeed don't have any metadata file in the current FS as well as the backups. This is just lost data, it seems.

Abstract contract

Error message: "The compiled contract bytecode is \"0x\". Are you trying to verify an abstract contract?"
Contracts in Mainnet: 8

I fail to understand how these contracts were verified. The compilationTargets are indeed abstract contracts.

Cannot insert contract

Error message: "cannot insert verified_contract address=0x8b2AA451F98cc7eA61f5c462c94eF76CD5F131Cf chainId=369\nerror: unsupported Unicode escape sequence"
Contracts in Mainnet: 7

These contracts seem to contain a \u0000 unicode escape sequence (NULL) in their sources in compiled_contracts. Postgres does not allow this character in jsonb columns.

Cannot generate contract artifacts

Error message: "cannot generate contract artifacts address=0xbdB2691fa62707daf356A6470325C7d7118388E1 chainId=369"
Contracts in Mainnet: 4

Found that these contracts have a different auxdata at the end of their bytecode and the verification process stops incorrectly. Fixed by #1594

Metadata match but not bytecodes

Error message: "contract's metadata hashes match but not the bytecodes. You should add all the files input to the compiler during compilation and remove all others"
Contracts in Mainnet: 5

Turns out they can be verified when they are verified with Import from Etherscan but the verification can't be reproduced via the files saved in the Sourcify repo. This is expected as the bug says exactly this, the info in the metadata does not reproduce the bytecode found onchain.

Selfdesctruct'ed contracts

These contracts return the below error. Upon inspecting couple of them I see the general pattern that these are the SELFDESTRUCTed contracts. I guess there isn't much we can do about it.

Error message: "Chain #1 does not have a contract deployed at 0x6fb34B28Dbb558E3232f0e85a7Cf0537037Ef344"
Contracts in Mainnet: 148

Bytecodes don't match

Error message: "The deployed and recompiled bytecode don't match."
Contracts in Mainnet: 30956

This is the majority of the contracts.

• One case I found is similar bytecodes but likely different because of the AST changes. IDK how these were verified in the first place
• Most other ones on Mainnet are proxies. Again, a question how these got verified in the first place.

Compiler Errors

Error message: "Compiler error...(full compiler error output)"
Contracts in Mainnet: 212

Here there can be multiple errors for each contract so it's difficult to get a full number.

These are also weird because it leaves me thinking how did they compile and get verified in the first place? Maybe an issue with the compiler binary platform?

There were couple different errors:

• "File outside of allowed directories.\nimport "@openzeppelin..."
• "Expected pragma, import directive or contract/interface/library/struct/enum definition"
• "Definition of base has to precede definition of derived contract"
  • These cases seem to be all 0.5.17 contracts that have a "multipart file" pattern in Etherscan. They are mostly CEther CErc20 something contracts. E.g. https://etherscan.io/address/0x7b34EbB043260D3802F151F3b9f673F6FDb2E5a0/contract#code. This one is another example with 0.8.9 0x309f49bD32B1098F4dF20a9DD51954e8680d77B1
  • However these contracts are verified in Etherscan and Blockscout.
• "Undeclared identifier"

I'll try to debug these a bit to see if I can catch a pattern.