Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] LE12 32bit arm RPI4 mariadb connection is failing (Bad handshake) #8187

Closed
the-dreamer opened this issue Oct 4, 2023 · 23 comments · Fixed by #8236
Closed

[BUG] LE12 32bit arm RPI4 mariadb connection is failing (Bad handshake) #8187

the-dreamer opened this issue Oct 4, 2023 · 23 comments · Fixed by #8236
Labels
LE 12.0 RPi

Comments

@the-dreamer
Copy link

Describe the bug

with a rpi4 build the mariadb connection is failing with bad handshake and later blocked by mariadb server due flooding connection retries.

Same code base is running on x86 system without issue.

LE11 was also running on RPI4 without issue.

same config is used for both systems. (advancedconfig.xml)

How to reproduce

Steps to reproduce the behavior:

  • use master, i used from 27th Sept.
  • have advanced settings with sql settings
    connection to sql server is failing

Information

  • LibreELEC Version: master from 27th sept
  • RPI 4

Log file

relevant part: Debug output has no more info
.....
2023-10-04 11:35:39.238 T:1324 error : Unable to open database: MyMusic71 [1043](Bad handshake)
2023-10-04 11:35:39.239 T:1324 error : Unable to open database: MyMusic70 [1043](Bad handshake)
2023-10-04 11:35:39.240 T:1324 error : Unable to open database: MyMusic69 [1043](Bad handshake)
2023-10-04 11:35:39.241 T:1324 error : Unable to open database: MyMusic68 [1043](Bad handshake)
2023-10-04 11:35:39.242 T:1324 error : Unable to open database: MyMusic67 [1043](Bad handshake)
2023-10-04 11:35:39.243 T:1324 error : Unable to open database: MyMusic66 [1043](Bad handshake)
2023-10-04 11:35:39.244 T:1324 error : Unable to open database: MyMusic65 [1043](Bad handshake)
2023-10-04 11:35:39.245 T:1324 error : Unable to open database: MyMusic64 [1043](Bad handshake)
2023-10-04 11:35:39.246 T:1324 error : Unable to open database: MyMusic63 [1043](Bad handshake)
2023-10-04 11:35:39.247 T:1324 error : Unable to open database: MyMusic62 [1043](Bad handshake)
2023-10-04 11:35:39.248 T:1324 error : Unable to open database: MyMusic61 [1043](Bad handshake)
2023-10-04 11:35:39.252 T:1324 error : Unable to open database: MyMusic60 [1043](Bad handshake)
2023-10-04 11:35:39.253 T:1324 error : Unable to open database: MyMusic59 [1043](Bad handshake)
2023-10-04 11:35:39.254 T:1324 error : Unable to open database: MyMusic58 [1043](Bad handshake)
2023-10-04 11:35:39.256 T:1324 error : Unable to open database: MyMusic57 [1043](Bad handshake)
2023-10-04 11:35:39.257 T:1324 error : Unable to open database: MyMusic56 [1043](Bad handshake)
2023-10-04 11:35:39.258 T:1324 error : Unable to open database: MyMusic55 [1043](Bad handshake)
2023-10-04 11:35:39.263 T:1324 error : Unable to open database: MyMusic54 [1043](Bad handshake)
2023-10-04 11:35:39.264 T:1324 error : Unable to open database: MyMusic53 [1043](Bad handshake)
2023-10-04 11:35:39.274 T:1324 error : Unable to open database: MyMusic52 [1043](Bad handshake)
2023-10-04 11:35:39.275 T:1324 error : Unable to open database: MyMusic51 [1043](Bad handshake)
2023-10-04 11:35:39.277 T:1324 error : Unable to open database: MyMusic50 [1043](Bad handshake)
2023-10-04 11:35:39.279 T:1324 error : Unable to open database: MyMusic49 [1043](Bad handshake)
2023-10-04 11:35:39.280 T:1324 error : Unable to open database: MyMusic48 [1043](Bad handshake)
2023-10-04 11:35:39.281 T:1324 error : Unable to open database: MyMusic47 [1043](Bad handshake)
2023-10-04 11:35:39.282 T:1324 error : Unable to open database: MyMusic46 [1043](Bad handshake)
2023-10-04 11:35:39.283 T:1324 error : Unable to open database: MyMusic45 [1043](Bad handshake)
2023-10-04 11:35:39.285 T:1324 error : Unable to open database: MyMusic44 [1043](Bad handshake)
2023-10-04 11:35:39.286 T:1324 error : Unable to open database: MyMusic43 [1043](Bad handshake)
2023-10-04 11:35:39.287 T:1324 error : Unable to open database: MyMusic42 [1043](Bad handshake)
2023-10-04 11:35:39.288 T:1324 error : Unable to open database: MyMusic41 [1043](Bad handshake)
2023-10-04 11:35:39.289 T:1324 error : Unable to open database: MyMusic40 [1043](Bad handshake)
2023-10-04 11:35:39.290 T:1324 error : Unable to open database: MyMusic39 [1043](Bad handshake)
2023-10-04 11:35:39.292 T:1324 error : Unable to open database: MyMusic38 [1043](Bad handshake)
2023-10-04 11:35:39.293 T:1324 error : Unable to open database: MyMusic37 [1043](Bad handshake)
2023-10-04 11:35:39.294 T:1324 error : Unable to open database: MyMusic36 [1043](Bad handshake)
2023-10-04 11:35:39.295 T:1324 error : Unable to open database: MyMusic35 [1043](Bad handshake)
2023-10-04 11:35:39.296 T:1324 error : Unable to open database: MyMusic34 [1043](Bad handshake)
2023-10-04 11:35:39.297 T:1324 error : Unable to open database: MyMusic33 [1043](Bad handshake)
2023-10-04 11:35:39.298 T:1324 error : Unable to open database: MyMusic32 [1043](Bad handshake)
2023-10-04 11:35:39.300 T:1324 error : Unable to open database: MyMusic82 [1043](Bad handshake)
2023-10-04 11:35:39.300 T:1324 error : Unable to create new database

on server side:

2023-10-04 11:35:39 4775679 [Warning] IP address '192.168.10.118' could not be resolved: Name or service not known
2023-10-04 11:35:39 4775679 [Warning] Aborted connection 4775679 to db: 'unconnected' user: 'unauthenticated' host: '192.168.10.118' (This connection closed normally without authentication)
2023-10-04 11:35:39 4775680 [Warning] Aborted connection 4775680 to db: 'unconnected' user: 'unauthenticated' host: '192.168.10.118' (This connection closed normally without authentication)
2023-10-04 11:35:39 4775681 [Warning] Aborted connection 4775681 to db: 'unconnected' user: 'unauthenticated' host: '192.168.10.118' (This connection closed normally without authentication)
2023-10-04 11:35:39 4775682 [Warning] Aborted connection 4775682 to db: 'unconnected' user: 'unauthenticated' host: '192.168.10.118' (This connection closed normally without authentication)
2023-10-04 11:35:39 4775683 [Warning] Aborted connection 4775683 to db: 'unconnected' user: 'unauthenticated' host: '192.168.10.118' (This connection closed normally without authentication)
2023-10-04 11:35:39 4775684 [Warning] Aborted connection 4775684 to db: 'unconnected' user: 'unauthenticated' host: '192.168.10.118' (This connection closed normally without authentication)
......

x86 log relevant log:

2023-10-04 11:40:41.216 T:1051 info : Running database version Addons33
2023-10-04 11:40:41.219 T:1051 info : Running database version ViewModes6
2023-10-04 11:40:41.346 T:1051 info : Running database version Textures13
2023-10-04 11:40:41.470 T:1051 info : MYSQL: Connected to version 10.5.21-MariaDB
2023-10-04 11:40:41.479 T:1051 info : Running database version MyMusic82
2023-10-04 11:40:41.498 T:1051 info : Running database version MyVideos122
2023-10-04 11:40:41.515 T:1051 info : creating version table
2023-10-04 11:40:41.515 T:1051 info : CreateTables: Creating PVR database tables
2023-10-04 11:40:41.517 T:1051 info : CreateAnalytics: Creating PVR database indices
2023-10-04 11:40:41.544 T:1051 info : creating version table
2023-10-04 11:40:41.545 T:1051 info : Creating EPG database tables

Context
@heitbaum
Copy link
Contributor

heitbaum commented Oct 4, 2023

There has been some updates in the mariadb-connector-c. Here are the changes and dates.

nothing obvious for 64bit RPi
if we can pinpoint the “version it occurred in” as a start.

@the-dreamer
Copy link
Author

when did it work last with LE12?
-> No clue, sorry, i did start directly from a LE 11 build from mid march.

nothing obvious for 64bit RPi
-> it was necessary to have the commit with the 32bit patch inside the build. otherwise i had build errors. I am using a 32bit build? The build host is a 64bit machine.

@heitbaum
Copy link
Contributor

heitbaum commented Oct 4, 2023 •
edited
Loading

Check your compile line? For LE12 RPi4 it should be PROJECT=RPi ARCH=aarch64 DEVICE=RPi4 make image for LE11 is should be ARCH=arm

if 3.3.7 is indeed the culprit to your bad handshake on a 32-bit build - test by recompiling ARCH=arm and reverting the patch and the bump to 3.3.7 and can go back to the mariadb team with further info.

point of note - LE11 is using 3.3.4 - so if it is indeed the mariadb-connector-c then it is somewhere between 3.3.4 and 3.3.7.

PKG_VERSION="3.3.4"
PKG_SHA256="ea6a23850d6a2f6f2e0d9e9fdb7d94fe905a4317f73842272cf121ed25903e1f"

@the-dreamer
Copy link
Author

Indeed it was wrong. Wiki has no LE12 infos :-(

i am rebuilding rpi4 with aarch64 now. Can take some time. i will let you know if it is working.

@heitbaum
Copy link
Contributor

heitbaum commented Oct 4, 2023

Indeed it was wrong. Wiki has no LE12 infos :-(

Very true - they will catchup once LE12 becomes stable. Not all of the 64bit ARM chips are being built with a 64bit userland yet - still a state of flux - but RPi4/5 and Allwinner are.

@the-dreamer
Copy link
Author

the-dreamer commented Oct 5, 2023 •
edited
Loading

Image is built, but upgrade is not working. Also the nightly LE12 is not updating. I was searching for log files, but i cannot see anywhere where a failed update is logged. On Screen there is nothing but the LE splash. The boot take longer when the system is trying to update. But without reboot it is starting kodi with old version.

Btw, i am using at least an aarch64 kernel. But i assume the userland is 32bit and with aarch64 this is changed.

LibreELEC (community): devel-20230927072201-f584472 (RPi4.arm)
LibreELECpi4:~ # uname -a
Linux LibreELECpi4 6.1.45 #1 SMP Wed Sep 27 11:45:01 CEST 2023 aarch64 GNU/Linux

LibreELECpi4:~ # dmesg
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 6.1.45 (vncuser@pluto) (aarch64-none-elf-gcc-13.2.0 (GCC) 13.2.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Sep 27 11:45:01 CEST 2023
[ 0.000000] random: crng init done
[ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.4
[ 0.000000] efi: UEFI not found.
...

EDIT: Okay i got it with touch /storage/.update/.nocompat

mariadb is working again. So 32bit userland has a problem. aarch64 is working.

Now i need to find out how to switch the kodi repo to aarch64 because pvr is not working...

@heitbaum
Copy link
Contributor

heitbaum commented Oct 5, 2023

The ARMv8 LE12 packages are built for both arm and aarch64 - though you will need to clear out the arm addons and addon cache for your new aarch64 userland image.

@heitbaum heitbaum changed the title [BUG] LE12 RPI4 mariadb connection is failing (Bad handshake) [BUG] LE12 32bit arm RPI4 mariadb connection is failing (Bad handshake) Oct 5, 2023
@misanthropos
Copy link

It recompiled current master 58c21f3 with mariadb-connector-3.4, 3.5, 3,6 - at least with 3.4 I can still see data from audiodb wheras with later versions that wont work either.

It looks like the conversion of the DB seems to be broken with mariadb-connector-3.3.4:

SQL: [kodivideo122] Undefined MySQL error: Code (1419)
                                                   Query: CREATE TRIGGER delete_movie AFTER DELETE ON movie FOR EACH ROW BEGIN DELE
TE FROM genre_link WHERE media_id=old.idMovie AND media_type='movie'; DELETE FROM actor_link WHERE media_id=old.idMovie AND media_t
ype='movie'; DELETE FROM director_link WHERE media_id=old.idMovie AND media_type='movie'; DELETE FROM studio_link WHERE media_id=ol
d.idMovie AND media_type='movie'; DELETE FROM country_link WHERE media_id=old.idMovie AND media_type='movie'; DELETE FROM writer_li
nk WHERE media_id=old.idMovie AND media_type='movie'; DELETE FROM movielinktvshow WHERE idMovie=old.idMovie; DELETE FROM art WHERE 
media_id=old.idMovie AND media_type='movie'; DELETE FROM tag_link WHERE media_id=old.idMovie AND media_type='movie'; DELETE FROM ra
ting WHERE media_id=old.idMovie AND media_type='movie'; DELETE FROM uniqueid WHERE media_id=old.idMovie AND media_type='movie'; END

                                                   
2023-10-09 15:47:16.519 T:1193    error <general>: Exception updating database kodivideo122 from version 121 to 122

@the-dreamer
Copy link
Author

the-dreamer commented Oct 9, 2023 •
edited
Loading

From my side i can tell, conversion is no problem on x86_64 and rpi with 64bit userland.

btw: your issue is a different one as described in this issue. This ticket is describing connection issues to mariadb when using 32 bit userland on at least RPI platform.

PS: and at least, you left complete no information about your system.

@misanthropos
Copy link

@the-dreamer PROJECT=RPi ARCH=arm DEVICE=RPi4

I have the exact same problem as you do and for that reason I tried different versions of mariadb-connector-c, in the hopes it would remedy the Bad Handshake problem.

@the-dreamer
Copy link
Author

the-dreamer commented Oct 9, 2023 •
edited
Loading

As for LE12 RPI4 is using a 64bit userland now, 32bit is not necessary anymore and i see no reason to stick on 32bit. I switched to 64bit and everythink is working now. As you can read above you, the build command is wrong for LE12.

The handshake issue is a 32bit issue.

@misanthropos
Copy link

Sorry, I missed that part. Kodi runs just fine now. However: After compiling LE12 with aarch64 no videodb122 was created/used (which has been done before). videodb121 is in use. Because of that I am not sure it the Bad Handshake issue has been handled by aarch64.

@the-dreamer
Copy link
Author

Log file for it? as i can see for the x86_64 db122 is used. I have not looked in log file for aarch64 because i cannot imagine there will be a difference, because it is part of kodi itself.

@misanthropos
Copy link

Oh my... I accidentally flashed an old working version from August without noticing. The upgrade process did not run through. Because of the failed trigger SQL statement.
kodi.log

@the-dreamer
Copy link
Author

I still belive you have another issue. My x86_64 intel system did not has any issue. RPI4 had issues because of 32bit userland. With 64bit userland i have no issues.

But indeed, my RPI4 did not update the DB because the x86_64 system did it already. maybe i will delete db 122 and try. but anyway, it would be another issue.

what is about your sql server logs? any hint?

@misanthropos
Copy link

It is clearly an SQL problem. This log has debug database enabled. I believe I had such a problem. I think I fixed it (in the past) by manually executing a modified sql statement.
kodi.log
Trying to rebuild the movie db does do something but kodi displays just an empty screen.
I will file an issue.

@wagnerch
Copy link
Contributor

@heitbaum I was able to track this down to the 3.3.7 bump of mariadb-connector-c, I reverted it to 3.3.6 and the patch for fixing the 32-bit build, and the problem seems to be fixed. Unclear if it is the "build fix" patch, since it won't built without it.

@heitbaum
Copy link
Contributor

@wagnerch - thank you for isolating it. I thought that 3.3.6 to 3.3.7 was the issue. As you have now confirmed this - I will look at it and see what we can do / upstream the issue.

So just confirming
A - "3.3.6 on 32bit connects without error"

B - "3.3.7 on 32bit connects WITH error"

C - "3.3.7 on 64bit connects without error"

@wagnerch
Copy link
Contributor

wagnerch commented Oct 19, 2023 •
edited
Loading

@heitbaum Correct, and it appears the issue is the patch. What I did now is rebuilt with 3.3.7, reverted the build fix, and applied this patch to revert the cflag changes for warnings/errors:
http://ix.io/4JrJ

Here are the changes that I am referring to, this isn't runtime tested on LE but it is on CE w/ 32-bit builds.
master...wagnerch:LibreELEC.tv:mariadb-connector-32

@wagnerch
Copy link
Contributor

Can confirm this bit of the patch is causing the issue. size_t is 32-bit on "ARCH=arm", and 64-bit on "ARCH=aarch64".

The other bits with the union initialization, no problems with that applied.

On aarch64:
//CHECK_TYPE_SIZE: sizeof(size_t)
SIZEOF_SIZE_T:INTERNAL=8

On armhf (armv8):
//CHECK_TYPE_SIZE: sizeof(size_t)
SIZEOF_SIZE_T:INTERNAL=4

diff --git a/libmariadb/mariadb_lib.c b/libmariadb/mariadb_lib.c
index 6b912d6f..32e66bea 100644
--- a/libmariadb/mariadb_lib.c
+++ b/libmariadb/mariadb_lib.c
@@ -117,7 +117,7 @@ extern int mthd_stmt_fetch_to_bind(MYSQL_STMT *stmt, unsigned char *row);
 extern int mthd_stmt_read_all_rows(MYSQL_STMT *stmt);
 extern void mthd_stmt_flush_unbuffered(MYSQL_STMT *stmt);
 extern my_bool _mariadb_read_options(MYSQL *mysql, const char *dir, const char *config_file, const char *group, unsigned int recursion);
-extern unsigned char *mysql_net_store_length(unsigned char *packet, size_t length);
+extern unsigned char *mysql_net_store_length(unsigned char *packet, ulonglong length);

 extern void
 my_context_install_suspend_resume_hook(struct mysql_async_context *b,
diff --git a/libmariadb/mariadb_stmt.c b/libmariadb/mariadb_stmt.c
index 3f610669..07cf6b16 100644
--- a/libmariadb/mariadb_stmt.c
+++ b/libmariadb/mariadb_stmt.c
@@ -481,7 +481,7 @@ MYSQL_RES *_mysql_stmt_use_result(MYSQL_STMT *stmt)
   return(NULL);
 }

-unsigned char *mysql_net_store_length(unsigned char *packet, size_t length)
+unsigned char *mysql_net_store_length(unsigned char *packet, ulonglong length)
 {
   if (length < (unsigned long long) L64(251)) {
     *packet = (unsigned char) length;

@wagnerch
Copy link
Contributor

Also wanted to point out that ulonglong is 64-bit on both platforms, so effectively this patch does nothing for aarch64, which is why it's unaffected. I am curious what the compiler was doing on that bit shift in int8store, because it is clearly wrong for a 32-bit platform. I am assuming callers are still declaring size_t and passing in 32-bit integers, so maybe the compiler is casting implicitly or who knows.

@wagnerch
Copy link
Contributor

I have a PR ready that should fix it, I'll push it over and you can take a look. The fix is pretty simple, just cast length to ulonglong before calling int8store.

@wagnerch wagnerch mentioned this issue Oct 21, 2023
Merged
@heitbaum
Copy link
Contributor

Fixed by #8236

@heitbaum heitbaum linked a pull request Oct 21, 2023 that will close this issue
mariadb-connector-c: Resolve bad handshake on 32-bit platforms #8236
Merged
BlackDex added a commit to BlackDex/rust-musl that referenced this issue Nov 8, 2023
@BlackDex
Down/Upgrade libraries
896256b 
- Downgrade MariaDB to v3.3.5 again.
  See: LibreELEC/LibreELEC.tv#8187
  And: mariadb-corporation/mariadb-connector-c@8320f0d
  And: https://jira.mariadb.org/projects/CONC/issues/CONC-668
- Upgrade SQLite to v3.44.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
LE 12.0 RPi
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

mariadb-connector-c: Resolve bad handshake on 32-bit platforms wagnerch/LibreELEC.tv
4 participants
@misanthropos @the-dreamer @heitbaum @wagnerch