TypeError: must be str, not bytes

[leematthewshome]

Versions

• Python: 3.5.3 (running in virtualenv and in root)
  (seems to be running Python 3.7.1 on Android based on the logs)
• OS: Ubuntu 17:04
• Kivy: 1.9.1
• Cython: 0.28.6
• Andoid 7.0 (on Samsung Galaxy S6)

Description

I have compiled a very basic helloWorld app as below. The app works fine on Ubuntu and also compiles fine into apk using buildozer.

__version__ = '1.0'
from kivy.app import App
from kivy.uix.button import Button
 
class Hello(App):
    def build(self):
        btn = Button(text='Hello World')
        return  btn
 
Hello().run()

When I run on Android the app immediately closes before displaying the hello world screen. After working through logs produced with adb logcat I see the following...

06-25 19:48:52.272  2100  2118 I python  :  Traceback (most recent call last):
06-25 19:48:52.272  2100  2118 I python  :    File "/home/lee/KivyProjects/helloWorld/.buildozer/android/app/main.py", line 3, in <module>
06-25 19:48:52.273  2100  2118 I python  :    File "/home/lee/KivyProjects/helloWorld/.buildozer/android/platform/build/build/python-installs/helloWorld/kivy/app.py", line 319, in <module>
06-25 19:48:52.273  2100  2118 I python  :    File "/home/lee/KivyProjects/helloWorld/.buildozer/android/platform/build/build/python-installs/helloWorld/kivy/base.py", line 26, in <module>
06-25 19:48:52.274  2100  2118 I python  :    File "/home/lee/KivyProjects/helloWorld/.buildozer/android/platform/build/build/python-installs/helloWorld/kivy/clock.py", line 410, in <module>
06-25 19:48:52.274  2100  2118 I python  :    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
06-25 19:48:52.274  2100  2118 I python  :    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
06-25 19:48:52.275  2100  2118 I python  :    File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
06-25 19:48:52.275  2100  2118 I python  :    File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
06-25 19:48:52.276  2100  2118 I python  :    File "/home/lee/KivyProjects/helloWorld/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/util.py", line 73, in <module>
06-25 19:48:52.276  2100  2118 I python  :    File "/home/lee/KivyProjects/helloWorld/.buildozer/android/platform/build/build/python-installs/helloWorld/android/__init__.py", line 8, in <module>
06-25 19:48:52.277  2100  2118 I python  :    File "android/_android.pyx", line 162, in init android._android
06-25 19:48:52.277  2100  2118 I python  :  TypeError: must be str, not bytes
06-25 19:48:52.277  2100  2118 I python  : Python for android ended.

This appears to be the same issue raised as 1691 under the title "unicode error during startup (python3, numpy, opencv) - patch included #1691". If I am reading this patch correctly the current file _android.pyx should have the following changes

removed this line
python_act = autoclass(JAVA_NAMESPACE + u'.PythonActivity')

added these lines
java_namespace = JAVA_NAMESPACE.decode() if isinstance(JAVA_NAMESPACE, bytes) else JAVA_NAMESPACE
python_act = autoclass(java_namespace + u'.PythonActivity')

Yet when I look at the current code on Github for _android.pyx I do not see these changes applied. Has this fix somehow been removed from the current version? How can i apply this fix to my system?

[AndreMiras]

Yes it looks like your are outdated. You seem to be using buildozer. Can you share the buildozer.spec?
And also I think it was something with cython, but can't remember what 😕

[leematthewshome]

Thanks. The current code on Github seems outdated to me...as the patch mentioned in 1691 doesnt seem to be applied. I expect that is feeding the pip and apt repos.
Spec below...

[app]

# (str) Title of your application
title = Hello world

# (str) Package name
package.name = helloWorld

# (str) Package domain (needed for android/ios packaging)
package.domain = org.test

# (str) Source code where the main.py live
source.dir = .

# (list) Source files to include (let empty to include all the files)
source.include_exts = py,png,jpg,kv,atlas

# (str) Application versioning (method 1)
version = 0.1

# (list) Application requirements
# comma separated e.g. requirements = sqlite3,kivy
requirements = python3,kivy

# (str) Custom source folders for requirements
# Sets custom source for any requirements with recipes
# requirements.source.kivy = ../../kivy


# (str) Supported orientation (one of landscape, sensorLandscape, portrait or all)
orientation = portrait


# change the major version of python used by the app
osx.python_version = 3

# Kivy version to use
osx.kivy_version = 1.9.1

#
# Android specific
#

# (bool) Indicate if the application should be fullscreen or not
fullscreen = 0


# (str) The Android arch to build for, choices: armeabi-v7a, arm64-v8a, x86
android.arch = armeabi-v7a

# Alternately, specify the URL and branch of a git checkout:
ios.kivy_ios_url = https://github.com/kivy/kivy-ios
ios.kivy_ios_branch = master

# Or specify URL and branch
ios.ios_deploy_url = https://github.com/phonegap/ios-deploy
ios.ios_deploy_branch = 1.7.0

[buildozer]

# (int) Log level (0 = error only, 1 = info, 2 = debug (with command output))
log_level = 1

# (int) Display warning if buildozer is run as root (0 = False, 1 = True)
warn_on_root = 1

[aswinmurali-io]

I am also facing the same issue. What he said is correct. The patch isn't applied in the master branch. Please update it

[ghost]

I think we fixed this differently, with JAVA_NAMESPACE now defined in android/__init__.py as java_ns = u'org.kivy.android'.

I can't see anything in that buildozer config pointing at the latest develop version of p4a, maybe that is hte issue? I'm not sure if it's fixed in the latest stable release of p4a already, maybe @AndreMiras knows

[aswinmurali-io]

Ok.... so I will explain. I did set p4a.branch=master when I used buildozer to run the build first time this error popped up. After searching on the internet I found the same problem in #1691. So what I did is reran the buildozer android debug command, obviously, the same error was shown. Now I when inside the .buildozer folder and navigated to python-for-android source downloaded by buildozer and then navigated to _android.pyx file and manually did the patch myself and also I deleted the build folder so that buildozer does a rebuild with the new patched python-for-android and it is now working properly. I didn't get any side effects by applying this patch.

[leematthewshome]

Thanks @SyncCodes.
That helps me understand a little more about the kivy build process. So buildozer fetches p4a when you compile the kivy project.
I found two copies of _android.pyx under my helloWorld app folder, under the .buildozer subfolder. It sounds like one of the paths this was found in should be deleted. Can you clarify which one?

I tried editing both to apply the patch but when I deployed to my phone I still got the same error. So perhaps it didnt overwrite the old APK on my phone. I will try changing the project name and see if that helps.

[ghost]

@SyncCodes I am importing android with python3 with p4a master though and have no issues, and there is no p4a.branch=master in your buildozer spec. If you want us to look into this, can you provide a minimal app example & spec so we can reproduce and fix it?

The thing is that this patch shouldn't be required as the code is set up right now, so it doesn't make too much sense to just blindly apply it without understanding what is going on. I am still suspecting you guys may have accidentally used an older p4a version without realizing, and we should rule that out (or find out whatever else is going on) before just committing unnecessary conditionals into the code

[leematthewshome]

Actually @SyncCodes said he/she did put p4a.branch=master into the buildozer.spec file and it still resulted in the same error. Regardless, I created a new app folder named helloTest1 and pasted in two files. These are main.py and buildozer.spec. The contents are listed below. I am compiling now and will let you know how it goes. Note that the spec file now contains p4a.branch=master

main.py

__version__ = '1.0'
from kivy.app import App
from kivy.uix.button import Button
 
class Hello(App):
    def build(self):
        btn = Button(text='Hello World')
        return  btn
 
Hello().run()

buildozer.spec

[app]

# (str) Title of your application
title = Hello Test V1

# (str) Package name
package.name = helloTest1

# (str) Package domain (needed for android/ios packaging)
package.domain = thisRobotAI.com

# (str) Source code where the main.py live
source.dir = .

# (list) Source files to include (let empty to include all the files)
source.include_exts = py,png,jpg,kv,atlas

# (str) Application versioning (method 1)
version = 0.1

# comma separated e.g. requirements = sqlite3,kivy
requirements = python3,kivy

# (str) Supported orientation (one of landscape, sensorLandscape, portrait or all)
orientation = portrait

# change the major version of python used by the app
osx.python_version = 3

# Kivy version to use
osx.kivy_version = 1.9.1

# (bool) Indicate if the application should be fullscreen or not
fullscreen = 0

# (list) Permissions
#android.permissions = INTERNET

# (str) python-for-android branch to use, defaults to master
p4a.branch = master

# (str) The Android arch to build for, choices: armeabi-v7a, arm64-v8a, x86
android.arch = armeabi-v7a

# (str) Path to a custom kivy-ios folder
#ios.kivy_ios_dir = ../kivy-ios
# Alternately, specify the URL and branch of a git checkout:
ios.kivy_ios_url = https://github.com/kivy/kivy-ios
ios.kivy_ios_branch = master

# Another platform dependency: ios-deploy
# Uncomment to use a custom checkout
#ios.ios_deploy_dir = ../ios_deploy
# Or specify URL and branch
ios.ios_deploy_url = https://github.com/phonegap/ios-deploy
ios.ios_deploy_branch = 1.7.0

[buildozer]

# (int) Log level (0 = error only, 1 = info, 2 = debug (with command output))
log_level = 1

# (int) Display warning if buildozer is run as root (0 = False, 1 = True)
warn_on_root = 1

[ghost]

@leematthewshome for what it's worth, if that p4a.branch=master line is put in later you may still be using the previous old p4a unless .buildozer is manually wiped beforehand. it's unfortunately a buildozer limitation many people run into

[leematthewshome]

Results unfortunately as before....

06-26 19:41:12.449 28669 28688 I python  : [INFO   ] [Kivy        ] v1.11.0
06-26 19:41:12.449 28669 28688 I python  : [INFO   ] [Kivy        ] Installed at "/data/user/0/thisrobotai.com.hellotest1/files/app/_python_bundle/site-packages/kivy/__init__.pyc"
06-26 19:41:12.449 28669 28688 I python  : [INFO   ] [Python      ] v3.7.1 (default, Jun 26 2019, 19:30:33) 
06-26 19:41:12.450 28669 28688 I python  : [INFO   ] [Python      ] Interpreter at "android_python"
06-26 19:41:12.483  3652  3940 D MdnieScenarioControlService:  packageName : thisrobotai.com.hellotest1    className : org.kivy.android.PythonActivity
06-26 19:41:12.524  3652  3681 I WindowManager_SurfaceController: Destroying surface Surface(name=Starting thisrobotai.com.hellotest1) called by com.android.server.wm.WindowStateAnimator.destroySurface:2907 com.android.server.wm.WindowStateAnimator.destroySurfaceLocked:1120 com.android.server.wm.WindowState.destroyOrSaveSurface:2654 com.android.server.wm.AppWindowToken.destroySurfaces:424 com.android.server.wm.AppWindowToken.destroySurfaces:388 com.android.server.wm.WindowStateAnimator.finishExit:686 com.android.server.wm.WindowStateAnimator.stepAnimationLocked:570 com.android.server.wm.WindowAnimator.updateWindowsLocked:439 
06-26 19:41:12.524  3100  3153 I SurfaceFlinger: id=800 Removed iellotest1 (7/10)
06-26 19:41:12.566 28669 28688 I python  :  Traceback (most recent call last):
06-26 19:41:12.566 28669 28688 I python  :    File "/home/lee/KivyProjects/helloTest1/.buildozer/android/app/main.py", line 3, in <module>
06-26 19:41:12.567 28669 28688 I python  :    File "/home/lee/KivyProjects/helloTest1/.buildozer/android/platform/build/build/python-installs/helloTest1/kivy/app.py", line 319, in <module>
06-26 19:41:12.567 28669 28688 I python  :    File "/home/lee/KivyProjects/helloTest1/.buildozer/android/platform/build/build/python-installs/helloTest1/kivy/base.py", line 26, in <module>
06-26 19:41:12.568 28669 28688 I python  :    File "/home/lee/KivyProjects/helloTest1/.buildozer/android/platform/build/build/python-installs/helloTest1/kivy/clock.py", line 410, in <module>
06-26 19:41:12.568 28669 28688 I python  :    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
06-26 19:41:12.568 28669 28688 I python  :    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
06-26 19:41:12.569 28669 28688 I python  :    File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
06-26 19:41:12.570 28669 28688 I python  :    File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
06-26 19:41:12.570 28669 28688 I python  :    File "/home/lee/KivyProjects/helloTest1/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/util.py", line 73, in <module>
06-26 19:41:12.570 28669 28688 I python  :    File "/home/lee/KivyProjects/helloTest1/.buildozer/android/platform/build/build/python-installs/helloTest1/android/__init__.py", line 8, in <module>
06-26 19:41:12.571 28669 28688 I python  :    File "android/_android.pyx", line 162, in init android._android
06-26 19:41:12.571 28669 28688 I python  :  TypeError: must be str, not bytes

[leematthewshome]

I assume by "manually wiping" .buildozer you mean deleting the .buildozer directory. That is why i actually started with a completely new folder structure with only main.py and buildozer.spec in the folder. There was no .buildozer as I did not copy the directory from the previous test, only the two files.

[aswinmurali-io]

@Jonast I did remove .buildozer. I did see a .buildozer folder both in user folder where the android build tools are stored and also one in my project directory. I deleted the .buildozer folder in my current project when i changed p4a.branch

[ghost]

oh right, I'm wondering... would p4a.branch=develop help by any chance?

[aswinmurali-io]

@Jonast what @leematthewshome did can you do the same and see if you can recreate the error? Can you test this in a fresh vm ?

[AndreMiras]

Thanks for the update guys. @Jonast I think develop and master are pretty aligned on that one. However something related to cython rings the bell, for instance if it's installed from pip2, maybe?
Could you guys which cython and share the result and also cat $(which cython) and share that too.

[leematthewshome]

OK. As per @SyncCodes I tried with the patch and it worked. Here is what i did:

• Copied the folder helloTest1 and renamed to helloTest2.
• Edited the buildozer.spec and changed the app name and desk to helloTest2 (from helloTest1)
• deleted the build directory under helloTest2/.buildozer/android/platform
• edited the _android.pyx using nano and applied the patch as per the below
• ran buildozer -v android debug in the helloTest2 folder to recreate the apk file
• installed on Android and it worked

patch applied:

#Lee fix bug
java_namespace = JAVA_NAMESPACE.decode() if isinstance(JAVA_NAMESPACE, bytes) else JAVA_NAMESPACE
python_act = autoclass(java_namespace + u'.PythonActivity')
#Lee old code
#python_act = autoclass(JAVA_NAMESPACE + u'.PythonActivity')

[aswinmurali-io]

command which cython
output /home/aswin/.local/bin/cython

second command cat $(which cython)
output

#!/usr/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from Cython.Compiler.Main import setuptools_main


if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(setuptools_main())

[aswinmurali-io]

it's using cython under python instead of python3 is that the issue? the bytes and str problem?

[leematthewshome]

"which cython" gives me...

/usr/local/bin/cython

cat $(which cython) gives me....

#!/usr/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from Cython.Compiler.Main import setuptools_main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(setuptools_main())

[aswinmurali-io]

According to buildozer.readthedocs.io/en/latest/installation.html
This is how they said to install cython and this is what I did. It's cython under python2
sudo pip install --upgrade cython==0.28.6

[leematthewshome]

Actually I I had trouble with the old Kivy install documentation that was mostly centered around Python2.7 So when I installed Cython I installed using BOTH pip2 and pip3. This was because I had seen some old posts on the net that implied python on android did not support python3, so I installed using both just in case.

If Cython was installed into both my Python2.7 and Python3.5 environments (both exist on my PC) and the buildozer spec says to use python3 then I would have thought it would use the Python3 install of Cython.

[aswinmurali-io]

Hello, @AndreMiras @Jonast any updates?

[ghost]

@SyncCodes did you try sudo pip uninstall cython, pip uninstall cython (to make sure it's gone from all places) and then sudo pip3 install cython? If it's really cython being installed via python 2 that should possibly fix it

[aswinmurali-io]

Hello @Jonast the problem is fixed now! I started a fresh VM with the latest ubuntu 18 and now only used the python3, python3-pip and cython under python3 and now it works. Thanks for the support. I would appreciate if you can change the buildozer documents to add support for pip3 installation of cython instead of pip. Because the documentation still uses cython under pip and not pip3. Sorry for the trouble

[aswinmurali-io]

Is it possible for the users to get warned by buildozer for the cython version they use? This can be a problem for various users. Like if buildozer is targetting python3 then buildozer will check if cython is from python3 and not python? That warning feature would be greatly appreciated

[leematthewshome]

Actually, if you put osx.python_version = 3 in the buildozer spec file shouldnt it just use cython from python3? I cant see a scenario where you would want to use python2.7 cython to build a python3 targeted app.

For an existing PC where I have cython installed with both pip and pip3, how can I fix my system so it uses the correct version of cython with buildozer? Will just uninstalling with pip result in it falling back to the pip3 installed version?

[ghost]

I just looked at the code in pythonforandroid/build.py and it seems to me this is a bug, it uses "Cython" as a binary rather than using sys.executable/python2/python3 -m cython which should probably fix this. I'll see if I can change that, it seems fixable to me

[ghost]

Reopening for now since the fix wasn't actually validated. If anyone affected would be interested in retesting, let me know so we can figure out if this actually helped with anything

(otherwise I will close this ticket in a week or two)

[leematthewshome]

I am interested in retesting. Its been a long time since i looked at Kivy as i have been focused on the pure Python devlopment of my project. I just started familiarising myself again. I assume to test i will need to update my buildozer install, is that correct? If so, how should i update it? Do i just run the following...

pip install --upgrade buildozer

[ghost]

I assume to test i will need to update my buildozer install, is that correct? no need, you just need to upgrade p4a/python-for-android to test. to do that, I think the best way is to wipe .buildozer (warning: this will afaik also wipe your distributions if you care about any of them!) and then just rerun from your buildozer spec so p4a will be fetched

(someone of the other devs please correct me if I got this wrong! I don't really use buildozer myself)

edit: oh and you need to set p4a to the dev version in your spec file of course, the develop branch. not stable, since the fix isn't in a stable release yet

[leematthewshome]

I tested today and deleted all .buildozer files. Ran a build again and watched it download py4a in the comments. The following is what it said:

git clone -b master --single-branch https://github.com/kivy/python-for-android.git python-for-android

Still got the same error with the resulting apk file when installed on a Samsung phone. The app failed with the "TypeError: must be str not bytes". (Error message detected using adb logcat when open the app). Tried running build with another demo app that I have not created before and got exactly the same result. Still failed with the same error.