Getting photos via mobile upload

[snowbollaanm]

Question

I am trying to create an element for taking a photo with a phones camera and have come up with this code:

from nicegui import ui

async def get_file():
    image = await ui.run_javascript('return document.getElementById("file-upload").files[0];')

    print(image)

ui_camera_input = ui.input(label="HELLO", on_change=lambda: get_file())
ui_camera_input.props('type="file" accept="image/* capture="camera" for="file-upload"')
ui_camera_input.classes('w-full')

ui.run()

Now this works 100% of the time when uploading photos with either a computer or through the phone's files however when taking the photo with the phones camera the awaited javascript times out even when set to 50 seconds.
Weirdly though every about 1 in 10 tries the photo uploads successfully.

Any help on how to get this to work correctly would be great!

Tried on Pixel 8, Chrome Firefox and Edge
iPhone 12 on Safari

[falkoschindler]

Hi @snowbollaanm,

It looks like files is empty:

async def get_file():
    print(await ui.run_javascript('return document.getElementById("file-upload").files'))

ui.input(on_change=get_file).props('type="file" accept="image/* capture="camera" for="file-upload"')

Output:

{}

But I'm not quite sure how to fix it. Does anyone else have an idea?

[snowbollaanm]

Yea, the thing I find strange however is every 1 in 20 or so tries it does actually work and successfully upload.

[rodja]

Maybe you experience the lack of #2410? Try to increase your reconnect timeout in ui.run.

[snowbollaanm]

I tried increasing the timeout time and even though it did increase the likelihood of the photo grab being successful it still fails quite often.

Aswell a new error has occured:

JavaScript did not respond within 1.0 s
Traceback (most recent call last):
  File "c:\venv\lib\site-packages\nicegui\elements\timer.py", line 95, in _invoke_callback
    await result
  File "c:\venv\lib\site-packages\nicegui\elements\notification.py", line 86, in try_delete
    if not await self.client.run_javascript(query):
  File "c:\venv\lib\site-packages\nicegui\client.py", line 211, in send_and_wait
    raise TimeoutError(f'JavaScript did not respond within {timeout:.1f} s')
TimeoutError: JavaScript did not respond within 1.0 s

[snowbollaanm]

Ive done some more testing using the console in Inspect Element, when running the same JS to get the files in the console it shows there is actually a photo there however once running it as a run_javascript it returns the timeout error. I'm quite unsure on where to go from here

[rodja]

ui.run_javascript has a timeout parameter which defaults to 1 s. You could try to increase it...

[snowbollaanm]

Alright, after a lot of tinkering I have managed to come up with a solution that seems to work reliably.

The problem from what I can tell has been caused by two main issues:

1. Reconnect Timeout on page, after setting it to 20 seconds it became more reliable but still worked less often than not
2. File size limitations, from my testing my phone was creating images roughly 3 - 5MB in size whereas I was uploading 300kb images on my computer. I realized this after putting document.getElementById("file-upload").files; into the inspect element console and it was showing there was a file present even with the timeout errors.

The solution I ended up coming to is this:

import asyncio

from nicegui import ui

@ui.page('/')
async def page():
    ui.add_body_html('''
    <canvas id="canvas" style="display:none;"></canvas>
    <script>
    window.addEventListener("DOMContentLoaded", function() {
        document.getElementById('file-upload').addEventListener('change', function(event) {
            const file = event.target.files[0];
            if (!file.type.match('image.*')) {
                alert('Please select an image file.');
                return;
            }

            const reader = new FileReader();
            reader.onload = function(readerEvent) {
                const img = new Image();
                img.onload = function() {
                    const maxWidth = 1024;
                    const maxHeight = 768;

                    let width = img.width;
                    let height = img.height;
                    if (width > height) {
                        if (width > maxWidth) {
                            height *= maxWidth / width;
                            width = maxWidth;
                        }
                    } else {
                        if (height > maxHeight) {
                            width *= maxHeight / height;
                            height = maxHeight;
                        }
                    }
                    const canvas = document.getElementById('canvas');
                    canvas.width = width;
                    canvas.height = height;
                    const ctx = canvas.getContext('2d');
                    ctx.drawImage(img, 0, 0, width, height);
                };
                img.src = readerEvent.target.result;
            };
            reader.readAsDataURL(file);
        });
    }, false);
    </script>
    ''')


    async def get_picture():
        await asyncio.sleep(0.5)
        file = await ui.run_javascript('var canvas = document.getElementById("canvas"); var imageURL = canvas.toDataURL(); return imageURL')
        with ui.dialog(value=True).classes('w-full'):
            ui.image(source=file).classes('w-full')

    ui_camera_input = ui.input(on_change=lambda: get_picture())
    ui_camera_input.props('type="file" accept="image/* capture="camera" for="file-upload"')
    ui_camera_input.classes('w-full').style('display: none;')

    ui.html(r'''
    <label for="file-upload" class="custom-file-upload q-icon text-blue-5 notranslate material-icons q-pa-sm" style="cursor: pointer; font-size: 64px; background-color: rgba(66, 165, 245, 0.3); border-radius: 10px;" aria-hidden="true" role="presentation" rounded="true">
        add_a_photo
    </label>
    ''')
ui.run(reconnect_timeout=20)

What this does is when a file is uploaded it downsizes the image into a canvas of 1024 x 768 and then getting the canvas image data as a base64 string that I can later turn into a jpg file. It seems when trying to create higher resolution images e.g. 2048 x 1536 that its still breaking when trying to return the canvas data.

What are your thoughts @falkoschindler @rodja

[falkoschindler]

Looks pretty complicated. But great if it works. 👍🏻
The prop accept="image/* is missing a closing " though.

[snowbollaanm]

I've had to do this workaround because of what I believe to be an issue with ui.run_javascript. It seems that when the awaited value is to large that it causes the timeout errors. Running the same code into the developer console spits out what run_javascript is expected to return. I don't know if there's anything that can be done about this but figured I would raise it either way

[snowbollaanm]

Hopefully final addition to this!
I've created a class that can be called to add this "camera" to the page. Note that it will only open a camera on a mobile device otherwise on a computer it will open a file explorer. Thanks!

Class:

from typing import Callable, Optional, Any
import asyncio

from nicegui import ui


class Camera():
    def __init__(
        self,
        icon: str,
        icon_color: str,
        background_color: str,
        for_id: Optional[str] = 'file-upload',
        canvas_id: Optional[str] = 'canvas',
        on_change: Optional[Callable[..., Any]] = None,
        compression: Optional[float] = 0.8,
    ):
        """Camera Input
                :param icon: The icon to be used for the camera
                :param icon_color: The color of the icon
                :param background_color: The color of the background, must be in rgba format
                :param for_id: The id of the input being used, this should be set if you're using this more than once on a page (default: file-upload)
                :param canvas_id: The id for the canvas, this should be set if you're using more than 1 canvas on the page (default: canvas)
                :param on_change: callback to execute when the value changes
                :param compression: Compression ammount used on uploaded images, change this if no image is being returned usually caused by large file sizes (default: 0.8)
        """
        self.compression = compression
        self.canvas_id = canvas_id
        ui.add_body_html(f'''
        <canvas id="{canvas_id}" style="display:none;"></canvas>
        <script>
        window.addEventListener("DOMContentLoaded", function() {{
            document.getElementById('{for_id}').addEventListener('change', function(event) {{
                const file = event.target.files[0];
                if (!file.type.match('image.*')) {{
                    alert('Please select an image file.');
                    return;
                }}

                const reader = new FileReader();
                reader.onload = function(readerEvent) {{
                    const img = new Image();
                    img.onload = function() {{
                        const maxWidth = 2048;
                        const maxHeight = 1536;

                        let width = img.width;
                        let height = img.height;
                        if (width > height) {{
                            if (width > maxWidth) {{
                                height *= maxWidth / width;
                                width = maxWidth;
                            }}
                        }} else {{
                            if (height > maxHeight) {{
                                width *= maxHeight / height;
                                height = maxHeight;
                            }}
                        }}
                        const canvas = document.getElementById('{canvas_id}');
                        canvas.width = width;
                        canvas.height = height;
                        const ctx = canvas.getContext('2d');
                        ctx.drawImage(img, 0, 0, width, height);
                    }};
                    img.src = readerEvent.target.result;
                }};
                reader.readAsDataURL(file);
            }});
        }}, false);
        </script>
        ''')
        self._camera_input = ui.input(on_change=on_change)
        self._camera_input.props(f'type="file" accept="image/*" capture="camera" for="{for_id}"')
        self._camera_input.classes('w-full').style('display: none;')

        ui.html(rf'''
        <label for="{for_id}" class="custom-{for_id} q-icon text-{icon_color} notranslate material-icons q-pa-sm" style="cursor: pointer; font-size: 64px; background-color: {background_color}; border-radius: 10px;" aria-hidden="true" role="presentation" rounded="true">
            {icon}
        </label>
        ''')

    
    async def get_image(self):
        await asyncio.sleep(0.5)
        file = await ui.run_javascript(f'var canvas = document.getElementById("{self.canvas_id}"); var imageURL = canvas.toDataURL("image/jpeg", {self.compression}); return imageURL', timeout=15)
        return file

Usage:

@ui.page('/')
async def content():
    async def generate_image():
        await asyncio.sleep(0.1)
        image = await test_camera.get_image()
        ui.image(image).classes('w-1/5')

    test_camera = Camera(
        icon='photo_camera', icon_color='blue-5',
        background_color='rgba(66, 165, 245, 0.3)', for_id='file-upload',
        canvas_id='canvas', on_change=lambda: generate_image()
    )

ui.run()

[snowbollaanm]

Hey @mattloose,

It looks like the issue is with document.getElementById('{for_id}').addEventListener('change', function(event) {. The problem arises because the input element is not yet in the DOM when the event listener is being attached. From what I can see, the HTML for the second step is only generated after the button is pressed.

To resolve this, I suggest changing the JavaScript code from window.addEventListener("DOMContentLoaded", function() { to a separate function, such as function readyCanvas() {. Then, you can call ui.run_javascript('readyCanvas()') when you reach the appropriate step.

This should ensure the canvas is set up appropriately. Have a crack at it and if you can't get it working Ill look into it more tomorrow

Unless you have a different take @falkoschindler

[mattloose]

Thanks - if I simply switch the line:

window.addEventListener("DOMContentLoaded", function() {

to:

function readyCanvas() {

I get a console error telling me there is no such function as readyCanvas()

I call ui.run_javascript('readyCanvas()') immediately after creating the camera object in the second step. Sorry - I've never been any good with javascript!

[mattloose]

Just bumping this - I've been unsucessful with the simple switch above - any suggestions?

[mattloose]

As an alternative idea, here I am making a single camera object but I've split the javascript and showing the camera icon. This means I can reshow the same camera. A little bit more modification and I should be able to pass difference on_change functions each time.

    def __init__(
        self,
        icon: str,
        icon_color: str,
        background_color: str,
        for_id: Optional[str] = 'file-upload',
        canvas_id: Optional[str] = 'canvas',
        on_change: Optional[Callable[..., Any]] = None,
        compression: Optional[float] = 0.8,
    ):
        """Camera Input
                :param icon: The icon to be used for the camera
                :param icon_color: The color of the icon
                :param background_color: The color of the background, must be in rgba format
                :param for_id: The id of the input being used, this should be set if you're using this more than once on a page (default: file-upload)
                :param canvas_id: The id for the canvas, this should be set if you're using more than 1 canvas on the page (default: canvas)
                :param on_change: callback to execute when the value changes
                :param compression: Compression ammount used on uploaded images, change this if no image is being returned usually caused by large file sizes (default: 0.8)
        """
        self.compression = compression
        self.canvas_id = canvas_id
        self.for_id = for_id
        self.icon = icon
        self.icon_color = icon_color
        self.background_color = background_color
        ui.add_body_html(f'''
        <canvas id="{canvas_id}" style="display:none;"></canvas>
        <script>
        window.addEventListener("DOMContentLoaded", function() {{
            document.getElementById('{self.for_id}').addEventListener('change', function(event) {{
                const file = event.target.files[0];
                if (!file.type.match('image.*')) {{
                    alert('Please select an image file.');
                    return;
                }}

                const reader = new FileReader();
                reader.onload = function(readerEvent) {{
                    const img = new Image();
                    img.onload = function() {{
                        const maxWidth = 2048;
                        const maxHeight = 1536;

                        let width = img.width;
                        let height = img.height;
                        if (width > height) {{
                            if (width > maxWidth) {{
                                height *= maxWidth / width;
                                width = maxWidth;
                            }}
                        }} else {{
                            if (height > maxHeight) {{
                                width *= maxHeight / height;
                                height = maxHeight;
                            }}
                        }}
                        const canvas = document.getElementById('{canvas_id}');
                        canvas.width = width;
                        canvas.height = height;
                        const ctx = canvas.getContext('2d');
                        ctx.drawImage(img, 0, 0, width, height);
                    }};
                    img.src = readerEvent.target.result;
                }};
                reader.readAsDataURL(file);
            }});
        }}, false);
        </script>
        ''')
        self._camera_input = ui.input(on_change=on_change)
        self._camera_input.props(f'type="file" accept="image/*" capture="camera" for="{for_id}"')
        self._camera_input.classes('w-full').style('display: none;')

    def show_camera(self):
        ui.html(rf'''
        <label for="{self.for_id}" class="custom-{self.for_id} q-icon text-{self.icon_color} notranslate material-icons q-pa-sm" style="cursor: pointer; font-size: 64px; background-color: {self.background_color}; border-radius: 10px;" aria-hidden="true" role="presentation" rounded="true">
            {self.icon}
        </label>
        ''')

    
    async def get_image(self):
        await asyncio.sleep(0.5)
        file = await ui.run_javascript(f'var canvas = document.getElementById("{self.canvas_id}"); var imageURL = canvas.toDataURL("image/jpeg", {self.compression}); return imageURL', timeout=15)
        return file```

[mattloose]

In case anyone else looks at this:

self._camera_input = ui.input(on_change=on_change)

is problemattic after assign the props. The on_change event only fires once. So if you change the file later it isn't redetected. Switching to

self._camera_input = ui.input().on(
            "update:model-value", on_change
        ) 

Means that any subsequent change of the input is also detected.