Feature request: Add support for image fill sampling and scaling

[crertel]

What problem does this feature solve?

So, I'm trying to build a simple little animated sprite renderer (similar to what you'd see in a game). This is done by using a sprite sheet, and by using the existing API to render.

Doing this, you call something like:

      |> rect( {vp_width, vp_height},
                id: :image_root,
                fill: {:image, {@image_hash, 255}})

And later, if you want to be more specific, you do:

      |> rect( {vp_width, vp_height},
                id: :image_root,
                fill: {:image, {@image_hash, start_x, start_y, image_width, image_height, theta, alpha, 255}})

(note)
Why is alpha on [0-255] instead of just the normal [0-1]? Theta (the rotation) is in radians as it should be. We should try to make this API unsurprising for people who've done graphics. :(
(end note)

So, the problem here is that if I want to scale up my little image rectangle, I have to scale the rectangle instead of the image--the texture sample isn't automatically stretched to fit the rectangle in Scenic, which is again unexpected from normal graphics conventions.

My source image is basically this:

And when I render it with the following code (cribbed from other scenic examples and modified):

def init(_, opts) do
    viewport = opts[:viewport]
    IO.inspect ViewPort.info(viewport), label: "viewport info"

    # calculate the transform that centers the parrot in the viewport
    {:ok, %ViewPort.Status{size: {vp_width, vp_height}}} = ViewPort.info(viewport)

    Scenic.Cache.File.load(@image_path, @image_hash)

    graph =
      @root_image_graph
      |> rect( {vp_width, vp_height},
                id: :image_root,
                fill: {:image, {@image_hash, 0,0,100,100,0,255}})
      |> group(
        fn g ->
          g
          |> line({{-10,0},{10,0}}, stroke: {2, :magenta})
          |> line({{0,-10},{0,10}}, stroke: {2, :magenta})
        end,
        id: :cursor_lines,
        transform: {0,0})
      |> push_graph()

    state = %{
      viewport: viewport,
      graph: graph,
      cursor_pos: {0,0},
      in_drag: false,
      start_point: nil,
      end_point: nil
    }

    {:ok, state}
  end

it renders something like this:

So, it looks like it's trying to scale the image down into a sub-rectangle which may or may not be transformed in the Scenic rectangle.

Additionally, you can kinda see image streaks: I'd suggest that that's because of an incorrect GL texture sampling mode, and that nanovg is deciding to clamp to edge. That's fine, but it's nice to be able to do other things (see glTexParameter for GL's take on this). At the least clamp to border would probably be the reasonable choice, or mirroring/repeating.

What is your proposed solution?

So, I'd like to propose a couple of things:

Change the image fill semantics for rectangles to stretch the entire image across the rectangle, always. There's no convenient way to, say, update the image transformation (image fill offset and scale) anyways, so we might as well just leave that up to the transform of the rectangle that is being drawn--this also means we don't have to fight with the scale property on the rectangle at the same time we fight with the image fill's width and height.

Allow users to set the image ST scaling on the rectangle. If I have an image fill (say, a simple brick image), and I want to tile it over the rectangle right now, I can't quite do it. It'd be cool if I could specify multipliers for ST (probably better understood as X and Y tiling scaling) so that the image, once stretch over the rectangle, is scaled out (values < 1) or tiled across (values > 1) properly. This might be the time to include the ability to clamp, repeat, or mirror as seems reasonable.

(note)
The docs erroneously state the image tuple as:
{:image, {image_key, start_x, start_y, end_x, end_y, angle, alpha}}
when it should be:
{:image, {image_key, start_x, start_y, width, height, angle, alpha}}
(end note)

In light of these things, and to make the API non-breaking with what's already been done, I'd propose a new fill type:

{:image_stretched, {image_key, scale_x, scale_y, x_repeat_mode, y_repeat_mode, alpha}}
{:image_stretched, {image_key, scale_x, scale_y, x_repeat_mode, y_repeat_mode, source_offset_x, source_offset_y, source_width, source_height, angle,alpha}}

• scale_x is the S (horizontal) scale factor for the image when rendered on the rectangle. 1 is normal, 0.1 would be showing the first tenth of the horizontal parts of the image, 10 would be tiling the horizontal parts 10 times.
• scale_y is the T (vertical) scale factor for the image when rendered on the rectangle. 1 is normal, 0.1 would be showing the first tenth of the vertical parts of the image, 10 would be tiling the vertical parts 10 times.
• x_repeat_mode would be one of :repeat, :clamp_to_border, :clamp_to_edge, :mirror.
• y_repeat_mode would be one of :repeat, :clamp_to_border, :clamp_to_edge, :mirror.
• source_offset_x and source_offset_y are the offset (in pixels) of the upper-left corner of the subset of the image to use when texturing the rectangle.
• source_width is the horizontal size, in pixels, of the subset of the image to use when texturing the rectangle.
• source_height is the vertical size, in pixels, of the subset of the image to use when texturing the rectangle.
• angle is the CCW rotation of the image in radians prior to sampling.
• alpha is the [0,1] transparency (fully transparent to fully opaque) of the image to use.

[crertel]

Any word on this one?

[boydm]

Been busy with fonts.

I have an alternate proposal, although it is vague. I'd like to see a separate mechanism to officially support sprite atlases. Doing it via the image fill means setting the texture every time you draw a sprite from it. This is slow.

Proper atlas support would look something like here is a single image. Now draw this list of locations out to these spots on the screen. They would all have to have the same transforms on them, but could still be translated around. Then it would set the texture once into the video card, do a bunch of copies, then unset it. Much faster.

This is essentially how rendering text works.

[crertel]

I like sprite atlases, but my counterargument is that this feature would still be helpful for dynamically-generated textures (say, from a camera) that wouldn't make sense to put into an atlas.

[jessejanderson]

@boydm Has there been any progress on this? I've been working on the same problem as @crertel and seeming had all the same assumptions and similar conclusions. I was not previously familiar with a sprite atlas, which does seem like a possible good solution here, though the tuple labelling remains confusing since end_x, end_y don't seem to match expectations.

In particular, I lost a bit of time due to the 0-255 alpha scale since I am used to alpha always being defined on a 0-1 scale and an alpha of 1 on a 0-255 scale is basically invisible. I thought my cache or something wasn't working, but it was just an alpha problem. Unfortunate that a common assumption (1.0 == 100% for an alpha value) will result in confusion and misdiagnosis.

Thanks!

[crertel]

@boydm any word on this sir?

[boydm]

No real progress as I've been swamped in OS level work.
I'd be happy to do a call with @cretel and @jessejanderson to discuss a plan. That might help it move forward.

[JEG2]

I hit this problem this weekend. Looking forward to spritesheet support!

[boydm]

Hi @JEG2. Would love to understand what you are doing with it.

[JEG2]

I did a programming contest last weekend where a quick GUI would have come in handy. I fired up Scenic and got what I needed going. It was cool!

That inspired me to go back and play around some more. I thought, I'll build a trivial sprite game to get to know it better. I pulled in a free spritesheet off the Internet, then lost two days of free time chasing {:image, ox, oy, ex, ey, angle, alpha} through Scenic's code. I mostly lost the war, but I've learned a fair bit. We'll call it even.

I might be interested in helping to add this feature. I'm really a terrible C programmer, but maybe if I had a roadmap. I understand if you don't have the cycles to provide guidance though.

I am quite surprised about the lack of posts I can find on Scenic. Googling Elixir Scenic sprite doesn't get a single useful hit (that I've found so far). I had to go through the example code to figure out what the expected values for alpha are for images and I couldn't find anything to explain angle, until I read this issue. Maybe I could add some docs if I got my head around it enough.

Anyways, I appreciate having Scenic to use in contests. Thanks for all your effort!

[boydm]

The big question is not how to do it on the C side. That is relatively easy. The question is what should the elixir side module and api look like. It isn't straight forward.

First, if possible, you want to blit out all the sprites at once. This is for efficiency reasons on the C side. Setting the image into the driver so that it can be copied from is relatively expensive. Once it is there, lots of copies from it are relatively cheap.

Sorry for the braindump feel for this but it is what has stopped me...

So... A Sprite module would have an image, or a reference to an image in the cache. Then have a list of from/to tuples to copy from. Rendering a list of tuples is nice and efficient from the C side. BUT... The sprites are updated in more of random-access pattern on the elixir side.. That wants to be a map. But if it is a map, then how do I know and efficiently traverse the order?

The graph had the same problem. Updates to it are random access, but rendering is like a list. So the structure of the graph is a map but the contents of each entry contains the key to the next entry. This is complicated and was difficult to get right. I hesitate to do the same thing again for the Sprite sheet api. But maybe that is what it has to be.

You have any ideas on the design from the Elixir side?

[boydm]

Not said directly above but... The Sprite Sheet would be a new primitive as the C side needs to know about. It would get rendered "all at once". All the copies in the tuple list/map/whatever would be rendered in order, but essentially atomically as far as the rest of the graph is concerned.

You could probably have multiple SpriteSheet objects in the graph, but you do not want too many of them as that is inefficient.

Think of it this way. Rendering text in the current implementation is essentially copying characters out of a sprite sheet. The font sheet is set once, then lots of individual characters are copied out of it. This is relatively easy as text is really predictable in where to render each item. (cough cough. Also really hard with kerning). This is also why there is no support for changing the font characteristics in the middle of a string. That would be changing out the font sheet set in the driver.

[JEG2]

Thanks for the tips!

I'm very busy at work right now, but I just made a note to fiddle with this a bit on the weekend. If I come up with anything worth sharing, I'll do so.

[crertel]

I'm a little worried we got distracted with the sprite sheet stuff. It's useful and exactly what you need for games, for example, but the more basic and general case of "I want to render this polygon with scaled repeating textures" was kinda lost in the shuffle I fear.

[boydm]

Hi @crertel. Yes. I think the original purpose got lost.

I know I had "I want to render this polygon with scaled repeating textures" working in the past. Maybe it regressed? Or maybe the docs are wrong.

Sorry for the latency figuring this out. I've been beyond swamped in other C code on a deadline...

[crertel]

Totally understandable, no offense taken. :)