[POSSIBLE NEW INTEGRATION] I found a video generator much faster and also uses depthmap

[davidmartinrius]

Hi everyone!

How are you doing?

I stopped using this extension because each 3d video requires a mesh and it is sooo slow to generate a mesh and then render a video.

I found a project that makes the same but much faster, with no need of a mesh. Also works with depth maps.

https://github.com/BrokenSource/DepthFlow

This extension is quick as lightning, I was amazed when I tried it.

I told this idea to integrate this to @Tremeschin in BrokenSource/DepthFlow#5 and he liked the idea.

Also, maybe it is easier to create a new extension from scratch starting from DepthFlow as a base.

What do you think about this @semjon00 @thygate ?

Thank you!

[davidmartinrius]

Also he developed https://github.com/BrokenSource/ShaderFlow

It makes the depth images move with the music / sound

[Tremeschin]

Thanks for reaching out to other projects for possible integrations @davidmartinrius ! ❤️

On the technical side of things, having my code base as a submodule can be troublesome, I'll need to scrap the idea of "skip cloning private submodules", which isn't standard at all on git. It's a non-essential feature but convenient for only me

Feel free to reach me out @thygate for any feature requests and cleanup on my side, or wait for proper PyPI releases on me

[semjon00]

Both projects are awesome! But I'm not sure how useful would an integration be - DepthFlow can be used conviniently as-is 🙂
But thanks for mentioning this project! I gonna add it to the Readme into the projects that might interest you section.

[semjon00]

@Tremeschin Thanks for passing by! I would like to ask you one thing. If you've got time and interest, could you please take a look at my stereoimage generation code? Currently it is not geometrically accurate, and I hope that you may have the knowledge to fix it.

The important part is this function, that takes the image and its depth, and for every row creates a polyline, where each dot has some deviation from its original position.

stable-diffusion-webui-depthmap-script/src/stereoimage_generation.py

Line 163 in 500ee72

def apply_stereo_divergence_polylines(

The lines crucial for geometric accuracy are

stable-diffusion-webui-depthmap-script/src/stereoimage_generation.py

Lines 182 to 183 in 500ee72

	coord_d = (normalized_depth[row][col] ** stereo_offset_exponent) * divergence_px
	coord_x = col + 0.5 + coord_d + separation_px

coord_d is a dot priority when drawing, coord_x is the horizontal position of the dot.

I would be happy if you could help 🙂

[Tremeschin]

Happy to help! And thanks for showcasing me on "projects that might interest you" 😭

~

It might take a bit until I understand your approach, but until then, I can give you some insight on how depthflow was coded

I mostly derived on my own a very similar math as seen here https://learnopengl.com/Advanced-Lighting/Parallax-Mapping

I've got a pretty standard plane-to-plane 3d camera doing projections based on the coordinates of the screen, depthflow's shader is quite "simple" and very inefficient in a way - here's my old development sketch on xournalpp (I'm yet to make a proper one)

(I call STUV as ShaderToy UV going from (0, 0) to (1, 1) and GLUV the OpenGL UV going from (-1, 1), the "a" version as in "astuv", "agluv" are the ones I just mentioned, absolute coords, the gluv and stuv have their xcomponent multiplied to take account aspect ratio and the top of the screen to always be coordinate 1)

(In my case, bottom plane is Z=1, I'm using the camera basis as XY on the plane of the screen and Z as "forward", so left handed Y up, found it the most convenient for projections and positioning things, even tho Z up and right handed is the "mathematical" way, doesn't matter in the end just interpretations of vectors or axis reflections (handedness))

The black line is a plane distance 1 from the camera, and the depth map can be scaled on a height h, I calculated the theoretical lambda point the camera would project to and sample the texture. However, we need to find the "earliest" occlusion of the depth that blocks its view, and that's the essential part of the parallax effect

Translating to math this means we walk from lambda to where the ray was projected (lambda -> P) and keep track of the last coordinate the depth map was higher than the "projection ray's height" (distance from the bottom plane) at that point (which is what the end for loop does, then draws the image)

We don't need to search further than the green plane as no depth's height can reach it (assuming normalized 0-1), I called the "util" walk distance beta, and delta the minimum we should care of, so we get a unit vector from lambda to P, walk in many small steps of beta*(the unit vector), keep saving the last sample point the depth is greater than the ray's height

Calculating theta can be tricky, be careful to take where the ray was projected from and not the actual camera's position (this only applies if you have isometric or dolly zoom parameters, which I both support)

The other math on the code is just algebraic manipulation / many triangles to find sides etc, tbh there's not much more to it!

One thing that helped me a lot was to simplify the problem: don't think of the 3d first, solve it in 2d or 1d then just rotate it around the normal axis, and naturally you've got a 2d/3d version, as evident by the 1d sketch and math done on the plane in the shader

Let me know if this helps or if I can try helping into the specifics of your code !