Skip to content

Commit

Permalink
Misc changes for image-to-image (#329)
Browse files Browse the repository at this point in the history
Co-authored-by: Pedro Cuenca <[email protected]>
  • Loading branch information
osanseviero and pcuenca authored Nov 22, 2023
1 parent 921533e commit dfa711c
Showing 1 changed file with 8 additions and 7 deletions.
15 changes: 8 additions & 7 deletions packages/tasks/src/image-to-image/about.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,22 @@

### Style transfer

One of the most popular use cases of image to image is the style transfer. Style transfer models can convert a regular photography into a painting in the style of a famous painter.
One of the most popular use cases of image-to-image is style transfer. Style transfer models can convert a normal photography into a painting in the style of a famous painter.

## Task Variants

### Image inpainting

Image inpainting is widely used during photography editing to remove unwanted objects, such as poles, wires or sensor
Image inpainting is widely used during photography editing to remove unwanted objects, such as poles, wires, or sensor
dust.

### Image colorization

Old, black and white images can be brought up to life using an image colorization model.
Old or black and white images can be brought up to life using an image colorization model.

### Super Resolution

Super resolution models increase the resolution of an image, allowing for higher quality viewing and printing.
Super-resolution models increase the resolution of an image, allowing for higher-quality viewing and printing.

## Inference

Expand Down Expand Up @@ -55,20 +55,21 @@ await inference.imageToImage({

## ControlNet

Controlling outputs of diffusion models only with a text prompt is a challenging problem. ControlNet is a neural network type that provides an image based control to diffusion models. These controls can be edges or landmarks in an image.
Controlling the outputs of diffusion models only with a text prompt is a challenging problem. ControlNet is a neural network model that provides image-based control to diffusion models. Control images can be edges or other landmarks extracted from a source image.

Many ControlNet models were trained in our community event, JAX Diffusers sprint. You can see the full list of the ControlNet models available [here](https://huggingface.co/spaces/jax-diffusers-event/leaderboard).

## Most Used Model for the Task

Pix2Pix is a popular model used for image to image translation tasks. It is based on a conditional-GAN (generative adversarial network) where instead of a noise vector a 2D image is given as input. More information about Pix2Pix can be retrieved from this [link](https://phillipi.github.io/pix2pix/) where the associated paper and the GitHub repository can be found.
Pix2Pix is a popular model used for image-to-image translation tasks. It is based on a conditional-GAN (generative adversarial network) where instead of a noise vector a 2D image is given as input. More information about Pix2Pix can be retrieved from this [link](https://phillipi.github.io/pix2pix/) where the associated paper and the GitHub repository can be found.

Below images show some of the examples shared in the paper that can be obtained using Pix2Pix. There are various cases this model can be applied on. It is capable of relatively simpler things, e.g. converting a grayscale image to its colored version. But more importantly, it can generate realistic pictures from rough sketches (can be seen in the purse example) or from painting-like images (can be seen in the street and facade examples below).
The images below show some examples extracted from the Pix2Pix paper. This model can be applied to various use cases. It is capable of relatively simpler things, e.g., converting a grayscale image to its colored version. But more importantly, it can generate realistic pictures from rough sketches (can be seen in the purse example) or from painting-like images (can be seen in the street and facade examples below).

![Examples](https://huggingface.co/datasets/huggingfacejs/tasks/resolve/main/image-to-image/pix2pix_examples.jpg)

## Useful Resources

- [Image-to-image guide with diffusers](https://huggingface.co/docs/diffusers/using-diffusers/img2img)
- [Train your ControlNet with diffusers 🧨](https://huggingface.co/blog/train-your-controlnet)
- [Ultra fast ControlNet with 🧨 Diffusers](https://huggingface.co/blog/controlnet)

Expand Down

0 comments on commit dfa711c

Please sign in to comment.