Generate a QRCode with StableDiffusion and ControlNet
0xkoji
Posted on July 4, 2023
As you may know last month, people were passionate about generating QR codes with Stable Diffusion.
Actually, I tried to generate some QR codes with Automatic1111.
It was fun but at the same time, it was kind of hard since the success rate to generate a scannable QR code was almost 50% 🥲
Then I found an interesting modal for generating QR codes on Hugging Face.
https://huggingface.co/DionTimmer/controlnet_qrcode-control_v1p_sd15
In this post, I will introduce a way to generate a QR code with controlnet_qrcode-control_v1p_sd15 and v2_1.
This post will use Google Colab (free plan).
generate a QR code
First, we need to generate a QR code.
You can generate a QR code with python package(I use that when I need to create multiple QR codes).
But for this case, we won't need to create many QR codes so I recommend you to use this, https://keremerkan.net/qr-code-and-2d-code-generator/
install packages
!pip install diffusers transformers accelerate torch xformers
import packages
import torch
from PIL import Image
from diffusers import StableDiffusionControlNetImg2ImgPipeline, ControlNetModel, DDIMScheduler
from diffusers.utils import load_image
load pre_trainned model controlnet_qrcode-control and stable diffusion model
If you want to try v1.5, please comment out the bottom 2 line and remove the comment from the lines for v1.5.
# v1.5
# controlnet = ControlNetModel.from_pretrained("DionTimmer/controlnet_qrcodecode-control_v1p_sd15", torch_dtype=torch.float16)
# pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16)
# v2.1
controlnet = ControlNetModel.from_pretrained("DionTimmer/controlnet_qrcode-control_v11p_sd21", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16)
pipe.enable_xformers_memory_efficient_attention()
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
generate a qr code
For this step, what you need to do is the followings.
- Upload the QR code you generated to your Google Colab folder
- Get the source image / the source image URL If you want an image that is uploaded somewhere, you can use the image URL directly The image must be square. The image must be square.
- Set prompt and negative prompt
- Run (it will take 1-2min)
def resize_image(input_image: Image, resolution: int):
input_image = input_image.convert("RGB")
W, H = input_image.size
k = float(resolution) / min(H, W)
H *= k
W *= k
H = int(round(H / 64.0)) * 64
W = int(round(W / 64.0)) * 64
img = input_image.resize((W, H), resample=Image.LANCZOS)
return img
orriginal_qr_code_image = load_image('/content/code_with_block.png')
# img_path = 'https://images.squarespace-cdn.com/content/v1/59413d96e6f2e1c6837c7ecd/1536503659130-R84NUPOY4QPQTEGCTSAI/15fe1e62172035.5a87280d713e4.png'
img_path = load_image('/content/test.png')
init_image = load_image(img_path)
condition_image = resize_image(orriginal_qr_code_image, 768)
init_image = resize_image(init_image, 768)
generator = torch.manual_seed(123121231)
image = pipe(prompt="8k, 3d, ((photo-realistic)), bonsai, Japanese style",
negative_prompt="ugly, disfigured, low quality, blurry, nsfw, worst quality, illustration, drawing",
image=init_image,
control_image=condition_image,
width=768,
height=768,
guidance_scale=20,
controlnet_conditioning_scale=2.5,
generator=generator,
strength=0.9,
num_inference_steps=150,
)
image.images[0]
Google Colab code
https://colab.research.google.com/drive/1D2V-5hebvOeHJOOK19l0Bujqh4MlcLTl?usp=sharing
Right now, I'm trying Gradio to add the UI for the above script.
Posted on July 4, 2023
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.