Stable Diffusion Models
Importing Stable Diffusion Models
Learn how to import stable diffusion models into Sogni
To import any Stable Diffusion model that has already been converted to Core ML, whether downloaded from the internet or trained by you, go to File > Import Stable Diffusion (Core ML) Model
Settings for Imported Model
When importing your model into Sogni you'll need to assign settings for your model.
Model Name: This is the name that will be displayed on the Stable Diffusion Model list, which is located in the Advanced module at the bottom of the Controls Bar.
Output Size: Select the correct output size of the model. Currently, the following output sizes are supported: 512x512, 786x786, XL 1024x1024.
Attention: The attention version of the model dictates the compatible processing units. Ensure you select the appropriate version. If the model you're importing is available in both the Original and Split-Einsum versions, it's advisable to import both. Use the Original version with 'CPU, GPU' processing.
Quantized: Enable this option if the model has been quantized (8-bit, 6-bit, etc.).
Model Folder: Select the location of the model's uncompressed folder. A copy of the folder's contents will be imported. You can discard the downloaded model file after importing.
Output Size: Select the correct output size of the model. Currently, the following output sizes are supported: 512x512, 786x786, XL 1024x1024.
Attention: The attention version of the model dictates the compatible processing units. Ensure you select the appropriate version. If the model you're importing is available in both the Original and Split-Einsum versions, it's advisable to import both. Use the Original version with 'CPU, GPU' processing.
Quantized: Enable this option if the model has been quantized (8-bit, 6-bit, etc.).
Model Folder: Select the location of the model's uncompressed folder. A copy of the folder's contents will be imported. You can discard the downloaded model file after importing.
Deleting Imported Model
To delete a model that you have previously imported, follow these steps:
1. Select the "Reveal in Finder" option from the Stable Diffusion Model list.
2. Locate the folder for the model you wish to delete and move it to the trash. If a shortcut with the same folder name but ending with '_original_compiled' exists, delete it too.
3. Click the refresh button next to the Stable Diffusion Model list to update it.
1. Select the "Reveal in Finder" option from the Stable Diffusion Model list.
2. Locate the folder for the model you wish to delete and move it to the trash. If a shortcut with the same folder name but ending with '_original_compiled' exists, delete it too.
3. Click the refresh button next to the Stable Diffusion Model list to update it.
How to Convert Stable Diffusion Model SafeTensors or CKPT Files to Core ML
Sogni utilizes Apple's Core ML to efficiently run stable diffusion models. To obtain the necessary bundle of compiled .mlmodelc and other files, the original Stable Diffusion model (CKPT or SafeTensors) must first be converted to Diffusers. Subsequently, the Diffusers format is transformed into .mlpackage files, from which the final compiled bundle is created.
RequirementsConversion: SafeTensors / CKPT → DiffusersConversion: Diffusers → CoreMLImport CoreML model into SogniTroubleshooting
Open Terminal and run all of the following commands :
Requirements
1. Install Homebrew and follow the "Next steps" instructions in Terminal
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"2. Install Wget
brew install wget3. Install Xcode
4. Select Xcode as the active Command Line Tools provider.
sudo xcode-select -s /Applications/Xcode.app5. Install Miniconda
6. Clone Apple's ml-stable-diffusion repository to your Mac:
git clone https://github.com/apple/ml-stable-diffusion.git7. Start a new Python environment and install other requirements:
conda create -n coreml_stable_diffusion python=3.8 -yconda activate coreml_stable_diffusioncd ml-stable-diffusionpip install -e .pip install omegaconfpip install safetensors8. Create a folder called "models" on your Desktop. Place the model's .safetensors or .ckpt file and this Python script inside the new folder.
9. Run the following commands to ensure you have the correct versions for these items:
pip install torch==2.1.0pip install scikit-learn==1.1.2pip install diffusers==0.23.1pip install huggingface-hub==0.19.4pip install transformers==4.34.110. Your system should now be ready to convert Stable Diffusion models!
Conversion: SafeTensors / CKPT → Diffusers
1. Activate the Conda environment.
conda activate coreml_stable_diffusion2. Navigate to the "models" folder on your Desktop
cd ../desktop/models3. Run conversion script. Replace <MODEL-NAME> with your model's file name.
SafeTensors:
python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path <MODEL-NAME>.safetensors --from_safetensors --device cpu --extract_ema --dump_path <MODEL-NAME>_diffusersCKPT:
python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path <MODEL-NAME>.ckpt --device cpu --extract_ema --dump_path <MODEL-NAME>_diffusersFor SDXL models, add the following flag to the conversion command you are using:
--pipeline_class_name StableDiffusionXLPipeline
--pipeline_class_name StableDiffusionXLPipeline
(This process takes about 1 minute.)
Conversion: Diffusers → Core ML
Run conversion script. Replace <MODEL-NAME> with your model's file name.
1. ORIGINAL attention-implementation (compatible with CPU, GPU only)
python -m python_coreml_stable_diffusion.torch2coreml --compute-unit CPU_AND_GPU --convert-vae-decoder --convert-vae-encoder --convert-unet --unet-support-controlnet --convert-text-encoder --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o original/<MODEL-NAME>_original && python -m python_coreml_stable_diffusion.torch2coreml --compute-unit CPU_AND_GPU --convert-unet --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o original/<MODEL-NAME>_original2. SPLIT_EINSUM attention-implementation (compatible with all processing units)
python -m python_coreml_stable_diffusion.torch2coreml --compute-unit ALL --convert-vae-decoder --convert-vae-encoder --convert-unet --unet-support-controlnet --convert-text-encoder --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation SPLIT_EINSUM -o split-einsum/<MODEL-NAME>_split-einsum && python -m python_coreml_stable_diffusion.torch2coreml --compute-unit ALL --convert-unet --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation SPLIT_EINSUM -o split-einsum/<MODEL-NAME>_split-einsumFor SDXL models:
1. ORIGINAL attention-implementation (compatible with CPU, GPU only)
python -m python_coreml_stable_diffusion.torch2coreml --xl-version --compute-unit CPU_AND_GPU --convert-vae-decoder --convert-vae-encoder --convert-unet --unet-support-controlnet --convert-text-encoder --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o original/<MODEL-NAME>_original && python -m python_coreml_stable_diffusion.torch2coreml --xl-version --compute-unit CPU_AND_GPU --convert-unet --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o original/<MODEL-NAME>_originalWeight Compression:
--quantize-nbits 8
or
--quantize-nbits 6
or
--quantize-nbits 6
Example:
python -m python_coreml_stable_diffusion.torch2coreml --xl-version --quantize-nbits 8 --compute-unit CPU_AND_GPU --convert-vae-decoder --convert-vae-encoder --convert-unet --unet-support-controlnet --convert-text-encoder --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o original/<MODEL-NAME>_original_8bit && python -m python_coreml_stable_diffusion.torch2coreml --xl-version --quantize-nbits 8 --compute-unit CPU_AND_GPU --convert-unet --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o original/<MODEL-NAME>_original_8bitCustom Size:
Modifying the output image size of the model is only possible with the ORIGINAL implementation. To do so use the following flags:
--latent-w <SIZE> --latent-h <SIZE>
Note: Size must be a multiple of 64, specify it divided by 8 (Eg. for 768x768 use --latent-w 96 --latent-h 96)
Example:
--latent-w <SIZE> --latent-h <SIZE>
Note: Size must be a multiple of 64, specify it divided by 8 (Eg. for 768x768 use --latent-w 96 --latent-h 96)
Example:
python -m python_coreml_stable_diffusion.torch2coreml --xl-version --quantize-nbits 8 --latent-w 96 --latent-h 96 --compute-unit CPU_AND_GPU --convert-vae-decoder --convert-vae-encoder --convert-unet --unet-support-controlnet --convert-text-encoder --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o original/<MODEL-NAME>_original_8bit_768x768 && python -m python_coreml_stable_diffusion.torch2coreml --xl-version --quantize-nbits 8 --latent-w 96 --latent-h 96 --compute-unit CPU_AND_GPU --convert-unet --model-version <MODEL-NAME>_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o original/<MODEL-NAME>_original_8bit_768x768(This process can take 30+ minutes depending on model size and quantization settings.)
Import CoreML model into Sogni
1. The compiled model folder will be created with the name "Resources" inside the "models" folder on your desktop, under the orginal/<MODEL-NAME> or split-einsum/<MODEL-NAME> folder. Give the "Resources" folder a more descriptive name.
2. You can discard the .mlpackage files from the orginal/<MODEL-NAME> or split-einsum/<MODEL-NAME>.
3. You can also discard the <MODEL-NAME>_diffusers folder if you are not going to perform other convertions from this model.
4. Open Sogni, and go to File > Import Stable Diffusion (CoreML) model. Select the folder with the compiled resources to import the model.
5. Select your new model from the Stable Diffusion models list in the "Advanced" tab of the Controls-Bar.
2. You can discard the .mlpackage files from the orginal/<MODEL-NAME> or split-einsum/<MODEL-NAME>.
3. You can also discard the <MODEL-NAME>_diffusers folder if you are not going to perform other convertions from this model.
4. Open Sogni, and go to File > Import Stable Diffusion (CoreML) model. Select the folder with the compiled resources to import the model.
5. Select your new model from the Stable Diffusion models list in the "Advanced" tab of the Controls-Bar.
Troubleshooting
Miniconda:
This package is incompatible with this version of macOS: after the "Software Licence Agreement" step, click on "Change Install Location..." and select "Install for me only"
This package is incompatible with this version of macOS: after the "Software Licence Agreement" step, click on "Change Install Location..." and select "Install for me only"
Terminal:
xcrun: error: unable to find utility "coremlcompiler", not a developer tool or in PATH: open Xcode and go to "Settings..." → "Locations" then click on the "Command Line Tools" drop-down menu and reselect the Command Line Tools version
ModuleNotFoundError: No module named 'xyz': while the conda coreml_stable_diffusion environment is active.
Run:
xcrun: error: unable to find utility "coremlcompiler", not a developer tool or in PATH: open Xcode and go to "Settings..." → "Locations" then click on the "Command Line Tools" drop-down menu and reselect the Command Line Tools version
ModuleNotFoundError: No module named 'xyz': while the conda coreml_stable_diffusion environment is active.
Run:
pip install xyzzsh: killed python: your Mac has run out of memory. Close other memory-hungry applications and retry.
Need help? Ask in our Discord!
For any questions regarding the use of Sogni, tips, suggestions, bugs, sharing art, discussing AI and life, or simply hanging out, please feel free to join our Discord today!