Hugging Face's logo

Diffusers documentation

AutoencoderKLMiniMaxH3Audio

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v0.39.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

AutoencoderKLMiniMaxH3Audio

The audio autoencoder used in MiniMax-H3 by MiniMax. It is waveform in and waveform out, with no mel front-end and no separate vocoder: a DAC-lineage strided convolutional encoder, a causal-attention projection onto the diffusion latent width, and a BigVGAN decoder.

The encoder hops 800 samples at 32 kHz, i.e. 40 latents per second, so a waveform of 800 * n samples encodes to n latents. Waveforms that are not a whole number of hops are right-padded.

The causal-attention projection goes through the attention dispatcher, so set_attention_backend applies to it; its mask is is_causal=True, which every backend honours except _native_npu, whose kernel takes no causal flag.

The autoencoder is mono, and it normalizes latents per channel with latents_mean / latents_std rather than a scalar scaling_factor. MiniMax-H3 carries stereo as two batch items, and it always consumes the posterior mean (latent_dist.mode()), never a sample.

import torch
from diffusers import AutoencoderKLMiniMaxH3Audio

audio_vae = AutoencoderKLMiniMaxH3Audio.from_pretrained(
    "MiniMaxAI/MiniMax-H3", subfolder="audio_vae", dtype=torch.float32
).to("cuda")

AutoencoderKLMiniMaxH3Audio

class diffusers.AutoencoderKLMiniMaxH3Audio

< >

( encoder_dim: int = 64encoder_rates: tuple = (2, 4, 4, 5, 5)latent_dim: int = 2048latent_channels: int = 32num_attention_heads: int = 8decoder_dim: int = 1024decoder_rates: tuple = (5, 5, 2, 2, 2, 2, 2)decoder_kernel_sizes: tuple = (9, 9, 4, 4, 4, 4, 4)resblock_kernel_sizes: tuple = (3, 7, 11)resblock_dilation_sizes: tuple = ((1, 3, 5), (1, 3, 5), (1, 3, 5))sampling_rate: int = 32000latents_mean: list[float] | None = Nonelatents_std: list[float] | None = None )

Parameters

  • encoder_dim (int, defaults to 64) — Channel width of the encoder’s first convolution; doubles at every downsampling stage.
  • encoder_rates (tuple[int], defaults to (2, 4, 4, 5, 5)) — Encoder strides. Their product (800) is the hop length, i.e. 40 latents/s at 32 kHz.
  • latent_dim (int, defaults to 2048) — Width of the encoder trunk and of the decoder input, before/after the latent projections.
  • latent_channels (int, defaults to 32) — Width of the diffusion latent, i.e. the mean_proj / logs_proj output channels.
  • num_attention_heads (int, defaults to 8) — Number of heads in the causal-attention projection pre_block.
  • decoder_dim (int, defaults to 1024) — BigVGAN initial channel count; halved at every upsampling stage.
  • decoder_rates (tuple[int], defaults to (5, 5, 2, 2, 2, 2, 2)) — BigVGAN upsampling rates. Their product must equal prod(encoder_rates).
  • decoder_kernel_sizes (tuple[int], defaults to (9, 9, 4, 4, 4, 4, 4)) — Transposed-convolution kernel size per upsampling stage.
  • resblock_kernel_sizes (tuple[int], defaults to (3, 7, 11)) — Kernel sizes of the parallel AMP residual blocks at each upsampling stage.
  • resblock_dilation_sizes (tuple[tuple[int]], defaults to ((1, 3, 5), (1, 3, 5), (1, 3, 5))) — Per-AMP-block dilations.
  • sampling_rate (int, defaults to 32000) — Waveform sampling rate.
  • latents_mean (list[float], optional) — Per-channel latent mean the pipeline uses to normalize / denormalize latents.
  • latents_std (list[float], optional) — Per-channel latent standard deviation the pipeline uses to normalize / denormalize latents.

The audio autoencoder used by MiniMax-H3: a DAC-lineage convolutional encoder and a BigVGAN decoder, operating directly on mono 32 kHz waveforms.

This model inherits from ModelMixin. Check the superclass documentation for the generic methods the library implements for all models (such as downloading or saving).

encode

< >

( sample: Tensorreturn_dict: bool = True ) → MiniMaxH3AudioEncoderOutput or tuple

Parameters

  • sample (torch.Tensor) — Mono waveform of shape [batch_size, 1, samples]. MiniMax-H3 passes the two stereo channels of a reference clip as batch_size = 2.
  • return_dict (bool, defaults to True) — Whether to return a MiniMaxH3AudioEncoderOutput instead of a plain tuple.

Returns

MiniMaxH3AudioEncoderOutput or tuple

The latent posterior over [batch_size, latent_channels, samples / 800].

Encode a waveform into the audio latent posterior.

The waveform is right-padded to a multiple of hop_length (800 samples) first. MiniMax-H3 always consumes the posterior mean (latent_dist.mode()) — the logs_proj head is never evaluated by the reference pipeline.

decode

< >

( latents: Tensorreturn_dict: bool = True ) → DecoderOutput or tuple

Parameters

  • latents (torch.Tensor) — Denormalized latents of shape [batch_size, latent_channels, num_frames]. MiniMax-H3 passes the two stereo channels as batch_size = 2.
  • return_dict (bool, defaults to True) — Whether to return a DecoderOutput instead of a plain tuple.

Returns

DecoderOutput or tuple

Waveform of shape [batch_size, 1, num_frames * 800], clamped to [-1, 1].

Decode audio latents into a waveform.

forward

< >

( sample: Tensorsample_posterior: bool = Falsereturn_dict: bool = Truegenerator: typing.Optional[torch.Generator] = None ) → DecoderOutput or tuple

Parameters

  • sample (torch.Tensor) — Mono waveform of shape [batch_size, 1, samples].
  • sample_posterior (bool, defaults to False) — Whether to sample the posterior instead of taking its mode. MiniMax-H3 uses the mode.
  • return_dict (bool, defaults to True) — Whether to return a DecoderOutput instead of a plain tuple.
  • generator (torch.Generator, optional) — Generator used when sample_posterior=True.

Returns

DecoderOutput or tuple

The round-tripped waveform of shape [batch_size, 1, num_frames * 800], clamped to [-1, 1].

Encode then decode a waveform.

Update on GitHub

←AutoencoderKLMiniMaxH3 AutoencoderKLMochi→