Video-Text-to-Text
Transformers
Safetensors
English
Chinese
moss_vl
feature-extraction
SFT
Video-Understanding
Image-Understanding
MOSS-VL
OpenMOSS
multimodal
video
vision-language
custom_code
Instructions to use OpenMOSS-Team/MOSS-VL-Instruct-0708 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OpenMOSS-Team/MOSS-VL-Instruct-0708 with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("OpenMOSS-Team/MOSS-VL-Instruct-0708", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
per-sample media budgets in processor; per-sample MRoPE position ids for batched vision inputs
Browse files- modeling_moss_vl.py +21 -0
modeling_moss_vl.py
CHANGED
|
@@ -1732,6 +1732,27 @@ class MossVLModel(MossVLPreTrainedModel):
|
|
| 1732 |
rope_deltas: (batch_size,) - position offset due to vision tokens
|
| 1733 |
"""
|
| 1734 |
batch_size, max_vision_seq_len, _ = cross_attention_states.shape
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1735 |
device = cross_attention_states.device
|
| 1736 |
image_token_id = self.config.image_token_id
|
| 1737 |
merge_size = self.visual.spatial_merge_size
|
|
|
|
| 1732 |
rope_deltas: (batch_size,) - position offset due to vision tokens
|
| 1733 |
"""
|
| 1734 |
batch_size, max_vision_seq_len, _ = cross_attention_states.shape
|
| 1735 |
+
if batch_size > 1:
|
| 1736 |
+
# Match frames to text markers within each sample. Blank images and
|
| 1737 |
+
# right-truncated media may have no corresponding marker; flattening
|
| 1738 |
+
# across the batch would assign their metadata to the next sample.
|
| 1739 |
+
# Keep the original single-sample computation, including in-place
|
| 1740 |
+
# text position updates and generation's per-sample rope_deltas.
|
| 1741 |
+
results = [
|
| 1742 |
+
self.compute_vision_position_ids(
|
| 1743 |
+
input_ids[i:i + 1],
|
| 1744 |
+
position_ids[:, i:i + 1],
|
| 1745 |
+
[vision_token_info[i]],
|
| 1746 |
+
cross_attention_states[i:i + 1],
|
| 1747 |
+
None if attention_mask is None else attention_mask[i:i + 1],
|
| 1748 |
+
)
|
| 1749 |
+
for i in range(batch_size)
|
| 1750 |
+
]
|
| 1751 |
+
return (
|
| 1752 |
+
torch.cat([result[0] for result in results], dim=1),
|
| 1753 |
+
position_ids,
|
| 1754 |
+
torch.cat([result[2] for result in results], dim=0),
|
| 1755 |
+
)
|
| 1756 |
device = cross_attention_states.device
|
| 1757 |
image_token_id = self.config.image_token_id
|
| 1758 |
merge_size = self.visual.spatial_merge_size
|