4.3 VLM: MLP and Linear Feature Projection - Computer Vision Across the Marine Sciences

A Vision-Language Model connects a visual perception encoder with an autoregressive language model. In foundational architectures such as LLaVA and InternVL, this bridge is a multi-layer perceptron (MLP) or linear projection module. The vision encoder extracts spatial patch embeddings from input imagery. The projection module transforms those visual vectors into the token embedding space of the language model.

This lesson examines the mechanics of MLP projection layers. You will inspect weight dimensions, calculate parameter counts, trace tensor dimension shifts, ingest marine imagery from the LILA BC Community Fish Detection dataset, and execute zero-shot marine organism identification.

Learning Objectives¶

Inspect the 2-layer MLP projection module bridging vision encoder to language model.
Trace tensor transformations from visual patch embeddings to language embeddings.
Toggle between supported open-weight vision-language models: Qwen2-VL-2B-Instruct and InternVL2-1B.
Ingest marine imagery directly from the LILA BC Community Fish Detection dataset with offline fallback support.
Execute zero-shot marine organism identification using compact open-weight VLMs.
Batch-process marine survey frames and export structured detections to a pandas DataFrame and CSV file.

Part 1 - Setup¶

1.1 Install Dependencies¶

This lesson requires transformers, accelerate, pillow, opencv-python, pandas, matplotlib, torchvision, and qwen-vl-utils for vision processing.

# Install required dependencies
!pip install -q transformers>=4.40.0 accelerate pillow opencv-python-headless huggingface_hub pandas matplotlib torchvision qwen-vl-utils

1.2 Check GPU and Hardware Fallback¶

A GPU provides fast model execution. On Google Colab, a free T4 GPU provides 16 GB VRAM. The code detects CUDA automatically and falls back to CPU execution if a GPU is unavailable.

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

if torch.cuda.is_available():
    gpu_name = torch.cuda.get_device_name(0)
    vram_gb = torch.cuda.get_device_properties(0).total_memory / 1e9
    print(f"[OK] GPU: {gpu_name} | VRAM: {vram_gb:.1f} GB | Precision: {dtype}")
else:
    print("[INFO] No GPU detected. Running on CPU with float32 precision.")

1.3 Hugging Face Authentication Handling¶

The models used in this lesson (Qwen2-VL-2B-Instruct and InternVL2-1B) are public and ungated. Authentication is optional. If you have a Hugging Face token configured in your environment, the cell below applies it without interrupting execution.

import os

hf_token = os.environ.get("HF_TOKEN", None)
if hf_token:
    try:
        from huggingface_hub import login
        login(token=hf_token)
        print("[OK] Hugging Face token applied.")
    except Exception as e:
        print(f"[INFO] Login skipped: {e}")
else:
    print("[INFO] No HF_TOKEN supplied. Using unauthenticated public access.")

Part 2 - Architecture and Model Loading¶

2.1 The Two-Layer MLP Projector Concept¶

A Vision Transformer processes an image into spatial patch embeddings:

X_{vis} \in \mathbb{R}^{N \times D_{vis}}

(1)

where $N$ is the number of patch tokens and $D_{vis}$ is the vision encoder hidden dimension. The language model expects input tokens in its native embedding dimension:

Z_{llm} \in \mathbb{R}^{N \times D_{llm}}

(2)

A 2-layer MLP projection module bridges this gap:

H = \text{GELU}(X_{vis} W_1 + b_1)

(3)

Z_{llm} = H W_2 + b_2

(4)

In Qwen2-VL, adjacent 2x2 spatial patch tokens merge prior to projection. Four adjacent tokens of dimension $D_{vis} = 1280$ concatenate into a merged vector of dimension $D_{in} = 5120$ . The 2-layer MLP maps $5120 \to 5120 \to 1536$ , aligning directly with the language model embedding dimension.

In InternVL2-1B, visual patches from an InternViT encoder pass through pixel unshuffle downsampling into an explicit 2-layer MLP module (mlp1). The first linear layer projects the concatenated patch features into intermediate representations with LayerNorm and GELU non-linearities, and the second linear layer projects into the language model embedding dimension ( $D_{llm} = 896$ ).

2.2 Model Selection and Projector Loading¶

We provide an interactive model selector toggle supporting both Qwen2-VL-2B-Instruct and InternVL2-1B. You can select either model below to inspect its architecture and run marine organism identification.

# Model selector toggle
# Supported options: "Qwen2-VL-2B-Instruct" or "InternVL2-1B"
MODEL_CHOICE = "Qwen2-VL-2B-Instruct"

print(f"Active model selection: {MODEL_CHOICE}")

if MODEL_CHOICE == "Qwen2-VL-2B-Instruct":
    from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
    MODEL_ID = "Qwen/Qwen2-VL-2B-Instruct"
    print(f"Loading {MODEL_ID}...")
    processor = AutoProcessor.from_pretrained(MODEL_ID)
    model = Qwen2VLForConditionalGeneration.from_pretrained(
        MODEL_ID,
        torch_dtype=dtype,
        device_map="auto" if torch.cuda.is_available() else None,
        low_cpu_mem_usage=True,
    ).eval()
    if not torch.cuda.is_available():
        model = model.to(device)
    tokenizer = None
    projector_module = model.model.visual.merger.mlp
    print(f"[OK] {MODEL_ID} loaded successfully.")
elif MODEL_CHOICE == "InternVL2-1B":
    from transformers import AutoTokenizer
    from transformers.dynamic_module_utils import get_class_from_dynamic_module
    MODEL_ID = "OpenGVLab/InternVL2-1B"
    print(f"Loading {MODEL_ID}...")
    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
    model_class = get_class_from_dynamic_module(
        "modeling_internvl_chat.InternVLChatModel",
        MODEL_ID,
    )
    if not hasattr(model_class, "all_tied_weights_keys"):
        model_class.all_tied_weights_keys = property(lambda self: getattr(self, "_all_tied_weights_keys", {}))
        model_class.all_tied_weights_keys = model_class.all_tied_weights_keys.setter(lambda self, v: setattr(self, "_all_tied_weights_keys", v))
    model = model_class.from_pretrained(
        MODEL_ID,
        torch_dtype=dtype,
        low_cpu_mem_usage=True,
    ).eval().to(device)
    processor = None
    projector_module = model.mlp1
    print(f"[OK] {MODEL_ID} loaded successfully.")
else:
    raise ValueError(f"Unknown MODEL_CHOICE: {MODEL_CHOICE}")

print("\nVisual Projector Module:")
print(projector_module)

2.3 Inspect Projector Weights and Trace Mathematical Transformation¶

Let us examine the weight shapes of the projection layers and simulate passing a batch of visual tokens through the projector module.

print("Detailed Layer Breakdown:")
total_projector_params = 0
for idx, layer in enumerate(projector_module):
    if hasattr(layer, "weight"):
        n_params = layer.weight.numel() + (layer.bias.numel() if getattr(layer, "bias", None) is not None else 0)
        total_projector_params += n_params
        bias_shape = list(layer.bias.shape) if getattr(layer, "bias", None) is not None else None
        print(f"  Layer {idx} ({layer.__class__.__name__}): weight={list(layer.weight.shape)}, bias={bias_shape}, params={n_params:,}")
    else:
        print(f"  Layer {idx} ({layer.__class__.__name__})")

print(f"\nTotal MLP Projector Parameters: {total_projector_params:,}")

# Trace dimensional transformation with synthetic visual tokens
batch_size = 1
num_tokens = 64
if MODEL_CHOICE == "Qwen2-VL-2B-Instruct":
    d_in = projector_module[0].in_features
    d_out = projector_module[2].out_features
else:
    d_in = projector_module[1].in_features
    d_out = projector_module[3].out_features

synthetic_input = torch.randn(batch_size, num_tokens, d_in, dtype=dtype, device=device)
with torch.no_grad():
    projected_output = projector_module(synthetic_input)

print(f"Synthetic patch input shape:  {list(synthetic_input.shape)}")
print(f"Projected LLM token output:   {list(projected_output.shape)}")
print(f"Target LLM hidden dimension:  {d_out}")
assert projected_output.shape[-1] == d_out, "Projected dimension must match language model hidden size."
print("[OK] Dimension alignment verified.")

Part 3 - LILA BC Marine Dataset Integration¶

3.1 Community Fish Detection Dataset Attribution¶

We integrate imagery from the Community Fish Detection dataset on LILA BC.

Dataset Name: Community Fish Detection Dataset
Source Catalog: https://lila.science/datasets/community-fish-detection-dataset/
Attribution: Filippo Varini et al., Community Fish Detector project
License: Community Data License Agreement - Permissive (CDLA-Permissive-1.0)
Azure Blob Storage: https://lilawildlife.blob.core.windows.net/lila-wildlife/community-fish-detection-dataset/

The dataset contains underwater camera frames capturing diverse marine teleost species and benthic habitats.

3.2 Download Sample Marine Images¶

We access individual image blobs directly from Azure storage over HTTP without requiring credentials. We set a default slice of 3 sample images with an adjustable parameter to expand data volume on demand. Network requests are wrapped in exception handlers with synthetic offline fallbacks to ensure uninterrupted execution.

import urllib.request
import io
from PIL import Image, ImageDraw

BASE_AZURE_URL = "https://lilawildlife.blob.core.windows.net/lila-wildlife/community-fish-detection-dataset/JPEGImages/"

# Expanded sample blobs from underwater transects (10 entries for volume expansion)
SAMPLE_BLOBS = [
    "marine_detect_3759335805_train.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:28.300000.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:29.300000.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:30.300000.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:28.400000.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:28.500000.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:28.600000.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:28.700000.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:28.800000.jpg",
    "CDFW-LakeCam-April-SpiderBlocks1_00:16:28.900000.jpg",
]

# Adjustable data slice parameter (default: 3)
num_samples = 3  # Increase up to len(SAMPLE_BLOBS) to download additional frames

loaded_samples = []
for blob_name in SAMPLE_BLOBS[:num_samples]:
    url = BASE_AZURE_URL + blob_name
    print(f"Fetching {blob_name}...")
    try:
        req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
        with urllib.request.urlopen(req, timeout=15) as resp:
            img_bytes = resp.read()
            pil_img = Image.open(io.BytesIO(img_bytes)).convert("RGB")
            loaded_samples.append({
                "filename": blob_name,
                "image": pil_img,
                "width": pil_img.width,
                "height": pil_img.height,
                "bytes": len(img_bytes),
                "size_bytes": len(img_bytes),
                "is_fallback": False,
            })
            print(f"  [OK] Loaded {blob_name} ({pil_img.width}x{pil_img.height}, {len(img_bytes):,} bytes)")
    except Exception as err:
        print(f"  [WARN] Remote fetch failed for {blob_name} ({err}). Generating synthetic marine fallback.")
        fallback_img = Image.new("RGB", (640, 480), color=(20, 70, 115))
        draw = ImageDraw.Draw(fallback_img)
        draw.rectangle([30, 30, 610, 450], outline=(80, 170, 230), width=3)
        draw.text((50, 50), f"Synthetic Fallback Frame: {blob_name}", fill=(220, 240, 255))
        loaded_samples.append({
            "filename": blob_name,
            "image": fallback_img,
            "width": 640,
            "height": 480,
            "bytes": 0,
            "size_bytes": 0,
            "is_fallback": True,
        })

print(f"\nTotal loaded samples: {len(loaded_samples)}")

3.3 Visualize Input Marine Frames¶

We inspect the loaded imagery to observe water clarity, benthic substrate texture, and species visibility.

import matplotlib.pyplot as plt

n_samples = max(1, len(loaded_samples))
fig, axes = plt.subplots(1, n_samples, figsize=(4 * n_samples, 3.5))
if n_samples == 1:
    axes = [axes]

for ax, sample in zip(axes, loaded_samples):
    ax.imshow(sample["image"])
    status = "(Fallback)" if sample.get("is_fallback") else f"{sample['width']}x{sample['height']}"
    ax.set_title(f"{sample['filename']}\n{status}", fontsize=8)
    ax.axis("off")

plt.suptitle("Sample Marine Imagery - Community Fish Detection", fontsize=11)
plt.tight_layout()
plt.show()

Part 4 - Inference and Marine Analysis Pipeline¶

4.1 Define Single-Image Marine Inference Engine¶

We implement an inference helper that formats the input image and textual query into model inputs. The function supports both Qwen2-VL and InternVL2 architectures.

from qwen_vl_utils import process_vision_info
import torchvision.transforms as T
from torchvision.transforms.functional import InterpolationMode

IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD = (0.229, 0.224, 0.225)

def build_internvl_transform(input_size=448):
    return T.Compose([
        T.Lambda(lambda img: img.convert("RGB") if img.mode != "RGB" else img),
        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
        T.ToTensor(),
        T.Normalize(mean=IMAGENET_MEAN, std=IMAGENET_STD),
    ])

@torch.no_grad()
def run_marine_inference(
    image: Image.Image,
    prompt_text: str,
    max_new_tokens: int = 128,
) -> str:
    if MODEL_CHOICE == "Qwen2-VL-2B-Instruct":
        messages = [
            {
                "role": "user",
                "content": [
                    {"type": "image", "image": image},
                    {"type": "text", "text": prompt_text},
                ],
            }
        ]

        text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
        image_inputs, video_inputs = process_vision_info(messages)
        inputs = processor(
            text=[text],
            images=image_inputs,
            videos=video_inputs,
            padding=True,
            return_tensors="pt",
        ).to(device)

        generated_ids = model.generate(**inputs, max_new_tokens=max_new_tokens)
        trimmed_ids = [out[len(inp):] for inp, out in zip(inputs.input_ids, generated_ids)]
        response = processor.batch_decode(
            trimmed_ids,
            skip_special_tokens=True,
            clean_up_tokenization_spaces=False,
        )[0]
        return response.strip()
    else:
        transform = build_internvl_transform(input_size=448)
        pixel_values = transform(image).unsqueeze(0).to(device=device, dtype=dtype)
        question = f"<image>\n{prompt_text}"
        response = model.chat(tokenizer, pixel_values, question, generation_config=dict(max_new_tokens=max_new_tokens))
        return response.strip()

print("[OK] Marine inference engine defined.")

4.2 Execute Marine Science Analysis Prompt¶

We run zero-shot inference on the first sample image, requesting species identification, count estimation, and substrate description.

if not loaded_samples:
    print("No samples available for inference (num_samples=0). Increase num_samples to at least 1.")
else:
    sample = loaded_samples[0]
    prompt = (
        "Identify any marine organisms, fish species, or benthic substrate in this image. "
        "Provide a concise ecological summary including approximate count and habitat description."
    )

    print(f"Analyzing {sample['filename']} with {MODEL_CHOICE}...")
    response = run_marine_inference(sample["image"], prompt, max_new_tokens=150)

    print("\nModel Analysis:")
    print("-" * 60)
    print(response)
    print("-" * 60)

Part 5 - Batch Evaluation and Visualization¶

5.1 Batch Processing with Latency Tracking and CSV Export¶

We evaluate all loaded marine images sequentially, recording per-image inference latencies and structured ecological descriptions.

import time
import pandas as pd

results = []
print(f"Beginning batch evaluation across {len(loaded_samples)} frames...")

for idx, sample in enumerate(loaded_samples):
    t0 = time.time()
    resp = run_marine_inference(
        sample["image"],
        "Identify the primary marine organisms and habitat types visible in this image.",
        max_new_tokens=100,
    )
    elapsed = time.time() - t0

    results.append({
        "frame_id": idx + 1,
        "filename": sample["filename"],
        "width": sample["width"],
        "height": sample["height"],
        "is_fallback": sample["is_fallback"],
        "latency_sec": round(elapsed, 3),
        "model": MODEL_CHOICE,
        "analysis": resp,
    })
    print(f"  Frame {idx+1}/{len(loaded_samples)}: {sample['filename']} analyzed in {elapsed:.2f}s")

df_results = pd.DataFrame(results)
csv_path = "marine_vlm_analysis.csv"
df_results.to_csv(csv_path, index=False)
print(f"\n[OK] Results exported to {csv_path} ({len(df_results)} records).")
if not df_results.empty:
    print(df_results[["frame_id", "filename", "latency_sec", "analysis"]])
else:
    print("No batch results available to summarize (num_samples=0).")

5.2 Visual Summary Display¶

We display each image alongside its generated identification summary.

if not loaded_samples:
    print("No samples to display (num_samples=0).")
else:
    fig, axes = plt.subplots(len(loaded_samples), 1, figsize=(8, 3.5 * len(loaded_samples)))
    if len(loaded_samples) == 1:
        axes = [axes]

    for ax, row, sample in zip(axes, results, loaded_samples):
        ax.imshow(sample["image"])
        summary_text = row["analysis"]
        if len(summary_text) > 80:
            summary_text = summary_text[:77] + "..."
        ax.set_title(f"{row['filename']} ({row['latency_sec']}s)\n{summary_text}", fontsize=8)
        ax.axis("off")

    plt.tight_layout()
    plt.show()

Part 6 - Discussion and Marine Science Applications¶

6.1 Architectural Tradeoffs: MLP vs Resamplers¶

The linear or 2-layer MLP projection module offers distinct characteristics compared to more complex visual bridges:

Information preservation: Because an MLP projects every visual patch token independently, no visual compression occurs. A 1000-patch image yields 1000 tokens for the language model. This maximizes fine spatial detail, which is critical for identifying small cryptic organisms (such as camouflaged flatfish or juvenile rockfish).
Context length scaling: In high-resolution or multi-frame video applications, preserving all patch tokens rapidly fills the LLM context window. Architecture variants such as Q-Former (Lesson 4.4) and Perceiver Resampler (Lesson 4.6) trade some fine spatial detail for fixed token budgets.
Training efficiency: Projector modules have minimal parameters (often 5 to 15 million parameters), enabling rapid pre-training on aligned image-text pairs while keeping vision and language backbones frozen.

6.2 Human-in-the-Loop Marine Survey Validation¶

When deploying vision-language models in automated ecological surveys:

Use structured JSON prompts to extract standardized Darwin Core biodiversity records.
Track confidence metrics and establish human review thresholds for rare or endangered species.
Pair zero-shot VLM outputs with traditional supervised object detectors to cross-validate taxonomic classifications.

References¶

LLaVA: Liu et al., Visual Instruction Tuning (NeurIPS 2023). https://arxiv.org/abs/2304.08485
InternVL 2.0: Chen et al., Expanding Performance Boundaries of Open-Source Multimodal Models (2024). https://arxiv.org/abs/2404.16821
Qwen2-VL: Wang et al., Qwen2-VL: To See the World More Clearly (2024). https://arxiv.org/abs/2409.12191
LILA BC Community Fish Detection Dataset: https://lila.science/datasets/community-fish-detection-dataset/