4.2 LocateAnything FirstPass Localization - Computer Vision Across the Marine Sciences

Learning Objectives¶

By the end of this lesson, you will be able to:

Understand Open-Vocabulary Grounding: Transition from closed-vocabulary detectors (such as the OceanCV FirstPass YOLO model) to open-vocabulary localization using NVIDIA LocateAnything (nvidia/LocateAnything-3B).
Sample ROV Video Transects: Download compressed deep-sea ROV transect footage and extract candidate frames at regular temporal intervals using OpenCV.
Master Non-Expert Morphological Prompting: Formulate visual descriptions in natural language that non-domain experts understand (describing visual morphology such as “large dark purple sea cucumber resting on seafloor” instead of scientific Latin binomials like Paelopatides confundens, “flattened white sea star”, and “translucent swimming jellyfish”).
Parse Structured Spatial Tokens: Extract coordinate tokens (<ref>, <box>) from autoregressive model output into pixel-space coordinates and normalized scales.
Export YOLO Annotations: Convert normalized detections into standard YOLO label format (<class_idx> <x_center> <y_center> <width> <height>).
Package Human-in-the-Loop Datasets: Create a compressed dataset zip archive containing extracted frames and candidate pre-labels for human review in annotation tools such as CVAT or Label Studio.
Evaluate Prompt Variations: Conduct structured exercises comparing prompt phrasing variations across taxonomy, descriptive granularity, substrate context, and distractor disambiguation.

The Paradigm Shift: From Fixed Categories to Open-Vocabulary Grounding¶

In Lesson 3.8, you utilized the OceanCV FirstPass YOLO detector to find candidate regions of interest. Traditional object detectors rely on a fixed classification layer: they only recognize pre-defined classes learned during supervised training. To detect a new deep-sea taxon, you must collect hundreds of bounding boxes, modify the model head, and retrain network weights.

Vision-Language Models (VLMs) fundamentally change this paradigm. Models such as NVIDIA LocateAnything project visual tokens and natural language embeddings into a shared multimodal space. Instead of being locked to pre-defined classes, LocateAnything locates any target described in free-form natural language text. In this lesson, we replicate the first-pass localization pipeline using LocateAnything, converting raw ROV transect imagery into verified candidate training datasets.

Part 1 - Environment Setup and Hardware Check¶

We install the necessary libraries for vision-language inference and computer vision processing:

transformers >= 4.40.0 and accelerate for model loading and pipeline execution
opencv-python-headless for video ingestion and frame sampling
pillow and matplotlib for image handling and bounding box rendering
pandas for structured tabular inspection of candidate detections

# Install required dependencies for LocateAnything and video processing
!pip install -q transformers>=4.40.0 accelerate pillow opencv-python-headless huggingface_hub pandas matplotlib

import os
import torch

# Verify GPU availability and memory
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

if device.type == "cuda":
    gpu_name = torch.cuda.get_device_name(0)
    vram_gb = torch.cuda.get_device_properties(0).total_memory / 1e9
    print(f"[OK] GPU: {gpu_name} | VRAM: {vram_gb:.1f} GB | Precision: {dtype}")
else:
    print("[INFO] No GPU detected. Running on CPU (inference will be slow).")

Part 2 - Frame Sampling from ROV Transect Video¶

Marine surveys record hours of continuous benthic video. Processing every consecutive frame at 30 fps is computationally redundant because adjacent frames show almost identical scenes.

We download a representative ROV benthic transect from Hugging Face (OceanCV/ROVTransectCompressed) and sample candidate frames every $N$ frames (e.g., sample rate = 32) using OpenCV.

import os
import urllib.request
import cv2

video_url = "https://huggingface.co/datasets/OceanCV/ROVTransectCompressed/resolve/main/transect_compressed.mp4?download=true"
video_path = "transect_compressed.mp4"

# Download the compressed ROV video transect if not present locally
if not os.path.exists(video_path):
    print("Downloading ROV transect video...")
    urllib.request.urlretrieve(video_url, video_path)
    print("Download complete.")
else:
    print(f"Using existing transect video: {video_path}")

# Create directory for extracted sample frames
subset_folder = "frames"
os.makedirs(subset_folder, exist_ok=True)

cap = cv2.VideoCapture(video_path)
sample_rate = 32  # Extract every 32nd frame
frame_count = 0
extracted_count = 0

print("Extracting transect frames...")
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    if frame_count % sample_rate == 0:
        frame_name = f"frame_{frame_count}.jpg"
        cv2.imwrite(os.path.join(subset_folder, frame_name), frame)
        extracted_count += 1
    frame_count += 1
cap.release()

print(f"Extracted {extracted_count} frames into '{subset_folder}/'.")

import matplotlib.pyplot as plt
from PIL import Image

sample_frame_path = os.path.join(subset_folder, "frame_0.jpg")
if os.path.exists(sample_frame_path):
    sample_img = Image.open(sample_frame_path).convert("RGB")
    plt.figure(figsize=(10, 6))
    plt.imshow(sample_img)
    plt.title(f"Sample ROV Transect Frame ({sample_img.width}x{sample_img.height})")
    plt.axis("off")
    plt.tight_layout()
    plt.show()
else:
    print("Sample frame not found. Verify video extraction.")

Part 3 - Non-Expert Morphological Prompting vs Scientific Taxonomy¶

Why Latin Binomials Fail in Web-Scale VLMs¶

Vision-Language Models learn multimodal associations from hundreds of millions of public image-caption pairs scraped from the internet. These corpora are authored by general internet users who write natural language descriptions of visual scenes (e.g., “purple sea cucumber”, “spiny red crab”, “white sea star”).

Specialized scientific Latin binomials (such as Paelopatides confundens, Benthopecten spinosus, or Bathynomus giganteus) appear with virtually zero frequency in general web datasets. As a result:

Asking a VLM for "Paelopatides confundens" produces zero detections or severe hallucinations because the text encoder lacks aligned visual tokens for the taxonomic string.
Asking the VLM for "large dark purple sea cucumber resting on seafloor" directly activates the model’s visual-semantic representations of color (dark purple), geometry (elongated sea cucumber), and substrate posture (resting on seafloor).

Guidelines for Formulating Non-Expert Morphological Prompts¶

When prompting LocateAnything for marine survey localization, think like a visual observer, not a taxonomist:

Scientific Taxonomy (Avoid)	Visual Morphology Prompt (Use)	Key Visual Cues Encoded
Paelopatides confundens	`"large dark purple sea cucumber resting on seafloor"`	Size, deep pigmentation, cylindrical shape, benthic contact
Benthopecten spinosus	`"flattened white sea star"`	Flat geometry, stark light color, radiating arm morphology
Scyphozoa / Solmissus	`"translucent swimming jellyfish"`	Optical transparency, gelatinous bell shape, pelagic posture
Lithodidae / Paralithodes	`"red spiny crab on sediment"`	Prominent coloration, exoskeleton spines, walking legs
Hexactinellida	`"branching white glass sponge"`	Porous texture, upright branching silhouette, high contrast

Part 4 - Loading NVIDIA LocateAnything-3B¶

NVIDIA LocateAnything is a 3-billion-parameter open-vocabulary grounding model. It ingests an image alongside target category strings separated by the </c> token delimiter.

The model autoregressively outputs structured text containing reference tags <ref>...</ref> and bounding coordinate tags <box><x1><y1><x2><y2></box>, where coordinates are normalized to the integer interval $[0, 1000]$ .

from transformers import AutoProcessor, AutoModel, AutoTokenizer

MODEL_ID = "nvidia/LocateAnything-3B"
print(f"Loading {MODEL_ID} processor, tokenizer, and weights...")

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID, trust_remote_code=True
)
processor = AutoProcessor.from_pretrained(
    MODEL_ID, trust_remote_code=True
)
model = AutoModel.from_pretrained(
    MODEL_ID,
    torch_dtype=dtype,
    _attn_implementation="sdpa" if device.type == "cuda" else "eager",
    trust_remote_code=True,
).to(device).eval()

print("[OK] LocateAnything model successfully initialized.")

Part 5 - Building Prompt Construction, Parsing, and Visualization Utilities¶

We implement the core components of the LocateAnything pipeline:

build_messages: Formats chat templates with categories joined by </c>.
parse_detections: Extracts coordinate tokens <box><x1><y1><x2><y2></box> and reference tags <ref> from the model’s textual output.
draw_detections: Renders semi-transparent bounding boxes and labeled tags on candidate frames.
locate_objects: Orchestrates inference with image scaling to prevent out-of-memory errors on GPU instances.

import re
import numpy as np
from PIL import Image, ImageDraw, ImageFont

# Deterministic color palette for bounding box overlays
PALETTE = [
    (8, 145, 178),   # Teal
    (220, 38, 38),   # Red
    (22, 163, 74),   # Green
    (37, 99, 235),   # Blue
    (217, 119, 6),   # Amber
    (147, 51, 234),  # Purple
    (234, 88, 12),   # Orange
    (13, 148, 136),  # Light sea green
]

def color_for_label(label: str) -> tuple[int, int, int]:
    """Assign a deterministic RGB color from the palette based on label text."""
    return PALETTE[sum(ord(c) for c in label) % len(PALETTE)]


def build_messages(image: Image.Image, categories: list[str]) -> list[dict]:
    """
    Build the multimodal chat message list for LocateAnything.
    Categories are joined with '</c>' as the delimiter token.
    """
    category_str = "</c>".join(categories)
    user_text = f"Locate all the instances that matches the following description: {category_str}."
    return [{
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": user_text},
        ]
    }]


def parse_detections(text: str, categories: list[str]) -> list[dict]:
    """
    Parse model output tokens into detection dictionaries.
    Coordinates are returned in normalized [0, 1000] coordinate space.
    """
    results = []
    expected = [c.strip().lower() for c in categories if c.strip()]
    current_label = None
    pattern = r"(<ref>.*?</ref>)|(<box>.*?</box>)"
    found_structured = False

    for m in re.finditer(pattern, text, flags=re.IGNORECASE | re.DOTALL):
        token = m.group(0)
        if token.lower().startswith("<ref>"):
            current_label = re.sub(r"</?ref>", "", token, flags=re.IGNORECASE).strip()
        else:
            content = re.sub(r"</?box>", "", token, flags=re.IGNORECASE)
            nums = re.findall(r"<\s*([0-9]+(?:\.[0-9]+)?)\s*>", content)
            coords = [float(n) for n in nums]
            if not coords:
                continue
            label = current_label or (expected[0] if expected else "object")
            if len(coords) == 4:
                results.append({"label": label, "type": "box", "coords": coords})
            elif len(coords) == 2:
                results.append({"label": label, "type": "point", "coords": coords})
            found_structured = True

    if found_structured:
        return results

    # Fallback parser for bare <box> tags without preceding <ref>
    box_pattern = r"<box>(.*?)</box>"
    parts = re.split(box_pattern, text)
    for i in range(1, len(parts), 2):
        preceding = parts[i - 1].lower()
        label = expected[0] if expected else "object"
        for cat in expected:
            if cat in preceding:
                label = cat
                break
        nums = re.findall(r"<\s*([0-9]+(?:\.[0-9]+)?)\s*>", parts[i])
        coords = [float(n) for n in nums]
        if len(coords) == 4:
            results.append({"label": label, "type": "box", "coords": coords})
        elif len(coords) == 2:
            results.append({"label": label, "type": "point", "coords": coords})

    return results


def draw_detections(image: Image.Image, detections: list[dict]) -> Image.Image:
    """
    Overlay semi-transparent bounding boxes and labeled badges on a PIL image.
    """
    img = image.convert("RGBA")
    overlay = Image.new("RGBA", img.size, (255, 255, 255, 0))
    draw = ImageDraw.Draw(overlay)

    try:
        font = ImageFont.truetype("DejaVuSans-Bold.ttf", 16)
    except Exception:
        font = ImageFont.load_default()

    W, H = image.size

    for det in detections:
        label = det.get("label", "object")
        color = color_for_label(label)
        c = det["coords"]

        if det["type"] == "box" and len(c) == 4:
            x1 = max(0.0, min(float(W), c[0] * W / 1000.0))
            y1 = max(0.0, min(float(H), c[1] * H / 1000.0))
            x2 = max(0.0, min(float(W), c[2] * W / 1000.0))
            y2 = max(0.0, min(float(H), c[3] * H / 1000.0))

            box_x1, box_x2 = min(x1, x2), max(x1, x2)
            box_y1, box_y2 = min(y1, y2), max(y1, y2)

            # Draw semi-transparent rectangle fill and solid bounding border
            draw.rectangle([box_x1, box_y1, box_x2, box_y2], fill=color + (60,), outline=color, width=3)

            # Draw text label banner
            try:
                tb = draw.textbbox((0, 0), label, font=font)
                tw, th = tb[2] - tb[0], tb[3] - tb[1]
            except Exception:
                tw, th = len(label) * 8, 14

            tag_y = box_y1 - th - 8 if box_y1 - th - 8 >= 0 else box_y2 + 2
            draw.rectangle([box_x1, tag_y, box_x1 + tw + 12, tag_y + th + 6], fill=color)
            draw.text((box_x1 + 6, tag_y + 3), label, fill="white", font=font)

        elif det["type"] == "point" and len(c) == 2:
            cx = max(0.0, min(float(W), c[0] * W / 1000.0))
            cy = max(0.0, min(float(H), c[1] * H / 1000.0))
            r = 8
            draw.ellipse([cx - r, cy - r, cx + r, cy + r], fill=color, outline="white", width=2)

    return Image.alpha_composite(img, overlay).convert("RGB")

print("[OK] Localization utilities compiled successfully.")

@torch.no_grad()
def locate_objects(
    image: Image.Image,
    categories: list[str],
    max_new_tokens: int = 2048,
    temperature: float = 0.7,
    top_p: float = 0.9,
) -> tuple[Image.Image, list[dict], str]:
    """
    Run NVIDIA LocateAnything on a single image with natural language categories.
    """
    w, h = image.size
    scaled_image = image.copy()
    if min(w, h) > 1024:
        scale = 1024 / min(w, h)
        scaled_image = scaled_image.resize((int(w * scale), int(h * scale)), Image.BILINEAR)

    messages = build_messages(scaled_image, categories)
    text = processor.py_apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    imgs, vids = processor.process_vision_info(messages)
    inputs = processor(
        text=[text], images=imgs, videos=vids, return_tensors="pt"
    ).to(device)

    pixel_values = inputs["pixel_values"].to(dtype)
    input_ids = inputs["input_ids"]
    attention_mask = inputs["attention_mask"]
    image_grid_hws = inputs.get("image_grid_hws", None)

    result = model.generate(
        pixel_values=pixel_values,
        input_ids=input_ids,
        attention_mask=attention_mask,
        image_grid_hws=image_grid_hws,
        tokenizer=tokenizer,
        max_new_tokens=max_new_tokens,
        use_cache=True,
        temperature=temperature,
        do_sample=True,
        top_p=top_p,
        repetition_penalty=1.1,
        verbose=False,
    )

    raw_text = result[0] if isinstance(result, tuple) else result
    detections = parse_detections(raw_text, categories)
    annotated = draw_detections(scaled_image, detections)
    return annotated, detections, raw_text

print("[OK] Inference wrapper locate_objects ready.")

Part 6 - Executing Open-Vocabulary Localization on Transect Imagery¶

We specify candidate organisms using non-expert morphological natural language descriptions:

"large dark purple sea cucumber resting on seafloor"
"flattened white sea star"
"translucent swimming jellyfish"
"red spiny crab on sediment"
"branching white glass sponge"

We execute localization on our sampled ROV frame, visualize detection boxes, and inspect the detected coordinates in a pandas DataFrame.

import time
import pandas as pd

# Define target candidate classes using descriptive visual morphology
MORPHOLOGICAL_CATEGORIES = [
    "large dark purple sea cucumber resting on seafloor",
    "flattened white sea star",
    "translucent swimming jellyfish",
    "red spiny crab on sediment",
    "branching white glass sponge",
]

print("Morphological query categories:")
for idx, cat in enumerate(MORPHOLOGICAL_CATEGORIES, 1):
    print(f"  {idx}. {cat}")

test_frame_path = os.path.join(subset_folder, "frame_0.jpg")
if os.path.exists(test_frame_path):
    target_img = Image.open(test_frame_path).convert("RGB")
    print(f"\nRunning LocateAnything inference on {test_frame_path}...")
    t0 = time.time()
    annotated_frame, detections, raw_output = locate_objects(target_img, MORPHOLOGICAL_CATEGORIES)
    elapsed = time.time() - t0
    print(f"Finished in {elapsed:.1f}s | {len(detections)} candidate detections found.")

    # Side-by-side visualization
    fig, axes = plt.subplots(1, 2, figsize=(16, 7))
    axes[0].imshow(target_img)
    axes[0].set_title("Original ROV Transect Frame")
    axes[0].axis("off")

    axes[1].imshow(annotated_frame)
    axes[1].set_title(f"LocateAnything Predictions ({len(detections)} found)")
    axes[1].axis("off")
    plt.tight_layout()
    plt.show()

    # Tabulate detections in pandas DataFrame
    if detections:
        W, H = target_img.size
        rows = []
        for i, d in enumerate(detections, 1):
            c = d["coords"]
            row = {"#": i, "label": d["label"], "type": d["type"]}
            if d["type"] == "box" and len(c) == 4:
                row["x1_px"] = int(c[0] * W / 1000)
                row["y1_px"] = int(c[1] * H / 1000)
                row["x2_px"] = int(c[2] * W / 1000)
                row["y2_px"] = int(c[3] * H / 1000)
            rows.append(row)
        df_results = pd.DataFrame(rows)
        print("\nDetected Entities Table:")
        print(df_results.to_string(index=False))
    else:
        print("No detections found for the given categories in this frame.")

Part 7 - Converting Coordinates to Normalized YOLO Format¶

LocateAnything outputs coordinates in the integer interval $[0, 1000]$ . Standard YOLO object detection format requires space-separated text records normalized to $[0, 1]$ :

\langle \text{class\_idx} \rangle \quad \langle x_{\text{center}} \rangle \quad \langle y_{\text{center}} \rangle \quad \langle \text{width} \rangle \quad \langle \text{height} \rangle

(1)

The coordinate normalization and bounding box conversion formulas are:

x_1' = \frac{x_1}{1000}, \quad y_1' = \frac{y_1}{1000}, \quad x_2' = \frac{x_2}{1000}, \quad y_2' = \frac{y_2}{1000}

(2)

x_{\text{center}} = \frac{x_1' + x_2'}{2}, \quad y_{\text{center}} = \frac{y_1' + y_2'}{2}

(3)

\text{width} = x_2' - x_1', \quad \text{height} = y_2' - y_1'

(4)

We implement detections_to_yolo and write candidate labels to disk.

def detections_to_yolo(detections: list[dict], category_to_idx: dict[str, int]) -> list[str]:
    """
    Convert LocateAnything [0, 1000] bounding boxes to normalized YOLO format strings.
    Format: <class_idx> <x_center> <y_center> <width> <height>
    """
    yolo_lines = []
    for d in detections:
        if d["type"] == "box" and len(d["coords"]) == 4:
            x1, y1, x2, y2 = [c / 1000.0 for c in d["coords"]]
            x_min, x_max = min(x1, x2), max(x1, x2)
            y_min, y_max = min(y1, y2), max(y1, y2)

            x_center = (x_min + x_max) / 2.0
            y_center = (y_min + y_max) / 2.0
            box_w = x_max - x_min
            box_h = y_max - y_min

            class_idx = category_to_idx.get(d["label"], 0)
            line = f"{class_idx} {x_center:.6f} {y_center:.6f} {box_w:.6f} {box_h:.6f}"
            yolo_lines.append(line)
    return yolo_lines

# Build category-to-index mapping
class_to_idx = {cat: i for i, cat in enumerate(MORPHOLOGICAL_CATEGORIES)}

# Setup directory for YOLO labels
labels_dir = os.path.join("localization", "first_pass", "labels")
os.makedirs(labels_dir, exist_ok=True)

# Generate and export YOLO labels for the sample frame
if "detections" in locals() and detections:
    sample_yolo = detections_to_yolo(detections, class_to_idx)
    label_file = os.path.join(labels_dir, "frame_0.txt")
    with open(label_file, "w", encoding="utf-8") as f:
        f.write("\n".join(sample_yolo) + "\n")
    print(f"Saved {len(sample_yolo)} YOLO annotations to '{label_file}':")
    for line in sample_yolo:
        print(" ", line)
else:
    print("Example YOLO label string format: 0 0.485120 0.621340 0.124500 0.082100")

Part 8 - Packaging Candidate Datasets for Human Review¶

Zero-shot VLM pre-labels provide an initial pass over raw survey footage. Before training an edge detector (such as YOLOv8 or YOLOv11), a human domain expert must verify the candidate detections, refine bounding box boundaries, and correct false positives.

We package the extracted frames and candidate YOLO label files into a compressed zip archive (dataset_for_labeling.zip). This archive can be uploaded directly to annotation tools such as Label Studio or CVAT.

import zipfile

def package_for_labeling(img_folder: str, label_folder: str, output_zip: str) -> None:
    """
    Package extracted video frames and candidate YOLO label files into a zip archive.
    """
    with zipfile.ZipFile(output_zip, "w", zipfile.ZIP_DEFLATED) as zipf:
        # Add frame images
        if os.path.exists(img_folder):
            for fname in os.listdir(img_folder):
                if fname.lower().endswith((".jpg", ".jpeg", ".png")):
                    fpath = os.path.join(img_folder, fname)
                    zipf.write(fpath, arcname=f"images/{fname}")
        # Add candidate label files
        if os.path.exists(label_folder):
            for fname in os.listdir(label_folder):
                if fname.lower().endswith(".txt"):
                    fpath = os.path.join(label_folder, fname)
                    zipf.write(fpath, arcname=f"labels/{fname}")

output_zip_name = "dataset_for_labeling.zip"
package_for_labeling(subset_folder, labels_dir, output_zip_name)
print(f"[OK] Candidate dataset packaged into '{output_zip_name}'.")
print("Download 'dataset_for_labeling.zip' to import into Label Studio or CVAT for human verification.")

Part 9 - Structured Exercises: Experimenting with Prompt Phrasing Variations¶

Because vision-language models ground language through open-vocabulary cross-modal attention, small adjustments in prompt phrasing produce measurable differences in localization recall and precision. Complete the following four structured exercises.

Exercise 1: Scientific Binomial vs Non-Expert Morphological Prompting¶

Objective: Measure the recall disparity between formal scientific taxonomy and lay visual descriptions.

Select an ROV frame containing a visible sea cucumber or echinoderm.
Run locate_objects using the Latin binomial:
- Taxonomy prompt: ["Paelopatides confundens"]
Run locate_objects on the same frame using descriptive visual morphology:
- Morphology prompt: ["large dark purple sea cucumber resting on seafloor"]
Record whether the model successfully detected the organism under each prompt.
Repeat the test for a sea star (["Benthopecten spinosus"] vs ["flattened white sea star"]) and a swimming jellyfish (["Solmissus"] vs ["translucent swimming jellyfish"]).

Discussion Question: Why does the model fail to recognize the Latin name despite being a multi-billion parameter foundation model?

Exercise 2: Descriptive Granularity and Specificity¶

Objective: Identify the optimal level of morphological detail for candidate pre-labeling. Test three tiers of descriptive granularity on the same frame:

Tier 1 (Single generic noun): ["sea star"]
Tier 2 (Color + Morphological noun): ["flattened white sea star"]
Tier 3 (Compound morphological and postural description): ["flattened white multi-armed sea star crawling on muddy seafloor sediment"]

Evaluation Criteria:

Does Tier 1 produce false positives (e.g., detecting pale rocks or sediment patches)?
Does Tier 3 overly restrict attention, causing the model to miss the organism?
Which tier produces the tightest, most accurate bounding box?

Exercise 3: Substrate and Contextual Conditioning¶

Objective: Use habitat and substrate context to filter ambiguous detections. In deep-sea video, water column organisms (pelagic) and benthic organisms (attached or crawling on sediment) require different survey treatment. Compare:

Unconditioned: ["jellyfish"]
Contextually conditioned: ["translucent swimming jellyfish in dark open water column"]
Benthic conditioned: ["sessile pale anemone anchored to exposed rocky substrate"]

Observe how explicit contextual keywords guide the model’s spatial attention away from seafloor clutter.

Exercise 4: Disambiguation and Distractor Suppression¶

Objective: Distinguish between target organisms and visually similar non-biological seafloor features. On deep-sea benthic plains, dark manganese nodules and basalt cobbles frequently mimic holothurians (sea cucumbers):

Baseline prompt: ["dark object on seafloor"]
Disambiguated organism prompt: ["elongated dark purple sea cucumber with smooth cylindrical body resting on sediment"]
Distractor prompt: ["angular dark gray basalt rock"]

Evaluate whether the model correctly assigns separate bounding boxes to the rock and the organism.

# Prompt Phrasing Experimentation Harness
# Modify the prompts below to test your hypotheses from Exercises 1 - 4

taxonomy_prompts = [
    "Paelopatides confundens",
    "Benthopecten spinosus",
    "Solmissus",
]

morphology_prompts = [
    "large dark purple sea cucumber resting on seafloor",
    "flattened white sea star",
    "translucent swimming jellyfish",
]

print("Experiment prompts initialized:")
print("Taxonomy set:", taxonomy_prompts)
print("Morphology set:", morphology_prompts)

# Uncomment the lines below when running in an active GPU environment:
# target_frame = Image.open(os.path.join(subset_folder, "frame_0.jpg")).convert("RGB")
# _, dets_tax, _ = locate_objects(target_frame, taxonomy_prompts)
# _, dets_morph, _ = locate_objects(target_frame, morphology_prompts)
# print(f"Taxonomy Detections: {len(dets_tax)} | Morphology Detections: {len(dets_morph)}")

Part 10 - Summary and Operational Takeaways¶

Key Principles of Open-Vocabulary Marine Grounding¶

Language as an Open Interface: Open-vocabulary models eliminate the constraint of fixed-index classification heads. You can query unannotated transect footage for rare or unmodeled taxa immediately without collecting training sets or fine-tuning weights.
Morphological Prompting is Mandatory: Web-scale models understand colloquial visual descriptions (color, shape, texture, posture) rather than scientific Latin taxonomy. Translating taxonomic taxa into non-expert visual descriptors is a critical domain adaptation skill for marine scientists.
Human-in-the-Loop Pre-Labeling: High-resolution VLMs generate candidate bounding boxes that drastically accelerate dataset creation. Exporting candidates to standardized YOLO format and packaging into zip archives enables human verification in tools like CVAT or Label Studio.

Operational Reflection Questions¶

When conducting a 100-kilometer ROV survey containing 200,000 frames, why is running an 8GB VLM on every frame impractical, and how does temporal sampling (e.g., every 32nd frame) mitigate this?
How does the quality of human verification in Label Studio affect downstream detector fine-tuning if the VLM produces a 15% false positive rate on manganese nodules?
In what scenarios would you choose to deploy a lightweight YOLOv8 detector over a 3B-parameter vision-language model on an autonomous underwater vehicle (AUV)?