16Multimodal model evaluation · Read 4 min

One vector space for everything? Benchmarking CLIP, FashionCLIP and OpenAI

No model won on both tasks: FashionCLIP ranked similar images more accurately, while a text-specialized model understood written queries better.

Article context

The same model can excel at visual similarity while remaining less suitable for textual product search.

CONTEXT

This post presents a benchmark I built to evaluate whether one vector space could power both e-commerce search and recommendation. I compared Contrastive Language–Image Pre-training (CLIP) ViT-B/32, FashionCLIP and an OpenAI text-embedding model on two public catalogues with very different characteristics.

SUMMARY

The protocol separates two tasks: retrieving products from the same category using an image, and retrieving a category from text deliberately removed from the indexed data. FashionCLIP improves the former; the dedicated text encoder retains a clear lead on the latter. Catalogue size and structure also change absolute scores.

Key diagram

One model, two tasks, very different rankings

01 · IMAGE → IMAGE

Visual recommendation

CLIP0.618

FashionCLIP0.666

CLIP · H&M0.876

Fashion · H&M0.916

+7.9% FashionCLIP on Amazon

NDCG@10→HIGHER = BETTER

02 · TEXT → PRODUCT

Semantic search

CLIP0.076

FashionCLIP0.296

OpenAI0.622

CLIP · H&M0.556

Fashion · H&M0.638

OpenAI · H&M0.727

0.727 OpenAI · H&M

DECISIONOne encoder per taskFashionCLIP → image · OpenAI → text

The bars reproduce the documented NDCG@10 scores. The benchmark explicitly separates image similarity from text-to-product search to avoid a misleading average conclusion.

A research-based explainer grounded in the sources listed below. Some mechanisms are simplified without changing the overall conclusions.

Data-informed decision16

Can one model power both textual search and visual recommendation?

The benchmark gives a task-dependent answer: FashionCLIP improves visual neighbours, while the OpenAI text encoder leads semantic queries. I therefore recommended keeping two spaces until a hybrid benchmark justifies merging them.

Next validation

Human similarity judgments

Per-category results

Real multilingual catalogues

Takeaways

A useful benchmark separates tasks, prevents the answer from leaking into the index and compares models across multiple catalogues before simplifying the architecture.

Separate text search and visual similarity: model rankings change with the task.
Remove answer shortcuts and inspect neighbors alongside category-based metrics.
Keep two specialized spaces until a hybrid benchmark justifies replacing them.

01 — Experimental question

Evaluate visual recommendation and semantic search separately.

For visual recommendation, every image becomes a query in turn. Its nearest neighbours count as relevant when they belong to the same category. For semantic search, the category name is used as a query while being deliberately removed from indexed text: the model must recover it from product names, brands and descriptions alone.

Metrics include Precision@K, Recall@K and Normalized Discounted Cumulative Gain at ten results (NDCG@10), which rewards relevant results placed early in the ranking. Experiments use roughly 1,880 Amazon Berkeley Objects products and a 1,000-product H&M subset.

02 — Visual results

FashionCLIP improves the ranking of visually similar products.

On Amazon Berkeley Objects, NDCG@10 increases from 0.618 with CLIP to 0.666 with FashionCLIP, a relative gain of about 7.9%. On H&M, it rises from 0.870 to 0.916, or roughly 4.6%.

The gap between catalogues is as informative as the gap between models. A dense catalogue with many items per category makes relevant neighbours easier to retrieve. A smaller, fragmented catalogue produces a more demanding benchmark.

03 — Text results

Fashion specialization narrows the gap without replacing a purpose-built text encoder.

On Amazon Berkeley Objects, FashionCLIP raises CLIP’s NDCG@10 from 0.076 to 0.296, yet the OpenAI text model reaches 0.622. On H&M, FashionCLIP reaches 0.638, compared with 0.556 for CLIP and 0.727 for OpenAI.

The most plausible explanation lies in the training objective: CLIP aligns short captions with images, while a dedicated text embedding organizes passages and queries by semantic proximity. FashionCLIP adds a fashion-specific vocabulary, reducing the gap on H&M.

04 — Architecture decision

Keep one encoder per use case until simplification is validated.

The results support a two-space architecture: CLIP or FashionCLIP for image similarity, and a dedicated text model for search. A shared space remains attractive for hybrid queries, but it needs its own benchmark.

Category membership is only a proxy for relevance and does not fully measure perceptual similarity. The next step should add human judgments, real multilingual catalogues and per-category analysis to verify that the measured gains match the intended product experience.

Primary source

CLIP, FashionCLIP and text-embedding benchmark

Protocol, notebooks and comparison report produced as part of applied visual-search research.

Selected references

CLIP ViT-B/32 — Radford et al. (2021) ↗
FashionCLIP — Chia et al. (2022) ↗
Amazon Berkeley Objects and H&M datasets
Precision@K, Recall@K and NDCG@10