# Late Chunking + Jina v4: Context-Aware Embeddings, Self-Hosted on GCP

**Authors:** [Pradipta Nanda](https://www.linkedin.com/in/pradipta-kumar-nanda-zeonai/) [Amlan Das](https://www.linkedin.com/in/amlan-das-a82b7a248/)  
  
*How we get retrieval that doesn't forget context, search that spans text and images, and full control over our embedding stack.*

If you've built a RAG system, you've hit the chunking problem. You split documents into passages, embed each one, and feed the top matches to an LLM. It works… until it doesn't — because the moment you cut a document into independent chunks, every chunk forgets everything around it.

We rebuilt our embedding layer around two ideas that fix this — **late chunking** and **Jina Embeddings v4** — and moved the whole thing onto our own **GCP GPUs**. Here's the what and the why.

![](https://cdn.hashnode.com/uploads/covers/6a69ec5ec58b56c90ccc7e94/b55f06f2-da13-4fcf-bb90-5f148f692d30.png align="center")

## The problem: naive chunking throws away context

The standard pipeline is *chunk → embed*: split a 20-page report into 300-token passages, embed each one in isolation, index the vectors.

The catch is that the model only ever sees the chunk you hand it. A passage that says *"this cut churn by 18%"* has no idea what "this" refers to — the antecedent was three chunks ago. Pronouns, defined terms, running topics, tables that straddle a boundary — all of it degrades retrieval. You end up with chunks that are locally fluent but globally clueless.

## Late chunking: embed first, chunk later

Late chunking (introduced by Jina AI) flips the order to **embed → chunk**:

1.  Run the **entire document** through a long-context transformer, producing a contextualized embedding for **every token** — each token now "knows" the whole document.
    
2.  **Then** apply your chunk boundaries and **mean-pool** the token embeddings inside each chunk.
    

Now every chunk vector is conditioned on the full document. "This" still carries the meaning of what it pointed to, even if that was pages earlier. Same chunk count, same index — dramatically more faithful retrieval.

Our service implements exactly this: it tokenizes the full text, encodes it (windowing long documents into overlapping spans with a stride so nothing is double-counted at the seams), returns **multi-vector** token embeddings, and mean-pools per chunk into the final vectors.

## Why Jina Embeddings v4

We standardized on `jina-embeddings-v4` because it checked every box we cared about:

*   **Multimodal** — text *and* images in one shared vector space. We can retrieve an image with a text query, or match a diagram to a paragraph, without bolting on a second model.
    
*   **Long context (~32K tokens)** — enough to embed whole documents in a single pass, which is what makes late chunking practical.
    
*   **Multi-vector output** — token-level (late-interaction / ColBERT-style) embeddings, not just one dense vector — so we can pool for chunks *or* keep fine-grained vectors for precise matching.
    
*   **Task-tuned & truncatable** — retrieval-specific behavior via query/passage prompts, plus Matryoshka dimensions to trade a little accuracy for a lot of storage.
    

One model — ~3.8B parameters — covering text, images, long context, and both dense and multi-vector retrieval.

## The migration: HuggingFace Spaces → our own GCP GPUs

We prototyped on **HuggingFace Spaces** — perfect for a demo, wrong for production. Shared and limited GPUs, cold starts, opaque batching, queueing under load, and our documents leaving our infrastructure.

So we moved it onto **GCP**, on a GPU node in our existing GKE cluster:

*   **Containerized** (Dockerfile + CI build-and-push) so it ships like every other service on our platform.
    
*   **Model pinned to a specific revision** — reproducible builds, no surprise upstream updates.
    
*   **A custom batching worker** — requests queue for a few milliseconds, then encode as a single batch on the GPU, with queries and passages on their own paths. This is where the throughput comes from.
    
*   **fp16 autocast on CUDA** — roughly halves memory and speeds up inference with negligible quality loss.
    
*   **Ephemeral weight cache** — the model is cached to local scratch at startup; nothing persistent to babysit.
    

Same model — now on infrastructure we control.

## By the numbers: HuggingFace Spaces vs. self-hosted GCP

Both platforms run the same GPU — **NVIDIA L4 (24 GB VRAM, Ada Lovelace)**. The difference isn't the chip; it's everything around it.

| Dimension | HuggingFace Spaces (L4) | Self-hosted GKE (L4) |
| --- | --- | --- |
| **GPU** | NVIDIA L4 · 24 GB VRAM | NVIDIA L4 · 24 GB VRAM |
| **Cold start / wake-up** | **3–5 min** after every sleep window | **< 1 s** — model lives in GPU VRAM |
| **Auto-sleep** | Yes — 1 hr inactivity timeout | None — pod always running |
| **Inference latency (warm)** | ~1–3 s / chunk | **0.14–0.8 s / chunk** ¹ |
| **Cost — always-on (24/7)** | ~$576 / mo ($0.80/hr) | ~$615 / mo on-demand; **~$286/mo on Spot** |
| **Cost — bursty (~8 hr/day)** | **~$192 / mo** (sleep saves the rest) | ~$286 / mo (Spot, always warm) |
| **Data residency** | Leaves your VPC | Stays in your GCP VPC |
| **Network path** | External HTTPS (CDN hops, TLS) | Internal cluster — sub-millisecond |
| **Custom batching** | ✗ HF-managed, opaque | ✓ Full control — queue, batch size, paths |
| **Model version** | ✗ Live from Hub — can silently drift | ✓ Pinned revision baked into image |
| **Uptime SLA** | None | GKE 99.5% control-plane SLA |

¹ *Measured live from our GKE deployment: short chunks ≈ 0.15 s, medium (~4 K chars) ≈ 0.22 s, long (~10 K chars) ≈ 0.58 s. Model permanently warm in VRAM; internal cluster call with no TLS overhead.*

**The honest trade-off:** If you only need embeddings during business hours and your data leaving your VPC is acceptable, HF Spaces wins on raw cost — ~$192/month versus our ~$286/month on Spot. The cold start (3–5 minutes after any hour of idle) is the dealbreaker for production indexing: any client with a shorter timeout errors out on the first request of the day, and you either need keep-alive pings or retry logic to paper over it. Once you need always-warm latency, data residency, and something you can actually batch-tune, the calculus tips the other way.

## What we get out of it

*   **Better RAG accuracy** — late chunking preserves the context naive chunking discards, so retrieved passages actually answer the question.
    
*   **One embedding space for everything** — text and images together means simpler pipelines and richer search.
    
*   **Whole-document understanding** — 32K context plus windowing lets long reports embed faithfully.
    
*   **Control** — latency, throughput, cost, and data residency are ours to tune. Batching keeps the GPU busy, the model version is locked, and documents never leave our VPC.
    
*   **It scales with us** — it's just another workload on our GKE GPU pool, deployed and observed like everything else.
    

## Takeaway

Chunking isn't a preprocessing detail — it's where most RAG systems quietly lose accuracy. **Late chunking** fixes the *what* (context-aware chunk vectors), **Jina v4** fixes the *scope* (multimodal, long-context, multi-vector), and **self-hosting on GCP** fixes the *how* (control, cost, privacy, throughput).

If you run RAG at any real scale, two questions are worth asking: *are my chunk embeddings context-aware?* and *do I actually control the model serving them?* For us, the answer to both is now yes.

*Building something similar? I'm always happy to compare notes.*