Perplexity Introduces Photon: A Rust-Based Retrieval Engine That Cuts p99 Latency From 800 ms to 65 ms

By

Perplexity has released Photon, an in-house retrieval and ranking engine written in Rust. It replaces an open-source engine Perplexity had forked for its AI-native search stack. Photon now handles retrieval and ranking for all production traffic. It also powers a new Fast Search mode in the Perplexity Search API. Perplexity reports single-call latency of 160 ms at p50 and 230 ms at p95.

Is it deployable? Yes, as a hosted API. Set search_type: "fast" on POST /search and pay $1 per 1,000 requests. Photon itself is not open source, so the engine cannot be self-hosted.

Why Perplexity Replaced its Old Engine

The old engine hit 3 limits as the index grew:

  • Tail latency: Production p99 sat near 800 ms. The dataset exceeded RAM, so mlock was not an option. Cold reads triggered major page faults that stalled queries.
  • Merge spikes: During disk index fusion, p99 climbed to about 1.2 s for 10 to 15 minutes.
  • Slow recovery: Deploying and syncing an extra cluster could take more than a week. Recovery also raised the share of partial responses.

Perplexity team concluded that building from scratch was simpler and cheaper than maintaining its fork.

How Photon Works

A load balancer routes each request to a Photon broker. The broker fans out to a shard group and watches for timeouts. Each shard runs retrieval, initial ranking, and second-stage ranking. The broker then merges candidates and fetches key document fields.

  • Adaptive posting lists: Short lists sit inline within a single page. Longer lists split into blocks of fixed document ID ranges. Sparse blocks store sorted offset arrays and use galloping search. Dense blocks use bitmaps, so membership becomes a single bit lookup.
  • Budgeted traversal: A WAND-like algorithm splits lists into driving lists and probe lists. Cheap presence checks bound each candidate’s maximum score first. Exact term frequencies are read only when a candidate can clear the threshold.
  • Docblob records: Each document gets a compact record of frequencies, field masks, and positions. Terms use Elias-Fano encoding, so ranking decodes only the matched terms. Ranking a candidate needs just 1 lookup per document.
  • Batched async reads: Record offsets are known upfront, so disk reads go out in batches through io_uring. The cache checks the whole batch first. Readers take no locks, and eviction uses CLOCK instead of a shared LRU list.
  • Separate build and serve: Indexers build versioned shard indexes from YTsaurus tables on dedicated nodes. A controller rotates serving groups one at a time and warms caches with replayed search-log queries.

A full web index now builds in a single-digit number of hours.

Interactive Explainer: Inside Photon

Production Results

  • p99 retrieval and ranking latency fell from about 800 ms to about 65 ms. This covers Photon’s stages only.
  • Photon runs on about 20% fewer serving machines than the old content nodes.
  • It stores about 2.5x as much data per document, which Perplexity used to improve ranking quality.
  • Pinning the same dataset with mlock would need an estimated 4.6x the resident memory Photon uses today.
  • Index version switches no longer cause latency spikes.

Fast Search: Speed and Cost for Agents

Fast Search pairs Photon with lighter ranking tuned for agentic workflows. Perplexity tested it on 6 benchmarks: WideSearch, BrowseComp, DSQA, FRAMES, SEAL-0, and SEAL-Hard. Across 3,554 tasks, Fast scored 64.3% at $59.73 in estimated model-plus-search cost. The default preset scored 64.0% at $187.60, so Fast was about 68% cheaper.

The trade-off shows up in broader search quality. On internal long-tail benchmarks, relevance (DCG) fell from 2.45 to 2.21. Answer availability dropped from 0.596 to 0.567, a loss of 2.9 percentage points. Perplexity recommends Fast for day-to-day agent loops and the default for hard, ambiguous queries.

Copy Code
curl -X POST 'https://api.perplexity.ai/search' \
  -H "Authorization: Bearer $PERPLEXITY_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"query": "latest stable Rust release", "search_type": "fast", "max_results": 5}'

On Python SDK 0.43.4 and 0.43.5, pass extra_body={"search_type": "fast"} per the docs.

Fast Search vs Closest Competitors

FeaturePerplexity Fast SearchExa InstantParallel Search TurboTavily ultra-fast
Request parametersearch_type: "fast"type: "instant"mode: "turbo"search_depth: "ultra-fast"
Vendor-reported latency160 ms p50, 230 ms p95 (blog)~250 ms typical (docs); sub-200 ms at launch~200 ms (docs)No figure published; lowest-latency depth (docs)
List price per 1K requests$1 (pricing)$4 for up to 10 results (pricing)$1 (docs)1 credit: $8 pay-as-you-go, $5 to $7.50 on plans (credits)
Results per request1 to 2010 in base price, $1 per 1K per extra resultNot specifiedNot specified
Known limitsLower relevance than default presetExtra results billed separatelyEnglish and Japanese queries onlyLower relevance than other depths
LaunchedSep 24, 2026Feb 12, 2026Jul 13, 2026 (blog)Jan 5, 2026 (blog)

All latency figures are vendor-reported under different setups, so they are not like-for-like.

Key Takeaways

  • Photon is Perplexity’s Rust retrieval and ranking engine, now serving all production traffic.
  • Production p99 latency dropped from about 800 ms to about 65 ms.
  • Fast Search reports 160 ms p50 and 230 ms p95 at $1 per 1,000 requests.
  • Fast cut estimated agent task cost by about 68% at comparable task quality.
  • It trades some retrieval relevance, so keep the default preset for hard queries.

Check out the technical details and Fast Search docs. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Perplexity Introduces Photon: A Rust-Based Retrieval Engine That Cuts p99 Latency From 800 ms to 65 ms appeared first on MarkTechPost.



from MarkTechPost https://ift.tt/fZd36IT
via IFTTT

Comments

Post a Comment

Popular posts from this blog

Microsoft AI Proposes BitNet Distillation (BitDistill): A Lightweight Pipeline that Delivers up to 10x Memory Savings and about 2.65x CPU Speedup

By
Image
Microsoft Research proposes BitNet Distillation , a pipeline that converts existing full precision LLMs into 1.58 bit BitNet students for specific tasks, while keeping accuracy close to the FP16 teacher and improving CPU efficiency. The method combines SubLN based architectural refinement , continued pre training , and dual signal distillation from logits and multi head attention relations. Reported results show up to 10× memory savings and about 2.65× faster CPU inference , with task metrics comparable to FP16 across multiple sizes. What BitNet Distillation changes? The community already showed that BitNet b1.58 can match full precision quality when trained from scratch, but converting a pretrained FP16 model directly to 1.58 bit often loses accuracy, and the gap grows as model size increases. BitNet Distillation targets this conversion problem for practical downstream deployment. It is designed to preserve accuracy while delivering CPU friendly ternary weights with INT8 activa...
Read more

Nanbeige4-3B-Thinking: How a 23T Token Pipeline Pushes 3B Models Past 30B Class Reasoning

By
Image
Can a 3B model deliver 30B class reasoning by fixing the training recipe instead of scaling parameters? Nanbeige LLM Lab at Boss Zhipin has released Nanbeige4-3B, a 3B parameter small language model family trained with an unusually heavy emphasis on data quality, curriculum scheduling, distillation, and reinforcement learning. The research team ships 2 primary checkpoints, Nanbeige4-3B-Base and Nanbeige4-3B-Thinking, and evaluates the reasoning tuned model against Qwen3 checkpoints from 4B up to 32B parameters. https://ift.tt/bMCdZs2 Benchmark results On AIME 2024, Nanbeige4-3B-2511 reports 90.4, while Qwen3-32B-2504 reports 81.4. On GPQA-Diamond, Nanbeige4-3B-2511 reports 82.2, while Qwen3-14B-2504 reports 64.0 and Qwen3-32B-2504 reports 68.7. These are the 2 benchmarks where the research’s “3B beats 10× larger” framing is directly supported. The research team also showcase strong tool use gains on BFCL-V4, Nanbeige4-3B reports 53.8 versus 47.9 for Qwen3-32B and 48.6 for Qwen...
Read more

Technical Deep Dive: Automating LLM Agent Mastery for Any MCP Server with MCP- RL and ART

By
Image
Table of contents Introduction What Is MCP- RL? ART: The Agent Reinforcement Trainer Code Walkthrough: Specializing LLMs with MCP- RL Explanation: Under the Hood: How MCP- RL Generalizes Real-World Impact and Benchmarks Architectural Overview Practical Integration Summary Introduction Empowering large language models (LLMs) to fluidly interact with dynamic, real-world environments is a new frontier for AI engineering. The Model Context Protocol (MCP) specification offers a standardized gateway through which LLMs can interface with arbitrary external systems—APIs, file systems, databases, applications, or tools—without needing custom glue code or brittle prompt hacks each time. Still, leveraging such toolsets programmatically, with robust reasoning across multi-step tasks, remains a formidable challenge. This is where the recent combination of MCP- RL (a reinforcement learning loop targeting MCP servers) and the open-source ART (Agent Reinforcement Trainer) library ...
Read more