Rajat Singhal

Agentic Engineering: The Optimization Series (1/50) "How do you design an agentic system that handles 10k RPS without blowing your token budget or hitting 5s P99 latency?" This is the "Senior Architect" interview question for any LLM-native role. If your answer is "use a smaller model," you’ve already lost. The real answer lies in Tiered Intent Routing and Deterministic Sifting. [ I will keep posting set of intriguing topics faced by me ] The Problem: The Monolithic Prompt Trap Most developers cram 20 tool definitions and a 1k-token system prompt into an agent. Every single user "Hello" costs you 1,500 tokens and 2 seconds of LLM "reasoning" just to figure out the user is being polite. The Solution: The "Sifter & Router" Architecture To keep MS (Latency) low and Tokens lean, you must move the decision-making logic out of the LLM and into a high-performance Data Science layer. 1. The L1 Gate (Classic NLP/SLM) Instead of an LLM, use a Bi-Encoder (like all-MiniLM-L6-v2) or a fine-tuned DistilBERT classifier. Mechanism: Map user input to a vector space and compare it against "Gold Standard" intent embeddings using Cosine Similarity. Result: Simple intents (e.g., "Order Status," "Refund Policy") are routed to a Deterministic Script or a Small Language Model (SLM) like Phi-3. Performance: Latency drops from ~2000ms to <40ms. 2. The Dynamic Context Injector For the remaining "High-Reasoning" queries, don't pass all 20 tools. Use the L1 Gate's output to inject only the relevant tool schemas into the prompt. Constraint: Prompt_Tokens Proportional to Relevant_Tools Impact: You reduce the "distraction" for the LLM, increasing tool-calling accuracy while slashing the Input Token bill by 70%. 3. The "Stateful" Fallback If the L1 confidence is $< 0.7$, only then do you engage the "Frontier Model" (GPT-4o/Claude 3.5) to perform an "Ambiguity Resolution" step. Practical Systems Impact By implementing a Router-Worker pattern: Cost: You decouple cost from volume. 80% of traffic hits the $0.00/1k$ token local classifier. Throughput: Your system handles massive spikes because the "Heavy" LLM is only touched for 20% of requests. Reliability: You eliminate "Tool Hallucination" because the LLM only sees the 2 tools it actually needs. Resources [Got this from my fellow students] To master this architecture, I recommend studying these core concepts: RouteLLM (UC Berkeley): An open-source framework specifically for learning how to route queries between "strong" and "weak" models to optimize cost/performance. Semantic Router (Aurelio AI): A great library for implementing high-speed schema routing using embeddings rather than LLM reasoning. Classification vs. Generation: Read up on Natural Language Understanding (NLU) fundamentals specifically "Intent Classification and Slot Filling" to see how classic DS solves 80% of agentic "logic" faster than a Generative model. #AgenticAI #SystemsDesign #LLMOps #DataScience #RetailFoundry #ai #genai

28

3 Yorum

🔥 Real Madrid spent $80M on Ronaldo, and Meta spent $100M to land Jiahui Yu from OpenAI. But the real game-changer? Trapit Bansal, an IIT Kanpur grad who went from Accenture analyst to IISc researcher to UMass Amherst PhD. He’s worked at Google, Facebook, Microsoft, and OpenAI, helped build the o1 reasoning model, and now joins Meta’s Superintelligence Labs with a rumored $100M payday. Meta didn’t stop there. They’ve poached a wave of core OpenAI talent: - Shengjia Zhao (ChatGPT and GPT-4 co-creator) - Jiahui Yu (o3, o4 mini, GPT-4.1) = Shuchao Bi and Hongyu Ren (GPT-4o creators) Meta is building its own $100M dream team. This isn’t just a hiring spree. It’s a strategic knockout move to drain OpenAI of its key people and leap ahead in AGI. 🧠 What this means: Meta is rewriting the roster with superstar engineers OpenAI risks serious talent loss even as it focuses on retention The AI battlefield now includes billion-dollar sign-on bonuses Bottom line: From an Accenture cubicle to Meta’s Superintelligence Labs, this high-stakes transfer saga could reshape the future of AI. Who’s on your fantasy team?

22

2 Yorum

🎩 The Case for LLM Gallantry Posts have gone viral recently citing that manners create "noisy context" and that your "please" is costing millions in compute. I respectfully disagree. 1. Prime for 'Expert Clusters'—Sophisticated reasoning usually inhabits the same domain as polite, professional, discourse. 🔷 Bluntness or rudeness can direct a model toward “internet argument” weighting, encouraging it to cite equally toned sources. Gallantry primes the model to navigate toward ‘Expert Clusters’ such as academic journals and expert forums. 2. Reduce Entropy—Bluntness does not equal directness. Abrasive or chaotic syntax introduces irrelevant emotional tokens that compete for compute attention. 🔷 Clear gallant framing acts as a high-fidelity signal. It minimises entropy, ensuring the model's $K, V$ matrices are focused on your logic problem, and not your emotional tone. 3. Use the Reward Model—LLMs are trained using RLHF, harsh prompting risks triggering safety guardrails or defensive hedging. 🔷 Collaborative, chivalrous tone aligns with model's reward model, creating a fast-lane to complex reasoning capabilities. 4. Tonal Drift & The Preservation of Excellence—The period (.) is 1500 years old, but modern texting culture has caused its meaning to morph into a symbol of passive-aggression. 🔷 Cultural preservation of respectful communication, is not a burden, but a necessary exercise of selflessness, lest your uncouth LLM-speak drift into your real-world communication. Manners maketh man. #AI #PromptEngineering #GenerativeAI #TechTips #LLM #Etiquette I used em dashes before they were cool.

4

Was reading a bit on serving LLMs and space is rapidly evolving. Found vLLM (Liang et al., 2023) quite interesting. It introduced a mechanism called PagedAttention, which virtualizes the KV (key-value) cache by managing it like a paged memory system. In transformer models, the KV cache stores intermediate attention values for previously seen tokens—essential for generating the next token without recomputing everything from scratch. Instead of keeping this entire cache in GPU memory, which grows linearly with prompt and response length, PagedAttention evicts and loads cache blocks on demand. This enables long-context inference without exhausting GPU memory and is a foundational step toward memory-aware serving. Google’s Pathways architecture takes a related route, by conditionally activating only the parts of a model relevant to a given input—known as expert routing. This sparsely activates subsets of model parameters (called "experts") rather than loading the full model, saving compute and memory. Although these architectures focus on model modularity rather than serving infra modularity. Microsoft’s DeepSpeed Inference and MII optimize serving by reducing the resource load of each inference phase and employ techniques such as KV cache sharing, where multiple concurrent requests can reuse the same cache entries instead of duplicating memory; and activation offloading, which temporarily moves intermediate computations (activations) from GPU memory to CPU or host memory during inference to avoid exceeding memory limits. These optimizations allow each GPU to serve more concurrent decode requests without degrading throughput or accuracy. NVIDIA’s Dynamo is currently one of the most complete implementations of fully disaggregated inference. It explicitly splits prefill and decode phases into different GPU pools. A Smart Router component directs decode requests to the GPUs where the relevant KV cache already resides, minimizing cache transfers or recomputation. A separate KV Cache Manager coordinates cache eviction and loading across a hierarchy of memory layers—including GPU VRAM, CPU RAM, SSD, and even networked object stores—similar to how virtual memory systems operate. Meta’s LlamaServe, built on Triton Inference Server, is still in early stages but is beginning to explore smarter scheduling and batching of requests. Across all of these systems, a common trend is taking shape. Inference is no longer treated as a uniform, atomic task. If you're working on inference infrastructure, I’d love to hear more on this from you! #inference

8

Getting into YC is easy. Only ~2% of applicants get in. Indian Railways? Try 0.3%. In the last drive, ~30 million people applied for 90,000 jobs. Odds can be 1,800 to 1. Six years of study is not unusual. Here’s the key difference: A YC application takes days. The building takes years. Years in which you’re creating something of value, whether YC accepts you or not. Exam prep takes years of cramming obscure facts. If you miss the cut‑off, you might emerge a decade later with no job and no work experience. Invest your best years in building something that compounds: a product, a company, and a skill set no one can take from you.

16

7 Yorum

An Indian startup just open-sourced a 105B model that outperforms DeepSeek R1 on math and agentic benchmarks. Trained entirely in India. On Indian government GPUs. With $53M in funding. Not $5B. And the weights are live on Hugging face right now: https://lnkd.in/djPhwvEB Here's why Sarvam 105B deserves your attention: The benchmarks that matter: → 88.3 on AIME 2025 (DeepSeek R1: 87.5) → 96.7 on AIME with tools → 49.5 on BrowseComp (DeepSeek R1: 3.2 — not a typo) → 68.3 on Tau2 agentic benchmark (beats o4-mini, DeepSeek R1, and Claude Sonnet 4) → 98.6 on Math500 These aren't cherry-picked comparisons against small models. They're head-to-head with DeepSeek R1, Gemini 2.5 Flash, o4-mini, and Claude Sonnet 4. The backstory makes this even more impressive. Last year, Sarvam caught heavy criticism for fine-tuning Mistral's model and calling it sovereign AI. Downloads were underwhelming. People questioned whether India could build foundational models at all. So they went back and trained from scratch. 12 trillion tokens for the 105B. 16 trillion for the 30B. 128 experts in a Mixture-of-Experts architecture. Custom tokenizer, custom kernels, custom RL pipeline. They threw out KL-divergence regularization entirely — broke from the textbook approach and it worked. The 30B model is equally wild — only 2.4B active parameters. It matches models 10x its active size on coding and reasoning benchmarks. This is what efficient scaling actually looks like. What makes this a signal, not just a model drop: ✅ Trained on IndiaAI Mission compute — sovereign infrastructure, not rented from US hyperscalers ✅ Supports all 22 official Indian languages natively ✅ Already in production — Samvaad (conversational agents) and Indus (reasoning/agentic workflows) ✅ Open-sourced under Apache License on Hugging face ✅ Demoed running on a feature phone with physical buttons That last one hit differently. A 105B-parameter model. On a dumbphone. The AI race isn't just US vs China anymore. A team of ~114 people in Bengaluru just shipped a globally competitive reasoning model on government GPUs, at a fraction of frontier lab costs. And they gave the weights away for free. Download it. Fine-tune it. Ship something with it. Hugging face link: https://lnkd.in/djPhwvEB What's your take — can sovereign AI models compete long-term with frontier labs? 👇 🔄 Repost if your network needs to see what India is building. #AI #OpenSource #SarvamAI #indi

189

4 Yorum

Most SDEs assume they’ve already done enough to level up. → They’ve worked on production systems. Solved real engineering problems. Operated in fast-moving teams. On paper, that should qualify them for the next step. Sounds sufficient, right? NOT REALLY. Because SDE-2/3 interviews don’t evaluate your past work the way you expect. They evaluate how you think, live, under constraints. And that changes everything. DSA still shows up first. And for many, that’s where the process ends, before System Design is even discussed. What actually makes the difference 👇 🔸 Stop treating DSA like a numbers game: Volume doesn’t translate to readiness. Pattern recognition does. Focus on how problems are structured, not how many you’ve solved. 🔸 Don’t postpone system design: Waiting until later is the usual mistake. Strong candidates build problem-solving and design intuition together. 🔸 Practice in interview conditions: Mocks expose hesitation, gaps, and communication issues that solo prep hides. The transition from SDE-1 to SDE-2/3 roles isn’t about writing cleaner code alone. It’s about demonstrating structured thinking, articulating decisions clearly, and breaking down unfamiliar problems quickly. That’s what companies are really screening for. So, when you’re managing prep alongside a full-time job, random effort won’t make any difference, you need direction. And that’s where Bosscoder’s Transformer program comes in, built to help you focus on what actually moves the needle, without wasting time on the wrong things. 🔗 Explore Transformer here: bcalinks.com/COZj4LK They’ve helped 2200+ engineers crack top PBC roles through: 💡 Structured curriculum to help you master DSA, System Design, and AI 💡 Real world, AI-powered projects & 24/7 doubt-solving 💡 Leadership skills for Senior roles with 1:1 mentorship from industry experts 💡 Resume reviews, mock interviews, and 100% job switch support. #dsa #sdeprep #pbc #tech #collab

428

54 Yorum

⚙️ One library, real DSP speed. There’s a gap in the developer ecosystem. Teams doing DSP/ML often migrate to Python, because that’s where most tooling lives. But Python wasn’t built for high-throughput I/O — under load, it spawns new interpreters/processes, creating duplicated memory, slow cold starts, and more containers. Compute stays fast, but orchestration cost explodes. Meanwhile, Node.js/TS scale I/O well, but were never meant for serious compute. Pure-JS DSP hits performance walls. Even worse, serializing data between Redis ↔ DSP/ML adds latency and forces batching — slowing real-time pipelines. So devs are stuck: Python → strong math, weak concurrency Node → strong networking, weak compute dspx closes that gap — TypeScript DX + native C++/SIMD (AVX2/SSE3/SSE2/NEON) under the hood. 🧩 How dspx works FIR / FFT / Conv1D run in optimized C++/SIMD Memory-safe circular buffers → O(1) throughput TS handles Kafka / Redis / WebSockets Redis persistence < 0.5 ms Batched logging avoids I/O stalls At tiny batch sizes, N-API overhead can make naive JS look competitive. At realistic scale, batched native pipelines win. ⚡ Benchmarks Dell OptiPlex 3000 Micro (i5-12600T · AVX2 · Node 22) FFT: 2.6× faster than fft.js, ~9× faster than tfjs-node FIR (51-tap): ~4× faster than fili / naive JS Conv1D (128-kernel): ~3.5× faster than tfjs-node Moving Avg (O(1)): 559× more throughput/sec than naive JS Redis save/load: sub-ms Logging: <3% overhead 📊 Full tables + charts in carousel 💬 Observations SIMD FMA + contiguous memory → big FIR/Conv wins O(1) moving-average design → massive throughput gain Sub-ms Redis + low I/O overhead → real-time persistence in pure Node 🧠 Open Source dspx is Apache 2.0, free for commercial/academic use. Looking for contributors interested in: ARM NEON / Apple M / Graviton tuning Audio / sensor / biomedical DSP Visualization + benchmark tooling 📦 npm i dspx 🔗 https://lnkd.in/e-tWxAgu 🧠 Real-time DSP for Node, TypeScript & Redis 💼 Note I’m exploring opportunities in full-stack, real-time systems, performance engineering, and DSP. If your team works in this space, I’d love to connect. My next post will cover why sub-ms Redis latency isn’t just technical — it’s economic. Lower serialization + compute overhead → lower infra + energy cost. 🔖 Tags & Mentions #NodeJS #TypeScript #DSP #PerformanceEngineering #EdgeComputing #OpenSource #SIMD #Redis #Cplusplus #RealTime NodeJS Developer TensorFlow Google Microsoft JavaScript Developer Amazon Web Services (AWS) Vercel

4

1 Yorum

We don't need that much focus on building LLMs, There are far more optimal things happening instead of just diving into a money-burning race. People don't realize it, but India has the largest surplus of software engineering talent. For the first time, the risk of starting something and the resources required to start have gone down so much so that people can finally build. In recent days, I have seen significantly more Indians, yes, us, building something new than anybody else in the world. It's happening.

114

7 Yorum

RAG vs Fine-Tuning for LLMs (2026): What Actually Works in Production RAG vs fine-tuning in 2026 explained with real tradeoffs, latest trends, and a practical decision framework for production LLM systems. What you'll find inside: - RAG - Fine-Tuning - LLM Engineering Curious how you'd approach this. Drop your take in the comments. Read the full post: https://lnkd.in/gKBrQs2M #RAG #FineTuning #LLMEngineering #AIArchitecture #GenAI2026

16

1 Yorum

🤗 Good news for builders in India. Supabase is now fully accessible across the country. The earlier network connectivity issue has been resolved and IP-level restrictions have been removed. All core components are operational: - Compute capacity - ap-south-1 - Auth - Edge Functions If you were affected, try clearing your DNS cache or restarting your network. ISP-side propagation may take up to 24 hours. This incident is also a reminder - infrastructure dependency risk is real. Always think about redundancy, monitoring, and fallback plans. Back to shipping.

5

1 Yorum

Context engineering → better tools for agents (not just better retrieval/RAG). Traditional retrieval works well on pointed questions over chunks/snippets. But struggles with holistic cross-document questions, forcing you to stuff entire docs into context. We need llms.txt for documents 🤖🗺️. Here's a demo inspired by talks and conversations at @aiDotEngineer: A "document navigator" AI agent (with Cursor + MCP) browsing a 250-page US Govt. document to answer: "summarize all parts of the document about US government debt". (skim to 1:10 and 2:10 for just the demo) Under the hood: - Contextual AI /parse API + auto-generated document_metadata.hierarchy (as llms.txt) - Cursor's agent loop + navigation tools via MCP - Verifiable attribution via interpretable tool call traces with rationales This simple demo uses purely “navigation” tools on one 250 page doc. But combining with solid traditional retrieval can scale context for agents to 10-100x more than can fit in context - while keeping all the agentic goodness intact 🚀. The bigger picture: LLMs can read, summarize, and index various cross-document metadata. The pattern emerging is: Index-time compute → smarter tools → more capable agents. Chat with the US Govt FY 24 Financial report in Cursor yourself: https://lnkd.in/gpvqUbqP

66

4 Yorum

The most ironic part of all this is, the subject they deem only theoretical often define our real world. AI functioning on matrices and theorems? That's not something which you can build or realise. It's a concept, a method, a thought process. And that very thought process now impacts the whole world. Pure Science, Mechanical Engineering, Aeronautics, Food Technology, Philosophy, Psychology, all these supposed niche fields require huge brain power, intellectual capability and creativity. The applied fields and concepts are nothing without their theoretical part. We must remember that.

6

🚨 If developer infrastructure can be blocked overnight… what does that mean for builders? Many users across India are currently unable to access Supabase. No notice. No migration window. Apps suddenly break. When infrastructure platforms get blocked at ISP/DNS level: → Student projects break → Startups lose customers → Production systems get affected → Trust in the ecosystem shakes This isn’t about one company. It’s about stability, transparency, and protecting the developer ecosystem. For now, some possible workarounds 👇 • Switch DNS to Cloudflare (1.1.1.1) or Google (8.8.8.8) • Use a VPN • Enable DNS proxying via a custom domain India wants to lead in tech. Let’s protect the builders powering it. #Supabase #IndianDevelopers #StartupIndia #BuildInIndia #DeveloperCommunity #TechPolicy

8

There is significant unmet demand for developers who understand AI. At the same time, because most universities have not yet adapted their curricula to the new reality of programming jobs being much more productive with AI tools, there is also an uptick in unemployment of recent CS graduates. When I interview AI engineers — people skilled at building AI applications — I look for people who can: - Use AI assistance to rapidly engineer software systems - Use AI building blocks like prompting, RAG, evals, agentic workflows, and machine learning to build applications - Prototype and iterate rapidly Someone with these skills can get a massively greater amount done than someone who writes code the way we did in 2022, before the advent of Generative AI. I talk to large businesses every week that would love to hire hundreds or more people with these skills, as well as startups that have great ideas but not enough engineers to build them. As more businesses adopt AI, I expect this talent shortage only to grow! At the same time, recent CS graduates face an increased unemployment rate, though the underemployment rate — of graduates doing work that doesn’t require a degree — is still lower than for most other majors. This is why we hear simultaneously anecdotes of unemployed CS graduates and also of rising salaries for in-demand AI engineers. When programming evolved from punchcards to keyboard and terminal, employers continued to hire punchcard programmers for a while. But eventually, all developers had to switch to the new way of coding. AI engineering is similarly creating a huge wave of change. There is a stereotype of “AI Native” fresh college graduates who outperform experienced developers. There is some truth to this. Multiple times, I have hired, for full-stack software engineering, a new grad who really knows AI over an experienced developer who still works 2022-style. But the best developers I know aren’t recent graduates (no offense to the fresh grads!). They are experienced developers who have been on top of changes in AI. The most productive programmers today deeply understand computers, how to architect software, and how to make complex tradeoffs — and who additionally are familiar with cutting-edge AI tools. Sure, some skills from 2022 are becoming obsolete. For example, a lot of coding syntax that we had to memorize back then is no longer important, since we no longer need to code by hand as much. But even if, say, 30% of CS knowledge is obsolete, the remaining 70% — complemented with modern AI knowledge — is what makes really productive developers. Without understanding how computers work, you can’t just “vibe code” your way to greatness. Fundamentals are still important, and for those who additionally understand AI, job opportunities are numerous! [Original text: https://lnkd.in/gHSWcKhr ]

10.863

641 Yorum

Here’s exactly how I would crack a $150K+ SWE job at Microsoft in 6 months. (A real strategy my client used to land interviews at Microsoft, Meta, and Stripe) Too many engineers prepare hard to crack MAANG. But they lack a solid strategy. Here’s the exact roadmap I'll follow if I wanted to land a SWE job at Microsoft. Step 1: Resume + Role Clarity → Reverse-engineer the JD Study at least 10+ Microsoft SWE job listings. Highlight recurring keywords, must-haves, and preferred tools. → Rewrite your resume like a product pitch Show measurable impact: Good: “Improved load time by 43%” Bad: “Worked on performance optimization.” → Make it keyword-optimized for ATS without sounding robotic. Step 2: Master DSA the Microsoft Way → Focus on patterns, not problems Microsoft LOVES: Sliding window Trees (esp. DFS/BFS) Graphs Dynamic programming → Suggested Platforms: Leetcode (Microsoft tag), Neetcode(dot)io roadmap, Grokking series on Educative → 2 problems/day + 1 mock interview/week = compounding prep Step 3: System Design (yes, even for junior roles) → Start with High-Level Design (HLD): Learn how to design APIs, caching, rate limiting, and DB scaling. → Build a project where you actually implement what you learn. Step 4: Microsoft-Specific Behavioral Prep → Microsoft uses structured behavioral interviews Focus on “3As” framework: Action → Approach → Aftermath → Use real projects to show: Collaboration Adaptability Customer focus Engineering rigor → Prep 8–10 STAR stories mapped to their values Step 5: Mock + Real-World Practice → Do 4–5 peer or mentor-led mock interviews (especially for behavioral + design) → Record yourself. Watch for filler words, unstructured answers, lack of metrics. → Apply for 5 roles/week—Microsoft + similar-sized companies (to get into interview flow) Step 6: Apply Strategically + Use Referrals → Connect with 3–5 engineers/recruiters/week on LinkedIn. Don’t ask for a referral right away - engage first. → Reach out using tailored messages like: “Hi [Name], I’ve been preparing for an SWE role at Microsoft. Your journey from [X] to [Microsoft] really stood out. If you're open, I’d love to learn more about your experience.” → Submit applications using referrals whenever possible + 1-click apply where relevant The client who followed this playbook now works at Microsoft Azure and had 3 competing offers before accepting. It’s not about doing everything. It’s about doing the right things in the right order. Give yourself 6 months and follow this roadmap to make it a reality. Share this with someone who dreams of working at Microsoft. P.S. Follow me if you are a tech job seeker in the U.S. I share practical advice that gets you hired. — Additional resources: https://lnkd.in/eKxQmYtP. https://lnkd.in/gwRWpXR9. https://lnkd.in/g94_Cziv

100

4 Yorum

There’s something broken in how we train engineers in India Especially in Computer Science And it shows Most fresh CS grads today are unemployable - not because they’re not smart, but because they’ve been poorly trained and terribly advised At 1811 Labs, we have worked with, interviewed and reviewed assignments of many young engineers. Here’s what I’ve seen up close: - They jump from tech to tech: React today, Go tomorrow, maybe a bit of AI on the side. No depth, just surface-level noise - Fundamentals? Weak. Code quality? Worse. Best practices? Never heard of them - AI: Blindly use AI without a clear understanding of the problem/use-case making things even worse - Communication? Most can’t explain what they built, let alone collaborate on a team You might argue, “An engineer just needs to build” But how do you build anything meaningful if you can’t explain your thoughts, write clean code, or understand a problem deeply? The saddest part? They still expect a job just because they have a degree And honestly, I don’t blame them That’s what the system promised But today, skills > degrees The market doesn’t care where you studied It cares what you can do So if you’re a junior dev, here’s the roadmap I wish more people followed: - Build rock-solid fundamentals - Pick one stack. Get decent at it before you jump around - Write clean, readable code. Follow best practices - Do real projects, not just for resumes, but to understand what “good” actually looks like - And please, don’t sleep on communication. It’s your second most important skill after coding Trust me, without solid understanding and skill, AI Agents will eat away at most low and mid-level dev jobs. We are slowly transitioning into an era where engineers will engineer and architect systems supervise progress and fix issues, while AI will do the heavy lifting. A tier-1 degree might fast-track access But a tier-1 skillset is what’ll fast-track growth We don’t need more engineers We need better ones

82

12 Yorum