seshank koganti

LiteLLM is an open-source proxy that connects your app to LLMs through one unified interface. Paired with Redis, it gives AI and ML teams a simple yet powerful way to unify access to LLMs, accelerate response times, and make AI apps real-time. Redis handles performance, memory, and data coordination that modern AI apps demand, while LiteLLM handles abstraction and routing. Our AI Product Marketing Manager Rini Vasan dives into: ▶️ What LiteLLM is ▶️ Why it works well with Redis ▶️ How LiteLLM and Redis work together Learn how to scale your LLM gateway with LiteLLM & Redis: https://lnkd.in/gqsTebF7

32

You are in a system design interview at Amazon for an SDE-3 role and the interviewer has given you the question to Design Netflix. He then asks a follow up to you: How does Netflix knows when to show: Are you still watching? Is it just time-based? If it were time-based, you would see it every time. So what is actually going on under the hood? Both look like a simple pop-up. Underneath, it is a mix of product thinking, client logic, and backend events. Btw, if you’re preparing for system design/coding interviews, check out our mock interview tool. You can use it for free here: https://lnkd.in/gpCn7t2T [1] Start from the product goals The feature is not only about nagging people. - Save bandwidth and CDN cost if the viewer has slept or walked away - Avoid autoplaying potentially sensitive content in an empty room - Protect kids if parents start a show and leave the TV on - Do all this without annoying active binge watchers So the design must answer one question: "When is the user probably not here any more" [2] Naive design - pure timer Simplest idea: - If playback has been running for 2 hours, show Are you still watching - If user clicks "Yes", reset the timer Why this is weak: - Someone can binge 5 episodes in a row and gets interrupted in the middle of an intense scene - Someone can start a show, walk away after 5 minutes, and the platform will keep streaming for the next 2 hours - It ignores how many episodes were auto played, device type, time of day, user habits  [3] Realistic design - session and engagement signals Think in terms of a "watch session" and "engagement events". Signals the client can track: - Play, pause, seek, volume change - Episode finished and next episode auto started - Remote or keyboard input, UI navigation - Screen on or off events from the device - For mobiles: app background or locked state A common heuristic could be: - Count how many episodes have auto played without any user interaction - Track how long it has been since the last button press or navigation - Only trigger the prompt when both are high enough Example rule: - If 3 episodes in a row have auto played - And there has been no interaction for 45 minutes - Then, before starting episode 4, show Are you still watching This feels much smarter: - Active viewers who pause, skip intro, change volume keep resetting the "engagement clock" - Sleeping viewers do not touch anything, so the next episode is blocked by the prompt  [4] Client heavy vs server heavy design You can talk about two design choices. Client first: - All logic runs inside the app on TV, mobile, or web - Client keeps an in memory session model and decides when to show the popup - It still sends telemetry events to backend for analytics and future tuning Pros: - Works even with flaky network - Highly responsive, no extra server round trip Cons: - Logic must be implemented and updated across many platforms - Harder to quickly roll out rule changes Continued ↓

184

12 Comments

JFrog Fly is redefining how engineering teams manage artifacts in an AI-driven world. As software delivery becomes faster, distributed, and increasingly automated by AI, traditional artifact management can’t keep up. JFrog Fly introduces a smarter, globally scalable way to store, replicate, and deliver artifacts—built for modern engineering velocity. 🔗 Read the full blog to explore what this shift means for your DevOps pipeline. Link - https://lnkd.in/dUK_UFd8 #JFrog #AI #DevOps #SoftwareDelivery #CloudNative

23

Tip: Scale large HNSW vector sets in Redis by splitting your dataset across multiple instances and querying them in parallel with multiplexing, enabling efficient parallel reads and writes that drastically improve performance and manage memory effectively. This approach leverages Redis's scalable vector support to handle hundreds of millions of vectors seamlessly. #Redis #HNSW #VectorSearch #ScalableAI #DataScience

1

When you join a new org or switch teams in the same, it is quite an unsettling and borderline anxious experience. Here are a few things that I did to ramp up faster. 1. I consciously made an effort to remain unblocked 2. I read a ton of code and its history, sometimes even unrelated 3. I asked questions about the past, the present, and the future 4. I took up relatively mundane tasks - cross-team work, tests, and docs 5. I extended a helping hand in whatever capacity I could 6. I proactively met skip-level leaders and managers to understand the vision Some companies do have a culture of pairing up with someone existing in the team, but even if you do not get a mentor, it is important that you still navigate the situation and ramp up as quickly as you can. The above list is not exhaustive, by any means, and hence you can always add things that you find helpful in your context. but the lowest common denominator is to show extreme intent and interest to get output and drive outcomes. Hope this helps.

995

25 Comments

Jyoti Bansal joins Mudassar Malik for the Season 4 premiere of The Builders Playbook to share an honest look at scaling without losing quality, speed, or accountability. They dig into Jyoti’s journey from engineer to founder, the investor question that pushed him to go all-in on AppDynamics, and the lessons he brings to building at Harness today. You’ll hear about: ✓ Turning one focused use case into a platform. ✓ Harness’s “startup within a startup” model. ✓ Why small, AI-enabled teams move faster. 🎧 If you’re building or scaling, this one’s packed with practical takeaways: https://lnkd.in/gFw-q9a6

7

Stop treating Load Balancers and Reverse Proxies as the same thing. In System Design interviews (and production incidents), the distinction matters more than you think. As we move from SDE1 to SDE2, we stop asking "How do I make this code work?" and start asking "How does this system scale and survive failure?" Here is the deep dive on two critical components that often get conflated. 1. The Load Balancer (The Scaler) Primary Goal: Availability & Horizontal Scaling. The Job: Spreading traffic across multiple compute resources to eliminate Single Points of Failure (SPOF). The SDE2 Nuance: You need to know the difference between Layer 4 (Transport) and Layer 7 (Application) balancing. L4 is fast; it forwards packets based on IP/Port without looking inside. L7 is smart; it terminates the connection, reads the HTTP headers/path, and routes /api differently than /images. 2. The Reverse Proxy (The Shield) Primary Goal: Security, Unification & Offloading. The Job: Sitting in front of backend servers to hide their topology and IP addresses. The SDE2 Nuance: SSL Termination. Decrypting HTTPS handshakes is CPU-intensive. A Reverse Proxy handles this heavy lifting at the edge, allowing your backend application servers to focus purely on business logic (and communicate via HTTP inside the private VPC). The Reality Check 💡 In modern architecture (and cloud environments like AWS ALB or tools like NGINX), the lines blur. We often use a single component to perform both roles simultaneously—terminating SSL (Reverse Proxy) and then distributing the request to a pool of instances (Load Balancer). But knowing which function you are tuning—and why—is what separates a mid-level engineer from a senior one. Questions for the network: Do you prefer handling SSL termination at the Load Balancer level or strictly on the application server for end-to-end encryption? #SystemDesign #SDE2 #SoftwareEngineering #Scalability #DistributedSystems #DevOps

66

1 Comment

Looking for a Kubernetes-native, serverless platform to run scalable, reliable event-driven applications, including inference on streaming data workloads? 👇 Hear directly from 𝐍𝐮𝐦𝐚𝐟𝐥𝐨𝐰 PM Sriharsha Yayi, Stuff Product Manager at Intuit and Krithika Vijayakumar, Senior Software Engineer at Intuit 🔥 𝐑𝐞𝐠𝐢𝐬𝐭𝐞𝐫 𝐟𝐨𝐫 𝐟𝐫𝐞𝐞 𝐟𝐨𝐫 𝐭𝐡𝐢𝐬 𝐚𝐧𝐝 𝐦𝐚𝐧𝐲 𝐦𝐨𝐫𝐞 𝐠𝐫𝐞𝐚𝐭 𝐬𝐞𝐬𝐬𝐢𝐨𝐧𝐬 𝐚𝐭 https://lnkd.in/eJkr7wqS 📆 𝐓𝐮𝐞𝐬𝐝𝐚𝐲, 𝐒𝐞𝐩𝐭𝐞𝐦𝐛𝐞𝐫 𝟐𝟑𝐫𝐝, @ 𝟒 𝐩𝐦 𝐂𝐄𝐓 | 𝟏𝟎 𝐚𝐦 𝐄𝐓 | 𝟕 𝐚𝐦 𝐏𝐓 Running inference on streaming data is complex. Teams must connect to event sources, process streams, and scale inference with minimal overhead. Numaflow, a Kubernetes-native open source platform, simplifies this by allowing teams to do stream processing and inference in the same pipelines natively on Kubernetes. It integrates with Kafka, Pulsar, SQS, and more to support diverse event sources and scales to handle high-throughput workloads. Numaflow (created by the creators of Argo Project) is currently in use by large enterprises. This talk shows how Numaflow enables ML engineers to achieve scalable stream processing and inference on streaming data without the infrastructure burden. #Numaflow #Kubernetes #InferenceStreaming #EDA #DevSecOps #KubeCrash #PlatformEngineering 

30

1 Comment