Appen

Great discussion at SlatorCon London 2026 around the evolving role of human expertise in AI development. One theme that came up repeatedly: as models become more capable, the bottleneck is no longer just data volume. It’s evaluation quality, domain expertise, and the ability to generate meaningful feedback signals for increasingly complex systems. A few areas Sergio Bruccoleri, VP, Delivery, Appen, touched on during the panel: • Evaluation is shifting beyond static benchmarks toward more dynamic, environment-based assessment • Domains like coding, STEM, legal, and finance require deeper subject matter expertise to properly evaluate reasoning quality and edge cases • As agentic systems evolve, high-quality human feedback becomes increasingly important for alignment, reliability, and model behavior in production settings Interesting conversations across the broader language AI ecosystem on where training, evaluation, and human-in-the-loop systems are headed next.

Appen recently completed an independent third-party evaluation of Subquadratic's SSA (Sparse Self-Attention) kernel. The core architectural claim: replacing O(n²) full self-attention with a learned sparse formulation that routes computation only to the most relevant key-value blocks, enabling near-linear scaling as context length increases. What we measured: Efficiency (NVIDIA B200, bfloat16, PyTorch 2.11.0) - 56.2× wall clock speedup vs. FlashAttention-2 at 1M tokens - 62.8× FLOP reduction vs. dense attention at 1M tokens - FLOP counts independently validated via torch.profiler (within 0.7–3.9% of theoretical) Long-context retrieval - RULER at 128K tokens - 95.6% average score across all evaluated tasks (LLM-judged via Claude Opus 4.6) - Perfect retrieval on all single-needle tasks Ultra-long context - MRCR at 512K–1M token context lengths - 86.2% average score on the hardest 8-needle retrieval bucket Coding - SWE-Bench Verified - 81.8% resolved rate with extended thinking enabled Evaluation was conducted independently with access scoped to API endpoints only. No model weights, training data, fine-tuning configurations, or ground-truth labels were provided in advance. The efficiency scaling results are particularly notable. Full report: https://lnkd.in/e-Q4FrEt #LongContextLLM #AttentionMechanism #AIBenchmarking #SparseAttention #NLP #MachineLearning

SOTA ASR models report near-human accuracy on public test sets. So why do they still fail users in the real world? The answer isn't the models, it's the benchmarks. Widely used benchmarks like LibriSpeech are built on clean, scripted, accent-narrow speech. Real production speech is spontaneous, noisy, accented, and multi-speaker. That gap doesn't show up on a leaderboard. It shows up when your product ships. There's also a compounding problem: benchmaxxing. Models optimised to climb public leaderboards don't generalise; they're tuned to the test, not the task. Our new whitepaper lays out how to fix this. We cover: → Why current benchmarks systematically overstate real-world ASR performance → The evidence: WERs that jump from ~12% on read speech to 42% on casual conversation → Our 5-stage methodology for building production-representative speech benchmarks (scoping → contributor sourcing → speech design → recording → transcription) → How private, held-out benchmark sets resist benchmaxxing → Our partnership with Hugging Face to make the Open ASR Leaderboard a more trustworthy signal If you're building or evaluating ASR systems, this is the benchmarking gap your eval stack may not be surfacing. Read the full whitepaper [link in comments] #SpeechRecognition #ASR #AIBenchmarking #NLP #SpeechAI #MachineLearning #DataQuality

Good to see Zoom Scribe API is competitive for real world speech recognition benchmarks.

Data makes all the difference. Especially when it reflects real production conditions. One of the biggest challenges in AI today is what we call “benchmaxxing” — training and testing models on the same public datasets, only to see performance drop once models hit the real world. That’s exactly why Appen partnered with Hugging Face on the #OpenASR Dashboard initiative. As AI solutions become more productized and customer-facing, production readiness matters more than leaderboard scores. Models need to perform under shifting datasets, noisy environments, and real operational constraints — not just in controlled benchmarks. In this article, we share: • How Appen contributes to the OpenASR Dashboard • Our methodology for building evaluation datasets • Why production-grade data is critical for trustworthy AI benchmarking • How we help ensure models are ranked based on true real-world performance Because the future of AI evaluation isn’t just about benchmark accuracy. It’s about reliability in production. Link to the article in the comments!

Seven private English ASR datasets by Appen are now powering a new evaluation track on the Hugging Face Open ASR Leaderboard. Better signal, harder to game. 📖 https://lnkd.in/eFibXcES https://lnkd.in/eXr54bcd

Appen partnered with Hugging Face to bring private, benchmarking-resistant ASR evaluation datasets to the Open ASR Leaderboard. The leaderboard has been visited over 710,000 times since 2023. But public benchmarks have a problem: models can be optimized to climb rankings without actually performing better in the real world. That's benchmaxxing. Our contribution: seven private English ASR datasets spanning scripted and conversational speech across American, Australian, Canadian, and Indian accents. Because they're kept private, they can't be gamed, making leaderboard results more trustworthy. The data speaks for itself. When our datasets are included, model rankings shift. That's the signal a public-only benchmark can't give you. Read the full story → https://lnkd.in/eFibXcES #SpeechAI #ASR #AIEvaluation #HuggingFace #Benchmarking

If you’re at the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) this week, join us for an evening with Appen and Hugging Face. May 6 (Wednesday) | 6:00-9:00 PM We’re bringing together researchers, engineers and practitioners working across speech, audio, NLP and multimodal AI for a casual happy hour. Whether you want to exchange ideas or just unwind after a full day of sessions, this is the space for it. No pitches. No demos. Just people, drinks and real conversations. If you're around, come by and say hi. Register here: https://luma.com/7j9nagtx

Barcelona next week 🇪🇸 Sergio Bruccoleri, our VP of Delivery, will be at the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2026, connecting with researchers working on speech, audio, and multimodal systems and hearing what’s actually evolving in real-world deployments. If you’re attending, it’d be great to see you there.