Latent Variable Models

Behind every complex dataset lies structure we can’t see directly. People differ in patterns, not just averages. Behaviors co-vary for reasons that aren’t obvious. Latent modeling helps uncover these hidden structures. Principal Component Analysis (PCA) takes many correlated variables and transforms them into fewer uncorrelated components that retain most of the original variance. Each component is a linear combination of the initial variables, capturing how they vary together. PCA simplifies data, reduces noise, and helps visualize multidimensional relationships. It relies on eigenvalues and eigenvectors of the correlation matrix and is data-driven; it describes structure without inferring causes. Factor Analysis (FA) goes further by assuming correlations among variables stem from hidden factors such as traits or abilities. Each observed measure reflects both common factors and unique variance. Exploratory FA searches for these latent dimensions, while Confirmatory FA tests whether a proposed model fits new data. FA accounts for measurement error and aims to reveal theoretical constructs rather than just summarize data. Estimation involves solving for factor loadings and variances through maximum likelihood or least squares and assessing how well the structure explains observed relationships. Latent Class Analysis (LCA) shifts focus from variables to people. It applies to categorical data such as survey responses or ratings and assumes the population contains unobserved subgroups defined by similar response patterns. Each person’s answers are explained by their membership in a latent class, and the model estimates both class sizes and membership probabilities. LCA reveals population heterogeneity, showing that similar averages can hide very different subgroups. Latent Profile Analysis (LPA) extends this idea to continuous data. It assumes individuals belong to profiles characterized by distinct response patterns; one group may show high scores, another moderate, another low. These profiles can be interpreted as types within a population. Like LCA, LPA is a finite mixture model estimated using algorithms such as Expectation - Maximization. Criteria like AIC, BIC, and entropy guide how many profiles best fit the data. LPA exposes structured diversity without forcing arbitrary cutoffs. Latent Dirichlet Allocation (LDA) applies the same principle to text. It models each document as a mixture of topics and each topic as a distribution of words. A document might contain several topics in varying proportions, revealing recurring themes across a corpus. LDA uses Bayesian inference through variational methods or Gibbs sampling to estimate these distributions. It supports large-scale qualitative analysis, identifying emergent ideas and linguistic patterns without manual coding. Topics are probabilistic, adapting as new data appear.

Beyond Regression: Why Structural Equation Modeling (SEM) is a Game Changer for UX Research In user experience research, the tools we choose can shape the depth and clarity of our findings. Regression analysis, a reliable workhorse for many researchers, often provides an excellent starting point for identifying relationships between variables. But when it comes to uncovering the nuanced and interconnected dynamics of user behavior, regression may not always be enough. This realization hit home during a project on optimizing visual design for memory recall using the Rule of Thirds, where structural equation modeling (SEM) proved invaluable. Initially, regression helped us establish a direct relationship between the visual alignment of elements and memory performance. It was quick and clear, showing a correlation that seemed actionable. However, the more we probed, the more evident it became that we were missing the full picture. Turning to SEM, we were able to model not just the direct effects but also the indirect relationships, like how visual attention mediated the link between alignment and memory. SEM also allowed us to explore latent variables, such as user focus, which regression couldn’t adequately address. The insights were richer, more actionable, and far better aligned with the complexity of real-world user interactions. So, what’s the real difference between regression and SEM? Regression shines when the relationships between variables are straightforward and linear. It’s efficient and excellent for testing direct effects. But UX research often deals with interconnected systems where user satisfaction, cognitive load, and task completion influence each other in intricate ways. SEM steps in here as a more advanced method that models these complexities. It allows you to include latent variables, account for indirect effects, and visualize the interplay between multiple factors, all within a single framework. One of the most valuable aspects of SEM is its ability to uncover relationships you might not even think to test using regression. For example, while regression can tell you that a particular design change improves task completion rates, SEM can show how that improvement is mediated by reduced cognitive load or increased user trust. This kind of insight is critical for designing experiences that go beyond surface-level success metrics and truly resonate with users. To be clear, this isn’t an argument to abandon regression altogether. Each method has its place in a researcher’s toolkit. Regression is great for quick analyses and when the problem is relatively simple. But when your research involves complex systems or layered relationships, SEM provides the depth and clarity needed to make sense of it all. Yes, it’s more resource-intensive and requires a steeper learning curve, but the payoff in terms of actionable insights makes it worth the effort.

What you know about Logistic regression, might only be the tip of the iceberg. There is a whole other model hidden below the surface - quite literally. Logistic regression can be interpreted through the lens of a latent variable model. The concept is elegantly straightforward. In logistic regression, we observe an outcome variable, y, which takes on the values of either 1 or 0. Now imagine there exists another unobserved variable y* that is determined by a linear model and can be any real number. This latent variable y* is the key to unlocking the observed outcomes. Specifically, we observe y=1, whenever y*>0. Otherwise, we get y=0. The intriguing twist comes into play with the distribution of error terms in this latent model for y*. If these errors follow a logistic distribution, the observed outcomes y follow a logistic model. On the other hand, if the errors are normally distributed, we step into the domain of the so-called Probit model. This perspective enriches our understanding of logistic regression and bridges our comprehension to other statistical models, illuminating the interconnectedness of seemingly disparate methods. #datascience #statistics

Humans build an intuitive “world model” largely from a continuous stream of observations: we watch what changes, infer what likely caused it, and gradually get better at predicting what will happen next, often using language to conceptualise the action taking place in between. That’s the inspiration behind our new preprint, “Self-Improving World Modelling with Latent Actions”, led by my amazing student Yifu QIU: we introduce SWIRL ꩜, a framework to help foundation models (LLMs and VLMs) learn world modelling from state-only sequences (just “before” → “after”), without requiring expensive action-labelled trajectories. The key idea is to treat the missing action as a *latent variable grounded on language* and iteratively refine two models (initialised with pre-trained weights): - Forward World Model (FWM): predicts the next state given the current state and latent action; - Inverse Dynamics Model (IDM): predicts the latent action given a state transition. SWIRL ꩜ alternates two RL phases (via GRPO), where each model in turn acts as a policy updated with the other frozen model’s reward: FWM is rewarded when IDM can reliably “explain” its generated futures (encourages identifiable dynamics). IDM is rewarded when FWM assigns a high probability to the observed transition (encourages fidelity to real data). SWIRL ꩜ was tested across visual and textual/digital environments using state-only sequences for iterative self-improvement. The environments include (1) open-world visual dynamics for VLMs (single-step next-observation prediction and multi-step rollouts), and (2) text-based simulated worlds, web/HTML interaction dynamics, and tool-use execution dynamics for LLMs. Across these settings, SWIRL ꩜ consistently improves over backbone models (i.e., Liquid as a VLM and Qwen as an LLM) after a short SFT warm-up, achieving gains of +16% (AURORA-BENCH), +28% (ByteMorph), +16% (WorldPredictionBench), +14% (StableToolBench). If you’re interested in scaling world modelling from unlabeled video/web/tool traces, I’d love to hear your thoughts! Authors: Yifu QIU, Zheng Zhao, Waylon Li, Yftah Ziser, Anna Korhonen, Shay Cohen and me. arXiv paper: https://lnkd.in/eZgYBQGs Code: https://lnkd.in/eEWf6Szv Huggingface paper: https://lnkd.in/esa5TUgc #WorldModels #LLM #VLM #ReinforcementLearning

𝐂𝐚𝐮𝐬𝐚𝐥 𝐄𝐬𝐭𝐢𝐦𝐚𝐭𝐢𝐨𝐧 𝐰𝐢𝐭𝐡 𝐋𝐚𝐭𝐞𝐧𝐭 𝐕𝐚𝐫𝐢𝐚𝐛𝐥𝐞𝐬 𝐢𝐧 𝐃𝐀𝐆𝐬 𝐔𝐬𝐢𝐧𝐠 𝐏𝐲𝐭𝐡𝐨𝐧 In observational studies, the unconfoundedness assumption is rarely satisfied in practice. However, for every challenge, a solution eventually emerges, even if it is partial or contingent on specific conditions. Several Python libraries address this issue by working directly with causal structures and latent variables. In this post, want to share some insights into how these tools handle such complexities. 🏗️ 𝐀𝐧𝐚𝐧𝐤𝐞 𝐚𝐧𝐝 𝐅𝐫𝐨𝐧𝐭-𝐝𝐨𝐨𝐫 / 𝐈𝐃 𝐀𝐥𝐠𝐨𝐫𝐢𝐭𝐡𝐦 Ananke focuses on causal graphs with latent variables (ADMGs), where hidden confounders are represented explicitly. ➡️ It implements the ID algorithm and supports front-door adjustment. ➡️ If a mediator M satisfies front-door conditions, the effect T → Y can be identified even in the presence of an unobserved U. ➡️ This is one of the few tools that answers the key question first: Is the effect identifiable at all under hidden confounding? ➡️ Only after that, estimation makes sense. 🔗 𝐏𝐫𝐨𝐱𝐢𝐦𝐚𝐥 𝐂𝐚𝐮𝐬𝐚𝐥 𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞: 𝐏𝐫𝐨𝐱𝐢𝐞𝐬 𝐈𝐧𝐬𝐭𝐞𝐚𝐝 𝐨𝐟 𝐈𝐧𝐬𝐭𝐫𝐮𝐦𝐞𝐧𝐭𝐬 Valid instrumental variables are rare in real business and behavioral data. Proximal methods replace instruments with proxy variables for latent confounders: ➡️ one proxy linked to treatment, ➡️ one proxy linked to outcome. Under completeness and bridge-function assumptions, these proxies allow consistent estimation despite hidden U. In practice, this approach is useful when: 🔸 hidden confounding is unavoidable, 🔸 good proxies exist, 🔸 IV assumptions are implausible. 👉 Currently supported via extensions in Ananke (Eff-AIPW) and CausalML (Proximal Learners). 🧩 𝐄𝐱𝐩𝐥𝐢𝐜𝐢𝐭 𝐋𝐚𝐭𝐞𝐧𝐭 𝐕𝐚𝐫𝐢𝐚𝐛𝐥𝐞 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 Instead of bypassing hidden factors, some approaches model them directly. In CausalNex library, latent-variable Bayesian networks with EM estimation allow researchers to introduce unobserved concepts such as engagement, risk attitude, or market pressure. This enables counterfactual reasoning with hidden states, but requires: ➡️ strong priors, ➡️ domain constraints, ➡️ partial supervision or expert input. ❗ Without these, solutions are often unstable and hard to interpret. 🗝️ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲𝐬 𝐟𝐨𝐫 𝐃𝐞𝐜𝐢𝐬𝐢𝐨𝐧 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 ➡️ Identification before estimation: never apply complex models to non-identifiable effects. Establishing a structural identification strategy must precede numerical optimization. ➡️ Assumptions as first-class objects: every estimate is only as credible as its underlying assumptions. Treat these as explicit, testable components of your model. ➡️ Transparent uncertainty: combining point estimates with formal sensitivity analysis provides stakeholders with explicit bounds on the risks posed by unobserved confounding. #CausalInference #DataScience #DecisionScience #DataDriven #DecisionMaking

New working paper "A Bayesian Gaussian Process Dynamic Factor Model" with Tony Chernis, Niko Hauzenberger, and haroon mumtaz. Link to arXiv: https://lnkd.in/dSy7CF9t We propose a dynamic factor model (DFM) where the latent factors are linked to observed variables with unknown and potentially nonlinear functions: ➡️ Nonparametric observation equation, specified via Gaussian Process (GP) priors for each series ➡️ Factor dynamics modeled with a standard VAR (straightforward computation and interpretation) ➡️ Empirical applications: (1) forecasting with FRED-QD, (2) extracting driving forces of global inflation dynamics and measuring international asymmetries #forecasting #macroeconomics #inflation #ML

Autoencoders and Variational Autoencoders often look almost identical in diagrams, an encoder, a latent space, and a decoder, but the difference between them completely changes what these models can and cannot do. A standard autoencoder learns a direct, deterministic mapping from image space to latent space and back. For a given input image, the encoder always produces the same latent vector, and the decoder always produces the same reconstruction. This makes autoencoders very good at learning compact representations, removing noise, compressing images, and detecting anomalies, because the latent space is optimized purely for reconstruction fidelity. The model is rewarded for being precise, not for being creative, and the latent space ends up reflecting that objective. The limitation appears the moment we treat the latent space as something we can sample from. An autoencoder does not learn how to organize its latent space in a smooth or continuous way. Two nearby latent points do not necessarily correspond to similar images, and random sampling usually produces meaningless outputs. This is not a failure of training, it is simply not what the model was designed to do. Variational Autoencoders change exactly this assumption. Instead of mapping an image to a single point in latent space, the encoder maps it to a distribution defined by a mean and a variance. The latent vector is then sampled from this distribution, which introduces controlled stochasticity into the model. During training, the latent space is explicitly regularized to follow a known prior distribution, which forces it to become smooth, continuous, and sample friendly. This single change has deep consequences. VAEs sacrifice some reconstruction sharpness in exchange for a latent space that can be meaningfully explored and sampled. Interpolations between points become meaningful. Random samples decode into plausible images. The model is no longer just compressing data; it is learning a structured generative representation. This distinction is why autoencoders are often used as representation learners, while VAEs are used as true generative models, and why modern diffusion pipelines still rely on autoencoder-style compression but move generation itself into a probabilistic framework. If you understand why autoencoders fail at sampling, you already understand why VAEs exist, and once that clicks, much of modern image generation starts to feel inevitable rather than mysterious. I have just released a 1 hour lecture on Vizuara Technologies Private Limited's YouTube channel on building the intuition + coding an autoencoder from scratch in a total of 60 mins. You can watch it here. Please note: At the moment, the video is members only: https://lnkd.in/dASdyt-N

Everyone’s talking about AI images and videos, but very few understand how they actually work. Here’s the universal pipeline that powers everything from Stable Diffusion to MidJourney, DALL-E, and the new wave of video models: 1. Prompt → Embedding Your text is tokenized and mapped into vectors by a text encoder (CLIP, T5, or even an LLM). Think of it as turning words into math - directions in a concept space. 2. Latent Space Models don’t paint pixels directly. They generate in a compressed latent space using a VAE (variational autoencoder). This makes training and generation faster, more stable, and less memory-intensive. 3. Diffusion This is the core engine. A UNet learns to remove noise step by step. At inference, you start from pure noise and denoise toward an image (or video). Different schedulers (DDPM, DDIM, etc.) change the path, trading speed for accuracy. 4. Guidance & Conditioning Prompts steer the denoising process. Classifier-free guidance blends unconditional and conditioned predictions, with a guidance scale controlling how strongly the model follows your words. Negative prompts push it away from unwanted features. Extra controls (depth maps, pose, edge detection, reference images) can lock in structure. 5. Extending to Video Instead of a 2D latent image, the model denoises a 3D block of frames. Temporal attention keeps characters stable and motion coherent. This is how new systems generate fluid, consistent video instead of flickering frame by frame. 🔧 Knobs that matter: number of steps, sampler type, seed, CFG scale, VAE quality, and resolution. 👉 That’s the skeleton. Swap in different encoders, architectures, or training tricks, but the blueprint is similar. Understanding how it works is the first step to securing it. Once you see the pipeline, you also see where to place guardrails, raining data governance, prompt/output filters in latent space, watermarking and provenance, red-teaming for abuse, and deepfake detection downstream. 🧠 I highly recommend watching the full breakdown by 3Blue1Brown here: https://lnkd.in/ehRJpCD4