Dr. Juan Camilo Orduz

Mediation Analysis and (In)Direct Effects with PyMC

Tue, 17 Mar 2026 00:00:00 +0000

Mediation analysis goes beyond asking “does the treatment work?” to ask “how does the treatment work?” Understanding the mechanisms by which an intervention achieves its effect can have important implications for what treatments or policy changes are preferable. For instance, a family intervention program during adolescence might reduce substance use disorder in young adulthood, but through which pathways? Should the intervention focus on reducing peer influence, or on curbing direct experimentation?

Intuition behind CRPS

Wed, 11 Mar 2026 00:00:00 +0000

In many forecasting applications, probabilistic methods have become a standard tool. The reason is that predicting the point forecast is not enough for decision-making. The predicted uncertainty provides valuable information that can be used to make better decisions, such as risk management or inventory control. How to evaluate the performance of probabilistic forecasts? A naive approach is to collapse the output of the probabilistic model into a single point forecast (say, using the mean) and then use the standard metrics for point forecasts. A better alternative is to use the Continuous Ranked Probability Score (CRPS). This metric is a generalization of the mean absolute error (MAE) to the probabilistic setting. It is computed as the integral of the squared difference between the cumulative distribution function (CDF) of the predicted distribution \(F\) and the empirical CDF of the observation \(y\):

Bayesian Power Analysis for A/B Testing

Thu, 05 Mar 2026 00:00:00 +0000

This notebook implements a simulation-based Bayesian power analysis for conversion-rate A/B tests and compares it against the classical frequentist two-proportion z-test. We follow the Kruschke HDI+ROPE framework from The Bayesian New Statistics (Section “Planning for precision and other goals: Better done Bayesian”), in the same way as in the introductory example: Introduction to Bayesian Power Analysis: Exclude a Null Value. The process is as follows:

Generate synthetic A/B test data across a grid of sample sizes and true relative lifts \(\omega\) (distributions).
For each cell, fit a Bayesian model via MCMC and compute the \(95\%\) HDI of the relative_lift posterior.
Power = fraction of simulations where the \(95\%\) HDI falls entirely outside a Region of Practical Equivalence (ROPE).

We compare three prior specifications motivated by Prior Predictive Modeling in Bayesian AB Testing (see also The Bet Test):

In-Sample \(R^2\) is Not a Good Metric for Decision Making

Sun, 22 Feb 2026 00:00:00 +0000

In many years of professional data science, I have seen many senior decision makers choose models based on in-sample \(R^2\) — the more the better. Many of them would not even tolerate a model with an \(R^2\) lower than \(0.8\). These decision makers base their decisions on hyper inflated \(R^2\) models of the orders of millions of dollars (think about marketing spend). Here is a hard pill to swallow: for optimization and decision making, good causal models are not necessarily the ones with the highest \(R^2\). People working on causal inference are very aware about this, but it is sad that this is not common knowledge across various industries. Motivated by my previous blog post on a related topic, “Using Data Science for Bad Decision-Making: A Case Study”, here is yet another simulation to illustrate this point. I hope this can be used by data scientists to influence decision makers to make better decisions from statistical models.

Fixed and Random Effects Models: A Simulated Study

Tue, 03 Feb 2026 00:00:00 +0000

In this notebook we reproduce the fantastic material from the video Statistical Rethinking 2026 Lecture B04 - Group-level confounding and intro to social networks by Richard McElreath. This is a great video to understand the difference between fixed and random effects models. It is a must watch! Here we use PyMC to fit the models and compare the results.

The Problem

We work out a simple example of group-level confounding. The DAG we are going to use is the following:

A Bayesian Decision Theory Workflow: Port to NumPyro

Sun, 25 Jan 2026 00:00:00 +0000

In this notebook I explore the Bayesian Decision Theory Workflow described in the amazing blog post A Bayesian Decision Theory Workflow by Daniel Saunders. It is a great resource to understand how to use Bayesian methods for optimization problems. In the original post, Daniel explains the theory and key concepts using PyMC and its computational backend PyTensor. He does a remarkable job presenting powerful PyTensor features to manipulate and operate with symbolic computational graphs. I can only recommend you to read it!

Forecasting Hierarchical Models - Part III

Mon, 05 Jan 2026 00:00:00 +0000

In this third notebook, I extend the hierarchical forecasting model from Part II by adding a neural network component to the state transition function. This creates a Hybrid Deep State-Space Model that combines probabilistic modeling with deep learning.

Why? This is a personal experiment to explore how to integrate neural networks with hierarchical models. It is not adding complexity for the sake of complexity. It is rather an exploratory exercise to see if this approach can lead to better forecasting performance.

Machine Learning for Optimization: Toy Example

Thu, 01 Jan 2026 00:00:00 +0000

Based on my experience, whenever someone asks for a prediction (or forecasting) model, they actually do not need a prediction model per se. They typically want to answer causal questions or do some kind of optimization. I have covered some case studies on causal questions in previous posts (for example, Introduction to Causal Inference with PPLs and “Using Data Science for Bad Decision-Making: A Case Study”).

In this blog post, I want to focus on optimization. I found a little nice use case when working on adtech, where one is interested in optimizing bids to maximize the revenue (or any other target, like ROAS or lifetime value). How to set the bids? This is a huge active research area so this is by no means an exhaustive treatment. I want to focus on a small component on a recent paper: “Lightweight Auto-bidding based on Traffic Prediction in Live Advertising” where the authors propose a method to set the bids by optimizing on the output of a fitted forecast model. I won’t go into the paper scope, but rather focus on a self contained problem: Algorithm 1 Algorithm BiCB. The basic idea is as follows: In order to set bids on time intervals we can fit a forecasting model to predict the cumulative cost over the day based on time features and the current bid value \(\text{bid}_t\). To set the next bid \(\text{bid}_{t + 1}\) we can compare the forecast against the desired target (say, the expected cumulative daily budget at \(t + 1\)). We can adjust the under/over pacing by minimizing this difference. In other words, we want to steer achieving the target using the bids values through a time machine (i.e. a forecasting model) to generate counterfactuals. The paper works out this in certain level of detail, but the concrete implementation is a bit open. So here we do it by plain brute force (why not?). The whole idea is not to solve this concrete algorithm but to experiment on how to use machine learning methods for optimization purposes.

CATE Estimation with Causal Effect Variational Autoencoders

Sat, 13 Dec 2025 00:00:00 +0000

In this notebook, we demonstrate how to estimate Conditional Average Treatment Effects (CATE) using a Causal Effect Variational Autoencoder (CEVAE) by implementing an example from scratch in NumPyro. This approach is particularly useful when we suspect the presence of unobserved confounders that affect both treatment assignment and outcomes.

Disclaimer: I am not an expert in this specific approach, so please take all the results with a grain of salt and please do not hesitate to provide feedback.

Causal Effect Estimation with Variational Inference and Latent Confounders

Wed, 10 Dec 2025 00:00:00 +0000

This notebook demonstrates how to estimate the Average Treatment Effect (ATE) using variational inference in the presence of unobserved confounders. The approach is based on the tutorial by Robert Ness from his book: Causal AI book. We port his Pyro code to NumPyro (this was the objective for me to learn the details of the method).

The fundamental challenge in causal inference from observational data is confounding: variables that affect both the treatment and outcome can bias naive estimates. When confounders are unobserved, traditional adjustment methods fail. However, as shown in the CEVAE paper (Louizos et al., NeurIPS 2017), latent variable models can simultaneously infer the hidden confounders and estimate causal effects. The variational autoencoder framework provides flexibility to model complex latent structures without strong parametric assumptions. The encoder (recognition network) learns to map observed data to the latent confounder distribution, enabling amortized inference.

Causal Inference with Multilevel Models: The Electric Company Example

Fri, 28 Nov 2025 00:00:00 +0000

Estimating causal effects from clustered or grouped data requires careful attention to the hierarchical structure of observations. When units are nested within groups such as students within classrooms, or patients within hospitals—ignoring this structure can lead to incorrect standard errors, inefficient estimates, and invalid causal inferences. Multilevel models provide a principled framework for handling such data while leveraging the advantages of partial pooling across groups.

This notebook reproduces and extends the analysis from Chapter 23 of Gelman and Hill’s “Data Analysis Using Regression and Multilevel/Hierarchical Models”. We demonstrate two complementary approaches to modeling treatment effects in hierarchical data: first, a model with varying intercepts that efficiently controls for group-level confounding, and second, a more flexible covariance model that allows treatment effects themselves to vary across groups. Together, these models illustrate how multilevel structures enhance both the efficiency and interpretability of causal effect estimation. In addition to reproducing the analysis, we show how to efficiently vectorize the model (across grades and pairs) using PyMC.

Introduction to Causal Inference with PPLs

Thu, 27 Nov 2025 00:00:00 +0000

Causal inference asks a deceptively simple question: “What would have happened if things were different?” Whether we’re evaluating a job training program, testing a new medical treatment, or analyzing the impact of a policy change, we want to understand the causal effect of an intervention not just observe correlations in the data.

Traditional statistical methods often struggle with causal questions because they conflate correlation with causation. When confounders variables that affect both treatment assignment and outcomes are present, naive comparisons can lead us astray. This notebook demonstrates how probabilistic programming languages (PPLs) provide a powerful framework for causal inference that makes confounding explicit, quantifies uncertainty properly, and enables us to answer counterfactual questions directly.

Bayesian Vector Autoregressive Models in NumPyro

Fri, 03 Oct 2025 00:00:00 +0000

In this notebook, we present how to implement and fit Bayesian Vector Autoregressive (VAR) models using NumPyro. We work out three components:

Specifying and fitting the model in NumPyro
Using the model to generate forecasts
Computing the Impulse Response Functions (IRFs)

We compare these three components with the ones obtained using the statsmodels implementation from the Vector Autoregressions tsa.vector_ar tutorial.

Prepare Notebook

from functools import partial

import arviz as az
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpy as np
import numpyro
import numpyro.distributions as dist
import pandas as pd
import statsmodels.api as sm
import xarray as xr
from jax import jit, lax, random, vmap
from jaxtyping import Array, Float
from numpyro.contrib.control_flow import scan
from numpyro.handlers import condition
from numpyro.infer import MCMC, NUTS
from statsmodels.tsa.api import VAR
from statsmodels.tsa.base.datetools import dates_from_str

numpyro.set_host_device_count(n=10)

rng_key = random.PRNGKey(seed=42)

az.style.use("arviz-darkgrid")
plt.rcParams["figure.figsize"] = [12, 7]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"

%load_ext autoreload
%autoreload 2
%load_ext jaxtyping
%jaxtyping.typechecker beartype.beartype
%config InlineBackend.figure_format = "retina"

Load Data

We are going to use a dataset from the statsmodels package. Specifically, we will use the macrodata dataset from Vector Autoregressions tsa.vector_ar tutorial. For the sake of reproducibility, we will keep the exact same code as in the tutorial.

PyData Berlin 2025: Introduction to Stochastic Variational Inference with NumPyro

Mon, 01 Sep 2025 00:00:00 +0000

In this notebook we provide a brief introduction to Stochastic Variational Inference (SVI) with NumPyro. We provide the key mathematical concepts, but we focus on the code implementation. This introductory notebook is meant for practitioners. We do this by working through two examples: a very simple parameter recovery model and a Bayesian Neural Network.

This work was presented at PyData Berlin 2025, you can find the slides here.

Overview

Stochastic Variational Inference (SVI) is a scalable approximate inference method that transforms the problem of posterior inference into an optimization problem. Instead of sampling from the posterior distribution (like MCMC), SVI finds the best approximation to the posterior within a family of simpler distributions.

Hierarchical Revenue & Retention Modeling

Mon, 04 Aug 2025 00:00:00 +0000

In this notebook, we extend the revenue-retention model introduced in the sequence of blog posts “Cohort Revenue & Retention Analysis: A Bayesian Approach” and “Cohort Revenue Retention Analysis with Flax and NumPyro” (plus the associated pre-print “Cohort Revenue & Retention Analysis: A Bayesian Approach”) to include analysis across different markets (or any type of grouping variable). The motivation for this extension is that in many real applications one is interested in understanding the revenue and retention patterns across different markets where typically we have different data sizes. Instead of modeling all separately, we take advantage of the hierarchical structure of the data to share information across markets. This will help have better forecasts for younger markets where limited data is available. We show through a simulation the power of this approach (and how relatively simple it is to implement once we have the basic model in place).

Vectorize ROC Curve for Bayesian Models

Mon, 30 Jun 2025 00:00:00 +0000

In this notebook, we present a simple example to illustrate how to vectorize the ROC curve computation over a Bayesian model. This is helpful when we want to compute the ROC curve using the implementations from scikit-learn. We use the classical moons dataset to generate the data and fit a Gaussian process models similarly as the previous post Scikit-Learn Example in PyMC: Gaussian Process Classifier.

Prepare Notebook

import arviz as az
import matplotlib.pyplot as plt
import numpy as np
import preliz as pz
import pymc as pm
import pytensor
import seaborn as sns
import xarray as xr
from sklearn.datasets import make_moons
from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split

az.style.use("arviz-darkgrid")
plt.rcParams["figure.figsize"] = [10, 6]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"

%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = "retina"

seed: int = sum(map(ord, "vectorize_roc_curve"))
rng: np.random.Generator = np.random.default_rng(seed=seed)

Generate Synthetic Data

# Generate data
x, y = make_moons(n_samples=150, noise=0.25, random_state=seed)
# Split data into training and test sets
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.3, random_state=seed
)
# Get number of samples in training and test sets
n_train = x_train.shape[0]
n_test = x_test.shape[0]
n = n_train + n_test
# Create indices for training and test sets
idx_train = range(n_train)
idx_test = range(n_train, n_train + n_test)
# Get dimension of the domain
domain_dim = x.shape[1]

Let’s start by visualizing the data.

Introduction to Bayesian Power Analysis: Exclude a Null Value

Tue, 29 Apr 2025 00:00:00 +0000

Recently, I have been thinking a lot about data-driven decision-making, particularly in the context of experimentation. Why? I am uncomfortable with the common practice of using p-values and frequentist null hypothesis significance testing to make decisions. I don’t feel confident about the approach. I think it is because I do not get it. For instance, when I am forced to explain the definition of a confidence interval precisely, it does not come naturally. I always need to check with a trustworthy source (it is common to find wrong explanations online). If I do not understand it, I cannot use it, especially for decision-making. I always play this exercise when thinking about business recommendations in real applications: “Would I bet my salary on this?” Whenever I work with p-values, the answer to this question is no.

Prior Predictive Modeling in Bayesian AB Testing

Tue, 11 Mar 2025 00:00:00 +0000

In this notebook we generate a simulation to reproduce the results of the great blog post “The Bet Test: Spotting Problems in Bayesian A/B Test Analysis”, where Tyler Buffington discussed about some caveats of using Bayesian AB testing when not thinking about the prior predictive distribution on the key metrics of interested. If you haven’t read it yet, please do it before continuing ;)

In a nutshell, the blog post showcases the risk of using informative priors to speed-up A/B tests in a naive way. To be fair, it is easy to fall into this trap (I have been there 🙈), so I think it is great to have these discussions to make us think twice about what we are doing.

Notes on Hierarchical Hilbert Space Gaussian Processes

Wed, 01 Jan 2025 00:00:00 +0000

In this notebook, we want to explore some ideas on hierarchical Hilbert Space Gaussian Processes following the fantastic exposition of the PyMC example notebook: “Gaussian Processes: HSGP Advanced Usage” by Bill Engels, Alexandre Andorra and Maxim Kochurov. I can only recommend to read the notebook and the references therein! For an introduction to Hilbert Space Gaussian Processes, please see my introductory blog post: “A Conceptual and Practical Introduction to Hilbert Space GPs Approximation Methods”.

Motivation: In many applications, one is interested in understanding the dynamic effect of one variable on another. For example, in marketing, one is interested in efficiency across many channels on sales (or conversions) as a function of time. For such a purpose, one typically uses a time-varying regression model with time-varying coefficients (for a simple example, see “Time-Varying Regression Coefficients via Hilbert Space Gaussian Process Approximation”). The main challenge is that, if not done carefully, the model will easily overfit the data. One can use a hierarchical model with a global component and a group-specific component to overcome this. The global component will capture the overall trend, while the group-specific component will capture the idiosyncratic behavior of each group. As a side effect, we will gain some regularization effect that will help to avoid overfitting. For the marketing example above, PyMC Labs has successfully applied this approach in the context of media mix modeling: “Bayesian Media Mix Models: Modelling changes in marketing effectiveness over time”. This hierarchical approach was motivated by the work “Hierarchical Bayesian modeling of gene expression time series across irregularly sampled replicates and clusters”. Here is a schematic diagram of the hierarchical approach (taken from the paper):

Hacking the TSB Model for Intermediate Time Series to Accommodate for Availability Constraints

Mon, 25 Nov 2024 00:00:00 +0000

In many demand forecasting problems, we face the challenge of predicting the demand for a product or service with very sparse data. This is especially true in retail, where many products are only sold occasionally. This sparsity can come from many sources, as greatly described in the (recommended!) blog post “Why zeroes happen” by Ivan Svetunkov. On one hand, the sparsity can simply come from the lack of demand for the product. On the other hand, the sparsity can also come from the lack of availability of the product. For example, a product might be only available for purchase during certain times of the year or only in certain geographical locations. There could be other reasons as well. In this notebook, we experiment with an extension on a classical TSB time-series model for intermediate time series to accommodate availability constraints.

Bayesian CUPED & Sensitivity Analysis

Tue, 29 Oct 2024 00:00:00 +0000

Motivated by the great blog post by Understanding CUPED by Matteo Courthoud, we explore a Bayesian approach to CUPED to understand its sensitivity with respect to the covariance parameter \(\theta\) of the pre-post mean model. I will assume that the reader is already familiar with CUPED and has read Matteo’s blog post (highly recommended!). Here we focus on the sensitivity component. We do not do this in full generality but rather focus on the specific example of Matteo’s blog post.

Electricity Demand Forecast: Dynamic Time-Series Model with Prior Calibration

Mon, 07 Oct 2024 00:00:00 +0000

We present an example of a dynamic forecasting time-series model that incorporates a prior calibration process to estimate the temperature effect on electricity demand. The model is based on the previous example Electricity Demand Forecast: Dynamic Time-Series Model. In this second iteration, we borrow the ideas from the Pyro great example Forecasting with Dynamic Linear Model (DLM) where they use a prior calibration process on a local level forecasting model. In our case, we use the same technique with a Hilbert Space Gaussian Process latent component model.

Electricity Demand Forecast: Dynamic Time-Series Model

Sun, 06 Oct 2024 00:00:00 +0000

We work out a classical electricity demand forecasting model form the case study Structural Time Series Modeling Case Studies: Atmospheric CO2 and Electricity Demand from the TensorFlow Probability documentation. The idea of this example is to use temperature as a linear covariate to model the electricity demand. In this example, we show how to use a (Hilbert Space Approximation) Gaussian process to model the non-linear relationship between temperature and electricity demand (for an introduction to the topic see A Conceptual and Practical Introduction to Hilbert Space GPs Approximation Methods). This technique improves the simple linear model in and out of sample predictions as we aro not using the Gaussian process to extrapolate over time, but rather to model the non-linear relationship between temperature and electricity demand, similarly as how it has done in the example Time-Varying Regression Coefficients via Hilbert Space Gaussian Process Approximation.

From Pyro to NumPyro: Forecasting Hierarchical Models - Part II

Sat, 05 Oct 2024 00:00:00 +0000

In this second notebook, we continue working on the NumPyro implementation of the hierarchical forecasting models presented in Pyro’s forecasting documentation: Forecasting III: hierarchical models. In this second part, we extend the model described in the first part From Pyro to NumPyro: Forecasting Hierarchical Models - Part I by adding all stations to the model.

Prepare Notebook

import arviz as az
import jax
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpyro
import numpyro.distributions as dist
from jax import random
from jaxtyping import Array, Float
from numpyro.contrib.control_flow import scan
from numpyro.infer import SVI, Predictive, Trace_ELBO
from numpyro.infer.autoguide import AutoNormal
from numpyro.infer.reparam import LocScaleReparam
from pyro.contrib.examples.bart import load_bart_od

az.style.use("arviz-darkgrid")
plt.rcParams["figure.figsize"] = [12, 7]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"

numpyro.set_host_device_count(n=4)

rng_key = random.PRNGKey(seed=42)

%load_ext autoreload
%autoreload 2
%load_ext jaxtyping
%jaxtyping.typechecker beartype.beartype
%config InlineBackend.figure_format = "retina"

The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
The jaxtyping extension is already loaded. To reload it, use:
  %reload_ext jaxtyping

Read Data

dataset = load_bart_od()
print(dataset.keys())
print(dataset["counts"].shape)
print(" ".join(dataset["stations"]))

dict_keys(['stations', 'start_date', 'counts'])
torch.Size([78888, 50, 50])
12TH 16TH 19TH 24TH ANTC ASHB BALB BAYF BERY CAST CIVC COLM COLS CONC DALY DBRK DELN DUBL EMBR FRMT FTVL GLEN HAYW LAFY LAKE MCAR MLBR MLPT MONT NBRK NCON OAKL ORIN PCTR PHIL PITT PLZA POWL RICH ROCK SANL SBRN SFIA SHAY SSAN UCTY WARM WCRK WDUB WOAK

In this second example, we model all the rides from all stations to all other stations.

From Pyro to NumPyro: Forecasting Hierarchical Models - Part I

Thu, 03 Oct 2024 00:00:00 +0000

In this notebook we provide a NumPyro implementation of the first model presented in the Pyro forecasting documentation: Forecasting III: hierarchical models. This model generalizes the local level model with seasonality presented in the univariate example Forecasting I: univariate, heavy tailed (see From Pyro to NumPyro: Forecasting a univariate, heavy tailed time series for the corresponding NumPyro implementation).

In this example, we continue working with the BART train ridership dataset.

Prepare Notebook

import arviz as az
import jax
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpyro
import numpyro.distributions as dist
import torch
from jax import random
from jaxtyping import Array, Float
from numpyro.contrib.control_flow import scan
from numpyro.infer import SVI, Predictive, Trace_ELBO
from numpyro.infer.autoguide import AutoNormal
from numpyro.infer.reparam import LocScaleReparam
from pyro.contrib.examples.bart import load_bart_od
from pyro.ops.tensor_utils import periodic_repeat

az.style.use("arviz-darkgrid")
plt.rcParams["figure.figsize"] = [12, 7]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"

numpyro.set_host_device_count(n=4)

rng_key = random.PRNGKey(seed=42)

%load_ext autoreload
%autoreload 2
%load_ext jaxtyping
%jaxtyping.typechecker beartype.beartype
%config InlineBackend.figure_format = "retina"

The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
The jaxtyping extension is already loaded. To reload it, use:
  %reload_ext jaxtyping

Read Data

dataset = load_bart_od()
print(dataset.keys())
print(dataset["counts"].shape)
print(" ".join(dataset["stations"]))

dict_keys(['stations', 'start_date', 'counts'])
torch.Size([78888, 50, 50])
12TH 16TH 19TH 24TH ANTC ASHB BALB BAYF BERY CAST CIVC COLM COLS CONC DALY DBRK DELN DUBL EMBR FRMT FTVL GLEN HAYW LAFY LAKE MCAR MLBR MLPT MONT NBRK NCON OAKL ORIN PCTR PHIL PITT PLZA POWL RICH ROCK SANL SBRN SFIA SHAY SSAN UCTY WARM WCRK WDUB WOAK

For this first model, we just model the rides to Embarcadero station, from each of the other \(50\) stations.

From Pyro to NumPyro: Forecasting a univariate, heavy tailed time series

Tue, 01 Oct 2024 00:00:00 +0000

In this notebooks we port the Pyro forecasting example Forecasting I: univariate, heavy tailed to NumPyro. The forecasting module in Pyro is fantastic as it provides an easy interface to develop custom forecasting models. It has also many helpful utility functions to generate features and for model evaluation. The purpose of this translation is to dig deeper into the some forecasting components and to show that translating Pyro code to NumPyro is not that hard, even though there are come caveats.

Hierarchical Pricing Elasticity Models

Thu, 01 Aug 2024 00:00:00 +0000

In this notebook we use a retail publicly available dataset to fit and compare various pricing elasticity models. This example can be seen as a continuation of the notebooks regarding Bayesian hierarchical models (see for example Multilevel Elasticities for a Single SKU - Part I). In this example we also see how Bayesian hierarchical models can help regularize elasticity estimates when taking advantage of the hierarchical structure of the data. In addition, this example shows that these models can scale well using stochastic variational inference in NumPyro.

Multilevel Elasticities for a Single SKU - Part III.

Fri, 19 Jul 2024 00:00:00 +0000

In this notebook we continue our simulation study for elasticities, see:

for an introduction to the problem and some models. We extend the covariance model to allow two covariance components on both the intercepts and slopes (coefficient of the median_ income variable). Also, to abstract from a specific framework, we do the implementation in NumPyro. The corresponding PyMC is very similar.

Hierarchical Exponential Smoothing Model

Fri, 07 Jun 2024 00:00:00 +0000

In this blog post, we experiment with a hierarchical exponential smoothing forecasting model, extending the ideas from the univariate case presented in the blog post “Notes on Exponential Smoothing with NumPyro”. We use NumPyro and compare the NUTS and SVI results. For such a purpose, we use Continuous Ranked Probability Score (CRPS). We also compare these forecasts with univariate statistical models like Holt-Winters, AutoETS and Seasonal Naive from the great Statsforecast package. These baseline models are, in general, hard to beat!

Demand Forecasting with Censored Likelihood

Sun, 21 Apr 2024 00:00:00 +0000

In this notebook we will explore the use of censored likelihoods for demand forecasting.

Business Problem: Let us assume we have a store with a single product. We want to forecast the (true!) demand for this product for the next 30 days using historical data. The historical data consists of the daily sales of the product for the last year (approximately). An important challenge is that our historical sales data is censored. This means that we only observe the sales of the product when it is in stock. If the product is out of stock, we do not observe any sales. This includes when the product is not available in the store or when we sold out of the product.

A Conceptual and Practical Introduction to Hilbert Space GPs Approximation Methods

Thu, 18 Apr 2024 00:00:00 +0000

In this notebook, we explore the (conceptual) ideas and (practical) implementation details of the Hilbert Space approximation for Gaussian processes introduced in the article “Hilbert space methods for reduced-rank Gaussian process regression” by Arno Solin and Simo Särkkä. We do not go deep into the mathematical details (proofs) but focus on the core ideas to help us understand the main concepts guiding the technical implementation. We provide examples, both in NumPyro and PyMC, so that users can learn from cross-framework comparison so that we abstract the core ideas.

Bayesian Censoring Data Modeling

Mon, 26 Feb 2024 00:00:00 +0000

In this notebook, we explore how we can use Bayesian modeling to estimate the parameters of a censored data set. These datasets are common in many fields, including survival analysis and supply chain management. I was motivated to write this notebook after reading the excellent blog post “Modeling Anything With First Principles: Demand under extreme stockouts” by Kyle Caron where he uses these techniques to model and balance demand under extreme stockouts and other constraints.

Zero-Inflated TSB Model

Tue, 20 Feb 2024 00:00:00 +0000

After going through the fundamentals of the TSB Method for Intermittent Time Series Forecasting in NumPyro in the previous notebook, we explore a variation of it that might be useful for certain applications. In a nutshell, we keep the same model structure of the TSB model, but we modify the likelihood function to account for the sparsity of the time series. Concretely, we replace the classic Gaussian likelihood function with a zero-inflated Negative Binomial likelihood function. One clear conceptual advantage of this approach that all our prediction and credible intervals will be non-negative.

TSB Method for Intermittent Time Series Forecasting in NumPyro

Sat, 17 Feb 2024 00:00:00 +0000

In this notebook we provide a NumPyro implementation of the TSB (Teunter, Syntetos and Babai) method for forecasting intermittent time series. The TSB method is similar to the Croston’s method in the sense that is constructs two different time series out of the original one and then forecast each of them separately, so that the final forecast is generated by combining the forecasts of the two time series. The main difference between the two methods is that the TSB method uses the demand probability instead of the demand periods. Consequently, let \(y_{t}\) denote the input time series then the TSB method is specified by the following equations:

Croston's Method for Intermittent Time Series Forecasting in NumPyro

Thu, 15 Feb 2024 00:00:00 +0000

In this notebook, we will implement Croston’s method for intermittent demand forecasting using NumPyro. Croston’s method is a popular forecasting method for intermittent demand data, which is characterized by a large number of zero values. The method is based on the idea of separating the demand size and the demand interval, and then forecasting them separately using simple exponential smoothing. We therefore can leverage on top of the previous post Notes on Exponential Smoothing with NumPyro. Once we have the forecasts for the demand size and the demand interval, we can combine them to get the final forecast. For a succinct explanation of Croston’s method, I recommend the following blog post: https://www.pmorgan.com.au/tutorials/crostons-method/.

Notes on an ARMA(1, 1) Model with NumPyro

Tue, 13 Feb 2024 00:00:00 +0000

This are some notes on how to implement an ARMA(1, 1) model using NumPyro for time series forecasting. The ARMA(1, 1) model is given by

\[y_t = \mu + \phi y_{t-1} + \theta \varepsilon_{t-1} + \varepsilon_t\]

where \(y_t\) is the time series, \(\mu\) is the mean, \(\phi\) is the autoregressive parameter, \(\theta\) is the moving average parameter, and \(\varepsilon_t\) is a white noise process with mean zero and variance \(\sigma^2\).

Notes on Exponential Smoothing with NumPyro

Sun, 11 Feb 2024 00:00:00 +0000

This notebook serves as personal notes on NumPyro’s implementation of the classic exponential smoothing forecasting method. I use Example: Holt-Winters Exponential Smoothing. The strategy is to go into the nitty-gritty details of the code presented in the example from the documentation: “Example: Holt-Winters Exponential Smoothing”. In particular, I want to understand the auto-regressive components using the scan function, which always confuses me 😅. After reproducing the example from the documentation, we go a step further and extend the algorithm to include a damped trend.

Media Mix Model and Experimental Calibration: A Simulation Study

Sun, 04 Feb 2024 00:00:00 +0000

In this notebook, we present a complete simulation study of the media mix model (MMM) and experimental calibration method presented in the paper “Media Mix Model Calibration With Bayesian Priors”, by Zhang, et al., where the authors propose a convenient parametrization the regression model in terms of the ROAs (return on advertising spend) instead of the classical regression (beta) coefficients. The benefit of this parametrization is that it allows for using Bayesian priors on the ROAS, which typically come from previous experiments or domain knowledge. Providing this information to the model is essential in real-life MMM applications, as biases and noise can easily fool us. Similar to the author’s paper, we show that the proposed method can provide better ROAS estimation when we have a bias due to missing covariates or omitted variables. We work out an example of the classical media mix model presented in the paper “Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects”, by Jin, et al.. I strongly recommend taking a look in too these two papers before reading this notebook. Reading them in parallel with this notebook is also a good alternative.

Cohort Revenue Retention Analysis with Flax and NumPyro

Mon, 08 Jan 2024 00:00:00 +0000

In this notebook we present an alternative implementation of the cohort-revenue-retention model presented in the blog post Cohort Revenue & Retention Analysis: A Bayesian Approach where we show how to replace the BART retention component with a general neural network implemented with Flax. This allows faster inference, as we can use NumPyro’s NUTS sampler or any of the stochastic variational inference (SVI) algorithms available. We could even use a wider family of samplers using the newly released package Bayeux or the great BlackJax (see for example, the MLP Classifier Example). We use the same simulated dataset to be able to compare the approaches. Overall, the retention and revenue in and out-of sample predictions, as well as the credible intervals are very similar to the ones obtained with the BART model.

Flax and NumPyro Toy Example

Fri, 05 Jan 2024 00:00:00 +0000

In this notebook I want to experiment with the numpyro/contrib/module.py module which allow us to integrate Flax models with NumPyro models. I am interested in this because I want to experiment with complex bayesian models with larger datasets.

Most of the main components can be found in the great blog post Bayesian Neural Networks with Flax and Numpyro. The author takes a different path working directly with potentials, but he also points out the recent addition of the numpyro/contrib/module.py module. The main difference with the model presented here is that I am using two components in the model (to model the mean and standard deviation of the data), I use stochastic variational inference instead of MCMC and I work with scaling transformations.

Time Series Modeling with HSGP: Baby Births Example

Tue, 02 Jan 2024 00:00:00 +0000

In this notebook we want to reproduce a classical example of using Gaussian processes to model time series data: The birthdays data set. I first encountered this example in the seminal book Chapter 21, Bayesian Data Analysis (Third edition) when learning about the subject. One thing I rapidly realized was that fitting these types of models in practice is very computationally expensive and sometimes almost infeasible for real industry applications where the data size is larger than all of these academic examples. Recently, there has been a lot of progress in approximation methods to speed up the computations. We investigate one such method: the Hilbert Space Gaussian Process (HSGP) approximation introduced in Hilbert space methods for reduced-rank Gaussian process regression. The main idea of this method relies on the Laplacian’s spectral decomposition to approximate kernels’ spectral measures as a function of basis functions. The key observation is that the basis functions in the reduced-rank approximation do not depend on the hyperparameters of the covariance function for the Gaussian process. This allows us to speed up the computations tremendously. We do not go into the mathematical details here (we might do this in a future post), as the original article is very well written and easy to follow (see also the great paper Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming). Instead, we reproduce this classical example using PyMC using a very raw implementation from NumPyro Docs - Example: Hilbert space approximation for Gaussian processes, which is a great resource to learn about the method internals (so it is also strongly recommended!).

Non-Parametric Product Life Cycle Modeling

Sun, 10 Dec 2023 00:00:00 +0000

In this notebook we present an example of how to use a combination of Bayesian hierarchical models and the non-parametric methods , namely bayesian additive trees (BART), to model the product life cycles. This approach is motivated by previous work in cohort analysis, see here.

As a case study we use the Google search index (trends) data for iPhones worldwide. We use the data of four different models to predict the development of the latest iPhone. The model presented for this specific example can be easily extended for other products and product life cycles structures.

NumPyro with Pathfinder

Mon, 04 Dec 2023 00:00:00 +0000

In this notebook we describe how to use blackjax’s pathfinder implementation to do inference with a numpyro model.

I am simply putting some pieces together from the following resources (strongly recommended to read):

References:

Blackjax docs: Use with Numpyro models
Blackjax Sampling Book: Pathfinder
Numpyro Issue #1485
PyMC Experimental - Pathfinder
Pathfinder: Parallel quasi-Newton variational inference

What and Why Pathfinder?

From the paper’s abstract:

What?

We propose Pathfinder, a variational method for approximately sampling from differentiable log densities. Starting from a random initialization, Pathfinder locates normal approximations to the target density along a quasi-Newton optimization path, with local covariance estimated using the inverse Hessian estimates produced by the optimizer. Pathfinder returns draws from the approximation with the lowest estimated Kullback-Leibler (KL) divergence to the true posterior.

Causal Bandits: Causality, Marketing & Simulations

Sun, 19 Nov 2023 00:00:00 +0000

I had the great opportunity to have a conversation with Aleksander Molak about causality, marketing and simulations as well as my career from academia to industry. Check it out!

Multilevel Elasticities for a Single SKU - Part II.

Mon, 13 Nov 2023 00:00:00 +0000

In this notebook we go deeper into the last covariance model presented in the previous blog post Multilevel Elasticities for a Single SKU. In particular we describe how to generate posterior predictive samples from an unseen region by the model. This can be useful for scenario planning: once can simulated outcome quantities from price ranges through the elasticity estimates (with uncertainty!)

We strongly recommend reading the previous blog post before reading this one as we will skip the EDA and baseline model comparison parts.

Multilevel Elasticities for a Single SKU - Part I.

Thu, 10 Aug 2023 00:00:00 +0000

In this notebook I want to experiment with some basic models for price elasticity estimation in the simple context of a simple sku across multiple stores and regions. The motivation is to have a concrete example of the elasticity models presented in the Chapter 11: Big Data Pricing Models of the book Pricing Analytics by Walter R. Paczkowski.

Elasticity Definition

Here I provide a very succinct definition of elasticity (there is a vast literature on this topic, see the reference above). The elasticity of a variable \(y(x, z)\) with respect to another variable \(x\) is defined as the percentage change in \(y\) for a one percent change in \(x\). Mathematically, this is written as \[ \eta = \frac{\partial \log(y(x, z))}{\partial \log(x)} \]

Time-Varying Regression Coefficients via Hilbert Space Gaussian Process Approximation

Wed, 05 Jul 2023 00:00:00 +0000

In this notebook we present an example of a regression model with time varying coefficients using Gaussian processes. In particular, we use a Hilbert space Gaussian process approximation in pymc to speed up the computations (see HSGP). We continue using the bikes dataset from the previous posts (Exploring Tools for Interpretable Machine Learning and Time-Varying Regression Coefficients via Gaussian Random Walk in PyMC). Please refer to those posts for more details on the dataset, EDA and base models. In essence, we are trying to model bike count rentals as a function of meteorological variables and seasonality. We are particularly interested in the marginal effect of temperature on bike rentals, which we expect to be non-linear.

Using Data Science for Bad Decision-Making: A Case Study

Fri, 16 Jun 2023 00:00:00 +0000

You will probably be intrigued by the title of this post. In this notebook I do not want to present a fancy data science trick or to test a novel technique. I would simply like to tell a story. A story about how data science can be used to make bad decisions. “How can this be?” you might ask. Everyone has been saying that data is the way to unlock insights to gain a competitive advantage. Well, it is true. But it is also true that data can be used to make decisions that can actually hurt your business. There are many possible reasons you could think of (and you might even have experienced some of them). The story I am about to tell won’t be about bad algorithms or bad data. It will be about thinking about data as simply

Regression Discontinuity with GLMs and Kernel Weighting

Sat, 10 Jun 2023 00:00:00 +0000

In this notebook we explore regression discontinuity design using generalized linear models (GLMs) and kernel weighting from a bayesian perspective. The motivation comes from applications when:

The data does not fit the usual linear regression OLS normal likelihood (e.g. modeling count data).
The data size is limited.

In addition, we experiment with kernel weighting to weight the data points near the cutoff more heavily. This is a common technique in RD analysis, but it is not always clear how to do this with GLMs in the bayesian framework. We show how to do this with the PyMC.

ATE Estimation for Count Data

Wed, 07 Jun 2023 00:00:00 +0000

This notebook is a continuation of the previous notebook on ATE estimation for binary data with logistic regression based on the sequence of (great!) posts by Solomon Kurz. In this notebook, we will focus on count data. We reproduce in python an example presented in the post Causal inference with count regression by Solomon Kurz. Our intention is to simply show how to port these type of model to bambi. In addition, as in the previous post, we compare the ATE estimation with a simple linear regression model.

ATE Estimation with Logistic Regression

Sun, 04 Jun 2023 00:00:00 +0000

In this notebook, I want to reproduce some components of the extensive blog post Causal inference with Bayesian models by Solomon Kurz. Specifically, I want to deep dive into the logistic regression model used to estimate the average treatment effect (ATE) of the study Internet-accessed sexually transmitted infection (e-STI) testing and results service: A randomised, single-blind, controlled trial by Wilson, et.al. I can only recommend to read the original sequence of posts Solomon has written on causal inference. They are very well written, easy to follow and provide a lot of insights into the topic.

Bayesian Methods in Modern Marketing Analytics Webinar with PyMC Labs

Wed, 31 May 2023 00:00:00 +0000

Here I want to share the recording and slides of the webinar Bayesian Methods in Modern Marketing Analytics in collaboration with PyMC Labs.

Abstract: During the webinar, we will discuss some of the most crucial topics in marketing analytics: media spend optimization through media mix models and experimentation, and customer lifetime value estimation. We will approach these topics from a Bayesian perspective, as it gives us great tools to have better models and more actionable insights. We will take this opportunity to describe our join with PyMC Labs in open-sourcing some of these tools in our brand-new pymc-marketing Python package.

How to vectorize an scikit-learn transformer over a numpy array?

Mon, 01 May 2023 00:00:00 +0000

In this short post, I show how to vectorize an scikit-learn transformer over a numpy array. That is, how to apply a transformer along a specific axes of a numpy array. I have found this to be particularly useful when working with output sample posterior distributions from a bayesian model where I want to apply a transformer to each sample. This is not particularly difficult, but I always forget how to do it, so I thought I would write it down once and for all 😄.

Counting the Number of Kitas per PLZ in Berlin using a Hierarchical Bayesian Model

Fri, 28 Apr 2023 00:00:00 +0000

This notebook is the continuation of data gathering and data analysis post Open Data: Berlin Kitas. In this second part we use the data gathered to model the number of Kitas per PLZ in Berlin using a hierarchical bayesian model. The hierarchy is defined by the Berlin districts. The objective is to develop a sound basic model which can be enhanced in the future with a richer data set.

Prepare Notebook

import arviz as az
import geopandas as gpd
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpy as np
import numpyro
import numpyro.distributions as dist
import pandas as pd
import seaborn as sns
from jax import random
from jaxlib.xla_extension import ArrayImpl
from numpyro.infer import MCMC, NUTS, Predictive
from sklearn.preprocessing import LabelEncoder

plt.style.use("bmh")
plt.rcParams["figure.figsize"] = [10, 6]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"

numpyro.set_host_device_count(n=4)

rng_key = random.PRNGKey(seed=0)

%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = "retina"

Read Data

Let’s start by reading the data we gathered in the previous post.

Simple Hierarchical Model with NumPyro: Cookie Chips Example

Mon, 24 Apr 2023 00:00:00 +0000

This notebook presents a simple example of a hierarchical model using NumPyro. The example is based on the cookie chips example in presented in the post Introduction to Bayesian Modeling with PyMC3. There are many great resources regarding bayesian hierarchical model and probabilistic programming NumPyro. This notebook aims to provide a succinct simple example to get started.

Remark: Well, the real reason is that I want to get acquainted other probabilistic programming libraries in order to abstract the core principles of probabilistic programming.

Experimentation, Non-Compliance and Instrumental Variables with PyMC

Mon, 20 Feb 2023 00:00:00 +0000

In this notebook we present an example of how to use PyMC to estimate the effect of a treatment in an experiment where there is non-compliance through the use of instrumental variables.

By non-compliance we mean that the treatment assignment does not guarantee that the treatment is actually received by the treated. The main challenge is that we can not simply estimate the treatment effect as a difference in means since the non-compliance mechanism is most of the time not at random and may introduce confounders. For a more detailed discussion of the problem see Chapter 9 in the (amazing!) book Causal Inference for The Brave and True by Matheus Facure Alves. One very common tool to tackle such problem are instrumental variables (see Chapter 8) for a nice introduction. In essence, we can use the variant assignment as an instrument to control for confounders introduced by the non-compliance mechanism.

Cohort Revenue & Retention Analysis: A Bayesian Approach

Mon, 23 Jan 2023 00:00:00 +0000

In this notebook we extend the cohort retention model presented in the post Cohort Retention Analysis with BART so that we just model retention and per cohort simultaneously (we recommend reading the referenced post before this one). The idea is to keep modeling the retention using a Bayesian Additive Regression Tree (BART) model (see pymc-bart) and linearly model the revenue per cohort using a Gamma distribution. We couple the retention and revenue components in a similar way as presented in the notebook Introduction to Bayesian A/B Testing. For this simulated example we use a synthetic data set, see the blog post A Simple Cohort Retention Analysis in PyMC For more details. Here you can find the data to reproduce the results.

Cohort Retention Analysis with BART

Mon, 02 Jan 2023 00:00:00 +0000

In this notebook we study an alternative approach for the cohort analysis problem presented in A Simple Cohort Retention Analysis in PyMC. Instead of using a linear model to estimate the retention rate, we use a Bayesian Additive Regression Tree (BART) model(see pymc-bart). The BART model is a flexible non-parametric model that can be used to model complex relationships between the response and the predictors.

Prepare Notebook

import arviz as az
import matplotlib.pyplot as plt
import matplotlib.ticker as mtick
import numpy as np
import pandas as pd
import pymc as pm
import pymc_bart as pmb
import pytensor.tensor as pt
import seaborn as sns

from scipy.special import expit, logit
from sklearn.preprocessing import LabelEncoder
from pymc_bart.split_rules import ContinuousSplitRule, SubsetSplitRule

az.style.use("arviz-darkgrid")
plt.rcParams["figure.figsize"] = [12, 7]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"

%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = "retina"

seed: int = sum(map(ord, "retention"))
rng: np.random.Generator = np.random.default_rng(seed=seed)

Read Data

Here we simply read the data from the previous post.

A Simple Cohort Retention Analysis in PyMC

Tue, 20 Dec 2022 00:00:00 +0000

In this notebook we present a simple approach to study cohort retention analysis through a simulated data set. The aim is to understand how retention rates change over time and provide a simple model to predict them (with uncertainty estimates!). We do not expect this technique to be a silver bullet for all retention problems, but rather a simple approach to get started with the problem.

Remark: A motivation for this notebook was the great post Bayesian Age/Period/Cohort Models in Python with PyMC by Austin Rochford.

Geo-Experimentation via Time Based Regression in PyMC

Thu, 01 Dec 2022 00:00:00 +0000

Introduction

In this notebook I describe and present an implementation of the time based regression (TBR) approach to marketing campaign analysis in the context of geo experimentation presented in the paper Estimating Ad Effectiveness using Geo Experiments in a Time-Based Regression Framework by Jouni Kerman, Peng Wang and Jon Vaver (Google, Inc. 2017). I strongly recommend reading the paper as it is quite clear in the exposition of the approach and presents some simulation results. Here I will focus on the basic model specification and on the implementation in PyMC.

Offline Campaign Analysis Measurement: A journey through causal impact, geo-experimentation and synthetic control

Tue, 25 Oct 2022 00:00:00 +0000

In October 2022 I had the opportunity to give a talk at the Helsinki Data Science Meetup hosted by Wolt. Here I want to share the recording of my talk.

Abstract: The talk will show how to measure offline campaigns using causal inference techniques. In particular it’ll focus on tapping into the potential of synthetic control, geo-experiments via time-based regression, and Google’s Causal-Impact Method.

Code to generate data

You can find the raw data here and the code here.

Scikit-Learn Example in PyMC: Gaussian Process Classifier

Sat, 24 Sep 2022 00:00:00 +0000

In this notebook we want to describe how to port a couple of classification examples from scikit-learn’s documentation (classifier comparison) to PyMC. We focus in the classical moons synthetic dataset.

Prepare Notebook

import arviz as az
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import pymc as pm
import pymc.sampling_jax
import seaborn as sns
from sklearn.datasets import make_moons
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split


plt.style.use("bmh")
plt.rcParams["figure.figsize"] = [8, 6]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"

%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = "retina"

Generate Raw Data

We generate synthetic data using the moons function from sklearn.datasets and split it into a training and test set.

Synthetic Control in PyMC

Tue, 09 Aug 2022 00:00:00 +0000

Synthetic control can be considered “the most important innovation in the policy evaluation literature in the last few years” (see The State of Applied Econometrics: Causality and Policy Evaluation by Susan Athey and Guido W. Imbens).

In this notebook we provide an example of how to implement a synthetic control problem in PyMC to answer a “what if this had happened?” type of question in the context of causal inference. We reproduce the results of the example provided in the great book Causal Inference for The Brave and True by Matheus Facure. Specifically, we look into the problem of estimating the effect of cigarette taxation on its consumption presented in Chapter 15 - Synthetic Control. The purpose of this notebook is not to present a general description of this method but rather show how to implement it in PyMC. Therefore, we strongly recommend to look into the book to understand the problem, motivation and details of the method.

Modeling Short Time Series with Prior Knowledge in PyMC

Tue, 19 Jul 2022 00:00:00 +0000

In this notebook I want to reproduce in PyMC the methodology described in the amazing blog post Modeling Short Time Series with Prior Knowledge by Tim Radtke to forecast short time series using bayesian transfer learning 🚀. The main idea is to transfer information (e.g. long term seasonality) from a long time series to a short time series via prior distributions. Tim’s blog post treats a very concrete example where all the concepts become very concrete. The challenge of the example is to generate long term forecast for a short time series of bike sales data. Specifically, the input sales data consists of three months of daily data and the objective is to generate at least a two years forecast. In general this is very hard to to with commonly available methods (as we will show below) due the fact we do not have enough historical data to capture seasonal patterns. For this concrete example, we do expect to have a strong yearly seasonal pattern as bike sales are usually much higher during summer than in winter. Hence, we could use temperature as a proxy for this seasonal pattern. However, as mentioned above, we can not simply try to use such data in a model with just 3 months of daily data … ¯\(ツ)/¯ …

Time-Varying Regression Coefficients via Gaussian Random Walk in PyMC

Sun, 03 Jul 2022 00:00:00 +0000

In this notebook we want to illustrate how to use PyMC to fit a time-varying coefficient regression model. The motivation comes from post Exploring Tools for Interpretable Machine Learning where we studied a time series problem, regarding the prediction of the number of bike rentals, from a machine learning perspective. Concretely, we fitted and compared two machine learning models: a linear regression with interactions and a gradient boost model (XGBoost). The models regressors were mainly meteorological data and seasonality features. One interesting feature we saw, through PDP and ICE plots was that the temperature feature had a non-constant effect over the bike rentals (see here). Indeed, when the temperature is high (more than 25 degrees approximately), the bike rentals are negatively impacted by the temperature (to be fair, this is when controlling by other regressors) on average. What we want to do in this notebook is to tackle the same problem from a different perspective. Namely, use to use a GaussianRandomWalk to model the interaction effect between the temperature and the bike rentals. We of course start with the simple regression baseline for comparison.

Data Talks Club: Machine Learning in Marketing

Tue, 17 May 2022 00:00:00 +0000

On Friday 13th of May 2022 I was invited to join Alexey Grigorev in an event organised by DataTalks.Club to talk about Machine Learning in Marketing. It was a really insightful discussion and i would like to thanks the organizers who make it possible. Here is the recording:

Here are some useful links and resources about the subject:

Relevant blog post I have written about the subject:
Books and references to learn more about marketing data science

PyConDE & PyData Berlin 2022: Introduction to Uplift Modeling

Mon, 11 Apr 2022 00:00:00 +0000

In this notebook we present a simple example of uplift modeling estimation via meta-models using causalml and scikit-uplift. For a more detailed introduction to uplift modeling, see:

Gamma-Gamma Model of Monetary Value in PyMC

Tue, 29 Mar 2022 00:00:00 +0000

In this notebook we describe how to fit Fader’s and Hardie’s gamma-gamma model presented in the paper “RFM and CLV: Using Iso-value Curves for Customer Base Analysis” and the note “The Gamma-Gamma Model of Monetary Value”. The approach is very similar as the one presented in the previous post BG/NBD Model in PyMC where we simply ported the log-likelihood of the lifetimes package from numpy to pytensor.

Prepare Notebook

import arviz as az
import matplotlib.pyplot as plt
import numpy as np
import pymc as pm
import pytensor.tensor as pt
import seaborn as sns
from lifetimes import GammaGammaFitter
from lifetimes.datasets import load_cdnow_summary_data_with_monetary_value

az.style.use("arviz-darkgrid")
plt.rcParams["figure.figsize"] = [12, 7]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"

seed = sum(map(ord, "juanitorduz"))
rng = np.random.default_rng(seed)

%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = "retina"

Load Data

We are going to use an existing data set from the lifetimes package documentation, see here.

BG/NBD Model in PyMC

Thu, 03 Mar 2022 00:00:00 +0000

In this notebook we show how to port the BG/NBD model from the the lifetimes (developed mainly by Cameron Davidson-Pilon) package to pymc. The BG/NBD model, introduced in the seminal paper “Counting Your Customers” the Easy Way: An Alternative to the Pareto/NBD Model by Peter S. Fader, Bruce G. S. Hardie and Ka Lok Lee in 2005, is used to

predict future purchasing patterns, which can then serve as an input into “lifetime value” calculations, in the “non-contractual” setting (i.e., where the opportunities for transactions are continuous and the time at which customers become inactive is unobserved).

Media Effect Estimation with PyMC: Adstock, Saturation & Diminishing Returns

Fri, 11 Feb 2022 00:00:00 +0000

In this notebook we present a concrete example of estimating the media effects via bayesian methods, following the strategy outlined in Google’s paper Jin, Yuxue, et al. “Bayesian methods for media mix modeling with carryover and shape effects.” (2017). This example can be considered the continuation of the post Media Effect Estimation with Orbit’s KTR Model. However, it is not strictly necessary to read before as we make this notebook self-contained. In addition, we provide some remarks and references regarding MMM projects in practice.

Media Effect Estimation with Orbit's KTR Model

Fri, 04 Feb 2022 00:00:00 +0000

In this notebook we want to experiment to the new KTR model included in the new orbit’s release (1.1). In particular, we are interested in its applications to media effects estimation in the context of media mix modeling. This is one of the applications for the KTR model by the Uber’s team, see the corresponding paper Edwin, Ng, et al. “Bayesian Time Varying Coefficient Model with Applications to Marketing Mix Modeling”.

KTR stands for kernel-based Time-varying regression as its main feature is to fit time-variant regression coefficients in a bayesian forecasting model. For details on the KTR model, see the paper mentioned above and KTR documentation where there is a great sequence of tutorials. In a nutshell, given a time series \(y_t\) the KTR models its components as \(y_t = l_t + s_t + r_t + \varepsilon_t\), where \(l_t\) is the local-trend (level) , \(s_t\) is the seasonal component, \(r_t\) are the regression terms and \(\varepsilon_t\) is a stationary random error process. This is very similar to the Prophet model (we actually compare these two models bellow). One of the main differences is that the regression coefficients are time-varying. From the model documentation:

Unobserved Components Model as a Bayesian Model with PyMC

Fri, 10 Dec 2021 00:00:00 +0000

In this notebook I want to deep-dive into the idea of wrapping a statsmodels UnobservedComponents model as a bayesian model with PyMC described in the (great!) post Fast Bayesian estimation of SARIMAX models. This is a nice excuse to get into some internals of how PyMC works. I hope this can serve as a complement to the original post mentioned above. This post has two parts: In the first one we fit a UnobservedComponents model to a simulated time series. In the second part we describe the process of wrapping the model as a PyMC model, running the MCMC and sampling and generating out of sample predictions.

ISLR2 - Survival Analysis Lab (lifelines)

Wed, 01 Sep 2021 00:00:00 +0000

In this notebook we provide a python implementation of the lab from the Survival Analysis - Chapter 11 of the second edition of the book An Introduction to Statistical Learning (Second Edition). You can find a free pdf version of the book here. We will use the lifelines python package, which you can find in this repository. There is a nice introduction into survival analysis on the documentation. There are also many concrete examples and guidelines to use the package.

Exploring Tools for Interpretable Machine Learning

Thu, 01 Jul 2021 00:00:00 +0000

In this notebook we want to test various ways of getting a better understanding on how non-trivial machine learning models generate predictions and how features interact with each other. This is in general not straight forward and key components are (1) understanding on the input data and (2) domain knowledge on the problem. Two great references on the subject are:

Note that the methods discussed in this notebook are not related with causality. I strongly recommend to refer to the article Be Careful When Interpreting Predictive Models in Search of Causal Insights by Scott Lundberg (one of the core developers of SHAP). The following are two references I have found particularly useful as an introduction to causal inference:

Feature Engineering: patsy as FormulaTransformer

Sat, 01 May 2021 00:00:00 +0000

In this notebook I want to describe how to create features inside scikit-learn pipelines using patsy-like formulas. I have used this approach to generate features in a previous post: GLM in PyMC3: Out-Of-Sample Predictions, so I will consider the same data set here for the sake of comparison.

Remark: Very recently (2021-09-01) I discovered there is an implementation of this transformer in scikit-lego, see PatsyTransformer. In addition, please refer to the great tutorial on patsy in calmcode.io.

GLM in PyMC3: Out-Of-Sample Predictions

Mon, 04 Jan 2021 00:00:00 +0000

In this notebook I explore the glm module of PyMC3. I am particularly interested in the model definition using patsy formulas, as it makes the model evaluation loop faster (easier to include features and/or interactions). There are many good resources on this subject, but most of them evaluate the model in-sample. For many applications we require doing predictions on out-of-sample data. This experiment was motivated by the discussion of the thread “Out of sample” predictions with the GLM sub-module on the (great!) forum discourse.pymc.io/, thank you all for your input!

Gaussian Processes for Time Series Forecasting with PyMC3

Sat, 02 Jan 2021 00:00:00 +0000

In this notebook we translate the forecasting models developed for the post on Gaussian Processes for Time Series Forecasting with Scikit-Learn to the probabilistic Bayesian framework PyMC3. I strongly recommend looking into the following references for more details and examples:

References:

Prepare Notebook1

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style(
    style='darkgrid', 
    rc={'axes.facecolor': '.9', 'grid.color': '.8'}
)
sns.set_palette(palette='deep')
sns_c = sns.color_palette(palette='deep')

import pymc3 as pm
import arviz as az

plt.rcParams['figure.figsize'] = [12, 6]
plt.rcParams['figure.dpi'] = 100

Generate Sample Data

Instead of the step-by-step approach we took in Gaussian Processes for Time sSeries Forecasting with Scikit-Learn, we are just going to develop one model.

Simple Bayesian Linear Regression with TensorFlow Probability

Tue, 06 Oct 2020 00:00:00 +0000

In this post we show how to fit a simple linear regression model using TensorFlow Probability by replicating the first example on the getting started guide for PyMC3. We are going to use Auto-Batched Joint Distributions as they simplify the model specification considerably. Moreover, there is a great resource to get deeper into this type of distribution: Auto-Batched Joint Distributions: A Gentle Tutorial, which I strongly recommend (see this post to get a brief introduction on TensorFlow probability distributions). In addition the tutorial: Bayesian Modeling with Joint Distribution is also a great reference to get started with linear models in TensorFlow Probability.

Open Data: Berlin Kitas

Sat, 19 Sep 2020 00:00:00 +0000

In this notebook I want to explore some data I found on the Berlin Open Data portal daten.berlin.de. The data source contains information of Kitas (Kindertagesstätte, i.e. kindergartens) in Berlin. This is a big topic as finding a spot in a Kita in Berlin is extremely difficult. We first provide an initial exploratory data analysis of the data set, then we merge it with population data to create some geo-location maps.

Disclaimer: In our way we do not expect to answer on how to find a Kita, but rather simpler questions about their location, distribution and sizes.

A Simple Hamiltonian Monte Carlo Example with TensorFlow Probability

Fri, 24 Jul 2020 00:00:00 +0000

In this post we want to revisit a simple bayesian inference example worked out in this blog post. This time we want to use TensorFlow Probability (TFP) instead of PyMC3.

References:

Statistical Rethinking is an amazing reference for Bayesian analysis. It also has a sequence of online lectures freely available on YouTube.
An introduction to probabilistic programming, now available in TensorFlow Probability
There are many examples on the TensorFlow’s GitHub repository. I am following the case study Bayesian Switchpoint Analysis for this example.

Regression Analysis & Visualization

Fri, 26 Jun 2020 00:00:00 +0000

In this notebook I want to collect some useful visualizations which can help model development and model evaluation in the context of regression analysis. I use many visualization resources not just only to share results but as a key component of my workflow: data QA, EDA, feature engineering, model development, model evaluation and communicating results. In this notebook I focus on a simple regression model (time series) with statsmodels and visualization with matplotlib and seaborn. The focus is not on the model development but rather visualization.

A Glimpse into TensorFlow Probability Distributions

Tue, 16 Jun 2020 00:00:00 +0000

In this notebook we want to go take a look into the distributions module of TensorFlow probability. The aim is to understand the fundamentals and then explore further this probabilistic programming framework. Here you can find an overview of TensorFlow Probability. We will concentrate on the first part of Layer 1: Statistical Building Blocks. As you could see from the distributions module documentation, there are many classes of distributions. We will explore a small sample of them in order to get an overall overview. I find the documentation itself a great place to start. In addition, there is a sample of notebooks with concrete examples on the GitHub repository. In particular, I will follow some of cases presented on the A_Tour_of_TensorFlow_Probability notebook, expand on some details and add some other additional examples.

Disease Spread Simulation (Animation)

Tue, 28 Apr 2020 00:00:00 +0000

We describe how to generate a basic disease spread simulation. We explore how to do animations in Matplotlib.

Getting Started with Spectral Clustering

Sat, 04 Apr 2020 00:00:00 +0000

In this post I want to explore the ideas behind spectral clustering. I do not intend to develop the theory. Instead, I will unravel a practical example to illustrate and motivate the intuition behind each step of the spectral clustering algorithm. I particularly recommend two references:

For an introduction/overview on the theory, see the lecture notes A Tutorial on Spectral Clustering by Prof. Dr. Ulrike von Luxburg.
For a concrete application of this clustering method you can see the PyData’s talk: Extracting relevant Metrics with Spectral Clustering by Dr. Evelyn Trautmann.

Prepare Notebook

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('darkgrid', {'axes.facecolor': '.9'})
sns.set_palette(palette='deep')
sns_c = sns.color_palette(palette='deep')
%matplotlib inline

from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()

Generate Sample Data

Let us generate some sample data. As we will see, spectral clustering is very effective for non-convex clusters. In this example, we consider concentric circles:

The Volume of the d-Ball via Monte Carlo Simulation

Mon, 24 Feb 2020 00:00:00 +0000

In this notebook we run Monte Carlo simulations to estimate the volume of the \(d\)-ball \[ B^{d}:=\{x \in \mathbb{R}^d : ||x|| \leq 1\}. \] There are many ways to obtain a closed formula for this volume , see for example this Wikipedia article. Here we do it via sampling just for fun!

Main Idea

Consider a square \(A_{d}\subset \mathbb{R}\) centered at the origin with side length \(2\). We estimate the volume of the \(d\)-ball \(B^{d}:=\{x \in \mathbb{R}^d : ||x|| \leq 1\}\subset A^{d}\) by sampling uniformly from \(A\) and computing the proportions of vectors having length less or equal than one. Note that \(\text{vol}(A^{d}) = 2^{d}\).

Forecasting Weekly Data with Prophet

Fri, 21 Feb 2020 00:00:00 +0000

In this notebook we are present an initial exploration of the Prophet package by Facebook. From the documentation:

Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.¹

Exploring TensorFlow Probability STS Forecasting

Tue, 11 Feb 2020 00:00:00 +0000

In this notebook we explore the Structural Time Series (STS) Module of TensorFlow Probability. We follow closely the use cases presented in their Medium blog. As described there: An STS model expresses an observed time series as the sum of simpler components ¹:

\[ f(t) = \sum_{k=1}^{N}f_{k}(t) + \varepsilon, \quad \text{where}\quad \varepsilon \sim N(0, \sigma^2). \]

Each summand \(f_{k}(t)\) has a particular structure, e.g. specific seasonality, trend, autoregressive terms, etc.

In this notebook we generate a time series sample and then present some techniques to recover its component structure. This is indeed a crucial step since in real applications we are not given the components separately. We then show how to fit and generate predictions using variational inference. Finally we run some diagnostics of the errors on the test set.

Intro ML in Production: Flask, Docker and GitHub Actions

Tue, 28 Jan 2020 00:00:00 +0000

We describe how to set up a toy-model repository to train and dockerize a machine learning model with data store on aws s3.

Drawing Manifolds in LaTeX with TikZ

Fri, 10 Jan 2020 00:00:00 +0000

We give some LaTex code to create figures of manifolds with boundaries.

Open Data: Germany Maps Viz

Tue, 07 Jan 2020 00:00:00 +0000

In this post I want to show how to use public available (open) data to create geo visualizations in python. Maps are a great way to communicate and compare information when working with geolocation data. There are many frameworks to plot maps, here I focus on matplotlib and geopandas (and give a glimpse of mplleaflet).

Reference: A very good introduction to matplotlib is the chapter on Visualization with Matplotlib from the Python Data Science Handbook by Jake VanderPlas.

The Graph Laplacian & Semi-Supervised Clustering

Thu, 05 Dec 2019 00:00:00 +0000

In this post we want to explore the semi-supervided algorithm presented Eldad Haber in the BMS Summer School 2019: Mathematics of Deep Learning, during 19 - 30 August 2019, at the Zuse Institute Berlin. He developed an implementation in Matlab which you can find in this GitHub repository. In addition, please find the corresponding slides here.

Prepare Notebook

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()

%matplotlib inline

Generate Sample Data

Let us generate the sample data as 3 concentric circles:

The Lapacian on the 2-Torus

Sun, 13 Oct 2019 00:00:00 +0000

In this blog post I want to describe the explicit computation of the Laplacian on differential forms on the \(2\)-Torus \(T^2\subset \mathbb{R}^3\). This surface can be obtained by rotating the circle \((x-a)^2+y^2=r^2\) around the \(z\)-axis (\(0<r<a\)). Locally, this surface can be parametrized by the equations \[ x = (a+r\cos u)\cos v,\\ y = (a+r\cos u)\sin v,\\ z = r\sin u, \]

where \(0<u,v<2\pi\).

Remark: Note that, topologically, \(T^2\) can be identified with the product of two circles. In particular, it is a Lie group.

PyData Berlin 2019: Gaussian Processes for Time Series Forecasting (scikit-learn)

Thu, 10 Oct 2019 00:00:00 +0000

In this notebook we run some experiments to demonstrate how we can use Gaussian Processes in the context of time series forecasting with scikit-learn. This material is part of a talk on Gaussian Process for Time Series Analysis presented at the PyCon DE & PyData 2019 Conference in Berlin.

Update: Additional material and plots were included for the Second Symposium on Machine Learning and Dynamical Systems at The Fields Institute (virtual event).

Videos

PyData Berlin 2019

Second Symposium on Machine Learning and Dynamical Systems 2020

Slides

satRday Berlin 2019: Remedies for Severe Class Imbalance

Sat, 15 Jun 2019 00:00:00 +0000

In this post I present a concrete case study illustrating some techniques to improve model performance in class-imbalanced classification problems. The methodologies described here are based on Chapter 16: Remedies for Severe Class Imbalance of the (great!) book Applied Predictive Modeling by Max Kuhn and Kjell Johnson. I absolutely recommend this reference to anyone interested in predictive modeling.

This notebook should serve as an extension of my talk given at satRday Berlin 2019: A conference for R users in Berlin. Here are the slides:

Seasonal Bump Functions

Thu, 11 Apr 2019 00:00:00 +0000

Motivated by the nice talk on Winning with Simple, even Linear, Models by Vincent D. Warmerdam, I briefly describe how to construct certain class of bump functions to encode seasonal variables in R.

Prepare Notebook

library(glue)
library(lubridate)
library(magrittr)
library(tidyverse)

Generate Data

Let us generate a time sequence variable stored in a tibble.

# Define time sequence. 
t <- seq.Date(from = as.Date("2017-07-01"), to = as.Date("2019-04-01"), by = "day")
# Store it in a tibble. 
df <- tibble(t = t)

Symmetric Bump Function

We want to generate a gaussian-like curve around a specific date. The width of the bump function is parametrized by a non-negative number \(\varepsilon \geq 0\). For \(\varepsilon = 0\) we generate a dummy variable. Namely, for \(\varepsilon \neq 0\),

An Introduction to Gaussian Process Regression

Mon, 08 Apr 2019 00:00:00 +0000

Updated Version: 2019/09/21 (Extension + Minor Corrections)

After a sequence of preliminary posts (Sampling from a Multivariate Normal Distribution and Regularized Bayesian Regression as a Gaussian Process), I want to explore a concrete example of a gaussian process regression. We continue following Gaussian Processes for Machine Learning, Ch 2.

Other recommended references are:

Gaussian Processes for Timeseries Modeling by S. Roberts, M. Osborne, M. Ebden, S. Reece, N. Gibson & S. Aigrain.
Bayesian Data Analyis, Chapter 21 by by Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin.

Prepare Notebook

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style(
    style='darkgrid', 
    rc={'axes.facecolor': '.9', 'grid.color': '.8'}
)
sns.set_palette(palette='deep')
%matplotlib inline
plt.rcParams['figure.figsize'] = [12, 6]
plt.rcParams['figure.dpi'] = 100

Generate Data Samples

We consider de model \(y = f(x) + \varepsilon\), where \(\varepsilon \sim N(0, \sigma_n)\). Here \(f\) does not need to be a linear function of \(x\). As a concrete example, let us consider (1-dim problem)

Bayesian Regression as a Gaussian Process

Mon, 01 Apr 2019 00:00:00 +0000

In this post we study the Bayesian Regression model to explore and compare the weight and function space and views of Gaussian Process Regression as described in the book Gaussian Processes for Machine Learning, Ch 2. We follow this reference very closely (and encourage to read it!). Our main objective is to illustrate the concepts and results through a concrete example. We use PyMC3 to run bayesian sampling.

References:

Gaussian Processes for Machine Learning, Carl Edward Rasmussen and Christopher K. I. Williams, MIT Press, 2006.
See this post for an introduction to bayesian methods and PyMC3.
Documentation of linear regression in PyMC3.
Documentation for the multivariate normal distribution in PyMC3.
Here is an Stackoverflow post which can help Mac OS users which might have problems with Theano after upgrading to Mojave.

Let us consider the model:

Sampling from a Multivariate Normal Distribution

Sat, 23 Mar 2019 00:00:00 +0000

In this post I want to describe how to sample from a multivariate normal distribution following section A.2 Gaussian Identities of the book Gaussian Processes for Machine Learning. This is a first step towards exploring and understanding Gaussian Processes methods in machine learning.

Multivariate Normal Distribution

Recall that a random vector \(X = (X_1, , X_d)\) has a multivariate normal (or Gaussian) distribution if every linear combination

\[ \sum_{i=1}^{d} a_iX_i, \quad a_i\in\mathbb{R} \] is normally distributed.

Dockerize a ShinyApp

Sat, 02 Mar 2019 00:00:00 +0000

In this post I want to describe how to dockerize a simple Shiny App. Docker is a great way of sharing and deploying projects. You can download it here.

Resources:

R Docker tutorial, recommended for Docker beginners.
Running a shiny app in a docker container by Mark Sellors (which is an updated and more complete version of this post).

Assume you have a project folder structure as follows:

.
+-- project.Rproj
+-- app.R
+-- R
|   +-- script_1.R
|   +-- script_2.R
+-- data
|   +-- data_df.rds
|   +-- raw_data.csv

The script app.R contains the code of the Shiny application.
The folder R contains auxiliary scripts.
The folder data contains a csv and an rds file which might be nessesary for the application (cleaned data set, for example).

To create a docker image you need to create a Dockerfile, which is a simple text file located in the same directory as app.R. The name of the file must be Dockerfile. A common structure of this file is the following:

The Spectral Theorem for Matrices

Sat, 02 Feb 2019 00:00:00 +0000

When working in data analysis it is almost impossible to avoid using linear algebra, even if it is on the background, e.g. simple linear regression. In this post I want to discuss one of the most important theorems of finite dimensional vector spaces: the spectral theorem. The objective is not to give a complete and rigorous treatment of the subject, but rather show the main ingredients, some examples and applications.

Movie Plots Text Generation with Keras

Sun, 13 Jan 2019 00:00:00 +0000

In this post I show some text generation experiments I ran using LSTM with Keras. For the preprocessing and tokenization I used SpaCy. The aim is not to present a completed project, but rather a first step which should be then iterated.

Resources

There are many great resources and blog posts about the subject (and similar experiments). Here I mention the ones I found particularly useful for the general theory:

Online Resources:
- Deep Learning Specialization, Coursera by Andrew Ng.

Exploring the Curse of Dimensionality - Part II.

Tue, 01 Jan 2019 00:00:00 +0000

I continue exploring the curse of dimensionality. Following the analysis form Part I., I want to discuss another consequence of sparse sampling in high dimensions: sample points are close to an edge of the sample. This post is based on The Elements of Statistical Learning, Section 2.5, which I encourage to read!

Uniform Sampling

Consider \(N\) data points uniformly distributed in a \(p\)-dimensional unit ball centered at the origin. Suppose we consider a nearest-neighbor estimate at the origin. The median distance from the origin to the closest data point is given by the expression \[ d(p,N) = \left(1-\frac{1}{2}^{1/N}\right)^{1/p}. \]

Text Mining, Networks and Visualization: Plebiscito Tweets

Thu, 20 Dec 2018 00:00:00 +0000

Nowadays social media generates a vast amount of raw data (text, images, videos, etc). It is a very interesting challenge to discover techniques to get insights on the content and development of social media data. In addition, as a fundamental component of the analysis, it is important to find ways of communicating the results, i.e. data visualization. In this post I want to present a small case study where I analyze Twitter text data. The aim is not to give a complete analysis (as it would require many interations), but rather to describe how to to start. The emphasis of this post is in the data manipulation and data visualization. In particular, I describe how networks (graphs) can be used as data structures to describe text relations (some measure of pairwise count occurrences).

Exploring the Curse of Dimensionality - Part I.

Sun, 09 Dec 2018 00:00:00 +0000

In this post I want to present the notion of curse of dimensionality following a suggested exercise (Chapter 4 - Ex. 4) of the book An Introduction to Statistical Learning, written by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.

When the number of features \(p\) is large, there tends to be a deterioration in the performance of KNN and other local approaches that perform prediction using only observations that are near the test observation for which a prediction must be made. This phenomenon is known as the curse of dimensionality, and it ties into the fact that non-parametric approaches often perform poorly when \(p\) is large. We will now investigate this curse.

From Pelican to Blogdown

Sun, 02 Dec 2018 00:00:00 +0000

Here I want to discuss my transition from Pelican to Blogdown and present some personal learnings. In June 2017 I decided to build a personal website/portafolio. I chose Pelican, because:

It is written in Python, which was the programing language I was mainly working on.
I wanted to include some Jupyter notebook I had already written.
A great post: Building a data science portfolio: Making a data science blog explaining the procedure and using GitHub Pages to publist it.

\(S^1\)-Equivariant Dirac operators on the Hopf Fibration

Sun, 11 Nov 2018 00:00:00 +0000

In this expository article I discuss the definition and basic properties of the Hopf fibration, with particular emphasis on Dirac-type operators induced, in the sense of Brüning and Heintze, by the Hodge-de Rham and spin-Dirac operators. In addition, we compute the Dirac-Schrödinger type operator introduced in my PhD thesis.

Introduction to R Plumber : Expose a Caret model to a web API

Fri, 12 Oct 2018 00:00:00 +0000

In this post we present a simple example of how to expose a prediction model to a web API using the Plumber package.

Circle Radius Fit for a Cloud of Points

Sun, 09 Sep 2018 00:00:00 +0000

We explore how to include an R notebook into a pelican post. As an example, we describe how to fit a circle onto a cloud of points.

From Bachelor to PhD: Geometric and Topological Methods for Quantum Field Theory

Thu, 02 Aug 2018 00:00:00 +0000

We give an introduction to PyMC3, a probabilistic programming framework written in Python. We revise the basic mathematical theory and present two concrete examples.

PyData Berlin 2018: On Laplacian Eigenmaps for Dimensionality Reduction

Sun, 08 Jul 2018 00:00:00 +0000

This post contains the slides and material from a talk I gave at PyData Berlin 2018. I presented the paper Laplacian Eigenmaps for Dimensionality Reduction and Data Representation by Mikhail Belkin and Partha Niyogi.

Probability that a given observation is part of a bootstrap sample?

Wed, 29 Nov 2017 00:00:00 +0000

We study the problem of computing the probability that a given observation is part of a bootstrap sample. We include some numerical simulations.

Induced Dirac-Schrödinger operators on semi-free circle quotients

Sat, 11 Nov 2017 00:00:00 +0000

I present the content of my PhD Thesis in mathematics, which has now been published in The Journal of Geometric Analysis.

Introduction to Bayesian Modeling with PyMC3

Sun, 13 Aug 2017 00:00:00 +0000

We give an introduction to PyMC3, a probabilistic programming framework written in Python. We revise the basic mahematical theory and present two concrete examples.

Web scraping with Beautiful Soup: Plebiscito Colombia (October 2nd)

Sun, 09 Jul 2017 00:00:00 +0000

We describe how to use Beautiful Soup to scrape the official goverment website in order to get the results of the peace referendum in Colombia.

The Dirac operator on the 2-sphere

Thu, 29 Jun 2017 00:00:00 +0000

The objective of this post is to explore MathJax, a JavaScript display engine for LaTeX. Being my first post writen with this tool, I want to present a short but fun example: I will give a description of the explicit computation of the spin-Dirac operator (of the unique complex spinor bundle!) on the 2-sphere \(S^2\) equipped with the standar round metric. A more detailed treatment can be found in my expository paper.

The Levi-Civita connection

Let us consider a 2-sphere of radius \(r>0\),

Python Exercise: Distance to Rectangle

Wed, 28 Jun 2017 00:00:00 +0000

In this first post we get started with a small python script to explore the basic capabilities of Pelican.

About

Mon, 01 Jan 0001 00:00:00 +0000