What is it?

Frequentist.de is my professional homepage as a Business Intelligence & Analytics freelancer based in Berlin. It lays out what I do, how I think about analytics, and who I typically work with. The site is written entirely in German, aimed at companies in the DACH region looking for hands-on BI expertise.

What’s on it

The site covers six service areas:

Who it’s for

The typical clients I expect to gain through frequentist.de are mid-sized companies, startups, and agencies in Berlin and the DACH region — either building internal data capabilities from scratch or professionalizing existing BI structures.

1 The Nine-Box Framework for Managing “Brain Fry”

One of the most common mistakes sales leaders make is “shoving the entire elephant” down their team’s throats by introducing too many changes at once. To combat this, Mark Kosoglow (CRO at Docebo) introduced a nine-box framework designed to measure a team’s “available capacity of change”.

The framework plots initiatives on two axes:

• Competency: Moving from Aware (passed a quiz) to Competent (internal role-play) to Mastery (proven in a real-world call).

• Cognitive Load: The amount of effort a rep must exert to change their existing behavior, categorized as Low, Medium, or High.

Under this system, a team is given a 17-point quarterly budget. A “High Cognitive Load Mastery” project is worth 9 points, meaning a team can only handle one such major behavioral shift per quarter. This forces leadership to make trade-offs rather than overwhelming reps with “brain fry”, where employee satisfaction plummets.

2 Moving from Call Samples to Aggregate Data Sets

Traditional coaching relies on managers manually reviewing a handful of “cherry-picked” calls — typically four or five a week — which provides a distorted view of performance. AI shifts this paradigm by providing an aggregate data set.

When every single call is automatically scored against a rubric, several things change:

• Accountability: Reps behave differently when they know every interaction is being reviewed.

• Action over Collection: Managers stop spending hours collecting data and instead spend those hours actioning.

• Personalization: Leaders can see team’s strengths and weaknesses, allowing them to tailor coaching to specific individual needs rather than generalities.

3 Forecasting via Binary Risk Assessment

Personalized coaching is described as the most critical driver of forecast accuracy. Many organizations struggle with “haircuts”, where each layer of management arbitrarily reduces a forecast because they don’t trust the underlying details.

To fix this, the speakers suggested creating a binary forecasting system. Instead of using vague terms like “best case” or “most likely”, AI-driven scorecards can determine risk based on objective, binary yes/no criteria.

By decoupling deal maturity (the sales stage) from risk assessment, one organization reported increasing their forecasting accuracy from 54% to 97%.

4 Rewarding the “Exposure of Risk”

AI tools now allow for proactive risk identification by tracking “competitive mentions” on calls and instantly alerting managers via Slack. This data is only useful if the organization fosters a culture where exposing risk is rewarded rather than punished.

The speakers emphasized that “losing alone is a fireable offense”. The goal of AI-supported deal reviews is to ensure that the minute a rep feels a deal is in danger, they flag it so the rest of the team can step in and work to save it.

AI provides the “receipts” to show reps that specific behaviors (like quantifying a pain point) can increase win rates significantly, turning the scorecard from a “Big Brother” surveillance tool into a “self-service” empowerment platform.

Network Architecture

This network is a Multilayer Perceptron (MLP) structured as a feed-forward architecture. It maps a single input scalar representing progress along a path through two hidden layers of 6 and 7 neurons to a 2D coordinate output.

• Input Layer (): A single node () representing the value , the amount traveled along the curve from 0 to 1.

• Hidden Layer 1 (): Six neurons using the logistic sigmoid activation function to capture initial non-linear features.

• Hidden Layer 2 (): Seven neurons that further process the signals to handle the complex geometric “lobes” of the heart curve.

• Output Layer (): Two nodes representing the and (or ) coordinates of the curve’s position in 2D space.

• Connectivity: The topology is fully connected, meaning every neuron in one layer is linked to every neuron in the next via a weight matrix () and a bias vector ().

Gradient Descent Process

For each iteration of the training loop, the network updates its parameters to minimize the distance between the predicted coordinates () and the target heart curve coordinates ().

Acknowledgments

This project is based on one of the lab exercises from Imperial College London’s “Multivariate Calculus for Machine Learning” course. I re-implemented the network in base R, translated the mathematical components from Python, and extended the project with visualization pipeline to produce a frame-by-frame learning animation.

Source Code

Source code is available on GitHub: Backpropagating Love.

See Also

Here are some related posts that you might find interesting:

	Title	Reading Time
	Animation of Spatial Data	7 min
	Nerdy Valentine’s in Python, R, and Matlab	4 min

No matching items

The problem isn’t price

Most travel today is leisure travel. And leisure travelers don’t optimize for the cheapest possible stay, but for the overall experience.

Price still matters, of course. But once price competition is saturated, reducing it further yields diminishing returns. At that point, the real differentiation comes from helping users make better decisions, not just cheaper ones.

One of the biggest hidden costs of a trip is time.

• Time spent commuting.

• Time spent navigating unfamiliar transit systems.

• Time spent realizing that a hotel was “well located” only on a map.

Optimization problem

When searching for a hotel, users are already solving a complex optimization problem.

They juggle price, amenities, aesthetics, availability, and location. On top of that, they carry a mental list of places they want to visit: attractions, museums, neighborhoods, restaurants.

Now add three constraints:

• unfamiliar city,

• multiple points of interest,

• spatial reasoning under uncertainty.

This quickly exceeds human cognitive limits. Most people cannot reliably reason about more than a handful of entities at once — and certainly not while doing spatial calculations in their head.

As a result, users fall back to heuristics. The most common one is proximity.

Beyond the proximity paradigm

“Close to the center” is the default shortcut, but proximity is a poor proxy for convenience:

• Distance travel time

• City center optimal access

• Straight-line proximity ignores transit topology

Two hotels equally distant from an attraction can have completely different travel times depending on transfers, line frequency, or network connectivity.

If travelers care about how much time they spend moving through a city (and they do) then optimizing for coordinates is the wrong abstraction.

We should optimize for time, not for distance.

Different concept: total transit time

Instead of asking how far from the city center a hotel is, we can ask a more relevant question:

How much time will a traveler spend getting to the places they want to visit?

The idea is simple:

• Let users select multiple points of interest.

• Calculate public transit time between hotels and those POIs.

• Rank accommodations by aggregate transit time.

There is no heavy machine learning in background involved. Just a deterministic algorithm precomputing transit times, and a simple aggregation function to rank hotels based on user-selected POIs.

Proof of concept

To test whether this idea is practical and meaningful, I built a proof of concept using Berlin as an example. The goal was to understand whether transit-time–based ranking reveals meaningful structure and whether it contradicts common assumptions.

I’ve gathered data on 455 hotels (Figure 1) and top 50 most popular points of interest in Berlin (Figure 2) and calculated 9,444 transit times between hotels and those attractions.

Then I simulated travelers visiting attractions based on different interest profiles. Assuming trip duration is 7 days, and traveler starts with a new location each day, I picked 7 random locations and calculated average transit time from each hotel to chosen locations (Figure 3, Figure 4, Figure 5).

Key findings

Looking at the results, several key insights emerge:

These effects are invisible when using distance-based filters, but immediately obvious when optimizing for time.

Product implications

From a product perspective, the implications are straightforward. This approach improves both user experience and marketplace balance.

Operational costs

From an engineering and analytics standpoint, this approach is lightweight.

• Transit times are precomputed.

• Routing APIs are inexpensive.

• Results are deterministic and predictable.

There is no real-time inference, no complex infrastructure, and no hidden operational risk. Running costs are close to zero and, more importantly, easy to forecast.

Risks and second-order effects

Any ranking change shifts market dynamics.

• Central hotels may face stronger competition as their implicit advantage erodes.

• Poorly connected hotels may lose visibility once transit time becomes explicit.

These are not unintended consequences. The filter surfaces information that already affects the user experience, whether platforms acknowledge it or not.

Next steps

Several steps would be needed to move from concept to product:

1. User research to validate perceived value (e.g. using Kano method).

2. Replicating the analysis across multiple cities.

3. A pilot implementation with a limited user segment.

4. A/B testing the impact on conversion, engagement, and satisfaction.

Conclusion

Optimizing hotel search for transit time rather than distance offers a more accurate and user-centric way to help travelers make informed decisions. It reveals hidden structure in the city that distance-based filters miss, and it has the potential to improve user experience and marketplace balance with minimal operational cost. As travel platforms seek new ways to differentiate, this approach provides a compelling opportunity to enhance decision support and create real value for users.

See Also

Here are some of my other posts related to product analytics, geospatial analysis, and data visualization:

	Title	Reading Time
	Practical Concepts for AI Driven Sales Coaching	5 min
	The 7-Step KPI Blueprint from Business Intelligence Analytics Perspective	14 min
	Minimum Detectable Effect (MDE) Calculation	6 min
	A/B Testing: Concepts and Techniques	37 min
	Animation of Spatial Data	7 min
	Kano Method for Prioritization of Features	10 min

No matching items

Key Features & Components

Implementation

Outcomes & Impact

Skills Demonstrated

LLM Orchestration • Retrieval-Augmented Generation (RAG) • Context-Aware Synthesis • Sales & Revenue Analytics • B2B Sales Methodology • Asynchronous API Design • Prompt Engineering • Data Visualization (Altair) • Multi-model LLM Architectures • Automated Insight Generation • End-to-End System Design

See Also

1. Create Objectives

This is where business intelligence analyst adds strategic value. The golden rule of KPI design I follow is simple:

From a data perspective, we can treat objectives as filters. They limit what deserves to be measured and prevent metrics overgrowth.

I look for three things in a good objective:

• It explicitly supports business strategy.
• It is material enough to influence decisions.
• It can be influenced by the organization (rather than external noise).

This step may require pushing back if stakeholders request KPIs before objectives are articulated. I treat that as a signal to pause and reframe.

2. Describe Results

In this step, we translate strategy into observable outcomes.

A common pitfall I see in KPI design is confusing activities with results.

I aim to describe results in language that is:

• Outcome-oriented.
• Free of vague terms like optimized, improved, or efficient.
• Observable and interpretable in the real world.

For analytics specialists, this step is critical because ambiguous results lead to ambiguous measures. If we cannot clearly observe success, we cannot reliably model it.

graph TD
    subgraph Strategic ["Level 1: Strategic"]
        A[Net Profit / ARR]
    end

    subgraph Tactical ["Level 2: Tactical"]
        B1[Customer Acquisition Cost]
        B2[Churn Rate]
        B3[Average Order Value]
    end

    subgraph Operational ["Level 3: Operational"]
        C1[Ad Spend / Clicks]
        C2[Support Ticket Vol]
        C3[Discount Usage]
        C4[Landing Page Conv %]
    end

    A --- B1
    A --- B2
    A --- B3

    B1 --- C1
    B1 --- C4
    B2 --- C2
    B3 --- C3

    style Strategic fill:#F8CECC,stroke:#333,stroke-width:2px
    style Tactical fill:#E1D5E7,stroke:#333,stroke-width:1px
    style Operational fill:#D5E8D4,stroke:#333,stroke-width:1px

3. Identify Measures

I believe a KPI should be expressible in one sentence containing countable entities. At this stage, I explicitly consider Lead vs. Lag vs. Diagnostic indicators.

4. Define Thresholds

A KPI without a threshold is just a data point; a KPI with a threshold is a call to action. For an analyst, the challenge is defining “Normal” vs. “Critical” without relying on gut feeling.

Beyond Static Targets

Many organizations use “flat” thresholds (e.g., Red if Sales < $100k). However, businesses are rarely static. I prefer a more mature BI approach:

Historical Baselines

Comparing performance against a rolling average or the same period last year (YoY) to account for seasonality.

Statistical Process Control (SPC)

Using standard deviations of the mean to define “Natural Variation”. If a metric falls within 1..2 SD, it’s a noise. If it crosses the 3rd, it’s a signal.

Dynamic Thresholds

Adjusting targets based on external variables (e.g., lower conversion targets during a known website migration).

RAG Model

The most common approach is the RAG model (Red, Amber, Green), where specific values are set to trigger a status change:

• Green: An acceptable result or on-target performance.
• Amber: A warning sign that requires investigation.
• Red: An unacceptable result requiring rectification.

Status	Threshold Logic	BI Implementation	Business Action
Green	of Target	Automated “Good News” report	Maintain current strategy
Amber	(Warning)	Trendline analysis & breakdown	Investigate root cause
Red	(Critical)	Real-time Slack/Email alert	Immediate tactical intervention

5. Model Structure and Data

This is where we see many KPI initiatives fail. Inexperienced teams build KPIs directly inside a visualization tool instead of building them into the data architecture.

6. Interpret Results

Interpretation is where we move from being a “data provider” to a “business partner.” Most dashboards fail because they show the what but leave the why as an exercise for the reader.

Three Pillars of Interpretation:

1. Contextual Benchmarking: Never present a number in isolation.
   • Bad: “Churn is 5%.”
   • Good: “Churn is 5%, which is a 12% increase MoM, primarily driven by the Enterprise segment.”
2. Cohort Analysis: Aggregates lie. A KPI might look stable while a specific cohort is collapsing. Always look beneath the surface to see if a segment is skewing the average.
3. Correlation vs. Causality: I use BI tools to overlay external events. Did the dip in “Engagement” happen exactly when the new UI was deployed? This helps us transform a correlation into a testable hypothesis.

7. Drive Action

The final stage is ensuring the data actually changes the trajectory of the business. I aim to move from Passive Monitoring to Active Orchestration.

KPI Definition Template

To put it all together, here is the template for documenting every core metric in a semantic layer. This ensures that the logic is transparent and the accountability is clear.

Download printable table: kpi-definition-template.pdf

Final Thoughts

KPI design is not a reporting task but a form of systems design. A strong framework creates alignment between strategy, data, and decisions.

See Also

Here are some of my other posts related to Business Intelligence, KPI design, and data visualization:

	Title	Reading Time
	Practical Concepts for AI Driven Sales Coaching	5 min
	Using Transit Time to Rethink Hotel Search	14 min
	Building an E-Commerce Dashboard with Power BI and R	5 min
	Building the Analytical Dashboard with Power BI and R	6 min
	Building a Credit Risk Dashboard with Power BI and R	8 min
	BI System Blueprint	2 min

No matching items

Microservices Approach

Building a local-first application that handles data munging, heavy UI rendering, and asynchronous secondary data fetching requires a robust architecture. I opted for a microservices-based approach orchestrated via Docker Compose. This ensures that the ML inference and data scraping doesn’t starve the web server of resources.

Technical Highlights

@lru_cache(maxsize=1024)
def get_embedding(text: str) -> list[float]:
    if not text.strip():
        return [0 for _ in range(768)]
    try:
        response = requests.post(
            TEXT_EMBEDDINGS_URL,
            json={"inputs": [text]},
        )
        response.raise_for_status()
        return response.json()[0]
    except Exception as e:
        # Fallback to zero-vector or log error
        return [0 for _ in range(768)]

# Identifying the representative name for Cluster #X
df["PosFreq"] = df.groupby("Position")["Cluster"].transform("count")
# Filter and sort to find the dominant title that isn't too short
positions = df.query("PosFreq == MaxFreq").drop_duplicates("Cluster")
# The resulting Cluster Name provides context like "Engineering Manager" or "Product Design"

System Robustness

An engineer’s work is defined not just by the “happy path”, but by how the system handles state, persistence, and deployment.

Future Roadmap

RAG (Retrieval-Augmented Generation): By indexing the parsed profiles into a vector database (like Chroma or Faiss), I could implement a private LLM chat interface to ask questions like: “Who in my network has experience with Kubernetes who been active in the last months?”

Conclusion

The My Professional Network project is more than just a dashboard; it’s a prototype for data agency. By moving the analytics layer from a centralized platform to a local, user-controlled environment, we reclaim the ability to navigate our professional lives with intent.

By combining modern frontend reactivity (Shiny), asynchronous infrastructure (Celery/Redis), and local ML inference (TEI), I’ve built a system that respects privacy while delivering the kind of “superpowers” usually reserved for big-tech internal tools.

In the end, your network is your most valuable professional asset — it’s time you had the tools to actually manage it.

See Also

Below are some of my other posts related to building applications, natural language processing, and data visualization:

	Title	Reading Time
	Agentic vs Deterministic Workflows: Designing a Reliable AI Application	7 min
	Building an E-Commerce Dashboard with Power BI and R	5 min
	Building the Analytical Dashboard with Power BI and R	6 min
	Building a Credit Risk Dashboard with Power BI and R	8 min
	Animation of Spatial Data	7 min
	Product Cards Creation Application	1 min
	Creating Anki Flashcards From List of Words	6 min
	Implementing a Local Retrieval-Augmented Generation System	12 min

No matching items

Key Features & Problems Solved

Implementation Highlights

Outcomes & Impact

Skills Demonstrated

Data Engineering • NLP & Embeddings • Unsupervised Learning • Network Analysis • Asynchronous Systems • Docker & DevOps • Full-Stack Data Science • Privacy-First System Design • System Architecture

Apply This to Your Business

If you have a business problem that requires data-driven solutions, feel free to reach out via contact page to discuss how I can help leverage data science, analytics, and automation to drive value for your organization.

See Also

Orchestration Logic

The Orchestrator’s intelligence comes from its ability to maintain a ‘Context Loop’ — remembering the results of Step 1 (Search) to inform Step 2 (Assessment).

In src/agents/orchestrator.py, the system instruction acts as a state machine. It doesn’t just say “Help the user”, it defines a strict protocol:

orchestrator_agent = LlmAgent(
    name="career_orchestrator",
    instruction="""
    Your responsibilities:
    ...
    3. Use the `recruitment_search_agent` to find relevant vacancies.
    4. Provide user with the list of relevant vacancies.
    5. Ask user to provide information about their skills and experiences.
    6. For each relevant vacancy chosen by the user:
        a. Use the `skills_assessment_agent` to analyze the fit...
        b. If the fit is good ... generate the tailored resume...
    """
)

By explicitly numbering the steps and restricting available tools at each phase, we reduce the degrees of freedom available to the LLM. The goal is not to make the model “smarter”, but to make incorrect behavior harder to express. This reliability is crucial for user trust.

Observability & Evaluation

A reliable agent system requires more than just code; it requires rigorous testing.

Conclusion

The Autonomous Career Agent app demonstrates that building useful GenAI applications goes beyond prompt engineering. It requires solid software engineering principles: modular architecture, secure integration patterns, and automated testing.

This project reaffirmed an important engineering reality: for the specific task of matching keywords and filling templates, a deterministic pipeline offers distinct advantages in speed and predictability. The agentic overhead here is non-trivial.

However, this architecture lays the groundwork for a much broader application; for example, Career Consulting. By grasping the observability and evaluation frameworks probed here, we can safely expand the agentic application to handle unstructured tasks like helping a user figure out what they want to do or pivoting strategies mid-search, where no deterministic application could succeed.

See Also

Here are some related articles you might find interesting:

	Title	Reading Time
	Building a Privacy-First LinkedIn Analytics Platform	8 min
	Product Cards Creation Application	1 min
	Creating Anki Flashcards From List of Words	6 min
	Implementing a Local Retrieval-Augmented Generation System	12 min
	Nerdy Valentine’s in Python, R, and Matlab	4 min
	Python Library for Russian Macroeconomics Data	6 min
	Kano Method for Prioritization of Features	10 min
	Merging Customers Records Using Graphs in Python	8 min

No matching items

Key Features

The dashboard includes several analytical features. The implementation of these involved data processing and modeling techniques, which I will outline below.

Revenue Forecast

For the executives it’s important to know ahead of time whether the revenue will meet the target for the set period. To address this, I implemented a time series forecasting model. The model takes into account seasonality and trends to provide accurate revenue predictions. Due to the limited data span of less than two years, I opted for a simpler forecasting approach using auto.arima function from R’s forecast package.

The model was used to generate revenue forecasts for various segments, including country, category, and channel. I integrated the forecasting results into the Power BI dashboard, enabling users to easily view and interact with forecasts alongside historical data, with smooth segment switching when applying cross-filtering.

ABC/XYZ Analysis

A classic way to segment products is by using ABC analysis. This method classifies products based on their revenue, placing items that contribute the most to revenue in category A, those with moderate contribution in category B, and the rest in category C.

To complement this, I applied XYZ analysis, which segments products based on their sales variability. Products with stable demand are classified as X, those with moderate variability as Y, and highly variable products as Z. Combining ABC and XYZ analyses provides a comprehensive view of product performance, enabling more informed inventory and marketing strategies.

RFM Segmentation

Another widely used customer segmentation technique is RFM analysis, which focuses on three key metrics: Recency (how recently a customer made a purchase), Frequency (how often they purchase), and Monetary value (how much they spend). By scoring customers on these dimensions, businesses can identify their most valuable customers and tailor marketing efforts accordingly.

To make the segmentation more intuitive, I defined nine segments (levels) based on RFM scores: “Champions”, “Loyal Customers”, “New Customers”, “Potential Loyalists”, “Promising”, “Needs Attention”, “Cannot Lose Them”, “At Risk”, “Hibernating”. This categorization helps in devising targeted strategies for customer retention and engagement.

Areas to Improve

Conclusion

I hope this overview of the dashboard construction process, from initial data cleaning to final model integration, gives the reader a clear picture of the analytical work involved.

References

• Project Repository

• Interactive Dashboard

See Also

Here are some of my other posts related to BI, statistics, machine learning, and data visualization:

	Title	Reading Time
	Implementing a Neural Network in Base R	9 min
	The 7-Step KPI Blueprint from Business Intelligence Analytics Perspective	14 min
	Building a Privacy-First LinkedIn Analytics Platform	8 min
	Propensity Score Matching for Causal Analysis	6 min
	Building the Analytical Dashboard with Power BI and R	6 min
	Building a Credit Risk Dashboard with Power BI and R	8 min
	Minimum Detectable Effect (MDE) Calculation	6 min
	A/B Testing: Concepts and Techniques	37 min
	Animation of Spatial Data	7 min
	BI System Blueprint	2 min
	CV Week 2024	10 min

No matching items

Key Features & Components

Dashboard Design

Outcomes & Impact

Skills Demonstrated

Power BI report development • R analytics & forecasting • Data modeling & ETL • Advanced segmentation • Interactive visualization • BI storytelling

Apply This to Your Business

If your organization could benefit from enhanced e-commerce analytics, you’re welcome to reach out via contact page to discuss how I can help leverage data science, analytics, and automation to drive value for your business.

See Also

Data Overview

First, let’s load the necessary libraries and the dataset.

library(MatchIt)
library(tidyverse)
library(lubridate)
library(ggplot2)
library(readr)
library(skimr)
library(here)

if (file.exists(here("bank-full.csv"))) {
  dataset <- read.csv(file.path(here(), "bank-full.csv"), sep = ";")
} else {
  url <- "https://archive.ics.uci.edu/static/public/222/bank+marketing.zip"
  temp <- tempfile()
  download.file(url, temp)
  unzip(temp, "bank.zip", exdir = here())
  unzip(here("bank.zip"), "bank-full.csv", exdir = here())
  dataset <- read.csv(file.path(here(), "bank-full.csv"), sep = ";")
  unlink(temp)
}
dataset |> glimpse()

Rows: 45,211
Columns: 17
$ age       <int> 58, 44, 33, 47, 33, 35, 28, 42, 58, 43, 41, 29, 53, 58, 57, …
$ job       <chr> "management", "technician", "entrepreneur", "blue-collar", "…
$ marital   <chr> "married", "single", "married", "married", "single", "marrie…
$ education <chr> "tertiary", "secondary", "secondary", "unknown", "unknown", …
$ default   <chr> "no", "no", "no", "no", "no", "no", "no", "yes", "no", "no",…
$ balance   <int> 2143, 29, 2, 1506, 1, 231, 447, 2, 121, 593, 270, 390, 6, 71…
$ housing   <chr> "yes", "yes", "yes", "yes", "no", "yes", "yes", "yes", "yes"…
$ loan      <chr> "no", "no", "yes", "no", "no", "no", "yes", "no", "no", "no"…
$ contact   <chr> "unknown", "unknown", "unknown", "unknown", "unknown", "unkn…
$ day       <int> 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, …
$ month     <chr> "may", "may", "may", "may", "may", "may", "may", "may", "may…
$ duration  <int> 261, 151, 76, 92, 198, 139, 217, 380, 50, 55, 222, 137, 517,…
$ campaign  <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
$ pdays     <int> -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, …
$ previous  <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
$ poutcome  <chr> "unknown", "unknown", "unknown", "unknown", "unknown", "unkn…
$ y         <chr> "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", …

dataset |> skim()

Correlation Matrix

library(PerformanceAnalytics)

set.seed(123)

dataset |>
  sample_n(1000) |>
  select_if(is.numeric) |>
  chart.Correlation(histogram = TRUE, pch = "+")

Defining Treatment and Outcome

Our treatment variable will be whether the client was previously contacted (pdays != -1), and the outcome variable will be whether the client subscribed to a term deposit (y == "yes").

dataset <- dataset |>
  mutate(
    treat = ifelse(pdays != -1, 1, 0),    # previously contacted
    outcome = ifelse(y == "yes", 1, 0)
  )

• treat = 1: client was previously contacted (pdays != -1);

• treat = 0: new client, not contacted before;

• outcome = 1: client subscribed to term deposit;

• outcome = 0: client did not subscribe.

Let’s check the basic rates:

dataset |>
  group_by(treat) |>
  summarise(
    n = n(),
    subscription_rate = mean(outcome)
  )

# A tibble: 2 × 3
  treat     n subscription_rate
  <dbl> <int>             <dbl>
1     0 36954            0.0916
2     1  8257            0.231 

We can clearly see that previously contacted clients (treat = 1) have a higher subscription rate — but this might be due to other factors like income or engagement.

t_test_all <- t.test(outcome ~ treat, data = dataset)
t_test_all

    Welch Two Sample t-test

data:  outcome by treat
t = -28.552, df = 10051, p-value < 2.2e-16
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
 -0.1486926 -0.1295874
sample estimates:
mean in group 0 mean in group 1 
     0.09157331      0.23071333 

Covariates and Model Formula

We include demographic and financial variables that can influence both being re-contacted and subscribing. We need to exclude campaign (number of contacts performed during this campaign and for this client), contact (contact communication type), previous (number of contacts performed before this campaign), and poutcome (outcome of the previous marketing campaign) to avoid leaking information about treatment assignment.

formula <- treat ~ age + job + marital + education + default +
  balance + housing + loan 

Estimating Propensity Scores and Matching

We now fit the PSM model using nearest-neighbor matching:

psm <- matchit(
  formula,
  data = dataset,
  method = "nearest",
  ratio = 1
)

Let’s inspect the summary. We won’t include the whole output of the summary function, just the number of matched pairs:

summary(psm)$nn

              Control Treated
All (ESS)       36954    8257
All             36954    8257
Matched (ESS)    8257    8257
Matched          8257    8257
Unmatched       28697       0
Discarded           0       0

Visualizing Balance

plot(psm, type = "jitter", interactive = FALSE)

plot(psm, type = "density", interactive = FALSE,
     which.xs = ~ age + job + marital + education + default + balance + 
    housing + loan)

library(cobalt)

love.plot(psm, threshold = 0.1)

Estimating the Treatment Effect

Extract the matched data and estimate the effect on subscription:

matched_data <- match.data(psm)  
t_test <- t.test(outcome ~ treat, data = matched_data)
t_test

    Welch Two Sample t-test

data:  outcome by treat
t = -18.501, df = 15563, p-value < 2.2e-16
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
 -0.1201447 -0.0971255
sample estimates:
mean in group 0 mean in group 1 
      0.1220782       0.2307133 

The difference in means represents the Average Treatment Effect on the Treated (ATT) — how much more likely previously contacted customers are to subscribe, compared to similar new customers.

diff_means <- unname(t_test$estimate[2] - t_test$estimate[1])
diff_means

[1] 0.1086351

Results and Interpretation

Before matching, customers who were previously contacted were percentage points more likely to subscribe to a term deposit. After controlling for demographics and financial variables via propensity score matching, previously contacted customers were percentage points more likely to subscribe than comparable new customers (), which is percentage points () less than the initial estimate.

• Before matching: previously contacted customers have a much higher subscription rate.

• After matching: the difference decreases, indicating that part of the initial gap was due to selection bias.

Discussion and Limitations

Unobserved confounding: we only matched on observed variables; factors like personality or spending habits might still bias the result.

Choice of treatment: we assumed “previous contact” is the cause; other definitions (e.g., contact channel, number of calls) could be explored.

Generalization: results apply to customers similar to those treated (ATT), not necessarily to all clients.

Conclusion

In this tutorial, we’ve demonstrated how to estimate a causal effect using Propensity Score Matching with real marketing data.

References

Ho D, Imai K, King G, Stuart E (2011). “MatchIt: Nonparametric Preprocessing for Parametric Causal Inference.” Journal of Statistical Software, 42(8), 1-28. doi:10.18637/jss.v042.i08 https://doi.org/10.18637/jss.v042.i08.

See Also

Here are some of my other posts related to A/B testing, statistics, and R:

	Title	Reading Time
	Implementing a Neural Network in Base R	9 min
	Building an E-Commerce Dashboard with Power BI and R	5 min
	Building the Analytical Dashboard with Power BI and R	6 min
	Building a Credit Risk Dashboard with Power BI and R	8 min
	Time-Series Clustering with R’s dtwclust Package	8 min
	Minimum Detectable Effect (MDE) Calculation	6 min
	A/B Testing: Concepts and Techniques	37 min
	Animation of Spatial Data	7 min
	Nerdy Valentine’s in Python, R, and Matlab	4 min
	European Tech Salaries	19 min
	Exploring Geospatial Insights with R and rnaturalearth	5 min

No matching items

Key Features & Components

Dashboard Design

Outcomes & Impact

Skills Demonstrated

Business Intelligence dashboarding • Public data analytics • Financial services domain knowledge • Data modeling & ETL • Power BI & DAX • Analytical storytelling • Regulatory and compliance analytics

Apply This to Your Business

If your organization could benefit from advanced analytics embedded into Power BI dashboards to support risk, compliance, or operational decision-making, let’s talk. I can help design and build tailored BI solutions that turn data into actionable insights.

See Also

Data Preparation and Exploratory Data Analysis (EDA)

The entire project started with ensuring the data was robust and ready for analysis. I performed Exploratory Data Analysis (EDA) to check the data structure, look at distributions, and establish correlations.

• Loading and Cleaning: I loaded two main sheets from the Excel file: the complaints data, which contained 62,516 rows and 19 columns, and the companies data, which contained 1,081 rows and 9 columns.

• Joining Tables: to create a single comprehensive analytical dataset, I joined the complaints data and the companies data. This resulted in a dataset comprising 62,516 rows and 27 variables.

Visualizing Distributions and Relationships

During EDA, I needed to check the univariate distributions of the numeric variables (Figure 1) and examine correlations between them to identify relationships and potential multicollinearity (Figure 2).

Note

See the document Exploratory Data Analysis for the detailed description of the EDA process.

Forecasting Complaint Trends (Time Series Analysis)

My next step was to create time series forecasts of complaint volumes segmented by state, which is essential for identifying upcoming trends.

monthly <- dataset |> 
    mutate(month = floor_date(as.Date(date_received), "month")) |>
    group_by(state, month) |>   
    summarise(complaints = n(), .groups = "drop") |>   
    arrange(state, month) 

Dashboard Visual: Complaints Over Time

The resulting forecast is presented visually in the dashboard’s “Overview” and “Geography” sections, showing the actual and forecasted complaint volumes over time.

Factor Analysis (Causal Modeling)

I performed causal analysis to estimate the effect of different company characteristics, such as enforcement history and size, on critical performance metrics like the complaint rate.

Advanced Techniques (Anomaly Detection and Clustering)

I also explored advanced techniques, though I ultimately chose to exclude their visuals from the final report to maintain focus and brevity.

Anomaly Detection

I used STL decomposition and z-score methods to identify anomalies in the monthly complaint volumes. This approach helped to detect only a couple of anomalies (one spike and one drop). I decided to skip including this visual in the final Power BI report because the low number of detected anomalies didn’t add substantial new information to the report at this stage.

Note

See Anomaly Detection for full details.

Clustering Analysis

I performed K-means clustering on various company features (like enforcement history, size, and complaint rate). I found that the optimal number of clusters was 3, based on the silhouette scores. These clusters effectively separated companies:

• Cluster 1 (Large/Medium, low complaints, near-zero enforcement),

• Cluster 2 (Small, high complaints, zero enforcement), and

• Cluster 3 (Medium/Small, high complaints, some enforcement history).

However, since visualizing these clusters simply reinforced the patterns already established by the causal analysis (the importance of size), I chose to skip the clustering visualizations in the final dashboard.

Note

See Clustering Analysis for full details.

Key Dashboard Visuals

The final dashboard provides users with an interactive, clear overview of the financial complaints landscape, structured across tabs for Overview, Geography, Products, Companies, and Factor Analysis.

Overview

The dashboard immediately shows key metrics.

Geography and Product Breakdowns

The geographical tab highlights totals by Census Region, for example, the South region accounted for 4K complaints, the West for 3K, and the Midwest for 1K.

In terms of products, corresponding report section provides breakdown of complaints by product, including YoY changes.

Companies

This report section provides descriptive statistics about companies.

Factor Analysis

This tab presents the results of the causal modeling, comparing the full and reduced models.

Conclusion

I hope this overview of the dashboard construction process, from initial data cleaning to final model integration, gives the reader a clear picture of the analytical work involved.

References

• Project Repository

• Interactive Dashboard

See Also

Here are some related articles you might find interesting:

	Title	Reading Time
	Implementing a Neural Network in Base R	9 min
	The 7-Step KPI Blueprint from Business Intelligence Analytics Perspective	14 min
	Building a Privacy-First LinkedIn Analytics Platform	8 min
	Building an E-Commerce Dashboard with Power BI and R	5 min
	Propensity Score Matching for Causal Analysis	6 min
	Building a Credit Risk Dashboard with Power BI and R	8 min
	Time-Series Clustering with R’s dtwclust Package	8 min
	Minimum Detectable Effect (MDE) Calculation	6 min
	A/B Testing: Concepts and Techniques	37 min
	Animation of Spatial Data	7 min
	Nerdy Valentine’s in Python, R, and Matlab	4 min
	BI System Blueprint	2 min
	CV Week 2024	10 min
	European Tech Salaries	19 min
	Exploring Geospatial Insights with R and rnaturalearth	5 min

No matching items

Phase 1: Data Cleaning and Engineering

The project began with a Credit Risk Dataset composed of 32,581 rows and 29 columns, split between 12 character (categorical) and 17 numeric variables. My first step, documented in detail in the Exploratory Data Analysis of Credit Risk Dataset, was ensuring the data was pristine for modeling.

1. Variable Selection and Redundancy

I immediately excluded variables that lacked predictive utility or caused issues in regression models. I removed the simple identifier client_ID and the geographical coordinates (city_latitude and city_longitude).

A key part of cleaning involved addressing redundancy. A correlation check revealed that loan_percent_income and loan_to_income_ratio were nearly perfectly correlated (0.9989417). The decision was made to drop loan_percent_income to eliminate multicollinearity and simplify the feature set.

2. Handling Missing Data and Outliers

I identified missing values primarily in person_emp_length (97% complete) and loan_int_rate (90% complete). Since no obvious pattern of missingness was found, I employed a simple and robust imputation method: replacing all missing numeric values with the median of their respective columns.

Outlier handling required careful judgment. I found extreme values in person_age and person_emp_length that seemed like data entry errors (e.g., ages greater than 100).

My logic was to replace these anomalies with the median to normalize the distributions without losing valid information. However, extreme values in other_debt and person_income were retained, as they were judged to be genuine observations.

Phase 2: Building and Optimizing Predictive Models

The core task was to predict the loan_status variable, where 1 signifies default and 0 signifies no default. I explored multiple models, prioritizing predictive power (AUC) and, for deployment, interpretability (Marginal Effects for the dashboard). The process is described in detail in Building Predictive Models for Credit Risk.

2. Advanced Modeling: Random Forest and XGBoost

To benchmark and potentially surpass the GLM performance, I implemented a Random Forest model. Since Random Forests can handle correlated features well, I initially trained this model on the full dataset (excluding only the two variables noted above).

The initial Random Forest model achieved an AUC of 0.8519. Using the variable importance metrics (Mean Decrease Accuracy and Mean Decrease Gini), I identified features contributing little to predictive power, such as gender, marital_status, and education_level. After dropping these weak predictors, the simplified Random Forest model improved performance, reaching an AUC of 0.861.

Finally, I tested XGBoost (Extreme Gradient Boosting), a powerful ensemble technique known for maximizing accuracy. After converting categorical variables to numeric matrices (a necessary step for XGBoost), this model delivered the best result: an AUC of 0.8963. XGBoost’s feature importance analysis indicated that loan_to_income_ratio and person_income were the most influential predictors, followed by person_home_ownership.

The superior performance of XGBoost led me to select it as the engine for the portfolio simulation component of the dashboard, while the GLM was retained for the interpretable marginal effects calculations.

Phase 3: Operationalizing Models via Power Query R Scripts

A critical technical challenge was integrating these analytical processes so the dashboard could update dynamically. I created R scripts to be embedded directly into Power Query, documented in Scripts for Power Query and Synthetic Data For Simulations.

Phase 4: Building Power BI Report

1. Power Query Integration

There are two ways to run R scripts in Power BI: through the R visual or via Power Query. The R visuals are mainly for creating plots and have multiple limitations. Power Query is more versatile as it runs to preprocess data before loading it into the report.

Conclusion

This project was a comprehensive exercise in data science, statistical modeling, and business intelligence. By employing R for data processing and modeling, and Power BI for reporting, I created a powerful tool for credit risk analysis.

References

• September 2025 DataDNA - Credit Risk Analytics Challenge - Onyx Data — challenge web-page.

• DataDNA Challenge Credit Risk Analytics September 2025 — GitHub repository.

• Interactive dashboard running in Power BI service.

See Also

Here are other related articles you might find interesting:

	Title	Reading Time
	Implementing a Neural Network in Base R	9 min
	The 7-Step KPI Blueprint from Business Intelligence Analytics Perspective	14 min
	Building a Privacy-First LinkedIn Analytics Platform	8 min
	Building an E-Commerce Dashboard with Power BI and R	5 min
	Propensity Score Matching for Causal Analysis	6 min
	Building the Analytical Dashboard with Power BI and R	6 min
	Time-Series Clustering with R’s dtwclust Package	8 min
	Minimum Detectable Effect (MDE) Calculation	6 min
	A/B Testing: Concepts and Techniques	37 min
	Animation of Spatial Data	7 min
	Nerdy Valentine’s in Python, R, and Matlab	4 min
	BI System Blueprint	2 min
	CV Week 2024	10 min
	European Tech Salaries	19 min
	Exploring Geospatial Insights with R and rnaturalearth	5 min

No matching items

1 What is Dynamic Time Warping (DTW)

Dynamic Time Warping (DTW) is a prominent distance measure used in shape-based time-series clustering. Unlike Euclidean distance, which compares points at the same time index, DTW allows for “warping” or stretching/compressing the time axis of one series to find an optimal alignment with another. This enables it to accurately measure similarity between time-series that may vary in speed, length, or have phase shifts, but exhibit similar overall shapes.

2 Data Preparation

We will use daily closing prices of various cryptocurrencies from the dYdX exchange. The data will be fetched using the httr2 package.

Let’s start by fetching the list of available perpetual markets from the dYdX API. We will then filter for popular cryptocurrencies and retrieve their daily closing prices.

library(httr2)
library(tibble)
library(dplyr)
library(tidyr)

base_url <- "https://indexer.dydx.trade/v4"
full_url <- paste0(base_url, "/perpetualMarkets")

req <- request(full_url) |>
  req_headers(Accept = "application/json")

resp <- req_perform(req)
body <- resp_body_json(resp, simplifyVector = TRUE)

markets <- bind_rows(body$markets) |> as_tibble()
markets <- markets |> mutate(volume24H = as.numeric(volume24H))
markets <- markets |> arrange(desc(volume24H)) 

markets |> select(ticker, oraclePrice, priceChange24H, volume24H) |> head(5)

# A tibble: 5 × 4
  ticker   oraclePrice  priceChange24H  volume24H
  <chr>    <chr>        <chr>               <dbl>
1 ETH-USD  4585.44      -10.614039     150676770.
2 BTC-USD  112855.34    2015.74841      43359731.
3 SOL-USD  212.08       7.72            18677423.
4 XRP-USD  2.9988498849 -0.0133507301    3201248.
5 LINK-USD 23.7155      -0.694490999     2551624.

Let’s take 30 the most traded markets.

tickers <- lapply(body$markets[1:31], function(market) {
  market$ticker
})
# remove MATIC-USD due to missing data
tickers <- tickers[!tickers %in% c("MATIC-USD")]
tickers <- tickers |>
  unlist() |>
  array()

Below are are helper functions to convert date-times to the required format and to fetch daily candle data from the dYdX API. For some reason at the time of writing this, the API ignores the fromIso parameter, so we will filter the data after fetching it.

to_iso8601_ns_utc <- function(datetime) {
  datetime_utc <- as.POSIXct(datetime, tz = "UTC")
  format(datetime_utc, format = "%Y-%m-%dT%H:%M:%OS9Z", usetz = FALSE)
}

get_dydx_daily_candles <- function(
    market,
    from_iso = NULL,
    to_iso = NULL,
    limit = NULL) {
  base_url <- "https://indexer.dydx.trade/v4"
  full_url <- paste0(base_url, "/candles/perpetualMarkets/", market)

  req <- request(full_url) |>
    req_headers(Accept = "application/json") |>
    req_url_query(resolution = "1DAY")

  from_iso <- if (!is.null(from_iso)) to_iso8601_ns_utc(from_iso) else NULL
  to_iso <- if (!is.null(to_iso)) to_iso8601_ns_utc(to_iso) else NULL

  if (!is.null(from_iso)) req <- req |> req_url_query(fromIso = from_iso)
  if (!is.null(to_iso)) req <- req |> req_url_query(toIso = to_iso)
  if (!is.null(limit)) req <- req |> req_url_query(limit = limit)

  resp <- req_perform(req)
  body <- resp_body_json(resp, simplifyVector = TRUE)

  df <- body$candles
  df <- df |> mutate(
    startedAt = as.POSIXct(startedAt,
      format = "%Y-%m-%dT%H:%M:%OSZ",
      tz = "UTC"
    ),
    open = as.numeric(open),
    high = as.numeric(high),
    low = as.numeric(low),
    close = as.numeric(close)
  )
  df |> as_tibble()
}

get_dydx_daily_candles("BTC-USD",
  from_iso = "2025-01-01T00:00:00Z",
  to_iso = "2025-08-26T00:00:00Z"
) |> head()

# A tibble: 6 × 13
  startedAt           ticker  resolution    low   high   open  close
  <dttm>              <chr>   <chr>       <dbl>  <dbl>  <dbl>  <dbl>
1 2025-08-28 00:00:00 BTC-USD 1DAY       110841 113400 111243 112888
2 2025-08-27 00:00:00 BTC-USD 1DAY       110395 112674 111809 111260
3 2025-08-26 00:00:00 BTC-USD 1DAY       108713 112397 110124 111797
4 2025-08-25 00:00:00 BTC-USD 1DAY       109296 113679 113537 110132
5 2025-08-24 00:00:00 BTC-USD 1DAY       110550 115662 115405 113539
6 2025-08-23 00:00:00 BTC-USD 1DAY       114569 117024 116948 115406
# ℹ 6 more variables: baseTokenVolume <chr>, usdVolume <chr>, trades <int>,
#   startingOpenInterest <chr>, orderbookMidPriceOpen <chr>,
#   orderbookMidPriceClose <chr>

Now, we will fetch daily closing prices for our selection of popular cryptocurrencies.

cryptos_list <- lapply(tickers, function(coin) {
  # cat("Fetching data for:", coin, "\n")
  df <- get_dydx_daily_candles(coin,
    from_iso = "2025-01-01T00:00:00Z",
    to_iso = "2025-08-26T00:00:00Z"
  )
  df <- df |> filter(startedAt >= as.POSIXct("2025-01-01", tz = "UTC"))
  df <- df |> select(ds = startedAt, y = close) |> arrange(ds)
  # z-score normalization
  df <- df |> mutate(y = (y - mean(y, na.rm = TRUE)) / sd(y, na.rm = TRUE))
  df$coin <- toupper(coin) %>% gsub("-USD", "", .)
  return(df)
})

cryptos_list <- bind_rows(cryptos_list)

Next, we will reshape the data into a wide format suitable for clustering, where each row represents a cryptocurrency and each column represents a daily closing price.

cryptos_list_wide <- cryptos_list  |> 
  pivot_wider(names_from = coin, values_from = y) |> 
  arrange(ds) |> 
  select(-ds) |>  
  as.list()

3 Performing Hierarchical Clustering

We will perform hierarchical clustering using the DTW distance and the “ward.D2” agglomeration method. Hierarchical clustering builds a hierarchy of groups without requiring a pre-specified number of clusters initially, and the process is deterministic.

• k = 4 specifies the desired number of clusters.

• type = "hierarchical" sets the clustering algorithm type.

• distance = "dtw" uses Dynamic Time Warping distance.

• seed = 42 for reproducibility of random initializations (if applicable).

• control = hierarchical_control(method = "ward.D2") specifies the linkage method.

• args = tsclust_args(dist = list(window.size = 7)) sets DTW window constraint.

library(dtwclust)

# Perform hierarchical clustering
hc_4_ward <- tsclust(cryptos_list_wide,
  k = 4,
  type = "hierarchical",
  distance = "dtw",
  seed = 42,
  control = hierarchical_control(method = "ward.D2"),
  args = tsclust_args(dist = list(window.size = 7))
)

# View the clustering summary
hc_4_ward

hierarchical clustering with 4 clusters
Using dtw distance
Using PAM (Hierarchical) centroids
Using method ward.D2 

Time required for analysis:
   user  system elapsed 
  2.958   0.047   3.005 

Cluster sizes with average intra-cluster distance:

  size  av_dist
1    4 87.52806
2    2 80.73321
3   12 66.03449
4   12 51.73463

The output provides details about the clustering, including the distance measure, centroid method, linkage method, and cluster sizes with their average intra-cluster distances.

4 Accessing Clustering Results

The tsclust() function returns an S4 object of class TSClusters. You can access its slots, such as the cluster assignments, using the @ operator.

# View cluster assignments for each time series
hc_4_ward@cluster

 BTC  ETH LINK  CRV  SOL  ADA AVAX  FIL  LTC DOGE ATOM  DOT  UNI  BCH  TRX NEAR 
   1    2    3    2    3    3    3    4    3    3    4    4    3    1    1    4 
 MKR  XLM  ETC COMP  WLD  APE  APT  ARB BLUR  LDO   OP PEPE  SEI SHIB 
   1    3    3    4    4    4    4    3    4    3    4    4    3    4 

5 Visualizing Clustering Results

The plot() method for TSClusters objects offers various visualization types.

5.1 Dendrogram

A dendrogram visually represents the hierarchy of clusters.

par(mar = c(0, 4, 2, 2)) # Adjust margins for better plot
plot(hc_4_ward,
  xlab = "", ylab = "", sub = "",
  main = "Hierarchical Clustering Dendrogram (DTW, Ward.D2)"
)

5.2 Series and Centroids

Visualize the time series grouped by cluster, along with their representative prototypes (centroids). By default, prototypes for hierarchical clustering with PAM centroids are actual series from the data.

plot(hc_4_ward, type = "sc") # sc = series + centroids

You can also plot a specific centroid, and even customize its appearance.

plot(hc_4_ward,
  type = "centroids", clus = 1,
  linetype = 1, size = 1, alpha = 0.8
)

6 Comparing Multiple Clustering Solutions and Evaluation

In practice, choosing the optimal number of clusters (k) and other parameters is crucial. dtwclust allows you to test multiple configurations simultaneously and evaluate them using Cluster Validity Indices (CVIs).

To accelerate the process, especially when testing many combinations, parallelization is highly recommended. dtwclust integrates with the foreach and doParallel packages for this purpose.

library(bigmemory)
library(doParallel)

# Define a range of k values and agglomeration methods to test
k_values <- 3:6
linkage_methods <- c("ward.D2", "average", "single", "complete")

# Initialize a parallel backend
# Use detectCores() - 1 to leave one core free
num_cores <- detectCores() - 1
if (num_cores < 1) num_cores <- 1 # Ensure at least one core is used

cl <- makeCluster(num_cores)
registerDoParallel(cl)

# Perform multiple hierarchical clusterings in parallel
hc_par <- tsclust(cryptos_list_wide,
  k = k_values,
  type = "hierarchical",
  distance = "dtw",
  seed = 42,
  control = hierarchical_control(method = linkage_methods),
  args = tsclust_args(dist = list(window.size = 7)),
  trace = TRUE
)

Calculating distance matrix...
Performing hierarchical clustering...
Extracting centroids...

    Elapsed time is 13.749 seconds.

# Stop the parallel cluster and revert to sequential computation
stopCluster(cl)
registerDoSEQ()

cvi_results <- lapply(hc_par, cvi, type = c("Sil", "D", "CH")) %>%
  do.call(rbind, .)

# Find the configuration that maximizes each CVI
optimal_indices <- apply(cvi_results, MARGIN = 2, FUN = which.max)

print(cvi_results)

            Sil         D        CH
 [1,] 0.3463459 0.1355412 21.828708
 [2,] 0.5144728 0.3810831  9.673635
 [3,] 0.1888799 0.2514864  2.709671
 [4,] 0.3369159 0.2050711 18.657689
 [5,] 0.2989078 0.1745498 13.778813
 [6,] 0.3477774 0.3091404  8.406351
 [7,] 0.3689743 0.3451739  6.165031
 [8,] 0.3290618 0.2245861 14.349598
 [9,] 0.2838572 0.2257579 12.063620
[10,] 0.3054947 0.3091404  6.059278
[11,] 0.3441495 0.4739975  5.473468
[12,] 0.2518630 0.2031676 11.466633
[13,] 0.2731625 0.2257579 10.492873
[14,] 0.3439047 0.3098938 10.261007
[15,] 0.2602655 0.4210699  7.605540
[16,] 0.2297632 0.2559787  9.802860

print(optimal_indices)

Sil   D  CH 
  2  11   1 

best_clustering_sil <- hc_par[[optimal_indices["Sil"]]]
best_clustering_sil

hierarchical clustering with 3 clusters
Using dtw distance
Using PAM (Hierarchical) centroids
Using method average 

Time required for analysis:
   user  system elapsed 
  1.364   0.122  13.749 

Cluster sizes with average intra-cluster distance:

  size  av_dist
1    4 80.06529
2   25 84.14893
3    1  0.00000

best_clustering_dunn <- hc_par[[optimal_indices["D"]]]
best_clustering_dunn

hierarchical clustering with 5 clusters
Using dtw distance
Using PAM (Hierarchical) centroids
Using method single 

Time required for analysis:
   user  system elapsed 
  1.364   0.122  13.749 

Cluster sizes with average intra-cluster distance:

  size  av_dist
1    3 62.58062
2    1  0.00000
3   24 78.78400
4    1  0.00000
5    1  0.00000

best_clustering_ch <- hc_par[[optimal_indices["CH"]]]
best_clustering_ch

hierarchical clustering with 3 clusters
Using dtw distance
Using PAM (Hierarchical) centroids
Using method ward.D2 

Time required for analysis:
   user  system elapsed 
  1.364   0.122  13.749 

Cluster sizes with average intra-cluster distance:

  size   av_dist
1    6 106.59632
2   12  66.03449
3   12  51.73463

7 Performing Clustering with Optimal Configuration

Let’s take a look at the clustering configuration that achieved the maximum results according to the Calinski-Harabasz index.

hc_3_ward <- tsclust(cryptos_list_wide,
  k = 3,
  type = "hierarchical",
  distance = "dtw",
  seed = 42,
  control = hierarchical_control(method = "ward.D2"),
  args = tsclust_args(dist = list(window.size = 7))
)

# View the clustering summary
hc_3_ward

hierarchical clustering with 3 clusters
Using dtw distance
Using PAM (Hierarchical) centroids
Using method ward.D2 

Time required for analysis:
   user  system elapsed 
  2.910   0.106   3.017 

Cluster sizes with average intra-cluster distance:

  size   av_dist
1    6 106.59632
2   12  66.03449
3   12  51.73463

# View cluster assignments for each time series
hc_3_ward@cluster

 BTC  ETH LINK  CRV  SOL  ADA AVAX  FIL  LTC DOGE ATOM  DOT  UNI  BCH  TRX NEAR 
   1    1    2    1    2    2    2    3    2    2    3    3    2    1    1    3 
 MKR  XLM  ETC COMP  WLD  APE  APT  ARB BLUR  LDO   OP PEPE  SEI SHIB 
   1    2    2    3    3    3    3    2    3    2    3    3    2    3 

Plot the dendrogram for this clustering.

# Plot the dendrogram
par(mar = c(0, 4, 2, 2))
plot(hc_3_ward,
  xlab = "", ylab = "", sub = "",
  main = "Hierarchical Clustering Dendrogram (DTW, Ward.D2)"
)

The line plots of the time series grouped by cluster show very distinct patterns.

# Plot time series and their centroids
plot(hc_3_ward, type = "sc") 

8 Conclusion

dtwclust provides a modular and efficient framework for time-series clustering in R, implementing various algorithms (especially DTW-related ones) and allowing for customization and comparison. It serves as a strong starting point for time-series clustering tasks.

How MDE is calculated in experimental design

The MDE calculation depends on several key parameters:

• Sample size ( for the test group, for the control group).
• Significance level (alpha, ): This is the probability of a Type I error (falsely rejecting the null hypothesis), typically set at 0.05.
• Statistical power (1-beta, ): This is the probability of correctly detecting an effect when one truly exists, commonly set at 0.8 (or 80%).
• Variance () of the metric being measured in the population, often estimated from historical data.
• Ratio of control to test group sizes ().

Calculating MDE for a given sample size

If you have a fixed number of users (n) or a defined sample size (m or n), you can determine the smallest effect (e) that your test can reliably detect:

and are the Z-scores corresponding to the desired significance level and power, respectively.

is the ratio of the control group size to the test group size (e.g., for a 1:1 split).

is the estimated variance of the metric.

is the minimal effect.

# Function to calculate MDE
calculate_mde <- function(n, alpha = 0.05, beta = 0.2, sigma = 1, k = 1) {
  z_alpha <- qnorm(1 - alpha / 2)
  z_beta <- qnorm(1 - beta)
  m <- n * k / (k + 1)
  e <- sqrt((z_alpha + z_beta)^2 * (1 + k) * sigma^2 / m)
  return(e)
}

calculate_mde(n = 100000, alpha = 0.05, beta = 0.2, sigma = 500, k = 2)

[1] 9.396802

Finding the required sample size for a given MDE

The formula used to determine the necessary sample size (e.g., for the control group, ) to detect a specific MDE () is:

# Function to calculate required sample size
calculate_sample_size <- function(e, alpha = 0.05, beta = 0.2, sigma = 1, k = 1) {
  z_alpha <- qnorm(1 - alpha / 2)
  z_beta <- qnorm(1 - beta)
  m <- (z_alpha + z_beta)^2 * (1 + k) * sigma^2 / e^2
  return(ceiling(m))
}
m <- calculate_sample_size(e = 10, alpha = 0.05, beta = 0.2, sigma = 500, k = 2)
n <- ceiling(m / 2)
total_users <- m + n

c(m, n, total_users)

[1] 58867 29434 88301

Conclusion

A/B testing is a powerful tool for product development, and understanding the concept of Minimal Detectable Effect (MDE) is crucial for designing effective experiments. By calculating the required sample size or MDE, you can ensure that your tests are statistically sound and capable of providing meaningful insights into user behavior and product performance.

1. What is A/B testing and what are its key components?

A/B testing is a method used to determine the impact of implemented changes on a product by isolating external factors. It involves dividing users into two groups: a control group (A) that experiences no changes, and a test group (B) that is exposed to a new feature.

The key components of an A/B test include:

• Infrastructure: A robust system is required to conduct and manage experiments.

• Customer Base: A large user base is necessary to ensure statistically significant results.

• Time: Sufficient time is needed for the experiment to run and for the data to be analyzed.

• Metrics: Carefully selected metrics are used to measure the effect of the changes. These can be “value metrics” (e.g., total cost of successful trips, number of unique completed orders) or “ratio metrics” (e.g., acceptance rate, completed rate, tips as a share of GMV).

• User Aggregation: Data is typically aggregated per user rather than per event to ensure independent observations, which is crucial for valid statistical analysis. Comparing raw event-level data can introduce dependencies that invalidate standard statistical tests.

2. What are the common types of metrics used in A/B testing and how are they handled?

Metrics in A/B testing are broadly categorized into:

• Value Metrics: These represent absolute values or sums, such as Gross Merchandise Value (GMV), total number of impressions, or total dwell time. For these metrics, the average (mean) is commonly compared between test and control groups.

• Ratio Metrics: These represent a proportion or ratio, such as Acceptance Rate (accepted offers to seen offers) or CTR (number of clicks to the number of views). These are more complex because they involve both a numerator and a denominator, and the simple t-test for means may not be appropriate due to the inherent correlation between the numerator and denominator within each user.

For ratio metrics, several advanced methods are used:

• Delta Method: This statistical technique estimates the variance of a ratio by using the variances and covariance of its numerator and denominator. It approximates the distribution of the ratio using a Taylor series expansion.

• Linearization: This method transforms the ratio into a linear approximation, allowing the use of standard t-tests on the transformed data. There are different types of linearization, typically involving a reference value (e.g., the control group’s ratio) to define the linear terms.

  NoteMore information

  Consistent Transformation of Ratio Metrics for Efficient Online Controlled Experiments (DOI:10.1145/3159652.3159699).

• Bucketization (Bucketing): Instead of analyzing individual user data, users (or their aggregated events) are grouped into “buckets”. The ratio is then calculated for each bucket, and a t-test is performed on the bucket-level ratios. This can help normalize the distribution and reduce the impact of outliers but may lead to loss of information or reduced power with too few buckets.

• Bootstrap: A non-parametric resampling technique that involves repeatedly drawing samples with replacement from the observed data to create an empirical distribution of the statistic of interest (e.g., the difference in ratios). This distribution is then used to construct confidence intervals and calculate p-values, making it robust to distributional assumptions. Poisson bootstrap is a variant suitable for large datasets, allowing parallelization.

3. What are Type I and Type II errors in A/B testing, and how do they relate to MDE?

In hypothesis testing:

• Null Hypothesis (H0): States there is no effect or difference between groups (e.g., the new feature has no impact).

• Alternative Hypothesis (H1): States there is an effect or difference.

The two types of errors are:

• Type I Error : Rejecting the null hypothesis when it is actually true. This is also known as the “level of significance” and represents the probability of falsely concluding that an effect exists when it doesn’t.

• Type II Error : Failing to reject the null hypothesis when the alternative hypothesis is true. This means failing to detect an effect that actually exists.

There’s an inverse relationship between Type I and Type II errors: decreasing alpha (making it harder to find an effect) will increase beta (making it harder to detect a real effect), and vice-versa.

Minimal Detectable Effect (MDE) is the smallest true difference between the control and test groups that an A/B test can reliably detect as statistically significant, given predefined values for:

• Sample Size : The number of users in each group.

• Significance Level : The probability of a Type I error (e.g., 0.05).

• Statistical Power : The probability of correctly detecting a true effect (e.g., 0.8 or 80%).

MDE is crucial for experiment design, helping to estimate the required sample size and understand the sensitivity of the test to detect meaningful changes.

4. How can the validity of a statistical criterion be checked using A/A tests?

A/A testing is a method where two identical groups are compared to each other, with no actual changes introduced. Since no effect is expected, an A/A test helps validate the statistical criterion used in A/B tests.

The primary principle for validation is that if the null hypothesis is true (i.e., there is no actual difference between the groups), the p-values obtained from the statistical tests should be uniformly distributed between 0 and 1.

Validation steps involve:

1. Synthetic Data Generation: Create simulated datasets for test and control groups where no effect is present.

2. Repeated Testing: Run the statistical criterion (e.g., t-test) many times (e.g., 10,000 times) on these synthetic A/A datasets.

3. P-value Distribution Analysis: Histogram of P-values: If the p-values are uniformly distributed, the histogram should appear flat. Any peaks or skews indicate issues with the criterion.

4. QQ-plot (Quantile-Quantile Plot): This plot compares the observed p-values’ quantiles against the theoretical quantiles of a uniform distribution. Points should fall approximately along a 45-degree line. Deviations suggest the p-values are not uniformly distributed.

5. Empirical Cumulative Distribution Function (ECDF): Plotting the ECDF of the p-values against the theoretical CDF of a uniform distribution (which is a straight line from 0,0 to 1,1). Similar to QQ-plots, a close fit indicates uniformity.

6. Kolmogorov-Smirnov (KS) Test: A non-parametric statistical test that formally assesses whether the observed p-values significantly differ from a uniform distribution. A high p-value from the KS test (e.g., > 0.05) would suggest uniformity.

7. Confidence Interval for Type I Error: Calculate a confidence interval for the proportion of times the null hypothesis was incorrectly rejected (Type I error rate). This observed error rate should ideally be close to the chosen alpha level (e.g., 0.05) and fall within its confidence interval.

If any of these checks fail, it indicates that the chosen statistical criterion is not valid for the given data and experiment setup, even before considering any actual effects.

5. What are the challenges with “peeking” at A/B test results and how can they be addressed?

“Peeking” or “p-hacking” refers to the practice of repeatedly checking the results of an A/B test as data accumulates and stopping the experiment as soon as a statistically significant result is observed.

6. What is the problem of multiple testing and how can it be mitigated?

The “multiple testing problem” arises when multiple statistical hypotheses are tested simultaneously. If you perform independent A/B tests, each with a Type I error rate (alpha) of, say, 0.05, the probability of making at least one false positive (Family-Wise Error Rate, FWER) increases significantly with .

7. What is CUPED and how does it improve A/B test sensitivity?

CUPED (Controlled-experiment Using Pre-Experiment Data) is a technique designed to improve the sensitivity (power) of A/B tests by reducing the variance of the metrics being analyzed. It achieves this by leveraging pre-experiment data (covariates) for each user.

8. What are the key differences between frequentist and Bayesian A/B testing approaches?

See Also

Here are some related articles you might find interesting:

	Title	Reading Time
	Using Transit Time to Rethink Hotel Search	14 min
	Building an E-Commerce Dashboard with Power BI and R	5 min
	Propensity Score Matching for Causal Analysis	6 min
	Building the Analytical Dashboard with Power BI and R	6 min
	Building a Credit Risk Dashboard with Power BI and R	8 min
	Minimum Detectable Effect (MDE) Calculation	6 min
	Kano Method for Prioritization of Features	10 min

No matching items

Load the stations data

Here we will download the stations data from the DWD website. The data contains information about weather stations, including their IDs, names, locations, and the time period they were active.

url <- "https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/subdaily/cloudiness/historical/"
stations_file <- "N_Terminwerte_Beschreibung_Stationen.txt"
if (!file.exists("data")) {
  dir.create("data")
}
if (!file.exists(paste0("data/", stations_file))) {
  download.file(paste0(url, stations_file), here("posts", "20250704-animation", "data", stations_file), mode = "wb")
}

Let’s read the stations data.

col_names <- c(
  "STATIONS_ID", "von_datum", "bis_datum", "Stationshoehe", "lat",
  "lon", "Stationsname", "Bundesland", "Abgabe"
)

stations <- read.fwf(
  here(
    "posts", "20250704-animation", "data",
    "N_Terminwerte_Beschreibung_Stationen.txt"
  ),
  widths = c(6, 9, 9, 15, 12, 10, 41, 41, 5), skip = 2,
  fileEncoding = "Windows-1252", col.names = col_names
) |> as.data.table()


stations[, von_datum := as.Date(str_trim(von_datum), format = "%Y%m%d")]
stations[, bis_datum := as.Date(str_trim(bis_datum), format = "%Y%m%d")]
stations[, lon := as.numeric(lon)]
stations[, lat := as.numeric(lat)]

Read the links to the data files

We will read the HTML content of the DWD website to extract the links to the cloudiness data files. The links will be filtered to include only those that contain the term “terminwerte”.

page_content <- read_html(url)

links <- page_content |>
  html_nodes("a") |>
  html_attr("href")

links <- links[!is.na(links) & str_detect(links, "terminwerte")]
links <- links |> data.table()
links |> head()

                                            links
                                           <char>
1: terminwerte_N_00001_19370101_19860630_hist.zip
2: terminwerte_N_00003_18910101_20110331_hist.zip
3: terminwerte_N_00044_19710301_20111231_hist.zip
4: terminwerte_N_00052_19730101_20011231_hist.zip
5: terminwerte_N_00061_19750701_19780831_hist.zip
6: terminwerte_N_00070_19730601_19860930_hist.zip

Extract the station IDs from the links. The station IDs are 5-digit numbers that are part of the file names.

links[, STATIONS_ID := str_extract(links, "[0-9]{5}")]
links[, STATIONS_ID := as.integer(STATIONS_ID)]

Filter the stations data

We will download only those stations data that were active during the specified time period.

stations <- stations[von_datum <= "2023-12-01" & bis_datum >= "2025-01-01"]
links <- links[stations, on = "STATIONS_ID"]

Download and process the data files

In this section, we will download the data files from the DWD website and process them to extract the cloud coverage and density observations. The data will be stored in a DuckDB database which is useful if we need to reuse the data later without downloading and parsing it again.

# check if the files exist
files <- list.files("data", full.names = TRUE)
files <- files[str_detect(files, "produkt_n_termin")]

if (length(files) == 0) {
  for (link in links$links) {
    download.file(paste0(url, link), here(
      "posts", "20250704-animation", "data", link
    ), mode = "wb")
    unzip(paste0("data/", link), exdir = "data")
    unlink("data/Metadaten*")
    unlink("data/*.html")
  }
}
unlink("data/*.zip")

Here is where parsing is done.

con <- dbConnect(duckdb(),
  dbdir = here("posts", "20250704-animation", "db", "weather.duckdb")
)
tables <- dbGetQuery(con, "SHOW ALL TABLES;")["name"]
start_date <- "2023-12-01"

if (!("cloudiness" %in% tables$name)) {
  files <- list.files("data", full.names = TRUE)
  files <- files[str_detect(files, "produkt_n_termin")]

  observations <- data.table()

  for (file in files) {
    temp_data <- read.csv(file, sep = ";") |> as.data.table()
    temp_data[, MESS_DATUM := as.Date(str_trim(MESS_DATUM), format = "%Y%m%d")]
    temp_data <- temp_data[MESS_DATUM >= start_date]
    observations <- rbind(
      observations,
      temp_data
    )
  }

  observations <- observations[N_TER != -999]
  observations[, CD_TER := ifelse(CD_TER == -999, NA, CD_TER)]

  if (!file.exists("db")) {
    dir.create("db")
  }

  con <- dbConnect(duckdb(),
    dbdir = here(
      "posts", "20250704-animation", "db", "weather.duckdb"
    )
  )
  dbWriteTable(con, "cloudiness", observations, overwrite = TRUE)
}

observations <- dbGetQuery(
  con,
  sprintf("SELECT * FROM cloudiness WHERE MESS_DATUM >= '%s'", start_date)
) |>
  as.data.table()
dbDisconnect(con)

observations |> glimpse()

Rows: 231,108
Columns: 6
$ STATIONS_ID <int> 4024, 4024, 4024, 4024, 4024, 4024, 4024, 4024, 4024, 4024…
$ MESS_DATUM  <date> 2023-12-01, 2023-12-01, 2023-12-01, 2023-12-02, 2023-12-0…
$ QN_4        <int> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9…
$ N_TER       <int> 4, 5, 8, 8, 8, 8, 8, 7, 8, 8, 8, 8, 7, 8, 8, 8, 7, 8, 8, 8…
$ CD_TER      <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ eor         <chr> "eor", "eor", "eor", "eor", "eor", "eor", "eor", "eor", "e…

As the data contains multiple observations per day for each station, we will aggregate the data to get the average cloud coverage and density per day per station.

observations <- observations[, .(
  .N,
  cloud_coverage = mean(N_TER, na.rm = TRUE),
  cloud_density = mean(CD_TER, na.rm = TRUE)
), by = c("STATIONS_ID", "MESS_DATUM")]

Let’s plot the cloud coverage for a specific station as a time series to visualize the data.

observations[MESS_DATUM > "2024-01-01" & STATIONS_ID == 433] |>
  ggplot(aes(x = MESS_DATUM, y = cloud_coverage)) +
  geom_line(color = "gray") +
  geom_line(aes(y = rollapply(cloud_coverage,
    width = 7, FUN = mean, align = "center", partial = TRUE
  )), color = "blue") +
  labs(title = NULL, x = NULL, y = "Cloud Coverage") +
  theme_gray()

Add H3 addresses

To visualize the data on a map, we will convert the latitude and longitude coordinates of the stations into H3 addresses. H3 is a geospatial indexing system that allows us to represent geographic locations as hexagonal cells.

points <- stations[, .(lon, lat)] |> unique()

points[, h3_address := point_to_cell(points, res = 4)]

Assuming columns 1 and 2 contain x, y coordinates in EPSG:4326

stations <- stations[, c(
  "STATIONS_ID", "von_datum", "bis_datum", "Stationshoehe",
  "lat", "lon", "Stationsname", "Bundesland", "Abgabe"
)]

stations[points, on = .(lon, lat), h3_address := h3_address]

stations <- stations[, geometry := cell_to_polygon(h3_address, simple = F)[2]]

Load the boundaries of Germany to use as a background for the map.

boundaries <- geoboundaries("Germany", release_type = "gbOpen", adm_lvl = "adm1")

Join the stations data with the observations data to have the geometry of the stations in the observations data. Calculate mean for each H3 address and drop duplicates.

observations <- observations[stations, on = "STATIONS_ID"]
observations[, coud_coverage := mean(cloud_coverage, na.rm = TRUE), by = h3_address]
observations <- observations |> unique(by = c("h3_address", "MESS_DATUM"))

Create animations of cloud coverage in Germany

In the following section, we will create animations of cloud coverage in Germany using the observations data in 2024. This code creates a series of maps showing the average cloud coverage for each day in 2024, with a rolling average of 7 days to smooth the data.

min_date <- observations[, MESS_DATUM] |> min(na.rm = TRUE)
max_date <- observations[, MESS_DATUM] |> max(na.rm = TRUE)

min_date <- max(c(min_date, as.Date("2024-01-01")))
max_date <- min(c(max_date, as.Date("2024-12-31")))

max_coverage <- observations[(MESS_DATUM >= min_date) &
  (MESS_DATUM <= max_date), cloud_coverage] |> max(na.rm = TRUE)
min_coverage <- observations[(MESS_DATUM >= min_date) &
  (MESS_DATUM <= max_date), cloud_coverage] |> min(na.rm = TRUE)

table_dates <- seq(min_date - 7, max_date + 7, by = 1) |>
  as.data.table() |>
  rename(MESS_DATUM = V1)
observations <- table_dates[observations, on = "MESS_DATUM"]

observations[, cloud_coverage_r7 := rollapply(cloud_coverage,
  width = 7, FUN = mean, align = "center", partial = TRUE
),
by = STATIONS_ID
]

dates <- seq(min_date, max_date, by = 1)

if (!file.exists("figures")) {
  dir.create("figures")
}

for (d in as.character(dates)) {
  p <- ggplot(aes(fill = cloud_coverage_r7),
    data = observations[MESS_DATUM == d] |> as.data.frame()
  ) +
    geom_sf(data = boundaries, fill = "gray78", color = "gray54") +
    geom_sf(aes(geometry = geometry), color = "gray78") +
    scale_fill_whitebox_c(
      palette = "deep",
      direction = 1,
      limits = c(min_coverage, max_coverage)
    ) +
    coord_sf(default_crs = sf::st_crs(4326)) +
    theme_void() +
    theme(
      legend.position = "bottom",
      legend.key.height = unit(4, "pt"),
      legend.key.width = unit(20, "pt"),
      legend.title.position = "top",
      panel.grid.minor = element_blank(),
      panel.grid.major = element_blank(),
      plot.background = element_rect(fill = NA, color = NA),
      title = element_text(size = 8, color = "gray35")
    ) +
    labs(title = d, fill = "Cloud Coverage")

  ggsave(
    here(
      "posts", "20250704-animation", "figures",
      paste0("cloudiness-", d, ".png")
    ),
    plot = p,
    units = "px",
    width = 1200,
    height = 1200,
    dpi = 300
  )
}

Create a GIF animation

Finally, we will create a GIF animation from the generated PNG files. The GIF will show the cloud coverage in Germany over the course of 2024, with each frame representing a day.

png_files <- list.files(
  here("posts", "20250704-animation", "figures"),
  full.names = TRUE, pattern = "cloudiness.+\\.png"
) |>
  sort() |>
  as.character()

gif_file <- here(
  "posts", "20250704-animation", "animation-cloudiness.gif"
)

gifski(png_files, gif_file, width = 1200, height = 1200, delay = 0.1, loop = TRUE)

[1] "/mnt/Projects/Blog/posts/20250704-animation/animation-cloudiness.gif"

unlink(here("posts", "20250704-animation", "figures", "cloudiness*"))

The resulting GIF animation shows the cloud coverage in Germany for each day in 2024, with a rolling average of 7 days to smooth the data. The animation provides a clear visual representation of how cloud coverage changed over time across different regions in Germany.

Source code available in the repository: Animations-in-R

See Also

Here are some related posts that you might find interesting:

	Title	Reading Time
	Implementing a Neural Network in Base R	9 min
	Using Transit Time to Rethink Hotel Search	14 min

No matching items

Application Features

See the detailed description of the application features in the Projects section: Product Cards Creation Application.

Discussion on LinkedIn

Here is a LinkedIn post where I discuss how I created this AI application:

See Also

Below you can find some related articles on building AI applications and working with LLMs:

	Title	Reading Time
	Agentic vs Deterministic Workflows: Designing a Reliable AI Application	7 min
	Implementing a Local Retrieval-Augmented Generation System	12 min

No matching items