Let’s review a simple example of how RAG works with LLM. Apple just reported earnings, and you want to know information like “What was iPhone revenue this quarter?” The transcript is 46,000 characters. You could read the whole thing. Or you could ask an LLM.

But the LLM has never seen this transcript. It was trained months ago. It has no idea what Apple reported last week. So how do you give it the information it needs without dumping 46,000 characters into the prompt?

More broadly, how can we ask questions against some text?

That’s what RAG does. Retrieval-Augmented Generation. It sounds fancy, but the core idea is simple: find the relevant pieces of a document, hand them to the LLM, and let it answer.

The whole thing is about 100 lines of Python. No frameworks. No vector databases. No LangChain. Just the raw mechanics, step by step.

The Source: An Earnings Call Transcript

We’re working with Apple’s Q4 FY2025 earnings call transcript.

“Good afternoon, and welcome to the Apple Q4 Fiscal Year 2025 Earnings Conference Call. Today, Apple is proud to report $102.5 billion in revenue, up 8% from a year ago and a new September quarter record...”

It goes on for about 8,000 words. Tim Cook gives prepared remarks, the CFO walks through financials, then analysts ask questions. Typical earnings call structure. You can read the full thing on Yahoo Finance or Motley Fool.

If we paste this entire transcript into a prompt, we’re wasting tokens on irrelevant sections (the legal disclaimer, the operator instructions, the analyst chitchat). We want to find just the parts that answer our question.

Chunking: Splitting the Document

First, we break the transcript into smaller pieces. Each piece is roughly 500 characters, about the length of a short paragraph. When a paragraph is longer than that, we split it with some overlap so we don’t lose context at the boundaries.

For this transcript, that gives us about 100 chunks. Each one is a self-contained snippet: one might be Tim Cook talking about iPhone, another might be the CFO discussing services revenue, another might be an analyst asking about China.

The tradeoff is straightforward. Chunks too small, and you lose context. “Revenue was $49 billion” means nothing without knowing it’s about iPhone. Chunks too large, and you dilute the signal with irrelevant text. 500 characters is a reasonable middle ground for earnings calls.

Embeddings: Turning Text into Numbers

Now we need a way to compare our question to each chunk. Computers can’t compare sentences directly, so we convert them into numbers.

An embedding model takes a piece of text and produces a list of numbers. “iPhone set a revenue record at $49 billion” becomes something like [0.038, -0.021, 0.087, 0.015, -0.044, ...]. The model was trained so that texts about similar topics produce similar lists of numbers.

We run every chunk through the same model. Each chunk becomes a list of numbers. Our question, “What was iPhone revenue this quarter?”, also becomes a list of numbers through the same model.

The key idea: because the model was trained on enormous amounts of text, it learned to place “iPhone revenue was $49 billion” and “What was iPhone revenue?” near each other in this number space. They’re about the same topic, so their numbers are similar.

Cosine Similarity: Measuring Closeness

How do we measure “similar”? We use cosine similarity. Think of each list of numbers as an arrow pointing in some direction. Two arrows pointing in nearly the same direction have high cosine similarity (close to 1.0). Two arrows pointing in completely different directions have low cosine similarity (close to 0).

The math is a dot product divided by the magnitudes. For our question and the iPhone revenue chunk: multiply each pair of matching numbers, add them up, divide by the lengths. The result: 0.92. Very similar.

For a chunk about something unrelated, like the legal disclaimer, the score might be 0.15. The numbers point in different directions because the topics have nothing in common.

We compute this score between our question and every single chunk. Then we take the top 3.

Where Similarity Gets It Wrong

This is worth pausing on. The highest-scoring chunk is not always the one with the answer. In our case, the chunk about forward guidance (”We expect revenue to grow 10-12%”) scored higher than the chunk that actually contains “$49 billion.” Why? Because the guidance chunk mentions “revenue” and “quarter” repeatedly, the same words as our question.

Cosine similarity measures word overlap in meaning, not whether the chunk answers the question. This is a real limitation. The system finds related text, not necessarily the right text.

But it doesn’t matter too much, because we send the top 3 chunks to the LLM, not just the top 1. As long as the answer is somewhere in those 3, the LLM can find it.

Generation: The LLM Reads and Answers

We take the top 3 chunks and build a prompt:

Context (from the earnings call):

- “iPhone set a revenue record at $49 billion, up 6% from a year ago, with growth in most markets...”

- “Apple is proud to report $102.5 billion in revenue, up 8%...”

- “We expect December quarter revenue to grow 10-12%...”

Question: What was iPhone revenue this quarter?

Answer using only the context above.

We send this to Claude. The LLM reads all three chunks, identifies the specific fact, and produces:

“iPhone revenue was $49 billion this quarter, up 6% year over year.”

Grounded in the transcript. Not hallucinated. The LLM didn’t need to know this from training. We gave it the information at query time.

The Full Pipeline (Animated)

The animation below walks through every step: starting from the raw transcript, splitting into chunks, converting to numbers, computing similarity, ranking, and generating the final answer.

Where RAG Breaks Down

RAG is not magic. It breaks in predictable ways.

Bad chunking is the most common failure. If a chunk splits a sentence in half, the embedding model sees a fragment without context. “Revenue was” in one chunk and “$49 billion from iPhone” in another. Neither chunk alone answers the question well.

Ambiguous questions cause problems too. “How did Apple do?” could match almost everything in the transcript. The retrieval returns generic chunks about overall performance, missing the specific detail you wanted.

And sometimes the LLM ignores the context anyway. You hand it three relevant chunks and it hallucinates a number that appears nowhere in the transcript. This happens less with explicit instructions (”answer using only the context above”) but it still happens.

Production systems address these issues with re-ranking, hybrid search, and better chunking strategies. But the core loop is exactly what we built: chunk, embed, retrieve, generate.

#AI #NLP #RAG

Suppose you’re looking at count data over time: disasters per year, trades per hour, defaults per month. You suspect the rate changed at some point, but you don’t know when. How do you find the change point and quantify your uncertainty?

This is a classic problem in Bayesian statistics. We’ll walk through it using the UK Coal Mining Disasters dataset (1851-1962), which records the number of major mining disasters per year. The data shows a clear drop in disaster frequency, but when exactly did it happen?

The early years average around 3 disasters per year. The later years average closer to 1. Somewhere in between, something changed. We want to find that transition point. Naruraly in the data above we can see that but how can we be more scientific about it?

The Historical Context

Coal mining in 19th century Britain was brutal. Before regulation, mines were death traps. Children as young as five worked underground. Explosions, collapses, and gas poisonings were routine.

Reform came in waves. The Mines and Collieries Act of 1842 banned women and children under 10 from underground work. The Coal Mines Regulation Act of 1860 raised the minimum age to 12 and introduced basic safety rules. But the real turning point came with the Coal Mines Regulation Act of 1872, which required certified mine managers, mandated proper ventilation, and assigned clear legal responsibility for safety to mine owners.

Further amendments in 1887 tightened enforcement. By the 1890s, the cumulative effect of decades of legislation, union pressure, and government inspection had fundamentally changed how mines operated.

Our model should find this. If the data reflects reality, we’d expect the change point to land somewhere in the 1880s or 1890s, after the major reforms had time to take effect.

The Poisson Distribution (For Counting Events)

When you’re counting events in a fixed period, the Poisson distribution is the standard model. It has one parameter: λ (lambda), the average rate.

P(k events) = λᵏ × e⁻λ / k!

Where:

• λ = average rate of events

• k = number of events (0, 1, 2, ...)

If λ = 3, you expect about 3 events per period. The distribution tells you the probability of seeing 0, 1, 2, 3, ... events.

Higher λ shifts probability toward larger counts. With λ = 1, seeing 5 events is rare. With λ = 5, seeing 5 events is common.

The Change Point Model

We believe the disaster rate changed at some unknown year τ (tau). Before τ, disasters follow Poisson(λ₁). After τ, they follow Poisson(λ₂).

Three unknowns:

• λ₁ = disaster rate in the early period

• λ₂ = disaster rate in the late period

• τ = the year when the rate changed

Our goal is to estimate all three.

What Are Priors?

In Bayesian statistics, we specify what we believe about each parameter before seeing the data. This is called a prior distribution.

Think of it as answering: “If I hadn’t looked at the data yet, what values would I consider plausible?”

For τ (the change point): We have no idea which year the change occurred. Any year is equally likely. So we use a Uniform prior.

τ ~ Uniform(1851, 1962)

Every year gets equal probability: 1/112.

For λ₁ and λ₂ (the rates): We know they must be positive (can’t have negative disasters). We think smaller rates are more plausible than huge ones, but we’re not very opinionated. So we use an Exponential prior.

λ₁ ~ Exponential(mean ≈ 1.7) λ₂ ~ Exponential(mean ≈ 1.7)

Poisson vs Exponential: Two Different Roles

This is where things get confusing. Both distributions involve λ. But they serve completely different purposes.

The Poisson distribution models the actual disaster counts. Given a rate λ, it tells you the probability of observing 0, 1, 2, ... disasters.

The Exponential distribution is our prior belief about λ itself. Before seeing data, we think λ is probably somewhere between 0 and 5, with smaller values more likely than larger ones.

Poisson = “How many disasters happen given rate λ?” Exponential = “What rates λ do we think are plausible before seeing data?”

The Exponential is not claiming disasters follow an exponential process. It’s just a way to express uncertainty about the rate parameter.

How the Model Finds the Answer

The model tests thousands of (τ, λ₁, λ₂) combinations. Combinations with high likelihood get kept more often. Combinations with low likelihood get rejected.

After sampling:

• τ clusters around 1890

• λ₁ clusters around 3.1 disasters/year

• λ₂ clusters around 0.9 disasters/year

The prior for τ was Uniform (all years equally likely). The posterior for τ is concentrated around 1890. The data transformed our belief.

The posterior shows high probability mass around 1889-1891. This is our best estimate for when the change occurred, with uncertainty quantified.

The two rate distributions are well separated. The early period had about 3 disasters per year. The late period had about 1. Strong evidence that a real change occurred.

The expected rate shows a clear transition. The uncertainty band captures when the switch happened.

Limitations

This model assumes exactly one change point. The data might actually have:

• No change point (constant rate)

• Two or more change points

• A gradual transition rather than an abrupt switch

You could extend the model to handle multiple change points, or use model comparison techniques to decide which structure fits best. For this dataset, one change point seems reasonable based on visual inspection and historical context.

Summary

The Bayesian change point model lets you:

• Estimate when a rate changed (τ)

• Estimate the rates before and after (λ₁, λ₂)

• Quantify uncertainty in all estimates

The Poisson distribution models count data. The Exponential distribution expresses prior uncertainty about the rate. Bayes’ theorem combines prior beliefs with observed data to produce the posterior.

For the coal mining data, the model finds a change around 1890, with the disaster rate dropping from about 3 per year to about 1. This aligns with the historical record: the Coal Mines Regulation Acts of 1872 and 1887 had fundamentally transformed the industry by this point. The model, knowing nothing about Victorian politics, independently arrives at the same conclusion historians would.

Some stocks move together. Coca-Cola and Pepsi. Visa and Mastercard. ExxonMobil and Chevron. They’re in the same industry, face the same market conditions, and tend to rise and fall in tandem.

But sometimes they diverge. One gets ahead of the other. And when that happens, they usually snap back together.

Pairs trading exploits this. When two related stocks drift apart, you bet they’ll converge again. You short the one that got too expensive and buy the one that got too cheap. When they meet in the middle, you profit from both sides.

The strategy has been around since the 1980s, pioneered by quants at Morgan Stanley. It still works today, though the easy pairs have been arbitraged away. Finding good pairs requires some statistical machinery. That’s what this post is about.

Stationarity: The Foundation

Before we can trade pairs, we need to understand stationarity.

A stationary time series has a constant mean. It doesn’t drift up or down over time. When it deviates from the mean, it reverts back.

Stock prices are not stationary. Apple went from $1 to $200 over the decades. It doesn’t revert to any fixed level.

Stock returns are roughly stationary. They bounce around zero. Big up days are followed by normal days. Big down days are followed by normal days. Returns don’t trend forever in one direction.

If something is stationary, we can predict it will return to its mean. That’s a tradeable pattern. If something is not stationary, it can drift forever. No edge there.

We test for stationarity using the Augmented Dickey-Fuller (ADF) test. It gives us a p-value:

- p-value < 0.05: The series IS stationary

- p-value > 0.05: The series is NOT stationary

The left chart oscillates around zero. It’s stationary. The right chart trends upward forever. It’s not stationary.

Cointegration: Finding Pairs That Move Together

Two stocks can be correlated but not cointegrated. Correlation means they move in the same direction. Cointegration means they stay close together.

Example: Two stocks both trend upward (correlated). But one grows 20% per year and the other grows 10% per year. The gap between them keeps widening. They’re correlated, but not cointegrated. You can’t trade that divergence because it never reverts.

Two stocks are cointegrated when:

1. Each stock individually is not stationary (they both trend)

2. But their difference (the spread) IS stationary

Think of two drunk friends walking home. Each one staggers randomly. Their individual paths are unpredictable. But they’re holding onto each other, so they can never get too far apart. The distance between them stays bounded.

That’s cointegration. Each series wanders, but the spread between them reverts to a mean.

The top chart shows two stocks that both trend upward. Neither is stationary on its own.

The bottom chart shows their spread. It bounces around a stable mean. The spread IS stationary. This pair is cointegrated.

The Trading Logic

If the spread is stationary, we can trade its mean reversion.

We measure how far the spread has deviated using the z-score:

Z-Score = (Spread - Mean) / Standard Deviation

A z-score of +2 means the spread is 2 standard deviations above its mean. Stock A is expensive relative to Stock B.

A z-score of -2 means the spread is 2 standard deviations below its mean. Stock A is cheap relative to Stock B.

The trading rules:

When z-score crosses above +1 (spread too high):

- Short Stock A (sell, expecting it to fall)

- Long Stock B (buy, expecting it to rise)

- The spread should compress

When z-score crosses below -1 (spread too low):

- Long Stock A (buy, expecting it to rise)

- Short Stock B (sell, expecting it to fall)

- The spread should expand back to normal

When z-score crosses below zero:

- Close all positions

- The spread has reverted to its mean

- Trade complete

[INSERT IMAGE: images/trading_signals.png]

The blue line is the z-score of the spread. The red and green zones show when we enter trades. We enter when z crosses the thresholds. We exit when z returns to zero.

# A Concrete Example

Let’s trace through exactly how this works with numbers.

Say we’re trading two stocks in the same sector. From historical data, we know the spread (Stock A minus Stock B) has a mean of 50 and a standard deviation of 3.

Day 1: A = $100, B = $50. Spread = 50. Z = 0. No trade.

Day 2: A = $102, B = $50. Spread = 52. Z = 0.67. No trade.

Day 3: A = $105, B = $51. Spread = 54. Z = 1.33. Above +1. ENTER.

- Short A at $105

- Long B at $51

Day 4: A = $107, B = $51. Spread = 56. Z = 2.00.

- Unrealized P&L: -1.9% (A went up, hurting our short)

Day 5-8: A starts falling back, B holds steady.

- Unrealized P&L improves as our thesis plays out

Day 9: A = $100, B = $51. Spread = 49. Z = -0.33. Below zero. EXIT.

- Close short A at $100: profit = (105-100)/105 = 4.76%

- Close long B at $51: profit = (51-51)/51 = 0%

- REALIZED PROFIT = 4.76%

The key distinction: while holding (Days 4-8), we had unrealized gains and losses. The position value fluctuated. We only locked in the profit when we exited on Day 9.

Simulated Trading:

Here’s a visual of the flow:

The blue line tracks the z-score over time. The green shaded region marks where z > +1, our entry zone for

betting the spread will shrink (short A, long B). The red shaded region marks where z < -1, our entry zone for

betting the spread will grow (long A, short B). The dashed line at zero is our exit. Once z crosses back to zero, we close both positions and collect our profit.

Today we build a simple momentum strategy using moving averages to gain some intution

What is Momentum?

Momentum is the tendency for recent performance to persist. Winners keep winning. Losers keep losing.

Why does this happen? A few theories:

Slow information diffusion. When good news hits, not everyone reacts immediately. Some investors learn about it days or weeks later. This creates a gradual price adjustment rather than an instant jump.

Herding. When a stock rises, it attracts attention. More buyers pile in. The buying pressure pushes prices higher still. Momentum feeds on itself.

Underreaction: Investors anchor to old prices. When fundamentals improve, they update their expectations slowly. Prices drift upward as reality gradually sinks in.

Whatever the cause, the pattern is real. Our job is to detect it and trade it.

Measuring Momentum with Moving Averages

A simple moving average (SMA) is the average price over the last N periods:

SMAₙ = (Pₜ₋₁ + Pₜ₋₂ + ... + Pₜ₋ₙ) / n

Where:

- Pₜ₋ᵢ = price i periods ago

- n = number of periods (window size)

Short-term SMAs react quickly to price changes. Long-term SMAs smooth out the noise and reveal the underlying trend.

The green line (SMA 3) hugs the price closely and captures recent moves. The red line (SMA 12) is smoother and shows where the stock has been over the past year.

When the short-term average rises above the long-term average, momentum is positive. The stock is trending up. When the short-term average falls below the long-term average, momentum is negative. The stock is trending down.

Building the Momentum Factor

Our momentum factor is simple:

Momentum Factor = SMA₃ − SMA₁₂**

If this number is positive, recent prices are above the long-term trend. Bullish. If negative, recent prices are below the long-term trend. Bearish.

One critical detail: we shift the moving averages by one period. Why? To avoid look-ahead bias.

When you sit at your desk on January 31st deciding whether to buy, you do not know January’s closing price yet. You only know data through December. The shift ensures our signal uses only information that was actually available at decision time.

The Median Threshold

Now we need to convert our momentum factor into a trading signal. We use the median.

**Signal Logic:**

- If Factor > Median → Go Long (+1)

- If Factor < Median → Go Short (−1)

The red dashed line marks the median. Everything to the right gets a long signal. Everything to the left gets a short signal.

No look-ahead bias here either. We calculate the median using only training data (the first 70% of our sample), then apply that fixed threshold to the test data (the remaining 30%). We never peek at future information.

Putting It Together

The full strategy works like this:

1. Calculate moving averages (SMA 3 and SMA 12), shifted by one period

2. Compute momentum factor (SMA₃ − SMA₁₂)

3. Normalize using z-scores from training data

4. Generate signal based on median threshold from training data

5. Calculate profit as signal × next month’s return

That last step deserves explanation. We see the momentum signal at the end of month T. We trade at the start of month T+1. We earn month T+1’s return. So profit equals last month’s signal multiplied by this month’s return.

The blue line shows our momentum strategy’s cumulative return on out-of-sample data. The gray dashed line shows a simple buy-and-hold benchmark.

What Can Go Wrong

This is a toy example, not investment advice. A few caveats:

Transaction costs. We trade monthly. Each trade has costs: commissions, bid-ask spreads, market impact. These add up. A strategy that looks great on paper can turn negative after costs.

Momentum crashes. Momentum works until it does not. When markets reverse sharply (like March 2009), momentum strategies get crushed. They hold yesterday’s winners, which become today’s biggest losers.

Overfitting. We picked 3-month and 12-month windows. Why not 2 and 10? Or 4 and 15? With enough parameter combinations, you can find something that worked in the past purely by chance. Out-of-sample testing helps, but is not foolproof.

Crowding. Momentum is well known. When everyone runs the same strategy, the edge gets arbitraged away. Signals become self-defeating.

You have $100,000 to invest across six ETFs. How do you split it?

You could go equal weight. Put $16,667 in each and call it a day. But that ignores everything we know about these assets. Some have higher expected returns. Some are less volatile. Some zig when others zag. Why not use that information?

That’s what portfolio optimization does. You feed it historical returns, and it spits out weights that balance return against risk. We’ll look at Mean-Variance optimization which aims to balance the returns and risk of a portfolio, and its sister, MVSK, which accounts for futher moments of skewness and kurtosis.

The Assets

We’re using six sector ETFs:

The data covers 2018 to 2024.

Correlations Matter

Before optimizing, look at how these assets move together.

A correlation of 1.0 means two assets move in lockstep. No diversification benefit there. Correlation near zero means they move independently. Negative correlation means they tend to move opposite directions.

GLD (Gold) has near-zero correlations with the equity ETFs. When stocks tank, gold often holds steady or goes up. That makes it useful even if its returns are nothing special on their own.

XLE (Energy) also marches to its own beat. Oil prices respond to supply shocks and geopolitics more than general market sentiment.

Mean-Variance Optimization

MV optimization maximizes a utility function that rewards returns and penalizes variance:

U = μ - (λ/2)σ²

Where:

• μ is the portfolio’s expected return (annualized)

• σ² is the portfolio’s variance (how much returns bounce around)

• λ is your risk aversion. Higher values mean you hate volatility more.

Think of λ as a dial. Set it to 1 and the optimizer goes aggressive, chasing returns. Set it to 10 and it gets conservative, prioritizing stability. A value around 3 is moderate.

With λ = 3, here’s what comes out:

Heavy allocation to XLK (Technology) and GLD (Gold). XLK had strong returns in this period. GLD helps because its low correlation with everything else brings down portfolio variance without hurting returns much.

The Problem with Variance

MV treats upside and downside volatility the same. A 5% gain and a 5% loss both add to variance equally.

But that’s not how people experience risk. Losses sting more than gains feel good. And real returns aren’t symmetric. Big drops happen more often than big rallies. The distribution has a fat left tail.

Two statistics capture what variance misses:

Skewness measures asymmetry. You take each return’s deviation from the mean, cube it, average those cubes, and divide by standard deviation cubed.

• Positive skewness: longer right tail, occasional big gains

• Negative skewness: longer left tail, occasional big losses

• Zero: symmetric

Most stocks have negative skewness. Crashes are more common than melt-ups.

Kurtosis measures how fat the tails are. Same idea but raise deviations to the fourth power instead of the third. Subtract 3 so a normal distribution scores zero.

• Positive kurtosis: fat tails, extreme events happen more than you’d expect

• Negative kurtosis: thin tails, extreme events are rare

• Zero: normal distribution

Most stocks have positive kurtosis. The 2008 crash and COVID’s March 2020 drop were supposedly “six sigma” events. Once in a million years stuff. Except they happen way more often than that.

MVSK Optimization

MVSK extends the utility function:

U = μ - (λ/2)σ² + (λₛ/6)S - (λₖ/24)K

Where:

• S is portfolio skewness. We reward positive values (more upside surprises).

• K is portfolio kurtosis. We penalize positive values (fewer extreme events).

• λₛ and λₖ control how much we care about each.

The 1/6 and 1/24 coefficients come from Taylor series math. The signs make sense: add skewness (we like it positive), subtract kurtosis (we want it low).

How do MVSK weights compare to MV?

Often the differences are subtle. MVSK might shift weight toward assets with better skewness even if they’re slightly more volatile. Trading variance for better tail behavior.

Out-of-Sample Test

Optimizing on historical data is easy. Does it work on data the optimizer never saw?

We trained on 2018-2021 and tested on 2021-2024. Here’s how $1 grows under each strategy:

During calm markets, they track closely. During rough patches like late 2022, differences emerge. Neither dominates everywhere.

The return distributions:

Similar centers, but different tails. That’s where MVSK focuses.

How Risk Aversion Changes Everything

λ matters a lot. Here’s how weights shift as you crank it up:

Low λ (aggressive): pile into XLK for returns. High λ (conservative): shift toward XLU and GLD for stability.

The vertical line is λ = 3. Balanced, not too hot, not too cold.

Summary

What to remember:

MV is the default for a reason. Simple, well understood, works fine when returns are roughly normal. Start here.

MVSK adds tail awareness. Worried about crashes? MVSK penalizes bad skewness and fat tails. Costs you some complexity.

Don’t sleep on equal weight. Zero estimation error. When your estimates are garbage (often true), equal weight beats fancy optimizers out of sample.

Always test out of sample. Any optimizer looks good on its training data. Reserve a holdout period.

λ controls the aggression. Start around 3, adjust to taste.

Returns & Noise

At its core, a return is just the percentage change in price:

• Rₜ = (Pₜ − Pₜ₋₁) / Pₜ₋₁**

Where Pₜ is today’s price and Pₜ₋₁ is yesterday’s price. If a stock goes from $100 to $102, that’s a 2% return. Simple enough.

The key insight here is that returns have two components: **signal** (the expected return) and **noise** (random fluctuations). When we look at daily stock returns, most of what we see is noise. The signal is buried in there somewhere, but extracting it is the whole game.

Volatility

Volatility measures how much returns bounce around. It’s the standard deviation of returns:

• σ = √[ Σ(Rᵢ − R̄)² / n ]**

Where:

- σ = volatility (standard deviation)

- Rᵢ = return on day i

- R̄ = mean return

- n = number of observations

A stock with 30% annual volatility will have wilder swings than one with 10% volatility.

We typically annualize daily volatility:

**σ_annual= σ_daily × √252**

Why √252? Because variance scales linearly with time, but standard deviation scales with the square root. There are 252 trading days in a year.

ₐ

Notice how the high-vol stock (red) has returns spread across a much wider range. More uncertainty, more risk. The low-vol fund (green) clusters tightly around zero.

Higher volatility means more uncertainty. If you’re saving for a house next year, a high-vol stock might keep you up at night. For a retirement account 30 years out? You can probably stomach the swings.

Beta

Beta tells you how much a stock moves relative to the market:

• β = Cov(Rᵢ, Rₘ) / Var(Rₘ)**

Where:

- β = beta

- Cov(Rᵢ, Rₘ) = covariance between stock and market returns

- Var(Rₘ) = variance of market returns

It’s the slope of the regression line when you plot stock returns against market returns.

**β = 1** means the stock moves with the market.

**β > 1** means the stock amplifies market moves (up more on good days, down more on bad days).

**β < 1** means the stock dampens market moves.

Look at the slopes. The high-beta stock (around 1.5) moves 1.5x the market. When the market is up 1%, this stock tends to be up 1.5%. The lower-beta fund (around 0.8) has a flatter slope—it doesn’t swing as hard.

Tech stocks tend to have high betas. Utilities tend to have low betas. This isn’t good or bad—it depends on what you’re trying to achieve.

## Alpha

Alpha is the holy grail. It’s the excess return beyond what beta would predict.

The logic comes from CAPM (Capital Asset Pricing Model), which says your expected return should be:

**E(Rᵢ) = Rꜰ + βᵢ × (Rₘ − Rꜰ)**

Where:

- E(Rᵢ) = expected return of asset i

- Rꜰ = risk-free rate (Treasury bills, ~5%)

- βᵢ = beta of asset i

- Rₘ = market return

- (Rₘ − Rꜰ) = market risk premium

Alpha is the difference between actual and expected return:

**α = Rᵢ − E(Rᵢ)**

**α = Rᵢ − [Rꜰ + β × (Rₘ − Rꜰ)]**

If CAPM says you should earn 10% given your beta, but you earned 12%, that 2% is your alpha. It’s the “skill” component—the value added beyond just riding the market.

The blue diagonal line shows where you SHOULD be based on your beta. The colored dots show where each fund ACTUALLY landed. The dashed lines show the alpha—the gap between expected and actual.

The Good Fund (green) sits above the line with positive alpha—it’s beating expectations. The Bad Fund (red) sits below the line with negative alpha—it’s destroying value. The High Beta Stock (blue) is roughly on the line—no skill, just market exposure.

Reality check on alpha

Most actively managed funds have α ≈ 0% or negative after fees. Consistent α of +2% annually is exceptional (top 10% of managers). Warren Buffett’s lifetime α is estimated at 4-6%—that’s GOAT territory.

Index funds are often used as baselines as why pay managers that deliver poor alpha?

Correlation

Correlation measures how two assets move together, ranging from -1 to +1:

• ρ(X,Y) = Cov(X,Y) / (σₓ × σᵧ)**

Where:

- ρ = correlation coefficient

- Cov(X,Y) = covariance between X and Y

- σₓ, σᵧ = standard deviations of X and Y

It’s covariance normalized to always fall between -1 and +1.

**ρ = +1** means perfect positive correlation (move together exactly).

**ρ = 0** means no correlation (independent movements).

**ρ = −1** means perfect negative correlation (move exactly opposite).

Tech A and Tech B are highly correlated (0.87)—makes sense, they’re in the same sector. Bonds are negatively correlated with the market (−0.80)—when stocks tank, investors flee to bonds. Gold just does its own thing (near zero correlation with everything).

This is why diversification works. If all your assets move together, you’re not actually diversified. Mix in some uncorrelated or negatively correlated assets and your portfolio’s overall volatility drops. Summary

• Volatility (σ)

  • Formula: σ = std(R)

  • What it tells you: How risky the asset is (how much returns fluctuate)

• Beta (β)

  • Formula: β = Cov(Rᵢ, Rₘ) / Var(Rₘ)

  • What it tells you: How the asset moves relative to the market

• Alpha (α)

  • Formula: α = Rᵢ − [Rꜰ + β(Rₘ − Rꜰ)]

  • What it tells you: Excess return beyond what beta predicts

• Correlation (ρ)

  • Formula: ρ = Cov(X, Y) / (σₓ × σᵧ)

  • What it tells you: How two assets move together

When evaluating an investment, you want to know: What’s the expected return? What’s the risk? And how does this fit with what I already own?

Since they’ll take random steps, we can multiple simulations of where they might end up after 10 steps. We can extrapolate this further over time to get the following

This isn’t telling us much. We just have a person just stumbling around and we’re just guesttimating their position. However, the key idea is that each step is independent. The step the person at step 100 is irrelevant to any position or step they made prior to to that. Hence the randomness. We can draw a parallel to modeling stock prices, where the stock price of tomorrow is independent of the stock price today. It’s just randomness that impacts the price.

Statistical Properties

We can model a random walk with above, where the position at some time t is just the previous step with some white noise. Now since the expectation is a linear operator, the expectation of a random walk is just the initial position. Meaning if we were to ask on average, where do we expect the drunk person to be, it would be their starting point. We can observe this as when we plot over time, the expectation just back to 0.

Likewise, if we take the variance, the variance is proportional to the square root of the number of steps.

Mirroring Stock Prices

Now this is simply a rudimentary example to demonstrate how this works. Suppose we have a stock priced at $100 and want to predict how the price is going to be. We’re not psychic, but using this random walk model, we can simulate the path of the prices it’s going to be in the future. We can do this a bunch of time which can help us give us a range of where we expect the prices to be . Assuming we have daily returns from a normal distrubtion with mu=0, and sigma=2%, we can get the following simulations.

Now they both have the same average return of 6.2% and similar annualized returns of 7.4% and 7.1% respectively. In addition to returns, we also want to factor in risk in our decision making. Notice portfolio 2 is more volatile. Imagine you’re saving up to buy a house in a couple years, you might have a lower risk tolerance given that you need the money sooner. Versus a retirement account you would be open to more risk. This is where we need a way to quantify the risk

Sharpe Ratio

The most common way to measure risk in a portfolio is the sharpe ratio.

Decomposing this, we are simply looking at the ratio of the excess returns vs its risk, where the risk is we measure as the standard deviation of returns. I.E, getting more bang for my buck. For each unit of risk, I want more return.

Now we use a risk free rate as a baseline to calculate the “excess return”. One can invest in treasury bill which are generally risk free to serve as a baseline as if you are going to be taking some risk, you want it have better returns than the baseline.

def get_sharpe_ratio(returns, annual_risk_free_rate=0.05):
    """
    Determines the Sharpe ratio of a strategy.

    Parameters
    ----------
    returns : pd.Series or np.ndarray
        Daily returns of the strategy, noncumulative.
    adjustment_factor : int, float
        Constant daily benchmark return throughout the period.
    Returns
    -------
    sharpe_ratio : float
    Note
    -----
    See https://en.wikipedia.org/wiki/Sharpe_ratio for more details.
    """
    daily_risk_free_rate = (1 + annual_risk_free_rate) ** (1 / 252) - 1
    returns_risk_adj = returns - daily_risk_free_rate
    excess_returns = returns.mean()-daily_risk_free_rate
    volatility = returns.std()
    return np.sqrt(252)*excess_returns/volatility #annaulize it

Sortino Ratio

In addition, we can look at the Sortino Ratio. Notice the formula is the exact same, excess this time we only look at the standard deviation of the downside. AKA, we only focus in the negative returns. Since we’re focusing on the negative returns, this will inform us about worrying about the big losses in the portfolio. Yes we can have positive volatility, but that’s still positive returns so in this case I am comfortable with that volatility vs swings in the other direction.

def get_sortino_ratio(returns, annual_risk_free_rate=0.05):
    """
    Determines the Sharpe ratio of a strategy.

    Parameters
    ----------
    returns : pd.Series or np.ndarray
        Daily returns of the strategy, noncumulative.
    adjustment_factor : int, float
        Constant daily benchmark return throughout the period.
    Returns
    -------
    sharpe_ratio : float
    Note
    -----
    See https://en.wikipedia.org/wiki/Sharpe_ratio for more details.
    """
    daily_risk_free_rate = (1 + annual_risk_free_rate) ** (1 / 252) - 1
    returns_risk_adj = returns - daily_risk_free_rate
    excess_returns = returns.mean()-daily_risk_free_rate
    negative_volatility = returns_risk_adj[returns_risk_adj<0].std()
    return np.sqrt(252)*excess_returns/negative_volatility  #Annualize it

Putting it all together

We usually have to deal with much more nuanced data than a 5 year return. But the beauty of this is all we need is returns data to calculate the ratio. Suppose I’m deciding in portfolios in just pure Tesla Stock or pure Apple stock. I see that the daily returns for the past 5 years below, which is ugly to look. So let’s just feed in the daily returns to measure our risk

We can observe that TSLA gives us a better Sharpe & Sortino Ratio than apple. In practice, we deal with large portfolios with varying stocks and weights. However it’s really just plug and play. I just need the returns of the portfolios, I don’t care about the compositions.