Featured image of post LynxCrypto实战指南:6步掌握pandas真实加密数据

LynxCrypto实战指南:6步掌握pandas真实加密数据

摒弃语法理论,通过ccxt获取真实BTC行情、pandas处理、vectorbt回测,6步走完加密量化中pandas的全部核心用法——从数据读取、重采样、手动计算RSI/MACD,到多资产对齐和回测引擎导入。

You can’t do crypto quant without pandas. Your market data is a table, your indicators are columns, and your backtesting engine consumes a time-indexed DataFrame — whether it’s vectorbt, Freqtrade, or whatever strategy class lives in your strategy/ directory, they all run on this same data structure.

But most people learn pandas the wrong way: they chew through syntax manuals and memorize APIs, then still can’t handle a single candlestick.

This is the pandas crash course I put together for myself, built around one principle: skip the grammar, learn by driving real crypto data. In quant, pandas only ever does four things — pull data in, shape it into a time-series table, calculate technical indicators, and run statistical analysis. I’ve broken it down into 6 steps, each pairing a concept set with runnable JupyterLab code. By the end, you’ll be able to independently pull data from ccxt → process it with pandas → backtest with vectorbt, closing the full loop.

Environment assumptions: You have a Python environment with ccxt + pandas + vectorbt installed (mine lives in the .venv of the LynxCrypto project), and JupyterLab is already running. If not, pip install ccxt pandas vectorbt jupyterlab has you covered.

If you’re brand new to pandas, spend 30–60 minutes working through these HTML tutorials first — all of them let you copy-paste and run code directly:

  • Kaggle Free Pandas Course — https://www.kaggle.com/learn/pandas Run code and exercises right in your browser, completely free. The best way to get comfortable with Series / DataFrame / index / grouping concepts.
  • Official Pandas 10 Minutes to pandas — https://pandas.pydata.org/docs/user_guide/10min.html The official quick tour — all code is copyable, gives you a solid big-picture sense of what pandas can do.
  • 《Python for Data Analysis》 (by pandas creator Wes McKinney), chapters 5–9, available free online. More practical, and great to keep open as a reference dictionary.

Go through any one of the above before coming back to the 6 steps below, and you’ll move through them significantly faster than if you just brute-read syntax.

Step 1 — Turn Market Data into a DataFrame (Reading + Structure)

Concepts covered: Series, DataFrame, index, columns, dtypes, head/tail, to_datetime

Task: Pull BTC/USDT OHLCV candles from an exchange using ccxt and convert them into a time-indexed table.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import ccxt
import pandas as pd

# Fetch BTC/USDT 1-minute candles from Binance — public data, no API key needed
# Connection issues? Make sure your proxy is running, or swap 'binance' for 'okx'/'gate'
ex = ccxt.binance()
ohlcv = ex.fetch_ohlcv('BTC/USDT', '1m', limit=500)  # 500 candles

# ccxt returns a list of lists: [timestamp, open, high, low, close, volume]
df = pd.DataFrame(ohlcv, columns=['ts', 'open', 'high', 'low', 'close', 'vol'])
df['ts'] = pd.to_datetime(df['ts'], unit='ms')   # millisecond timestamp → datetime
df = df.set_index('ts')                            # make time the index
df.head()                                          # preview the first 5 rows
df.dtypes                                          # check each column's data type

Key takeaway: Each column in the table is a Series; the whole table is a DataFrame; the leftmost column (time) is the index. The index is the foundation of all pandas time-series operations — everything from resampling to slicing depends on it.

Step 2 — Time-Series Cleaning and Slicing (Indexing, Slicing, Resampling)

Concepts covered: DatetimeIndex, loc/iloc slicing, resample, time zones

Task: Resample 1-minute candles into 4-hour candles and grab the most recent period.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# Slice by time: grab a date range using loc + string dates
recent = df.loc['2026-09-05':]

# Resample: 1m → 4h OHLC (the financial-standard aggregation pattern)
df_4h = df.resample('4h').agg({
    'open': 'first',   # interval open = first candle in the window
    'high': 'max',      # interval high = max of all highs
    'low': 'min',
    'close': 'last',   # interval close = last candle in the window
    'vol': 'sum'       # interval volume = sum of all volumes
})
df_4h.head()

# Position-based slicing (iloc, by row number)
df.iloc[:10]      # first 10 rows
df.iloc[-5:]      # last 5 rows

Key takeaway: loc slices by label (time strings / index values), while iloc slices by position (which row number). resample is the core tool for compressing high-frequency data into lower frequencies — 1m→4h, 1h→1d, all of it relies on this.

Step 3 — Calculate Technical Indicators by Hand (Column Ops, Rolling, Shift, EWM)

Concepts covered: column operations, rolling (rolling windows), shift (lag), ewm (exponentially weighted), apply/lambda

Task: Compute MA, RSI, and MACD from scratch instead of importing a ready-made library — understand how the indicators actually work.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# Moving average MA(20): rolling mean of the past 20 candles
df['ma20'] = df['close'].rolling(20).mean()

# RSI(14): compute price changes, then smooth with EWM
delta = df['close'].diff()                     # diff = current minus previous
gain = delta.clip(lower=0)                      # keep only the positive part (gains)
loss = -delta.clip(upper=0)                     # keep only the negative part (losses), flip sign
avg_gain = gain.ewm(alpha=1/14, adjust=False).mean()
avg_loss = loss.ewm(alpha=1/14, adjust=False).mean()
rs = avg_gain / avg_loss
df['rsi'] = 100 - 100 / (1 + rs)

# MACD: difference between two EMAs
ema12 = df['close'].ewm(span=12, adjust=False).mean()
ema26 = df['close'].ewm(span=26, adjust=False).mean()
df['macd'] = ema12 - ema26
df['signal'] = df['macd'].ewm(span=9, adjust=False).mean()
df['hist'] = df['macd'] - df['signal']

df[['close', 'ma20', 'rsi']].tail()             # peek at the most recent rows

Key takeaway: rolling(20).mean() opens a 20-element sliding window and computes the mean — nearly every trend indicator comes from this pattern. shift(1) nudges data back one step, which is how you compare “today’s value against yesterday’s.” ewm is the decay-weighted mean that RSI and MACD both rely on. Walking through these by hand makes it immediately clear what’s happening under the hood of libraries like ft-pandas-ta.

Step 4 — Multi-Asset Alignment (Concat, Merge, Correlation)

Concepts covered: concat, merge/join, multi-asset time alignment, fillna, corr (correlation)

Task: Align BTC and ETH closing prices into a single table and compute their correlation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Pull ETH close prices using the same pattern
eth = ex.fetch_ohlcv('ETH/USDT', '1m', limit=500)
eth_df = pd.DataFrame(eth, columns=['ts','o','h','l','c','v'])
eth_df['ts'] = pd.to_datetime(eth_df['ts'], unit='ms')
eth_close = eth_df.set_index('ts')['c']

# Horizontal merge: stack BTC close and ETH close side by side (axis=1 = by column)
both = pd.concat({'BTC': df['close'], 'ETH': eth_close}, axis=1)
both = both.dropna()            # drop rows that didn't align

both.corr()                     # correlation matrix
both['spread'] = both['BTC'] - both['ETH']   # new column = difference between the two
both.plot()                     # plot directly (pandas has built-in plotting)

Key takeaway: The core of any multi-asset strategy (pairs trading, cross-asset arb) is time alignment — if timestamps don’t line up between two coins, you either fillna or dropna, and concat(axis=1) is your main tool for horizontal merging. BTC and ETH’s high correlation is a basic fact of crypto markets, and both pairs trading and statistical arbitrage are built on top of it.

Step 5 — Statistics and Visualization (describe, groupby, Plotting)

Concepts covered: describe, pct_change (returns), groupby (grouping), agg (aggregation), plot

Task: Compute return distributions, group by day/hour to find patterns, and visualize.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
df['ret'] = df['close'].pct_change()          # returns = (today - yesterday) / yesterday
df['ret'].describe()                          # quick stats: mean / std / quantiles

# Group by hour: see which hour has the biggest volatility
df['hour'] = df.index.hour
df.groupby('hour')['ret'].agg(['mean', 'std', 'count'])

# Aggregate daily returns
df['day'] = df.index.date
daily_ret = df.groupby('day')['ret'].sum()

# Plot (inline in the notebook)
df[['close','ma20']].plot(title='BTC close + MA20')
df['ret'].hist(bins=50)                        # returns distribution histogram

Key takeaway: describe() is the fastest health check you can run on any dataset. groupby is “split by a column, then compute separately for each group” — finding patterns by hour / day of week / month in quant all depend on it. The shape of your returns distribution (normal? fat-tailed?) determines your strategy’s risk assumptions. Crypto returns have pronounced fat tails — which is exactly why martingale loses every time long-term (see my previous post on the math of blowups).

Step 6 — Connect to vectorbt (Feeding Your DataFrame into a Backtest Engine)

Concepts covered: nothing new in pandas here — this step is about understanding how your research output enters the backtester. A pandas DataFrame is vectorbt’s standard input.

Task: Use the MA20 you calculated in Step 3 to generate crossover signals and feed them into a vectorbt backtest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import vectorbt as vbt

# Signal: buy when close crosses above MA20, sell when it crosses below (boolean Series)
entries = df['close'] > df['ma20']
exits   = df['close'] <= df['ma20']

# vectorbt takes close (Series) + boolean entry/exit signals
pf = vbt.Portfolio.from_signals(df['close'], entries, exits, init_cash=10000)
pf.stats()                  # backtest stats: total return / win rate / max drawdown, etc.
pf.plot()                   # backtest equity curve

Key takeaway: By this point, pandas has produced your finished data product — vectorbt, your strategy classes in strategy/, everything consumes this same time-indexed DataFrame. The notebook is where you prototype ideas during research (these 6 steps); once the logic is locked in, you distill it into a .py module under strategy/. That’s the division of labor between Jupyter and production code: notebook as scratch paper, .py as deliverable.

What You Can Do After This

  • You’ve worked through all 6 core pandas operations in quant (read / slice / compute indicators / align / statistics / feed to backtest) using real crypto data
  • You can independently run the full chain: ccxt → pandas → vectorbt
  • Next up: map these hand-calculated indicators and signals against the strategies already in your strategy/ directory (I compared mine against MacdCross) and internalize the relationship between the “notebook exploration” version and the “production module” version

Four Anti-Patterns (I Fell into These — Don’t)

  • Don’t cram ten different ideas into one notebook — one notebook per research question, or you won’t be able to read it yourself in three months.
  • Write your assumptions and conclusions at the top of every notebook — when you come back to it, read those two sections first, then the code.
  • Distill finalized logic back into .py — keep notebooks for research only; don’t let them turn into a stew.
  • Don’t just read — run it — type out every snippet, tweak parameters, watch the output change. You won’t learn by only looking.

This is a practical introduction in the LynxCrypto series. Other posts in the series: Research Blueprint · From Grid Martingale to Alpha (Part 1) · Falsification · WFO Verdict · Tools Roadmap