Skip to main content

Introduction into Machine Learning

Hans Dembinski

Overview

  • Caution: Field is riddled with buzzwords

    • Machine learning, artificial intelligence, deep learning, belief networks, attention-based networks, long short-term memory, features, hyperparameters …

  • Anthropomorphized and technical names hide what are usually basic mathematical algorithms

  • Machine learning: umbrella term for multivariate statistical methods for classification and regression

    • Multivariate: algorithms use more than one variable as input, e.g. (x, y, z)
    • Statistical: algorithms can handle noisy input and are founded on statistical theory
    • Classification: sort inputs based on input variables into categories
    • Regression: find a mapping of input vectors to output vectors (fitting)

  • Rise of the machines: Why do we need them?
    • Humans good at working with 1D or 2D data, because we can visualize it
    • Humans bad at working with higer-dimensional data
    • Algorithms (“machines”) scale well to high-dimensional data

Notable software

Learning resources

  • Internet provides excellent resources for autodidactic learning

    • Example code
    • Tutorials
    • Documentation

  • Scikit-Learn

  • Kaggle

  • All methods can do classification and regression and work with any input

  • What to use?

    • Structured data (i.e. tables of numbers): Ensembles of decision trees
      • Good results with default settings
      • Can process continuous and categorical input (m/w, red/green/blue, …)
      • No data preprocessing required
      • Compute feature importance
    • Unstructured data (i.e. images, waveforms, text…): Neural networks
      • Deep networks learn to extract features automatically
      • Requires data preprocessing
      • Networks difficult to optimize for non-experts
      • Often need very large data sets for training

Some definitions * sample: one or more numbers which describe outcome of stochastic process * dataset: consists of samples * features: subset of numbers from each sample which carry interesting information, also numbers computed from each sample * signal: samples which originate from stochastic process of interest, typically particle decay * background: samples which do not originate from process of interest

Discuss how the typical decay \(K^0_S \to \pi^+ \pi^-\) is reconstructed and why in general there is always some background.

Clarify how the previous terms would be used in the context of this example.

Classification

  • Setting: dataset is mix of two or more components, typically signal + one or more backgrounds
  • Want only signal component, invert a cut to select signal candidates
  • Features are variables which allow to discriminate between signal and background, for example, reconstructed mass of decay candidate
scipy 1.10.0
matplotlib 3.7.1
numpy 1.21.6
sklearn 1.2.2
xgboost 1.7.4

Exercise: * Propose simple selection (“cut”) that keeps most of the signal and removes most of the background * Is it possible to eliminate the background completely?

Efficiency and purity of a selection

  • Efficiency: fraction of signal left after applying selection
    • Number of true signal samples after cut / Number of all true signal samples
  • Purity: fraction of background still contained in selected sample
    • Number of true signal samples after cut / Number of selected samples
  • Want to maximize efficiency and purity

Single-parameter cut

efficiency at max purity 0.904

  • How to choose cut?

    • Want to maximize efficiency and purity, but what trade-off between the two?

  • Often subjective choice is acceptable, because exact answer is difficult

  • Objective answer given by figure-of-merit (FOM)

    • Popular is Punzi FOM
    • Most FOM (including Punzi FOM) only minimize statistical uncertainty
    • Correct FOM has to minimize total uncertainty = statistical ⊕ systematic
    • Systematic uncertainty often difficult to quantify: fallback to Punzi or subjective choice

BDT-based cut

  • We can do better with two cuts, on xmin and xmax

  • Space of cut parameters then already two-dimensional

  • With 2D data

    • Four rectangular cuts possible (4 cut parameters)
    • Infinitely many non-rectangular cuts possible

  • Use machine instead, needs to be trained on labeled data

  • Use BDT classifier from XGBoost

  • Gradient boosting machines regularly win Kaggle competitions

  • Well documented, fast, parallizable, tunable

Exercise: * Vary some hyperparameters in XGBClassifier to see how score, runtime, and predicted probability changes * n_estimators try e.g. 2, 5, 10 * max_depth try e.g. 1, 5, 10 * learning_rate try e.g. 0.001, 0.05, 0.5, 1

  • BDT predicts signal probability
  • We use this to implement our cut: select regions with high signal probability

  • Physicists like efficiency and purity, but ROC curve is standard
  • Receiver-operator characteristic
  • ROC plots true positive rate (fraction of signal that passes cut) over false positive rate (fraction of background that passes cut)
  • Scikit-Learn provides a quick way to generate this plot

  • area under curve (AUC) is a common metric to compare the performance of two different classifiers
  • can also use Scikit-Learn to compute ROC points and AUC and plot these ourselves

  • Exercise: run scikit-learn example
  • Try adding XGBClassifier into the comparison
  • Play around with the hyperparameters of the classifiers

Regression

  • Regression (aka fitting): “learn” function y = f(x) from points(x, y), where both x and y can be vectors

  • Machines can do this, but we rarely need this in HEP

  • We usually prefer to use theoretically motivated models or hand-crafted low-order polynoms

  • Let’s try regression on one-dimensional function

  • generate random points on this curve and add some noise to make it more realistic

  • (X, y) are inputs for regression similar to before, but y is a real number now, not a label (integer)

Neural network

  • We use MLPRegressor from Scikit-Learn, a basic neural network
  • Neural networks always need preprocessing
  • Scikit-Learn offers pipelines to automate this, use them

  • Exercise: play around with hyperparameters of MLPRegressor
  • What happens if you remove the StandardScaler?
  • What performs better, deep or wide networks?

Gaussian Process

  • For low-dimensional data, Gaussian Process Regression can work well
  • Not easy to set up: need to select suitable kernels and configure them
  • Advantage: Predicts uncertainty and automatically provides smooth solution