ainesmile

Mean Shift

Wed, 18 Nov 2020 00:00:00 +0000

1. Kernel Density Estimation

Kernel density estimation(KDE) is a non-parametric method to estimate the probability density function(pdf). A non-parametric density estimator assumes no pre-specified function form for the pdf. Assume \(X = (x_1, x_2, ... x_n), x_i \in \mathbf{R}^d\) are independent and identically distributed(i.i.d.) random variables with continuous density function \(f\), the kernel density estimator is defined as

\[\begin{equation} \hat{f}(x;H)= \sum_{i=1}^{n} \alpha_i \mathbf{K}(x-x_i, H) \label{eq1} \end{equation}\]

where \(\mathbf{K}\) is the kernel, \(H\) is the bandwidth, and \(\alpha_i\) is the weighting coefficient.

1.1 Interpretation

In my understanding, the form of equation \(\eqref{eq1}\) can be interpreted like this:

The kernel \(K_i = K(x-x_i, H)\) is the estimated density function for data point \(x_i\)
The coefficient \(\alpha_i\) is the weight for data point \(x_i\), usually set as \(1/n\)
\(\sum_{i=1}^{n} \alpha_i \mathbf{K}(x-x_i, H)\) is the weighted estimate probability at \(x\)

Given a single data point \(x_i\), when we try to estimate the density function \(K_i\) for \(x_i\), it is nature to assume that:

the maximum should locate at \(x_i\)
only the distance to \(x_i\) matters
the closer to \(x_i\), the bigger the probability

Based on these, a kernel usually satisfies:

Non-negative, \(K(x) \geq 0\) for all \(x\), probability should be non-negative
Normalization, \(\int_{-\infty}^{\infty} K(x)d(x) = 1\), results in a pdf
\(K(0)\) is the maxima
Non-increasing for distance, \(K(a) \geq K(b)\) if \(\|a \| < \|b\|\)
Symmetry, \(K(x) = K(-x)\)

The choice of the kernel function is not crucial, the most widely used kernel is the normal kernel

\[\begin{equation} K(x-x_i;H) = \frac{\exp(-\frac{1}{2}(x-x_i)^{T}H^{-1}(x-x_i))}{\sqrt{(2\pi)^d|H|}} \end{equation}\]

For more common used kernel functions, please see wiki Kernel (statistics).

1.2 Bandwidth

The bandwidth matrix \(H\) is a symmetric, positive matrix. The choice of bandwidth plays an important role. Selecting an optimal bandwidth is a bias-variance trade-off which is not easy. A large bandwidth has low variance but can leads to oversmoothing. To reduce the complexity, the bandwidth matrix \(H\) is often chosen either as a diagonal matrix \(H = diag[h_1^{2}, ..., h_{d}^{2}]\) with \(d\) bandwidth parameters or a diagonal matrix \(H = h^{2}I_{d}\) with a single bandwidth parameters \(h \neq 0\). This article will consider the later case, combined with the normal kernel, we have

\[\begin{equation} \begin{aligned} \hat{f}(x) & = \frac{1}{n} \sum_{i=1}^n K(x-x_i; H) \\ & = \frac{1}{n} \sum_{i=1}^n \frac{\exp(-\frac{1}{2}(x-x_i)^{T}H^{-1}(x-x_i))}{\sqrt{(2\pi)^d|H|}} \\ & = \frac{1}{n} \sum_{i=1}^n \frac{\exp(-\frac{1}{2h^2}(x-x_i)^{T}(x-x_i))}{h^{d}\sqrt{(2\pi)^{d}}} \end{aligned} \end{equation}\]

To make the estimated density function to integrate to one,

\[\begin{equation} \hat{f}(x) = \frac{c_{k, d}}{nh^{d}} \sum_{i=1}^n \exp(-\frac{1}{2} (\frac{x-x_i}{h})^{T} (\frac{x-x_i}{h})) \label{eqnor} \end{equation}\]

where \(c_{k, d} > 0\) is the normalization constant.

2. The Mean Shift

The mean shift procedure is an adaptive gradient ascent method to locate the modes of a density function. Given the unknown density function \(f(x)\), the mean shift procedure is a way to locate the modes without estimating the density function. In this case, the modes are the maxima located at the zeros of the gradient \(\bigtriangledown f(x) = 0\). The mean shift procedure is based on the kernel density estimation.

2.1 Equations

In a general way, equation \(\eqref{eqnor}\) can be rewritten as

\[\begin{equation} \hat{f}_{h, k}(x) = \frac{c_{k, d}}{nh^{d}} \sum_{i=1}^n k(\beta {\left\| \frac{x-x_i}{h} \right\|}^{2}) \end{equation}\]

where \(\beta < 0\). In the normal kernel case, \(\beta = -\frac{1}{2}\). In this way, the density gradient estimator becomes

\[\begin{equation} \begin{aligned} \hat{\bigtriangledown}f_{h, k}(x) & = \frac{2 \beta c_{k,d}}{nh^{d+2}} \sum_{i=1}^n (x-x_i)k^{'} \\ & = \frac{2 \beta c_{k,d}}{nh^{d+2}} \left( \sum_{i=1}^n x k^{'} - \sum_{i=1}^n x_i k^{'} \right) \\ & = \frac{-2 \beta c_{k,d}}{nh^{d+2}} \left( \sum_{i=1}^n k^{'} \right) \left( \frac{\sum_{i=1}^n x_i k^{'}}{ \sum_{i=1}^n k^{'}} - x \right) \end{aligned} \end{equation}\]

where

\[\begin{equation} k^{'} = k^{'}( \beta {\left\| \frac{ x-x_i}{h} \right\| }^2) = \partial \left( k( \beta {\left\| \frac{ x-x_i}{h} \right\| }^2) \right) / \partial \left( \beta {\left\| \frac{ x-x_i}{h} \right\| }^2 \right) \end{equation}\]

After assigning

\[\begin{equation} g(x) = -\text{sgn}(\beta)k^{'}(x) = k^{'}(x) \end{equation}\]

and

\[\begin{equation} y_i = \beta {\left\| \frac{ x-x_i}{h} \right\| }^2 \end{equation}\]

we have

\[\begin{equation} \begin{aligned} \hat{\bigtriangledown}f_{h, k}(x) & = \frac{-2 \beta c_{k,d}}{nh^{d+2}} \left[ \sum_{i=1}^n g(y_i) \right] \left[ \frac{\sum_{i=1}^n x_i g(y_i)}{\sum_{i=1}^n g(y_i)} - x \right] \\ & = \frac{c}{nh^{d+2}} \left[ \sum_{i=1}^n g(y_i) \right] m(x) \\ \end{aligned} \label{gradient} \end{equation}\]

where \(c > 0\) is the positive constant, and

\[\begin{equation} m(x) = \frac{\sum_{i=1}^n x_i g(y_i)}{\sum_{i=1}^n g(y_i)} - x \label{meanshift1} \end{equation}\]

is the mean shift, i.e., the difference between the weighted mean and \(x\) the center of the kernel. Therefore, we have

\[\begin{equation} m(x) = \frac{nh^{d+2}}{c} \frac{\hat{\bigtriangledown}f_{h, k}(x)}{\sum_{i=1}^n g(y_i)} \label{meanshift2} \end{equation}\]

2.2 The Mean Shift Procedure

The mean shift procedure aims to find the maxima location of \(\hat{f}_{h,k}(x)\). According to equation \(\eqref{meanshift2}\), since \(\frac{nh^{d+2}}{c} > 0\) and \(g(y_i) > 0\) for all \(i\), we have

\[\begin{equation} \hat{\bigtriangledown}f_{h, k}(x) = 0 \iff m(x) = 0 \end{equation}\]

Referring to equation \(\eqref{meanshift1}\), we define the weighted mean as \(w(x)\), using \(g\) for weights

\[\begin{equation} w(x) = \frac{\sum_{i=1}^n x_i g(y_i)}{\sum_{i=1}^n g(y_i)} \label{weightedmean} \end{equation}\]

Therefore,

\[\begin{equation} m(x) = w(x) - x \end{equation}\]

The mean shift procedure as follows:

for each instance \(x_i\), at step \(t \geq 0\):

1.1. computes weighted mean \(w(x_t)\) according to equation \(\eqref{weightedmean}\)

1.2 assign \(x_{t+1} = x_t + m(x_t) = w(x_t)\)
repeat the above step until converge or reach the max_iter

With the mean shift procedure, we can get the centroid for each data point \(x_i\), and the data points share the same centroid will be labeled as the same cluster. Therefore, mean shift can be applied as clustering.

2.3 Adaptive Gradient Ascent

In my understanding, we can explain the adaptive gradient ascent in this way. According to equation \(\eqref{meanshift2}\), we have

\[\begin{equation} \begin{aligned} m(x) & = \frac{nh^{d+2}}{c \sum_{i=1}^n g(y_i)} \hat{\bigtriangledown f_{h, k}}(x) \\ & = \gamma_{t} \hat{\bigtriangledown}f_{h, k}(x) \\ \end{aligned} \end{equation}\]

where

\[\begin{equation} \gamma_t = \frac{nh^{d+2}}{c \sum_{i=1}^n g(y_i)} \end{equation}\]

And

\[\begin{equation} \begin{aligned} x_{t+1} & = x_t + m(x_t) \\ & = x_t + \gamma_{t} \hat{\bigtriangledown}f_{h, k}(x) \\ \end{aligned} \end{equation}\]

During the process of \(x_t\) approaching to the optimal value \(x^*\), \(f_{h, k}(x)\) becoming bigger, that is, \(\sum_{i=1}^n k(y_i)\) becoming bigger. After applying the normal kernel to \(k\), it is easy to see

\[\begin{equation} \begin{aligned} g(y_i) & = k^{'}(y_i) = \partial k(y_i) / \partial y_i \\ & = \partial \exp(y_i) / y_i = \exp(y_i) \\ & = k(y_i) \end{aligned} \end{equation}\]

Therefore, the bigger \(f_{h, k}(x)\), the bigger \(\sum_{i=1}^n g(y_i) = \sum_{i=1}^n k(y_i)\) will be, the smaller \(\gamma_t\). In this case, if \(\gamma_t\) is small enough, the mean shift procedure is an adaptive gradient ascent.

3. A Numpy Python Implementation For Gaussian Mean Shift Clustering

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
# 1. import data
X, y = make_blobs(n_samples=500, centers=3, n_features=2, random_state=42)

# 2. find centroids
# update centroid using gaussian kernel
# $$g_i = np.exp(-\frac{1}{h^2}(x_t-x_i)^T(x_t-x_i))$$
# $$G = (g_1, ..., g_n)^T$$, h = bandwidth
def update_weighted_mean(X, x_weighted, bandwidth):
    diff = x_weighted - X
    diff_square = diff*diff
    diff_square_sum = np.sum(diff_square, axis=1)
    G = np.exp(-diff_square_sum/(2*np.square(bandwidth)))
    
    numerator = X.T.dot(G)
    denominator = np.sum(G)
    
    return numerator/denominator

# check if centroid already exist, if exist, return the cluster label
def has_mean(new_mean, means, tol):
    for idx, mean in enumerate(means):
        if np.allclose(new_mean, mean, atol=tol):
            return idx

# 3. the meanshift function
def meanshift(X, bandwidth, max_iter, tol, need_track, track_point_idx):
    it = 0
    means = None
    labels = []
    means_track = []
        
    for idx, x in enumerate(X):
        x_weighted = x
        while it < max_iter:
            x_weighted_new = update_weighted_mean(X, x_weighted, bandwidth)
            # stop iteration if converged
            if np.allclose(x_weighted, x_weighted_new):
                break
            else:
                x_weighted = x_weighted_new
            # only track the particular data point
            if need_track and idx == track_point_idx:
                means_track.append(x_weighted)
            it += 1
        if it == max_iter:
            print('case', idx, 'did not converged')
        if idx == 0:
            means = np.array([x_weighted])
            label = idx
        else:
            mean_idx = has_mean(x_weighted, means, tol)
            if mean_idx is None:
                means = np.vstack((means, x_weighted))
                label += 1
            else:
                label = mean_idx
        labels.append(label)
    return means, np.array(labels), np.array(means_track)

# 4. get results
# In this article, we use sklearn to estimate bandwidth
from sklearn.cluster import estimate_bandwidth
bandwidth = estimate_bandwidth(X)
# In this case, the weighted means converage slowly, so we set max_iter=10000
means, labels, means_track = meanshift(X, bandwidth, 10000, 1e-03, True, 0)
print(means.shape, means_track.shape)

# 5. show results with pyplot and compare with sklearn MeanShift
# 5.1 get sklearn MeanShift results
from sklearn.cluster import MeanShift
sk_ms = MeanShift().fit(X)

def plot_scatter(ax, X, labels, centroids, title):
    ax.scatter(X[:, 0], X[:, 1], c=labels, cmap='cividis')
    for centroid in centroids:
        ax.scatter(centroid[0], centroid[1], c='red')
    ax.set_title(title)

# plot the centroid moving track
def plot_track(ax, X_sub, means_track):
    start_point = X_sub[0]
    end_point= means_track[-1]
    dx = end_point[0] - start_point[0]
    dy = end_point[1] - start_point[1]
    ax.arrow(start_point[0], start_point[1], dx, dy,
             alpha=0.5, head_width=0.5, width=0.1, color='red', zorder=10)

fig, axes = plt.subplots(2, 2, figsize=(16, 10))
plot_scatter(axes[0, 0], X, y, [], 'Blobs')
plot_scatter(axes[0, 1], X, labels, means, 'Mean Shift')
plot_scatter(axes[1, 0], X, labels, means, 'Mean Shift Centroid Moving For First Cluster')
plot_scatter(axes[1, 1], X, sk_ms.labels_, sk_ms.cluster_centers_, 'Sklearn Mean Shift')

plot_track(axes[1, 0], X[labels==0], means_track)

4. References

Dorin Comaniciu, Peter Meer(2002). “Mean Shift: A Robust Approach toward Feature Space Analysis”.
José E. Chacón, Tarn Duong(2018). “Multivariate Kernel Smoothing and Its Applications”.
M. Herbin, N. Bonnet, and P. Vautrot(1996). “A Clustering Method based on the Estimation of the Probability Density Function and on the Skeleton by Influence Zones. Application to Image Processing”.

Affinity Propagation

Thu, 15 Oct 2020 00:00:00 +0000

1. Before Starting

If you are not familiar with Affinity Propagation, please see Wikipedia or the original paper “Clustering by Passing Messages Between Data Points(2007)” by Brendan J. Frey, Delbert Dueck.

2. Notions

Similarity \(s(i,k)\): the similarity between point \(k\) and point \(i\). It could be the negative Euclidean distance as follower:

\[\begin{equation} s(i,k) = - \|x_i - x_k \|^2, i \neq k \end{equation}\]

Preferences \(s(k, k)\): data points with larger values of \(s(k, k)\) are more likely to be chosen as exemplars. \(s(k,k)\) could be the median of the similarities(resulting in a moderate of clusters) or their minimum(resulting in a small number of clusters). In this article, we set each \(s(k, k)\) to the median as default.

\[\begin{equation} s(k, k) = median(S) \end{equation}\]

Responsibility \(r(i, k), i \rightarrow k\), send from point \(i\) to point \(k\), reflects the accumulated evidence that how well-suited point \(k\) is to serve as the exemplar for point \(i\). For candidate \(k\), it only needs to compete with the best other candidate.

\[\begin{equation} r(i, k) \gets s(i, k) - \underset{k' \neq k}{\operatorname{max}}{\{a(i, k') + s(i, k')\}} \label{eqres} \end{equation}\]

Self-responsibility \(r(k, k)\), reflects accumulated evidence that point \(k\) is an exemplar. \(r(k,k) \leq 0\) means it is better for point \(k\) not to be an exemplar.

\[\begin{equation} r(k, k) = s(k, k) - \underset{k' \neq k}{\operatorname{max}}{\{a(k, k') + s(k, k')\}} \end{equation}\]

Availability \(a(i, k), k \rightarrow i\), send from point \(k\) to point \(i\), reflects the accumulated evidence for how appropriate it would be for point \(i\) to choose point \(k\) as its exemplar.

\[\begin{equation} a(i, k) \gets min\{0, r(k, k)+\sum_{i' \notin \{i, k\}}max\{0, r(i', k)\}\} \label{eqava} \end{equation}\]

Self-availability \(a(k, k)\), reflects accumulated evidence that point \(k\) is an exemplar, based on the positive responsibilities sent to \(k\) from other points.

\[\begin{equation} a(k,k) \gets \sum_{i' \neq k} max\{0, r(i', k)\} \end{equation}\]

Criterion \(c(i, k)\), is the combination of the two messages, used to identify exemplars.

\[\begin{equation} c(i, k) \gets r(i, k) + a(i, k) \label{eqcri} \end{equation}\]

Damping factor \(\lambda\), used to smooth the message-updating procedure. At each iteration step \(t+1(t \geq 0)\),

\[\begin{equation} r_{t+1}(i, k) = \lambda \cdot r_{t}(i, k) + (1-\lambda) \cdot r_{t+1}(i, k) \\ a_{t+1}(i, k) = \lambda\cdot a_{t}(i, k) + (1-\lambda) \cdot a_{t+1}(i, k) \end{equation}\]

Best exemplar \(k^*\), for each point \(i\), the exemplar will be

\[\begin{equation} k^* = \underset{k}{\operatorname{argmax}} c(i, k) \end{equation}\]

3. How To Understand?

For \(r(i, k)\), we can roughly see it as the indicator measuring how appropriate for point \(i\) to choose candidate \(k\) as its exemplar compared with other candidates. Larger similarity, other candidate’s smaller availability, and other candidate’s smaller responsibility will bring on a larger responsibility for candidate \(k\).

As for \(a(i, k)\), we can roughly see it as the candidate \(k\) promotes itself to point \(i\) using the sum of the self-responsibility and the positive responsibilities received from other points. Since a particular candidate doesn’t need to serve the whole data set, it doesn’t matter how bad for a candidate be the exemplar for some points. Therefore, AP only calculates the positive responsibilities. And also, to limit the really large responsibilities from some points or even a single point, AP sets an upper limits(which is 0) to \(a(i, k)\), makes \(a(i, k) \leq 0\). What’s more, a negative \(a(i, k)\) can insure a candidate with largest similarity receives a positive responsibility.

For damping factor \(\lambda\), it is used to avoid numerical oscillations. It is import to damping the responsibility matrix \(R\) first, then using the damped \(R\) to update the availability matrix \(A\).

4. When To Converge And How To Find Exemplars?

It will be converged when the exemplars stay constant for some number of iteration, or the changes in the messages fall below a threshold. In this article, we use the first one. When is converged, AP use the criterion matrix \(C\) to find the exemplars. In each row \(i\) of the criterion matrix \(C\), the point with the max value(e.g. point \(k\)) is either the exemplar for point i(if k != i) or is the exemplar(if k == i).

5. A Numpy Python Implementation

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
from sklearn.metrics.pairwise import euclidean_distances
import copy

# 1. import data
X, y = make_blobs(n_samples=100)
X.shape, y.shape

# 2. calculate Similarity Matrix, equation (1), (2)
def cal_Similarity(X, preference):
    S = -euclidean_distances(X, X)
    if preference == 'min':
        preference = np.min(S)
    else:
        preference = np.median(S)
    np.fill_diagonal(S, preference)
    return S

# 3. calculate Responsity Matrix, equation (3)
# 3.1 find max{a_ik'+s_ik'}, k' != k
def cal_largest_criterion_except_k(S, A):
    n = S.shape[0]
    row_index = np.arange(n)
    
    C = A + S
    max_index_each_row = np.argmax(C, axis=1)
    
    C_copy = copy.deepcopy(C)
    # replace the max value to -np.inf to find the second max value
    C_copy[row_index, max_index_each_row] = -np.inf
    second_max_index_each_row = np.argmax(C_copy, axis=1)

    # get the max value for each row
    max_value_each_row = C[row_index, max_index_each_row]
    # get the second max value for each row
    second_max_value_each_row = C[row_index, second_max_index_each_row]
    
    R_to_substract = np.zeros(shape=(n, n))
    R_to_substract[row_index] = max_value_each_row.reshape(-1, 1)
    R_to_substract[row_index, max_index_each_row] = second_max_value_each_row
    return R_to_substract

def cal_Responsity(S, A):
    R_to_substract = cal_largest_criterion_except_k(S, A)
    return S - R_to_substract

# 4. cal Availability Matrix A, equation (5),(6)
def cal_Availability(S, R):
    n = S.shape[0]
    R_dia = R.diagonal()
    
    R_ramp = np.maximum(R, 0)
    R_ramp_dia = R_ramp.diagonal()
    R_ramp_sum = np.sum(R_ramp, axis=0)
    
    # a(i, k) = min{0, R(k, k) + R_ramp_sum[k] - R_ramp(i, k) - R_ramp(k, k)}
    A = -R_ramp - R_ramp_dia + R_dia + R_ramp_sum
    A = np.minimum(A, 0)
    
    # fill the diagonal with a(k, k) = sum_{i'!=k} max{0, r(i', k)}
    R_ramp_copy = copy.deepcopy(R_ramp)
    # fill diagonal with 0 to exculde
    np.fill_diagonal(R_ramp_copy, 0)
    A_dia = np.sum(R_ramp_copy, axis=0)
    
    np.fill_diagonal(A, A_dia)
    return A

# 5. and 6. Find exemplars and labels
# for each the max value of c(i, k)
# either identifies point i as an exemplar if k = i,
# or identifies the data point that is the exemplar for point i
# 5. find exemplars
    # also, exemplars equal to the points whoese diagonal value > 0, that is
    # exemplars = row_index[C.diagonal() > 0]
def cal_exemplars(C):
    n = C.shape[0]
    row_index = np.arange(n)
    max_index_each_row = np.argmax(C, axis=1)
    exemplars = row_index[row_index == max_index_each_row]
    return exemplars

# 6. find labels
def cal_labels(C_res, exemplars_res):
    n = C_res.shape[0]
    row_index = np.arange(n)
    max_index_each_row = np.argmax(C_res, axis=1)
    k = exemplars_res.shape[0]
    labels = np.zeros(shape=(n, ))
    for i in range(k):
        exemplar = exemplars_res[i]
        cluster = row_index[max_index_each_row == exemplar]
        labels[cluster] = i
    return labels

# 7. Put together
    # import note: use the damped R to update A!!!
    # the paper didn't mention that
def ap(X, preference, damping, max_iter, stop_iter):
    n = X.shape[0]
    # 1. calculate Similarity
    S = cal_Similarity(X, preference)
    # 2. init Availabiliy, Responsibility to zeros
    A = np.zeros(shape=(n, n))
    R = np.zeros(shape=(n,n))
    C = R + A
    exemplars = cal_exemplars(C)
    accum_count = 0
    left_iter = max_iter
    while left_iter > 0:
        # 3. calculate Responsity, Availability
        R_updated = cal_Responsity(S, A)
        # Damping First! Very Important! Or May Not Converge
        R = damping*R + (1-damping)*R_updated
        
        A_updated = cal_Availability(S, R)
        A = damping*A + (1-damping)*A_updated
        
        C_updated = R + A
        exemplars_updated = cal_exemplars(C_updated)
        
        num = exemplars.shape[0]
        num_updated = exemplars_updated.shape[0]
        
        if num > 0 and (num == num_updated) and np.allclose(exemplars, exemplars_updated):
            accum_count += 1
            if accum_count == stop_iter:
                labels = cal_labels(C_updated, exemplars_updated)
                print('Converge at iter', max_iter - left_iter)
                return labels, exemplars_updated
        exemplars = exemplars_updated
        C = C_updated
        left_iter -= 1
    print('Not converge.')
    return np.array([-1]*n), []

# 8. Compare and Show Results
# 8.1 with median preference
labels_median, cluster_centers_median = ap(X, 'median', 0.5, 100, 20)

# 8.2 with min preference
labels_min, cluster_centers_min = ap(X, 'min', 0.5, 100, 20)

# 8.3 Sklearn
from sklearn.cluster import AffinityPropagation
ap_sklearn = AffinityPropagation(random_state=42).fit(X)
labels_sklearn = ap_sklearn.labels_
cluster_centers_sklearn = ap_sklearn.cluster_centers_indices_

# 8.4 show results
def plot_scatter(ax, X, labels, title):
    ax.scatter(X[:, 0], X[:, 1], c=labels)
    ax.set_title(title)

fig, axes = plt.subplots(2, 2, figsize=(16, 10))
plot_scatter(axes[0, 0], X, y, 'Blobs')
plot_scatter(axes[0, 1], X, labels_median, 'AP-median-preference')
plot_scatter(axes[1, 0], X, labels_min, 'AP-min-preference')
plot_scatter(axes[1, 1], X, labels_sklearn, 'AP-sklearn')

6. References

Brendan J. Frey, Delbert Dueck(2007). “Clustering by Passing Messages Between Data Points”.

A Simple Note for Gaussian Mixture Models

Thu, 01 Oct 2020 00:00:00 +0000

1. Gaussian Mixture Model

A Gaussian mixture is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters.

scikit-learn.org

Gaussian mixture model(GMM) is an clustering algorithm similar to k-means. Unlike k-means, GMM is a soft clustering, each instance associates with a cluster with a probability. Formally, given a set of observations \(X = (x_1, x_2, ..., x_n)\), \(x_i \in R^{d}\), and \(i=1, 2, ..., n\), Gaussian mixture model assumes:

there are \(K\) mixture components, component value denoted as \(k \in \{1, 2, ..., K\}\)
\(x_i\) comes from component \(z_i\), \(Z = (z_1, z_2, ..., z_n)\) is the latent variables
\(\begin{equation} P(z_i=k) = \pi_k \end{equation}\), s.t. \(\sum_{k=1}^K \pi_k = 1\), \(\pi = (\pi_1, \pi_2, ...\pi_K)\) is the mixing proportions
\(\begin{equation} P(x_i=x | z_i=k) \sim N(\mu_k, \Sigma_k) \end{equation}\), \(N(\mu_k, \Sigma_k)\) is a multivariate Gaussian distribution, the density function is

\[\begin{equation} f(x_i) = \frac{1}{\sqrt{(2 \pi)^d \det \Sigma_k}} \exp\left( -\frac{1}{2} (x_i - \mu_k)^T \Sigma_k^{-1} (x_i - \mu_k) \right), \end{equation}\]

where \(\Sigma_k\) is the \(d \times d\) covariance matrix for component \(k\). The objective of GMM is to estimate the probabilities that each instance belongs to each component, that is

\[\begin{equation} P(z_i=k | x_i), k = 1, 2, ... K, i=1, 2, ... n \label{obj} \end{equation}\]

2. Bayes and Maximum Likelihood Estimation

With Bayes rule, we have

\[\begin{equation} \begin{aligned} P(z_i=k|x_i) & = \frac{P(x_i|z_i)P(z_i=k)}{P(x_i)} \\ & = \frac{P(x_i|z_i)P(z_i=k)}{\sum_{k=1}^KP(x_i|z_i)P(z_i=k)} \\ & = \frac{\pi_k*N(x_i, \mu_k, \Sigma_k)}{\sum_{k=1}^K\pi_k*N(x_i, \mu_k, \Sigma_k)} \\ & = w_{ik} \end{aligned} \label{weights} \end{equation}\]

If we know parameters \(\theta = (\pi_1, \pi_2, ..., \pi_K, \mu_1, \mu_2, ..., \mu_K, \Sigma_1, \Sigma_2, ..., \Sigma_K)\), we will get the objective. Therefore, the target becomes to estimate the unknown parameter \(\theta\). To estimate \(\theta\), GMM uses maximum likelihood estimation(MLE),

\[\begin{equation} \begin{aligned} L(\theta|X, Z) & = P(X, Z|\theta) \\ & = \prod_{i=1}^n \sum_{k=1}^K P(x_i=x|z_i=k)P(z_i=k) \\ & = \prod_{i=1}^n \sum_{k=1}^K \pi_k N(\mu_k, \Sigma_k) \end{aligned} \end{equation}\]

log-likelihood becomes

\[\begin{equation} \begin{aligned} l(\theta|X, Z) & = \sum_{i=1}^n log(\,\sum_{k=1}^K \pi_k N(\mu_k, \Sigma_k))\, \end{aligned} \end{equation}\]

Normally, we can get

\[\begin{equation} \theta^{*} = \underset{\theta}{\operatorname{argmax}} l(\theta|X, Z) \end{equation}\]

by set

\[\begin{equation} \frac{\partial l}{\partial \theta} = 0 \end{equation}\]

However, it’s hard to solve \(\theta\) with closed-form expressions now. So GMM uses the Expectation—maximization algorithm(EM).

3. Expectation—Maximization Algorithm

In statistics, an expectation–maximization (EM) algorithm is an iterative method to find (local) maximum likelihood or maximum a posteriori (MAP) estimates of parameters in statistical models, where the model depends on unobserved latent variables.

wiki

EM iterates the expectation step(E-step) and the maximization step(M-step) until converge:

initializes \(\theta = (\pi_1, \pi_2, ..., \pi_K, \mu_1, \mu_2, ..., \mu_K, \Sigma_1, \Sigma_2, ..., \Sigma_K)\) to \(\theta_0\)
start iteration until convergence or reach the max iterate number, at step \(t\geq0\):

2.1 E-step: with \(\theta_{t}\), calculates \(\begin{equation} P(z_i=k|x_i) \end{equation}\) through equation \(\eqref{weights}\) for \(k=1, 2, ..., K\) and \(i=1, 2, ..., n\)

2.2 M-step: estimates \(\theta_{t+1}\) by Maximum Likelihood Estimation

In E-step, EM defines the expected value of the log-likelihood function of \(\theta\) as

\[\begin{equation} \mathbf{Q}(\theta|\theta_{t}) = \mathbf{E}_{Z|X, \theta^t}(logL(\theta;X,Z)) \end{equation}\]

In M-step, EM estimates next \(\theta\) as

\[\begin{equation} \theta^{t+1} = \underset{\theta}{\operatorname{argmax}} \mathbf{Q}(\theta|\theta_{t}) \quad s.t. \ \sum_{k=1}^K \pi_k = 1 \end{equation}\]

4. GMM with EM

In GMM situation, we have

\[\begin{equation} \begin{aligned} L(\theta;x_i, z_i) & = \prod_{k=1}^K I(z_i=k) P(x_i, z_i|\theta) \\ & = \prod_{k=1}^K I(z_i=k) P(x_i|z_i, \theta) P(z_i | \theta) \\ & = \prod_{k=1}^K I(z_i=k) N(x_i|\mu_k, \Sigma_k) \pi_k \end{aligned} \end{equation}\]

where \(I(z_i=k)\) is a indicator function. And

\[\begin{equation} L(\theta|X, Z) = \prod_{i=1}^n L(\theta;x_i, z_i) \end{equation}\]

In E-step, the expected value of the log-likelihood function becomes

\[\begin{equation} \begin{aligned} \mathbf{Q}(\theta|\theta_{t}) & = \mathbf{E}_{Z|X, \theta^t}(logL(\theta;X,Z)) \\ & = \mathbf{E}_{Z|X, \theta^t}\sum_{i=1}^nlog(L(\theta;x_i, z_i)) \\ & = \sum_{i=1}^n \mathbf{E}_{z_i|x_i, \theta^t} log(L(\theta;x_i, z_i)) \\ & = \sum_{i=1}^n \mathbf{E}_{z_i|x_i, \theta^t} \sum_{k=1}^K I(z_i=k) *(log\pi_k + logN(x_i|\mu_k, \Sigma_k)) \\ & = \sum_{i=1}^n \sum_{k=1}^K P(z_i=k|x_i) * (log\pi_k + logN(x_i|\mu_k, \Sigma_k)) \end{aligned} \end{equation}\]

Combine the current \(\theta_{t}\) and equation \(\eqref{weights}\), we have

\[\begin{equation} w_{ik} = P(z_i=k|x_i) \end{equation}\]

where \(w_{ik}\) is a constant. Thus

\[\begin{equation} \mathbf{Q}(\theta|\theta_{t}) = \sum_{i=1}^n \sum_{k=1}^K w_{ik} * (log\pi_k + logN(x_i|\mu_k, \Sigma_k)) \end{equation}\]

In M-step, the optimal problem becomes

\[\begin{equation} \begin{aligned} \underset{\theta}{\operatorname{max}} \quad & \mathbf{Q}(\theta|\theta_{t}) \\ \text{s.t.} \quad & \sum_{k=1}^K \pi_k = 1 \end{aligned} \end{equation}\]

which is a Lagrangian problem. Therefore,

\[\begin{equation} \begin{aligned} H(\theta, \lambda) = \mathbf{Q}(\theta|\theta_{t}) + \lambda(\sum_{k=1}^K \pi_k - 1) \end{aligned} \end{equation}\]

Take partial derivatives, we have

\[\begin{equation} \begin{aligned} \frac{\partial H}{\partial \mu_k} & = \sum_{i=1}^n w_{ik} * \frac{\partial logN(x_i|\mu_k, \Sigma_k)}{\partial \mu_k} \\ & = \sum_{i=1}^n w_{ik} \Sigma_{k}^{-1}(x_i - \mu_k) \end{aligned} \label{partialmu} \end{equation}\]

and

\[\begin{equation} \frac{\partial H}{\partial \Sigma_k} = \frac{1}{2}\{ \sum_{i=1}^n w_{ik} \Sigma_{k}^{-T} (x_i - \mu_k)(x_i-\mu_k)^{T}(\Sigma^{-T} - I) \} \label{partialSigma} \end{equation}\]

and

\[\begin{equation} \frac{\partial H}{\partial \pi_k} = \sum_{i=1}^n w_{ik} / \pi_k + \lambda \label{pi_k} \end{equation}\]

and

\[\begin{equation} \frac{\partial H}{\partial \lambda} = \sum_{k=1}^K \pi_k - 1 \label{lambda} \end{equation}\]

Set \(\eqref{partialmu}\) equal to zero, we have

\[\begin{equation} \mu_k^* = \frac{\sum_{i=1}^n w_{ik}x_i}{\sum_{i=1}^n w_{ik}} \end{equation}\]

Set \(\eqref{partialSigma}\) equal to zero, and combine with \(\mu_k^*\), we have

\[\begin{equation} \Sigma_k^* = \frac{\sum_{i=1}^n w_{ik}(x_i-\mu_k)(x_i-\mu_k)^{T} }{\sum_{i=1}^n w_{ik}} \end{equation}\]

Set \(\eqref{pi_k}\) and \(\eqref{lambda}\) equal to zero, we have

\[\begin{equation} \pi_k^* = \frac{\sum_{i=1}^n w_{ik}}{\sum_{k=1}^K \sum_{i=1}^n w_{ik}} \end{equation}\]

Therefore, we got \(\theta_{t+1} = (\mu_1^*, \mu_2^*, ... \mu_K^*, \Sigma_1^*, \Sigma_2^*, ... \Sigma_K^*, \pi_1^*, \pi_2^*, ... \pi_K^*)\).

5. A numpy python implementation

First, we need to initialize \(\theta_0\). Here, we use kmeans++, kmeans and random to initialize parameters. After initialization, we calculate the weights equation \(\eqref{weights}\), which is the E-step.

# 0. import
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
from scipy.stats import multivariate_normal

X, y = make_blobs()

# 1. EM: Initlization and E-step

def _kmeans_init(X, n_component, kmeans_method):
    n_samples = X.shape[0]
    if kmeans_method == 'kmeans++':
        kmeans = KMeans(n_clusters=n_components).fit(X)
    else:
        kmeans = KMeans(n_clusters=n_components, init='random').fit(X)
    labels = kmeans.labels_
    
    weight_matrix = None
    
    for i in range(n_component):
        sub_component = X[labels==i]
        
        pi = sub_component.shape[0]/n_samples
        mean = np.mean(sub_component, axis=0)
        cov = np.cov(sub_component.T)
        
        pdf_value = multivariate_normal.pdf(X, mean, cov)
        weight = pi*pdf_value
        
        if weight_matrix is None:
            weight_matrix = weight
        else:
            weight_matrix = np.column_stack((weight_matrix, weight))
        
    weight_sum = np.sum(weight_matrix, axis=1)[:, np.newaxis]
    weights = weight_matrix/weight_sum
    return weights

def _initialize(X, n_component, method):
    # default random
    n_samples = X.shape[0]
    if method == 'kmeans++':
        weights = _kmeans_init(X, n_component, 'kmeans++')
    elif method == 'kmeans_random':
        weights = _kmeans_init(X, n_component, 'kmeans_random')
    else:
        weights = np.random.rand(n_samples, n_components)
        w = weights.sum(axis=1)
        weight_sum = weights.sum(axis=1)[:, np.newaxis]
        weights /= weight_sum
        
    return weights

Given the value of weights equation \(\eqref{weights}\), we update the value of \(\theta\) through MLE, which is the M-step. To avoid forloop, convert the numerator of equation \(\eqref{partialSigma}\) as:

\[\begin{equation} \begin{aligned} A & = \sum_{i=1}^n w_{ik}(x_i - \mu_k)(x_i - \mu_k)^T \\ \Rightarrow \quad A_{st} &= \sum_{i=1}^n w_{ik}x_{is}x_{it} - \sum_{i=1}^nw_{ik}\mu_{ks}x_{it} - \sum_{i=1}^nw_{ik}\mu_{kt}x_{is} + \sum_{i=1}^nw_{ik}\mu_{ks}\mu_{kt} \\ \Rightarrow \quad A_{st} & = A1_{st} - A2_{st} - A3_{st} + A4_{st} \end{aligned} \end{equation}\]

for \(1 \leq s, t \leq d\), where

\[\begin{equation} \begin{aligned} A1 & = w_k*(X.T) \\ A2 & = A3.T \\ A3 & = \mu_k((w_k*(X.T)).dot(H)) \\ A4 & = (\sum_{i=1}^n w_{ik})*(\mu_k.dot(\mu_k.T)) \\ w_k & = np.array([w_k1, w_k2, ... w_kn]) \\ \mu_k & = np.array([\mu_k1, \mu_k2, ..., \mu_kd]) \\ H & = np.ones(shape=(n, d)) \end{aligned} \end{equation}\]

# 2. EM: M-step

def _update_means(X, weights):
    weight_sum = np.sum(weights, axis=0)
    means = X.T.dot(weights)/weight_sum
    return means

def _update_covs(X, n_components, weights, means):
    n, m = X.shape
    covs = None
    H = np.ones(shape=(n, m))
    
    for k in range(n_components):
        weight = weights[:, k]
        weight_sum = np.sum(weight)
        mean = means[:, k]
        weighted_X_T = weight*(X.T)
        
        a1 = (weighted_X_T).dot(X)
        a3 = mean*(weighted_X_T.dot(H))
        a2 = a3.T
        mean_reshaped = mean.reshape(-1, 1)
        a4 = weight_sum*((mean_reshaped).dot(mean_reshaped.T))
        a = a1 - a2 - a3 + a4
        
        cov = a/weight_sum
        if covs is None:
            covs = cov
        else:
            covs = np.dstack((covs, cov))
    return covs

def _update_pis(weights):
    w_each_component = np.sum(weights, axis=0)
    w_sum = np.sum(w_each_component)
    pis = w_each_component/w_sum
    return pis

def _update_weights(X, n_components, means, covs, pis):
    weighted_pdf_array = None
    for k in range(n_components):
        mean = means[:, k]
        cov = covs[:,:,k]
        pi = pis[k]
        pdf_values = multivariate_normal.pdf(X, mean, cov)
        weighted_pdf_values = pi*pdf_values
        if weighted_pdf_array is None:
            weighted_pdf_array = weighted_pdf_values
        else:
            weighted_pdf_array = np.column_stack((weighted_pdf_array, weighted_pdf_values))
    
    weighted_sum = np.sum(weighted_pdf_array, axis=1)[:, np.newaxis]
    weights = weighted_pdf_array/weighted_sum
    return weights

# 3. gmm fit
def gmm_fit(X, n_components, init_method, max_iter=100):
    weights = _initialize(X, n_components, init_method)
    iter_left = max_iter
    while iter_left > 0:
        means = _update_means(X, weights)
        covs = _update_covs(X, n_components, weights, means)
        pis = _update_pis(weights)
        weights_next = _update_weights(X, n_components, means, covs, pis)
        
        if np.allclose(weights, weights_next):
            print('converage at step', max_iter-iter_left)
            break
        else:
            weights = weights_next
            iter_left -= 1
    return means, covs, weights_next

# 4. predict labels
def gmm_predict(X, n_components, means, covs, weights):
    pdf_values = None
    for k in range(n_components):
        mean = means[:, k]
        cov = covs[:, :, k]
        pdf_value = multivariate_normal.pdf(X, mean, cov)
        if pdf_values is None:
            pdf_values = pdf_value
        else:
            pdf_values = np.column_stack((pdf_values, pdf_value))
    labels = np.argmax(pdf_values, axis=1)
    return labels

# 5. gmm
def gmm_fit_predict(X, n_components, init_method, max_iter):
    means, covs, weights = gmm_fit(X, n_components, init_method, max_iter)
    labels = gmm_predict(X, n_components, means, covs, weights)
    return labels

# 6. result and demonstration
def plot_scatter(ax, X, labels, title):
    ax.scatter(X[:, 0], X[:, 1], c=labels)
    ax.set_title(title)

n_components = 3
gmm_labels_kmeans_plus_plus = gmm_fit_predict(X, n_components, 'kmeans++', max_iter=1000)
gmm_labels_kmeans_random = gmm_fit_predict(X, n_components, 'kmeans_random', max_iter=1000)
gmm_labels_random = gmm_fit_predict(X, n_components, 'random', max_iter=1000)

fig, axes = plt.subplots(2, 2, figsize=(16, 10))
plot_scatter(axes[0, 0], X, y, 'Blobs')
plot_scatter(axes[0, 1], X, gmm_labels_kmeans_plus_plus, 'kmeans++')
plot_scatter(axes[1, 0], X, gmm_labels_kmeans_random, 'kmeans random')
plot_scatter(axes[1, 1], X, gmm_labels_random, 'random')

A Simple Note for k-means clustering

Mon, 14 Sep 2020 00:00:00 +0000

1. What is k-means clustering

In short, K-means is an unsupervised clustering algorithm for partitioning a d-dimensional population into \(k\) sets. Formally, given a set of observations \(X = (x_1, x_2, ..., x_n), x_i \in R^{d}, i=1, 2, ..., n\), each observation is a d-dimensional vector, k-means

initializes \(k\) centroids denoted as \(C = \{c_1, c_2,... c_k\}\) to represent \(k\) clusters, denoted as \(S = \{S_1, S_2, ..., S_k\}\)
in each step \(t \geq 1\),

2.1 for each \(x_i\), finds the first nearest centroid, denoted as \(c^*_{t, i} = \underset{c_{t,j}}{\operatorname{argmin}} {\| x_i - c_{t,j} \|}^2, j=1, 2,..., k\)

2.2 expectation step: adds \(x_i\) into the cluster represented by \(c^*_{t, i}\), \(S\) updated

2.3 maximization step: assigns \(c_{t+1, j} = \frac{1}{ \vert S_{t, j} \vert } \sum_\limits{x_i \in S_{t, j}} x_i, j=1, 2, ..., k\), \(C\) updated.
iterate step \(t\) until converged or reach the max iterate number

The objective is to minimize the within-cluster sum of squares(WCSS),

\[\begin{equation} \underset{C}{\operatorname{argmin}} \sum_{j=1}^k \sum_{x \in S_j} {\| x - c_j \|}^{2} \end{equation}\]

which is an NP-hard problem. Kmeans does not guarantee convergence to the global optimum.

2. Initialization

Centroids initialization plays an important part in kmeans. The result may rely on the initial centroids badly. One simply method is to run kmeans multiple times. There are several initialization methods. The commonly used methods are Forgy random, random partition and k-means++.

Forgy random uniformly randomly selects \(k\) instances from the population as centroids.
Random partition randomly partition the population into \(k\) clusters.
K-means++ weights the probability for a data point to be chosen as center according to it’s distance from the closest already chosen centers, unlike uniformly random.

2.1 k-means++

find first center: uniformly randomly choose an instance as center from \(X\), denoted as \(c_1\), makes the already chosen centers as \(C=\{c_1\}\)
find the next \(k-1\) centers: repeat the below steps \(k-1\) times

2.1 for each instance \(x_i\), find the first nearest center, denoted as \(c_i\), \(c_i \in C\)

2.2 calculates the square of the distance between \(x_i\) and \(c_i\), denoted as \(d_{i}^2 = \| x_i - c_i \|^2\)

2.3 calculates the probability density function as \(P(x_i) = {d_i^2}/{\sum_\limits{i=1}^n{d_i^2}}\)

2.4 chooses center randomly with the above pdf, denoted as \(c_{next}\), updates \(C= C + \{c_{next}\}\)

3. A numpy python implementation

import numpy as np
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt

# 1. create samples
X, y = make_blobs()

# 2. centroids initialization
def _cal_min_distances(X, centers):
    distances = np.sum(np.square(X[:, np.newaxis, :] - centers), axis=2)
    min_distances = np.min(distances, axis=1)
    return min_distances

def _Lloyd_centroids_init(X, k):
    N = X.shape[0]
    idx = np.random.choice(N, size=k, replace=False)
    return X[idx]

def _kmeans_plus_plus(X, k):
    N = X.shape[0]
    
    first_center = X[np.random.choice(N)]
    centers = [first_center]
    
    for i in range(k-1):
        distances_min = _cal_min_distances(X, np.array(centers))
        pdf = distances_min/np.sum(distances_min)
        
        # with pdf like this,
        # the already chosen instances will be excluded, and
        # the instances far away from centers will have higher probablities to be chosen
        new_center_idx = np.random.choice(N, p=pdf)
        new_center = X[new_center_idx]
        centers.append(new_center)
        
    return centers

def centroids_init(X, k, method):
    # defalut method 'kmeans++'
    if method == 'random':
        return _Lloyd_centroids_init(X, k)
    return _kmeans_plus_plus(X, k)

# 3. utils functions
def _find_first_nearest_centroid(X, centroids):
    distances = np.sum(np.square(X[:, np.newaxis, :] - centroids), axis=2)
    first_nearest_centroid_idx = np.argmin(distances, axis=1)
    return first_nearest_centroid_idx

def _ndarray(arrs):
    return np.array(arrs, dtype=object)

# 4. kmeans
def kmeans(X, k, method, max_iter=100):
    centroids = centroids_init(X, k, method)
    iterate_left = max_iter
    while iterate_left > 0:
        centroid_idx = _find_first_nearest_centroid(X, centroids)
        
        clusters = [X[centroid_idx==i] for i in range(k)]
        new_centroids = [np.mean(clusters[i], axis=0) for i in range(k)]
        
        if np.allclose(centroids, new_centroids):
            print('converage at step', max_iter-iterate_left)
            return _ndarray(clusters), _ndarray(new_centroids), centroid_idx
        else:
            centroids = new_centroids
            iterate_left -= 1
    return _ndarray(clusters), _ndarray(new_centroids), centroid_idx

# 5. result and demonstration
clusters_plus_plus, centroids_plus_plus, labels_plus_plus = kmeans(X, 3, 'kmeans++', 100)
clusters_plus_plus2, centroids_plus_plus2, labels_plus_plus2 = kmeans(X, 3, 'kmeans++', 100)
clusters_plus_forgy, centroids_forgy, labels_forgy = kmeans(X, 3, 'random', 100)

# plot
def plot_scatter(ax, X, labels, title):
    ax.scatter(X[:, 0], X[:, 1], c=labels)
    ax.set_title(title)


fig, axes = plt.subplots(2, 2, figsize=(16, 10))
plot_scatter(axes[0, 0], X, y, 'Blobs')
plot_scatter(axes[0, 1], X, labels_plus_plus, 'kmeans++')
plot_scatter(axes[1, 0], X, labels_plus_plus2, 'kmeans++2')
plot_scatter(axes[1, 1], X, labels_forgy, 'Forgy Random')
# 

How to Select Data In Pandas

Wed, 24 Jun 2020 00:00:00 +0000

In this article, we will talk about different ways of selecting data in Pandas, including selecting by label, selecting by position, selecting by bool condition, selecting by value, and selecting by dtypes. Among these, using .loc[] and .iloc[] through labels and indices are the most common ways.

1. Prepare Data

# Series
>>> data_s = np.arange(5)
>>> index_s = list('abcde')
>>> s = pd.Series(data_s, index=index_s)
# DataFrame
>>> data_df = np.arange(20).reshape(5, 4)
>>> columns_df = list('ABCD')
>>> index_df = list('abcde')
>>> df = pd.DataFrame(data_df, index=index_df, columns=columns_df)
>>> s
a    0
b    1
c    2
d    3
e    4
dtype: int64
>>> df
	A	B	C	D
a	0	1	2	3
b	4	5	6	7
c	8	9	10	11
d	12	13	14	15
e	16	17	18	19
# multi-dtypes DataFrame
>>> df_types = pd.DataFrame({
    'int': ['a', 'b', 'c'],
    'number': [1, 2, 3],
    'bool': [True, False, True],
    'datetime': pd.to_datetime(['20200101', '20200102', '20200103']),
    'cat': pd.Categorical(['a', 'b', 'c']),
    'timedelta': [pd.Timedelta(1, unit='d'), pd.Timedelta(2, unit='d'), pd.Timedelta(3, unit='d')]
})
>>> df_types.dtypes
str                   object
int                    int64
bool                    bool
datetime      datetime64[ns]
cat                 category
timedelta    timedelta64[ns]
dtype: object
# Datatime DataFrame
>>> ts_index = pd.date_range('2020-01-01', periods=30, freq='4h30min')
>>> ts = pd.DataFrame({'A': np.arange(0, 30), 'B': np.arange(30, 60)}, index=ts_index)

2. Selecting Data By Label

In Pandas, we mainly use [], .loc, .xs, .at, .get, and .lookup to select data by label. Basically, we use

a single label
a list of labels
a slice of labels

as selecting parameters. Raise KeyError if a label does not exist in DataFrame or Series. In this section, we will show how to select data by label with some examples.

2.1 `[]`

DataFrame[] and Series[] support a single label and a list of labels. DataFrame[] also supports a slice of row labels, both the start and the stop are included.

# 1. a single label
>>> s['a']
0
>>> df['A']
a     0
b     4
c     8
d    12
e    16
Name: A, dtype: int64
# 2. a list of labels
>>> s[['a', 'b']]
a    0
b    1
dtype: int64
>>> df[['A', 'B']]
	A	B
a	0	1
b	4	5
c	8	9
d	12	13
e	16	17
# 3. a slice of row labels
>>> df['a':'c']
	A	B	C	D
a	0	1	2	3
b	4	5	6	7
c	8	9	10	11

2.2 `.loc[]`

Series.loc and DataFrame.loc:

Access a group of rows and columns by label(s) or a boolean array. Axes left out of the specification are assumed to be :.

Support a single label, a list of labels and a slice object with labels.

# 1. a single label
# equals to df.loc['a', :]
>>> df.loc['a']
A    0
B    1
C    2
D    3
Name: a, dtype: int64
>>> df.loc['a', 'A']
0
>>> s.loc['a']
0
# 2. a list or array of labels
# equals to df.loc[['a', 'b'], :]
>>> df.loc[['a', 'b']]
	A	B	C	D
a	0	1	2	3
b	4	5	6	7
>>> df.loc[['a', 'b'], ['A', 'B']]
	A	B
a	0	1
b	4	5
>>> s.loc[['a', 'b']]
a    0
b    1
dtype: int64
# 3. a slice object with labels, both the start and the stop are included
# equals to df.loc['a': 'c', :]
>>> df.loc['a': 'c']
	A	B	C	D
a	0	1	2	3
b	4	5	6	7
c	8	9	10	11
>>> df.loc['a': 'c', 'A':'C']
	A	B	C
a	0	1	2
b	4	5	6
c	8	9	10
>>> df.loc[['a', 'b'], ['A', 'B']]
	A	B
a	0	1
b	4	5
>>> s.loc['a':'b']
a    0
b    1
dtype: int64

2.3 `.get()`

DataFrame.get(self, key, default=None):

Get item from object for given key. Returns default value if not found.

>>> s.get('a')
0
>>> s.get(key='A', default='default')
'default'
# DataFrame: column names only
>>> df.get(key='A')
a     0
b     4
c     8
d    12
e    16
Name: A, dtype: int64
>>> df.get(key='a', default='default')
'default'

2.4 `.at()`

DataFrame.at and Series.at

Access a single value for a row/column label pair. Use at if you only need to get or set a single value in a DataFrame or Series.

>>> s.at['a']
0
>>> df.at['a', 'A']
0

2.5 `.xs()`

DataFrame.xs(self, key, axis=0, level=None, drop_level:bool=True)

Return cross-section from the Series/DataFrame.

Suit for MultiIndex.

>>> s.xs(key='a')
0
# equals to df.xs(key='a', axis=0)
>>> df.xs('a')
A    0
B    1
C    2
D    3
Name: a, dtype: int64
# equals to df.xs(key='A', axis=1)
>>> df.xs('A', axis=1)
a     0
b     4
c     8
d    12
e    16
Name: A, dtype: int64

2.6 `.lookup()`

DataFrame.lookup(self, row_labels, col_labels) -> numpy.ndarry:

Label-based “fancy indexing” function for DataFrame. Given equal-length arrays of row and column labels, return an array of the values corresponding to each (row, col) pair.

>>> df.lookup(row_labels=['a', 'a'], col_labels=['A', 'C'])
array([0, 2])

Raise ValueError first if row labels and column labels are not the same size.

2.7 `.filter()`

DataFrame.filter(items=None, like=None, regex=None, axis=None) -> DataFrame or Series:

Subset the dataframe rows or columns according to the specified index labels. The filter is applied to the labels of the index.

Keyword arguments items, like, or regex are mutually exclusive

# 1. a single label
>>> df.filter(items='A')
	A
a	0
b	4
c	8
d	12
e	16
>>> df.filter(items='a', axis=0)
	A	B	C	D
a	0	1	2	3
>>> s.filter(items='a')
a    0
dtype: int64
# 2. a list of labels
>>> df.filter(items=['A', 'C', 'Z'])
	A	C
a	0	2
b	4	6
c	8	10
d	12	14
e	16	18
>>> df.filter(items=['a', 'c'], axis=0)
	A	B	C	D
a	0	1	2	3
c	8	9	10	11
>>> s.filter(items=['a', 'c'])
a    0
c    2
dtype: int64
# 3. containing str
>>> df_types.filter(like='date')
	datetime
0	2020-01-01
1	2020-01-02
2	2020-01-03
# 4. regular expression
>>> df_types.filter(regex='time?')
datetime	timedelta
0	2020-01-01	1 days
1	2020-01-02	2 days
2	2020-01-03	3 days

2.8 Summary Table Of Selecting By Label

Table1: Selecting From DataFrame By Label
DataFrame		`df[]`	`df.loc[]`	`df.get()`	`df.at()`	`df.xs()`	`df.lookup()`	`df.filter()`
Row Label	a single row label`'a'`	✘	✔`s`	✔Default	✘	✔`s`	✘	✔`axis=0, ->s`
	a list of row labels `['a', 'b']`	✘	✔`df`	✘	✘	✘	✘	✔`axis=0, ->s`
	a slice of row labels `slice('a', 'c')`	✔`df`	✔`df`	✔`df`	✘	✘	✘	✘
Column Label	a single column label`'A'`	✔`s`	✘	✔`s`	✘	✔`axis=1,->s`	✘	✔`s`
	a list of column labels `['A', 'B']`	✔`df`	✘	✔`df`	✘	✘	✘	✔`s`
	a slice of column labels `slice('A', 'B')`	✔`empty df`	✔`empty df`	✘	✘	✘	✘	✘
Combined	single label `'a', 'A'`	✘	✔`Scalar`	✔'A'	✔`Scalar`	✘	✘	✘
	list of labels `['a', 'b'], ['A', 'B']`	✘	✔`df`	✔`['A', 'B']`	✘	✘	✔`np.ndarray`	✘
	slice of labels `slice('a','b'), slice('A','B')`	✘	✔`df`	✔`df`	✘	✘	✘	✘

3. Selecting Data By Position

In Pandas, we mainly use [], .head, .tail, .iloc, .iat, and take to select data by position. Raise IndexError if a requested indexer is out-of-bounds, except slice indexers which allow out-of-bounds indexing(same with python/numpy slice semantics).

3.1 `[]`

Besides labels, DataFrame[] and Series[] also support slice object of integers, and Series[] supports integers. The slice object slice(i, j, k) has the same semantics as NumPy. Therefore, for negative values, a negative i equals to n+i, and a negative j equals to n+j, and a negative k means go in reverse order.

# 1. a single int, Series only
>>> s[0]
0
>>> df[0]
KeyError
# 2. a list of ints, Series only
>>> s[[0, 1]]
a    0
b    1
dtype: int64
# 3. a slice object, both for DataFrame and Series
# equals to s.iloc[0:2]
>>> s[0:2]
a    0
b    1
dtype: int64
>>> df[0:4:2]
	A	B	C	D
a	0	1	2	3
c	8	9	10	11
# euqals to df[5-3:5-1]
>>> df[-3:-1]
	A	B	C	D
c	8	9	10	11
d	12	13	14	15

3.2 `.head()` and `.tail()`

DataFrame.head(n:int=5) and Series.head(n:int=5): return the first n rows. DataFrame.tail(n:int=5) and Series.head(n:int=5): return the last n rows.

For n<0, df.head(n) equals to df.head(len+n), and df.tail(n) equals to df.tail(len+n), where len is the number of rows.

For n=0, returns empty Series or DataFrame.

>>> s.head(n=2)
a    0
b    1
dtype: int64
# equals to df.head(n=5-3)
>>> df.head(n=-3)
	A	B	C	D
a	0	1	2	3
b	4	5	6	7

3.3 `.iloc[]`

Series.iloc and DataFrame.iloc

Purely integer-location based indexing for selection by position.

# 1. a single integer
# equals to df.iloc[0, :]
>>> df.iloc[0]
A    0
B    1
C    2
D    3
Name: a, dtype: int64
>>> s.iloc[0]
0
# 2. a list or array of integers
# euqals to df.iloc[[0, 2], :]
>>> df.iloc[[0, 2]]
	A	B	C	D
a	0	1	2	3
c	8	9	10	11
>>> s.iloc[[0, 1]]
a    0
b    1
dtype: int64
# 3. a slice object with ints, the stop excluded
# equals to df.iloc[0:2, :]
>>> df.iloc[0:2]
	A	B	C	D
a	0	1	2	3
b	4	5	6	7
>>> df.iloc[0:2, 0:2]
	A	B
a	0	1
b	4	5
>>> s.iloc[0:2]
a    0
b    1
dtype: int64

3.4 `.iat[]`

DataFrame.iat and Series.iat

Access a single value for a row/column pair by integer position. Use iat if you only need to get or set a single value in a DataFrame or Series.

>>> s.iat[0]
0
>>> df.iat[0, 1]
1

3.5 `.take()`

DataFrame.take(indices, axis=0, **kwargs):

Return the elements in the given positional indices along an axis. This means that we are not indexing according to actual values in the index attribute of the object. We are indexing according to the atual position of the element in the object.

For n<0, equals to len+n.

>>> df.take([0, 2], axis=0)
	A	B	C	D
a	0	1	2	3
c	8	9	10	11
# equals to df.take([5-3, 5-1], axis=1)
>>> df.take([-3, -1], axis=1)
	B	D
a	1	3
b	5	7
c	9	11
d	13	15
e	17	19

4. Selecting Data By Value

4.1 Selecting By Bool

In Pandas, we can use .[], .loc, and .query() to subset based on bool.

>>> bool_row = pd.Series({'a': False, 'b': False, 'c': False, 'd': True, 'e': True})
>>> bool_col = pd.Series({'A': False, 'B': True, 'C': False, 'D': True})
>>> df.loc[bool_row, bool_col]
	B	D
d	13	15
e	17	19
# equals to df.loc[df['A']>10]
>>> df[df['A'] > 10]
	A	B	C	D
d	12	13	14	15
e	16	17	18	19

DataFrame.query(expr, inplace=False, **kwargs):

Query the columns of a DataFrame with a boolean expression. This method uses the top-level eval() function to evaluae the passed query. The result of the evaluation of this expression is first passed to DataFrame.loc.

# equals to df.loc[eval('df.A>10')]
>>> df.query('A>10')
	A	B	C	D
d	12	13	14	15
e	16	17	18	19

4.2 Selecting By Value Order

DataFrame.nlargest(n, columns, keep='first) -> DataFrame:

Return the first n rows ordered by columns in descending order. The columns that are not specified are returned as well, but not used for ordering. This method is equivalent to df.sore_values(columns, ascending=False).head(n), but more performant.

Similarly, DataFrame.nsmallest(n, columns, keep='first') -> DataFrame:

Return the first n rows ordered by columns in ascending order. This method is equivalent to df.sort_values(columns, ascending=True).head(n), but more performant.

>>> df.nlargest(n=2, columns=['A'])
	A	B	C	D
e	16	17	18	19
d	12	13	14	15
>>> df.nsmallest(n=2, columns=['A'])
	A	B	C	D
a	0	1	2	3
b	4	5	6	7

4.3 Selecting By Datatime

For datetime data, we can use DataFrame.first(), DataFrame.last(), DataFrame.at_time(), DataFrame.bewteen_time() to select subset based on datetime.

# subset initial periods of time series data based on a date offset
>>> ts.first('12h')
	A	B
2020-01-01 00:00:00	0	30
2020-01-01 04:30:00	1	31
2020-01-01 09:00:00	2	32
# subset final periods of the time series data based on a date offset
>>> ts.last('12h')
	A	B
2020-01-06 01:30:00	27	57
2020-01-06 06:00:00	28	58
2020-01-06 10:30:00	29	59
# select values at particulat time of day
>>> ts.at_time('12:00')
	A	B
2020-01-02 12:00:00	8	38
2020-01-05 12:00:00	24	54
# select values between particular times of the day
>>> ts.between_time('10:00', '12:00')
	A	B
2020-01-02 12:00:00	8	38
2020-01-03 10:30:00	13	43
2020-01-05 12:00:00	24	54
2020-01-06 10:30:00	29	59

5. Selection Data By Dtypes

pd.DataFrame.select_dtypes(self, include=None, exclude=None) -> DataFrame:

Return a subset of the DataFrame’s columns based on the column dtypes.

Table 2: Selecting Data By Dtypes

dtype	parameters
numeric	np.number/number
strings	object
datetimes	np.datetime64/datetime/datetime64
timedeltas	np.timedelta64/timedelta/timedelta64
categorical	category
datetimetz	datetimetz/datetime64[ns, tz]

>>> df_types.select_dtypes(['object'])
	str
0	a
1	b
2	c
>>> df_types.select_dtypes(['number'])
	int	timedelta
0	1	1 days
1	2	2 days
2	3	3 days
>>> df_types.select_dtypes(['timedelta'])
	timedelta
0	1 days
1	2 days
2	3 days
>>> df_types.select_dtypes(exclude=['category', 'datetime'])
	str	int	bool	timedelta
0	a	1	True	1 days
1	b	2	False	2 days
2	c	3	True	3 days

NumPy: Indexing

Mon, 27 Apr 2020 00:00:00 +0000

NumPy provides two fundamental objects: an N-dimensional array object(ndarray) and a universal function object(ufunc). An ndarray is a homogeneous collection of “items” indexed using N integers. There are two essential parts of an ndarray: 1) the shape, and 2) the data-type(dtype). Here, homogeneous means the dtype of the “items” are all same, each item occupies the same size block of memory, and each block of memory is interpreted in exactly the same way.

>>> x = np.array([[1, 2, 3], [2, 3, 4]])
>>> x.shape
(2, 3)
>>> x.dtype
dtype('int64') # 64-bit platform

In this article, we will talk about indexing through:

Memory layout of ndarray and strides
View and copy
Indexing styles
Indexing

From the perspective of the memory layout and strides, we can understand what indexing really are. And view and copy are the two different results of the indexing operators.

1. Memory Layout of ndarray and Strides

On a fundamental level, an ndarray is just a one-dimensional sequence of memory with fancy indexing code that maps an N-dimensional index into a one-dimensional index.

In general, from the perspective of the memory layout, an ndarray is just a 1-dimensional sequence of memory, addressed by a single index(the memory address). However, it is not convenient for us to use a 1-dimensional indexing to locate an N-dimensional ndarray. We prefer to use an N-dimensional fancy indexing to locate an N-dimensional ndarray.

NumPy uses strides to map the fancy N-dimensional indexing into the one-dimensional memory address. For an ndarray with n dimensions, the strides of this ndarray is a n elements tuple, each element means the number of bytes needed to jump to the next element in that dimension.

What’s more, with strides we can easily locate an ndarray that the underlying memory are even not contiguous which is not often to see.

>>> x = np.arange(24, dtype=np.int32).reshape(4, 3, 2)
>>> x.strides
(24, 8, 4)
>>> bytearray(x.data)
bytearray(b'
\x00\x00\x00\x00
\x01\x00\x00\x00
\x02\x00\x00\x00
\x03\x00\x00\x00
\x04\x00\x00\x00
\x05\x00\x00\x00
\x06\x00\x00\x00
\x07\x00\x00\x00
\x08\x00\x00\x00
\t\x00\x00\x00
\n\x00\x00\x00
\x0b\x00\x00\x00
\x0c\x00\x00\x00
\r\x00\x00\x00
\x0e\x00\x00\x00
\x0f\x00\x00\x00
\x10\x00\x00\x00
\x11\x00\x00\x00
\x12\x00\x00\x00
\x13\x00\x00\x00
\x14\x00\x00\x00
\x15\x00\x00\x00
\x16\x00\x00\x00
\x17\x00\x00\x00')
>>> x[0, 0, 1], x[3, 2, 0]
(1, 22)
# offset = (0-1)*4 + (2-0)*8 + (3-0)*24 = 84 bytes
>>> int.from_bytes(b'\x16\x00\x00\x00', 'little')
22

Ndarray x’s dtype is int32, so each element occupies 4 bytes, the whole data occupy 4*24=96 bytes. In the last dimension, the stride is 4=32/8 and there are 2 elements, so the stride in the second dimension is 4*2=8. Similarly, for the first dimension, the stride is 8*3=24. Therefore, x.strides=(24, 8, 4). In this case, if we want to move from x[0, 0, 1] to x[3, 2, 0], we need to move (0-1)*4 + (2-0)*8 + (3-0)*24 = 84 bytes, means from b'\x01\x00\x00\x00 to b'\x16\x00\x00\x00.

And if we change the dtype, the number of memory block that each item occupied will change, so the stride will change too.

>>> data = np.arange(24, dtype='int8')
>>> x = data.reshape(2, 3, 4)
>>> x.strides
(12, 4, 1)
>>> x.dtype = 'int16'
>>> x.strides
(12, 4, 2)

2. View and Copy

In NumPy, view means

an array that does not own its data, but referees to another array’s data instead.

When we create a view of an array, the two arrays share the same underlying data buffer, which is faster and can save memory. While when we create a copy of an array, a new data buffer will be created. So the copied array and base array are independent, which means any changes made to the array will not affect the other.

>>> data = np.array([0, 1, 2])
>>> x = data.view()
>>> y = data.copy()
>>> x[0] = 3
>>> y[0] = 4
>>> data
array([3, 1, 2])

3. Indexing Styles(Orders)

There are two styles of N-dimensional indexing for an ndarray: the C-style(row-major) and the Fortran-style(column-major). In the C-style, the rightmost dimension indexing “varies the fastest”. That is, to move to the next block of memory, the last dimension index changes first. New NumPy arrays are by default in row-major order. To the contrary, in the Fortran-style, the leftmost dimension indexing “varies the fastest”.

>>> data = np.arange(6, dtype='int8')
>>> x = np.reshape(data, (2, 3), 'C')
>>> y = np.reshape(data, (2, 3), 'F')
# data, x, and y share the same underlying data buffer
>>> np.shares_memory(data, x), np.shares_memory(data, y)
(True, True)
>>> x
array([[0, 1, 2],
       [3, 4, 5]], dtype='int8')
>>> y
array([[0, 2, 4],
       [1, 3, 5]], dtype='int8')

Here, data, x, and y share the same underlying data buffer in memory, which occupy 6 blocks of memory in 1-dimension. Each block represents a 8 bytes int scaler, the block i represents the number i(i=0, 1, 2, 3, 4, 5). With different orders, the indexing for x and y are different although they share the same blocks of memory.

Table1: Indexing Styles

Memory Address	C-style index(x)	Fortran-style index(y)
0	(0, 0)	(0, 0)
1	(0, 1)	(1, 0)
2	(0, 2)	(0, 1)
3	(1, 0)	(1, 1)
4	(1, 1)	(0, 2)
5	(1, 2)	(1, 2)

Obviously, an ndarray’s indexing style will affect its strides. If two ndarraies share the same data buffer having different orders, their strides will be different. And if we change the strides, the array’s order will change too.

# x and y share the same data buffer, with different orders and strides
>>> data = np.arange(120)
>>> x = data.reshape(2, 3, 4, 5, order='C')
>>> y = data.reshape(2, 3, 4, 5, order='F')
>>> x.strides, y.strides
((480, 160, 40, 8), (8, 16, 48, 192))
>>> x.flags['C_CONTIGUOUS'], x.flags['F_CONTIGUOUS']
(True, False)
>>> y.flags['C_CONTIGUOUS'], y.flags['F_CONTIGUOUS']
(False, True)

# Now, change y's strides to x's
>>> y.strides = (480, 160, 40, 8)
# y will equal to x
>>> np.allclose(x, y)
True
# and y's order flag changes too, equals to x's
>>> y.flags['C_CONTIGUOUS'], y.flags['F_CONTIGUOUS']
(True, False)

4. Array Indexing

There are three kinds of indexing available using the x[obj] syntax: filed access, basic slicing, and advanced indexing, which one occurs depends on obj. For an N-dimension array, we use , to separate dimensions, each dimension could be sliced through the same syntax. In this article, we will talk about basic indexing and advanced indexing.

4.1 Basic Indexing

Basic indexing always returns another view of the array. Basic indexing occurs when obj is:

integers
slice objects, x[start:stop:step] or x[slice(i, j, k)]
tuples of slice objects and integers, e.g. x[(slice(i, j, k), slice(i, j, k))], x[(slice(i, j, k), 0, slice(i, j, k))]
Ellipsis objects, e.g.x[...,0]
newaxis objects, e.g.x[np.newaxis, 0]

4.1.1 Slice object

The basic syntax for slicing is seq[i:j:k],

start-index i, included, default value is 0 for k > 0 and n-1 for k < 0
stop-index j, excluded, default value is n for k > 0 and -n-1 for k < 0
step-size k, default value is 1, cann’t be 0

where n is the number of elements in the corresponding dimension. seq[i:j:k] is actually just a nice shorthand for seq[slice(i, j, k)].

# 1. normal values
>>> x = np.arange(10)
>>> x[2:8:2]
array([2, 4, 6])
# 2. shorthand, seq[i:j:k] == seq[slice(i, j, k)]
>>> np.allclose(x[2:8:2], x[slice(2,8,2)])
True

Some examples for the dafault values:

# 3. default values
# k_default = 1, i_default = 0
>>> x[:5]
array([0, 1, 2, 3, 4])
# k > 0, then i_default = 0, j_default = 10
>>> x[::2]
array([0, 2, 4, 6, 8])
# k < 0, then i_default = 9, j_default = -11
>>> x[::-1]
array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0])

There are some special cases for slicing, including None, negative values, and out-of-bounds start/stop values. None is literately None, thus seq[:] == seq[::] = seq[None:None:None].

For negative values, a negative i equals to n+i, and a negative j equals to n+j, and a negative k means go in reverse order:

# 4. negative values
# negative i or j, i = n + i, j = n + j
>>> x[-6:8], x[-6:-2]
(array([4, 5, 6, 7]), array([4, 5, 6, 7]))
# negative k, from right to left
>>> x[4:2:-1]
array([4, 3])

For out-of-bound values, the start index i will become the first available index, and the stop index j will become the last available index:

# 5. out-of-bound i or j
>>> x[-10:20]
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> x[20:-10:-1]
array([9, 8, 7, 6, 5, 4, 3, 2, 1])

With the right syntax, we can still get an empty array. If the start index i in the left/right of the stop index j, and k in the reverse/normal order, empty array will be returned. This empty array in a view of the original array, but shares no data(there is no data to share).

# 6. in wrong slice order, an empty array returned, and shares no memory
>>> y = x[2:4:-1]
>>> y
array([], dtype='int64')
>>> z = x[4:2:1]
>>> z
array([], dtype='int64')
>>> np.shares_memory(x, y), np.shares_memory(x, z), np.shares_memory(y, z)
(False, False, False) 

Table2: Slicing

seq[i:j:k]	included	default value	negative value	None	out-of-bound
strat-index i	Yes	k>0: 0; k<0: n-1	i=n+i	:	k>0: 0 k<0: n-1
stop-index j	No	k>0: n; k<0: -n-1	j=n+j	:	k>0: n k<0: 0
step-size k	N/A	1	reverse order	1	N/A

4.1.2 Newaxis and Ellipsis

We use newaxis to expand dimensions and use ... to represent all the non-explicit dimensions.

>>> x = np.arange(6)
>>> x[np.newaxis, 1:3:1, np.newaxis]
array([[[1],
        [2]]])
>>> x[np.newaxis, 1:3:1, np.newaxis].shape
(1, 2, 1)
>>> y = np.arange(120).reshape(2, 3, 4, 5)
>>> y[0,...,1]
array([[ 1,  6, 11, 16],
       [21, 26, 31, 36],
       [41, 46, 51, 56]])

4.2 Advanced Indexing

Advanced indexing is triggered when the obj is

a non-tuple sequence object
an ndarray(of data type integer or bool)
a tuple with at least one sequence object or ndarray(of data type integer or bool)

There are two types of advanced indexing: integer and Boolean. Advanced indexing always returns a copy of the data.

4.2.1 Integer Array Indexing

Integer array indexing allows selection of arbitrary items in the array based on their N-dimensional index. For an ndarray with n dimension, we need n integer index arrays; one for each dimension. If the n integer index arrays don’t have the same shape, broadcasting may be applied. In final, all the integer index arrays should have the same shape.

>>> x = np.arange(24).reshape(2, 3, 4)
>>> index1 = np.array([0, 1])
>>> index2 = np.array([[2, 1], [0, 2]])
>>> index3 = np.array([[3, 2], [1, 0]])
>>> x[index1, index2, index3]
array([[11, 18],
       [ 1, 20]])

In this example, x has 3 dimensions, thus we need 3 integer index arrays. index1 is for the first dimension, index2 is for the second dimension, and index3 is for the third dimension. As index1’s shape is (2, ), broadcasting will be appiled, and index1 turned into np.array([[0, 1], [0, 1]]). In this way, the first resulting element is x[0, 2, 3], the second is x[1, 1, 2], and so on. After broadcasting, each integer index array has the shape (2, 2), therefore the resulting array have the shape (2, 2).

4.2.2 Boolean Array Indexing

Boolean array indexing is used to select elements from an array based on logical conditions. The simplest case is like

>>> x = np.arange(5)
>>> index = [True, True, False, False, True]
>>> x[index]
array([0, 1, 4])

A common use case for Boolean index array is filtering for desired element values, e.g. select all elements which are not NaN.

>>> x = np.array([[0, 1], [np.nan, 2], [np.nan, np.nan]])
>>> x[~np.isnan(x)]
array([0, 1, 2])

4.2.3 Combing Integer and Boolean Array Indexing

>>> x = np.arange(24).reshape(2, 3, 4)
>>> index1 = np.array([True, False])
>>> index2 = np.array([[2, 1], [0, 2]])
>>> index3 = np.array([[3, 2], [1, 0]])
>>> x[index1, index2, index3]
array([[11,  6],
       [ 1,  8]])

In this example, index1 for the first dimension, means select the first element in the first dimension. Therefore, the indeies are (0, 2, 3), (0, 1, 2), (0, 0, 1), (0, 2, 0).

4.3 Combing Basic and Advanced Indexing

Actually, when combing basic and advanced indexing, the rules are still same, element will be chosen one by one, and a copy of the original array will be returned.

>>> x = np.arange(24).reshape(2, 3, 4)
>>> index1 = np.array([0, 1])
>>> index3 = np.array([[3, 2], [0, 2]])
# index2 use the basic indexing
>>> y = x[index1, :, index3]
>>> y
array([[[ 3,  7, 11],
        [14, 18, 22]],

       [[ 0,  4,  8],
        [14, 18, 22]]])
>>> np.shares_memory(x, y)
False

In this example, broadcasting are applied first, turn index1 into np.array([[0, 1], [0, 1]]). : in the second dimension is a basic indexing, means select all the elements in the second dimension, that is [0, 1, 2]. Therefore, the result will be:

>>> result = np.array([
    [
        [x[0, 0, 3], x[0, 1, 3], x[0, 2, 3]],
        [x[1, 0, 2], x[1, 1, 2], x[1, 2, 2]]
    ],
    [
        [x[0, 0, 0], x[0, 1, 0], x[0, 2, 0]],
        [x[1, 0, 2], x[1, 1, 2], x[1, 2, 2]]
    ]
])
>>> np.allclose(result, y)
True
>>> result.shape
(2, 2, 3)

Another example:

>>> x = np.arange(24).reshape(2, 3, 4)
>>> index1 = np.array([0, 1])
>>> index2 = np.array([[1, 2], [0, 2]])
>>> y = x[index1, index2, 0]
>>> y
array([[ 4, 20],
       [ 0, 20]])
>>> result = np.array([
    [x[0, 1, 0], x[1, 2, 0]],
    [x[0, 0, 0], x[1, 2, 0]]
])
>>> np.allclose(result, y)
True
>>> np.shares_memory(x, y)
False

4.4 Summary Table

Table3: Basic Indexing and Advanced Indexing

	Basic Indexing	Advanced Indexing	Combined
obj	integer, slice, ellipsis, newaxis	contains non-tuple seq obj or ndarray	combined
view or copy	view	copy	copy

NumPy: Joining Arrays Together

Mon, 06 Apr 2020 00:00:00 +0000

NumPy provides several functions that allow us to join arrays, including stack, concatenate, vstack, hstack, dstack, column_stack, and block. In this article, we will:

describe these joining array functions from the perspective of shape, ndim and axis
give more formalize descriptions and more examples
compare them to each other

1. np.stack

numpy.stack(arrays, axis=0, out=None): Join a sequence of arrays along a new axis.

So the number of dimensions will increase 1. And all the input arrays must have the same shape. With input arrays a_0, a_1, ... a_(n-1), each array has the same shape (s_0, s_1, ... s_(m-1)) and the same number of dimensions m, then stacked_array = np.stack((a_0, a_1, ... a_(n-1)), axis=d)) will have the shape

for 0 <= d <= m-1,

if d == 0: shape = (n, s_0, ..., s_(m-1)), ndim = m + 1

if d == i(0<i<m-1): shape = (s_0, ..., s_(i-1), n, s_(i+1), ..., s_(m-1)), ndim = m + 1

if d == m-1: shape = (s_0, s_1, ..., n, s_(m-1)), ndim = m + 1

For example:

>>> shape = (2, 3, 4, 5)
>>> a1 = np.arange(120).reshape(shape)
>>> a2 = np.arange(120, 120*2).reshape(shape)
>>> a3 = np.arange(120*2, 120*3).reshape(shape)
>>> a4 = np.arange(120*3, 120*4).reshape(shape)
>>> a5 = np.arange(120*4, 120*5).reshape(shape)
>>> a6 = np.arange(120*5, 120*6).reshape(shape)
>>> arrays = [a1, a2, a3, a4, a5, a6]
>>> r0 = np.stack(arrays, axis=0)
>>> r1 = np.stack(arrays, axis=1)
>>> r2 = np.stack(arrays, axis=2)
>>> r3 = np.stack(arrays, axis=3)
>>> r4 = np.stack(arrays, axis=4)
>>> r0.shape, r1.shape, r2.shape, r3.shape, r4.shape
((6, 2, 3, 4, 5),
 (2, 6, 3, 4, 5),
 (2, 3, 6, 4, 5),
 (2, 3, 4, 6, 5),
 (2, 3, 4, 5, 6))

2. np.concatenate

np.concatenate(arrays, axis=0, out=None): Join a sequence of arrays along an existing axis.

So the number of dimension will not increase. It requires all arrays must have the same shape, except in the dimension corresponding to axis. If axis=None, the number of dimension will be 1, arrays will be flattened.

If we want to execute concatenated_array=np.concatenate((a_0, ..., a_(n-1)), axis=d, out=None), where d is an integer, the input arrays a_0, ..., a_(n-1) must be

for 0 <= d <= m-1 and for all 0 <= j <= n-1: 

if d = 0, a_j.shape = (t_j, s_1, ..., s_(m-1))
if d = i(0<i<m-1), a_j.shape = (s_0, ..., s_(i-1), t_j, s_(i+1), ..., s_(m-1))
if d = m-1, a_j.shape = (s_0, ..., s_(m-2), t_j)

then the concatenated_array will remain the same shape except the corresponding axis, which will be the sum of all the arrays, that is

for 0 <= d <= m-1, 

if d = 0, concatenated_array.shape = (t, s_1, ..., s_(m-1))
if d = i(0<i<m-1), concatenated_array.shape = (s_0, ..., s_(i-1), t, s_(i+1), ..., s_(m-1))
if d = m-1, concatenated_array.shape = (s_0, ..., s_(m-2), t)

where t = sum(t_0, ..., t_(n-1))

For example:

>>> a0 = np.arange(120).reshape(2, 3, 4, 5)
>>> a1 = np.arange(120, 150).reshape(2, 3, 1, 5)
>>> a2 = np.arange(150, 330).reshape(2, 3, 6, 5)
>>> concatenate_array = np.concatenate((a0, a1, a2), axis=2)
>>> concatenate_array.shape, concatenate_array.ndim
((2, 3, 11, 5), 4) # 11 = 4+1+6

3. np.vstack

np.vstack(tup): Stack arrays in sequence vertically(row wise). This is equivalent to concatenation along the first axis after 1-D arrays of shape(N, ) have been reshaped to (1, N).

np.vstack requires the input arrays have the same shape along all but the first axis. 1-D arrays must have the same length. Therefore, np.vstack(arrays) == np.concatenate(arrays, axis=0).

For example:

>>> a0 = np.arange(120).reshape(4, 2, 3, 5)
>>> a1 = np.arange(120, 150).reshape(1, 2, 3, 5)
>>> a2 = np.arange(150, 330).reshape(6, 2, 3, 5)
>>> r0 = np.vstack((a0, a1, a2))
>>> r1 = np.concatenate((a0, a1, a2), axis=0)
>>> np.allclose(r0, r1)
True

4. np.hstack

np.hstack(tup): Stack arrays in sequence horizontally(column wise). This is equivalent to concatenation along the second axis, except for 1-D arrays where it concatenates along the first axis.

Therefore,

1-D arrays:
np.hstack(arrays) == np.concatenate(arrays, axis=0) == np.concatenate(arrays, axis=None)

arrays >= 2-D:
np.hstack(arrays) == np.concatenate(arrays, axis=1)

For example,

# 1-D arrays
>>> a = np.array([1,2,3])
>>> b = np.array([2,3,4])
>>> hstacked = np.hstack((a,b))
>>> concatenated_0 = np.concatenate((a, b), axis=0)
>>> concatenated_1 = np.concatenate((a, b), axis=None)
>>> np.allclose(hstacked, concatenated_0)
True
>>> np.allclose(hstacked, concatenated_1)
True

# arrays >= 2-D
>>> a0 = np.arange(120).reshape(2, 4, 3, 5)
>>> a1 = np.arange(120, 150).reshape(2, 1, 3, 5)
>>> a2 = np.arange(150, 330).reshape(2, 6, 3, 5)
>>> r0 = np.hstack((a0, a1, a2))
>>> r1 = np.concatenate((a0, a1, a2), axis=1)
>>> np.allclose(r0, r1)
True

5. np.dstack

np.dstack(tup): Stack arrays in sequence depth wise(along third axis).

This is equivalent to concatenation along the third axis. For arrays more than 2-D, np.dstack(arrays) == np.concatenate(arrays, axis=2). 1-D arrays with shape(N,) will be reshaped to (1,N,1), and 2-D arrays with shape(M, N) will be reshaped to (M, N, 1). After reshape, 1-D arrays and 2-D arrays have at least 3 dimensions, axis=2 will be okay.

Therefore,

1-D arrays with shape (N, ):
np.dstack(arrays) == np.concatenate(arrays.reshape(1, N, 1), axis=2)

2-D arrays with shape (M, N):
np.dstack(arrays) == np.concatenate(arrays.reshape(M, N, 1), axis=2)

>= 3-D arrays:
np.dstack(arrays) == np.concatenate(arrays, axis=2)

For example:

1-D arrays
>>> a0 = np.arange(6)
>>> a1 = np.arange(6, 12)
>>> a2 = np.arange(12, 18)
>>> r0 = np.concatenate((a0.reshape(1, 6, 1), a1.reshape(1, 6, 1), a2.reshape(1, 6, 1)), axis=2)
>>> r1 = np.dstack((a0, a1, a2))
>>> np.allclose(r0, r1)
True

2-D arrays
>>> a0 = np.arange(6).reshape(2, 3)
>>> a1 = np.arange(6, 12).reshape(2, 3)
>>> a2 = np.arange(12, 18).reshape(2, 3)
>>> r0 = np.concatenate((a0.reshape(2, 3, 1), a1.reshape(2, 3, 1), a2.reshape(2, 3, 1)), axis=2)
>>> r1 = np.dstack((a0, a1, a2))
>>> np.allclose(r0, r1)
True

>= 3-D arrays
>>> a0 = np.arange(120).reshape(2, 3, 4, 5)
>>> a1 = np.arange(120, 150).reshape(2, 3, 1, 5)
>>> a2 = np.arange(150, 330).reshape(2, 3, 6, 5)
>>> r0 = np.concatenate((a0, a1, a2), axis=2)
>>> r1 = np.dstack((a0, a1, a2))
>>> np.allclose(r0, r1)

6. np.column_stack

np.column_stack(tup) equals to concatenate arrays along the second axis, 1-D arrays with shape(N, ) will be reshape to (N, 1).

Therefore,

1-D arrays:
np.column_stack(arrays) == np.concatenate(arrays.reshape(-1, 1), axis=1)

>= 2-D arrays:
np.column_stack(arrays) == np.concatenate(arrays, axis=1) == np.hstack(arrays)

For example:

# 1-D arrays
>>> a = np.array([1,2,3])
>>> b = np.array([2,3,4])
>>> r0 = column_stack((a, b))
>>> r1 = np.concatenate((a.reshape(-1, 1), b.reshape(-1, 1)), axis=1)
>>> np.allclose(r0, r1)
>>> True

# arrays >= 2-D
>>> a0 = np.arange(120).reshape(2, 4, 3, 5)
>>> a1 = np.arange(120, 150).reshape(2, 1, 3, 5)
>>> a2 = np.arange(150, 330).reshape(2, 6, 3, 5)
>>> r0 = np.column_stack((a0, a1, a2))
>>> r1 = np.concatenate((a0, a1, a2), axis=1)
>>> r2 = np.hstack((a0, a1, a2))
>>> np.allclose(r0, r1), np.allclose(r0, r1)
(True, True)

7. np.block

np.block(arrays): Assemble an ndarray from nested lists of blocks.

Blocks in the innermost lists are concatenated along the last dimension(-1), then these are concatenated along the second-last dimension(-2), and so on until the outermost list is reached.

Therefore, np.block(arrs) will execute r_0_i = concatenate((innermost_lists_i), axis=-1) first, and then execute r_1_j = concatenate([..., r_0_i...,], axis=-2), and so on until the outermost list is reached.

For example:

>>> A11 = np.ones((2, 3, 1))
>>> A12 = np.zeros((2, 3, 5))
>>> A21 = np.ones((2, 3, 4))
>>> A22 = np.zeros((2, 3, 2))

>>> B11 = np.zeros((1, 3, 2))
>>> B12 = np.ones((1, 3, 4))
>>> B21 = np.zeros((1, 3, 1))
>>> B22 = np.ones((1, 3, 5))

# A11, A12 required to be sampe shape except the last axis
>>> A1 = np.concatenate((A11, A12), axis=-1)
>>> A2 = np.concatenate((A21, A22), axis=-1)
>>> A1.shape, A2.shape
((2, 3, 6), (2, 3, 6))

# A1, A2 required to be sampe shape except the last second axis
>>> A_con = np.concatenate((A1, A2), axis=-2)
>>> A_block = np.block([
    [[A11, A12], [A21, A22]]
])
>>> np.allclose(A_con, A_block)
True
>>> A_block.shape
(2, 6, 6)

>>> B1 = np.concatenate((B11, B12), axis=-1)
>>> B2 = np.concatenate((B21, B22), axis=-1)
>>> B_con = np.concatenate((B1, B2), axis=-2)
>>> B_block = np.block([
    [[B11, B12], [B21, B22]]
])

# A_con, B_con required to be same shape except last third axis
>>> result_con = np.concatenate((A_con, B_con), axis=-3)
>>> result_block = np.block([
    [[A11, A12], [A21, A22]],
    [[B11, B12], [B21, B22]],
])

>>> np.allclose(result_con, result_block)
True
>>> result_block.shape
(3, 6, 6)

8. summary

The different between stack and concatenate is easy to see. The main differences are including the input arrays’ shape requirement and the returned array’s dimension number.

Table1: Compare Stack With Concatenate

Table1	stack	concatenate
arrays’ shape requirement	exactly same shape	same shape except the corresponding axis
axis values	int	int or None
ndim	+1	unchanged or = 1

The main differences between vstack, hstack, dstack and column_stack are and the input arrays’ shape requirement and the concatenate axis. They all can be implemented through concatenate. Table2 describes how to implement them through concatenate.

Table2: Implement Through Concatenate

Table2	vstack	hstack	dstack	column_stack	block
along axis for >= 2-D arrays	0	1	2	1	from last to first
along axis for 1-D arrays	0	0 or None	2	1	None
reshape required for 2-D arrays	No	No	Yes	No	No
reshape required for 1-D arrays	No	No	Yes	Yes	No

A Super Boring Guide To Support Vector Machines

Wed, 04 Mar 2020 00:00:00 +0000

1. Overview

A support vector machine(SVM) is a supervised Machine Learning algorithm used for classification, regression and outliers detection. The basic idea of SVM is constructing a generalized optimal hyperplane that seperates the data with the maximal margin while makes the smallest number of errors.

For a particular hyperplane, the nearest instances are support vectors. The distance between support vectors and the hyperplane is margin. Basically, SVM Classification is a maximum-margin classifier.

2. Notations

We denote \(\mathbb{S}\) as a finite training set, with \(X\) as training features, and \(y\) as training labels, where

\[\begin{equation} X = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1m} \\ \vdots & \vdots & \vdots & \vdots \\ x_{n1} &x_{n2} & \cdots & x_{nm} \end{bmatrix} \end{equation}\]

and

\[\begin{equation} y = \begin{bmatrix} y_{1} \\ y_{2} \\ \cdots \\ y_{n} \end{bmatrix} \end{equation}\]

\(n\) is the sample number, \(m\) is the feature number. The label of instance is

\[\begin{equation} y_{i} \in \{ -1, 1 \}, \quad i = 1, 2, ..., n \end{equation}\]

and the features of an instance is

\[\begin{equation} x_{i} = \begin{bmatrix} x_{i1} & x_{i2} & \cdots & x_{im} \end{bmatrix}, \quad i = 1, 2, ..., n \end{equation}\]

A hyperplane is denoted as

\[\begin{equation} \phi(x)w+b = 0 \end{equation}\]

where

\[\begin{equation} w = \begin{bmatrix} w_{1} \\ w_{2} \\ \vdots \\ w_{m} \end{bmatrix} \end{equation}\]

\(\phi(x)\) is a feature mapping function, and \(b\) is a constant.

3. SVM: hard margin

To make things easy, we’ll consider a binary hard margin classification first. Here, we try to find a hyperplane that maximize the minimum distance between the samples and the hyperplane. The distance bewteen \(\forall x_{i}\) and the hyperplane is

\[\begin{equation} d_i = \frac{y_{i}(\phi(x_{i})w+b)}{||w||} \end{equation}\]

So the margin is

\[\begin{equation} \min \limits_{i} \ d_{i}, \quad i=1, 2, ... n \end{equation}\]

Assume instance \((x^*, y^*)\) is the nearest one to the hyperplane, therefore the hard margin optimal problem \(\mathcal{H}\) becomes

\[\begin{equation} \begin{aligned} \max \limits_{w, b} \quad & \frac{y^*(\phi(x^*)w+b)}{||w||} \\ \text{s.t.} \quad & \frac{y^*(\phi(x^*)w+b)}{||w||} \leq d_i, \quad i=1, 2, ... n \end{aligned} \end{equation}\]

Since for all constant \(a \neq 0\), we have

\[\begin{equation} y^* \left( \phi(x^*)(aw)+ab \right) = a*y^* \left( \phi(x^*)w+b \right) \end{equation}\]

so, there must exists a constant \(c \neq 0\), such that

\[c* y^*(\phi(x^*)w+b) = 1\]

therefore, \(\mathcal{H}\) becomes

\[\begin{equation} \begin{aligned} \max \limits_{w, b} \quad & \frac{1}{c||w||} \\ \text{s.t.} \quad & 1 - y_{i}(\phi(x_{i})w+b) \leq 0, \quad i=1, 2, ... n \end{aligned} \end{equation}\]

Equals to

\[\begin{equation} \begin{aligned} \min\limits_{w} \quad & \frac{1}{2}w^Tw \\ \text{s.t.} \quad & 1 - y_{i}(\phi(x_{i})w+b) \leq 0, \quad i=1, 2, ... n \end{aligned} \end{equation}\]

Combine with Lagrangian dual problem(see section Lagrangian dual problem), \(\mathcal{H}\) equals to \(\eqref{dual}\)

\[\begin{equation} \begin{aligned} \max \limits_{\lambda, \nu} \min \limits_{w} L(w, b, \lambda, \nu) = \frac{1}{2}w^Tw + \sum_{i=1}^{m}\lambda_{i}(1 - y_{i}(\phi(x_{i})w+b)) \end{aligned} \end{equation}\]

\(L(w, b, \lambda, \nu)\) is convex about \(w\), and \(b\), therefore \(w\), and \(b\) can be drived from \(\lambda\)

\[\begin{equation} \begin{aligned} & \begin{cases} \partial L(w, \lambda) / \partial w^* = 0 \\ \partial L(w, \lambda) / \partial b^* = 0 \end{cases} \\ \Rightarrow & \begin{cases} w_k^* = \sum_{i=1}^n \lambda_i y_i \phi(x_{ik}), \quad k=1,...m \\ \sum_{i=1}^n \lambda_i y_i = 0 \end{cases} \\ \end{aligned} \end{equation}\]

\(L(w, b, \lambda, \nu)\) becomes

\[\begin{equation} \begin{aligned} L(\lambda) & = \sum_{i=1}^n \lambda_i - \frac{1}{2}\sum_{k=1}^m \left( \sum_{i=1}^n \lambda_i y_i \phi(x_{ik}) \right) ^2 \\ & = \sum_{i=1}^n \lambda_i - \frac{1}{2} \sum_{i,j=1}^n \lambda_i y_i \lambda_j y_j \phi(x_i) \phi(x_j)^T \\ & = \sum_{i=1}^n \lambda_i - \frac{1}{2} \sum_{i,j=1}^n \lambda_i y_i \lambda_j y_j K(x_i, x_j) \end{aligned} \end{equation}\]

where \(K(x_i, x_j)\) is the kernel function. Therefore, the problem equals to

\[\begin{equation} \begin{aligned} & \max \limits_\lambda L(\lambda) = \sum_{i=1}^n \lambda_i - \frac{1}{2} \sum_{i,j=1}^n \lambda_i \lambda_j y_i y_j K(x_i, x_j)\\ & \text{s.t.} \quad \sum_{i=1}^n \lambda_i y_i = 0, \quad \lambda_i \geq 0, \quad i=1, 2, ... n \end{aligned} \label{SMOProblem} \end{equation}\]

4. SVM: soft margin

However, in real situation, a hard margin hyperplane may not exists or generalizes well. So we introduce soft margin hyperplane to find a good balance between maximal margin and smallest number of errors. The optimal problem becomes \(\mathcal{H_S}\)

\[\begin{equation} \begin{aligned} \min\limits_{w} \quad & \frac{1}{2}w^Tw + C\sum_{i=1}^n \zeta_i \\ \text{s.t.} \quad & 1 - y_{i}(\phi(x_{i})w+b) \leq \zeta_i, \quad i = 1, 2, ... n \end{aligned} \end{equation}\]

where \(C\) controls the trade-off and \(\zeta_i\) is the hinge loss function

\[\begin{equation} \zeta_i = \max(0, 1 - y_{i}(\phi(x_{i})w+b)) \end{equation}\]

The dual becomes

\[\begin{equation} \begin{aligned} & \max \limits_\lambda L(\lambda) = \sum_{i=1}^n \lambda_i - \frac{1}{2} \sum_{i,j=1}^n \lambda_i \lambda_j y_i y_j K(x_i, x_j)\\ & \text{s.t.} \quad \sum_{i=1}^n \lambda_i y_i = 0, \quad 0 \leq \lambda_i \leq C, \quad i=1, 2, ... n \end{aligned} \label{SoftDualProblem} \end{equation}\]

5. Kernels

Kernels are what make SVM magical and it is not limited to SVMs. A linear model can become a non-linear model by kernel functions. Through kernel functions, we can increase feature dimensions without even actually computing the feature mapping or even knowing the feature mapping function. The most common kernel functions are:

Gaussian kernel: \(K(x_i, x_j) = exp(- \gamma {\| x_i - x_j \|}^2)\), \(\gamma \ geq 0\).
Polynomial kernel: \(K(x_i, x_j) = (<x_i, x_j> + c)^d\), d means degree.
Expoenetial kernel: \(K(x_i, x_j) = exp(- \gamma \| x_i - x_j \|)\).
Sigmoid kernel: \(K(x_i, x_j) = tanh(\gamma<x_i, x_j> + c)\).

6. Iterative Methods

The traditional iterative method is sequential minimal optimization(SMO). SMO is an extreme case of the decomposition method used in LIBSVM. LIBSVM is a popular open source library for SVMs. Decomposition methods translate a problem into a new one that is easier to solve, by grouping variables into sets, and solving a subproblem for each set. SMO updates only two \(\lambda\) per iteration, other lagrange multipliers considered as constants, thus problem becomes two variables QP optimization per iteration. With the constraint as \(\eqref{SoftDualProblem}\), we have

\[\begin{equation} \begin{aligned} & \sum_{i=1}^n \lambda_i y_i = 0 \\ \Rightarrow \quad & \lambda_i y_i + \lambda_j y_j = - \sum_{k \neq i, j}^n \lambda_k y_k \\ \Rightarrow \quad & \lambda_j = (- \sum_{k \neq i, j}^n \lambda_k y_k - \lambda_i y_i)y_j \\ \Rightarrow \quad & \lambda_j = (a - \lambda_i y_i)y_j \end{aligned} \end{equation}\]

where \(a = - \sum_{k \neq i, j}^n \lambda_k y_k\). The Langrange in \(\eqref{SoftDualProblem}\) becomes

\[\begin{equation} \begin{aligned} L(\lambda) & = \sum_{k=1}^n \lambda_k - \frac{1}{2} \sum_{k,h=1}^n \lambda_k \lambda_h y_k y_h K_{kh} \\ & = \lambda_i(1 - y_i y_j) - \frac{1}{2} [\, \lambda_i^2K_{ii} + (a - \lambda_i y_i)^2K_{jj} \\ & \quad + 2\lambda_i y_i (a - \lambda_i y_i)K_{ij} \\ & \quad + \lambda_i y_i \sum_{k \neq i, j}^n \lambda_k y_k (K_{ik} - K_{jk}) ]\, + constant \end{aligned} \end{equation}\]

where \(K_{ij} = K(x_i, x_j)\). Therefore, we have

\[\begin{equation} \begin{aligned} \partial L / \partial \lambda_i & = 1 - y_i y_j - (K_{ii} + K_{jj} - 2K_{ij})\lambda_i \\ & \quad - a y_i (K_{ij} - K_{jj}) - \frac{1}{2} y_i \sum_{k \neq i, j}^n \lambda_k y_k (K_{ik} - K_{jk}) \end{aligned} \end{equation}\]

and

\[\begin{equation} \begin{aligned} \partial^2 L / \partial \lambda_i^2 & = - (K_{ii} + K_{jj} - 2K_{ij}) \\ & = - \sum_{h=1}^m(\phi^2(x_{ih})-2\phi(x_{ih})\phi(x_{jh})+\phi^2(x_{jh})) \\ & = - \sum_{h=1}^m(\phi(x_{ih}) - \phi(x_{jh}))^2 \leq 0 \end{aligned} \end{equation}\]

Therefore,

\[\begin{equation} \begin{aligned} (\lambda_i^{u})^{*} = \frac{1- y_i y_j + \frac{1}{2}y_i (\, \sum_{k \neq i, j}^n \lambda_k y_k(2K_{ij} - 2K_{jj} + K_{jk} - K_{ik}) )\, } {K_{ii} + K_{jj} - 2K_{ij}} \\ \end{aligned} \end{equation}\]

With the constraint \(\eqref{SoftDualProblem}\), we have

\[\begin{equation} \begin{aligned} & \begin{cases} 0 \leq \lambda_i \leq C \\ 0 \leq \lambda_j \leq C \\ \lambda_i^t y_i + \lambda_j^t y_j + \sum_{k \neq i, j}^n \lambda_k^t y_k = 0 \\ \lambda_i^{t-1} y_i + \lambda_j^{t-1} y_j + \sum_{k \neq i, j}^n \lambda_k^{t-1} y_k = 0 \\ \lambda_k^t = \lambda_k^{t-1}, k \neq i, j \end{cases} \\ \Rightarrow & \begin{cases} L = \max \{0, \lambda_i^{t-1} + \lambda_j^{t-1} - C \}, \ H = \min \{ C, \lambda_i^{t-1} + \lambda_j^{t-1} \}, \quad if \ y_i y_j = 1\\ L = \max \{ 0, \lambda_i^{t-1} - \lambda_j^{t-1} \}, \ H = \min \{ C, C + \lambda_i^{t-1} - \lambda_j^{t-1} \}, \quad if \ y_i y_j = -1 \end{cases} \end{aligned} \end{equation}\]

where \(L\) is the lower bounder, \(H\) is the high bounder, and \(t\) means step \(t\). Therefore,

\[\begin{equation} \begin{aligned} (\lambda_i^{u,t})^{*} = \frac{1- y_i y_j + \frac{1}{2}y_i (\, \sum_{k \neq i, j}^n \lambda_k^{t-1} y_k(2K_{ij} - 2K_{jj} + K_{jk} - K_{ik}) )\, } {K_{ii} + K_{jj} - 2K_{ij}} \\ \end{aligned} \end{equation}\] \[\begin{equation} \begin{aligned} (\lambda_i^{t})^{*} = \begin{cases} L, \qquad if \ (\lambda_i^{u,t})^{*} < L \\ \lambda_i^{*, u}, \qquad if \ L \leq (\lambda_i^{u,t})^{*} \leq H \\ H, \qquad if \ (\lambda_i^{u,t})^{*} > H \\ \end{cases} \end{aligned} \end{equation}\]

After solving the optimal Lagrange multipliers per iteration, the next problem is how to choose Lagrange multipliers at each iteration. In LIBSVM, it is called working set selection. There are several methods working with it. The current one used in LIBSVM is via the “maximal violating pair”. For more details, please reference Chang and Lin(2001) or Fan, Chen, and Lin(2005).

Other iterative methods includes subgradient descent. Subgradient methods are iterative methods for solving convex minimization problems.(via wiki) Basically, it is very similar to gradient descent. However, subgradient descent can apply to non-differentiable objective function.

7. Lagrangian dual problem

Given the primal problem \(\mathcal{P_0}\)

\[\begin{equation} \begin{aligned} \min\limits_{x \in \mathbb{D}} \quad & f(x) \\ \text{s.t.} \quad & g_{i}(x) \leq 0, \quad i=1, 2, ... m \\ \quad & h_{j}(x) = 0, \quad j = 1, 2, ... p \end{aligned} \label{originalPrimal} \end{equation}\]

with the domain \(\mathbb{D} \subset \mathbb{R^n}\), the \(\textit{Lagrangian function}\) \(\mathbb{L} : \mathbb{R^n} \times \mathbb{R^m} \times \mathbb{R^p} \to \mathbb{R}\) is defined as

\[\begin{equation} L(x, \lambda, \nu) = f(x) + \sum_{i=1}^{m}\lambda_{i}g_i(x) + \sum_{j=1}^{p}\nu_{j}h_j(x) \label{lagrangian} \end{equation}\]

where

\[\begin{equation} \lambda = \begin{bmatrix} \lambda_{1} \\ \lambda_{2} \\ \vdots \\ \lambda_{m} \end{bmatrix}, \quad \nu = \begin{bmatrix} \nu_{1} \\ \nu_{2} \\ \vdots \\ \nu_{p} \end{bmatrix} \end{equation}\]

are \(\textit{Lagrange muptiplier vectors}\), \(f(x)\) and \(g_i(x)\) are convex. With \(\textit{Lagrangian function} \eqref{lagrangian}\) , we denoted the \(\textit{primal problem}\) as

\[\begin{equation} \mathcal{P}: \quad \min \limits_{x} \max \limits_{\lambda, \nu } L(x, \lambda, \nu) \label{primal} \end{equation}\]

and the \(\textit{dual problem}\) as

\[\begin{equation} \mathcal{D}: \quad \max \limits_{\lambda, \nu} \min \limits_{x} L(x, \lambda, \nu) \label{dual} \end{equation}\]

\(p^*\) and \(d^*\) are the optimal values.

THEOREM 1. If \(\lambda_i \geq 0\), \(g_i(x) \leq 0\) and \(h_j(x) = 0\), we have \(\max \limits_{\lambda, \nu}L(x, \lambda, \nu) = f(x)\) and \(p^*\) is the optimal value for primal problem \(\ref{originalPrimal}\).

PROOF 1:

\[\begin{equation} \begin{aligned} & \partial L / \partial \lambda_i = g_i(x) \leq 0 \\ \Rightarrow \quad & \begin{cases} \lambda_i^* = 0, \quad g_i(x) < 0 \\ \lambda_i^* \geq 0, \quad g_i(x) = 0 \end{cases} \\ \Rightarrow \quad & \lambda_i^* g_i(x) = 0 \\ \Rightarrow \quad & \max \limits_{\lambda, \nu} L(x, \lambda, \nu) = L(x, \lambda^*, \nu^*) = f(x) \end{aligned}\end{equation}\]

THEOREM 2. If \(f(x)\) and \(g_i(x)\) are convex and continuously differentiable at a point \(x^{*}\), \(\lambda_i \geq 0\), \(g_i(x) \leq 0\) and \(h_j(x) = 0\), therefore

\[\begin{equation} p^* = d^* \end{equation}\]

PROOF 2: Suppose \((x_p^*, \lambda_p^*, \nu_p^*)\) are the solutions to primal problem \(\mathcal{P}\) \(\eqref{primal}\), same as Proof 1, we have

\[\begin{equation} \begin{aligned} & \partial \mathcal{P} / \partial \lambda_i = g_i(x) \leq 0 \\ \Rightarrow & \quad \lambda_{p,i}^* g_i(x) = 0 \\ \Rightarrow & \quad \mathcal{P} = \min \limits_{x} f(x) \\ \Rightarrow & \quad \partial f(x) / \partial x_p^* = 0 \end{aligned} \label{proof2.1} \end{equation}\]

Similarly, suppose \((x_d^*, \lambda_d^*, \nu_d^*)\) are the solutions to primal problem \(\mathcal{D}\) \(\eqref{dual}\), we have

\[\begin{equation} \begin{aligned} & \mathcal{D} = \max \limits_{\lambda} f(x_d^*) + \sum_{i=1}^{m}\lambda_{i}g_i(x_d^*) \\ \Rightarrow & \quad \partial \mathcal{D} / \partial \lambda_i = g_i(x_d^*) \leq 0 \\ \Rightarrow & \quad \lambda_{d,i}^* g_i(x_d^*) = 0 \\ \Rightarrow & \quad \mathcal{D} = f(x_d^*) \\ \Rightarrow & \quad \partial f(x) / \partial x_d^* = 0 \end{aligned} \label{proof2.2} \end{equation}\]

According to \eqref{proof2.1} and \eqref{proof2.2}, given f(x) is convex, we have

\[\begin{equation} \begin{aligned} f(x_p^*) = f(x_d^*) \quad \Leftrightarrow \quad p^* = d^* \end{aligned} \end{equation}\]

THEOREM 3. If \(f(x)\) and \(g_i(x)\) are convex and continuously differentiable at a point \(x^{*}\), \(\lambda_i \geq 0\), \(g_i(x) \leq 0\) and \(h_j(x) = 0\), therefore

\[\begin{equation} \mathcal{P_0} \ \Leftrightarrow \ \mathcal{P} \ \Leftrightarrow \ \mathcal{D} \end{equation}\]

8. References

[1] Chih-Chung Chang and Chih-Jen Lin(2001). LIBSVM: a library for support vector machines.

[2] John C. Platt(1998). Fast Training of Support Vector Machines Using Sequential Minimal Optimization.

[3] Rong-En Fan, Pai-Hsuen Chen, and Chih-Jen Lin(2005). Working Set Selection Using Second Order Information for Training Support Vector Machines.

在新西兰追尾记录

Wed, 23 May 2018 00:00:00 +0000

追尾当下

昨天收到保险公司的邮件，给出了最后的解决方案，这个case算是结束啦。这篇文章想要记录一下在新西兰发生了追尾事故及之后处理的过程，希望对你没有帮助又有帮助。

事故发生在 2018年2月11日，在 Auckland Harbour Bridge。那个路段的限度是80，当时下着暴雨所以大家开的都没有很快，车速在50-60的样子。前车突然急刹车，我们虽然刹车了但还是撞上了，后面还有一辆车跟我们发生了同样的情况，结果就是三车追尾。很幸运的是，都没有人受伤。通常情况下，在新西兰开车，车距并没有很大。如果是雨天之类的坏天气，尽量低速，特别注意保持车距哦。

后车打了电话报警，于是我们就在车里等着警察来，不知道这样子处理是否正确的。很快的，警车就来啦，带着我们开到了一个小路上，这样比较不妨碍交通。警察带着小本本各自询问车里人的信息并做记录，最后留下一个 Event number 各自保存。询问了警察我们需要做什么被告知不用做什么之后，我们就各回各家啦。在新西兰，追尾事故 100% 是后车的责任，因为是后车负责保持车距。在这个情况下，第二辆车需要对第一辆车负责，第三辆车需要对前面两辆车负责。

修车和保险

接下来就是保险和修车的事情了。

保险买了AA的第三方责任险，下次我可能会选择全险，还是要认真的看看啊~ 我们电话给我们的保险公司 make a claim，我们提供信息，并且交了前车的垫底费。由于海外驾照拿到时间不够2年，垫底费需要按800刀算。

打完电话我们决定去修车，傻傻的就出现了。我看了 AA Auto Centre 的介绍说是可以修车。于是我们就开着破车去了那里，被告知外壳坏掉了呢他们不修，需要去 panel beater。我们又开着破车去了 AA Auto Centre 给我们推荐的附近的 panel beater。等 panel beater 工作人员检查了一遍之后。我们想询问下修车费，被告知现在还不知道。panel beater 的人问我们 claim number，我们一脸懵比。啊，原来没有 claim number 是不能修车的呀。于是我们又开着破车回家了。

我们电话给 AA 问 claim number。AA 说他们联系不到后车，问我们有没有他们的信息，如果没有他们的信息等着警察走完流程的话可能需要4-6周并建议我们可以跟警察联系得到后车的信息，这样可以加速。于是我们联系了我们所在地的警察局，被告知他们不能给我们信息。

于是6周之后，我们给 AA 电话希望可以加速，但是 AA 建议我们自己去联系警察。于是我们又去联系了警察，接电话的警察非常耐心的跟我们说我们可以亲自去警察局。但是某人对于警察局有莫名的恐惧（雾，于是我们选择电话询问。电话询问后等了一周继续无果。于是我们去了当地的警察局，当面把各信息给了接待的警官，跟我说这个 case 应该被 report 了，但是并没有，所以他查不到信息。那个警官跟我说当时处理我们这个交通事故的警察会在下午1点上班，他下午应该会打你电话。我问万一没有打电话怎么办，于是被给了我们处理这个追尾的警察的电话和邮件。果然没有接到电话，第二天发了邮件询问。

终于收到邮件啦，有了后车的信息。乘胜追击电话给了 AA，被告知需要自己打电话给后车的保险，也就是BNZ insurance。于是等到了下一个工作日，给了 BNZ 电话。BNZ 的电话可能比较难打通，一定要有耐心，半个小时也等着哦。给了 BNZ 信息之后的几个工作日，终于等到了 claim number啦。 BNZ 的工作人员建议我们去他们指定的 panel beater 修车，这样只需要几个工作日，否则可能会慢很多。在这期间，警察那边终于 report 啦，我们也收到了一封 warning（2333

于是我们联系了 BNZ 指定的 panel beater 询问他们哪个时间段是合适的，比较了几家拖车公司，选了价格比较便宜的，预约了。第二天大早起来等着拖车公司的人，结果沟通有误，我们在家里等着，拖车公司的人在路边等着，彼此等了很久。打电话给了拖车公司才发现了这个误会呀~

因为长时间没发动，电池没有电啦，忘记定时去发动一下充电啦。鉴于前车盖被撞了，我们试着打开但没有打开成功，以为不能充电啦。结果拖车公司的人一下子就打开了，拿着电瓶充了下电，车子就可以开动啦。啊呀，经验不足又费了 110 刀的拖车费。

故事的结尾

经历了之前的龟速，这次上心多啦，比较及时的电话询问进展。在经过几个工作日的评估之后，我们打电话给了 panel beater 询问评估的情况，被告知已经评估好了，去联系 BNZ 就可以啦。于是我们去联系了 BNZ 说他们需要几个工作日来确定车子要不要修。几个工作日之后，我们漏接了电话但是有 voicemail 建议我们选择报废。于是我们电话给 BNZ，在经历了我的口语搞笑之后，BNZ 温柔的客服妹子说他们有专门的 manager 来处理这个事情，她并不是很清楚，我们等着那个 manager 来联系，但是会记录下我们打过电话了。

接下来的周一，又错过了 manager 的电话并收到了 voicemail 说他们会有专门的处理赔偿金钱的 manager 来联系我们。后面的那个 manager 给我们邮件说了下选择报废的方案和选择继续修车的方案，并给出了建议。

因为前车盖部分的损害是由我们自己撞了前车，和后车撞了我们导致我们再次撞了前车造成的。但是这两部分的责任比例并不能很好的鉴定，保险公司给出的方案是各自负责50%。所以就是后车盖由保险公司全部负责，前车盖保险公司和我们各自负责50%。

根据 BNZ 的建议，车子的修车费用与车子的估值相差不大，而且选择修车的话需要reregister，这个费用最高可以有2000刀，虽然我觉得费用应该没有那么高。。虽然有点舍不得，但是还是选择了报废，想着下一辆车买什么啦，心疼刚换的轮胎呀。

啊是不是还要去注销牌照并且拿剩下的路费。整个下来可能是损失了2000-3000刀，还有投入的时间精力。当然啦，所有人都没事已经足够好啦。

总结一下

总结下来就是犯了好多错误，需要好多改进：

要好好开车，保持足够的车距，特别是坏天气。
买保险的时候要认真哦，而且每次买的时候都要重新看一下哦。
在事故发生后，应该询问各车的保险信息、联系电话等等，正确的做法可以参考 https://www.aa.co.nz/insurance/car-and-vehicle-insurance/car-accident/。
当某个环节被卡住的时候，不要拖延，要积极打电话加速进程。

进程终止

Wed, 24 Jan 2018 00:00:00 +0000

1. 进程终止方式

进程终止的方式有5种正常终止方式以及3种异常终止方式。正常终止方式主要包括在 main 函数返回、调用 exit 函数等。异常终止方式包括调用 abort 、接受信号等。

不管进程如何终止，最后都会执行内核中的同一段代码，关闭进程所以打开描述符，释放它所使用大部分资源。

2. 进程终止状态

父进程可能会对子进程的执行情况感兴趣。在进程终止的时候，我们希望能够将子进程的终止状态（termination status）告知父进程。

2.1 终止状态如何产生

当使用终止函数（exit, _exit 和 _Exit）终止进程的时候，进程的退出状态（exit status）作为参数传递给终止函数，内核将退出状态转化为终止状态；
在异常终止的情况下，内核将产生一个指示其异常终止原因的终止状态。

2.2 终止状态如何传递

父子进程的终止顺序可以有先后，如何保证子进程的终止状态告知父进程？

操作系统会维护一张进程表（process table），进程表的每个条目包含了进程所有必须的信息。当进程终止的时候，进程占用的大部分资源都被释放了，但仍保留了少量的信息，包括进程的 ID 、进程的终止状态码、以及进程所使用的 CPU 时间总量。

当子进程先于父进程终止时，父进程可以通过 wait 和 waitpid 等函数获取内核保留的终止状态。

当父进程先于子进程终止时，子进程会被 init 进程领养，init 进程称为该进程的父进程。而 init 进程无论何时，只要有一个子进程终止，都会调用 wait 函数来获取其终止状态。

3. zombie 进程

zombie 进程是指一个已经终止，但其父进程尚未对其进行善后处理的进程。因为内核还保留着关于该进程的终止状态等信息，所以该进程仍然占用资源。

那么问题是，为什么要确保除了交换进程（swapper process，PID=0）以外，每个进程都有一个父进程呢？想一想哦~

4. 笔记来源

Advanced Programming in the UNIX Environment, 3rd Edition

ainesmile

Mean Shift

1. Kernel Density Estimation

1.1 Interpretation

1.2 Bandwidth

2. The Mean Shift

2.1 Equations

2.2 The Mean Shift Procedure

2.3 Adaptive Gradient Ascent

3. A Numpy Python Implementation For Gaussian Mean Shift Clustering

4. References

Affinity Propagation

1. Before Starting

2. Notions

3. How To Understand?

4. When To Converge And How To Find Exemplars?

5. A Numpy Python Implementation

6. References

A Simple Note for Gaussian Mixture Models

1. Gaussian Mixture Model

2. Bayes and Maximum Likelihood Estimation

3. Expectation—Maximization Algorithm

4. GMM with EM

5. A numpy python implementation

A Simple Note for k-means clustering

1. What is k-means clustering

2. Initialization

2.1 k-means++

3. A numpy python implementation

How to Select Data In Pandas

1. Prepare Data

2. Selecting Data By Label

2.1 []

2.2 .loc[]

2.3 .get()

2.4 .at()

2.5 .xs()

2.6 .lookup()

2.7 .filter()

2.8 Summary Table Of Selecting By Label

3. Selecting Data By Position

3.1 []

3.2 .head() and .tail()

3.3 .iloc[]

3.4 .iat[]

3.5 .take()

4. Selecting Data By Value

4.1 Selecting By Bool

4.2 Selecting By Value Order

4.3 Selecting By Datatime

5. Selection Data By Dtypes

NumPy: Indexing

1. Memory Layout of ndarray and Strides

2. View and Copy

3. Indexing Styles(Orders)

4. Array Indexing

4.1 Basic Indexing

4.1.1 Slice object

4.1.2 Newaxis and Ellipsis

4.2 Advanced Indexing

4.2.1 Integer Array Indexing

4.2.2 Boolean Array Indexing

4.2.3 Combing Integer and Boolean Array Indexing

4.3 Combing Basic and Advanced Indexing

4.4 Summary Table

NumPy: Joining Arrays Together

1. np.stack

2. np.concatenate

3. np.vstack

4. np.hstack

5. np.dstack

6. np.column_stack

7. np.block

8. summary

A Super Boring Guide To Support Vector Machines

1. Overview

2. Notations

3. SVM: hard margin

4. SVM: soft margin

5. Kernels

2.1 `[]`

2.2 `.loc[]`

2.3 `.get()`

2.4 `.at()`

2.5 `.xs()`

2.6 `.lookup()`

2.7 `.filter()`

3.1 `[]`

3.2 `.head()` and `.tail()`

3.3 `.iloc[]`

3.4 `.iat[]`

3.5 `.take()`