Q8Artificial Intelligence Techniques

(a) Support Vector Machine

A Support Vector Machine (SVM) is a powerful and widely-used supervised learning algorithm, primarily applied to binary classification problems, which seeks to find the optimal separating hyperplane between two classes of labeled training data, specifically choosing the hyperplane that maximizes the margin (the perpendicular distance) between the hyperplane and the nearest training data points of each class (the support vectors), as discussed in detail in the corresponding earlier answer on positive and negative margin.

Linear SVM formulation: for linearly separable training data, the SVM seeks weight vector w and bias b defining the separating hyperplane w.x+b=0, chosen to maximize the margin 2/||w|| subject to the constraint that every training example is correctly classified with at least unit margin, formulated as the constrained optimization problem: minimize (1/2)||w||^2, subject to yi(w.xi+b)>=1 for all training examples i, where yi is the class label (+1 or -1) of training example xi.

Soft-margin extension: for training data that is not perfectly linearly separable (the more common, realistic case), the soft-margin SVM formulation introduces slack variables permitting a controlled degree of margin violation or misclassification for individual training points, with a regularization parameter (commonly denoted C) controlling the trade-off between maximizing the margin width and minimizing the total penalty incurred by these margin violations — a larger C value penalizes margin violations more heavily (potentially leading to a narrower margin that more strictly fits the training data, with increased risk of overfitting), while a smaller C value tolerates more margin violations in exchange for a wider, potentially better-generalizing margin.

Kernel trick for non-linear classification: many real-world classification problems are not linearly separable in their original input feature space, but SVMs can be extended to handle such non-linear classification problems through the kernel trick — rather than explicitly transforming the input data into some higher-dimensional feature space where linear separation might become possible (which could be computationally very expensive, or even involve an infinite-dimensional feature space), the kernel trick replaces the explicit dot-product computation in the SVM's optimization formulation with a kernel function K(xi,xj) that implicitly computes the equivalent dot product as if the data had already been transformed into this higher-dimensional space, without ever needing to explicitly perform that transformation. Commonly used kernel functions include the polynomial kernel and the Radial Basis Function (RBF/Gaussian) kernel, each implicitly corresponding to a different, often very high-dimensional (or even infinite-dimensional, for the RBF kernel) feature space, allowing the SVM to learn highly flexible, non-linear decision boundaries in the original input space while still solving a computationally tractable convex optimization problem in the underlying (never explicitly computed) transformed feature space.

(b) Supervised vs Unsupervised Learning

Supervised learning trains a model using a labeled training dataset, consisting of input examples each paired with a known, correct target output (label) — the learning algorithm's objective is to learn a mapping function from inputs to outputs that generalizes well to new, previously unseen input examples, evaluated by how accurately it predicts the correct output for these new inputs. Supervised learning tasks are further divided into classification (predicting a discrete class label, such as determining whether an email is spam or not-spam) and regression (predicting a continuous numerical value, such as forecasting electrical load demand), with linear regression and support vector machines, both discussed elsewhere in this paper, being representative supervised learning algorithms.

Unsupervised learning, by contrast, works with a training dataset consisting only of input examples, with no corresponding target output labels provided at all — the learning algorithm's objective instead is to discover inherent structure, patterns, or groupings present within the unlabeled data itself, without any predefined notion of a 'correct' output to predict. Common unsupervised learning tasks include clustering (grouping similar data points together into clusters, such as the widely-used k-means clustering algorithm), dimensionality reduction (finding a lower-dimensional representation of the data that preserves its most important structure or variance, such as Principal Component Analysis), and anomaly/outlier detection (identifying data points that differ significantly from the general pattern of the rest of the dataset).

Key Differences

• Availability of labeled data: supervised learning requires a labeled training dataset (with known, correct target outputs provided for each training example), which can be expensive and time-consuming to obtain, particularly for large datasets requiring manual human labeling — unsupervised learning requires only unlabeled data, which is generally far more readily and cheaply available.
• Learning objective: supervised learning explicitly learns to predict a specific target output value, with performance directly measurable by comparing predictions against known correct answers — unsupervised learning instead seeks to discover useful, often more open-ended structural patterns in the data, without any single, objectively 'correct' target output against which to directly measure performance, making evaluation of unsupervised learning results often somewhat more subjective or task-dependent.
• Typical applications: supervised learning is used when the specific prediction task and the availability of correctly labeled training examples are both well defined in advance (such as fault classification, load forecasting, or image recognition with labeled training images), while unsupervised learning is used for exploratory data analysis, discovering natural groupings or segments within data (such as customer segmentation, or grouping similar operational patterns in a power system without predefined categories), and as a preprocessing/feature-learning step that can sometimes improve the performance of a subsequent supervised learning stage.
• Semi-supervised and reinforcement learning as intermediate/complementary categories: it is worth noting that semi-supervised learning (using a combination of a small amount of labeled data together with a larger amount of unlabeled data) and reinforcement learning (learning through trial-and-error interaction with an environment guided by reward feedback, rather than from a fixed, pre-collected dataset of either labeled or unlabeled examples) represent further important categories of machine learning that do not fit cleanly into either the purely supervised or purely unsupervised classification, reflecting the broader diversity of learning paradigms available within the overall field of machine learning.

Beyond the linear-separability case, Support Vector Machines extend naturally to non-linearly-separable classification problems through the kernel trick, implicitly mapping the original input features into a much higher-dimensional feature space (without ever explicitly computing the coordinates in that higher-dimensional space, which would often be computationally prohibitive) where a linear separating hyperplane can be found, using kernel functions such as the polynomial kernel or the radial basis function (RBF) kernel to compute the necessary inner products directly in the original input space - this kernel-trick capability is a major reason SVMs remained one of the most powerful and widely used classification algorithms in machine learning practice for many years before the more recent rise of deep neural networks for large-scale, high-dimensional classification tasks.