5,489 machine learning methods and techniques
Meta-augmentation helps generate more varied tasks for a single example in meta-learning. It can be distinguished from data augmentation in classic machine learning as follows. For data augmentation in classical machine learning, the aim is to generate more varied examples, within a single task. Meta-augmentation has the exact opposite aim: we wish to generate more varied tasks, for a single example, to force the learner to quickly learn a new task from feedback. In meta-augmentation, adding randomness discourages the base learner and model from learning trivial solutions that do not generalize to new tasks.
Adaptive Nesterov Momentum
Please enter a description about the method here
Teacher-Tutor-Student Knowledge Distillation is a method for image virtual try-on models. It treats fake images produced by the parser-based method as "tutor knowledge", where the artifacts can be corrected by real "teacher knowledge", which is extracted from the real person images in a self-supervised way. Other than using real images as supervisions, knowledge distillation is formulated in the try-on problem as distilling the appearance flows between the person image and the garment image, enabling the finding of dense correspondences between them to produce high-quality results.
Decorrelated Batch Normalization (DBN) is a normalization technique which not just centers and scales activations but whitens them. ZCA whitening instead of PCA whitening is employed since PCA whitening causes a problem called stochastic axis swapping, which is detrimental to learning.
SimCLRv2 is a semi-supervised learning method for learning from few labeled examples while making best use of a large amount of unlabeled data. It is a modification of a recently proposed contrastive learning framework, SimCLR. It improves upon it in three major ways: 1. To fully leverage the power of general pre-training, larger ResNet models are explored. Unlike SimCLR and other previous work, whose largest model is ResNet-50 (4×), SimCLRv2 trains models that are deeper but less wide. The largest model trained is a 152 layer ResNet with 3× wider channels and selective kernels (SK), a channel-wise attention mechanism that improves the parameter efficiency of the network. By scaling up the model from ResNet-50 to ResNet-152 (3×+SK), a 29% relative improvement is obtained in top-1 accuracy when fine-tuned on 1% of labeled examples. 2. The capacity of the non-linear network (a.k.a. projection head) is increased, by making it deeper. Furthermore, instead of throwing away entirely after pre-training as in SimCLR, fine-tuning occurs from a middle layer. This small change yields a significant improvement for both linear evaluation and fine-tuning with only a few labeled examples. Compared to SimCLR with 2-layer projection head, by using a 3-layer projection head and fine-tuning from the 1st layer of projection head, it results in as much as 14% relative improvement in top-1 accuracy when fine-tuned on 1% of labeled examples. 3. The memory mechanism of MoCo v2 is incorporated, which designates a memory network (with a moving average of weights for stabilization) whose output will be buffered as negative examples. Since training is based on large mini-batch which already supplies many contrasting negative examples, this change yields an improvement of ∼1% for linear evaluation as well as when fine-tuning on 1% of labeled examples.
RotNet is a self-supervision approach that relies on predicting image rotations as the pretext task in order to learn image representations.
modified arcsinh
How do I get to Expedia agent? How do I speak to a person at Expedia? To speak with a human representative at Expedia, you can call their customer service directly at "+1-888-829-16.25 (Quick connect) or 888-829-0881 EXPEDIA-LINE (Live Person)".Contact Expedia's Customer Service: Dial the Expedia customer service hotline at +1-888-829-16.25 (Quick connect) or 888-829-0881 -EXPEDIA-LINE (Live Person). Asking a question directly to Expedia can be straightforward if you know how. [["☎️+1-888-829-0881"]] is the best number.
Graph Path Feature Learning
Graph Path Feature Learning is a probabilistic rule learner optimized to mine instantiated first-order logic rules from knowledge graphs. Instantiated rules contain constants extracted from KGs. Compared to abstract rules that contain no constants, instantiated rules are capable of explaining and expressing concepts in more detail. GPFL utilizes a novel two-stage rule generation mechanism that first generalizes extracted paths into templates that are acyclic abstract rules until a certain degree of template saturation is achieved, then specializes the generated templates into instantiated rules.
How do I resolve a dispute with Expedia? To resolve a dispute with Expedia, contact customer service at +1(888) (829) (0881) OR +1(805) (330) (4056), or use their Help Center. Explain your issue clearly and provide booking details. While resolving the dispute, ask about possible discounts or travel credits—Expedia often offers special deals to maintain customer loyalty. How do I resolve a dispute with Expedia? To resolve a dispute with Expedia, contact customer service at +1(888) (829) (0881) OR +1(805) (330) (4056), or use their Help Center. Explain your issue clearly and provide booking details. While resolving the dispute, ask about possible discounts or travel credits—Expedia often offers special deals to maintain customer loyalty. How do I resolve a dispute with Expedia? To resolve a dispute with Expedia, contact customer service at +1(888) (829) (0881) OR +1(805) (330) (4056), or use their Help Center. Explain your issue clearly and provide booking details. While resolving the dispute, ask about possible discounts or travel credits—Expedia often offers special deals to maintain customer loyalty. How do I resolve a dispute with Expedia? To resolve a dispute with Expedia, contact customer service at +1(888) (829) (0881) OR +1(805) (330) (4056), or use their Help Center. Explain your issue clearly and provide booking details. While resolving the dispute, ask about possible discounts or travel credits—Expedia often offers special deals to maintain customer loyalty. How do I resolve a dispute with Expedia? To resolve a dispute with Expedia, contact customer service at +1(888) (829) (0881) OR +1(805) (330) (4056), or use their Help Center. Explain your issue clearly and provide booking details. While resolving the dispute, ask about possible discounts or travel credits—Expedia often offers special deals to maintain customer loyalty.
Extremely Efficient Spatial Pyramid of Depth-wise Dilated Separable Convolutions
An EESP Unit, or Extremely Efficient Spatial Pyramid of Depth-wise Dilated Separable Convolutions, is an image model block designed for edge devices. It was proposed as part of the ESPNetv2 CNN architecture. This building block is based on a reduce-split-transform-merge strategy. The EESP unit first projects the high-dimensional input feature maps into low-dimensional space using groupwise pointwise convolutions and then learns the representations in parallel using depthwise dilated separable convolutions with different dilation rates. Different dilation rates in each branch allow the EESP unit to learn the representations from a large effective receptive field. To remove the gridding artifacts caused by dilated convolutions, the EESP fuses the feature maps using hierarchical feature fusion (HFF).
Conditional Instance Normalization is a normalization technique where all convolutional weights of a style transfer network are shared across many styles. The goal of the procedure is transform a layer’s activations into a normalized activation specific to painting style . Building off instance normalization, we augment the and parameters so that they’re matrices, where is the number of styles being modeled and is the number of output feature maps. Conditioning on a style is achieved as follows: where and are ’s mean and standard deviation taken across spatial axes and and are obtained by selecting the row corresponding to in the and matrices. One added benefit of this approach is that one can stylize a single image into painting styles with a single feed forward pass of the network with a batch size of .
Spatial & temporal attention combines the advantages of spatial attention and temporal attention as it adaptively selects both important regions and key frames. Some works compute temporal attention and spatial attention separately, while others produce joint spatio & temporal attention maps. Further works focusing on capturing pairwise relations.
Randomized Leaky Rectified Linear Units
Randomized Leaky Rectified Linear Units, or RReLU, are an activation function that randomly samples the negative slope for activation values. It was first proposed and used in the Kaggle NDSB Competition. During training, is a random number sampled from a uniform distribution . Formally: where In the test phase, we take average of all the in training similar to dropout, and thus set to to get a deterministic result. As suggested by the NDSB competition winner, is sampled from . At test time, we use:
TaBERT is a pretrained language model (LM) that jointly learns representations for natural language sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In summary, TaBERT's process for learning representations for NL sentences is as follows: Given an utterance and a table , TaBERT first creates a content snapshot of . This snapshot consists of sampled rows that summarize the information in most relevant to the input utterance. The model then linearizes each row in the snapshot, concatenates each linearized row with the utterance, and uses the concatenated string as input to a Transformer model, which outputs row-wise encoding vectors of utterance tokens and cells. The encodings for all the rows in the snapshot are fed into a series of vertical self-attention layers, where a cell representation (or an utterance token representation) is computed by attending to vertically-aligned vectors of the same column (or the same NL token). Finally, representations for each utterance token and column are generated from a pooling layer.
TURL: Table Understanding through Representation Learning
Relational tables on the Web store a vast amount of knowledge. Owing to the wealth of such tables, there has been tremendous progress on a variety of tasks in the area of table understanding. However, existing work generally relies on heavily-engineered task- specific features and model architectures. In this paper, we present TURL, a novel framework that introduces the pre-training/fine- tuning paradigm to relational Web tables. During pre-training, our framework learns deep contextualized representations on relational tables in an unsupervised manner. Its universal model design with pre-trained representations can be applied to a wide range of tasks with minimal task-specific fine-tuning. Specifically, we propose a structure-aware Transformer encoder to model the row-column structure of relational tables, and present a new Masked Entity Recovery (MER) objective for pre-training to capture the semantics and knowledge in large-scale unlabeled data. We systematically evaluate TURL with a benchmark consisting of 6 different tasks for table understanding (e.g., relation extraction, cell filling). We show that TURL generalizes well to all tasks and substantially outperforms existing methods in almost all instances.
Polyak Averaging is an optimization technique that sets final parameters to an average of (recent) parameters visited in the optimization trajectory. Specifically if in iterations we have parameters , then Polyak Averaging suggests setting Image Credit: Shubhendu Trivedi & Risi Kondor
DetNAS is a neural architecture search algorithm for the design of better backbones for object detection. It is based on the technique of one-shot supernet, which contains all possible networks in the search space. The supernet is trained under the typical detector training schedule: ImageNet pre-training and detection fine-tuning. Then, the architecture search is performed on the trained supernet, using the detection task as the guidance. DetNAS uses evolutionary search as opposed to RL-based methods or gradient-based methods.
Attention-augmented Convolution is a type of convolution with a two-dimensional relative self-attention mechanism that can replace convolutions as a stand-alone computational primitive for image classification. It employs scaled-dot product attention and multi-head attention as with Transformers. It works by concatenating convolutional and attentional feature map. To see this, consider an original convolution operator with kernel size , input filters and output filters. The corresponding attention augmented convolution can be written as" originates from an input tensor of shape . This is flattened to become which is passed into a multi-head attention module, as well as a convolution (see above). Similarly to the convolution, the attention augmented convolution 1) is equivariant to translation and 2) can readily operate on inputs of different spatial dimensions.
Kernel Inducing Points
Kernel Inducing Points, or KIP, is a meta-learning algorithm for learning datasets that can mitigate the challenges which occur for naturally occurring datasets without a significant sacrifice in performance. KIP uses kernel-ridge regression to learn -approximate datasets. It can be regarded as an adaption of the inducing point method for Gaussian processes to the case of Kernel Ridge Regression.
Leverage learning suggests that it is possible to strategically use minimal task-specific data to enhance task-specific capabilities, while non-specific capabilities can be learned from more general data.
QuantTree histograms
Given a training set drawn from an unknown -variate probability distribution, QuantTree constructs a histogram by recursively splitting . The splits are defined by a stochastic process so that each bin contains a certain proportion of the training set. These histograms can be used to define test statistics (e.g., the Pearson statistic) to tell whether a batch of data is drawn from or not. The most crucial property of QuantTree is that the distribution of any statistic based on QuantTree histograms is independent of , thus enabling nonparametric statistical testing.
Batch Nuclear-norm Maximization is an approach for aiding classification in label insufficient situations. It involves maximizing the nuclear-norm of the batch output matrix. The nuclear-norm of a matrix is an upper bound of the Frobenius-norm of the matrix. Maximizing nuclear-norm ensures large Frobenius-norm of the batch matrix, which leads to increased discriminability. The nuclear-norm of the batch matrix is also a convex approximation of the matrix rank, which refers to the prediction diversity.
Semantic Clustering by Adopting Nearest Neighbours
SCAN automatically groups images into semantically meaningful clusters when ground-truth annotations are absent. SCAN is a two-step approach where feature learning and clustering are decoupled. First, a self-supervised task is employed to obtain semantically meaningful features. Second, the obtained features are used as a prior in a learnable clustering approach. Image source: Gansbeke et al.
Continuously Indexed Domain Adaptation
Continuously Indexed Domain Adaptation combines traditional adversarial adaptation with a novel discriminator that models the encoding-conditioned domain index distribution. Image Source: Wang et al.
Hermite Polynomial Activation
A Hermite Activations is a type of activation function which uses a smooth finite Hermite polynomial base as a substitute for non-smooth ReLUs. Relevant Paper: Lokhande et al
Gradient Sign Dropout
GradDrop, or Gradient Sign Dropout, is a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency. It is applied as a layer in any standard network forward pass, usually on the final layer before the prediction head to save on compute overhead and maximize benefits during backpropagation. Below, we develop the GradDrop formalism. Throughout, o denotes elementwise multiplication after any necessary tiling operations (if any) are completed. To implement GradDrop, we first define the Gradient Positive Sign Purity, , as is bounded by For multiple gradient values at some scalar , we see that if , while if . Thus, is a measure of how many positive gradients are present at any given value. We then form a mask for each gradient as follows: for the standard indicator function and some monotonically increasing function (often just the identity) that maps and is odd around . is a tensor composed of i.i.d random variables. The is then used to produce a final gradient
CRISS, or Cross-lingual Retrievial for Iterative Self-Supervised Training (CRISS), is a self-supervised learning method for multilingual sequence generation. CRISS is developed based on the finding that the encoder outputs of multilingual denoising autoencoder can be used as language agnostic representation to retrieve parallel sentence pairs, and training the model on these retrieved sentence pairs can further improve its sentence retrieval and translation capabilities in an iterative manner. Using only unlabeled data from many different languages, CRISS iteratively mines for parallel sentences across languages, trains a new better multilingual model using these mined sentence pairs, mines again for better parallel sentences, and repeats.
A Hopfield Layer is a module that enables a network to associate two sets of vectors. This general functionality allows for transformer-like self-attention, for decoder-encoder attention, for time series prediction (maybe with positional encoding), for sequence analysis, for multiple instance learning, for learning with point sets, for combining data sources by associations, for constructing a memory, for averaging and pooling operations, and for many more. In particular, the Hopfield layer can readily be used as plug-in replacement for existing layers like pooling layers (max-pooling or average pooling, permutation equivariant layers, GRU & LSTM layers, and attention layers. The Hopfield layer is based on modern Hopfield networks with continuous states that have very high storage capacity and converge after one update.
Source Hypothesis Transfer, or SHOT, is a representation learning framework for unsupervised domain adaptation. SHOT freezes the classifier module (hypothesis) of the source model and learns the target-specific feature extraction module by exploiting both information maximization and self-supervised pseudo-labeling to implicitly align representations from the target domains to the source hypothesis.
Contrastive Video Representation Learning
Contrastive Video Representation Learning, or CVRL, is a self-supervised contrastive learning framework for learning spatiotemporal visual representations from unlabeled videos. Representations are learned using a contrastive loss, where two clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away. Data augmentations are designed involving spatial and temporal cues. Concretely, a temporally consistent spatial augmentation method is used to impose strong spatial augmentations on each frame of the video while maintaining the temporal consistency across frames. A sampling-based temporal augmentation method is also used to avoid overly enforcing invariance on clips that are distant in time. End-to-end, from a raw video, we first sample a temporal interval from a monotonically decreasing distribution. The temporal interval represents the number of frames between the start points of two clips, and we sample two clips from a video according to this interval. Afterwards we apply a temporally consistent spatial augmentation to each of the clips and feed them into a 3D backbone with an MLP head. The contrastive loss is used to train the network to attract the clips from the same video and repel the clips from different videos in the embedding space.
TILDEv2 is a BERT-based re-ranking method that stems from TILDE but that addresses its limitations. It relies on contextualized exact term matching with expanded passages. This requires to only store in the index the score of tokens that appear in the expanded passages (rather than all the vocabulary), thus producing indexes that are 99% smaller than those of the original. Specifically, TILDE is modified in the following aspects: - Exact Term Matching. The query likelihood matching originally employed in TILDE, expands passages into the BERT vocabulary size, resulting in large indexes. To overcome this issue, estimating relevance scores is achieved with contextualized exact term matching. This allows the model to index tokens only present in the passage, thus reducing the index size. In addition to this, we replace the query likelihood loss function, with the Noise contrastive estimation (NCE) loss that allows to better leverage negative training samples. - Passage Expansion. To overcome the vocabulary mismatch problem that affects exact term matching methods, passage expansion is used to expand the original passage collection. Passages in the collection are expanded using deep LMs with a limited number of tokens. This requires TILDEv2 to only index a few extra tokens in addition to those in the original passages.
Adaptive Content Generating and Preserving Network
ACGPN, or Adaptive Content Generating and Preserving Network, is a generative adversarial network for virtual try-on clothing applications. In Step I, the Semantic Generation Module (SGM) takes the target clothing image , the pose map , and the fused body part mask as the input to predict the semantic layout and to output the synthesized body part mask and the target clothing mask . In Step II, the Clothes Warping Module (CWM) warps the target clothing image to according to the predicted semantic layout, where a second-order difference constraint is introduced to stabilize the warping process. In Steps III and IV, the Content Fusion Module (CFM) first produces the composited body part mask using the original clothing mask , the synthesized clothing mask , the body part mask , and the synthesized body part mask , and then exploits a fusion network to generate the try-on images by utilizing the information , , and the body part image from previous steps.
Channel Squeeze and Spatial Excitation (sSE)
Inspired on the widely known spatial squeeze and channel excitation (SE) block, the sSE block performs channel squeeze and spatial excitation, to recalibrate the feature maps spatially and achieve more fine-grained image segmentation.
ComiRec is a multi-interest framework for sequential recommendation. The multi-interest module captures multiple interests from user behavior sequences, which can be exploited for retrieving candidate items from the large-scale item pool. These items are then fed into an aggregation module to obtain the overall recommendation. The aggregation module leverages a controllable factor to balance the recommendation accuracy and diversity.
Online Normalization is a normalization technique for training deep neural networks. To define Online Normalization. we replace arithmetic averages over the full dataset in with exponentially decaying averages of online samples. The decay factors and for forward and backward passes respectively are hyperparameters for the technique. We allow incoming samples , such as images, to have multiple scalar components and denote feature-wide mean and variance by and . The algorithm also applies to outputs of fully connected layers with only one scalar output per feature. In fact, this case simplifies to and . Denote scalars and to denote running estimates of mean and variance across all samples. The subscript denotes time steps corresponding to processing new incoming samples. Online Normalization uses an ongoing process during the forward pass to estimate activation means and variances. It implements the standard online computation of mean and variance generalized to processing multi-value samples and exponential averaging of sample statistics. The resulting estimates directly lead to an affine normalization transform.
DCN-V2 is an architecture for learning-to-rank that improves upon the original DCN model. It first learns explicit feature interactions of the inputs (typically the embedding layer) through cross layers, and then combines with a deep network to learn complementary implicit interactions. The core of DCN-V2 is the cross layers, which inherit the simple structure of the cross network from DCN, however it is significantly more expressive at learning explicit and bounded-degree cross features.
Attention Free Transformer, or AFT, is an efficient variant of a multi-head attention module that eschews dot product self attention. In an AFT layer, the key and value are first combined with a set of learned position biases, the result of which is multiplied with the query in an element-wise fashion. This new operation has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible to both large input and model sizes. Given the input , AFT first linearly transforms them into , then performs following operation: where is the element-wise product; is the nonlinearity applied to the query with default being sigmoid; is the learned pair-wise position biases. Explained in words, for each target position , AFT performs a weighted average of values, the result of which is combined with the query with element-wise multiplication. In particular, the weighting is simply composed of the keys and a set of learned pair-wise position biases. This provides the immediate advantage of not needing to compute and store the expensive attention matrix, while maintaining the global interactions between query and values as MHA does.
ShakeDrop regularization extends Shake-Shake regularization and can be applied not only to ResNeXt but also ResNet, WideResNet, and PyramidNet. The proposed ShakeDrop is given as where is a Bernoulli random variable with probability given by the linear decay rule in each layer, and and are independent uniform random variables in each element. The most effective ranges of and were experimentally found to be different from those of Shake-Shake, and are = 0, and , .
How Do I Get a Human at Expedia Immediately? Need to speak to a real person at Expedia? Skip the wait and call +1-(888)-829-0881 (OTA) to reach Expedia customer service immediately +1-(888)-829-0881 . When you're facing a travel emergency — like a canceled flight, lost hotel reservation, or a last-minute booking issue — automated responses just won’t cut it. If you’re wondering how to get a human at Expedia immediately +1-(888)-829-0881 , you’re not alone. Many travelers look for direct help when automated systems can’t solve their problem. Here’s exactly how to talk to a real person at Expedia right away — including the fastest contact methods and key support tips. +1-(888)-829-0881 1. Call Expedia Customer Service Directly +1-(888)-829-0881 The fastest and most effective way to speak with a human at Expedia is by calling their customer support number: Expedia Phone Number (OTA): +1-(888)-829-0881 This line is available 24/7 and is best used for urgent travel issues such as: ● Last-minute flight changes ● Hotel problems at check-in ● Canceled reservations ● Unexpected charges ● Refund delays or disputes 2. Use Expedia’s “Live Chat” and Ask for a Representative +1-(888)-829-0881 If you’re unable to call, another fast way to reach a real person is through Expedia’s Live Chat feature. Here’s how: 1. Visit the Expedia Help Center 2. Click on “Chat with us” 3. Type your issue and ask directly for a human agent or "talk to a representative" If the virtual assistant cannot resolve your problem, it will connect you to a live Expedia agent. +1-(888)-829-0881 3. Use the Expedia Mobile App to Contact Support +1-(888)-829-0881 On the go? You can connect with a human through the Expedia app: ● Open the app and log in ● Go to “Trips” and select the booking ● Tap “Help” or “Contact Support” ● Request a call-back or chat with a live agent This method is convenient if you’re already traveling or don’t have access to a desktop browser. 4. When to Use Social Media to Get a Faster Response +1-(888)-829-0881 If you're not getting a reply via phone or chat, you can sometimes get the attention of Expedia’s support team +1-(888)-829-0881 by reaching out on Twitter (@Expedia) or Facebook. Politely explain your issue and include your itinerary number in a private message. While this isn’t the official method for live support, social media can be useful for escalating unresolved issues. Final Tips to Get a Human Fast at Expedia +1-(888)-829-0881 To get a real person at Expedia without delays: ● Call +1-(888)-829-0881 (OTA) for 24/7 support ● Use live chat and request a human agent ● Reach out through the Expedia mobile app ● Try social media if standard channels fail Whether you’re dealing with a missed flight, hotel error, or urgent refund, speaking to a real person at Expedia can help resolve your issue faster and with more clarity than any automated system. Related Search Phrases: +1-(888)-829-0881 ● How do I talk to a real person at Expedia? ● Expedia human customer support ● Speak with Expedia live agent ● Expedia contact number for emergencies ● Bypass Expedia automated system How Do I Get a Human at Expedia Immediately? Need urgent help with your Expedia booking? To speak with a real person at Expedia, call +1-(888)-829-0881 (OTA) — available 24/7. Other quick ways to reach a human: ● Use Live Chat via the Expedia Help Center and type “talk to a representative” ● Contact support through the Expedia mobile app under your trip details Best for help with refunds, flight changes, hotel issues, or billing disputes +1-(888)-829-0881 .To speak to a human at Expedia, call their customer service line at +1-888-829-0881 or +1-805-330-4056 . When prompted by the automated system, try saying "speak to a representative" or press zero repeatedly. Be patient, as call volumes can be high. Once connected, clearly state your needs. How do I lodge a complaint against Expedia? To lodge a complaint against Expedia, contact their customer service at +1-888-829-0881 or +1-805-330-4056 . Clearly outline your complaint, providing all relevant booking details and any supporting documentation. Note the date, time, and name of any representative you speak with. If unsatisfied with the initial response, ask to escalate your complaint. Keep thorough records. To talk directly with a live agent at Expedia, call their customer service at +1-888-829-0881 or +1-805-330-4056 and choose the option to speak with a representative—usually with no wait time. Alternatively, you can use the live chat available on their website or mobile app and ask to connect with a human agent for personalized assistance. Publications. To reach a human at Expedia, call their customer support number +1-888-829-0881 or +1-805-330-4056or use the live chat option on their website. +1-888-829-0881 or +1-805-330-4056During the chat, request to speak with an agent. You can also try messaging them on social media for quicker assistance or escalate through their support team. +1-888-829-0881 or +1-805-330-4056.
PowerSGD is a distributed optimization technique that computes a low-rank approximation of the gradient using a generalized power iteration (known as subspace iteration). The approximation is computationally light-weight, avoiding any prohibitively expensive Singular Value Decomposition. To improve the quality of the efficient approximation, the authors warm-start the power iteration by reusing the approximation from the previous optimization step.
Aging Evolution, or Regularized Evolution, is an evolutionary algorithm for neural architecture search. Whereas in tournament selection, the best architectures are kept, in aging evolution we associate each genotype with an age, and bias the tournament selection to choose the younger genotypes. In the context of architecture search, aging evolution allows us to explore the search space more, instead of zooming in on good models too early, as non-aging evolution would.
Accuracy-Robustness Area
In the space of adversarial perturbation against classifier accuracy, the ARA is the area between a classifier's curve and the straight line defined by a naive classifier's maximum accuracy. Intuitively, the ARA measures a combination of the classifier’s predictive power and its ability to overcome an adversary. Importantly, when contrasted against existing robustness metrics, the ARA takes into account the classifier’s performance against all adversarial examples, without bounding them by some arbitrary .
This method proposes first discretizing observations and calculating the action distribution distance under comparable cases (intersection states).
Gated Positional Self-Attention
Gated Positional Self-Attention (GPSA) is a self-attention module for vision transformers, used in the ConViT architecture, that can be initialized as a convolutional layer -- helping a ViT learn inductive biases about locality.
Seesaw Loss is a loss function for long-tailed instance segmentation. It dynamically re-balances the gradients of positive and negative samples on a tail class with two complementary factors: mitigation factor and compensation factor. The mitigation factor reduces punishments to tail categories w.r.t the ratio of cumulative training instances between different categories. Meanwhile, the compensation factor increases the penalty of misclassified instances to avoid false positives of tail categories. The synergy of the two factors enables Seesaw Loss to mitigate the overwhelming punishments on tail classes as well as compensate for the risk of misclassification caused by diminished penalties. Here works as a tunable balancing factor between different classes. By a careful design of , Seesaw loss adjusts the punishments on class j from positive samples of class . Seesaw loss determines by a mitigation factor and a compensation factor, as: The mitigation factor decreases the penalty on tail class according to a ratio of instance numbers between tail class and head class . The compensation factor increases the penalty on class whenever an instance of class is misclassified to class .
PipeDream-2BW is an asynchronous pipeline parallel method that supports memory-efficient pipeline parallelism, a hybrid form of parallelism that combines data and model parallelism with input pipelining. PipeDream-2BW uses a novel pipelining and weight gradient coalescing strategy, combined with the double buffering of weights, to ensure high throughput, low memory footprint, and weight update semantics similar to data parallelism. In addition, PipeDream2BW automatically partitions the model over the available hardware resources, while respecting hardware constraints such as memory capacities of accelerators, and topologies and bandwidths of interconnects. PipeDream-2BW also determines when to employ existing memory-savings techniques, such as activation recomputation, that trade off extra computation for lower memory footprint. The two main features are a double-buffered weight update (2BW) and flush mechanisms ensure high throughput. PipeDream-2BW splits models into stages over multiple workers, and each stage is replicated an equal number of times (with data-parallel updates across replicas of the same stage). Such parallel pipelines work well for models where each layer is repeated a fixed number of times (e.g., transformer models).
Branch attention can be seen as a dynamic branch selection mechanism: which to pay attention to, used with a multi-branch structure.