Fine-tuning pre-trained models has been ubiquitously proven to be effective in a wide range of NLP tasks. However, fine-tuning the whole model is parameter inefficient as it always yields an entirely new model for each task. Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks. These methods achieve surprisingly good performance and are shown to be more stable than their corresponding fully fine-tuned counterparts. However, such kind of methods is still not well understood. Some natural questions arise: How does the parameter sparsity lead to promising performance? Why is the model more stable than the fully fine-tuned models? How to choose the tunable parameters? In this paper, we first categorize the existing methods into random approaches, rule-based approaches, and projection-based approaches based on how they choose which parameters to tune. Then, we show that all of the methods are actually sparse fine-tuned models and conduct a novel theoretical analysis of them. We indicate that the sparsity is actually imposing a regularization on the original model by controlling the upper bound of the stability. Such stability leads to better generalization capability which has been empirically observed in a lot of recent research works. On the other hand, the random and rule-based methods do not utilize task-specific data information while the projection-based approaches suffer from the projection discontinuity problem. To solve these problems, we propose a novel Second-order Approximation Method (SAM) which approximates the original problem with an analytically solvable optimization function. The tunable parameters are determined by directly optimizing the approximation function. We conduct extensive experiments on several tasks. The experimental results show that our proposed SAM model outperforms many strong baseline models and it also verifies our theoretical analysis.

On the Effectiveness of Parameter-Efficient Fine-Tuning

The growing interest in complex decision-making and language modeling problems highlights the importance of sample-efficient learning over very long horizons. This work takes a step in this direction by investigating contextual linear bandits where the current reward depends on at most $s$ prior contexts, up to a time horizon of $h$. In order to avoid polynomial dependence on $h$, we propose new algorithms that leverage sparsity to discover the dependence pattern and arm parameters jointly. We consider both the data-poor ($T<h$) and data-rich  ($T\ge h$) regimes, and derive  respective regret upper bounds $\tilde O(\sqrt{sdT} +\min\{ q, T\})$ and $\tilde O(\sqrt{sdT})$ that are both optimal in $d$ and $T$,  with sparsity $s$, feature dimension $d$,  total time horizon $T$, and $q$ that is adaptive to the reward dependence pattern. 
Complementing upper bounds, we also show that learning over a single trajectory brings inherent challenges: While the dependence pattern and arm parameters form a rank-1 matrix, circulant matrices are not isometric over rank-1 manifolds and sample complexity indeed benefits from the sparse reward structure. Our results necessitate a new analysis to address long-range temporal dependencies across data and avoid polynomial dependence on the reward horizon $h$. Specifically, we utilize connections to the restricted isometry property of circulant matrices formed by dependent sub-Gaussian vectors and establish new guarantees that are also of independent interest.

Stochastic Contextual Bandits with Long Horizon Rewards

We investigate top-m arm identification, a basic problem in bandit theory, in a multi-agent learning model in which agents collaborate to learn an objective function.  We are interested in designing collaborative learning algorithms that achieve maximum speedup (compared to single-agent learning algorithms) using minimum communication cost, as communication is frequently the bottleneck in multi-agent learning.  We give both algorithmic and impossibility results, and conduct a set of experiments to demonstrate the effectiveness of our algorithms. 

Communication-Efficient Collaborative Best Arm Identification

The generalization ability often determines the success of machine learning algorithms in practice. Therefore, it is of great theoretical and practical importance to understand and bound the generalization error of machine learning algorithms. In this paper, we provide the first generalization results of the popular stochastic gradient descent (SGD) algorithm in the distributed asynchronous decentralized setting. Our analysis is based on the uniform stability tool, where stable means that the learned model does not change much in small variations of the training set. Under some mild assumptions, we perform a comprehensive generalizability analysis of the asynchronous decentralized SGD, including generalization error and excess generalization error bounds for the strongly convex, convex, and non-convex cases. Our theoretical results reveal the effects of the learning rate, training data size, training iterations, decentralized communication topology, and asynchronous delay on the generalization performance of the asynchronous decentralized SGD. We also study the optimization error regarding the objective function values and investigate how the initial point affects the excess generalization error. Finally, we conduct extensive experiments on MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets to validate the theoretical findings.

Stability-Based Generalization Analysis of the Asynchronous Decentralized SGD

The study of implicit regularization induced by gradient-based optimization in deep learning is a long-standing pursuit. In the present paper, we characterize the implicit regularization of momentum gradient descent (MGD) in the continuous-time view, so-called momentum gradient flow (MGF).   We show that the components of weight vector are learned for a depth-$N$ linear neural network at  the different evolution rates, and this evolution gap increases with the depth $N$. Firstly, we show that if $N=1$, the evolution gap between the weight components is linear, which is consistent with the performance of ridge. In particular, we establish a tight coupling between MGF and ridge which for least squares regression. In details, we prove that, under the calibration $t=\sqrt{2/\lambda}$, where $t$ is the time  parameter in MGF and $\lambda$ is the tuning parameter in ridge, the risk of MGF is no more than 1.54 times that of ridge and their relative Bayesian risk is hardly different. Secondly,  if the model becomes deeper, i.e. $N\geq2$, the evolution gap becomes more significant, which implies an implicit bias towards sparse solutions. The numerical experiments strongly support our theoretical results.

The Implicit Regularization of Momentum Gradient Descent in Overparametrized Models



Lack of consumers' agricultural literacy and awareness of agritourism operations and underestimating the importance of agricultural heritage preservation against many current and future threats is one of the concerns highlighted in the statewide plan for Agritourism development in Florida. This study is centered on agritourism brand awareness and investigating the pro-conservation behavior of agritourism stakeholders in the State of Florida. Four groups of stakeholders were the participants, including Florida farmers who adopted agritourism (providers), students at the University of Florida as actual and potential visitors (consumers), and Florida extension agents who worked with adopters (developers or heritage holders). Furthermore, academic researchers in three fields of tourism, rural studies, and ecology (focused on the cultural ecosystem services studies) who had published at least one article related to agritourism in the leading peer-reviewed journals were invited to participate as knowledge holders.Pro-conservation behavior was operationalized using the willingness to pay (WTP) concept, which was framed in the contingent valuation technique and grounded in the Integrated Leisure Constraints Model (ILCM). The concept of consumers' pro-conservation behaviors toward agricultural heritage was discussed. The identified significant factors on stakeholders' pro-conservation behavior offer a practical tool for providers, developers, and policymakers in Florida.Brand awareness results showed that the Agritourism brand has all the preferred criteria of a memorable and relevant brand for all four stakeholders, followed by Farm Visit. One critical finding based on the qualitative analysis was that agritourism in Florida needs to be re-branded separately for each stakeholder group.This study has implications for developers and practitioners in the agritourism industry to promote visitor participation.The implications addressing the characteristics of stakeholders' preferences (e.g., brand name and agricultural settings) shed light on segmenting the target groups efficiently and advertising the agritourism services more effectively.

Branding and pro-conservation behavior toward agritourism in Florida, integrative leisure constraints model

Extent, Impacts and Prospects of genetically engineered crops in Africa

Using agronomic parameters to assess quinoa cultivars tolerance under saline irrigation in semi arid climate in Eastern Morocco

Screening of Potetial Sunflower Genotypes in Response to Water Stress at Early Growth stage

Overexpression of GhKTI12 Enhances Salinity and Drought Stress Tolerance in Nicotiana tabacum

Strawberry (Fragaria × ananassa) is the most cultivated horticultural crop among the small berries due to its rich, healthy components. The major barrier in strawberry cultivation is caused by gray mold fungus (Botrytis cinerea Pers.) that causes an extensive loss in production. The predominant use of copper-based fungicides for controlling the gray mold fungus has been toxic to the plants, and fungal strains were developing resistance. Therefore, the present work aimed to evaluate and identify an alternative approach for controlling the gray mold fungus by screening the strawberry germplasm for resistance. Developing new cultivars requires the effective identification of resistant strawberry genotypes; hence, the current study was aimed at the evaluation of strawberry genetic resources under lab conditions by establishing an invitro leaf disc assay. The leaf discs from six different strawberry varieties were screened on 0.6% agar media and infected with actively growing gray mold fungus mycelial discs and incubated in the growth chambers maintained at 25˚C with 80% of humidity for two days under dark and 5 days under 16/8 hours of light/dark conditions. The fungal infection injury rate was measured at 7 days after infection. We identified two highly resistant, two moderate resistant, and two sensitive genotypes among the six varieties screened. The fungal infection was successful on leaf discs and the resistant genotypes showed less necrotic lesions, and sensitive genotypes showed more lesions. This assay provided a simple and rapid method to identify the strawberry genotype's resistance to gray mold fungus and potential donor genotypes.

Evaluation of Strawberry Germplasm for Gray Mold Fugal Resistance

Blueberries (Vaccinium corymbosum L.) are a very important crop because of their unique health benefits. The major hurdle for growing blueberries is the pH in the soil, as they need acidic soil. Hence improving the pH level tolerance in the blueberries provides a viable option for the growers to cultivate in a wide range of soil types and allow them to grow in several regions of the world. With this, our objective of the study was to screen the blueberry germplasm variation for high pH levels and identifying the tolerant lines. We used an in-vitro plant twig assay to screen the population using phosphate buffer solutions containing several pH gradients ranging from 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, and 7.0. In the assay, we used 6 different varieties of blueberries for testing the variation for pH level. Among six varieties, we screened, identified one highly resistant, 4 moderately resistant genotypes, and one sensitive genotype to the high pH. The resistance lines maintained a stay green nature at the high pH level, but the sensitive lines turned to brownish and chlorophyll degradation was observed. Developing new cultivars for high pH tolerance requires the effective identification of tolerant blueberry genotypes; therefore, the current approach was aimed at evaluating blueberry genetic resources under laboratory conditions by establishing an invitro plant twig assay. This assay provided an easy and time-consuming method to identify the blueberry genotypes tolerant to high pH and a better genetic source for developing the pH tolerant varieties.

Underline digital video library

On the Effectiveness of Parameter-Efficient Fine-Tuning

Stochastic Contextual Bandits with Long Horizon Rewards

Communication-Efficient Collaborative Best Arm Identification

Stability-Based Generalization Analysis of the Asynchronous Decentralized SGD

The Implicit Regularization of Momentum Gradient Descent in Overparametrized Models

Branding and pro-conservation behavior toward agritourism in Florida, integrative leisure constraints model

Extent, Impacts and Prospects of genetically engineered crops in Africa

Using agronomic parameters to assess quinoa cultivars tolerance under saline irrigation in semi arid climate in Eastern Morocco

Screening of Potetial Sunflower Genotypes in Response to Water Stress at Early Growth stage

Overexpression of GhKTI12 Enhances Salinity and Drought Stress Tolerance in Nicotiana tabacum

Evaluation of Strawberry Germplasm for Gray Mold Fugal Resistance

Screening of Blueberry Varieties for pH Variation

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES