Huiyi Chen, Jiawei Peng, Dehai Min, Changchang Sun, Kaijie Chen, Yan Yan, Xu Yang, Lu Cheng
Evaluating the robustness of Large Vision-Language Models (LVLMs) is essential for their continued development and responsible deployment in real-world applications. However, existing robustness benchmarks typically focus on hallucination or misleading textual inputs, while largely overlooking the equally critical challenge posed by misleading visual inputs in assessing visual understanding. To fill this important gap, we introduce MVI-Bench, the first comprehensive benchmark specially designed for evaluating how Misleading Visual Inputs undermine the robustness of LVLMs. Grounded in fundamental visual primitives, the design of MVI-Bench centers on three hierarchical levels of misleading visual inputs: Visual Concept, Visual Attribute, and Visual Relationship. Using this taxonomy, we curate six representative categories and compile 1,248 expertly annotated VQA instances. To facilitate fine-grained robustness evaluation, we further introduce MVI-Sensitivity, a novel metric that characterizes LVLM robustness at a granular level. Empirical results across 18 state-of-the-art LVLMs uncover pronounced vulnerabilities to misleading visual inputs, and our in-depth analyses on MVI-Bench provide actionable insights that can guide the development of more reliable and robust LVLMs. The benchmark and codebase can be accessed at https://github.com/chenyil6/MVI-Bench.
Huiyi Chen, Jiawei Peng, Kaihua Tang, Xin Geng, Xu Yang
In-context learning (ICL) enables Large Vision-Language Models (LVLMs) to adapt to new tasks without parameter updates, using a few demonstrations from a large support set. However, selecting informative demonstrations leads to high computational and memory costs. While some methods explore selecting a small and representative coreset in the text classification, evaluating all support set samples remains costly, and discarded samples lead to unnecessary information loss. These methods may also be less effective for image classification due to differences in feature spaces. Given these limitations, we propose Key-based Coreset Optimization (KeCO), a novel framework that leverages untapped data to construct a compact and informative coreset. We introduce visual features as keys within the coreset, which serve as the anchor for identifying samples to be updated through different selection strategies. By leveraging untapped samples from the support set, we update the keys of selected coreset samples, enabling the randomly initialized coreset to evolve into a more informative coreset under low computational cost. Through extensive experiments on coarse-grained and fine-grained image classification benchmarks, we demonstrate that KeCO effectively enhances ICL performance for image classification task, achieving an average improvement of more than 20\%. Notably, we evaluate KeCO under a simulated online scenario, and the strong performance in this scenario highlights the practical value of our framework for resource-constrained real-world scenarios.
Wenxiong Li, Huiyi Chen, Suiyin Chen, Zhiwei Liu
In this paper, a force-based beam finite element model based on a modified higher-order shear deformation theory is proposed for the accurate analysis of functionally graded beams. In the modified higher-order shear deformation theory, the distribution of transverse shear stress across the beam's thickness is obtained from the differential equilibrium equation, and a modified shear stiffness is derived to take the effect of transverse shear stress distribution into consideration. In the proposed beam element model, unlike traditional beam finite elements that regard generalized displacements as unknown fields, the internal forces are considered as the unknown fields, and they are predefined by using the closed-form solutions of the differential equilibrium equations of higher-order shear beam. Then, the generalized displacements are expressed by the internal forces with the introduction of geometric relations and constitutive equations, and the equation system of the beam element is constructed based on the equilibrium conditions at the boundaries and the compatibility condition within the element. Numerical examples underscore the accuracy and efficacy of the proposed higher-order beam element model in the static analysis of functionally graded sandwich beams, particularly in terms of true transverse shear stress distribution.
Liuyi Wang, Zongtao He, Ronghao Dang, Huiyi Chen, Chengju Liu, Qijun Chen
Vision-and-Language Navigation (VLN) has gained significant research interest in recent years due to its potential applications in real-world scenarios. However, existing VLN methods struggle with the issue of spurious associations, resulting in poor generalization with a significant performance gap between seen and unseen environments. In this paper, we tackle this challenge by proposing a unified framework CausalVLN based on the causal learning paradigm to train a robust navigator capable of learning unbiased feature representations. Specifically, we establish reasonable assumptions about confounders for vision and language in VLN using the structured causal model (SCM). Building upon this, we propose an iterative backdoor-based representation learning (IBRL) method that allows for the adaptive and effective intervention on confounders. Furthermore, we introduce the visual and linguistic backdoor causal encoders to enable unbiased feature expression for multi-modalities during training and validation, enhancing the agent's capability to generalize across different environments. Experiments on three VLN datasets (R2R, RxR, and REVERIE) showcase the superiority of our proposed method over previous state-of-the-art approaches. Moreover, detailed visualization analysis demonstrates the effectiveness of CausalVLN in significantly narrowing down the performance gap between seen and unseen environments, underscoring its strong generalization capability.
William W. Symes, Huiyi Chen, Susan E. Minkoff
Iterative inversion of seismic, ultrasonic, and other wave data by local gradient-based optimization of mean-square data prediction error (Full Waveform Inversion or FWI) can fail to converge to useful model estimates if started from an initial model predicting wave arrival times in error by more than half a wavelength (a phenomenon known as cycle skipping). Matched Source Waveform Inversion (MSWI) extends the wave propagation model by a filter that shifts predicted waves to fit observed data. The MSWI objective adds a penalty for deviation of this filter from the identity to the mean-square data misfit . The extension allows the inversion to make large model adjustments while maintaining data fit and so reduces the chances of local optimization iterates stagnating at non-informative model estimates. Theory suggests that MSWI applied to acoustic transmission data with single-arrival wavefronts may produce an estimate of refractive index similar to the result of travel time inversion, but without requiring explicit identification of travel times. Numerical experiments conform to this expectation, in that MSWI applied to single arrival transmission data gives reasonable model estimates in cases where FWI fails. This MSWI model can then be used to jumpstart FWI for further refinement of the model. The addition of moderate amounts of noise (30\%) does not negatively impact MSWI's ability to converge. However, MSWI applied to data with multiple arrivals is no longer theoretically equivalent to travel-time tomography and exhibits the same tendency to cycle-skip as does FWI.
Liuyi Wang, Chengju Liu, Zongtao He, Shu Li, Qingqing Yan, Huiyi Chen, Qijun Chen
Vision-and-language navigation (VLN) is a crucial but challenging cross-modal navigation task. One powerful technique to enhance the generalization performance in VLN is the use of an independent speaker model to provide pseudo instructions for data augmentation. However, current speaker models based on Long-Short Term Memory (LSTM) lack the ability to attend to features relevant at different locations and time steps. To address this, we propose a novel progress-aware spatio-temporal transformer speaker (PASTS) model that uses the transformer as the core of the network. PASTS uses a spatio-temporal encoder to fuse panoramic representations and encode intermediate connections through steps. Besides, to avoid the misalignment problem that could result in incorrect supervision, a speaker progress monitor (SPM) is proposed to enable the model to estimate the progress of instruction generation and facilitate more fine-grained caption results. Additionally, a multifeature dropout (MFD) strategy is introduced to alleviate overfitting. The proposed PASTS is flexible to be combined with existing VLN models. The experimental results demonstrate that PASTS outperforms all existing speaker models and successfully improves the performance of previous VLN models, achieving state-of-the-art performance on the standard Room-to-Room (R2R) dataset.
Dohyun Ahn, Huiyi Chen, Lewen Zheng
Standard rare-event simulation techniques require exact distributional specifications, which limits their effectiveness in the presence of distributional uncertainty. To address this, we develop a novel framework for estimating rare-event probabilities subject to such distributional model risk. Specifically, we focus on computing worst-case rare-event probabilities, defined as a distributionally robust bound against a Wasserstein ambiguity set centered at a specific nominal distribution. By exploiting a dual characterization of this bound, we propose Distributionally Robust Importance Sampling (DRIS), a computationally tractable methodology designed to substantially reduce the variance associated with estimating the dual components. The proposed method is simple to implement and requires low sampling costs. Most importantly, it achieves vanishing relative error, the strongest efficiency guarantee that is notoriously difficult to establish in rare-event simulation. Our numerical studies confirm the superior performance of DRIS against existing benchmarks.
Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, Xu Yang
Inspired by the success of Large Language Models in dealing with new tasks via In-Context Learning (ICL) in NLP, researchers have also developed Large Vision-Language Models (LVLMs) with ICL capabilities. However, when implementing ICL using these LVLMs, researchers usually resort to the simplest way like random sampling to configure the in-context sequence, thus leading to sub-optimal results. To enhance the ICL performance, in this study, we use Visual Question Answering (VQA) as case study to explore diverse in-context configurations to find the powerful ones. Additionally, through observing the changes of the LVLM outputs by altering the in-context sequence, we gain insights into the inner properties of LVLMs, improving our understanding of them. Specifically, to explore in-context configurations, we design diverse retrieval methods and employ different strategies to manipulate the retrieved demonstrations. Through exhaustive experiments on three VQA datasets: VQAv2, VizWiz, and OK-VQA, we uncover three important inner properties of the applied LVLM and demonstrate which strategies can consistently improve the ICL VQA performance. Our code is provided in: https://github.com/GaryJiajia/OFv2_ICL_VQA.
William W. Symes, Huiyi Chen, Susan E. Minkoff
Study of a simple single-trace transmission example shows how an extended source formulation of full-waveform inversion can produce an optimization problem without spurious local minima ("cycle skipping"), hence efficiently solvable via Newton-like local optimization methods. The data consist of a single trace extracted from a causal pressure field, propagating in a homogeneous fluid according to linear acoustics, and recorded at a given distance from a transient point energy source. The source intensity ("wavelet") is presumed quasi-impulsive, with zero energy for time lags greater than a specified maximum lag. The inverse problem is: from the recorded trace, recover both the sound velocity or slowness and source wavelet with specified support, so that the data is fit with prescribed RMS relative error. The least-squares objective function has multiple large residual minimizers. The extended inverse problem permits source energy to spread in time, and replaces the maximum lag constraint by a weighted quadratic penalty. A companion paper shows that for proper choice of weight operator, any stationary point of the extended objective produces a good approximation of the global minimizer of the least squares objective, with slowness error bounded by a multiple of the maximum lag and the assumed noise level. This paper summarizes the theory developed in the companion paper and presents numerical experiments demonstrating the accuracy of the predictions in concrete instances. We also show how to dynamically adjust the penalty scale during iterative optimization to improve the accuracy of the slowness estimate.